You need to have ArcGIS for Desktop, ArcGIS for server, and ArcSDE installed, and you have created a geodatabase
I tried to add it with the admin site, always unsuccessful, and then add it successfully with Arccatelog. The process of registering a managed database with Arccatelog is documented here
1. Add a database connection
Click "Add Database Connection" On the left, open the dialog box and fill in the content according to your environment.
Click OK to add it.
2. Add ArcGIS Server to Arccatelog
, double-click ADD ArcGIS Server to open the dialog box administer GIS server
Click Next to open the dialog box and fill in the contents according to your own configuration
Click Finish to do it.
3. Registering a Managed database
Right-click "Server Properties" on the "ArcGIS on localhost:6080 (admin)"
The following dialog box appears, select the DataSource tab
Click the drop-down arrow next to the plus sign in the upper-right corner, note the drop-down arrow, and select Register database as ArcGIS Server Managed database
Fill in a unique name, then click Import
Note the top drop-down list select "Database Connections" and then choose the connection file added in the first step, click "Select"
Last Click OK.