The network groups are contributed to the RelationshipService. Here is the default contribution:

<extension target="org.nuxeo.ecm.social.relationship.service.RelationshipService"
    point="relationshipKinds">
     <kind group="circle" name="friends" />
     <kind group="circle" name="coworkers" />
</extension>

You need to use the group “circle” so that your newly contributed groups will be displayed in the pop-up when adding a new relation.

If you want to add a new one, and disable the “friends” group, you can do the following:

<require>org.nuxeo.ecm.social.user.relationship.contrib</require>

<extension target="org.nuxeo.ecm.social.relationship.service.RelationshipService"
    point="relationshipKinds">
     <kind group="circle" name="friends" enabled="false" />
     <kind group="circle" name="my new group" />
</extension>

Don't forget the require element so that the “friends” group will be overridden.