Normally the procedure for creating a new family is something like this: You go into Catalog manager>You Go into a particular catalog in which you want create the family>click family in that catalog>You see all the families listed in the right navigation pane>right click on any family and click on new>a New/modify dialogue box will open up>enter the family name>then the you would be able to assign parent and child for that family.
Note: unless you enter the name for the new family, the roles (i.e parent and child) would be grayed out.
Please make sure you are doing the above steps correctly.