For me it helps to think of a segment as a search that allows you to save the results. Although you can run a segment based on group membership, you can also add parameters that transcend group membership such as contact data or response history. In other words, segments aren't a part of your groups but their own seperate entity. When you delete a segment, you're just deleting those search parameters that you've previously saved and not the contacts from their group.
I hope that answers your question, please don't hesitate to let me know if I've missed the mark and I'd be happy to elaborate further for you :)