I'm in a similar situation and finally came to the same realization as you but it wasn't easy getting to this point. I was so heartbroken, first, when I couldn't see DS on a regular basis and then my GC, we don't have GP's equal status, but it's ok now, DH and I have a lot of catching up to do, when we see them we make the best of it. We have two other children so we'll have other opportunities...
Hugs.