In the family that I grew up in ILs were family too. It made it really hard for me when later I had IL issues as it had never occurred to me that I might. We have my Aunt's two BILs over for every family event b/c her DH's FOO is very small and they have no one else. I always referred to them as my 'Uncles' and it never occurred to me that they weren't really family. My cousin had her DH leave her a few years ago for another woman. It was horrible. She and her new DH (a wonderful loving man) go to the movies with her children, his children, her former MIL and FIL, her former SIL and BIL and their children. When her former SIL needed someone to take her daughter on a last minute trip cross country due to some unexpected emergency she called my cousin and my cousin was happy to do it. My grandmother divorced her husband in the 60's. She is spending today with his sister at the nursing home as it is her former SIL's birthday. My grandfather on the other side was jokingly referred to as the 'favorite son' of his MIL. Yes, we did have some IL issues along the way (grandma had it rough!), but for the most part family is family in my FOO. I think it all depends on where you come from as to what seems 'normal'.
No, you don't have to spend time with people if you don't want to. But in this case, what could it hurt? You don't have to make plans with them every night, but these are the ILs to your son, the parents to your DIL and the other grandparents to your GK. One day you may very well appreciate taking the time out to get to know them.