So quick back story... my bf and I have only been dating for a few months. As most relationships it started out really good and probably to fast, but I really like him and we get along great and can see us going for a long time with a future. I have 2 kids and so does he with full custody (4 kids total under 10) and we've already met the kids and we all get along wonderfully. Where I'm getting confused is that for the past couple weeks I've been having to start most conversations and what used to seem like i was welcome to come over to his place anytime, I'm having to ask for myself to come over (so not getting invited by him hardly anymore). Or his family members he lives with ask me to come over and hang out more then he does. I don't always say yes to his family members cause I don't want to seem forceful and in his face. We had a talk last week and I expressed my concern about our communication not being as open and how I felt, but he told me we're fine, and that he really loves me and is sorry, and he's just exhausted (his hours from work recently increased and he works without much AC and having kids full time of course). He does have family that lives with him so they help a lot at the house. We used to see each other 3 times a week at least, and now it's maybe once every week-1.5 weeks.
I haven't heard from him for a couple days since after the talk, and I'm kind of upset because I've had the last few days off work and he did too. A month ago we had planned to hang out and spend time together during those days even with his kids because we don't get much time with our schedules. I texted the first night asking if it was okay for me to come over, and he nicely mentioned that it wasn't good cause he was spending quality time with a family member. I do trust him and I have been nothing but supportive cause he's been really stressed lately, but he hasn't called or messaged me at all since and I feel like I'm being ignored. Honestly I'm getting kind of mad cause i feel like I'm doing all of the work and hate feeling ignored by someone who loves me, or being taken for granted. I hate playing games, but should i just wait til he calls/text next to bring it up or wait a couple more days til I call? We usually talk everyday and now it's on day 3

I also don't want to be needy or be expecting to much cause we're comfortable with each other and know we're only exclusive. Any advice or ideas would be greatly appreciated!