Yeah, the sticky part to all of this is what happens when you don't show up? Does that create some 'issues' that become amplified when Mr. Good Guy decides to drink a few that night? You never know how things can escalate...
If you're there - you can always use your best judgement to leave and/or have the children leave the room/house while you address the situation.
Sounds like your mom is in a bad place and feels that she needs this guy's company because ... well, she doesn't like being alone and while this guy is likely 100% bad news, she tries to deal with it.
Either way, it's nearly a lose/lose situation for you. Do you deprive your kids time at grandmas? Do you leave your mom high and dry at an emotional time of year? This could make her tie to this guy even stronger.
I dunno... I do feel bad for you and your mom though. I think you need to suck it up and do what's right even if you don't like it or think it's best. There's a time and place to stand your ground, I'm not certain this is it.