My first instinct was the "Stick to your guns." and "don't go." response.
Then, I realized that the "if you don't go and he does, then he wins, because he controls her life." response is correct.
I say go, but leave the kids. It lets the guy know that you aren't going to chicken out just because he is there, but it lets your mother know that this relationship has consequences (no grandkids around). Your mom will blame you for not going, but not seeing her grandkids is the REAL and ONLY punishment she will understand.
This way you both stick to your guns and stick by your mom.