I'd still have her come to your house. Then you are on your turf. He could come if he wants... I suspect he wouldn't... and if he chooses to not let her come to your house, then he looks like the bad guy that he is.
EDIT: and while it may be an inconvenience for you, you could always offer to go get her, and take her back home if necessary.