Well the obvious workaround would be to set up your voice settings to friends only in your xbox live privacy settings.
“I get a kick out of being an outsider constantly. It allows me to be creative. I don't like anything in the mainstream and they don't like me.”
-Bill Hicks, another dead hero