Honestly, I am mystified.
Yeah, I know a lot of people are talking about carrying a small balance, and having a higher credit limit would be good, etc. But that should be the difference between "only" having a low to mid 700 score, and being among the elite. Those are the things you do to get that last little bit in your score. How the hell are you mid-600's? When I got my mortgage a few years ago, my situation was worse than yours and I had high-600's.
I know you said you had a bank card, but is that being treated as a regularly-reported credit card? I mean, if you had ZERO revolving credit, then maybe that'll explain it. Get a credit card, use it once in a blue moon, and that should fix it.
Other than that, are you sure you don't have a ding or late-pay reported on there somewhere? mid-600's is awfully low given the info in the OP.
<ptp> how many emo kids does it take to change a lightbulb?
<Willy> HOW MANY?!
<ptp> none they just sit in the dark and cry