Honestly, I am mystified.
Yeah, I know a lot of people are talking about carrying a small balance, and having a higher credit limit would be good, etc. But that should be the difference between "only" having a low to mid 700 score, and being among the elite. Those are the things you do to get that last little bit in your score. How the hell are you mid-600's? When I got my mortgage a few years ago, my situation was worse than yours and I had high-600's.
I know you said you had a bank card, but is that being treated as a regularly-reported credit card? I mean, if you had ZERO revolving credit, then maybe that'll explain it. Get a credit card, use it once in a blue moon, and that should fix it.
Other than that, are you sure you don't have a ding or late-pay reported on there somewhere? mid-600's is awfully low given the info in the OP.
<WarMoose> Think about how stupid the average person is. Now realize that half of them are dumber than that.
<Chunda> Why half?