I don't even like Cassel, but man, that's kind of a nutty argument. I mean, they won 11 games, but 9 of them don't count because they came against teams .500 or below? And all losses go on Cassel regardless?
Let's take a look.
One of the losses against a winning team came against the Dolphins, Cassel's second start. They only scored 13, and lost by 25. They avenged that loss later in the year by scoring 48 and winning by 20.
San Diego beat them by 20 week 6 (start #4), 30-10.
(Week 7 they beat Denver 41-7, but that doesn't count, since the

went .500).
They also split with the Jets, and the game that they lost the Patriots offense put up 31 points.
3 point loss on the road week 9 to the Colts, who ended the season 12-4. Only scored 15 in that one.
The Steelers beat them badly week 13, 33-10.
(The Patriots beat the Cardinals 47-7 in week 16 - that's one of their 2 wins).
I didn't realize it until I looked at the schedule, but apparently they didn't lose to a single team with a losing record. SD was 8-8. The Jets were 9-7. The Dolphins were 11-5. The Colts and Steelers were 12-4. That's 3 division winners, a 12-4 wild card team and a division opponent.