There could be 50 different possibilities for that type of behavior. And I wouldn't rule out your ISP just yet either.
How are you configuring your router? Is it set for multiple speeds? i.e. 802.11b+g+n? Do you have uPnP turned on? Is you connection secured? Have you done any line tests(both through router and straight from the modem)?
__________________
|