To some degree casual games do cut into the sales of PC/Console games. Even though I'm not going to cross shop iTetris with Starcraft 2, the sales of one will alter sales of the other.
For example, I love tower defense flash games (I started with Warcraft 3 mods funnily enough). In the last few years I've invested more hours into gaming that in previous years, but because I've been playing so many flash games I'd estimate that I've actually cut down my 'real' gaming time by 20% or so.
The problem is that most days I only have maybe an hour or 2 to play a game. Previously I'd be playing some online FPS/RTS, but now often I'll just play play a couple of flash games. To play a single player game really requires at least a few hours of commitment if you really want to get into it. The way most people's lives are nowadays they just don't have a few spare hours that often. I'm lucky if I can get more than 3-4 hours in over a whole weekend.