A lot of people do unfortunately, so I believe you are wrong in that assumption.
If you can't afford to go, you can't afford to go and that is the case for many, many fans.
I know of a lot of people who can't afford to go now, let alone if the match day cost was to increase even further.