It isn't a common misconception, if a referee gets the "big calls" correct nobody talks about them
BTW, the first person I ever heard make the comment about not being noticed was Neil Midgley, though I guess he didn't referee at the same level you did