This is the 21st century. We have the technology and digital watches. Wouldn't be at all hard to have the ref's watch linked to the scoreboard.
I just think there is far too much for the referees to already contend with what with players surrounding them, dissent, being right up in their faces...