Maybe those followers are from the time when he was with the EDL. Walking away from the EDL is one thing hoping that no one there follows you, that's another.
Also the followers are probably from other far right groups like: Combat 18, BNP, National Front and others.