I think you're right and it's beyond doubt that this issue transcends party politics, especially in the main two parties.
I think it's also beyond doubt, no matter what people say, that within the leave vote there are various factions with a different idea as to what Brexit should mean.