Of course not but surely you have to define neutral? Just because they cannot place you as a Brummie, Geordie, Scouser etc. doesn't mean you have a neutral accent.
It just means your accent isn't as strong as others, which isn't a bad thing, but just because they cannot place it doesn't make it...