I don’t know what the truth is, but judging by his more recent get-up on the Last Shadow Puppets videos, I suspect the accusation could be correct.
However, controversy or otherwise, I think the discussion only draws more attention to the Mod style as people go investigating to try to find out the truth! In a few of these videos from his second album, he's rocking it. He even has a collar bar and (possibly, though can't be sure) a knitted tie.
Someone somewhere is a Mod - even if it is just the costumes designer!