Hah! Glad I'm not the only one who thought so.
I think it's his face that made my alarm bells ring. He looks too plain. Think Aizen in Bleach.
Just the perfectly innocent face to suddenly transform with a villain reveal. So I think you're on to something. A plain-faced character being part of an "inner circle" of otherwise flashy people? Red flag.
He will start removing people's glasses and start breaking them to show off or something...
But other than that, I think that he is envious of the current holder of All-Might's power, One for All. As you said, he is very plain looking, meaning that he too is just one of the Quirkless people in the world, making One for All his only chance to not only be one of the cool kids, but be the cool kid.
Or so I think, at the very least.