Many big names have been linked the show recently, including Rita Ora and even Kim Kardashian, but new host Olly, 31, has reminded us that the panel wasn't always made up of big names.
He told OK! Online: "You've got to remember Louis, Simon and Sharon back in the day when they were on the panel were't celebrities at the time so anyone can do it.
"Well not anyone could do it, you've got to be a strong personality with a good opinion – who knows it could be someone completely unknown."
Just like the rest of us though, Olly is being kept in the dark over who will make the cut. But the singer does know who he would like on the show.
He continued: "I tried to convince Robbie Williams to do it. I thought he would be amazing but it was a no go for him – whoever does it will be great."
Olly previously hosted the Xtra Factor for two years, and it was confirmed in April that he and his former co-host Caroline Flack, 35, were taking over from Dermot O'Leary as presenters of the main show.
Speaking about the job, Olly said: "I'm buzzing! I've been working really hard focusing on the music and touring so it will be a nice end to the year hosting and presenting."