The 54-year-old claimed that Cheryl, 31, and former Spice Girl Mel, 39, aren't interested in finding the next big thing, as this will mean more competition in the industry.
PICTURE: TULISA MAKES X FACTOR RETURN AT JUDGE'S HOUSES IN BERMUDA - CLICK TO SEE
Speaking on a US chat show, Simon said: "Singers don't want want to find stars. It's not in their DNA. They only think about themselves. That is the nature of being a solo artist."
But Simon is confident nothing will cloud his judgement, adding: "I own a record label. So the only think I'm interested in is finding a star.
READ: TULISA CONFIRMS SHE IS GETTING 'BACK TO BUSINESS' AND RECORDING NEW MUSIC
"You put a singer in my seat and somebody 10 years younger than her comes on and is better, she does not want her to do well. It's a fact."
We think former Girls Aloud singer Cheryl may disagree with Simon on this one.
Remember in 2008 when she mentored the amazing Alexandra Burke to victory?
Simon also said that he doesn't understand people who become singers and then complain about press intrusion.
READ: LOUIS WALSH ON TULISA'S DRUGS TRIAL – "IT WAS COMPLETE ENTRAPMENT, I HOPE HE GOES TO JAIL"
He said: "What I don't get are people on TV, or making records, who bleat about the paparazzi. I always say, 'Do a different job'.
"You put yourself in that position. The people are buying your records, watching your shows, the least you could do is shake their hands."