Olly Murs, 30, is heavily rumoured to be replacing Dermot, 41, as the main host of the show and longest-serving judge Louis Walsh, 62, could be next out.
According to show insiders, Robbie Williams, 41, will fill Louis seat on the judging panel.
A source told the Daily Star: "It is not 100 per cent finalised yet but Simon knows he needs to make the show current and credible and Louis no longer ticks that box.
"It is hard for him personally as they are great friends. But the decision is all but done. Everyone in power has agreed it is time Louis went."
But it's not just Louis who is in trouble: the insider said that reinstated judge Cheryl Fernandez-Versini, 31, and 39-year-old Mel B could also face the chop.
"Simon will be coming back but Cheryl and Mel will have to fight for their seats," the source added.
Meanwhile a spokesperson for the ITV show has confirmed that it was Dermot's decision to quit the show.
They told the MailOnline: "Dermot chose to step down and was not axed."
Who would you like to see on the judging panel?