He'll be leaving at the end of this year
His comments follow reports Gary might be quitting to concentrate on a new Take That album and tour in 2014.
Dermot let the news slip at the premiere of One Direction: This Is Us on Tuesday when he was asked about Simon Cowell’s possible return to the panel.
He told reporters: ‘If Simon comes back, the dynamic would be very different. Gary wouldn't be there - he'll be leaving at the end of this year.
‘Simon is a great judge so we'd welcome him back, but Gary's been such a great head judge as well.’
Simon was rumoured to be making a comeback after ITV bosses reportedly pleaded with him as part of contract negotiations to keep the show at the channel.
Gary's departure would mean at least three spots on the judging panel would need to be filled for series 11 – both Sharon Osbourne and Louis Walsh have confirmed they will leave at the end of the new series.
However, the Louis revealed earlier this week that he would reconsider if Simon was to return as a judge.
He said: ‘I've been doing it ten years. It's an awful long time to be on the show. I don't want to be pushed. I'd prefer to leave than be pushed. To leave on my terms.
‘Now if Cowell came back, I might come back. The two of us, we would not stop laughing. If Simon called me up and said... but, oh well. Let's see what happens.’