The X Factor host, 41, has been married to Dee for two years but they have been together for 10.
Speaking to Fabulous, Dermot said: “I definitely want kids, but I’ve got a very busy wife with a very busy life.
“It’s not fair for me to say, ‘I want kids now’. I do want kids with my wife, but I want them when we both think it’s the right thing to happen.”
Dermot admitted he and Dee are so busy that they even forgot to remember their second wedding anniversary this year.
He said: “Circumstances and life takes over and, before you know it, you haven’t seen each other properly for a week or two.”
But rumours that Dermot’s schedule is about to become a lot less hectic are false: it was thought that he might not be returning to The X Factor this year.
He said: “Do you know what? If they don’t want me to do it next year, I’ve been blessed to have eight great years.
“It’s pretty much eight full-on months of the year – there’s no way I could do this if I didn’t like it.”
But when reports began to emerge this week suggesting that he’d quit the show, he tweeted: “It ain’t so [sic]”