Olly, 31, and model Francesca broke up last month after three years of dating, with the singer suggesting she struggled to cope with his intense female fans.
It was reported last week that Olly had moved on with Diana, 24, but he’s hit back.
Speaking to the Mirror, the X Factor host said: “It’s so ridiculous. It’s a load of rubbish!
“Me and Diana are friends, we’ve known each other for a long time. We laughed about it.
“What? We’re casually dating? No, we’re not. We find it funny.”
In his new book On The Road, Olly revealed how his passionate Murs Army caused problems for him and 26-year-old Francesca.
He said: “A lot of my fans are girls, so it can cause problems, obviously. Sometimes I’ll ask Francesca to stay in the people carrier when a crowd comes around.
“From her prospective she’s thinking, ‘Why am I a secret? Why are you putting me in a car?’ And I say, ‘You can come with me if you want’, but if someone comes up and asks for a picture and says, ‘Olly I really fancy you, you should dump your girlfriend. You should get rid of her and go out with me’. It’s not nice.”