The music mogul, 54, is returning to judge the show this year, alongside Cheryl Fernandez-Versini, 31, 61-year-old Louis Walsh and new judge Mel B, 39.
PICTURE: TULISA MAKES X FACTOR RETURN AT JUDGE'S HOUSES IN BERMUDA - CLICK TO SEE
But speaking of last year's show, particularly Sam Bailey's win, Simon said it was predictable'.
Instead he admitted that he and Cheryl would have preferred to see Tamera Foster make it to the final.
Simon told the Daily Mail: "Cheryl and I said if we'd been mentoring Tamera, we would've got her into the final. That would have made it more exciting.
READ: TULISA CONFIRMS SHE IS GETTING 'BACK TO BUSINESS' AND RECORDING NEW MUSIC
"Last year was OK, but it wasn't my favourite year."
Of former prison warden Sam, Simon said: "If was predictable right from the get-go who was going to win. Once the live shows started, Sam was in a different league to everybody else."
After their fall-out over her unceremonious sacking from the X Factor USA, Simon said that he and Cheryl are getting on just fine.
He said: "She's more confident, and though she probably won't admit it, she enjoys working with me. I love working with her."
READ: LOUIS WALSH ON TULISA'S DRUGS TRIAL – "IT WAS COMPLETE ENTRAPMENT, I HOPE HE GOES TO JAIL"
One person Cheryl does not enjoy working with is Louis Walsh. After she admitted that would have preferred him not to return as a judge this year, there has been rumours of a 'toxic feud' between the pair since the auditions started.
Of their rumoured feud, Cheryl said coyly: "We've never really liked each other."