Nicole has been a judge on the hit ITV show since 2012, and won the show during her first year as a judge with contestant Jahmeme Douglas.
But according to a source, the 35-year-old singer is not set to return to the show in 2014.
They said: ‘It breaks her heart, but Nicole is sad not to be able to go back to the show this year.’
‘She has agonised over the decision for months.’
The reports come following an announcement from Simon Cowell saying that the mogul is returning to the UK X Factor.
There have also been rumours that former judge Cheryl Cole has been in talks to return to the judging panel this year, though singer Rita Ora is also allegedly in the running to fill Nicole’s shoes on the judging panel.
Simon is reportedly keen to boost ratings after the talent show saw a sharp fall in numbers of people tuning into last year’s series.
According to 3am, a friend of Simons said about Nicole: ‘She will always be part of the X Factor family.’