During Sunday's episode of the ITV talent show, 21-year-old Josh Daniel's performed emotional Labrinth hit Jealous and dedicated it to his best friend Thomas Woolley, who passed away in 2013.
But Josh's audition was filmed just days after 55-year-old Simon's mother Julie died in July, and brought everything back for the music mogul.
Speaking to The Sun, Simon said: "Of course, it brought it all back. It is a very difficult thing to know what to do. The contestant didn't know the situation. He wasn't doing it to make me upset. He sang the song for his own reasons.
"It was more important that he did a good audition. But it was very raw and it did take me back a moment, but I think in quite a healthy way. Being upset is a human thing."
Julie, 89, passed away just days before this year's auditions kicked off, and Simon previously admitted that he found it hard getting on with things.
He said: "I'm not going to lie, there were times it was incredibly difficult. But then there were other times I was able to switch my mind off and concentrate on why was happening and the contestants."
Following Josh's beautiful rendition of Jealous, a visibly upset Simon walked off set – leaving Cheryl Fernandez-Versini concerned for her friend.
She told their fellow judge Nick Grimshaw: "I don't know what to do.What do I do? Do I go? Do I stay?"
The X Factor continues tonight at 8pm on ITV.