James Turner Street was the setting for the show that made her famous but, according to her agent Barry Tomes, the "nut-jobs and idiots" made her decide to leave.
READ: CBB HOUSEMATES 'OUTRAGED' AFTER GARY BUSEY 'FLASHED' DEE KELLY
Dee, who placed fifth on the Celebrity Big Brother final on Friday night, is mum to seven-year-old son Gerrard and daughter Caitlin, 16, who are both of mixed heritage.
Barry claimed that Dee's children have become the subjects of abuse, even shockingly being labelled "liquorice allsorts" by online trolls.
Dee's boyfriend, who is black, was also targeted by the trolls.
Despite the fact she's made the decision to leave James Turner Street, Dee is staying in Birmingham and keeping the house, so Gerrard and Caitlin can continue to go to the same school.
Barry added: "It just got a bit silly. It's just a small proportion of nut-jobs and idiots – 90 per cent of people love her."
Meanwhile, Dee has slammed her former CBB housemate and eventual winner Gary Busey. She's accused the 70-year-old actor of being a 'serial perv' and insisted he flashed all of the women in the house.
READ: WHITE DEE SLAMMED BY FRIENDS – "SHE'S STABBED IN THE HEART"
She the Daily Star: "He has exposed himself on purpose to a lot of other young girls and some of the lads in the house. To me that's indecent exposure.
"If my daughter was in the house, I personally would have kicked that door down, run in, either dragged her out or beat him up. It's absolutely disgusting."