Ken, 71, was thrown off the Channel 5 show for using "unacceptable language" including repeated use of the word "negro".
In his first TV appearance since leaving the house, he told ITV's Loose Women that he had used "an outdated expression", but denied he was racist.
"I apologise unreservedly to everybody who watched that programme," he said.
"I apologise if I disappointed you by saying what I said.
"It was banter in an odd situation, but I understand entirely I've upset a lot of people from everywhere in the world... I am not a racist. Never have been."
While on the show Morley used the n-word in a conversation with fellow contestant, singer, Alexander O'Neal and later in an anecdote about former boxer Frank Bruno.
More than 250 people complained to media regulator Ofcom about Morley, who also angered viewers with his comments about his female housemates.
On Monday, a Channel 5 spokesman said: "Ken Morley has been removed from the Celebrity Big Brother house for using unacceptable and offensive language following his formal warning."
Morley was the second contestant to be ejected by producers.