The Facebook photo, taken in a Connecticut gun store of six-month-old Genevieve Duffy, sparked outrage from the local community - who are still grieving from the school massacre which saw 20 children and 6 teachers shot dead in December 2012.
Woodbridge Firearms, who originally said it was proud of the photo, has now deleted the image from its Facebook account.
However, the child's father, Christopher Duffy, remains unapologetic about the matter.
'I don't force their kids to do anything. My hand was on it, the bolt was open, it was safety checked.'
He added that the photograph wasn't wide enough to show that his daughter's hands were placed safely on the firearm.
Duffy was searching for a new rifle in the store when his daughter latched onto the gun and the picture was taken.
And despite many local people feeling the photo was insensitive to a community who have suffered so much, others in the gun store see it as no big deal.
Tom Wayland, who was browsing in Woodbridge Firearms said: 'It's nothing, don't know why that would bother people to be honest with you.'
In April, it was announced the gun store that sold Adam Lanza, the Sandy Hook shooter, his gun, had lost its licence to sell firearms.
The people of Sandy Hook have been campaigning ruthlessly for tighter gun controls, after the tragedy that devastated their community.
Adam Lanza walked into the school and killed 26 people, before turning the gun on himself.