The 17-year-old star was pictured last week in London with the rapper, 24, before they headed to France and Greece this week.
KIM AND KHLOE KARDASHIAN HAVEN'T SPOKEN FOR SIX WEEKS: 'THEY'RE PROBLEMS AREN'T FIXABLE'
But Bruce, Kim, 34, Kourtney, 35, and 29-year-old Khloe are said to be upset as Kylie is still so young.
A source told Radar Online; "They blame Kris [Jenner] because Kylie is only 17-years-old and Kris needs to have more authority over her.
"Kim, Khloe and Kourtney are all extremely angry because Kylie is their baby and they think she should not be globetrotting Europe unsupervised.
KYLIE JENNER SPARKS 'BOOB JOB' RUMOURS WITH REVEALING NEW SELFIE
"She told Kris that she is going to help with her music career and learn the ins-and-outs, but they simply don't buy it."
But it's former Olympian Bruce who is the most angry. Following his split from Kris, he feels like he has no say over what his youngest daughter does.
KHLOE KARDASHIAN'S 'REAL FATHER IS LIONEL RICHIE': STRIKING RESEMBLANCE TO SOFIA RICHIE
The source continued: "Bruce told Kris that Kylie is still his child and, even though he does not live there, he must be informed and give his approval before allowing her to fly around the globe.
"When she is 18 she can do what she wants, but not until then."
Kylie was recently featured on Time magazine's most influential teen list, alongside supermodel sister Kendall.
Do you think Kylie should be allowed to travel unsupervised?