Inside the Moment Prince William and Prince Harry's Personalities Flipped as Kids

Prince William and Prince Harry knew early on that they were heading down very different paths.

The fractured marriage of their parents Prince Charles and Princess Diana — stressed by the heir's ongoing affair with current wife, Camilla — had an impact on William and Harry from a young age, as described by historian Robert Lacey's new book Battle of Brothers: The Inside Story of a Family in Tumult, excerpted in this week's issue of PEOPLE.

"From the ages of 4 and 2, William and Harry grew up in the care of two parents who were not sharing the same bed, who were more inclined to talk to the press than to each other, and who were also engaging in patterns of systematic deceit," Lacey writes.

While a "once-rambunctious" Prince William grew "more reflective" with a "noticeably quiet character — definitely introverted" as he headed off to boarding school, Prince Harry's personality was also changing.

"Ken Wharfe, bodyguard to Diana and the two boys in the late 1980s, recalls a telling incident when the family were travelling from London to Highgrove for the weekend," Lacey says in the book. "According to Wharfe, Harry and his brother got embroiled in an argument in the back seat of the car, with their nanny vainly seeking to referee the dispute. 'You’ll be king one day,' said the 4-year-old Harry. 'I won’t. So I can do what I want.' "

A stunned Princess Diana replied with, "Where the hell did he get that from?"

"Both brothers have been damaged by their upbringings; both have reacted by finding different solutions," Lacey, who is also the historical consultant for Netflix’s The Crown, tells PEOPLE. "There is so much pain and trauma in this story, going right back to the beginning."

Lacey says the brothers "were raised to be close and protective of each other," but tensions grew as William was prepped for his role as future monarch while Harry was left without a defined role of his own.

Can't get enough of PEOPLE's Royals coverage? Sign up for our free Royals newsletter to get the latest updates on Kate Middleton, Meghan Markle and more!

Lacey adds the brothers — now residing in different countries after Prince Harry and Meghan Markle relocated to California after stepping down from their senior roles within the royal family — face "potential tragedy" if they can't reconcile.

"This compares with Edward VIII’s abdication in 1936 and the death of Princess Diana in 1997," says Lacey. "It’s of that scale. And it's a challenge that’s yet to be resolved."