Former Secret Service Agent Describes JFK's Unbearable Grief After Losing Son Patrick When He Was Just 2 Days Old

In the three-and-a-half years he was Jackie Kennedy’s Secret Service agent, Clint Hill grew very close to the Kennedy family. The period with them, he says, was his favorite.

Hill, whose career in the Secret Service spanned 17 years and five presidents – from Eisenhower to Ford – says he saw the human side of the men who made history.

“I saw them when they were happy and successful and I saw them when they were down,” Hill tells PEOPLE. “I saw them go through extremely sad times.” In his new memoir Five Presidents Hill, 84, reveals the devastating story of the death of JFK’s son, Patrick Bouvier Kennedy, who died when he was two days old.

Born five and a half weeks prematurely on August 7, 1963, Patrick weighed just four pounds, ten and a half ounces. According to Hill, he “had a condition known as hyaline membrane disease, a common affliction in premature babies due to incomplete lung development.”

Soon after the baby’s birth, JFK arrived at the Otis Air Force Base hospital from Washington, D.C., to see Jackie and the baby. When he saw Hill, he asked him to find the base chaplain, telling him: “We need to baptize the baby right away.”

After Patrick was moved to Boston’s Children Hospital, JFK began shuttling back and forth between Boston and Otis Air Ford Base, where his wife remained.

But two days later, on August 9, Hill got a phone call at 4:15 in the morning from Jerry Behn, the special agent in charge of the White House detail, who told him baby Patrick had died.

When Hill saw JFK upon his return to Otis Air Force Base, he writes: “The president looked like he had been to hell and back with swollen eyes and sorrow etched all over his face.” As Hill stood outside their private room, he writes, “I could hear the muffled sobs, as they grieved the loss of their son.”

Later on, Special Agent Behn told Hill the doctors had called JFK in the middle of the night, knowing Patrick’s death was imminent. According to Hill, when JFK arrived at the hospital to say goodbye, he “was able to hold his son in his arms, for the first and last time.”

Jackie, recovering from the Cesarean operation, was too weak to attend the funeral the next day, August 10, in Boston. Describing the scene at Brookline Cemetery, Hill writes how his fellow agents had to fight back tears as “President Kennedy’s shoulders heaved up and down at the sight of the tiny white coffin being placed into the ground of the Kennedy family plot.”

The months that followed were somber and full of grief. “They had [a nursery] decorated and painted,” says Hill. “After Patrick died, they arranged to have it completely torn down and brought back to the way it was before.”

“They looked so forward to this birth, and to the new addition to the family and to have it taken away like that, was so terrible,” he says. “It was very difficult for both of them.”

As for their young children, John and Caroline, Hill says, “They understood there was sadness with their mother. The President brought the children to see her, to try and cheer her up somewhat and they helped. They didn’t discuss the baby much. And John was too young to understand what was going on.”

In the months that followed, Hill noticed a change in president and first lady’s relationship.

“Prior to that time, the President and Mrs. Kennedy didn’t display emotions towards each other in public,” he says. “We would see it occasionally in private but in public it was never seen. But after Patrick died, they held hands and they embraced in public it became obvious that their whole attitude had changed.”