Photo: Chris Jackson - WPA Pool/Getty

WINDSOR, ENGLAND - SEPTEMBER 10: Prince Harry, Duke of Sussex and Meghan, Duchess of Sussex speak with members of the public at Windsor Castle on September 10, 2022 in Windsor, England. Crowds have gathered and tributes left at the gates of Windsor Castle to Queen Elizabeth II, who died at Balmoral Castle on 8 September, 2022. (Photo by Chris Jackson - WPA Pool/Getty Images)

Prince Harryis missing his grandmother.

The Duke of Sussex, 37, connected with the crowd during asurprise walkaboutat Windsor Castle Saturday following the death of his grandmotherQueen Elizabethon Thursday. Prince Harry stepped out hand-in-hand with wifeMeghan Marklebeside brotherPrince Williamand sister-in-lawKate Middleton, where he made an emotional admission about how things already felt different inside the royal residence.

“Whichever room she was in, you felt her presence throughout,” he added with a smile.

Kate Middleton, Prince William, Prince Harry and Meghan Markle.Chris Jackson - WPA Pool/Getty

WINDSOR, ENGLAND - SEPTEMBER 10: Catherine, Princess of Wales, Prince William, Prince of Wales, Prince Harry, Duke of Sussex, and Meghan, Duchess of Sussex on the long Walk at Windsor Castle on September 10, 2022 in Windsor, England. Crowds have gathered and tributes left at the gates of Windsor Castle to Queen Elizabeth II, who died at Balmoral Castle on 8 September, 2022. (Photo by Chris Jackson - WPA Pool/Getty Images)

The Queenrelocatedher home base from Buckingham Palace in London to Windsor Castle when the coronavirus pandemic began in March 2020 and spent most of her final years there.

For more on Queen Elizabeth, listen below to our daily podcast PEOPLE Every Day.

Prince Harry, Meghan,Prince Williamand Kateleft the gates of Windsor togetherSaturday to tour the many tributes honoring the late monarch, who died “peacefully” died at age 96 on Thursday.

The outing for the group once known as the “Fab Four” was especially poignant as relations have been strained between the California couple and members of the royal family in recent years.

Kensington Palace says William, 40, invited the Duke and Duchess of Sussex to join him and Kate, 40, who was recentlynamed Princess of Wales, alongside her husband’s newPrince of Wales title.

Prince Harry, Meghan Markle, Prince William and Kate Middleton.Chris Jackson - WPA Pool/Getty

WINDSOR, ENGLAND - SEPTEMBER 10: Prince Harry, Duke of Sussex, Meghan, Duchess of Sussex, Prince William, Prince of Wales and Catherine, Princess of Wales look at floral tributes laid by members of the public on the Long walk at Windsor Castle on September 10, 2022 in Windsor, England. Crowds have gathered and tributes left at the gates of Windsor Castle to Queen Elizabeth II, who died at Balmoral Castle on 8 September, 2022. (Photo by Chris Jackson - WPA Pool/Getty Images)

Another source adds that security was not expecting both couples at the walkabout.

Prince Williamthought the step out to greet the crowds “was an important show of unity at an incredibly difficult time for the family,” a royal source tells PEOPLE.

Can’t get enough ofPEOPLE’s Royals coverage?Sign up for our free Royals newsletterto get the latest updates onKate Middleton,Meghan Markleand more!

A palace insider tells PEOPLE: “It’s such an extraordinary historical moment and also a deeply personal one for the family that you’d hope and think that all members of the family would unite and support [the King] especially. And perhaps some of those wounds can be healed in the process.”

source: people.com