Built on a plantation, Prince George's home is steeped in centuries of history