True, Green Castle could well be Camp David, per Lady Jane.
I recently saw one of those old crumbs about GREEN CASTLE/RED CASTLE and IIRC, a presidential residence would fit. Must look back.
Q
Maybe Red Castle could be Mar-a-lago? It has a red/orange tile roof.