The official website for The Settlers of Catan

Question about The “Age of Darkness” Expansion Castle - In the Theme Game, may I remove the Castle and then build it again, so that the opponent can't build it?

Answer

You may remove the Castle and build it again. This, however, will not prevent the opponent from building a Castle, because in the Theme Game a removed Castle is placed face up again.