The official website for The Settlers of Catan

Question about The Rivals for Catan Placed cards - Is it allowed to remove buildings/units if there are still unoccupied building sites in the principality?

Answer

Yes.