The official website for The Settlers of Catan

Question about “Cities & Knights” Expansion City Improvements - May I perform city improvements only in an area that matches the color of the city gate rolled on the symbol die?

Answer

No. The symbol die has no influence on your building activity.