The official website for The Settlers of Catan

Question about The Cards Master Builder - Can I renounce my right to exchange a card?

Answer

Only indirectly, by returning the selected card to the bottom of a stack.