The official website for The Settlers of Catan

Question about Special Cards Irmgard the Keeper of the Light - Can my opponent take Irmgard away from me?

Answer

No. The only risk is that he might be the first one to place Irmgard.