The official website for
the world of Catan

Question about Special Cards Irmgard the Keeper of the Light - Can my opponent take Irmgard away from me?

Answer

No. The only risk is that he might be the first one to place Irmgard.