The official website for The Settlers of Catan

Question about Base Game Largest Army – (See also: Development Cards “Knight”) - Do I have to hand over the Largest Army card and subtract the 2 victory points from my score if another player has more Knight Cards?

Answer

Yes. That’s how it’s specified in the game rules.