Some people push you until you react. Sometimes one needs to react when people cross boundaries.
If people insult you and continuously make derogatory remarks, you have to put a stop to it, otherwise people will continually take advantage of you.
You need to make them realize that there is a limit to everything. You do not want to pick every battle, but pick those that matter.