Argthtjtr

I am a beginning graduate student in Mathematics. I have a professor who always brings up the fact that my GPA is not great in my undergraduate degree even though my performance is above the norm in the institution I am enrolled at the moment, so far I got straight A's in the graduate courses that I have completed. He always tries to point out at my flaws in my undergraduate years to humiliate me in front of other professors and my colleagues. What would be a wise reply/reaction to such abuse?

I feel bad about it cause I feel like I am harassed and I don't reply.
I am also currently taking a course with that professor.

As I understand you, you are being publicly shamed (in front of fellow grad students) by this professor by dint of your undergrad record.
If you are in the US, this is violating FERPA by revealing your academic record. Other countries have similar privacy laws. File a FERPA complaint and wipe that smug smile off his face.

I'm no expert and I suspect each school is different. But this is a federal law and I think most schools would take such a blatant violation seriously. I'm pretty sure this prof would get his leash yanked pretty hard.

Note that it could backfire. You could an enemy who will make your graduate experience miserable. But I'm betting against it. There have been a number of times when I just let a jerk be a jerk because I thought my life would go more smoothly, but found out later that if I had stood up for myself, I would have gotten a standing ovation from 99% of the people around me. This prof is a jerk and his colleagues will likely appreciate him being called out.
Also, in the case of harassment and rights violations, the perpetrator is warned against retaliation in any form. And someone will be watching for it. Someone behind those closed doors will tattle. I think you are perfectly safe.

My experience

When I taught small classes, I would write all the test scores on the board, so that students could see where they ranked. One young lady thought I was violating FERPA with this tattled on me. I got hauled into the chairs office where he was accompanied with one of those university JD types (the law students who never pass the bar, but get jobs at universities being annoying.) and the dean. They were ready to have a field day with me. They had already talked to other students and had corroboration that I had, in fact, written all the scores on the board. The JD was salivating.

We talked at cross purposes for a while, then they figured out that I was writing numbers only. No names. No personal information was being displayed. They were so disappointed. An administrator gets to be administrative so rarely and here they had an open-and-shut case go up in smoke.

So the point here is that at this school at this time, FERPA violations were a big deal. I suspect that in the current safe-room environment, they might be a bigger deal. Telling the class that that guy right there has a low GPA could traumatize him for life, eh?

There are several possibilities. Some may be appropriate. Some may work. Or not.

Avoidance. If possible, just avoid this person. Don't have anything to do with him. Difficult, I know. There are probably limited options to do that.

Ignore his taunts. My guess is that he disgraces himself when he does this. If he does this publicly, other students probably see it for what it is. But a public, angry, response from you would probably do yourself more harm than it is worth.

Formal Complaint. This will have consequences all 'round, but might be effective. His department chair might be interested to hear what you have to say, especially if the professor is un-tenured. But a complaint from a group of students would be more effective than one from a single student. And make the complaint in person or using a formal mail. Email is too easy to ignore, for this.

Try not to feel bad. The actions of the prof are inexcusable and aren't due to anything in you or that you can actually correct. Know it for what it is: unprofessional behavior.

Seek advice from a trusted mentor.

If you have another professor or advisor who you believe you can speak to about this, I would encourage you to explore that option. Not only is it important to have a positive influence to counteract the negative impact of this professor in your personal development, but quality mentorship is also a component in your future career success.

A mentor who has already passed these trials and tribulations in their career may have very well witnessed and experienced these same behaviors. They can offer a more informed plan for how to treat this with your best interests in mind. What your mentor advises may boil down to the same options @Buffy has laid-out. In the case that you should choose formal action, a mentor supporting you in this could be very influential in how it concludes.

One of the unfortunate realities of academia currently is that institutional mechanisms to discourage and rectify this type of behavior are (in my opinion) rare and frequently ineffective. For better or worse, your professors often have an inordinate amount of influence on your career once you are at the graduate level. For this reason, it's really difficult for a student to utilize formal recourse options. Some schools do have specific anti-bullying resources, and you should investigate if these exist at your institution.

I feel you have to be careful not to go nuclear, because if he is your professor you have to be careful not to move from "annoyance" to "threat" in his eyes, given he presumably grades your work. However, it seems like he's already put himself in a bad position since you said he "tries ... to humiliate me in front of other professors", not just in class.

I believe to reduce risk of this going nuclear, you have to present as though you're concerned and not accusing.

So go to his boss. Say you're trying to look out for the organisation as a whole. Point out he's breaching privacy laws by repeatedly bringing this up in front of other professors, and if anyone makes an official complaint for any reason then things aren't going to end well for anyone. Ask his boss if they can have a quiet word to "nip this in the bud" and mitigate reputational risk to the organisation, and by implication also to this guy's boss.

Play up your concern for the organisation and try not to get personal. Mention some of the other staff members he's done this in front of. Stress that you believe it's better for all concerned if this just quietly stops happening.

Afterwards, write down what you believe was discussed in the meeting as clearly and succinctly as you can. Send an email to the person you've just met with, thanking them for their time and including your notes on the "informal meeting" (that is, you're explicitly not making a formal complaint right now) "just for their reference" and to "clear things up if I misinterpreted or misunderstood anything".

Assuming this person's boss doesn't argue your email, you've now established yourself on record as trying to deal with it quietly without reputational risk to the organisation. If it escalates or you need to file a formal complaint later, you've established the moral high ground.