My group was in this cottage gathering clues when someone kicked open the door while wielding a battle axe. I as a cleric reacted to this by using my action in order to make the now open door slam shut on his face pushing him outside using thaumaturgy:

You instantaneously cause an unlocked door or window to fly open or slam shut.

So what happens to the guy who had kicked the door?

The guy was standing right at the entrance breathing heavily and I won initiative.

The rules don't say, thus it is up to the DM

Thaumaturgy only has this to say about the door closing effect:

You instantaneously cause an unlocked door or window to fly open or slam shut.

Now, 5e spells are designed to do only the things that they say in the spell description. However, in this case, that doesn't really help settle the dilemma. Does the door shut, regardless of obstacles because the spell says that the door shuts? Or will obstacles prevent the door from shutting because there is nothing to indicate that the door slamming shut is in any way immune to the effects of normal physics with respect to things blocking it?

The thing is, the rules don't say. And either ruling, when applied uniformly, can lead to nonsensical results. For example, the door flinging enormous boulders out of the way to close.

And that is another thing 5e spells are designed for. Many spells are written in language that purposefully doesn't try to cover every detail or circumstance instead choosing to leave it up to the DM to adjudicate the results in those cases.

This is one of those cases. Ask your DM.

Reasonable ruling: it acts like a normal door slammed by a normal person

The way I would rule it at my table is that the door is not imbued with any kind of supernatural strength, but instead is simply closed as if I had slammed it myself.

So, what would happen to the guy in the door? It would depend on the exact circumstances of where they where, but if they were in enough to block the door, likely the only thing it would do to them is stub their toe or bruise their nose (not enough to do HP damage) just as a normal slammed door would do and the door would remain ajar.

This is my ruling that I would probably make a my table. It makes sense to me because we all know how doors normally work and thus it makes it easy to keep my rulings consistent and reasonable.

Your DM should consider what option is best fit for their table that will make sense to them and maximise fun.

Not much happens to the guy in the door

As spells only do what they say they do, what happens when you try to slam the door shut with someone standing in them? The door does not shut. Will it be painful for them? Maybe. Will it do any damage? No, the spell would say so. Will it smack into their face? Depends on where they stand – I would presume that after kicking the door open, they are well on the way through that door. Would it have any mechanical effect if it did? No, see above. (I assume the door opens into the room; in any case the it may also be destroyed by the aforementioned kicking in which case there is not much to use the spell on.)

Obviously, this is a rather strict, RAW reading. Your DM may rule otherwise, allowing for fun consequences.

You can try anything, the DM will narrate the result

Based on the core model of play being: 1. The DM describes the environment; 2. The players describe what they want to do; 3. The DM narrates the results of the adventurers’ actions(Basic Rules, p. 4), a recommended ruling is to apply opposed ability checks for a mini-shove.

The guy was standing right at the entrance breathing heavily and I won initiative.

This case looks like a way to use the cantrip's ability to slam the door shut in a tactical way without trying to turn that spell into a weapon/damage causing spell - none of its features indicate that it damages opponents. The opponent should get a chance to resist the effects since he's already in the doorway.

In order to see if the door slams in the opponent's face and closes (which would push him back to 'just outside the door', use the contest rules. A full "shove" action (knock the opponent back 5') does not seem appropriate for what is being attempted.

Contests

Sometimes one character’s or monster’s efforts are directly
opposed to another’s. This can occur when both of them are trying to
do the same thing and only one can succeed, such as attempting to
snatch up a magic ring that has fallen on the floor. This situation
also applies when one of them is trying to prevent the other one from
accomplishing a goal—for example, when a monster tries to force open a
door that an adventurer is holding closed.

The situation you describe is very similar to this. A contest is an opposed ability check. This does not seem to be a case where an automatic win should be granted to slam the door. There needs to be a chance that the door bounces off of the guy with the ax, and does not knock him back.

How to adjudicate the contest: oppose the cleric's spell casting DC with the opponent's Athletics(Strength) ability check. The outcome is that either the cleric succeeds, or if the opponent is strong enough, or quick enough, to resist the door slamming in his face. Tie goes to the guy in the doorway

If the contest results in a tie, the situation remains the same as it was before the contest

Discuss this with your DM

This is a case where rulings over rules, a 5e design paradigm, can work to make the game fun. Rewarding innovative use of game features - any DM ought to be on the lookout for that chance. In this case, the DM might offer advantage to the guy with the ax, or to the cleric, depending on how the rest of the situation is set up.

Rules As Fun-embrace it.

After the initiative is rolled, let us talk in terms of combat abstractions.

If the traditional combat grid is assumed, then the door is probably the border between two squares - which means the barbarian can't end his turn in the doorway - he is occupying either the 5' square outside the door, or the 5' square inside the building.

Shutting the door by any means (Thaumaturgy, Mage Hand, an action) on your turn does only that - closes the door. It will close in front of him if he is considered occupying an outside square, or will shut behind him otherwise. With no effect on the barbarian himself.

It is reasonable to expect a barbarian ready for combat, within the game's abstracted 6-second round, to be able to take a small step inside a 5' square he controls to not get hit by a closing door.