I'm trying to create basically realistic mermaids, and working on developing body language and vocalizations. I have most of the vocalizations I can think of (suggestions are still appreciated), but definitely need a lot more body language. Ideas?

Edit: They are a highly social colony species, so it should be somewhat complex, and interactions between colony members would be especially helpful. I'm also trying to avoid overly human interactions such as kissing or holding hands.

They could use clicks and whistles like a dolphin, or they could also communicate through bioluminescence like a bobtail squid. They could also cummunicate through whale song.

Sonar

They might have a specialized organ much like porpoises do. It could have bandwidth enough to communicate complex states of mind, to a point where it could be considered a limited telepathy (something of the kind happens with the Tines in Vernor Vinge's A Fire upon the Deep).

Human-like body language has the problem of interfering with swimming. You could render sonar interactions as you wished, though - being to all intents and purposes a sixth sense, mermaids might "hear" it as a voice plus anything else.

Dolphins are social animals and they love to "play" in a variety of ways - some of those appear in David Brin's Uplift cycle, suitably adapted to the uplifted dolphins' increased intelligence.

The reason we rely on our hands and faces so much when communicating is because they are very good for it. The hands and face have lots of muscles allowing for very precise control. This also means our brains have very good ability to control hands and face. And ability to understand movements of hands and face by others. The two are linked due to mirror neurons.

So unless you want them to be very different from normal humans when it comes to hands and face, which you probably would have mentioned as it is kind of critical, their body language communication would heavily rely on hands and face and work fairly similarly to human body language of hands and face.

Some differences.

If you are floating in water instead of standing on a solid surface large movements will have unintended consequences so body language will avoid large movements and focus on just hands and face. Same applies to physical contact such as holding hands or kissing. I think such would be preserved for private moments when you are entangled and not moving.

Underwater sound carries better and light worse than in air. So there should be clearer distinction between private and close range with hand gestures and facial expressions and with public and extended range without those components.

I think those might be a solid starting point for you. Having, from our point of view, much stronger distinctions between public, private, and intimate modes of communication, possibly to the point where even the vocal language used is noticeably different is good flavour that should be visible in pretty much everything and be fairly simple to highlight without needing boring exposition.

Their body language would be transmitted by sound.

Living in the dark underwater restricts vision to close distances. So visual communication would be like whispering in that it is covert and short range.

These merfolk would have exceptionally precise hearing, because sound travels farther and faster underwater. It's the best way to detect oncoming animals, and if they use sonar, they could see stationary barriers.

This means that they could 'see' people's body language through sound. When you move your hand really fast in the air, you can hear it 'whoosh.' This effect would happen all the time with merfolk. Body language would be subtle sounds of your body moving through the water.

As for the content, do the merfolk inhabit a similar world to ours? Because unless they have some hyperadvanced humanlike modern civilization underwater, they probably have to deal with the various predators or prey they interact with, resulting in primal emotion communication.

Since they are basically fish in terms of their movement, they might convey emotions of fear or aversion by twitch-like motions, reminiscent of a fish darting in response to a predator. A turn of the tail helps rotate you around, so maybe it's a gesture to indicate turning your back on someone, or changing your mind.

Since their environment of motion is four dimensional, you could consider a complex dance as a means of communication. The bees do in the hive to indicate pertinents about nectar flowers.

These Merfolk are intelligent and presumably sentient people, so the relatively simple dances of their ancestors -- warning of dangers, locations of food sources, approach of other Merfolk -- will have evolved into a means of telling stories, describing the loot to be found in a shipwreck, courtship and any of ten thousand other things they might to dance about.

A combination of signed language and vocalizations is your best bet. Because both hearing and sight can be compromised under water, a redundant system allows for maximum communication in a variety of situations. I'm envisioning both systems used simultaneously but with communication possible if it's dark or if there is a safety issue where the merfolk have to be quiet (predators, humans), or there is too much ambient noise.

I will also add tactile sign language. This is how (most) people who are deaf and blind communicate and also Deaf people with normal vision who happen to be in the dark or who need to focus visually on something else (a couple at night, friends at the movie theater, etc).

I have very basic sign language skills, and my spouse even less, yet it's remarkably easy to figure out what someone is saying by only laying your hand on theirs. If it's a sign or letter you know visually, you'll get it most of the time tactilely. Obviously someone who uses the skill regularly would be expert in it.

A colony species would be physically close together so communication can spread from one hand to another until everyone gets the message. A human observer might think the merfolk are "holding hands" but, since their hands are touching for communicative purposes, it will have very different cultural meaning and practical applications.

Semaphore (probably without real flags)

This would be great because it would look like the mermaids were communicating by doing an underwater dance. It could also be expanded to include their tail as a third flag. Note the Beatles spelled out their album's title "Help!" using semaphor.

If it's good enough for them, why not for the mermaids?