Argthtjtr

I think that Peter Shor's analysis is very likely the correct one. In support of it, I offer two early instances of "low key" in the sense of low vocal pitch and two early instances of figurative use of "low key." All are drawn from the Elephind newspaper database.

The earliest literal use of "low key" that the Elephind search turns up is from "Interesting Story Extracted from the 'Pioneers'," in the [Vincennes, Indiana] Western Sun (April 12, 1823):

But when aroused by this cry from Louisa, Miss Temple turned, she saw the dog with his eyes keenly set on some distant object, with his head bent near the ground, and his hair actually rising on his body, either through fright or anger. It was most probably the latter for he was growling in a low key and occasionally showing his teeth in a manner that would have terrified his mistress, had she not so well known his good qualities.

Although "low key" might refer to a deep (or bass) pitch—treating the growl as a note of music—I think that it more probably signifies low in loudness—a soft but ominous growl.

And from "Parliamentary Portraits," in the Sydney Gazette and New South Wales Advertiser (November 9, 1837):

He [Mr. Aglionby] speaks with singular rapidity : I am not sure whether he does not speak a greater quantity in a given time than any member in the house. No reporter could, if he wished, follow the hon. member through his speeches; that, however, for the reasons I have already given, is never attempted. His voice is not strong, but it is clear. It is easier to hear than to follow him. He never raises his voice: he continues in the same low key throughout.

Here, "low key" may refer both to loudness and to pitch, as the latter tends to go up when the former increases.

The earliest figurative use of "low key" is from "A Musical Definition," in the [Leesburg, Virginia] Genius of Liberty (October 23, 1830):

A gentleman, whose real name was George Sharp, but who generally went by the appellation, amongst his musical friends, of G sharp, on entering the company, and looking rather dull, a common friend observed that Mr. G. Sharp, was rather on a low key, that evening.—"O," replied a lady, with a good deal of naivette, "every body knows that G sharp is A flat."

The joke turns on the naive lady's literal interpretation of "low key"—but that interpretation wouldn't be a source of mirth if the common friend hadn't meant "low key" in the sense of "looking rather dull."

From "Jenny Lind in the Concert-Room," in the [Hobart, Tasmania] Courier (December 18, 1847):

The pieces set down for the prima donna were " Quando lascini la Normandia," from "Roberto il Diavolo," " Su l'aria," from "Il Nozze di Figaro;" lastly, the simplest, but most alluring entry in the whole list, was "Swedish melodies—by desire." By the desire of whom we know not; but are quite sure that, whoever they may be, there is not one of the audience who is not ready to return them his very hearty thanks for having suggested the enjoyment these melodies afforded. Those who have witnessed Jenny Lind's dramatic powers pitched, in all probability, their expectations in a low key, when they went to hear her with the stage excitements and opportunities of dramatic expression withdrawn, as they are at a day-light concert. If they did they were they were wise, for during the first performance of the first piece the singer was labouring under a degree of nervousness which extended its effects to her voice. The encore, however, reassured her, and she produced more effect than could have been anticipated.

The sense of "low key" here clearly involves restraint rather than literal quietness, but the writer explicitly frames it as a form of metaphorical voice: "pitched ... their expectations in a low key."

Also of possible interest is this instance from Charles Dickens, The Life and Adventures of Martin Chuzzlewit (1844):

She [Mrs. Gamp] continued to sidle at Mr. Chuffey with looks of sharp hostility, and to defy him with many other ironical remarks, uttered in that low key which commonly denotes suppressed indignation; until the entrance of the tea-board, and a request from Mrs. Jonas that she would make tea at a side-table for the party that had unexpectedly assembled, restored her to herself.

This example is significant because it associates "low key" not simply with low vocal pitch but with the suppression or restraint of the character's natural inclination to express her indignation loudly. As Peter Shor observes, it would be no great leap for this notion of "low key" in the sense of restrained or subdued to emerge from an earlier, more literal use of "low key" in the sense of quiet or low pitched.

The 1857 book Introductory Lessons in Reading and Elocution, found on Google books, has a different use from which the figurative use could very easily follow. Here is the relevant excerpt:

Every person has three keys, or pitches, of the voice, called THE HIGH, THE MIDDLE, and THE LOW KEY.
The HIGH KEY is that which is used in calling to a person at a distance.
The MIDDLE KEY is that which is used in common conversation.
The LOW KEY is that which is used when we wish no one to hear, except the person to whom we speak.

This doesn't appear to refer to just musical pitch, but more to loudness.
(Although pitch is related to loudness.) Metaphorically, doing something low key would thus be doing something so as not to attract any attention to it. This could easily metamorphose into the present meaning.

The most attractive possible source of the use of 'low key' in the figurative sense of "muted, restrained" is the very early literal use with reference to the low keys of the organ, as pictured here:

These 'low keys' are typically used to produce the underlying bass tones of the melody being played.

The 'key' of this 'low key' is a transferred sense from the earliest literal uses of 'key' in the sense of "an instrument designed to be inserted into a lock and turned" (OED, Old English from around 1000, earliest attestation from Ælfric, Lives of Saints).

Very early uses of 'low key' in this literal sense were in two French dictionaries, published in 1688 and 1719:

Pedale, ...a pedal, or low key of some Organs to be touched with the feet.

The Great French Dictionary. In Two Parts. Miege, Guy, 1688.

PEDAL, Subst. (a low Key of some Organs) ....

Dictionaire Royal, François-Anglois, et Anglois-François, by Abel Boyer et al., 1719.

By at least 1722, 'low key' was in quasi-figurative use referring to "muted, restrained" in the sense of 'low volume, tone or pitch':

A poor Beggar loaded with Wine, and not being able to carry it any further, laid himself down in a Ditch hard by, and fell into a deep Sleep. When this nocturnal Crew came together, the noise they made awak'd him; and he saw by the light of the Moon, that they were all Soldiers, and shew'd an extraordinary Respect to one amongst 'em. He tried to hear what was said, but they speaking in a low key, and there being a great many in the way, and in an open Place, he listned to no purpose.

The Life and Actions of Lewis Dominique Cartouche, Daniel Defoe, 1722. Emphasis mine.

OED supplies an earlier, 1709, attestation of the use of 'low key' referring plainly, if figuratively, to the "pitch or tone" of a voice:

G. Berkeley Ess. New Theory of Vision §46 Men speak in a high or a low key.

Note that evidence from OED suggests that the musical sense of 'low key' appears as early as a1450, although the 'low key' collocation is not attested:

key, n.¹ and adj.

....
IV. Senses relating to pitch or tone.
17. Music.
†a. A note, a tone; esp. the first or lowest note or tone of a scale or sequence of notes, the keynote. Obs.

a1450 Musical Treat. in Speculum (1935) 10 262 (MED) This same rwle may ye kepe betwene Dsolre, Dlasolre, & al oþer base keyys, whan þe countersight goþ low.

Figurative uses in the sense of the "tone or tenor of a piece of writing, situation, etc.; the intensity or force of a feeling or action" (OED key, sense 18a, cross-referenced to low key in the figurative sense of "muted, restrained"), the "pitch or tone of a person's voice" (18b), and the "prevailing range of tones in a painting; the relative intensity of a particular colour scheme" (18c) are not attested until a1530, 1600, and 1713 respectively (OED).

Attractive as the origin story implicating the lower keys of the organ (and likewise the lower, or white, keys of the piano) in the development of use of 'low key' in the later figurative sense of "not elaborate, showy, or intense; muted, restrained; modest" (OED, low key, fig., sense 2, emphasis mine) may be, the more convincing story overall is also more straightforward and plain.

Simply, 'low key' in the specific figurative sense at hand is intrinsically comparative, and developed from the correlated comparative phrase 'lower key'. For its part, 'lower key', as used in the later figurative sense, first (to the best of my knowledge) appeared in two primary contexts in the mid-1600s: the context of religion, and the context of rhetoric. The latter, rhetoric, is in this case an element of religious persuasion by means of the ornamentation of speech or writing, and so partakes so much of the former, the religious context, that the two, although I have here artificially separated them, are perhaps but one context.

The evidence supporting my conclusion is, like the phrase derivation itself, simple and straightforward. Very early uses of 'lower key', and subsequent uses in the intervening years between those early uses and later uses such as those cited by OED from 1941 and after, are in the specific figurative sense (OED sense 2 of 'low-key'). For example, this from the 1657 publication, The Protestants Evidence, by Simon Birckbek:

By this strain of rhetorick Chrysostome (as his manner is) perswadeth the people to come to the Lords Table with no lesse reverence, than if they were to receive a fiery coal (as Esay did in his vision) from one of the glorious Seaphims. Chrysostome had no intent that the bread was transubstantiated, no more than that the Priest was changed into an Angel, or his hand into a pair of tongs, or the body of Christ into a coal of fire; and he useth the same amplification in both the speeches, the same phrase [think you] and at the same time, and to the same people: so that if one be (as certainly it is) a strain of Rhetorick, why not the other also? Sixtus Senensis gives a good rule for interpretation of the Fathers speeches, specially in this argument; (z) The sayings of Preachers are not to be urged in that rigour of their words; for after the manner of Orators, they use to speak many times hyperbolically, and in exesse. ... Now these and the like sayings must be favourably construed, as being improper speeches, rhetoricall strains, purposely uttered to move affections, stirre up devotion, and bring the Sacrament out of contempt, that so the Communicants eyes may not be finally fixed on the outward elements of bread and wine...but to have their hearts elevated and lift up by faith to behold the very body of Christ, which is represented in these mysteries. Otherwise, the Fathers come down to a lower key, when they come to speak to the point, yes or no....

[Bold emphasis mine.]

While in the foregoing passage the sense is, as must be expected from the comparative form of 'lower', "not [as] elaborate, showy, or intense; muted, restrained, modest", the essential sense, stripped of its comparative force, is the same as the later figurative sense.

Additionally, observe that an early figurative sense of 'key' plays into the meaning conveyed to contemporaneous readers and auditors by the foregoing passage from The Protestants Evidence. That figurative sense is specifically religious:

II. Figurative and allusive uses.
3.
a. [With allusion to Matthew 16:19 (see quot. c1384).] R.C. Church. The spiritual authority believed to have been transmitted from Christ to St Peter, and so to the Pope considered as his successor, as symbolized by the keys to the kingdom of heaven. Also in a wider sense: the power or authority of the Christian Church or its priests.

OED, key, n.¹ and adj.

Thus, while the 'key' of 'low key' or 'lower key' remains a transferred sense from the earliest uses (see above), and while the first literal uses certainly influenced the development of the figurative sense of "not elaborate, etc." deployed in the passage from The Protestants Evidence (and elsewhere), as did the later semi-literal uses referring to the physically lower keys of the organ and piano, the immediate progenitor of the figurative sense of "not elaborate, etc." was the very similar earlier figurative use:

18. fig.
a. The tone or tenor of a piece of writing, situation, etc.; the intensity or force of a feeling or action.
a1530 T. Lupset Compend. Treat. Dyenge Well (1534) sig. E.iiii In this tenor and key sowneth al our holy scripture.

OED, key, n.¹ and adj.

As shown by the earliest attestation, from around 1530, this figurative sense of 'key', like the later figurative sense of 'key' in 'lower key' from The Protestants Evidence, appeared in the context of religious persuasion (rhetoric). The influence of music on the development of the figurative "not elaborate, etc." sense, while present in force (much more so than any later uses with reference to the 'key' of light in painting), takes a back seat to the earlier and more general influence of sound, and specifically the relative pitch of sound.

I believe 'low key' began life as 'low quay'.

Quay, [...], has its origin in the Proto-Celtic language. Before it changed to its current form under influence of the modern French quai, its Middle English spelling was key, keye or caye. This in turn also came from the Old North French cai (Old French chai), both roughly meaning "sand bank". The Old French term came from Gaulish caium, ultimately tracing back to the Proto-Celtic *kagio- "to encompass, enclose". Modern cognates include Welsh cae "fence, hedge" and Cornish ke "hedge". [7]

This is a reference to a two-tier dock system created for the shipping industry to deal with low and high tides. The normal (high tide) level would be the main quay, and to access to ships during low-tide would be via steps and/or a 'low quay'.

According to stevedores and dockers (stevedores work on the ships during loading/unloading of cargo, dockers on the dock), working the 'low quay' could be dangerous work, and certainly more difficult than working high tides.

Popular culture romantized 'low quay' into a ground for illicit activity - out of sight and the incoming tide could remove evidence from the scene, and gradually the meaning morphed into what is known as 'low key' today.

I can't fully answer this question, but I wanted to pass along two early examples.

First from the The Westminster Magazine, Or, The Pantheon of Taste, volume 4, (1776):

"It is very plain,” continued he, “that you have never been married, and are totally unacquainted with the artifices of the sex, or you would with half an eye have seen through all this.”

“Why,” replied I, in the same low key, “I do see that she is almost worn to a skeleton..."

In volume 5 (1777) of the same magazine there is:

The approach of the Chaplain and his Lady to pay their compliments, put a stop to the above conversation, carrie on in a low key, with a number of significant nods, shrugs, and other supplemental graces.

More literally, and much earlier (1597), the term "low key" is used in A plaine and easie introduction to practicall musicke set downe in forme of a dialogue. One of the four uses of the term in this book is:

Now must you diligentlie marke that in which of all these compasses you make your musicke, you must not suffer any part to goe without the compasse of his rules, except one note at the most aboue or below, without it be vpon an extremity for the ditties sake or in notes taken for Diapasons in the base. It is true that the high and lowe keyes come both to one pitch, or rather compasse, but you must vnderstand that those songs which are made for the high key be made for more life, the other in the low key with more grauetie and staidnesse, so that if you sing them in contrarie keyes, they wil loose their grace and wil be wrested as it were out of their nature: for take an instrument, as a Lute Orphadrion, Pandorae or such like, being in the naturall pitch, and set it a note or two lower it wil go much heauier and duller, and far from that spirit which it had before, much more being foure notes lower then the naturall pitch.

Note also that in some of the 1597 examples "low" is spelled "lowe".