Argthtjtr

The following should resolve most solutions (and are listed in order of their hierarchy):

echo 'kern.maxfiles=20480' | sudo tee -a /etc/sysctl.conf

echo -e 'limit maxfiles 8192 20480nlimit maxproc 1000 2000' | sudo tee -a /etc/launchd.conf

echo 'ulimit -n 4096' | sudo tee -a /etc/profile

Notes:

1. You will need to restart for these changes to take effect.


2. AFAIK you can no longer set limits to 'unlimited' under OS X


3. launchctl maxfiles are bounded by sysctl maxfiles, and
   therefore cannot exceed them


4. sysctl seems to inherit kern.maxfilesperproc from launchctl
   maxfiles


5. ulimit seems to inherit it's 'open files' value from
   launchctl by default


6. you can set a custom ulimit within /etc/profile, or
   ~/.profile ; while this isn't required I've provided an example


7. Be cautious when setting any of these values to a very high number when compared with their default - the features exist stability/security. I've taken these example numbers that I believe to be reasonable, written on other websites.

It seems like there is an entirely different method for changing the open files limit for each version of OS X!

For OS X Sierra (10.12.X) you need to:

1.
In Library/LaunchDaemons create a file named limit.maxfiles.plist and paste the following in (feel free to change the two numbers (which are the soft and hard limits, respectively):

<?xml version="1.0" encoding="UTF-8"?>  

<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"  

        "http://www.apple.com/DTDs/PropertyList-1.0.dtd">

<plist version="1.0">  

  <dict>

    <key>Label</key>

    <string>limit.maxfiles</string>

    <key>ProgramArguments</key>

    <array>

      <string>launchctl</string>

      <string>limit</string>

      <string>maxfiles</string>

      <string>64000</string>

      <string>524288</string>

    </array>

    <key>RunAtLoad</key>

    <true/>

    <key>ServiceIPC</key>

    <false/>

  </dict>

</plist> 

2.
Change the owner of your new file:

sudo chown root:wheel /Library/LaunchDaemons/limit.maxfiles.plist

3.
Load these new settings:

sudo launchctl load -w /Library/LaunchDaemons/limit.maxfiles.plist

4.
Finally, check that the limits are correct:

launchctl limit maxfiles

It seems that OS X Lion will not permit "unlimited" as a value:

% sudo launchctl limit maxfiles 8192 unlimited

Neither the hard nor soft limit for "maxfiles" can be unlimited. Please use a numeric parameter for both.

Providing numerical values for both the soft and the hard limit does the job:

% sudo launchctl limit maxfiles 4096 8192

On Mavericks its simple. As a regular user:

ulimit -n 8192

You can check the updated settings via

ulimit -a

On my machine:

ulimit -a

-t: cpu time (seconds)              unlimited

-f: file size (blocks)              unlimited

-d: data seg size (kbytes)          unlimited

-s: stack size (kbytes)             8192

-c: core file size (blocks)         0

-v: address space (kbytes)          unlimited

-l: locked-in-memory size (kbytes)  unlimited

-u: processes                       709

-n: file descriptors                8192

On OS X 10.13.6 this works for me:

$ sysctl kern.maxfiles

kern.maxfiles: 12288

$ sysctl kern.maxfilesperproc

kern.maxfilesperproc: 10240

$ sudo sysctl -w kern.maxfiles=1048600

kern.maxfiles: 12288 -> 1048600

$ sudo sysctl -w kern.maxfilesperproc=1048576

kern.maxfilesperproc: 10240 -> 1048576

$ ulimit -S -n

256

$ ulimit -S -n 1048576

$ ulimit -S -n

1048576

Some limits cannot be changed by ulimit, because they only apply to the current shell, therefore launchctl command should be used to change the limits globally, e.g.

sudo launchctl limit maxfiles 100000 unlimited

Please note that applying these limits in /etc/launchd.conf (as suggested in other answers) is no longer supported in the recent macOS releases. Although can still use launchd.plist (see: man launchd.plist) such as per-user or system-wide plist configuration files as suggested here and there.

To make these limits persistent, you can use /etc/sysctl.conf file and add for example:

kern.maxprocperuid=1000

kern.maxproc=2000

kern.maxfilesperproc=20000

kern.maxfiles=50000

^{For changes to take the effect, it requires reboot.}

To see the current limits, run: launchctl limit or sysctl -a | grep ^kern.max.

See also: How to persist ulimit settings in macOS?