Solving 'too many open files' problem on NetBSD

Submitted by Fekete Zoltán on Mon, 01/23/2017 - 18:00

When you see 'too many open files' message in the log of any userspace program, you have to tweak a couple of things to eliminate.

1. Check how many files are open parallel

$ fstat | wc -l

2. Check what is the limitation of the user

$ ulimit -n

3. Check what parameters have the system as soft and hard limits

# sysctl | grep kern.maxfiles
# sysctl | grep proc.curproc.rlimit.descriptors.hard
# sysctl | grep proc.curproc.rlimit.descriptors.soft

4. Set those sysctl parameters according to your needs

# sysctl -w kern.maxfiles=65535
# sysctl -w proc.curproc.rlimit.descriptors.hard=unlimited
# sysctl -w proc.curproc.rlimit.descriptors.soft=13196

The above example assumes that you would not exceed 13196 open files/process. If you need more, just try. If everything works well, write your own values into /etc/sysctl.conf.  

5. Set the ulimit for the user

Edit the file /etc/login.conf after reading how to. Set openfiles-cur to your soft sysctl limit.
Restart the system, and check  all parameters as described in 2nd and 3rd section. If parameters are ok, try your program again. If not, read on, because your kernel limits the resources.

6. Build a new kernel

Read how to build a kernel for NetBSD. See the attached link. Set the parameter maxusers higher then the current (GENERIC kernel is set up to 64). I have tried with 256, the config script warned, but compilation and run of the new kernel was perfect, so it's up to you.

After building a good working new kernel you should be able to hack parameters as described in 4. 5. sections.

Good luck!