Indexing with Camel. Too many files. Socket Closed

Elastic Stack Elasticsearch
1

I'm using Apache Camel to index data from a Enterprise Service. I'm testing it out on a local elasticsearch instance. It appears to index about 12000 files then starts giving me a Too many files error. I set my ulimit to 64000 and some other settings:

sudo sysctl kern.maxfiles=64000
sudo sysctl kern.maxfilesperproc=64000
sudo ulimit 64000
sudo ulimit -Hn 64000
sudo ulimit -Sn 64000

This got rid of the too many open files issue, but I'm now running into a Socket Closed issue. I'm not sure if these problems will go away when I'm running multiple nodes in a hosted environment instead of my local machine.

2

Can you run lsof to see which files (and which process) are opening so many files?

For ES to index just 12K documents it should need anywhere near that number...

Mike McCandless

3

Once it indexes those files, does it release them or are those files always open for the life of the elasticsearch process?

Edit: I just ran my camel Job. lsof | wc -l seemed to top out around 15531 and it was able to index around 13962 document, before the sockets closed and the camel job failed. I'm not too familar with sockets and I'm not sure how to increase my socket availablity or release sockets. Do I need to throttle my Camel job so it indexes more slowly?

4

Would bulk indexing possible solve this issue or would it use up the same amount of files and sockets?

5

I suspect your Camel job is somehow holding the files open. ES, with non-bulk and bulk index, would never open so many file handles for so few documents.

lsof will tell you which file names (or sockets) are held open and which processes have them open.

Mike McCandless

6

Interesting. I'm pretty new to camel so I'll bring this up to my camel guy and see what's happening. Thanks @mikemccand!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.