# Indexing with Camel. Too many files. Socket Closed **URL:** \ **Category:** Elasticsearch\ **Created:** [December 16, 2016, 6:32pm UTC](https://discuss.elastic.co/t/indexing-with-camel-too-many-files-socket-closed/69313 "2016-12-16T18:32:32Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![JonathanAaron](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jonathanaaron/32/24405_2.png) [@JonathanAaron](https://discuss.elastic.co/u/JonathanAaron)\ **Post date:** [December 16, 2016, 6:32pm UTC](https://discuss.elastic.co/t/indexing-with-camel-too-many-files-socket-closed/69313/1 "2016-12-16T18:32:32Z")
I'm using Apache Camel to index data from a Enterprise Service. I'm testing it out on a local elasticsearch instance. It appears to index about 12000 files then starts giving me a Too many files error. I set my ulimit to 64000 and some other settings: ``` sudo sysctl kern.maxfiles=64000 sudo sysctl kern.maxfilesperproc=64000 sudo ulimit 64000 sudo ulimit -Hn 64000 sudo ulimit -Sn 64000 ``` This got rid of the too many open files issue, but I'm now running into a Socket Closed issue. I'm not sure if these problems will go away when I'm running multiple nodes in a hosted environment instead of my local machine. ---
**Author:** ![mikemccand](https://avatars.discourse-cdn.com/v4/letter/m/f04885/32.png) [@mikemccand](https://discuss.elastic.co/u/mikemccand)\ **Post date:** [December 16, 2016, 7:08pm UTC](https://discuss.elastic.co/t/indexing-with-camel-too-many-files-socket-closed/69313/2 "2016-12-16T19:08:15Z")
Can you run `lsof` to see which files (and which process) are opening so many files? For ES to index just 12K documents it should need anywhere near that number... Mike McCandless ---
**Author:** ![JonathanAaron](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jonathanaaron/32/24405_2.png) [@JonathanAaron](https://discuss.elastic.co/u/JonathanAaron)\ **Post date:** [December 16, 2016, 7:15pm UTC](https://discuss.elastic.co/t/indexing-with-camel-too-many-files-socket-closed/69313/3 "2016-12-16T19:15:17Z")
Once it indexes those files, does it release them or are those files always open for the life of the elasticsearch process? Edit: I just ran my camel Job. `lsof | wc -l` seemed to top out around `15531` and it was able to index around `13962` document, before the sockets closed and the camel job failed. I'm not too familar with sockets and I'm not sure how to increase my socket availablity or release sockets. Do I need to throttle my Camel job so it indexes more slowly? ---
**Author:** ![JonathanAaron](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jonathanaaron/32/24405_2.png) [@JonathanAaron](https://discuss.elastic.co/u/JonathanAaron)\ **Post date:** [December 16, 2016, 7:54pm UTC](https://discuss.elastic.co/t/indexing-with-camel-too-many-files-socket-closed/69313/4 "2016-12-16T19:54:43Z")
Would bulk indexing possible solve this issue or would it use up the same amount of files and sockets? ---
**Author:** ![mikemccand](https://avatars.discourse-cdn.com/v4/letter/m/f04885/32.png) [@mikemccand](https://discuss.elastic.co/u/mikemccand)\ **Post date:** [December 16, 2016, 7:56pm UTC](https://discuss.elastic.co/t/indexing-with-camel-too-many-files-socket-closed/69313/5 "2016-12-16T19:56:42Z")
I suspect your Camel job is somehow holding the files open. ES, with non-bulk and bulk index, would never open so many file handles for so few documents. `lsof` will tell you which file names (or sockets) are held open and which processes have them open. Mike McCandless ---
**Author:** ![JonathanAaron](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jonathanaaron/32/24405_2.png) [@JonathanAaron](https://discuss.elastic.co/u/JonathanAaron)\ **Post date:** [December 16, 2016, 7:57pm UTC](https://discuss.elastic.co/t/indexing-with-camel-too-many-files-socket-closed/69313/6 "2016-12-16T19:57:53Z")
Interesting. I'm pretty new to camel so I'll bring this up to my camel guy and see what's happening. Thanks @mikemccand! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 13, 2017, 7:58pm UTC](https://discuss.elastic.co/t/indexing-with-camel-too-many-files-socket-closed/69313/7 "2017-01-13T19:58:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.