Strange no. of file descriptors eating up

Kailashnath_Reddy · September 5, 2013, 4:01am

Recently we've been running migration and we see a lot of errors with
elasticsearch, after sometime it just doesn't accept any more connections.

All I see in the logs is:

Caused by: java.io.FileNotFoundException:
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/4/index/_6t86.fdx
(Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at
org.apache.lucene.store.FSDirectory$FSIndexInput.(FSDirectory.java:409)
at
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:123)
at
org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:80)
at
org.apache.lucene.store.RateLimitedFSDirectory.openInput(RateLimitedFSDirectory.java:99)
at
org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:436)
at
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.(CompressingStoredFieldsReader.java:98)
at
org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsReader(CompressingStoredFieldsFormat.java:113)
at
org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:147)
at
org.apache.lucene.index.SegmentReader.(SegmentReader.java:56)
at
org.apache.lucene.index.ReadersAndLiveDocs.getReader(ReadersAndLiveDocs.java:121)
at
org.apache.lucene.index.ReadersAndLiveDocs.getReadOnlyClone(ReadersAndLiveDocs.java:218)
at
org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:96)
at
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:369)
at
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:111)
at
org.apache.lucene.search.SearcherManager.(SearcherManager.java:89)
at
org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1412)
at
org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:271)
... 7 more
[2013-09-05 05:52:11,722][WARN ][cluster.action.shard ] [Slave] sending
failed shard for [verbose][4], node[mFU1CeLwSr6sa4Jb0sqZpQ], [R],
s[INITIALIZING], reason [Failed to start shard, message
[RecoveryFailedException[[verbose][4]: Recovery failed from
[Master][4Zno5nzmQFqdh_o5TbGf5A][inet[/10.223.12.31:9300]]{master=true}
into
[Slave][mFU1CeLwSr6sa4Jb0sqZpQ][inet[/10.223.12.32:9300]]{master=false}];
nested:
RemoteTransportException[[Master][inet[/10.223.12.31:9300]][index/shard/recovery/startRecovery]];
nested: RecoveryEngineException[[verbose][4] Phase[2] Execution failed];
nested:
RemoteTransportException[[Slave][inet[/10.223.12.32:9300]][index/shard/recovery/prepareTranslog]];
nested: EngineCreationFailureException[[verbose][4] failed to open reader
on writer]; nested:
FileNotFoundException[/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/4/index/_6t86.fdx
(Too many open files)]; ]]
FileNotFoundException[/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/4/index/_6t86.fdx
(Too many open files)]; ]]

The ulimit in production machines are 65535 and when I try to do lsof -a -p
, I see something like this:

6291457
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/translog/translog-1369668960033
java 21548 kailashnath 848r REG 8,17 1941 10010832
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/_536h_es090_0.pos
java 21548 kailashnath 849r REG 8,17 15174 9971253
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdk.nvd
java 21548 kailashnath 850r REG 8,17 114071 9971246
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdk_es090_0.doc
java 21548 kailashnath 851r REG 8,17 2090 10010846
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/_536h_es090_0.pay
java 21548 kailashnath 852r REG 8,17 606 10010847
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/_536h_es090_0.tim
java 21548 kailashnath 853r REG 8,17 855 10010690
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/_536h.fdt
java 21548 kailashnath 854r REG 8,17 899 9970974
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mda.fdx
java 21548 kailashnath 855r REG 8,17 47 10010689
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/_536h.fdx
java 21548 kailashnath 856r REG 8,17 162098 10010696
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578p_es090_0.pos
java 21548 kailashnath 857r REG 8,17 34 10010702
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578p_es090_0.pay
java 21548 kailashnath 858r REG 8,17 391138 10010703
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578p_es090_0.tim
java 21548 kailashnath 859r REG 8,17 389565 10010694
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578p.fdt
java 21548 kailashnath 860r REG 8,17 158 10010693
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578p.fdx
java 21548 kailashnath 861r REG 8,17 400 10010915
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578q.nvd
java 21548 kailashnath 862r REG 8,17 1183 10010905
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578q_es090_0.doc
java 21548 kailashnath 863r REG 8,17 2578 10010906
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578q_es090_0.pos
java 21548 kailashnath 864r REG 8,17 34 10010907
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578q_es090_0.pay
java 21548 kailashnath 865r REG 8,17 517 10010912
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578q_es090_0.tim
java 21548 kailashnath 866r REG 8,17 590 10010692
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578q.fdt
java 21548 kailashnath 867r REG 8,17 45 10010691
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578q.fdx
java 21548 kailashnath 869r REG 8,17 412 9971197
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/_9w36.nvd
java 21548 kailashnath 870r REG 8,17 4116 9971130
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/_9w36_es090_0.doc
java 21548 kailashnath 871r REG 8,17 4306 9971133
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/_9w36_es090_0.pos
java 21548 kailashnath 872r REG 8,17 34 9971191
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/_9w36_es090_0.pay
java 21548 kailashnath 873r REG 8,17 13628 9971192
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/_9w36_es090_0.tim
java 21548 kailashnath 874r REG 8,17 19938 9971127
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/_9w36.fdt
java 21548 kailashnath 875r REG 8,17 55 9971117
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/_9w36.fdx
java 21548 kailashnath 876r REG 8,17 136468 9971247
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdk_es090_0.pos
java 21548 kailashnath 877r REG 8,17 34 9971248
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdk_es090_0.pay
java 21548 kailashnath 878r REG 8,17 235767 9971249
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdk_es090_0.tim
java 21548 kailashnath 879r REG 8,17 670494 9971245
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdk.fdt
java 21548 kailashnath 880r REG 8,17 736 9971244
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdk.fdx
java 21548 kailashnath 881r REG 8,17 424 9971270
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdl.nvd
java 21548 kailashnath 882r REG 8,17 4285 9971242
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdl_es090_0.doc
java 21548 kailashnath 883r REG 8,17 4441 9971243
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdl_es090_0.pos
java 21548 kailashnath 884r REG 8,17 34 9971264
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdl_es090_0.pay
java 21548 kailashnath 885r REG 8,17 14708 9971266
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdl_es090_0.tim
java 21548 kailashnath 886r REG 8,17 21322 9971060
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdl.fdt
java 21548 kailashnath 887r REG 8,17 54 9971059
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdl.fdx
java 21548 kailashnath 889r REG 8,17 16262 9970705
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/_9wdr.nvd
java 21548 kailashnath 890r REG 8,17 155969 9970687
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/_9wdr_es090_0.doc
java 21548 kailashnath 910r REG 8,17 159939 9970689
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/_9wdr_es090_0.pos
java 21548 kailashnath 911r REG 8,17 34 9970690
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/_9wdr_es090_0.pay
java 21548 kailashnath 912r REG 8,17 341259 9970693
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/_9wdr_es090_0.tim
java 21548 kailashnath 913r REG 8,17 821871 9970644
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/_9wdr.fdt
java 21548 kailashnath 914r REG 8,17 804 9970641
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/_9wdr.fdx

Not sure what is the issue

/K

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · September 5, 2013, 4:57am

How many shards do you have on this node?

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 5 sept. 2013 à 06:01, Kailashnath Reddy retadedgeek@gmail.com a écrit :

Recently we've been running migration and we see a lot of errors with elasticsearch, after sometime it just doesn't accept any more connections.

All I see in the logs is:

Caused by: java.io.FileNotFoundException: /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/4/index/_6t86.fdx (Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at org.apache.lucene.store.FSDirectory$FSIndexInput.(FSDirectory.java:409)
at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:123)
at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:80)
at org.apache.lucene.store.RateLimitedFSDirectory.openInput(RateLimitedFSDirectory.java:99)
at org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:436)
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.(CompressingStoredFieldsReader.java:98)
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsReader(CompressingStoredFieldsFormat.java:113)
at org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:147)
at org.apache.lucene.index.SegmentReader.(SegmentReader.java:56)
at org.apache.lucene.index.ReadersAndLiveDocs.getReader(ReadersAndLiveDocs.java:121)
at org.apache.lucene.index.ReadersAndLiveDocs.getReadOnlyClone(ReadersAndLiveDocs.java:218)
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:96)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:369)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:111)
at org.apache.lucene.search.SearcherManager.(SearcherManager.java:89)
at org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1412)
at org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:271)
... 7 more
[2013-09-05 05:52:11,722][WARN ][cluster.action.shard ] [Slave] sending failed shard for [verbose][4], node[mFU1CeLwSr6sa4Jb0sqZpQ], [R], s[INITIALIZING], reason [Failed to start shard, message [RecoveryFailedException[[verbose][4]: Recovery failed from [Master][4Zno5nzmQFqdh_o5TbGf5A][inet[/10.223.12.31:9300]]{master=true} into [Slave][mFU1CeLwSr6sa4Jb0sqZpQ][inet[/10.223.12.32:9300]]{master=false}]; nested: RemoteTransportException[[Master][inet[/10.223.12.31:9300]][index/shard/recovery/startRecovery]]; nested: RecoveryEngineException[[verbose][4] Phase[2] Execution failed]; nested: RemoteTransportException[[Slave][inet[/10.223.12.32:9300]][index/shard/recovery/prepareTranslog]]; nested: EngineCreationFailureException[[verbose][4] failed to open reader on writer]; nested: FileNotFoundException[/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/4/index/_6t86.fdx (Too many open files)]; ]]
FileNotFoundException[/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/4/index/_6t86.fdx (Too many open files)]; ]]

The ulimit in production machines are 65535 and when I try to do lsof -a -p , I see something like this:

6291457 /opt/isystem/dbs/elasticsearc java 21548 kailashnath 848r REG java 21548 kailashnath 849r REG java 21548 kailashnath 850r REG java 21548 kailashnath 851r REG java 21548 kailashnath 852r REG java 21548 kailashnath 853r REG java 21548 kailashnath 854r REG java 21548 kailashnath 855r REG java 21548 kailashnath 856r REG java 21548 kailashnath 857r REG java 21548 kailashnath 858r REG java 21548 kailashnath 859r REG java 21548 kailashnath 860r REG java 21548 kailashnath 861r REG java 21548 kailashnath 862r REG java 21548 kailashnath 863r REG java 21548 kailashnath 864r REG java 21548 kailashnath 865r REG java 21548 kailashnath 866r REG java 21548 kailashnath 867r REG java 21548 kailashnath 869r REG java 21548 kailashnath 870r REG java 21548 kailashnath 871r REG java 21548 kailashnath 872r REG java 21548 kailashnath 873r REG java 21548 kailashnath 874r REG java 21548 kailashnath 875r REG java 21548 kailashnath 876r REG java 21548 kailashnath 877r REG java 21548 kailashnath 878r REG java 21548 kailashnath 879r REG java 21548 kailashnath 880r REG java 21548 kailashnath 881r REG java 21548 kailashnath 882r REG java 21548 kailashnath 883r REG java 21548 kailashnath 884r REG java 21548 kailashnath 885r REG java 21548 kailashnath 886r REG java 21548 kailashnath 887r REG java 21548 kailashnath 889r REG java 21548 kailashnath 890r REG java 21548 kailashnath 910r REG java 21548 kailashnath 911r REG java 21548 kailashnath 912r REG java 21548 kailashnath 913r REG java 21548 kailashnath 914r REG h/Projectplace/nodes/0/indices/users/3/translog/translog-1369668960033
8,17 1941 10010832 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/_536h_es090_0.pos
8,17 15174 9971253 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdk.nvd
8,17 114071 9971246 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdk_es090_0.doc
8,17 2090 10010846 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/_536h_es090_0.pay
8,17 606 10010847 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/_536h_es090_0.tim
8,17 855 10010690 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/_536h.fdt
8,17 899 9970974 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mda.fdx
8,17 47 10010689 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/_536h.fdx
8,17 162098 10010696 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578p_es090_0.pos
8,17 34 10010702 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578p_es090_0.pay
8,17 391138 10010703 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578p_es090_0.tim
8,17 389565 10010694 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578p.fdt
8,17 158 10010693 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578p.fdx
8,17 400 10010915 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578q.nvd
8,17 1183 10010905 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578q_es090_0.doc
8,17 2578 10010906 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578q_es090_0.pos
8,17 34 10010907 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578q_es090_0.pay
8,17 517 10010912 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578q_es090_0.tim
8,17 590 10010692 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578q.fdt
8,17 45 10010691 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578q.fdx
8,17 412 9971197 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/_9w36.nvd
8,17 4116 9971130 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/_9w36_es090_0.doc
8,17 4306 9971133 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/_9w36_es090_0.pos
8,17 34 9971191 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/_9w36_es090_0.pay
8,17 13628 9971192 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/_9w36_es090_0.tim
8,17 19938 9971127 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/_9w36.fdt
8,17 55 9971117 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/_9w36.fdx
8,17 136468 9971247 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdk_es090_0.pos
8,17 34 9971248 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdk_es090_0.pay
8,17 235767 9971249 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdk_es090_0.tim
8,17 670494 9971245 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdk.fdt
8,17 736 9971244 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdk.fdx
8,17 424 9971270 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdl.nvd
8,17 4285 9971242 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdl_es090_0.doc
8,17 4441 9971243 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdl_es090_0.pos
8,17 34 9971264 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdl_es090_0.pay
8,17 14708 9971266 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdl_es090_0.tim
8,17 21322 9971060 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdl.fdt
8,17 54 9971059 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdl.fdx
8,17 16262 9970705 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/_9wdr.nvd
8,17 155969 9970687 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/_9wdr_es090_0.doc
8,17 159939 9970689 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/_9wdr_es090_0.pos
8,17 34 9970690 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/_9wdr_es090_0.pay
8,17 341259 9970693 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/_9wdr_es090_0.tim
8,17 821871 9970644 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/_9wdr.fdt
8,17 804 9970641 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/_9wdr.fdx

Not sure what is the issue

/K

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Kailashnath_Reddy · September 5, 2013, 6:46am

Hi David,
It's currently set to default 5.

Set the number of shards (splits) of an index (5 by default):

index.number_of_shards: 5

Set the number of replicas (additional copies) of an index (1 by default):

index.number_of_replicas: 1

Note, that for development on a local machine, with small indices, it

usually

makes sense to "disable" the distributed features:

index.number_of_shards: 1

index.number_of_replicas: 0

/K

On Thursday, September 5, 2013 10:27:33 AM UTC+5:30, David Pilato wrote:

How many shards do you have on this node?

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 5 sept. 2013 à 06:01, Kailashnath Reddy <retad...@gmail.com<javascript:>>
a écrit :

Recently we've been running migration and we see a lot of errors with
elasticsearch, after sometime it just doesn't accept any more connections.

All I see in the logs is:

Caused by: java.io.FileNotFoundException:
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/4/index/_6t86.fdx
(Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at
org.apache.lucene.store.FSDirectory$FSIndexInput.(FSDirectory.java:409)
at
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:123)
at
org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:80)
at
org.apache.lucene.store.RateLimitedFSDirectory.openInput(RateLimitedFSDirectory.java:99)
at
org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:436)
at
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.(CompressingStoredFieldsReader.java:98)
at
org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsReader(CompressingStoredFieldsFormat.java:113)
at
org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:147)
at
org.apache.lucene.index.SegmentReader.(SegmentReader.java:56)
at
org.apache.lucene.index.ReadersAndLiveDocs.getReader(ReadersAndLiveDocs.java:121)
at
org.apache.lucene.index.ReadersAndLiveDocs.getReadOnlyClone(ReadersAndLiveDocs.java:218)
at
org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:96)
at
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:369)
at
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:111)
at
org.apache.lucene.search.SearcherManager.(SearcherManager.java:89)
at
org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1412)
at
org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:271)
... 7 more
[2013-09-05 05:52:11,722][WARN ][cluster.action.shard ] [Slave]
sending failed shard for [verbose][4], node[mFU1CeLwSr6sa4Jb0sqZpQ], [R],
s[INITIALIZING], reason [Failed to start shard, message
[RecoveryFailedException[[verbose][4]: Recovery failed from
[Master][4Zno5nzmQFqdh_o5TbGf5A][inet[/10.223.12.31:9300]]{master=true}
into
[Slave][mFU1CeLwSr6sa4Jb0sqZpQ][inet[/10.223.12.32:9300]]{master=false}];
nested:
RemoteTransportException[[Master][inet[/10.223.12.31:9300]][index/shard/recovery/startRecovery]];
nested: RecoveryEngineException[[verbose][4] Phase[2] Execution failed];
nested:
RemoteTransportException[[Slave][inet[/10.223.12.32:9300]][index/shard/recovery/prepareTranslog]];
nested: EngineCreationFailureException[[verbose][4] failed to open reader
on writer]; nested:
FileNotFoundException[/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/4/index/_6t86.fdx
(Too many open files)]; ]]
FileNotFoundException[/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/4/index/_6t86.fdx
(Too many open files)]; ]]

The ulimit in production machines are 65535 and when I try to do lsof -a
-p , I see something like this:

6291457
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/translog/translog-1369668960033
java 21548 kailashnath 848r REG 8,17 1941
10010832
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/_536h_es090_0.pos
java 21548 kailashnath 849r REG 8,17 15174
9971253
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdk.nvd
java 21548 kailashnath 850r REG 8,17 114071
9971246
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdk_es090_0.doc
java 21548 kailashnath 851r REG 8,17 2090
10010846
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/_536h_es090_0.pay
java 21548 kailashnath 852r REG 8,17 606
10010847
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/_536h_es090_0.tim
java 21548 kailashnath 853r REG 8,17 855
10010690
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/_536h.fdt
java 21548 kailashnath 854r REG 8,17 899
9970974
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mda.fdx
java 21548 kailashnath 855r REG 8,17 47
10010689
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/_536h.fdx
java 21548 kailashnath 856r REG 8,17 162098
10010696
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578p_es090_0.pos
java 21548 kailashnath 857r REG 8,17 34
10010702
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578p_es090_0.pay
java 21548 kailashnath 858r REG 8,17 391138
10010703
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578p_es090_0.tim
java 21548 kailashnath 859r REG 8,17 389565
10010694
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578p.fdt
java 21548 kailashnath 860r REG 8,17 158
10010693
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578p.fdx
java 21548 kailashnath 861r REG 8,17 400
10010915
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578q.nvd
java 21548 kailashnath 862r REG 8,17 1183
10010905
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578q_es090_0.doc
java 21548 kailashnath 863r REG 8,17 2578
10010906
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578q_es090_0.pos
java 21548 kailashnath 864r REG 8,17 34
10010907
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578q_es090_0.pay
java 21548 kailashnath 865r REG 8,17 517
10010912
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578q_es090_0.tim
java 21548 kailashnath 866r REG 8,17 590
10010692
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578q.fdt
java 21548 kailashnath 867r REG 8,17 45
10010691
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578q.fdx
java 21548 kailashnath 869r REG 8,17 412
9971197
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/_9w36.nvd
java 21548 kailashnath 870r REG 8,17 4116
9971130
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/_9w36_es090_0.doc
java 21548 kailashnath 871r REG 8,17 4306
9971133
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/_9w36_es090_0.pos
java 21548 kailashnath 872r REG 8,17 34
9971191
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/_9w36_es090_0.pay
java 21548 kailashnath 873r REG 8,17 13628
9971192
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/_9w36_es090_0.tim
java 21548 kailashnath 874r REG 8,17 19938
9971127
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/_9w36.fdt
java 21548 kailashnath 875r REG 8,17 55
9971117
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/_9w36.fdx
java 21548 kailashnath 876r REG 8,17 136468
9971247
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdk_es090_0.pos
java 21548 kailashnath 877r REG 8,17 34
9971248
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdk_es090_0.pay
java 21548 kailashnath 878r REG 8,17 235767
9971249
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdk_es090_0.tim
java 21548 kailashnath 879r REG 8,17 670494
9971245
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdk.fdt
java 21548 kailashnath 880r REG 8,17 736
9971244
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdk.fdx
java 21548 kailashnath 881r REG 8,17 424
9971270
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdl.nvd
java 21548 kailashnath 882r REG 8,17 4285
9971242
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdl_es090_0.doc
java 21548 kailashnath 883r REG 8,17 4441
9971243
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdl_es090_0.pos
java 21548 kailashnath 884r REG 8,17 34
9971264
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdl_es090_0.pay
java 21548 kailashnath 885r REG 8,17 14708
9971266
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdl_es090_0.tim
java 21548 kailashnath 886r REG 8,17 21322
9971060
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdl.fdt
java 21548 kailashnath 887r REG 8,17 54
9971059
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdl.fdx
java 21548 kailashnath 889r REG 8,17 16262
9970705
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/_9wdr.nvd
java 21548 kailashnath 890r REG 8,17 155969
9970687
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/_9wdr_es090_0.doc
java 21548 kailashnath 910r REG 8,17 159939
9970689
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/_9wdr_es090_0.pos
java 21548 kailashnath 911r REG 8,17 34
9970690
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/_9wdr_es090_0.pay
java 21548 kailashnath 912r REG 8,17 341259
9970693
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/_9wdr_es090_0.tim
java 21548 kailashnath 913r REG 8,17 821871
9970644
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/_9wdr.fdt
java 21548 kailashnath 914r REG 8,17 804
9970641
/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/_9wdr.fdx

Not sure what is the issue

/K

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · September 5, 2013, 6:48am

How many index do you have?

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 5 sept. 2013 à 08:46, Kailashnath Reddy retadedgeek@gmail.com a écrit :

Hi David,
It's currently set to default 5.

Set the number of shards (splits) of an index (5 by default):

index.number_of_shards: 5

Set the number of replicas (additional copies) of an index (1 by default):

index.number_of_replicas: 1

Note, that for development on a local machine, with small indices, it usually

makes sense to "disable" the distributed features:

index.number_of_shards: 1

index.number_of_replicas: 0

/K

On Thursday, September 5, 2013 10:27:33 AM UTC+5:30, David Pilato wrote:

How many shards do you have on this node?

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 5 sept. 2013 à 06:01, Kailashnath Reddy retad...@gmail.com a écrit :

Recently we've been running migration and we see a lot of errors with elasticsearch, after sometime it just doesn't accept any more connections.

All I see in the logs is:

Caused by: java.io.FileNotFoundException: /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/4/index/_6t86.fdx (Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at org.apache.lucene.store.FSDirectory$FSIndexInput.(FSDirectory.java:409)
at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:123)
at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:80)
at org.apache.lucene.store.RateLimitedFSDirectory.openInput(RateLimitedFSDirectory.java:99)
at org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:436)
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.(CompressingStoredFieldsReader.java:98)
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsReader(CompressingStoredFieldsFormat.java:113)
at org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:147)
at org.apache.lucene.index.SegmentReader.(SegmentReader.java:56)
at org.apache.lucene.index.ReadersAndLiveDocs.getReader(ReadersAndLiveDocs.java:121)
at org.apache.lucene.index.ReadersAndLiveDocs.getReadOnlyClone(ReadersAndLiveDocs.java:218)
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:96)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:369)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:111)
at org.apache.lucene.search.SearcherManager.(SearcherManager.java:89)
at org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1412)
at org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:271)
... 7 more
[2013-09-05 05:52:11,722][WARN ][cluster.action.shard ] [Slave] sending failed shard for [verbose][4], node[mFU1CeLwSr6sa4Jb0sqZpQ], [R], s[INITIALIZING], reason [Failed to start shard, message [RecoveryFailedException[[verbose][4]: Recovery failed from [Master][4Zno5nzmQFqdh_o5TbGf5A][inet[/10.223.12.31:9300]]{master=true} into [Slave][mFU1CeLwSr6sa4Jb0sqZpQ][inet[/10.223.12.32:9300]]{master=false}]; nested: RemoteTransportException[[Master][inet[/10.223.12.31:9300]][index/shard/recovery/startRecovery]]; nested: RecoveryEngineException[[verbose][4] Phase[2] Execution failed]; nested: RemoteTransportException[[Slave][inet[/10.223.12.32:9300]][index/shard/recovery/prepareTranslog]]; nested: EngineCreationFailureException[[verbose][4] failed to open reader on writer]; nested: FileNotFoundException[/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/4/index/_6t86.fdx (Too many open files)]; ]]
FileNotFoundException[/opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/4/index/_6t86.fdx (Too many open files)]; ]]

The ulimit in production machines are 65535 and when I try to do lsof -a -p , I see something like this:

6291457 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/translog/translog-1369668960033
java 21548 kailashnath 848r REG 8,17 1941 10010832 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/_536h_es090_0.pos
java 21548 kailashnath 849r REG 8,17 15174 9971253 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdk.nvd
java 21548 kailashnath 850r REG 8,17 114071 9971246 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdk_es090_0.doc
java 21548 kailashnath 851r REG 8,17 2090 10010846 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/_536h_es090_0.pay
java 21548 kailashnath 852r REG 8,17 606 10010847 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/_536h_es090_0.tim
java 21548 kailashnath 853r REG 8,17 855 10010690 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/_536h.fdt
java 21548 kailashnath 854r REG 8,17 899 9970974 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mda.fdx
java 21548 kailashnath 855r REG 8,17 47 10010689 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/3/index/_536h.fdx
java 21548 kailashnath 856r REG 8,17 162098 10010696 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578p_es090_0.pos
java 21548 kailashnath 857r REG 8,17 34 10010702 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578p_es090_0.pay
java 21548 kailashnath 858r REG 8,17 391138 10010703 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578p_es090_0.tim
java 21548 kailashnath 859r REG 8,17 389565 10010694 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578p.fdt
java 21548 kailashnath 860r REG 8,17 158 10010693 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578p.fdx
java 21548 kailashnath 861r REG 8,17 400 10010915 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578q.nvd
java 21548 kailashnath 862r REG 8,17 1183 10010905 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578q_es090_0.doc
java 21548 kailashnath 863r REG 8,17 2578 10010906 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578q_es090_0.pos
java 21548 kailashnath 864r REG 8,17 34 10010907 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578q_es090_0.pay
java 21548 kailashnath 865r REG 8,17 517 10010912 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578q_es090_0.tim
java 21548 kailashnath 866r REG 8,17 590 10010692 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578q.fdt
java 21548 kailashnath 867r REG 8,17 45 10010691 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/users/1/index/_578q.fdx
java 21548 kailashnath 869r REG 8,17 412 9971197 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/_9w36.nvd
java 21548 kailashnath 870r REG 8,17 4116 9971130 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/_9w36_es090_0.doc
java 21548 kailashnath 871r REG 8,17 4306 9971133 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/_9w36_es090_0.pos
java 21548 kailashnath 872r REG 8,17 34 9971191 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/_9w36_es090_0.pay
java 21548 kailashnath 873r REG 8,17 13628 9971192 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/_9w36_es090_0.tim
java 21548 kailashnath 874r REG 8,17 19938 9971127 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/_9w36.fdt
java 21548 kailashnath 875r REG 8,17 55 9971117 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/1/index/_9w36.fdx
java 21548 kailashnath 876r REG 8,17 136468 9971247 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdk_es090_0.pos
java 21548 kailashnath 877r REG 8,17 34 9971248 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdk_es090_0.pay
java 21548 kailashnath 878r REG 8,17 235767 9971249 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdk_es090_0.tim
java 21548 kailashnath 879r REG 8,17 670494 9971245 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdk.fdt
java 21548 kailashnath 880r REG 8,17 736 9971244 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdk.fdx
java 21548 kailashnath 881r REG 8,17 424 9971270 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdl.nvd
java 21548 kailashnath 882r REG 8,17 4285 9971242 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdl_es090_0.doc
java 21548 kailashnath 883r REG 8,17 4441 9971243 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdl_es090_0.pos
java 21548 kailashnath 884r REG 8,17 34 9971264 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdl_es090_0.pay
java 21548 kailashnath 885r REG 8,17 14708 9971266 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdl_es090_0.tim
java 21548 kailashnath 886r REG 8,17 21322 9971060 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdl.fdt
java 21548 kailashnath 887r REG 8,17 54 9971059 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/2/index/_4mdl.fdx
java 21548 kailashnath 889r REG 8,17 16262 9970705 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/_9wdr.nvd
java 21548 kailashnath 890r REG 8,17 155969 9970687 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/_9wdr_es090_0.doc
java 21548 kailashnath 910r REG 8,17 159939 9970689 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/_9wdr_es090_0.pos
java 21548 kailashnath 911r REG 8,17 34 9970690 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/_9wdr_es090_0.pay
java 21548 kailashnath 912r REG 8,17 341259 9970693 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/_9wdr_es090_0.tim
java 21548 kailashnath 913r REG 8,17 821871 9970644 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/_9wdr.fdt
java 21548 kailashnath 914r REG 8,17 804 9970641 /opt/isystem/dbs/elasticsearch/Projectplace/nodes/0/indices/verbose/3/index/_9wdr.fdx

Not sure what is the issue

/K

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Kailashnath_Reddy · September 5, 2013, 6:59am

Hi David,
PFB the stats.
There are two indices, "users" and "verbose" as shown in the below json!

{
"ok" : true,
"_shards" : {
"total" : 20,
"successful" : 19,
"failed" : 0
},
"_all" : {
"primaries" : {
"docs" : {
"count" : 7090542,
"deleted" : 236979
},
"store" : {
"size" : "1.2gb",
"size_in_bytes" : 1376458012,
"throttle_time" : "0s",
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 1328383,
"index_time" : "1.7m",
"index_time_in_millis" : 106482,
"index_current" : 0,
"delete_total" : 0,
"delete_time" : "0s",
"delete_time_in_millis" : 0,
"delete_current" : 0
},
"get" : {
"total" : 0,
"get_time" : "0s",
"time_in_millis" : 0,
"exists_total" : 0,
"exists_time" : "0s",
"exists_time_in_millis" : 0,
"missing_total" : 0,
"missing_time" : "0s",
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 0,
"query_total" : 85,
"query_time" : "3.1s",
"query_time_in_millis" : 3135,
"query_current" : 0,
"fetch_total" : 32,
"fetch_time" : "92ms",
"fetch_time_in_millis" : 92,
"fetch_current" : 0
}
},
"total" : {
"docs" : {
"count" : 12944611,
"deleted" : 436068
},
"store" : {
"size" : "2.3gb",
"size_in_bytes" : 2506949989,
"throttle_time" : "0s",
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 2512365,
"index_time" : "3.4m",
"index_time_in_millis" : 204570,
"index_current" : 0,
"delete_total" : 0,
"delete_time" : "0s",
"delete_time_in_millis" : 0,
"delete_current" : 0
},
"get" : {
"total" : 0,
"get_time" : "0s",
"time_in_millis" : 0,
"exists_total" : 0,
"exists_time" : "0s",
"exists_time_in_millis" : 0,
"missing_total" : 0,
"missing_time" : "0s",
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 0,
"query_total" : 144,
"query_time" : "6.8s",
"query_time_in_millis" : 6867,
"query_current" : 0,
"fetch_total" : 53,
"fetch_time" : "360ms",
"fetch_time_in_millis" : 360,
"fetch_current" : 0
}
}
},
"indices" : {
"users" : {
"primaries" : {
"docs" : {
"count" : 60347,
"deleted" : 5751
},
"store" : {
"size" : "7.6mb",
"size_in_bytes" : 7979376,
"throttle_time" : "0s",
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 665000,
"index_time" : "30.3s",
"index_time_in_millis" : 30392,
"index_current" : 0,
"delete_total" : 0,
"delete_time" : "0s",
"delete_time_in_millis" : 0,
"delete_current" : 0
},
"get" : {
"total" : 0,
"get_time" : "0s",
"time_in_millis" : 0,
"exists_total" : 0,
"exists_time" : "0s",
"exists_time_in_millis" : 0,
"missing_total" : 0,
"missing_time" : "0s",
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 0,
"query_total" : 0,
"query_time" : "0s",
"query_time_in_millis" : 0,
"query_current" : 0,
"fetch_total" : 0,
"fetch_time" : "0s",
"fetch_time_in_millis" : 0,
"fetch_current" : 0
}
},
"total" : {
"docs" : {
"count" : 120691,
"deleted" : 10869
},
"store" : {
"size" : "16.2mb",
"size_in_bytes" : 17090438,
"throttle_time" : "0s",
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 1321446,
"index_time" : "58.7s",
"index_time_in_millis" : 58751,
"index_current" : 0,
"delete_total" : 0,
"delete_time" : "0s",
"delete_time_in_millis" : 0,
"delete_current" : 0
},
"get" : {
"total" : 0,
"get_time" : "0s",
"time_in_millis" : 0,
"exists_total" : 0,
"exists_time" : "0s",
"exists_time_in_millis" : 0,
"missing_total" : 0,
"missing_time" : "0s",
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 0,
"query_total" : 0,
"query_time" : "0s",
"query_time_in_millis" : 0,
"query_current" : 0,
"fetch_total" : 0,
"fetch_time" : "0s",
"fetch_time_in_millis" : 0,
"fetch_current" : 0
}
}
},
"verbose" : {
"primaries" : {
"docs" : {
"count" : 7030195,
"deleted" : 231228
},
"store" : {
"size" : "1.2gb",
"size_in_bytes" : 1368478636,
"throttle_time" : "0s",
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 663383,
"index_time" : "1.2m",
"index_time_in_millis" : 76090,
"index_current" : 0,
"delete_total" : 0,
"delete_time" : "0s",
"delete_time_in_millis" : 0,
"delete_current" : 0
},
"get" : {
"total" : 0,
"get_time" : "0s",
"time_in_millis" : 0,
"exists_total" : 0,
"exists_time" : "0s",
"exists_time_in_millis" : 0,
"missing_total" : 0,
"missing_time" : "0s",
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 0,
"query_total" : 85,
"query_time" : "3.1s",
"query_time_in_millis" : 3135,
"query_current" : 0,
"fetch_total" : 32,
"fetch_time" : "92ms",
"fetch_time_in_millis" : 92,
"fetch_current" : 0
}
},
"total" : {
"docs" : {
"count" : 12823920,
"deleted" : 425199
},
"store" : {
"size" : "2.3gb",
"size_in_bytes" : 2489859551,
"throttle_time" : "0s",
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 1190919,
"index_time" : "2.4m",
"index_time_in_millis" : 145819,
"index_current" : 0,
"delete_total" : 0,
"delete_time" : "0s",
"delete_time_in_millis" : 0,
"delete_current" : 0
},
"get" : {
"total" : 0,
"get_time" : "0s",
"time_in_millis" : 0,
"exists_total" : 0,
"exists_time" : "0s",
"exists_time_in_millis" : 0,
"missing_total" : 0,
"missing_time" : "0s",
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 0,
"query_total" : 144,
"query_time" : "6.8s",
"query_time_in_millis" : 6867,
"query_current" : 0,
"fetch_total" : 53,
"fetch_time" : "360ms",
"fetch_time_in_millis" : 360,
"fetch_current" : 0
}
}
}
}
}

/K

On Thu, Sep 5, 2013 at 12:18 PM, David Pilato david@pilato.fr wrote:

How many index do you have?

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 5 sept. 2013 à 08:46, Kailashnath Reddy retadedgeek@gmail.com a
écrit :

Hi David,
It's currently set to default 5.

Set the number of shards (splits) of an index (5 by default):

index.number_of_shards: 5

Set the number of replicas (additional copies) of an index (1 by

default):

index.number_of_replicas: 1

Note, that for development on a local machine, with small indices, it

usually

makes sense to "disable" the distributed features:

index.number_of_shards: 1

index.number_of_replicas: 0

/K

On Thursday, September 5, 2013 10:27:33 AM UTC+5:30, David Pilato wrote:

How many shards do you have on this node?

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 5 sept. 2013 à 06:01, Kailashnath Reddy retad...@gmail.com a écrit :

Recently we've been running migration and we see a lot of errors with
elasticsearch, after sometime it just doesn't accept any more connections.

All I see in the logs is:

Caused by: java.io.FileNotFoundException: /opt/isystem/dbs/**
elasticsearch/Projectplace/**nodes/0/indices/verbose/4/**index/6t86.fdx
(Too many open files)
at java.io.RandomAccessFile.open(**Native Method)
at java.io.RandomAccessFile.<**init>(RandomAccessFile.java:**233)
at org.apache.lucene.store.FSDirectory$FSIndexInput.<
init>(FSDirectory.java:409)
at org.apache.lucene.store.NIOFSDirectory$
NIOFSIndexInput.(**NIOFSDirectory.java:123)
at org.apache.lucene.store.NIOFSDirectory.openInput(
NIOFSDirectory.java:80)
at org.apache.lucene.store.**RateLimitedFSDirectory.*openInput(
*RateLimitedFSDirectory.java:99)
at org.elasticsearch.index.store.Store$StoreDirectory.
openInput(Store.java:436)
at org.apache.lucene.codecs.compressing.
CompressingStoredFieldsReader.(CompressingStoredFieldsReader.
java:98)
at org.apache.lucene.codecs.compressing.
CompressingStoredFieldsFormat.fieldsReader(
CompressingStoredFieldsFormat.**java:113)
at org.apache.lucene.index.SegmentCoreReaders.(
SegmentCoreReaders.java:147)
at org.apache.lucene.index.SegmentReader.(
SegmentReader.java:56)
at org.apache.lucene.index.ReadersAndLiveDocs.getReader(
ReadersAndLiveDocs.java:121)
at org.apache.lucene.index.ReadersAndLiveDocs.
getReadOnlyClone(**ReadersAndLiveDocs.java:218)
at org.apache.lucene.index.StandardDirectoryReader.open(
StandardDirectoryReader.java:**96)
at org.apache.lucene.index.IndexWriter.getReader(
IndexWriter.java:369)
at org.apache.lucene.index.DirectoryReader.open(
DirectoryReader.java:111)
at org.apache.lucene.search.SearcherManager.(
SearcherManager.java:89)
at org.elasticsearch.index.engine.robin.RobinEngine.
buildSearchManager(**RobinEngine.java:1412)
at org.elasticsearch.index.engine.robin.RobinEngine.
start(RobinEngine.java:271)
... 7 more
[2013-09-05 05:52:11,722][WARN ][cluster.action.shard ] [Slave]
sending failed shard for [verbose][4], node[mFU1CeLwSr6sa4Jb0sqZpQ], [R],
s[INITIALIZING], reason [Failed to start shard, message
[RecoveryFailedException[[**verbose][4]: Recovery failed from
[Master][4Zno5nzmQFqdh**o5TbGf5A][inet[/10.223.12.31:**9300]]{master=true}
into [Slave][**mFU1CeLwSr6sa4Jb0sqZpQ][inet[/**10.223.12.32:9300
]]{master=**false}]; nested: RemoteTransportException[[**Master][inet[/
10.223.12.31:**9300]][index/shard/recovery/**startRecovery]]; nested:
RecoveryEngineException[[**verbose][4] Phase[2] Execution failed];
nested: RemoteTransportException[[Slave][inet[/10.223.12.32:9300
]][index/shard/recovery/prepareTranslog]]; nested:
EngineCreationFailureException[[verbose][4] failed to open reader on
writer]; nested: FileNotFoundException[/opt/*isystem/dbs/elasticsearch/
*Projectplace/nodes/0/indices/**verbose/4/index/_6t86.fdx (Too many open
files)]; ]]
FileNotFoundException[/opt/isystem/dbs/elasticsearch/
Projectplace/nodes/0/indices/**verbose/4/index/_6t86.fdx (Too many open
files)]; ]]

The ulimit in production machines are 65535 and when I try to do lsof -a
-p , I see something like this:

6291457 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/users/3/translog/translog-1369668960033
java 21548 kailashnath 848r REG 8,17 1941
10010832 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/users/3/index/_536h_es090_0.pos
java 21548 kailashnath 849r REG 8,17 15174
9971253 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/verbose/2/index/_4mdk.nvd
java 21548 kailashnath 850r REG 8,17 114071
9971246 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/verbose/2/index/_4mdk_es090_0.doc
java 21548 kailashnath 851r REG 8,17 2090
10010846 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/users/3/index/_536h_es090_0.pay
java 21548 kailashnath 852r REG 8,17 606
10010847 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/users/3/index/_536h_es090_0.tim
java 21548 kailashnath 853r REG 8,17 855
10010690 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/users/3/index/_536h.fdt
java 21548 kailashnath 854r REG 8,17 899
9970974 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/verbose/2/index/_4mda.fdx
java 21548 kailashnath 855r REG 8,17 47
10010689 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/users/3/index/_536h.fdx
java 21548 kailashnath 856r REG 8,17 162098
10010696 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/users/1/index/_578p_es090_0.pos
java 21548 kailashnath 857r REG 8,17 34
10010702 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/users/1/index/_578p_es090_0.pay
java 21548 kailashnath 858r REG 8,17 391138
10010703 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/users/1/index/_578p_es090_0.tim
java 21548 kailashnath 859r REG 8,17 389565
10010694 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/users/1/index/_578p.fdt
java 21548 kailashnath 860r REG 8,17 158
10010693 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/users/1/index/_578p.fdx
java 21548 kailashnath 861r REG 8,17 400
10010915 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/users/1/index/_578q.nvd
java 21548 kailashnath 862r REG 8,17 1183
10010905 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/users/1/index/_578q_es090_0.doc
java 21548 kailashnath 863r REG 8,17 2578
10010906 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/users/1/index/_578q_es090_0.pos
java 21548 kailashnath 864r REG 8,17 34
10010907 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/users/1/index/_578q_es090_0.pay
java 21548 kailashnath 865r REG 8,17 517
10010912 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/users/1/index/_578q_es090_0.tim
java 21548 kailashnath 866r REG 8,17 590
10010692 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/users/1/index/_578q.fdt
java 21548 kailashnath 867r REG 8,17 45
10010691 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/users/1/index/_578q.fdx
java 21548 kailashnath 869r REG 8,17 412
9971197 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/verbose/1/**index/_9w36.nvd
java 21548 kailashnath 870r REG 8,17 4116
9971130 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/verbose/1/**index/_9w36_es090_0.doc
java 21548 kailashnath 871r REG 8,17 4306
9971133 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/verbose/1/**index/_9w36_es090_0.pos
java 21548 kailashnath 872r REG 8,17 34
9971191 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/verbose/1/**index/_9w36_es090_0.pay
java 21548 kailashnath 873r REG 8,17 13628
9971192 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/verbose/1/**index/_9w36_es090_0.tim
java 21548 kailashnath 874r REG 8,17 19938
9971127 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/verbose/1/**index/_9w36.fdt
java 21548 kailashnath 875r REG 8,17 55
9971117 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/verbose/1/**index/_9w36.fdx
java 21548 kailashnath 876r REG 8,17 136468
9971247 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/verbose/2/**index/_4mdk_es090_0.pos
java 21548 kailashnath 877r REG 8,17 34
9971248 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/verbose/2/**index/_4mdk_es090_0.pay
java 21548 kailashnath 878r REG 8,17 235767
9971249 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/verbose/2/**index/_4mdk_es090_0.tim
java 21548 kailashnath 879r REG 8,17 670494
9971245 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/verbose/2/**index/_4mdk.fdt
java 21548 kailashnath 880r REG 8,17 736
9971244 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/verbose/2/**index/_4mdk.fdx
java 21548 kailashnath 881r REG 8,17 424
9971270 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/verbose/2/**index/_4mdl.nvd
java 21548 kailashnath 882r REG 8,17 4285
9971242 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/verbose/2/**index/_4mdl_es090_0.doc
java 21548 kailashnath 883r REG 8,17 4441
9971243 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/verbose/2/**index/_4mdl_es090_0.pos
java 21548 kailashnath 884r REG 8,17 34
9971264 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/verbose/2/**index/_4mdl_es090_0.pay
java 21548 kailashnath 885r REG 8,17 14708
9971266 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/verbose/2/**index/_4mdl_es090_0.tim
java 21548 kailashnath 886r REG 8,17 21322
9971060 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/verbose/2/**index/_4mdl.fdt
java 21548 kailashnath 887r REG 8,17 54
9971059 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/verbose/2/**index/_4mdl.fdx
java 21548 kailashnath 889r REG 8,17 16262
9970705 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/verbose/3/**index/_9wdr.nvd
java 21548 kailashnath 890r REG 8,17 155969
9970687 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/verbose/3/**index/_9wdr_es090_0.doc
java 21548 kailashnath 910r REG 8,17 159939
9970689 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/verbose/3/**index/_9wdr_es090_0.pos
java 21548 kailashnath 911r REG 8,17 34
9970690 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/verbose/3/**index/_9wdr_es090_0.pay
java 21548 kailashnath 912r REG 8,17 341259
9970693 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/verbose/3/**index/_9wdr_es090_0.tim
java 21548 kailashnath 913r REG 8,17 821871
9970644 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/verbose/3/**index/_9wdr.fdt
java 21548 kailashnath 914r REG 8,17 804
9970641 /opt/isystem/dbs/elasticsearch/Projectplace/
nodes/0/indices/verbose/3/**index/_9wdr.fdx

Not sure what is the issue

/K

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@**googlegroups.com.
For more options, visit https://groups.google.com/**groups/opt_out https://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/Vs5jSFY9xBw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

jprante · September 5, 2013, 7:14am

Check with curl localhost:9200/_cluster/nodes/stats?process the current
number of open files.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Kailashnath_Reddy · September 5, 2013, 7:44am

Hi Jorg,
PFB the response. It seems to be stable now, but still I seriously
doubt what could have been the reason.

Restarted both the master/slave ES instances!

{
"cluster_name": "xxxxxxxxxx",
"nodes": {
"rnlGU2e-QiOWACCtXsmi5Q": {
"timestamp": 1378366972509,
"name": "Slave",
"transport_address": "inet[/10.223.12.32:9300]",
"hostname": "pp-feedsdb-2",
"attributes": {
"master": "false"
},
"indices": {
"docs": {
"count": 6020937,
"deleted": 203431
},
"store": {
"size": "1gb",
"size_in_bytes": 1159436704,
"throttle_time": "0s",
"throttle_time_in_millis": 0
},
"indexing": {
"index_total": 1574046,
"index_time": "2.1m",
"index_time_in_millis": 130263,
"index_current": 0,
"delete_total": 0,
"delete_time": "0s",
"delete_time_in_millis": 0,
"delete_current": 0
},
"get": {
"total": 0,
"get_time": "0s",
"time_in_millis": 0,
"exists_total": 0,
"exists_time": "0s",
"exists_time_in_millis": 0,
"missing_total": 0,
"missing_time": "0s",
"missing_time_in_millis": 0,
"current": 0
},
"search": {
"open_contexts": 0,
"query_total": 75,
"query_time": "4.4s",
"query_time_in_millis": 4412,
"query_current": 0,
"fetch_total": 31,
"fetch_time": "738ms",
"fetch_time_in_millis": 738,
"fetch_current": 0
}
},
"process": {
"timestamp": 1378366972509,
"open_file_descriptors": 862,
"cpu": {
"percent": 1,
"sys": "27 seconds and 950 milliseconds",
"sys_in_millis": 27950,
"user": "4 minutes, 51 seconds and 310 milliseconds",
"user_in_millis": 291310,
"total": "5 minutes, 19 seconds and 260 milliseconds",
"total_in_millis": 319260
},
"mem": {
"resident": "389.2mb",
"resident_in_bytes": 408113152,
"share": "10.1mb",
"share_in_bytes": 10625024,
"total_virtual": "4.3gb",
"total_virtual_in_bytes": 4683698176
}
}
},
"FpLkq6O6SviFsA9PXINnWw": {
"timestamp": 1378366972506,
"name": "Master",
"transport_address": "inet[/10.223.12.31:9300]",
"hostname": "pp-feedsdb-1",
"attributes": {
"master": "true"
},
"indices": {
"docs": {
"count": 7299347,
"deleted": 241932
},
"store": {
"size": "1.3gb",
"size_in_bytes": 1408533151,
"throttle_time": "0s",
"throttle_time_in_millis": 0
},
"indexing": {
"index_total": 1745124,
"index_time": "2.2m",
"index_time_in_millis": 136104,
"index_current": 0,
"delete_total": 0,
"delete_time": "0s",
"delete_time_in_millis": 0,
"delete_current": 0
},
"get": {
"total": 0,
"get_time": "0s",
"time_in_millis": 0,
"exists_total": 0,
"exists_time": "0s",
"exists_time_in_millis": 0,
"missing_total": 0,
"missing_time": "0s",
"missing_time_in_millis": 0,
"current": 0
},
"search": {
"open_contexts": 0,
"query_total": 110,
"query_time": "3.9s",
"query_time_in_millis": 3922,
"query_current": 0,
"fetch_total": 48,
"fetch_time": "159ms",
"fetch_time_in_millis": 159,
"fetch_current": 0
}
},
"process": {
"timestamp": 1378366972506,
"open_file_descriptors": 1004,
"cpu": {
"percent": 2,
"sys": "34 seconds and 370 milliseconds",
"sys_in_millis": 34370,
"user": "5 minutes, 28 seconds and 710 milliseconds",
"user_in_millis": 328710,
"total": "6 minutes, 3 seconds and 80 milliseconds",
"total_in_millis": 363080
},
"mem": {
"resident": "397.2mb",
"resident_in_bytes": 416587776,
"share": "10.2mb",
"share_in_bytes": 10788864,
"total_virtual": "4.3gb",
"total_virtual_in_bytes": 4688175104
}
}
}
}
}

On Thu, Sep 5, 2013 at 12:44 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Check with curl localhost:9200/_cluster/nodes/stats?process the current
number of open files.

Jörg

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/Vs5jSFY9xBw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · September 5, 2013, 7:50am

Did you run an _optimize or something?
Or did you launch a big "Delete" query?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 5 sept. 2013 à 09:44, Kailash ź Reddy retadedgeek@gmail.com a écrit :

Hi Jorg,
PFB the response. It seems to be stable now, but still I seriously doubt what could have been the reason.

Restarted both the master/slave ES instances!

{
"cluster_name": "xxxxxxxxxx",
"nodes": {
"rnlGU2e-QiOWACCtXsmi5Q": {
"timestamp": 1378366972509,
"name": "Slave",
"transport_address": "inet[/10.223.12.32:9300]",
"hostname": "pp-feedsdb-2",
"attributes": {
"master": "false"
},
"indices": {
"docs": {
"count": 6020937,
"deleted": 203431
},
"store": {
"size": "1gb",
"size_in_bytes": 1159436704,
"throttle_time": "0s",
"throttle_time_in_millis": 0
},
"indexing": {
"index_total": 1574046,
"index_time": "2.1m",
"index_time_in_millis": 130263,
"index_current": 0,
"delete_total": 0,
"delete_time": "0s",
"delete_time_in_millis": 0,
"delete_current": 0
},
"get": {
"total": 0,
"get_time": "0s",
"time_in_millis": 0,
"exists_total": 0,
"exists_time": "0s",
"exists_time_in_millis": 0,
"missing_total": 0,
"missing_time": "0s",
"missing_time_in_millis": 0,
"current": 0
},
"search": {
"open_contexts": 0,
"query_total": 75,
"query_time": "4.4s",
"query_time_in_millis": 4412,
"query_current": 0,
"fetch_total": 31,
"fetch_time": "738ms",
"fetch_time_in_millis": 738,
"fetch_current": 0
}
},
"process": {
"timestamp": 1378366972509,
"open_file_descriptors": 862,
"cpu": {
"percent": 1,
"sys": "27 seconds and 950 milliseconds",
"sys_in_millis": 27950,
"user": "4 minutes, 51 seconds and 310 milliseconds",
"user_in_millis": 291310,
"total": "5 minutes, 19 seconds and 260 milliseconds",
"total_in_millis": 319260
},
"mem": {
"resident": "389.2mb",
"resident_in_bytes": 408113152,
"share": "10.1mb",
"share_in_bytes": 10625024,
"total_virtual": "4.3gb",
"total_virtual_in_bytes": 4683698176
}
}
},
"FpLkq6O6SviFsA9PXINnWw": {
"timestamp": 1378366972506,
"name": "Master",
"transport_address": "inet[/10.223.12.31:9300]",
"hostname": "pp-feedsdb-1",
"attributes": {
"master": "true"
},
"indices": {
"docs": {
"count": 7299347,
"deleted": 241932
},
"store": {
"size": "1.3gb",
"size_in_bytes": 1408533151,
"throttle_time": "0s",
"throttle_time_in_millis": 0
},
"indexing": {
"index_total": 1745124,
"index_time": "2.2m",
"index_time_in_millis": 136104,
"index_current": 0,
"delete_total": 0,
"delete_time": "0s",
"delete_time_in_millis": 0,
"delete_current": 0
},
"get": {
"total": 0,
"get_time": "0s",
"time_in_millis": 0,
"exists_total": 0,
"exists_time": "0s",
"exists_time_in_millis": 0,
"missing_total": 0,
"missing_time": "0s",
"missing_time_in_millis": 0,
"current": 0
},
"search": {
"open_contexts": 0,
"query_total": 110,
"query_time": "3.9s",
"query_time_in_millis": 3922,
"query_current": 0,
"fetch_total": 48,
"fetch_time": "159ms",
"fetch_time_in_millis": 159,
"fetch_current": 0
}
},
"process": {
"timestamp": 1378366972506,
"open_file_descriptors": 1004,
"cpu": {
"percent": 2,
"sys": "34 seconds and 370 milliseconds",
"sys_in_millis": 34370,
"user": "5 minutes, 28 seconds and 710 milliseconds",
"user_in_millis": 328710,
"total": "6 minutes, 3 seconds and 80 milliseconds",
"total_in_millis": 363080
},
"mem": {
"resident": "397.2mb",
"resident_in_bytes": 416587776,
"share": "10.2mb",
"share_in_bytes": 10788864,
"total_virtual": "4.3gb",
"total_virtual_in_bytes": 4688175104
}
}
}
}
}

On Thu, Sep 5, 2013 at 12:44 PM, joergprante@gmail.com joergprante@gmail.com wrote:
Check with curl localhost:9200/_cluster/nodes/stats?process the current number of open files.

Jörg

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/Vs5jSFY9xBw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Kailashnath_Reddy · September 5, 2013, 7:56am

No, we didn't run any delete query or optimize at all and there are no
delete possibilities in the code too!

/K

On Thu, Sep 5, 2013 at 1:20 PM, David Pilato david@pilato.fr wrote:

Did you run an _optimize or something?
Or did you launch a big "Delete" query?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr
| @scrutmydocs https://twitter.com/scrutmydocs

Le 5 sept. 2013 à 09:44, Kailash ź Reddy retadedgeek@gmail.com a écrit :

Hi Jorg,
PFB the response. It seems to be stable now, but still I seriously
doubt what could have been the reason.

Restarted both the master/slave ES instances!

{
"cluster_name": "xxxxxxxxxx",
"nodes": {
"rnlGU2e-QiOWACCtXsmi5Q": {
"timestamp": 1378366972509,
"name": "Slave",
"transport_address": "inet[/10.223.12.32:9300]",
"hostname": "pp-feedsdb-2",
"attributes": {
"master": "false"
},
"indices": {
"docs": {
"count": 6020937,
"deleted": 203431
},
"store": {
"size": "1gb",
"size_in_bytes": 1159436704,
"throttle_time": "0s",
"throttle_time_in_millis": 0
},
"indexing": {
"index_total": 1574046,
"index_time": "2.1m",
"index_time_in_millis": 130263,
"index_current": 0,
"delete_total": 0,
"delete_time": "0s",
"delete_time_in_millis": 0,
"delete_current": 0
},
"get": {
"total": 0,
"get_time": "0s",
"time_in_millis": 0,
"exists_total": 0,
"exists_time": "0s",
"exists_time_in_millis": 0,
"missing_total": 0,
"missing_time": "0s",
"missing_time_in_millis": 0,
"current": 0
},
"search": {
"open_contexts": 0,
"query_total": 75,
"query_time": "4.4s",
"query_time_in_millis": 4412,
"query_current": 0,
"fetch_total": 31,
"fetch_time": "738ms",
"fetch_time_in_millis": 738,
"fetch_current": 0
}
},
"process": {
"timestamp": 1378366972509,
"open_file_descriptors": 862,
"cpu": {
"percent": 1,
"sys": "27 seconds and 950 milliseconds",
"sys_in_millis": 27950,
"user": "4 minutes, 51 seconds and 310 milliseconds",
"user_in_millis": 291310,
"total": "5 minutes, 19 seconds and 260 milliseconds",
"total_in_millis": 319260
},
"mem": {
"resident": "389.2mb",
"resident_in_bytes": 408113152,
"share": "10.1mb",
"share_in_bytes": 10625024,
"total_virtual": "4.3gb",
"total_virtual_in_bytes": 4683698176
}
}
},
"FpLkq6O6SviFsA9PXINnWw": {
"timestamp": 1378366972506,
"name": "Master",
"transport_address": "inet[/10.223.12.31:9300]",
"hostname": "pp-feedsdb-1",
"attributes": {
"master": "true"
},
"indices": {
"docs": {
"count": 7299347,
"deleted": 241932
},
"store": {
"size": "1.3gb",
"size_in_bytes": 1408533151,
"throttle_time": "0s",
"throttle_time_in_millis": 0
},
"indexing": {
"index_total": 1745124,
"index_time": "2.2m",
"index_time_in_millis": 136104,
"index_current": 0,
"delete_total": 0,
"delete_time": "0s",
"delete_time_in_millis": 0,
"delete_current": 0
},
"get": {
"total": 0,
"get_time": "0s",
"time_in_millis": 0,
"exists_total": 0,
"exists_time": "0s",
"exists_time_in_millis": 0,
"missing_total": 0,
"missing_time": "0s",
"missing_time_in_millis": 0,
"current": 0
},
"search": {
"open_contexts": 0,
"query_total": 110,
"query_time": "3.9s",
"query_time_in_millis": 3922,
"query_current": 0,
"fetch_total": 48,
"fetch_time": "159ms",
"fetch_time_in_millis": 159,
"fetch_current": 0
}
},
"process": {
"timestamp": 1378366972506,
"open_file_descriptors": 1004,
"cpu": {
"percent": 2,
"sys": "34 seconds and 370 milliseconds",
"sys_in_millis": 34370,
"user": "5 minutes, 28 seconds and 710 milliseconds",
"user_in_millis": 328710,
"total": "6 minutes, 3 seconds and 80 milliseconds",
"total_in_millis": 363080
},
"mem": {
"resident": "397.2mb",
"resident_in_bytes": 416587776,
"share": "10.2mb",
"share_in_bytes": 10788864,
"total_virtual": "4.3gb",
"total_virtual_in_bytes": 4688175104
}
}
}
}
}

On Thu, Sep 5, 2013 at 12:44 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Check with curl localhost:9200/_cluster/nodes/stats?process the current
number of open files.

Jörg

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/Vs5jSFY9xBw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · September 5, 2013, 8:01am

But you have many updates then.
It causes delete+insert behind the scene.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 5 sept. 2013 à 09:56, Kailash ź Reddy retadedgeek@gmail.com a écrit :

No, we didn't run any delete query or optimize at all and there are no delete possibilities in the code too!

/K

On Thu, Sep 5, 2013 at 1:20 PM, David Pilato david@pilato.fr wrote:
Did you run an _optimize or something?
Or did you launch a big "Delete" query?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 5 sept. 2013 à 09:44, Kailash ź Reddy retadedgeek@gmail.com a écrit :

Hi Jorg,
PFB the response. It seems to be stable now, but still I seriously doubt what could have been the reason.

Restarted both the master/slave ES instances!

{
"cluster_name": "xxxxxxxxxx",
"nodes": {
"rnlGU2e-QiOWACCtXsmi5Q": {
"timestamp": 1378366972509,
"name": "Slave",
"transport_address": "inet[/10.223.12.32:9300]",
"hostname": "pp-feedsdb-2",
"attributes": {
"master": "false"
},
"indices": {
"docs": {
"count": 6020937,
"deleted": 203431
},
"store": {
"size": "1gb",
"size_in_bytes": 1159436704,
"throttle_time": "0s",
"throttle_time_in_millis": 0
},
"indexing": {
"index_total": 1574046,
"index_time": "2.1m",
"index_time_in_millis": 130263,
"index_current": 0,
"delete_total": 0,
"delete_time": "0s",
"delete_time_in_millis": 0,
"delete_current": 0
},
"get": {
"total": 0,
"get_time": "0s",
"time_in_millis": 0,
"exists_total": 0,
"exists_time": "0s",
"exists_time_in_millis": 0,
"missing_total": 0,
"missing_time": "0s",
"missing_time_in_millis": 0,
"current": 0
},
"search": {
"open_contexts": 0,
"query_total": 75,
"query_time": "4.4s",
"query_time_in_millis": 4412,
"query_current": 0,
"fetch_total": 31,
"fetch_time": "738ms",
"fetch_time_in_millis": 738,
"fetch_current": 0
}
},
"process": {
"timestamp": 1378366972509,
"open_file_descriptors": 862,
"cpu": {
"percent": 1,
"sys": "27 seconds and 950 milliseconds",
"sys_in_millis": 27950,
"user": "4 minutes, 51 seconds and 310 milliseconds",
"user_in_millis": 291310,
"total": "5 minutes, 19 seconds and 260 milliseconds",
"total_in_millis": 319260
},
"mem": {
"resident": "389.2mb",
"resident_in_bytes": 408113152,
"share": "10.1mb",
"share_in_bytes": 10625024,
"total_virtual": "4.3gb",
"total_virtual_in_bytes": 4683698176
}
}
},
"FpLkq6O6SviFsA9PXINnWw": {
"timestamp": 1378366972506,
"name": "Master",
"transport_address": "inet[/10.223.12.31:9300]",
"hostname": "pp-feedsdb-1",
"attributes": {
"master": "true"
},
"indices": {
"docs": {
"count": 7299347,
"deleted": 241932
},
"store": {
"size": "1.3gb",
"size_in_bytes": 1408533151,
"throttle_time": "0s",
"throttle_time_in_millis": 0
},
"indexing": {
"index_total": 1745124,
"index_time": "2.2m",
"index_time_in_millis": 136104,
"index_current": 0,
"delete_total": 0,
"delete_time": "0s",
"delete_time_in_millis": 0,
"delete_current": 0
},
"get": {
"total": 0,
"get_time": "0s",
"time_in_millis": 0,
"exists_total": 0,
"exists_time": "0s",
"exists_time_in_millis": 0,
"missing_total": 0,
"missing_time": "0s",
"missing_time_in_millis": 0,
"current": 0
},
"search": {
"open_contexts": 0,
"query_total": 110,
"query_time": "3.9s",
"query_time_in_millis": 3922,
"query_current": 0,
"fetch_total": 48,
"fetch_time": "159ms",
"fetch_time_in_millis": 159,
"fetch_current": 0
}
},
"process": {
"timestamp": 1378366972506,
"open_file_descriptors": 1004,
"cpu": {
"percent": 2,
"sys": "34 seconds and 370 milliseconds",
"sys_in_millis": 34370,
"user": "5 minutes, 28 seconds and 710 milliseconds",
"user_in_millis": 328710,
"total": "6 minutes, 3 seconds and 80 milliseconds",
"total_in_millis": 363080
},
"mem": {
"resident": "397.2mb",
"resident_in_bytes": 416587776,
"share": "10.2mb",
"share_in_bytes": 10788864,
"total_virtual": "4.3gb",
"total_virtual_in_bytes": 4688175104
}
}
}
}
}

On Thu, Sep 5, 2013 at 12:44 PM, joergprante@gmail.com joergprante@gmail.com wrote:
Check with curl localhost:9200/_cluster/nodes/stats?process the current number of open files.

Jörg

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/Vs5jSFY9xBw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

jprante · September 5, 2013, 8:28am

During migration, the segments were rebuilt behind the scenes, with
deletions and inserts during renumbering.

The current file descriptor usage is low, and during the migration it was
certainly higher.

Please recheck if ulimit setting of 65535 is really active in the current
process. You need to set it in /etc/security.d/limits.conf and not only in
ulimit, and a full re-login of the ES user and re-start of the ES process.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Kailashnath_Reddy · September 5, 2013, 9:18am

Hi Jorg,
I did set the limits in limits.conf very early on:

           hard    nofile          70000

           soft    nofile          70000

/K

On Thu, Sep 5, 2013 at 1:58 PM, joergprante@gmail.com <joergprante@gmail.com

wrote:

During migration, the segments were rebuilt behind the scenes, with
deletions and inserts during renumbering.

The current file descriptor usage is low, and during the migration it was
certainly higher.

Please recheck if ulimit setting of 65535 is really active in the current
process. You need to set it in /etc/security.d/limits.conf and not only in
ulimit, and a full re-login of the ES user and re-start of the ES process.

Jörg

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/Vs5jSFY9xBw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

jprante · September 5, 2013, 2:03pm

The "nofile" setting with an asterisk (*) may not work (you set the default
value only). The Elasticsearch user must be set in limits.conf.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Kailashnath_Reddy · September 8, 2013, 8:47am

Sorry for getting back late, sure will try that Jörg, thanks!

On Thu, Sep 5, 2013 at 7:33 PM, joergprante@gmail.com <joergprante@gmail.com

wrote:

The "nofile" setting with an asterisk (*) may not work (you set the
default value only). The Elasticsearch user must be set in limits.conf.

Jörg

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/Vs5jSFY9xBw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.