Elasticsearch problem!


(Thibault Desaules) #1

Hi mailling list ! How are you today :slight_smile:

I have an issue (I think with elasticsearch) after that, graylog2 crash -_-

the elasticsearch logs:
[2012-03-26 16:36:43,875][DEBUG][action.bulk ] [elasticsearch-master] [graylog2][3] failed to bulk item (index) index {[graylog2][message][BEREdZawSUeX7NuVg4Clag], source[{"_IpSource":"10.12.18.26%10","host":"www.promovacances.com","_LoadBallancer":"10.12.8.27","full_message":"Mar 26 16:36:43 10.12.8.27 vs=vs-fo1-http80 cip=174.35.7.23%10 sip=10.12.18.26%10 sp=10000 vh=www.promovacances.com uri=/showsearchresults-weathernews?destinationCountries=55 m=GET v=1.1 s=200 l= t=22 r=http://www.promovacances.com/vacances/vacances-et-sejours/voyage/?departureCities1=1188&departureCities=1188&destinationCountries1=54&destinationCountries=54&departureDateDay=14&departureDateMonthYear=01%2F04%2F2012&departureDate=14%2F04%2F2012&departureDateRange=3&priceRange=&numberOfNightsRanges1=%5B0-%3E6%29&numberOfNightsRanges=%5B0-%3E6%29&theme=7&sejList=false ua="Mozilla/5.0 (compatible; Panther)"","_SourcePort":"10000","line":139,"_VirtualServer":"vs-fo1-http80","message":"Mar 26 16:36:43 10.12.8.27 vs=vs-fo1-http80 cip=174.35.7.23%10 sip=10.12.18.26%10 sp=10000 vh=www.promovacances.com uri=/showsearchresults-weathernews?destinationCountries=55 m=GET v=1.1 s=200 l= t=22 r=http://www.promovacances.com/vacances/vacances-et-sejours/voyage/?departureCities1=1188&departureCities=1188&destinationCountries1=54&destinationCountries=54&departureDateDay=14&departureDateMonthYear=01%2F04%2F2012&departureDate=14%2F04%2F2012&departureDateRange=3&priceRange=&numberOfNightsRanges1=%5B0-%3E6%29&numberOfNightsRanges=%5B0-%3E6%29&theme=7&sejList=false ua="Mozilla/5.0 (compatible; Panther)"","_Methode":"GET","_Uri":"/showsearchresults-weathernews?destinationCountries=55","_StatusHttp":"200","_ReponseTime":"22","_VirtualHost":"www.promovacances.com","level":7,"_UserAgent":""Mozilla/5.0 (compatible; Panther)"","facility":"F5Access","file":"file:/data/logstash/logstash.jar!/logstash/outputs/gelf.rb","_AccessUrl":"http://www.promovacances.com/vacances/vacances-et-sejours/voyage/?departureCities1=1188&departureCities=1188&destinationCountries1=54&destinationCountries=54&departureDateDay=14&departureDateMonthYear=01%2F04%2F2012&departureDate=14%2F04%2F2012&departureDateRange=3&priceRange=&numberOfNightsRanges1=[0->6)&numberOfNightsRanges=[0->6)&theme=7&sejList=false","created_at":1.332772603E9,"_VersionHttp":"1.1","streams":[],"_IpClient":"174.35.7.23","_Timestamp":"Mar 26 16:36:43"}]}
org.elasticsearch.index.engine.CreateFailedEngineException: [graylog2][3]Create failedfor [message#BEREdZawSUeX7NuVg4Clag]
at org.elasticsearch.index.engine.robin.RobinEngine.create(RobinEngine.java:362)
at org.elasticsearch.index.shard.service.InternalIndexShard.create(InternalIndexShard.java:301)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:163)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:529)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:427)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.io.FileNotFoundException:/data/elasticsearch/data/elasticsearch-graylog2/nodes/0/indices/graylog2/3/index/_1l63.fdt (Trop de fichiers ouverts)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:441)
at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:306)
at org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:418)
at org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:390)
at org.apache.lucene.index.FieldsWriter.(FieldsWriter.java:83)
at org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:65)
at org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:108)
at org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:152)
at org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1404)
at org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1424)
at org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1043)
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:772)
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2066)
at org.elasticsearch.index.engine.robin.RobinEngine.innerCreate(RobinEngine.java:462)
at org.elasticsearch.index.engine.robin.RobinEngine.create(RobinEngine.java:357)
...7 more
[2012-03-26 16:36:44,602][WARN ][netty.channel.socket.nio.NioServerSocketPipelineSink]Failed to accept a connection.
java.io.IOException:Trop de fichiers ouverts
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:236)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
[2012-03-26 16:36:44,770][WARN ][netty.channel.socket.nio.NioServerSocketPipelineSink]Failed to accept a connection.
java.io.IOException:Trop de fichiers ouverts
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:236)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
[2012-03-26 16:36:45,771][WARN ][netty.channel.socket.nio.NioServerSocketPipelineSink]Failed to accept a connection.
java.io.IOException:Trop de fichiers ouverts
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:236)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
[2012-03-26 16:36:46,602][WARN ][netty.channel.socket.nio.NioServerSocketPipelineSink]Failed to accept a connection.
java.io.IOException:Trop de fichiers ouverts
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:236)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
the graylog2 server log :
2012-03-26 16:40:49,931 DEBUG: org.graylog2.messagehandlers.gelf.SimpleGELFClientHandler - Handling ZLIB compressedSimpleGELFClient
2012-03-26 16:40:49,931 DEBUG: org.graylog2.messagehandlers.gelf.SimpleGELFClientHandler - Got GELF message: shortMessage: Mar 26 16:40:49 10.12.8.27 vs=vs-fo1-http80 cip=193.17.19.232%10 sip=cache sp=706 vh=www.promovacances.com uri=/resources/promovacances/static/images/product2/pictos/star.gif m=GET v=1.1 s=2o3 l=310 t=0 r=http:/ (...)
2012-03-26 16:40:49,970 WARN : org.graylog2.indexer.Indexer - IO error when trying to index messages
java.io.IOException: Error writing to server
at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:577)
at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:589)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1160)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:397)
at org.graylog2.indexer.Indexer.bulkIndex(Indexer.java:138)
at org.graylog2.periodical.BulkIndexerThread.run(BulkIndexerThread.java:63)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2012-03-26 16:40:49,970 WARN : org.graylog2.indexer.Indexer - Indexer response code was not 200, but 0
2012-03-26 16:40:49,970 INFO : org.graylog2.periodical.BulkIndexerThread - About to index max 4000 messages. You have a total of 126 messages in the queue.[freq:1s]
2012-03-26 16:40:49,970 DEBUG: org.graylog2.messagequeue.MessageQueue - Read 126 messages from queue.
2012-03-26 16:40:49,970 INFO : org.graylog2.periodical.BulkIndexerThread - ... indexing 126 messages.
2012-03-26 16:40:49,986 WARN : org.graylog2.indexer.Indexer - IO error when trying to index messages
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:185)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:688)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:653)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1162)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:397)
at org.graylog2.indexer.Indexer.bulkIndex(Indexer.java:138)
at org.graylog2.periodical.BulkIndexerThread.run(BulkIndexerThread.java:63)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2012-03-26 16:40:49,986 WARN : org.graylog2.indexer.Indexer - Indexer response code was not 200, but 0
2012-03-26 16:40:49,986 INFO : org.graylog2.periodical.BulkIndexerThread - About to index max 4000 messages. You have a total of 0 messages in the queue.[freq:1s]
Any idea ?

--
Karavel - Thibault Desaules
Administrateur Systèmes & Réseau (Apprenti)
Mail : tdesaules@karavel.com
Tel : +33 (0)170 36 48 01
Mobile : +33 (0)699 38 40 55


(Radu Gheorghe) #2

Hi Thibault,

Seems like the open files limit has been reached. Can you try to
increase it?

We've added this to the startup script, before loading Elasticsearch:

ulimit -n 64000

On Mar 26, 5:42 pm, Thibault Desaules tdesau...@karavel.com wrote:

Hi mailling list ! How are you today :slight_smile:

I have an issue (I think with elasticsearch) after that, graylog2 crash -_-

the elasticsearch logs:
[2012-03-26 16:36:43,875][DEBUG][action.bulk ] [elasticsearch-master] [graylog2][3] failed to bulk item (index) index {[graylog2][message][BEREdZawSUeX7NuVg4Clag], source[{"_IpSource":"10.12.18.26%10","host":"www.promovacances.com","_LoadBallancer":"10.12.8.27","full_message":"Mar 26 16:36:43 10.12.8.27 vs=vs-fo1-http80 cip=174.35.7.23%10 sip=10.12.18.26%10 sp=10000 vh=www.promovacances.comuri=/showsearchresults-weathernews?destinationCountries=55 m=GET v=1.1 s=200 l= t=22 r=http://www.promovacances.com/vacances/vacances-et-sejours/voyage/?departureCities1=1188&departureCities=1188&destinationCountries1=54&destinationCountries=54&departureDateDay=14&departureDateMonthYear=01%2F04%2F2012&departureDate=14%2F04%2F2012&departureDateRange=3&priceRange=&numberOfNightsRanges1=%5B0-%3E6%29&numberOfNightsRanges=%5B0-%3E6%29&theme=7&sejList=false ua="Mozilla/5.0 (compatible; Panther)"","_SourcePort":"10000","line":139,"_VirtualServer":"vs-fo1-http80","message":"Mar 26 16:36:43 10.12.8.27 vs=vs-fo1-http80 cip=174.35.7.23%10 sip=10.12.18.26%10 sp=10000 vh=www.promovacances.comuri=/showsearchresults-weathernews?destinationCountries=55 m=GET v=1.1 s=200 l= t=22 r=http://www.promovacances.com/vacances/vacances-et-sejours/voyage/?departureCities1=1188&departureCities=1188&destinationCountries1=54&destinationCountries=54&departureDateDay=14&departureDateMonthYear=01%2F04%2F2012&departureDate=14%2F04%2F2012&departureDateRange=3&priceRange=&numberOfNightsRanges1=%5B0-%3E6%29&numberOfNightsRanges=%5B0-%3E6%29&theme=7&sejList=false ua="Mozilla/5.0 (compatible; Panther)"","_Methode":"GET","_Uri":"/showsearchresults-weathernews?destinationCountries=55","_StatusHttp":"200","_ReponseTime":"22","_VirtualHost":"www.promovacances.com","level":7,"_UserAgent":""Mozilla/5.0 (compatible; Panther)"","facility":"F5Access","file":"file:/data/logstash/logstash.jar!/logstash/outputs/gelf.rb","_AccessUrl":"http://www.promovacances.com/vacances/vacances-et-sejours/voyage/?departureCities1=1188&departureCities=1188&destinationCountries1=54&destinationCountries=54&departureDateDay=14&departureDateMonthYear=01%2F04%2F2012&departureDate=14%2F04%2F2012&departureDateRange=3&priceRange=&numberOfNightsRanges1=[0->6)&numberOfNightsRanges=[0->6)&theme=7&sejList=false","created_at":1.332772603E9,"_VersionHttp":"1.1","streams":[],"_IpClient":"174.35.7.23","_Timestamp":"Mar 26 16:36:43"}]}
org.elasticsearch.index.engine.CreateFailedEngineException: [graylog2][3]Create failedfor [message#BEREdZawSUeX7NuVg4Clag]
at org.elasticsearch.index.engine.robin.RobinEngine.create(RobinEngine.java:362)
at org.elasticsearch.index.shard.service.InternalIndexShard.create(InternalIndexShard.java:301)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:163)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:529)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:427)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.io.FileNotFoundException:/data/elasticsearch/data/elasticsearch-graylog2/nodes/0/indices/graylog2/3/index/_1l63.fdt (Trop de fichiers ouverts)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:441)
at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:306)
at org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:418)
at org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:390)
at org.apache.lucene.index.FieldsWriter.(FieldsWriter.java:83)
at org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:65)
at org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:108)
at org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:152)
at org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1404)
at org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1424)
at org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1043)
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:772)
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2066)
at org.elasticsearch.index.engine.robin.RobinEngine.innerCreate(RobinEngine.java:462)
at org.elasticsearch.index.engine.robin.RobinEngine.create(RobinEngine.java:357)
...7 more
[2012-03-26 16:36:44,602][WARN ][netty.channel.socket.nio.NioServerSocketPipelineSink]Failed to accept a connection.
java.io.IOException:Trop de fichiers ouverts
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:236)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
[2012-03-26 16:36:44,770][WARN ][netty.channel.socket.nio.NioServerSocketPipelineSink]Failed to accept a connection.
java.io.IOException:Trop de fichiers ouverts
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:236)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
[2012-03-26 16:36:45,771][WARN ][netty.channel.socket.nio.NioServerSocketPipelineSink]Failed to accept a connection.
java.io.IOException:Trop de fichiers ouverts
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:236)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
[2012-03-26 16:36:46,602][WARN ][netty.channel.socket.nio.NioServerSocketPipelineSink]Failed to accept a connection.
java.io.IOException:Trop de fichiers ouverts
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:236)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
the graylog2 server log :
2012-03-26 16:40:49,931 DEBUG: org.graylog2.messagehandlers.gelf.SimpleGELFClientHandler - Handling ZLIB compressedSimpleGELFClient
2012-03-26 16:40:49,931 DEBUG: org.graylog2.messagehandlers.gelf.SimpleGELFClientHandler - Got GELF message: shortMessage: Mar 26 16:40:49 10.12.8.27 vs=vs-fo1-http80 cip=193.17.19.232%10 sip=cache sp=706 vh=www.promovacances.com uri=/resources/promovacances/static/images/product2/pictos/star.gif m=GET v=1.1 s=2o3 l=310 t=0 r=http:/ (...)
2012-03-26 16:40:49,970 WARN : org.graylog2.indexer.Indexer - IO error when trying to index messages
java.io.IOException: Error writing to server
at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.j...)
at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.j...)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection....)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:397)
at org.graylog2.indexer.Indexer.bulkIndex(Indexer.java:138)
at org.graylog2.periodical.BulkIndexerThread.run(BulkIndexerThread.java:63)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2012-03-26 16:40:49,970 WARN : org.graylog2.indexer.Indexer - Indexer response code was not 200, but 0
2012-03-26 16:40:49,970 INFO : org.graylog2.periodical.BulkIndexerThread - About to index max 4000 messages. You have a total of 126 messages in the queue.[freq:1s]
2012-03-26 16:40:49,970 DEBUG: org.graylog2.messagequeue.MessageQueue - Read 126 messages from queue.
2012-03-26 16:40:49,970 INFO : org.graylog2.periodical.BulkIndexerThread - ... indexing 126 messages.
2012-03-26 16:40:49,986 WARN : org.graylog2.indexer.Indexer - IO error when trying to index messages
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:185)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:688)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:653)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection....)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:397)
at org.graylog2.indexer.Indexer.bulkIndex(Indexer.java:138)
at org.graylog2.periodical.BulkIndexerThread.run(BulkIndexerThread.java:63)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2012-03-26 16:40:49,986 WARN : org.graylog2.indexer.Indexer - Indexer response code was not 200, but 0
2012-03-26 16:40:49,986 INFO : org.graylog2.periodical.BulkIndexerThread - About to index max 4000 messages. You have a total of 0 messages in the queue.[freq:1s]
Any idea ?

--
Karavel - Thibault Desaules
Administrateur Systèmes & Réseau (Apprenti)
Mail : tdesau...@karavel.com
Tel : +33 (0)170 36 48 01
Mobile : +33 (0)699 38 40 55


(Thibault Desaules) #3

it seems good :slight_smile:
thanks !
Karavel - Thibault Desaules
Administrateur Systèmes & Réseau (Apprenti)
Mail : tdesaules@karavel.com
Tel : +33 (0)170 36 48 01
Mobile : +33 (0)699 38 40 55

Radu Gheorghe a écrit:

Hi Thibault,

Seems like the open files limit has been reached. Can you try to
increase it?

We've added this to the startup script, before loading Elasticsearch:

ulimit -n 64000

On Mar 26, 5:42 pm, Thibault Desaulestdesau...@karavel.com wrote:

Hi mailling list ! How are you today :slight_smile:

I have an issue (I think with elasticsearch) after that, graylog2 crash -_-

the elasticsearch logs:
[2012-03-26 16:36:43,875][DEBUG][action.bulk ] [elasticsearch-master] [graylog2][3] failed to bulk item (index) index {[graylog2][message][BEREdZawSUeX7NuVg4Clag], source[{"_IpSource":"10.12.18.26%10","host":"www.promovacances.com","_LoadBallancer":"10.12.8.27","full_message":"Mar 26 16:36:43 10.12.8.27 vs=vs-fo1-http80 cip=174.35.7.23%10 sip=10.12.18.26%10 sp=10000 vh=www.promovacances.comuri=/showsearchresults-weathernews?destinationCountries=55 m=GET v=1.1 s=200 l= t=22 r=http://www.promovacances.com/vacances/vacances-et-sejours/voyage/?departureCities1=1188&departureCities=1188&destinationCountries1=54&destinationCountries=54&departureDateDay=14&departureDateMonthYear=01%2F04%2F2012&departureDate=14%2F04%2F2012&departureDateRange=3&priceRange=&numberOfNightsRanges1=%5B0-%3E6%29&numberOfNightsRanges=%5B0-%3E6%29&theme=7&sejList=false ua="Mozilla/5.0 (compatible; Panther)"","_SourcePort":"10000","line":139,"_VirtualServer":"vs-fo1-http80","message":"Mar 26 16:36:43 10.12.8.27 vs=vs-fo1-http80 cip=174.35.7.23%10 sip=10.12.18.26%10 sp=10000 vh=www.promovacances.comuri=/showsearchresults-weathernews?destinationCountries=55 m=GET v=1.1 s=200 l= t=22 r=http://www.promovacances.com/vacances/vacances-et-sejours/voyage/?departureCities1=1188&departureCities=1188&destinationCountries1=54&destinationCountries=54&departureDateDay=14&departureDateMonthYear=01%2F04%2F2012&departureDate=14%2F04%2F2012&departureDateRange=3&priceRange=&numberOfNightsRanges1=%5B0-%3E6%29&numberOfNightsRanges=%5B0-%3E6%29&theme=7&sejList=false ua="Mozilla/5.0 (compatible; Panther)"","_Methode":"GET","_Uri":"/showsearchresults-weathernews?destinationCountries=55","_StatusHttp":"200","_ReponseTime":"22","_VirtualHost":"www.promovacances.com","level":7,"_UserAgent":""Mozilla/5.0 (compatible; Panther)"","facility":"F5Access","file":"file:/data/logstash/logstash.jar!/logstash/outputs/gelf.rb","_AccessUrl":"http://www.promovacances.com/vacances/vacances-et-sejours/voyage/?departureCities1=1188&departureCities=1188&destinationCountries1=54&destinationCountries=54&departureDateDay=14&departureDateMonthYear=01%2F04%2F2012&departureDate=14%2F04%2F2012&departureDateRange=3&priceRange=&numberOfNightsRanges1=[0->6)&numberOfNightsRanges=[0->6)&theme=7&sejList=false","created_at":1.332772603E9,"_VersionHttp":"1.1","streams":[],"_IpClient":"174.35.7.23","_Timestamp":"Mar 26 16:36:43"}]}
org.elasticsearch.index.engine.CreateFailedEngineException: [graylog2][3]Create failedfor [message#BEREdZawSUeX7NuVg4Clag]
at org.elasticsearch.index.engine.robin.RobinEngine.create(RobinEngine.java:362)
at org.elasticsearch.index.shard.service.InternalIndexShard.create(InternalIndexShard.java:301)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:163)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:529)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:427)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.io.FileNotFoundException:/data/elasticsearch/data/elasticsearch-graylog2/nodes/0/indices/graylog2/3/index/_1l63.fdt (Trop de fichiers ouverts)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:441)
at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:306)
at org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:418)
at org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:390)
at org.apache.lucene.index.FieldsWriter.(FieldsWriter.java:83)
at org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:65)
at org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:108)
at org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:152)
at org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1404)
at org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1424)
at org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1043)
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:772)
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2066)
at org.elasticsearch.index.engine.robin.RobinEngine.innerCreate(RobinEngine.java:462)
at org.elasticsearch.index.engine.robin.RobinEngine.create(RobinEngine.java:357)
...7 more
[2012-03-26 16:36:44,602][WARN ][netty.channel.socket.nio.NioServerSocketPipelineSink]Failed to accept a connection.
java.io.IOException:Trop de fichiers ouverts
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:236)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
[2012-03-26 16:36:44,770][WARN ][netty.channel.socket.nio.NioServerSocketPipelineSink]Failed to accept a connection.
java.io.IOException:Trop de fichiers ouverts
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:236)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
[2012-03-26 16:36:45,771][WARN ][netty.channel.socket.nio.NioServerSocketPipelineSink]Failed to accept a connection.
java.io.IOException:Trop de fichiers ouverts
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:236)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
[2012-03-26 16:36:46,602][WARN ][netty.channel.socket.nio.NioServerSocketPipelineSink]Failed to accept a connection.
java.io.IOException:Trop de fichiers ouverts
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:236)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
the graylog2 server log :
2012-03-26 16:40:49,931 DEBUG: org.graylog2.messagehandlers.gelf.SimpleGELFClientHandler - Handling ZLIB compressedSimpleGELFClient
2012-03-26 16:40:49,931 DEBUG: org.graylog2.messagehandlers.gelf.SimpleGELFClientHandler - Got GELF message: shortMessage: Mar 26 16:40:49 10.12.8.27 vs=vs-fo1-http80 cip=193.17.19.232%10 sip=cache sp=706 vh=www.promovacances.com uri=/resources/promovacances/static/images/product2/pictos/star.gif m=GET v=1.1 s=2o3 l=310 t=0 r=http:/ (...)
2012-03-26 16:40:49,970 WARN : org.graylog2.indexer.Indexer - IO error when trying to index messages
java.io.IOException: Error writing to server
at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.j...)
at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.j...)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection....)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:397)
at org.graylog2.indexer.Indexer.bulkIndex(Indexer.java:138)
at org.graylog2.periodical.BulkIndexerThread.run(BulkIndexerThread.java:63)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2012-03-26 16:40:49,970 WARN : org.graylog2.indexer.Indexer - Indexer response code was not 200, but 0
2012-03-26 16:40:49,970 INFO : org.graylog2.periodical.BulkIndexerThread - About to index max 4000 messages. You have a total of 126 messages in the queue.[freq:1s]
2012-03-26 16:40:49,970 DEBUG: org.graylog2.messagequeue.MessageQueue - Read 126 messages from queue.
2012-03-26 16:40:49,970 INFO : org.graylog2.periodical.BulkIndexerThread - ... indexing 126 messages.
2012-03-26 16:40:49,986 WARN : org.graylog2.indexer.Indexer - IO error when trying to index messages
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:185)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:688)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:653)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection....)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:397)
at org.graylog2.indexer.Indexer.bulkIndex(Indexer.java:138)
at org.graylog2.periodical.BulkIndexerThread.run(BulkIndexerThread.java:63)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2012-03-26 16:40:49,986 WARN : org.graylog2.indexer.Indexer - Indexer response code was not 200, but 0
2012-03-26 16:40:49,986 INFO : org.graylog2.periodical.BulkIndexerThread - About to index max 4000 messages. You have a total of 0 messages in the queue.[freq:1s]
Any idea ?

--
Karavel - Thibault Desaules
Administrateur Systèmes& Réseau (Apprenti)
Mail : tdesau...@karavel.com
Tel : +33 (0)170 36 48 01
Mobile : +33 (0)699 38 40 55


(system) #4