Hi mailling list ! How are you today 
I have an issue (I think with elasticsearch) after that, graylog2 crash -_-
the elasticsearch logs:
[2012-03-26 16:36:43,875][DEBUG][action.bulk ] [elasticsearch-master] [graylog2][3] failed to bulk item (index) index {[graylog2][message][BEREdZawSUeX7NuVg4Clag], source[{"_IpSource":"10.12.18.26%10","host":"www.promovacances.com","_LoadBallancer":"10.12.8.27","full_message":"Mar 26 16:36:43 10.12.8.27 vs=vs-fo1-http80 cip=174.35.7.23%10 sip=10.12.18.26%10 sp=10000 vh=www.promovacances.com uri=/showsearchresults-weathernews?destinationCountries=55 m=GET v=1.1 s=200 l= t=22 r=http://www.promovacances.com/vacances/vacances-et-sejours/voyage/?departureCities1=1188&departureCities=1188&destinationCountries1=54&destinationCountries=54&departureDateDay=14&departureDateMonthYear=01%2F04%2F2012&departureDate=14%2F04%2F2012&departureDateRange=3&priceRange=&numberOfNightsRanges1=%5B0-%3E6%29&numberOfNightsRanges=%5B0-%3E6%29&theme=7&sejList=false ua="Mozilla/5.0 (compatible; Panther)"","_SourcePort":"10000","line":139,"_VirtualServer":"vs-fo1-http80","message":"Mar 26 16:36:43 10.12.8.27 vs=vs-fo1-http80 cip=174.35.7.23%10 sip=10.12.18.26%10 sp=10000 vh=www.promovacances.com uri=/showsearchresults-weathernews?destinationCountries=55 m=GET v=1.1 s=200 l= t=22 r=http://www.promovacances.com/vacances/vacances-et-sejours/voyage/?departureCities1=1188&departureCities=1188&destinationCountries1=54&destinationCountries=54&departureDateDay=14&departureDateMonthYear=01%2F04%2F2012&departureDate=14%2F04%2F2012&departureDateRange=3&priceRange=&numberOfNightsRanges1=%5B0-%3E6%29&numberOfNightsRanges=%5B0-%3E6%29&theme=7&sejList=false ua="Mozilla/5.0 (compatible; Panther)"","_Methode":"GET","_Uri":"/showsearchresults-weathernews?destinationCountries=55","_StatusHttp":"200","_ReponseTime":"22","_VirtualHost":"www.promovacances.com","level":7,"_UserAgent":""Mozilla/5.0 (compatible; Panther)"","facility":"F5Access","file":"file:/data/logstash/logstash.jar!/logstash/outputs/gelf.rb","_AccessUrl":"http://www.promovacances.com/vacances/vacances-et-sejours/voyage/?departureCities1=1188&departureCities=1188&destinationCountries1=54&destinationCountries=54&departureDateDay=14&departureDateMonthYear=01%2F04%2F2012&departureDate=14%2F04%2F2012&departureDateRange=3&priceRange=&numberOfNightsRanges1=%5B0-%3E6%29&numberOfNightsRanges=%5B0-%3E6%29&theme=7&sejList=false","created_at":1.332772603E9,"_VersionHttp":"1.1","streams":,"_IpClient":"174.35.7.23%10","_Timestamp":"Mar 26 16:36:43"}]}
org.elasticsearch.index.engine.CreateFailedEngineException: [graylog2][3]Create failedfor [message#BEREdZawSUeX7NuVg4Clag]
at org.elasticsearch.index.engine.robin.RobinEngine.create(RobinEngine.java:362)
at org.elasticsearch.index.shard.service.InternalIndexShard.create(InternalIndexShard.java:301)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:163)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:529)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:427)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.io.FileNotFoundException:/data/elasticsearch/data/elasticsearch-graylog2/nodes/0/indices/graylog2/3/index/_1l63.fdt (Trop de fichiers ouverts)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:441)
at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:306)
at org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:418)
at org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:390)
at org.apache.lucene.index.FieldsWriter.(FieldsWriter.java:83)
at org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:65)
at org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:108)
at org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:152)
at org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1404)
at org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1424)
at org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1043)
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:772)
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2066)
at org.elasticsearch.index.engine.robin.RobinEngine.innerCreate(RobinEngine.java:462)
at org.elasticsearch.index.engine.robin.RobinEngine.create(RobinEngine.java:357)
...7 more
[2012-03-26 16:36:44,602][WARN ][netty.channel.socket.nio.NioServerSocketPipelineSink]Failed to accept a connection.
java.io.IOException:Trop de fichiers ouverts
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:236)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
[2012-03-26 16:36:44,770][WARN ][netty.channel.socket.nio.NioServerSocketPipelineSink]Failed to accept a connection.
java.io.IOException:Trop de fichiers ouverts
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:236)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
[2012-03-26 16:36:45,771][WARN ][netty.channel.socket.nio.NioServerSocketPipelineSink]Failed to accept a connection.
java.io.IOException:Trop de fichiers ouverts
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:236)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
[2012-03-26 16:36:46,602][WARN ][netty.channel.socket.nio.NioServerSocketPipelineSink]Failed to accept a connection.
java.io.IOException:Trop de fichiers ouverts
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:236)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
the graylog2 server log :
2012-03-26 16:40:49,931 DEBUG: org.graylog2.messagehandlers.gelf.SimpleGELFClientHandler - Handling ZLIB compressedSimpleGELFClient
2012-03-26 16:40:49,931 DEBUG: org.graylog2.messagehandlers.gelf.SimpleGELFClientHandler - Got GELF message: shortMessage: Mar 26 16:40:49 10.12.8.27 vs=vs-fo1-http80 cip=193.17.19.232%10 sip=cache sp=706 vh=www.promovacances.com uri=/resources/promovacances/static/images/product2/pictos/star.gif m=GET v=1.1 s=2o3 l=310 t=0 r=http:/ (...)
2012-03-26 16:40:49,970 WARN : org.graylog2.indexer.Indexer - IO error when trying to index messages
java.io.IOException: Error writing to server
at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:577)
at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:589)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1160)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:397)
at org.graylog2.indexer.Indexer.bulkIndex(Indexer.java:138)
at org.graylog2.periodical.BulkIndexerThread.run(BulkIndexerThread.java:63)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2012-03-26 16:40:49,970 WARN : org.graylog2.indexer.Indexer - Indexer response code was not 200, but 0
2012-03-26 16:40:49,970 INFO : org.graylog2.periodical.BulkIndexerThread - About to index max 4000 messages. You have a total of 126 messages in the queue.[freq:1s]
2012-03-26 16:40:49,970 DEBUG: org.graylog2.messagequeue.MessageQueue - Read 126 messages from queue.
2012-03-26 16:40:49,970 INFO : org.graylog2.periodical.BulkIndexerThread - ... indexing 126 messages.
2012-03-26 16:40:49,986 WARN : org.graylog2.indexer.Indexer - IO error when trying to index messages
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:185)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:688)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:653)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1162)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:397)
at org.graylog2.indexer.Indexer.bulkIndex(Indexer.java:138)
at org.graylog2.periodical.BulkIndexerThread.run(BulkIndexerThread.java:63)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2012-03-26 16:40:49,986 WARN : org.graylog2.indexer.Indexer - Indexer response code was not 200, but 0
2012-03-26 16:40:49,986 INFO : org.graylog2.periodical.BulkIndexerThread - About to index max 4000 messages. You have a total of 0 messages in the queue.[freq:1s]
Any idea ?
--
Karavel - Thibault Desaules
Administrateur Systèmes & Réseau (Apprenti)
Mail : tdesaules@karavel.com
Tel : +33 (0)170 36 48 01
Mobile : +33 (0)699 38 40 55
Hi Thibault,
Seems like the open files limit has been reached. Can you try to
increase it?
We've added this to the startup script, before loading Elasticsearch:
ulimit -n 64000
On Mar 26, 5:42 pm, Thibault Desaules tdesau...@karavel.com wrote:
Hi mailling list ! How are you today
I have an issue (I think with elasticsearch) after that, graylog2 crash -_-
the elasticsearch logs:
[2012-03-26 16:36:43,875][DEBUG][action.bulk ] [elasticsearch-master] [graylog2][3] failed to bulk item (index) index {[graylog2][message][BEREdZawSUeX7NuVg4Clag], source[{"_IpSource":"10.12.18.26%10","host":"www.promovacances.com","_LoadBallancer":"10.12.8.27","full_message":"Mar 26 16:36:43 10.12.8.27 vs=vs-fo1-http80 cip=174.35.7.23%10 sip=10.12.18.26%10 sp=10000 vh=www.promovacances.comuri=/showsearchresults-weathernews?destinationCountries=55 m=GET v=1.1 s=200 l= t=22 r=http://www.promovacances.com/vacances/vacances-et-sejours/voyage/?departureCities1=1188&departureCities=1188&destinationCountries1=54&destinationCountries=54&departureDateDay=14&departureDateMonthYear=01%2F04%2F2012&departureDate=14%2F04%2F2012&departureDateRange=3&priceRange=&numberOfNightsRanges1=%5B0-%3E6%29&numberOfNightsRanges=%5B0-%3E6%29&theme=7&sejList=false ua="Mozilla/5.0 (compatible; Panther)"","_SourcePort":"10000","line":139,"_VirtualServer":"vs-fo1-http80","message":"Mar 26 16:36:43 10.12.8.27 vs=vs-fo1-http80 cip=174.35.7.23%10 sip=10.12.18.26%10 sp=10000 vh=www.promovacances.comuri=/showsearchresults-weathernews?destinationCountries=55 m=GET v=1.1 s=200 l= t=22 r=http://www.promovacances.com/vacances/vacances-et-sejours/voyage/?departureCities1=1188&departureCities=1188&destinationCountries1=54&destinationCountries=54&departureDateDay=14&departureDateMonthYear=01%2F04%2F2012&departureDate=14%2F04%2F2012&departureDateRange=3&priceRange=&numberOfNightsRanges1=%5B0-%3E6%29&numberOfNightsRanges=%5B0-%3E6%29&theme=7&sejList=false ua="Mozilla/5.0 (compatible; Panther)"","_Methode":"GET","_Uri":"/showsearchresults-weathernews?destinationCountries=55","_StatusHttp":"200","_ReponseTime":"22","_VirtualHost":"www.promovacances.com","level":7,"_UserAgent":""Mozilla/5.0 (compatible; Panther)"","facility":"F5Access","file":"file:/data/logstash/logstash.jar!/logstash/outputs/gelf.rb","_AccessUrl":"http://www.promovacances.com/vacances/vacances-et-sejours/voyage/?departureCities1=1188&departureCities=1188&destinationCountries1=54&destinationCountries=54&departureDateDay=14&departureDateMonthYear=01%2F04%2F2012&departureDate=14%2F04%2F2012&departureDateRange=3&priceRange=&numberOfNightsRanges1=%5B0-%3E6%29&numberOfNightsRanges=%5B0-%3E6%29&theme=7&sejList=false","created_at":1.332772603E9,"_VersionHttp":"1.1","streams":,"_IpClient":"174.35.7.23%10","_Timestamp":"Mar 26 16:36:43"}]}
org.elasticsearch.index.engine.CreateFailedEngineException: [graylog2][3]Create failedfor [message#BEREdZawSUeX7NuVg4Clag]
at org.elasticsearch.index.engine.robin.RobinEngine.create(RobinEngine.java:362)
at org.elasticsearch.index.shard.service.InternalIndexShard.create(InternalIndexShard.java:301)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:163)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:529)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:427)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.io.FileNotFoundException:/data/elasticsearch/data/elasticsearch-graylog2/nodes/0/indices/graylog2/3/index/_1l63.fdt (Trop de fichiers ouverts)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:441)
at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:306)
at org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:418)
at org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:390)
at org.apache.lucene.index.FieldsWriter.(FieldsWriter.java:83)
at org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:65)
at org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:108)
at org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:152)
at org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1404)
at org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1424)
at org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1043)
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:772)
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2066)
at org.elasticsearch.index.engine.robin.RobinEngine.innerCreate(RobinEngine.java:462)
at org.elasticsearch.index.engine.robin.RobinEngine.create(RobinEngine.java:357)
...7 more
[2012-03-26 16:36:44,602][WARN ][netty.channel.socket.nio.NioServerSocketPipelineSink]Failed to accept a connection.
java.io.IOException:Trop de fichiers ouverts
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:236)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
[2012-03-26 16:36:44,770][WARN ][netty.channel.socket.nio.NioServerSocketPipelineSink]Failed to accept a connection.
java.io.IOException:Trop de fichiers ouverts
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:236)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
[2012-03-26 16:36:45,771][WARN ][netty.channel.socket.nio.NioServerSocketPipelineSink]Failed to accept a connection.
java.io.IOException:Trop de fichiers ouverts
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:236)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
[2012-03-26 16:36:46,602][WARN ][netty.channel.socket.nio.NioServerSocketPipelineSink]Failed to accept a connection.
java.io.IOException:Trop de fichiers ouverts
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:236)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
the graylog2 server log :
2012-03-26 16:40:49,931 DEBUG: org.graylog2.messagehandlers.gelf.SimpleGELFClientHandler - Handling ZLIB compressedSimpleGELFClient
2012-03-26 16:40:49,931 DEBUG: org.graylog2.messagehandlers.gelf.SimpleGELFClientHandler - Got GELF message: shortMessage: Mar 26 16:40:49 10.12.8.27 vs=vs-fo1-http80 cip=193.17.19.232%10 sip=cache sp=706 vh=www.promovacances.com uri=/resources/promovacances/static/images/product2/pictos/star.gif m=GET v=1.1 s=2o3 l=310 t=0 r=http:/ (...)
2012-03-26 16:40:49,970 WARN : org.graylog2.indexer.Indexer - IO error when trying to index messages
java.io.IOException: Error writing to server
at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.j...)
at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.j...)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection....)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:397)
at org.graylog2.indexer.Indexer.bulkIndex(Indexer.java:138)
at org.graylog2.periodical.BulkIndexerThread.run(BulkIndexerThread.java:63)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2012-03-26 16:40:49,970 WARN : org.graylog2.indexer.Indexer - Indexer response code was not 200, but 0
2012-03-26 16:40:49,970 INFO : org.graylog2.periodical.BulkIndexerThread - About to index max 4000 messages. You have a total of 126 messages in the queue.[freq:1s]
2012-03-26 16:40:49,970 DEBUG: org.graylog2.messagequeue.MessageQueue - Read 126 messages from queue.
2012-03-26 16:40:49,970 INFO : org.graylog2.periodical.BulkIndexerThread - ... indexing 126 messages.
2012-03-26 16:40:49,986 WARN : org.graylog2.indexer.Indexer - IO error when trying to index messages
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:185)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:688)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:653)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection....)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:397)
at org.graylog2.indexer.Indexer.bulkIndex(Indexer.java:138)
at org.graylog2.periodical.BulkIndexerThread.run(BulkIndexerThread.java:63)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2012-03-26 16:40:49,986 WARN : org.graylog2.indexer.Indexer - Indexer response code was not 200, but 0
2012-03-26 16:40:49,986 INFO : org.graylog2.periodical.BulkIndexerThread - About to index max 4000 messages. You have a total of 0 messages in the queue.[freq:1s]
Any idea ?
--
Karavel - Thibault Desaules
Administrateur Systèmes & Réseau (Apprenti)
Mail : tdesau...@karavel.com
Tel : +33 (0)170 36 48 01
Mobile : +33 (0)699 38 40 55
it seems good
thanks !
Karavel - Thibault Desaules
Administrateur Systèmes & Réseau (Apprenti)
Mail : tdesaules@karavel.com
Tel : +33 (0)170 36 48 01
Mobile : +33 (0)699 38 40 55
Radu Gheorghe a écrit:
Hi Thibault,
Seems like the open files limit has been reached. Can you try to
increase it?
We've added this to the startup script, before loading Elasticsearch:
ulimit -n 64000
On Mar 26, 5:42 pm, Thibault Desaulestdesau...@karavel.com wrote:
Hi mailling list ! How are you today
I have an issue (I think with elasticsearch) after that, graylog2 crash -_-
the elasticsearch logs:
[2012-03-26 16:36:43,875][DEBUG][action.bulk ] [elasticsearch-master] [graylog2][3] failed to bulk item (index) index {[graylog2][message][BEREdZawSUeX7NuVg4Clag], source[{"_IpSource":"10.12.18.26%10","host":"www.promovacances.com","_LoadBallancer":"10.12.8.27","full_message":"Mar 26 16:36:43 10.12.8.27 vs=vs-fo1-http80 cip=174.35.7.23%10 sip=10.12.18.26%10 sp=10000 vh=www.promovacances.comuri=/showsearchresults-weathernews?destinationCountries=55 m=GET v=1.1 s=200 l= t=22 r=http://www.promovacances.com/vacances/vacances-et-sejours/voyage/?departureCities1=1188&departureCities=1188&destinationCountries1=54&destinationCountries=54&departureDateDay=14&departureDateMonthYear=01%2F04%2F2012&departureDate=14%2F04%2F2012&departureDateRange=3&priceRange=&numberOfNightsRanges1=%5B0-%3E6%29&numberOfNightsRanges=%5B0-%3E6%29&theme=7&sejList=false ua="Mozilla/5.0 (compatible; Panther)"","_SourcePort":"10000","line":139,"_VirtualServer":"vs-fo1-http80","message":"Mar 26 16:36:43 10.12.8.27 vs=vs-fo1-http80 cip=174.35.7.23%10 sip=10.12.18.26%10 sp=10000 vh=www.promovacances.comuri=/showsearchresults-weathernews?destinationCountries=55 m=GET v=1.1 s=200 l= t=22 r=http://www.promovacances.com/vacances/vacances-et-sejours/voyage/?departureCities1=1188&departureCities=1188&destinationCountries1=54&destinationCountries=54&departureDateDay=14&departureDateMonthYear=01%2F04%2F2012&departureDate=14%2F04%2F2012&departureDateRange=3&priceRange=&numberOfNightsRanges1=%5B0-%3E6%29&numberOfNightsRanges=%5B0-%3E6%29&theme=7&sejList=false ua="Mozilla/5.0 (compatible; Panther)"","_Methode":"GET","_Uri":"/showsearchresults-weathernews?destinationCountries=55","_StatusHttp":"200","_ReponseTime":"22","_VirtualHost":"www.promovacances.com","level":7,"_UserAgent":""Mozilla/5.0 (compatible; Panther)"","facility":"F5Access","file":"file:/data/logstash/logstash.jar!/logstash/outputs/gelf.rb","_AccessUrl":"http://www.promovacances.com/vacances/vacances-et-sejours/voyage/?departureCities1=1188&departureCities=1188&destinationCountries1=54&destinationCountries=54&departureDateDay=14&departureDateMonthYear=01%2F04%2F2012&departureDate=14%2F04%2F2012&departureDateRange=3&priceRange=&numberOfNightsRanges1=%5B0-%3E6%29&numberOfNightsRanges=%5B0-%3E6%29&theme=7&sejList=false","created_at":1.332772603E9,"_VersionHttp":"1.1","streams":,"_IpClient":"174.35.7.23%10","_Timestamp":"Mar 26 16:36:43"}]}
org.elasticsearch.index.engine.CreateFailedEngineException: [graylog2][3]Create failedfor [message#BEREdZawSUeX7NuVg4Clag]
at org.elasticsearch.index.engine.robin.RobinEngine.create(RobinEngine.java:362)
at org.elasticsearch.index.shard.service.InternalIndexShard.create(InternalIndexShard.java:301)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:163)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:529)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:427)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.io.FileNotFoundException:/data/elasticsearch/data/elasticsearch-graylog2/nodes/0/indices/graylog2/3/index/_1l63.fdt (Trop de fichiers ouverts)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:441)
at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:306)
at org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:418)
at org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:390)
at org.apache.lucene.index.FieldsWriter.(FieldsWriter.java:83)
at org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:65)
at org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:108)
at org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:152)
at org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1404)
at org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1424)
at org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1043)
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:772)
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2066)
at org.elasticsearch.index.engine.robin.RobinEngine.innerCreate(RobinEngine.java:462)
at org.elasticsearch.index.engine.robin.RobinEngine.create(RobinEngine.java:357)
...7 more
[2012-03-26 16:36:44,602][WARN ][netty.channel.socket.nio.NioServerSocketPipelineSink]Failed to accept a connection.
java.io.IOException:Trop de fichiers ouverts
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:236)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
[2012-03-26 16:36:44,770][WARN ][netty.channel.socket.nio.NioServerSocketPipelineSink]Failed to accept a connection.
java.io.IOException:Trop de fichiers ouverts
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:236)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
[2012-03-26 16:36:45,771][WARN ][netty.channel.socket.nio.NioServerSocketPipelineSink]Failed to accept a connection.
java.io.IOException:Trop de fichiers ouverts
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:236)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
[2012-03-26 16:36:46,602][WARN ][netty.channel.socket.nio.NioServerSocketPipelineSink]Failed to accept a connection.
java.io.IOException:Trop de fichiers ouverts
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:236)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
the graylog2 server log :
2012-03-26 16:40:49,931 DEBUG: org.graylog2.messagehandlers.gelf.SimpleGELFClientHandler - Handling ZLIB compressedSimpleGELFClient
2012-03-26 16:40:49,931 DEBUG: org.graylog2.messagehandlers.gelf.SimpleGELFClientHandler - Got GELF message: shortMessage: Mar 26 16:40:49 10.12.8.27 vs=vs-fo1-http80 cip=193.17.19.232%10 sip=cache sp=706 vh=www.promovacances.com uri=/resources/promovacances/static/images/product2/pictos/star.gif m=GET v=1.1 s=2o3 l=310 t=0 r=http:/ (...)
2012-03-26 16:40:49,970 WARN : org.graylog2.indexer.Indexer - IO error when trying to index messages
java.io.IOException: Error writing to server
at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.j...)
at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.j...)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection....)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:397)
at org.graylog2.indexer.Indexer.bulkIndex(Indexer.java:138)
at org.graylog2.periodical.BulkIndexerThread.run(BulkIndexerThread.java:63)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2012-03-26 16:40:49,970 WARN : org.graylog2.indexer.Indexer - Indexer response code was not 200, but 0
2012-03-26 16:40:49,970 INFO : org.graylog2.periodical.BulkIndexerThread - About to index max 4000 messages. You have a total of 126 messages in the queue.[freq:1s]
2012-03-26 16:40:49,970 DEBUG: org.graylog2.messagequeue.MessageQueue - Read 126 messages from queue.
2012-03-26 16:40:49,970 INFO : org.graylog2.periodical.BulkIndexerThread - ... indexing 126 messages.
2012-03-26 16:40:49,986 WARN : org.graylog2.indexer.Indexer - IO error when trying to index messages
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:185)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:688)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:653)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection....)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:397)
at org.graylog2.indexer.Indexer.bulkIndex(Indexer.java:138)
at org.graylog2.periodical.BulkIndexerThread.run(BulkIndexerThread.java:63)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2012-03-26 16:40:49,986 WARN : org.graylog2.indexer.Indexer - Indexer response code was not 200, but 0
2012-03-26 16:40:49,986 INFO : org.graylog2.periodical.BulkIndexerThread - About to index max 4000 messages. You have a total of 0 messages in the queue.[freq:1s]
Any idea ?
--
Karavel - Thibault Desaules
Administrateur Systèmes& Réseau (Apprenti)
Mail : tdesau...@karavel.com
Tel : +33 (0)170 36 48 01
Mobile : +33 (0)699 38 40 55