Hi,
all was working fine till few days ago. than i started to get this message: Request Timeout after 1500ms.
i can't find any info about it in the log files.
how can i debug this problem?
thanks,
guy
Where did you get this error?
There's not much for us in your post to go on unfortunately 
You should look in your Elasticsearch logs at the time that this happened.
is that help?
at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:241)
at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
[2016-05-04 12:39:03,915][DEBUG][action.admin.indices.stats] [Fever Pitch] failed to execute [indices:monitor/stats] on node [s3_zNTUiS0OeJWkVkWr40A]
RemoteTransportException[[Fever Pitch][localhost/127.0.0.1:9300][indices:monitor/stats[n]]]; nested: OutOfMemoryError[Java heap space];
Caused by: java.lang.OutOfMemoryError: Java heap space
this is the "top" output, how come it's Out Of Memory?
KiB Mem : 16171692 total, 10265008 free, 2295560 used, 3611124 buff/cache
KiB Swap: 8191996 total, 8191996 free, 0 used. 13473676 avail Mem
Probably because you did not allocate enough memory to elasticsearch?
You should review your settings and/or your architecture.
i changed the HEAP_SIZE with the commend: export ES_HEAP_SIZE=10g and it solved the problem,
Thanks.
hi,
i was happy to soon...
at org.elasticsearch.action.bulk.TransportBulkAction$2.onFailure(TransportBulkAction.java:351)
at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:95)
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:567)
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.retry(TransportReplicationAction.java:540)
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleException(TransportReplicationAction.java:525)
at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:855)
at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:833)
at org.elasticsearch.transport.DelegatingTransportChannel.sendResponse(DelegatingTransportChannel.java:68)
at org.elasticsearch.transport.RequestHandlerRegistry$TransportChannelWrapper.sendResponse(RequestHandlerRegistry.java:146)
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.finishAsFailed(TransportReplicationAction.java:730)
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:656)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:279)
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:271)
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
i will be grateful for some clue.
thanks,
guy
What do you have in elasticsearch logs? (Please format them).
the tail of the elasticsearch log:
java.lang.OutOfMemoryError: Java heap space
[2016-05-05 12:38:16,966][DEBUG][action.admin.indices.stats] [Chimera] [indices:monitor/stats] failed to execute operation for shard [[logstash-2016.04.15][0], node[pO0v0tCfQ922FVoXxU5Org], [P], v[58], s[STARTED], a[id=rlt48r71QPe4Dbj5RBRF-g]]
java.lang.OutOfMemoryError: Java heap space
[2016-05-05 12:38:17,070][DEBUG][action.fieldstats ] [Chimera] [logstash-2016.04.01][1], node[pO0v0tCfQ922FVoXxU5Org], [P], v[66], s[STARTED], a[id=hKsdWcElThGlyPRSxOfmKg]: failed to execute [org.elasticsearch.action.fieldstats.FieldStatsRequest@607e173e]
RemoteTransportException[[Chimera][localhost/127.0.0.1:9300][indices:data/read/field_stats[s]]]; nested: OutOfMemoryError[Java heap space];
Caused by: java.lang.OutOfMemoryError: Java heap space
[2016-05-05 12:38:21,532][INFO ][index.shard ] [Chimera] [logstash-2016.05.05][2] ignoring recovery of a corrupt translog entry
MapperParsingException[failed to parse]; nested: OutOfMemoryError[Java heap space];
at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:154)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:309)
at org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:529)
at org.elasticsearch.index.shard.TranslogRecoveryPerformer.performRecoveryOperation(TranslogRecoveryPerformer.java:175)
at org.elasticsearch.index.shard.TranslogRecoveryPerformer.recoveryFromSnapshot(TranslogRecoveryPerformer.java:107)
at org.elasticsearch.index.shard.IndexShard$1.recoveryFromSnapshot(IndexShard.java:1584)
at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:238)
at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:174)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1515)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1499)
at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:972)
at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:944)
at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:241)
at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
[2016-05-05 12:38:21,532][WARN ][indices.ttl ] [Chimera] failed to execute ttl purge
java.lang.OutOfMemoryError: Java heap space
Wow, lots of problems there
Don't use TTL, you have a few OOMs and you also appear to have shard failures.
what should i do then? what is the TTL?
help, someone?
It looks like you are still having issues with memory, can you increase the heap more?
same results...
How much heap do you have now, how much data in the cluster, how many indices and how many shards?
one cluster, about 80G Data
epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1462783198 11:39:58 elasticsearch red 1 1 266 266 0 4 277 2 1.5s 48.6%
help, someone? something i can do?
Not sure if entirely relevant but have a read of New ELK env. stopping working