Debug " Request Timeout after 1500ms"


(Mark Walkom) #2

Where did you get this error?
There's not much for us in your post to go on unfortunately :frowning:


(Guy) #3

Hi,

attached the most update error massage:


(Mark Walkom) #4

You should look in your Elasticsearch logs at the time that this happened.


(Guy) #5

is that help?

at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:241)
at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
[2016-05-04 12:39:03,915][DEBUG][action.admin.indices.stats] [Fever Pitch] failed to execute [indices:monitor/stats] on node [s3_zNTUiS0OeJWkVkWr40A]
RemoteTransportException[[Fever Pitch][localhost/127.0.0.1:9300][indices:monitor/stats[n]]]; nested: OutOfMemoryError[Java heap space];
Caused by: java.lang.OutOfMemoryError: Java heap space


(Guy) #6

this is the "top" output, how come it's Out Of Memory?

KiB Mem : 16171692 total, 10265008 free, 2295560 used, 3611124 buff/cache
KiB Swap: 8191996 total, 8191996 free, 0 used. 13473676 avail Mem


(David Pilato) #7

Probably because you did not allocate enough memory to elasticsearch?

You should review your settings and/or your architecture.


(Guy) #8

i changed the HEAP_SIZE with the commend: export ES_HEAP_SIZE=10g and it solved the problem,

Thanks.


(Guy) #9

hi,

i was happy to soon...

at org.elasticsearch.action.bulk.TransportBulkAction$2.onFailure(TransportBulkAction.java:351)
at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:95)
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:567)
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.retry(TransportReplicationAction.java:540)
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleException(TransportReplicationAction.java:525)
at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:855)
at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:833)
at org.elasticsearch.transport.DelegatingTransportChannel.sendResponse(DelegatingTransportChannel.java:68)
at org.elasticsearch.transport.RequestHandlerRegistry$TransportChannelWrapper.sendResponse(RequestHandlerRegistry.java:146)
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.finishAsFailed(TransportReplicationAction.java:730)
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:656)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:279)
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:271)
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

i will be grateful for some clue.

thanks,
guy


(David Pilato) #10

What do you have in elasticsearch logs? (Please format them).


(Guy) #11

the tail of the elasticsearch log:

java.lang.OutOfMemoryError: Java heap space
[2016-05-05 12:38:16,966][DEBUG][action.admin.indices.stats] [Chimera] [indices:monitor/stats] failed to execute operation for shard [[logstash-2016.04.15][0], node[pO0v0tCfQ922FVoXxU5Org], [P], v[58], s[STARTED], a[id=rlt48r71QPe4Dbj5RBRF-g]]
java.lang.OutOfMemoryError: Java heap space
[2016-05-05 12:38:17,070][DEBUG][action.fieldstats ] [Chimera] [logstash-2016.04.01][1], node[pO0v0tCfQ922FVoXxU5Org], [P], v[66], s[STARTED], a[id=hKsdWcElThGlyPRSxOfmKg]: failed to execute [org.elasticsearch.action.fieldstats.FieldStatsRequest@607e173e]
RemoteTransportException[[Chimera][localhost/127.0.0.1:9300][indices:data/read/field_stats[s]]]; nested: OutOfMemoryError[Java heap space];
Caused by: java.lang.OutOfMemoryError: Java heap space
[2016-05-05 12:38:21,532][INFO ][index.shard ] [Chimera] [logstash-2016.05.05][2] ignoring recovery of a corrupt translog entry
MapperParsingException[failed to parse]; nested: OutOfMemoryError[Java heap space];
at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:154)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:309)
at org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:529)
at org.elasticsearch.index.shard.TranslogRecoveryPerformer.performRecoveryOperation(TranslogRecoveryPerformer.java:175)
at org.elasticsearch.index.shard.TranslogRecoveryPerformer.recoveryFromSnapshot(TranslogRecoveryPerformer.java:107)
at org.elasticsearch.index.shard.IndexShard$1.recoveryFromSnapshot(IndexShard.java:1584)
at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:238)
at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:174)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1515)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1499)
at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:972)
at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:944)
at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:241)
at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
[2016-05-05 12:38:21,532][WARN ][indices.ttl ] [Chimera] failed to execute ttl purge
java.lang.OutOfMemoryError: Java heap space


(Mark Walkom) #12

Wow, lots of problems there :frowning:

Don't use TTL, you have a few OOMs and you also appear to have shard failures.


(Guy) #13

what should i do then? what is the TTL?


(Guy) #14

help, someone?


(Mark Walkom) #15

It looks like you are still having issues with memory, can you increase the heap more?


(Guy) #16

same results...


(Mark Walkom) #17

How much heap do you have now, how much data in the cluster, how many indices and how many shards?


(Guy) #18

one cluster, about 80G Data

epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1462783198 11:39:58 elasticsearch red 1 1 266 266 0 4 277 2 1.5s 48.6%


(Guy) #19

help, someone? something i can do?


(paul) #20

Not sure if entirely relevant but have a read of New ELK env. stopping working


(system) #21