# Is this stacktrace a reason for cluster instability? **URL:** \ **Category:** Elasticsearch\ **Created:** [March 19, 2014, 10:17am UTC](https://discuss.elastic.co/t/is-this-stacktrace-a-reason-for-cluster-instability/16458 "2014-03-19T10:17:23Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Jelle\_Smet](https://avatars.discourse-cdn.com/v4/letter/j/35a633/32.png) [@Jelle\_Smet](https://discuss.elastic.co/u/Jelle_Smet)\ **Post date:** [March 19, 2014, 10:17am UTC](https://discuss.elastic.co/t/is-this-stacktrace-a-reason-for-cluster-instability/16458/1 "2014-03-19T10:17:23Z")
Hi List, I'm running ES 1.0.1 in a 6 node configuration. Some days ago we have experienced instability issues with our cluster. At a certain moment no documents could be indexed anymore. After restarting the indexing processes (Logstash), indexing worked again for a short period of time only to stall again after a brief period. The ES cluster had to be restarted to restore normal behavior. After that, some (recent) index shards stayed unassigned. I dropped the replication value for those indexes to 0 in order to clear the unassigned shards. Enabling replication resulted again into unassigned shards for the impacted indexes. I have left the replication level to 0 for the troublesome indexes in order to clear the cluster status. Meanwhile, indexing and replication works again for any newly created indexes. What we noticed: - The logs didn't reveal any immediate cause. The only reported issue we saw in the logs prior to the incident is the below mentioned stack trace. Afterwards, we have countered this issue by creating a template which enforces the offending field to be treated as a string, which should prevent the below mentioned error. - Our ES collected metrics revealed a sudden drop of total number of Java threads at the exact same moment. My question: "Could the below mentioned stracktrace be the cause of any cluster instability?" Thanks, Jelle [2014-03-15 03:15:38,414][DEBUG][action.bulk] [xxxx-xxxxxxx-logs-001] [logstash-2014.03.15][0] failed to execute bulk item (index) index {[logstash-2014.03.15][logs][MBzzcesjQTSjUBhryTAgzQ], source[{"@source":"tcp://:0:0:0:0:0:0:1:60186/","@tags":[],"@fields":{"timestamp":["Mar 15 04:15:37"],"logsource":["xxxxxxx-sss01"],"program":["snmptrapd"],"snmptrapsource":["xxxxxxx-hdp04"],"snmptrapseverity":["INFORMATIONAL"],"message":["CLI/Telnet user logout: iRMC S2 CLI/Telnet user '' logout from xxx.xxx.xxx.xxx"]},"@timestamp":"2014-03-15T03:15:37.931+00:00","@message":"\<13\>Mar 15 04:15:37 xxxxxxx-sss01 snmptrapd: xxxxxxx-hdp04 INFORMATIONAL CLI/Telnet user logout: iRMC S2 CLI/Telnet user '' logout from xxx.xxx.xxx.xxx\n","@type":"snmptrapd","@collector":["xxxxxxx-sss01.xxx.xxxxxx"],"@version":"1"}]} org.elasticsearch.index.mapper.MapperParsingException: failed to parse [@fields.snmptrapsource] at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:418) at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:616) at org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMapper.java:604) at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:461) at org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:517) at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:459) at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:515) at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:462) at org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:371) at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:400) at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:153) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:556) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:426) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: org.elasticsearch.index.mapper.MapperParsingException: failed to parse date field [xxxxxxx-hdp04], tried both date format [dateOptionalTime], and timestamp number with locale [] at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:582) at org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:510) at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:215) at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:408) ... 15 more Caused by: java.lang.IllegalArgumentException: Invalid format: "xxxxxxx-hdp04" at org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:754) at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:576) ... 18 more [2014-03-15 03:15:57,635][DEBUG][action.bulk] [xxxx-xxxxxxx-logs-001] [logstash-2014.03.15][0] failed to execute bulk item (index) index {[logstash-2014.03.15][logs][jnTLutANRxeQKP4z\_SF5Jw], source[{"@source":"tcp://:0:0:0:0:0:0:1:51511/","@tags":[],"@fields":{"timestamp":["Mar 15 04:15:57"],"logsource":["xxxxxxx-xxxxx"],"program":["snmptrapd"],"snmptrapsource":["xxxxxxx-hdp13"],"snmptrapseverity":["INFORMATIONAL"],"message":["CLI/Telnet user logout: iRMC S2 CLI/Telnet user '' logout from xxx.xxx.xxx.xxx"]},"@timestamp":"2014-03-15T03:15:57.514+00:00","@message":"\<13\>Mar 15 04:15:57 xxxxxxx-xxxxx snmptrapd: xxxxxxx-hdp13 INFORMATIONAL CLI/Telnet user logout: iRMC S2 CLI/Telnet user '' logout from xxx.xxx.xxx.xxx\n","@type":"snmptrapd","@collector":["xxxxxxx-xxxxx.xxx.xxxxxx"],"@version":"1"}]} org.elasticsearch.index.mapper.MapperParsingException: failed to parse [@fields.snmptrapsource] at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:418) at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:616) at org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMapper.java:604) at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:461) at org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:517) at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:459) at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:515) at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:462) at org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:371) at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:400) at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:153) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:556) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:426) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: org.elasticsearch.index.mapper.MapperParsingException: failed to parse date field [xxxxxxx-hdp13], tried both date format [dateOptionalTime], and timestamp number with locale [] at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:582) at org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:510) at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:215) at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:408) ... 15 more Caused by: java.lang.IllegalArgumentException: Invalid format: "xxxxxxx-hdp13" at org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:754) at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:576) ... 18 more -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/77769ff2-7e46-4339-ae2b-22409e76f415%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/77769ff2-7e46-4339-ae2b-22409e76f415%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Jelle\_Smet](https://avatars.discourse-cdn.com/v4/letter/j/35a633/32.png) [@Jelle\_Smet](https://discuss.elastic.co/u/Jelle_Smet)\ **Post date:** [April 7, 2014, 12:39pm UTC](https://discuss.elastic.co/t/is-this-stacktrace-a-reason-for-cluster-instability/16458/3 "2014-04-07T12:39:37Z")
> So i dont think this is the reason for your troubles. > Nore that the message is with severity DEBUG , and not Error or CRIT True, this was indeed not the cause of the problems we ran into. Cheers, -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/9673b328-2af3-419f-abb6-1ac5b8559562%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9673b328-2af3-419f-abb6-1ac5b8559562%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:37am UTC](https://discuss.elastic.co/t/is-this-stacktrace-a-reason-for-cluster-instability/16458/4 "2017-07-06T01:37:50Z")