Hi!
I have a ELK cluster with 1, client, 1 master and 3 datanodes, before with ES1.7 everything was working fine, but now with ES2.x and LS2.x always after 1-2 days 3rd data node starts to have following errors. (Nodes are running on VMs and I have created few times on top of different HW, but problem persists always on 3rd data node).
Somehow few bytes from message field are lost and node cannot parse message field?
Only way i have found to recover from this is to delete whole index (there is index for each day). Also seems that shard fails to start again on 3rd data node if I restart elasticsearch on 3rd data node. So somehow it's messed up...
Also, it seems like this only happens if primary shard gets located to 3rd datanode. If that does not happen everything is running smoothly.
Any ideas what might cause this? Thanks!
-Marko
[2015-12-30 09:43:43,429][DEBUG][action.bulk ] [Data3] [logstash-2015.12.30][0] failed to execute bulk item (index) index {[logstash-2015$
MapperParsingException[failed to parse [message]]; nested: IllegalArgumentException[Invalid format: "084665 30.12 04:45:29.993 [192.168.2..." is malfo$
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:339)
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:314)
at org.elasticsearch.index.mapper.DocumentParser.parseAndMergeUpdate(DocumentParser.java:762)
at org.elasticsearch.index.mapper.DocumentParser.parseDynamicValue(DocumentParser.java:676)
at org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:447)
at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:267)
at org.elasticsearch.index.mapper.DocumentParser.innerParseDocument(DocumentParser.java:127)
at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:79)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:304)
at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:551)
at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:542)
at org.elasticsearch.action.support.replication.TransportReplicationAction.prepareIndexOperationOnPrimary(TransportReplicationAction.java:1049)
at org.elasticsearch.action.support.replication.TransportReplicationAction.executeIndexRequestOnPrimary(TransportReplicationAction.java:1060)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:338)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:131)
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.performOnPrimary(TransportReplicationAction.java:579)
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase$1.doRun(TransportReplicationAction.java:452)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Invalid format: "084665 30.12 04:45:29.993 [192.168.2..." is malformed at "65 30.12 04:45:29.993 [192.1$
at org.joda.time.format.DateTimeParserBucket.doParseMillis(DateTimeParserBucket.java:187)
at org.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:780)
at org.elasticsearch.index.mapper.core.DateFieldMapper$DateFieldType.parseStringValue(DateFieldMapper.java:360)
at org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:526)
at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:213)
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:331)