hi all,
I have three servers which one elasticesearch , one logstash and one kibana is installed in each server. therefore, the ELK cluster has three nodes. the resource of servers is as following:
servers1: RAM 12, CPU cores:16
servers2: RAM 12, CPU cores:16
servers3: RAM 12, CPU cores:16
JVM option of elasticsearch in each server: -Xms8g, -Xmx8g
JVM option of logstash in each server: -Xms1g, -Xmx1g
the artichecture of ELK is as following:
filebeat (which located on different server) send logs to logstash, then logstash filter them and creates an index named as "my_indexg-%{PDate}". in the logstash configuration file, a date field has been defined as "PDate"
everthing works great but 2 days ago an error had been found as following, notably the PDate field is not based on Gregorian calendar and some conflicts happen between PDate and the default calendar (PDate has been valued as 1398-02-29 but february has just 28 days),
[DEBUG][o.e.a.b.TransportShardBulkAction] [node-1] [my_indexg-1398-02-29][3] failed to execute bulk item (index) index {[my_indexg-1398-02-29][doc][gbdszGoBUuTyDqW325Tz], source[{"source":"D:\\logs\\data_0229.log","PDate":"1398-02-29","action":"Start send-recieve Successfully","Date":"2019-05-19","PDate-Month":"1398-02","@timestamp":"2019-05-18T19:30:46.495Z","prospector":{"type":"log"},"offset":394,"beat":{},"PDate_Time":"1398-02-29T00:00:43.977+0430","fields":{"log_type":"my_indexg"},"Date_Time":"2019-05-19T00:00:43.977+0430","Time":"00:00:43.977","Content":"Start send-recieve Successfully, listenerID:[b8f495e7-961f-4d55-8b6b-589b871dab54-10.0.49.12:60899-10.0.57.51:7000-13980229000043] ,RemoteEndPoint:10.0.49.12:60899 ,LocalEndPoint:10.0.57.51:7000","input":{"type":"log"},"tags":["beats_input_codec_plain_applied","_grokparsefailure"],"@version":"1","host":{"name":"SRV1"}}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [PDate] of type [date]
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:301) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:482) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:606) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:404) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:381) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:96) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:69) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:280) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:748) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:725) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:705) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$executeIndexRequestOnPrimary$3(TransportShardBulkAction.java:461) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeOnPrimaryWhileHandlingMappingUpdates(TransportShardBulkAction.java:483) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:459) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:216) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:159) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:151) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:139) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:79) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1022) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1000) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:102) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:356) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:296) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:963) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:960) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:271) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:238) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2327) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:972) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction.access$500(TransportReplicationAction.java:97) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:317) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:292) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:279) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:251) [x-pack-security-6.5.4.jar:6.5.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:309) [x-pack-security-6.5.4.jar:6.5.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:309) [x-pack-security-6.5.4.jar:6.5.4]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:717) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.4.jar:6.5.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_152]
at java.lang.Thread.run(Unknown Source) [?:1.8.0_152]
Caused by: org.joda.time.IllegalFieldValueException: Cannot parse "1398-02-29": Value 29 for dayOfMonth must be in the range [1,28]
at org.joda.time.field.FieldUtils.verifyValueBounds(FieldUtils.java:259) ~[joda-time-2.10.1.jar:2.10.1]
at org.joda.time.field.PreciseDurationDateTimeField.set(PreciseDurationDateTimeField.java:79) ~[joda-time-2.10.1.jar:2.10.1]
at org.joda.time.DateTimeField.setExtended(DateTimeField.java:392) ~[joda-time-2.10.1.jar:2.10.1]
at org.joda.time.format.DateTimeParserBucket$SavedField.set(DateTimeParserBucket.java:570) ~[joda-time-2.10.1.jar:2.10.1]
at org.joda.time.format.DateTimeParserBucket.computeMillis(DateTimeParserBucket.java:447) ~[joda-time-2.10.1.jar:2.10.1]
at org.joda.time.format.DateTimeParserBucket.doParseMillis(DateTimeParserBucket.java:182) ~[joda-time-2.10.1.jar:2.10.1]
at org.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:826) ~[joda-time-2.10.1.jar:2.10.1]
at org.elasticsearch.index.mapper.DateFieldMapper$DateFieldType.parse(DateFieldMapper.java:249) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.mapper.DateFieldMapper.parseCreateField(DateFieldMapper.java:457) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:295) ~[elasticsearch-6.5.4.jar:6.5.4]
... 44 more
to handle this issue i defined a template and changed the type of PDate from "date" type to "text" as following:
"PDate" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
on the other hand, daily about 8000000 records are shipping to elasticsearch,
but after this issue, elasticsearch falls behined and still is loading the data of yesterday and just load 6000000 of 8000000 and in the case of today just 20000 records have been loaded. it is noted that, ELK is working but falling behinds.
how can i handle this issue? how can i make elasticsearch to do faster?
in addition i checked the number of records based on the @timestamp in a time range for a usual day and yesterday as following:
number of records in time range [12-13] for a usual day: 1369246
number of records in time range [12-13] for yesterday: 226604
this shows that elk is working much slower than previous days.
is it possible that the ELK being slow due to using a text field "PDate" in name of index?
hi, these two cases are different. in this case, we don't define any template and due to date mismatching elasticsearch getting slow
but in (Template defining makes the elasticsearch slower) after template defining elasticsearch was more slower
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.