# Delay in data shiping to ELK
**URL:** https://discuss.elastic.co/t/delay-in-data-shiping-to-elk/181942
**Category:** Elasticsearch
**Created:** [May 21, 2019, 7:56am UTC](https://discuss.elastic.co/t/delay-in-data-shiping-to-elk/181942 "2019-05-21T07:56:03Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@sahere37](https://discuss.elastic.co/u/sahere37)
#### Post date: [May 21, 2019, 7:56am UTC](https://discuss.elastic.co/t/delay-in-data-shiping-to-elk/181942/1 "2019-05-21T07:56:03Z")
hi all,
I have three servers which one elasticesearch , one logstash and one kibana is installed in each server. therefore, the ELK cluster has three nodes. the resource of servers is as following:
servers1: RAM 12, CPU cores:16
servers2: RAM 12, CPU cores:16
servers3: RAM 12, CPU cores:16
JVM option of elasticsearch in each server: -Xms8g, -Xmx8g
JVM option of logstash in each server: -Xms1g, -Xmx1g
the artichecture of ELK is as following:
filebeat (which located on different server) send logs to logstash, then logstash filter them and creates an index named as "my\_indexg-%{PDate}". in the logstash configuration file, a date field has been defined as "PDate"
everthing works great but 2 days ago an error had been found as following, notably the PDate field is not based on Gregorian calendar and some conflicts happen between PDate and the default calendar (PDate has been valued as 1398-02-29 but february has just 28 days),
---
### Author:  [@sahere37](https://discuss.elastic.co/u/sahere37)
#### Post date: [May 21, 2019, 7:57am UTC](https://discuss.elastic.co/t/delay-in-data-shiping-to-elk/181942/2 "2019-05-21T07:57:45Z")
in elasticsearch log:
```
[DEBUG][o.e.a.b.TransportShardBulkAction] [node-1] [my_indexg-1398-02-29][3] failed to execute bulk item (index) index {[my_indexg-1398-02-29][doc][gbdszGoBUuTyDqW325Tz], source[{"source":"D:\\logs\\data_0229.log","PDate":"1398-02-29","action":"Start send-recieve Successfully","Date":"2019-05-19","PDate-Month":"1398-02","@timestamp":"2019-05-18T19:30:46.495Z","prospector":{"type":"log"},"offset":394,"beat":{},"PDate_Time":"1398-02-29T00:00:43.977+0430","fields":{"log_type":"my_indexg"},"Date_Time":"2019-05-19T00:00:43.977+0430","Time":"00:00:43.977","Content":"Start send-recieve Successfully, listenerID:[b8f495e7-961f-4d55-8b6b-589b871dab54-10.0.49.12:60899-10.0.57.51:7000-13980229000043] ,RemoteEndPoint:10.0.49.12:60899 ,LocalEndPoint:10.0.57.51:7000","input":{"type":"log"},"tags":["beats_input_codec_plain_applied","_grokparsefailure"],"@version":"1","host":{"name":"SRV1"}}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [PDate] of type [date]
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:301) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:482) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:606) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:404) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:381) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:96) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:69) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:280) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:748) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:725) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:705) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$executeIndexRequestOnPrimary$3(TransportShardBulkAction.java:461) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeOnPrimaryWhileHandlingMappingUpdates(TransportShardBulkAction.java:483) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:459) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:216) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:159) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:151) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:139) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:79) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1022) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1000) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:102) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:356) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:296) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:963) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:960) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:271) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:238) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2327) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:972) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction.access$500(TransportReplicationAction.java:97) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:317) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:292) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:279) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:251) [x-pack-security-6.5.4.jar:6.5.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:309) [x-pack-security-6.5.4.jar:6.5.4]
```
---
### Author:  [@sahere37](https://discuss.elastic.co/u/sahere37)
#### Post date: [May 21, 2019, 7:59am UTC](https://discuss.elastic.co/t/delay-in-data-shiping-to-elk/181942/3 "2019-05-21T07:59:21Z")
```
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:309) [x-pack-security-6.5.4.jar:6.5.4]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:717) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.4.jar:6.5.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_152]
at java.lang.Thread.run(Unknown Source) [?:1.8.0_152]
Caused by: org.joda.time.IllegalFieldValueException: Cannot parse "1398-02-29": Value 29 for dayOfMonth must be in the range [1,28]
at org.joda.time.field.FieldUtils.verifyValueBounds(FieldUtils.java:259) ~[joda-time-2.10.1.jar:2.10.1]
at org.joda.time.field.PreciseDurationDateTimeField.set(PreciseDurationDateTimeField.java:79) ~[joda-time-2.10.1.jar:2.10.1]
at org.joda.time.DateTimeField.setExtended(DateTimeField.java:392) ~[joda-time-2.10.1.jar:2.10.1]
at org.joda.time.format.DateTimeParserBucket$SavedField.set(DateTimeParserBucket.java:570) ~[joda-time-2.10.1.jar:2.10.1]
at org.joda.time.format.DateTimeParserBucket.computeMillis(DateTimeParserBucket.java:447) ~[joda-time-2.10.1.jar:2.10.1]
at org.joda.time.format.DateTimeParserBucket.doParseMillis(DateTimeParserBucket.java:182) ~[joda-time-2.10.1.jar:2.10.1]
at org.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:826) ~[joda-time-2.10.1.jar:2.10.1]
at org.elasticsearch.index.mapper.DateFieldMapper$DateFieldType.parse(DateFieldMapper.java:249) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.mapper.DateFieldMapper.parseCreateField(DateFieldMapper.java:457) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:295) ~[elasticsearch-6.5.4.jar:6.5.4]
... 44 more
```
---
### Author:  [@sahere37](https://discuss.elastic.co/u/sahere37)
#### Post date: [May 21, 2019, 8:03am UTC](https://discuss.elastic.co/t/delay-in-data-shiping-to-elk/181942/4 "2019-05-21T08:03:11Z")
to handle this issue i defined a template and changed the type of PDate from "date" type to "text" as following:
"PDate" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore\_above" : 256
}
}
},
on the other hand, daily about 8000000 records are shipping to elasticsearch,
but after this issue, elasticsearch falls behined and still is loading the data of yesterday and just load 6000000 of 8000000 and in the case of today just 20000 records have been loaded. it is noted that, ELK is working but falling behinds.
how can i handle this issue? how can i make elasticsearch to do faster?
in addition i checked the number of records based on the @timestamp in a time range for a usual day and yesterday as following:
number of records in time range [12-13] for a usual day: 1369246
number of records in time range [12-13] for yesterday: 226604
this shows that elk is working much slower than previous days.
is it possible that the ELK being slow due to using a text field "PDate" in name of index?
---
### Author:  [@sahere37](https://discuss.elastic.co/u/sahere37)
#### Post date: [May 21, 2019, 8:17am UTC](https://discuss.elastic.co/t/delay-in-data-shiping-to-elk/181942/5 "2019-05-21T08:17:34Z")
could you please advise me? it is so urgent
@Brandon_Kobel
@bargs
---
### Author:  [@sahere37](https://discuss.elastic.co/u/sahere37)
#### Post date: [May 25, 2019, 6:37am UTC](https://discuss.elastic.co/t/delay-in-data-shiping-to-elk/181942/6 "2019-05-25T06:37:20Z")
could anyone advise me?
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [May 25, 2019, 10:46am UTC](https://discuss.elastic.co/t/delay-in-data-shiping-to-elk/181942/7 "2019-05-25T10:46:44Z")
> [@sahere37](#):
>
> is it possible that the ELK being slow due to using a text field "PDate" in name of index?
You asked the same question in [Template defining makes the elasticsearch slower - #7 by dadoonet](https://discuss.elastic.co/t/template-defining-makes-the-elasticsearch-slower/182306/7) so let's keep the discussion there.
---
### Author:  [@sahere37](https://discuss.elastic.co/u/sahere37)
#### Post date: [May 29, 2019, 4:47am UTC](https://discuss.elastic.co/t/delay-in-data-shiping-to-elk/181942/8 "2019-05-29T04:47:00Z")
hi, these two cases are different. in this case, we don't define any template and due to date mismatching elasticsearch getting slow
but in ([Template defining makes the elasticsearch slower](https://discuss.elastic.co/t/template-defining-makes-the-elasticsearch-slower/182306/5)) after template defining elasticsearch was more slower
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 26, 2019, 4:47am UTC](https://discuss.elastic.co/t/delay-in-data-shiping-to-elk/181942/9 "2019-06-26T04:47:00Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.