# Logstash/Elastic slow indexing

**URL:** <https://discuss.elastic.co/t/logstash-elastic-slow-indexing/192388>\
**Category:** Logstash\
**Created:** [July 26, 2019, 8:34am UTC](https://discuss.elastic.co/t/logstash-elastic-slow-indexing/192388 "2019-07-26T08:34:34Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![thaideval](https://avatars.discourse-cdn.com/v4/letter/t/a8b319/32.png) [@thaideval](https://discuss.elastic.co/u/thaideval)\
**Post date:** [July 26, 2019, 8:34am UTC](https://discuss.elastic.co/t/logstash-elastic-slow-indexing/192388/1 "2019-07-26T08:34:34Z")

</div>

Hello,  
I am new to ELK and maybe my problem is dumm, but i cant get over it.  
I have very slow indexing( approx: 4k docs per min. sometimes its 8k)

My setup - Centos 7 - Logstash 7.2 - Elastic 7.2 - Kibana 7.2 - (ELK all on one VM)  
Centos is 6 CPU 24G Memory

Logstash input - just a directory with file (file is pretty big ~7Gb)  
Codec - multiline  
Filter - grok  
Output into Elastic with template overwrite

shards: 2  
refresh\_interval: -1  
replicas: 0

pipeline.workers: 4 (i tried more)  
pipeline.batch.size: 1000

no errors in both logs(logstash or elastic)  
no grok failures

Here is my logstash config:  
input {  
file {  
codec =\> multiline {  
pattern =\> "^%{DATESTAMP\_EVENTLOG}"  
what =\> "previous"  
negate =\> true  
}  
path =\> "my-path"  
sincedb\_path =\> "my-path-sincedb"  
start\_position =\> "beginning"  
stat\_interval =\> 120  
type =\> "my-db-name"  
}

```
  }

```

filter {  
mutate {}  
mutate {}  
ruby {}  
grok {}

date {}  
}  
output {  
elasticsearch {  
hosts =\> ["127.0.0.1:9200"]  
manage\_template =\> true  
template\_overwrite =\> true  
template\_name =\> "template"  
template =\> "path/template"  
index =\> "my index"  
}  
}

What can i do to understand the reason of slow work?  
Could it be the file size?

---

<div class="post-metadata">

**Author:** ![thaideval](https://avatars.discourse-cdn.com/v4/letter/t/a8b319/32.png) [@thaideval](https://discuss.elastic.co/u/thaideval)\
**Post date:** [July 29, 2019, 10:47am UTC](https://discuss.elastic.co/t/logstash-elastic-slow-indexing/192388/2 "2019-07-29T10:47:57Z")

</div>

Looks like i found my problem.  
its unique for my type of logs.  
seems like pattern in multiline was parsing every symbol of 7GB file.. changed pattern so he would work only after separator.  
and now i have 200k docs per min.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 26, 2019, 10:48am UTC](https://discuss.elastic.co/t/logstash-elastic-slow-indexing/192388/3 "2019-08-26T10:48:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.