# Loading log file into ES: poor performance

**URL:** <https://discuss.elastic.co/t/loading-log-file-into-es-poor-performance/73977>\
**Category:** Elasticsearch\
**Created:** [February 5, 2017, 11:20am UTC](https://discuss.elastic.co/t/loading-log-file-into-es-poor-performance/73977 "2017-02-05T11:20:33Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![John16](https://avatars.discourse-cdn.com/v4/letter/j/ea666f/32.png) [@John16](https://discuss.elastic.co/u/John16)\
**Post date:** [February 5, 2017, 11:20am UTC](https://discuss.elastic.co/t/loading-log-file-into-es-poor-performance/73977/1 "2017-02-05T11:20:33Z")

</div>

Hello,

I am loading the existing log file into ES using filebeat. I have a single-node setup.

filebeat.yml:  
filebeat.prospectors:

- paths: ["/data/test.log"]

output.elasticsearch:  
hosts: ["localhost:9200"]  
pipeline: test-pipeline

My pipeline is defined as follows:  
"processors" : [  
{  
"grok" : {  
"field": "message",  
"patterns": [ "%{TIMESTAMP\_ISO8601:date} %{NUMBER:reqtime} %{NUMBER:http  
code} %{IP:ip} %{GREEDYDATA:text}" ]  
},  
"date": {  
"field": "date",  
"target\_field": "@timestamp",  
"formats": ["yyyy-MM-dd HH:mm:ss"]  
},  
"date": {  
"field": "date",  
"target\_field": "date",  
"formats": ["yyyy-MM-dd HH:mm:ss"]  
},  
"date\_index\_name" : {  
"field" : "date",  
"index\_name\_prefix" : "testidx-",  
"date\_rounding" : "d"  
}  
}  
]

Log lines are loaded about 100 per second, filebeat process consumes about 2% of CPU, java process (elasticsearch) consumes about 20%-40% of CPU (this is 32 core server).

path.data is on software RAID6, and I see disks participating is this raid6 are about 50% busy (looks pretty high load provided log lines are just small HTTP requests).

Anything special should I tune to speed this process up (well, at least 10x times)?

Thanks.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 5, 2017, 11:20am UTC](https://discuss.elastic.co/t/loading-log-file-into-es-poor-performance/73977/2 "2017-03-05T11:20:43Z")

</div>

Closed on March 5, 2017, 11:20am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.