# Logstash is calculating itself to sudden death

**URL:** <https://discuss.elastic.co/t/logstash-is-calculating-itself-to-sudden-death/111505>\
**Category:** Logstash\
**Created:** [December 13, 2017, 8:01am UTC](https://discuss.elastic.co/t/logstash-is-calculating-itself-to-sudden-death/111505 "2017-12-13T08:01:49Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Haenschen](https://avatars.discourse-cdn.com/v4/letter/h/ba8739/32.png) [@Haenschen](https://discuss.elastic.co/u/Haenschen)\
**Post date:** [December 13, 2017, 8:01am UTC](https://discuss.elastic.co/t/logstash-is-calculating-itself-to-sudden-death/111505/1 "2017-12-13T08:01:49Z")

</div>

Hi folks,

I need your advice solving following problem: I am going to parse the Logfiles from Akamai CDN using Logstash. In order to do this I'm using the following Logstash config:

input {  
file {  
max\_open\_files =\> 1000000  
id =\> "akamai-cdn"  
type =\> "akamai-plain"  
path =\> "/data/logs/akamai-decompressed/\*"  
start\_position =\> "beginning"  
codec =\> plain  
}  
}  
filter {  
grok {  
match =\> { "message" =\> "%{DATE\_EU:date}\t%{TIME:time}\s%{IP:clientip}\s%{WORD:httpmethod}\t%{PATH:requestedpage}\s%{NUMBER:responsecode}\s%{NUMBER:bytessent}\s%{NUMBER:timetaken}\t%{DATA:csreferrer}\t%{DATA:csuseragent}\t%{DATA:cscookie}" }  
add\_tag =\> ["akamai-cdn"]  
}  
date {  
match =\> ["timestamp", "yyyy-MM-dd HH:hh:ss"]  
target =\> "@timestamp"  
add\_field =\> {"debug" =\> "timestampisfixed"}  
}

}

#and the es-output...

########## sample log lines#################

| 2017-11-19 | 15:14:33 | 95.90.212.131 | GET | /cdn-aka-ee-xxxxxxxx.xxxxxxxx/mall/shopde/pic/tbild3/tbild3-xxxxxxxx.JPG | 200 | 1841 | 0 | "-" | "-" | "-" |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 2017-11-19 | 15:14:33 | 95.90.212.131 | GET | /cdn-aka-ee-xxxxxxxx.xxxxxxxxe/mall/shopde/pic/bild0/Bild0-xxxxxxxx.JPG | 200 | 5030 | 0 | "-" | "-" | "-" |
| 2017-11-19 | 15:14:33 | 95.90.212.131 | GET | /cdn-aka-ee-xxxxxxxx.xxxxxxxxe/mall/shopde/pic/tbild1/tbild1-xxxxxxxx.JPG | 200 | 1715 | 0 | "-" | "-" | "-" |

########################################  
the "|" chars are in tabs or spaces in the log lines

When I start the pipeline the Logstash is calculating the hell out of itself, all cores are used completly to parse the logs. It's not that big amount of Data, we are talking about 1 million lines...  
do i have a bad filter ?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 14, 2017, 7:00am UTC](https://discuss.elastic.co/t/logstash-is-calculating-itself-to-sudden-death/111505/2 "2017-12-14T07:00:47Z")

</div>

> When I start the pipeline the Logstash is calculating the hell out of itself, all cores are used completly to parse the logs.

Yes. By default Logstash starts as many pipeline workers as you have CPU cores. This is configurable.

> It's not that big amount of Data, we are talking about 1 million lines...

The amount of data doesn't affect the CPU usage, a rate measure, only the accumulated CPU time and the wall clock time.

Similarly, an electric stove uses the same amount of power (1 kW or whatever) when you heat water regardless of how much water you have, but if you have more water it'll take longer and the amount of energy used will be higher.

So, if we rephrase your question to "how can I make Logstash use less CPU time" you can start by replacing the DATA patterns with NOTSPACE. Having more than one DATA or GREEDYDATA in the same grok pattern is almost always a mistake.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 11, 2018, 7:01am UTC](https://discuss.elastic.co/t/logstash-is-calculating-itself-to-sudden-death/111505/3 "2018-01-11T07:01:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.