# Logstash two different log types
**URL:** https://discuss.elastic.co/t/logstash-two-different-log-types/130002
**Category:** Logstash
**Created:** [April 30, 2018, 9:16am UTC](https://discuss.elastic.co/t/logstash-two-different-log-types/130002 "2018-04-30T09:16:41Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@JensVanDeynse](https://discuss.elastic.co/u/JensVanDeynse)
#### Post date: [April 30, 2018, 9:16am UTC](https://discuss.elastic.co/t/logstash-two-different-log-types/130002/1 "2018-04-30T09:16:42Z")
Hello there
I've been experimenting with logstash but ran into a problem when trying to parse haproxy logs.
The issue is as follows, I've been able to implement a grok parser function for the most common logs but sometimes a different type of line appears in the log which can't be parsed:
GROK function:
match =\> ["message", "%{SYSLOGTIMESTAMP:timestamp} %{IPORHOST} %{PROG}(?:[%{POSINT}])?: %{IPORHOST:source}:%{POSINT:port} | {%{DATA:tenant}|%{DATA:userAgent}} | %{NOTSPACE:frontend} %{NOTSPACE:backend}/%{NOTSPACE:server} %{INT:timeClientRequest}/%{INT:timeQueue}/%{INT:timeTCP}/%{INT:timeServer}/%{INT:timeTotal} %{INT:statusCode} %{INT:bytes} %{INT:connectionsProcess}/%{INT:concurrentConnectionsFrontend}/%{INT:concurrentConnectionsBackend}/%{INT:concurrentConnectionsServer}/%{INT:retries} %{INT:connectionsServer}/%{INT:connectionsBackend} "%{DATA:verb} %{DATA:request}""]
Types of lines:
Mar 21 03:25:02 localhost haproxy[14415]: 188.93.156.49:50499 | {header1|Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)} | main~ server\_2.2/appServer02 1/0/0/1/2 302 165 3/3/0/1/0 0/0 "GET /server-2.2 HTTP/1.1"
Mar 22 01:27:39 localhost haproxy[14415]: Server updates/appServer02 is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 3ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Mar 22 00:16:13 localhost haproxy[14415]: 64.41.200.105:60240 [22/Mar/2018:00:16:13.574] main/1: SSL handshake failure
Now my question is the following:
- Is there a possibility to detect if the log is the normal format (see logline 1) and if not to just apply GREEDYDATA to it.
Thank you
---
### Author:  [@guyboertje](https://discuss.elastic.co/u/guyboertje)
#### Post date: [April 30, 2018, 11:41am UTC](https://discuss.elastic.co/t/logstash-two-different-log-types/130002/2 "2018-04-30T11:41:28Z")
Sure you can. However you have to get the first pattern spot on correct because the second pattern will match all so you will not see a `_grokparsefailure` tag.
Make sure you anchor the start with `^` and escape the pipe and square brackets `|` and `[]`.
```auto
input {
generator {
lines => [
'Mar 21 03:25:02 localhost haproxy[14415]: 188.93.156.49:50499 | {header1|Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)} | main~ server_2.2/appServer02 1/0/0/1/2 302 165 3/3/0/1/0 0/0 "GET /server-2.2 HTTP/1.1"',
'Mar 22 01:27:39 localhost haproxy[14415]: Server updates/appServer02 is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 3ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.',
'Mar 22 00:16:13 localhost haproxy[14415]: 64.41.200.105:60240 [22/Mar/2018:00:16:13.574] main/1: SSL handshake failure'
]
count => 1
}
}
filter {
grok {
match => {
"message" => [
'^%{SYSLOGTIMESTAMP:timestamp} %{IPORHOST} %{PROG}(?:\[%{POSINT}\])?: %{IPORHOST:source}:%{POSINT:port} \| {%{DATA:tenant}\|%{DATA:userAgent}} \| %{NOTSPACE:frontend} %{NOTSPACE:backend}/%{NOTSPACE:server} %{INT:timeClientRequest}/%{INT:timeQueue}/%{INT:timeTCP}/%{INT:timeServer}/%{INT:timeTotal} %{INT:statusCode} %{INT:bytes} %{INT:connectionsProcess}/%{INT:concurrentConnectionsFrontend}/%{INT:concurrentConnectionsBackend}/%{INT:concurrentConnectionsServer}/%{INT:retries} %{INT:connectionsServer}/%{INT:connectionsBackend} "%{DATA:verb} %{DATA:request}"',
'^%{SYSLOGTIMESTAMP:timestamp} %{IPORHOST} %{PROG}(?:\[%{POSINT}\])?: %{GREEDYDATA:unexpected_message}'
]
}
break_on_match => true
}
}
output {
stdout {
codec => rubydebug
}
}
```
Gives:
```auto
{
"message" => "Mar 22 00:16:13 localhost haproxy[14415]: 64.41.200.105:60240 [22/Mar/2018:00:16:13.574] main/1: SSL handshake failure",
"sequence" => 0,
"timestamp" => "Mar 22 00:16:13",
"@timestamp" => 2018-04-30T11:30:25.906Z,
"host" => "Elastics-MacBook-Pro.local",
"@version" => "1",
"unexpected_message" => "64.41.200.105:60240 [22/Mar/2018:00:16:13.574] main/1: SSL handshake failure"
}
{
"timestamp" => "Mar 21 03:25:02",
"timeTCP" => "0",
"timeTotal" => "2",
"concurrentConnectionsServer" => "1",
"bytes" => "165",
"retries" => "0",
"connectionsBackend" => "0",
"connectionsProcess" => "3",
"connectionsServer" => "0",
"statusCode" => "302",
"backend" => "server_2.2",
"timeServer" => "1",
"concurrentConnectionsFrontend" => "3",
"host" => "Elastics-MacBook-Pro.local",
"message" => "Mar 21 03:25:02 localhost haproxy[14415]: 188.93.156.49:50499 | {header1|Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)} | main~ server_2.2/appServer02 1/0/0/1/2 302 165 3/3/0/1/0 0/0 \"GET /server-2.2 HTTP/1.1\"",
"request" => "/server-2.2 HTTP/1.1",
"@timestamp" => 2018-04-30T11:30:25.773Z,
"tenant" => "header1",
"userAgent" => "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)",
"timeClientRequest" => "1",
"verb" => "GET",
"@version" => "1",
"sequence" => 0,
"timeQueue" => "0",
"port" => "50499",
"frontend" => "main~",
"source" => "188.93.156.49",
"concurrentConnectionsBackend" => "0",
"server" => "appServer02"
}
{
"message" => "Mar 22 01:27:39 localhost haproxy[14415]: Server updates/appServer02 is UP, reason: Layer7 check passed, code: 200, info: \"OK\", check duration: 3ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.",
"sequence" => 0,
"timestamp" => "Mar 22 01:27:39",
"@timestamp" => 2018-04-30T11:30:25.904Z,
"host" => "Elastics-MacBook-Pro.local",
"@version" => "1",
"unexpected_message" => "Server updates/appServer02 is UP, reason: Layer7 check passed, code: 200, info: \"OK\", check duration: 3ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue."
}
```
---
### Author:  [@JensVanDeynse](https://discuss.elastic.co/u/JensVanDeynse)
#### Post date: [May 2, 2018, 9:36am UTC](https://discuss.elastic.co/t/logstash-two-different-log-types/130002/3 "2018-05-02T09:36:15Z")
Thanks, this works
---
### Author:  [@JensVanDeynse](https://discuss.elastic.co/u/JensVanDeynse)
#### Post date: [May 18, 2018, 7:07am UTC](https://discuss.elastic.co/t/logstash-two-different-log-types/130002/4 "2018-05-18T07:07:50Z")
Hello there
I've implemented this in our logstash configuration using a total of 3 different expressions but now our Logstash server uses all of it's CPU (8 cores).
I've also enabled x-pack for logstash and there I can see (in the pipelines schema) that this grok expression uses 75% cpu and has a latency of 122ms/e.
Is this a known issue when using multiple grok expressions because the grok expression isn't that complex?
I read somewhere that when a grok expression doesn't match the logs it could take up more CPU but I don't know if that's true.
Any help or tips would be extreme helpful and appreciated
Thank you
---
### Author:  [@guyboertje](https://discuss.elastic.co/u/guyboertje)
#### Post date: [May 21, 2018, 11:46am UTC](https://discuss.elastic.co/t/logstash-two-different-log-types/130002/5 "2018-05-21T11:46:45Z")
What version of LS and Grok are you using?
We did have a problem when patterns did not match but IIRC this is much better in the most recent versions of `grok`.
---
### Author:  [@JensVanDeynse](https://discuss.elastic.co/u/JensVanDeynse)
#### Post date: [May 23, 2018, 5:51am UTC](https://discuss.elastic.co/t/logstash-two-different-log-types/130002/6 "2018-05-23T05:51:12Z")
We are using the latest version of logstash -\> 6.2.4
I don't know how to find the version number of grok, I assume this is equal to the logstash version.
EDIT -\> Grok version: 4.0.2
---
### Author:  [@guyboertje](https://discuss.elastic.co/u/guyboertje)
#### Post date: [May 23, 2018, 12:30pm UTC](https://discuss.elastic.co/t/logstash-two-different-log-types/130002/7 "2018-05-23T12:30:00Z")
That version includes the "much better" code.
Did you read [this blog post](https://www.elastic.co/blog/do-you-grok-grok) from my colleague Joao?
---
### Author:  [@JensVanDeynse](https://discuss.elastic.co/u/JensVanDeynse)
#### Post date: [May 29, 2018, 11:56am UTC](https://discuss.elastic.co/t/logstash-two-different-log-types/130002/8 "2018-05-29T11:56:02Z")
I just read the post and I see what it could be, I didn't close the logentry with a $ sign so I suppose grok tries too much instead of just going to the next grok parser
I'll try this out
Thank you
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 26, 2018, 11:57am UTC](https://discuss.elastic.co/t/logstash-two-different-log-types/130002/9 "2018-06-26T11:57:24Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.