# Logstash - parse to fields with grok

**URL:** <https://discuss.elastic.co/t/logstash-parse-to-fields-with-grok/242033>\
**Category:** Logstash\
**Created:** [July 21, 2020, 1:15pm UTC](https://discuss.elastic.co/t/logstash-parse-to-fields-with-grok/242033 "2020-07-21T13:15:06Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![elk6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/elk6/32/72282_2.png) [@elk6](https://discuss.elastic.co/u/elk6)\
**Post date:** [July 21, 2020, 1:15pm UTC](https://discuss.elastic.co/t/logstash-parse-to-fields-with-grok/242033/1 "2020-07-21T13:15:06Z")

</div>

TL;DR at bottom

I have a custom generated log file that I made that lists commands users run (along with some other thing). For that, I have got a grok script that successfully parses the message into fields. I've started encountering errors when I was asked to add some really old servers that generated different timestamp.

This is the the log on the new servers:

```
[2020-07-21 12:59:31] SERVER-DB-230 john:USER=root PWD=/root PID=[22714] CMD="echo test9" Exit=[0] CONNECTION=
[2020-07-21 12:59:33] SERVER-DB-230 john:USER=root PWD=/root PID=[22714] CMD="echo test10" Exit=[0] CONNECTION=
[2020-07-21 12:59:35] SERVER-DB-230 john:USER=root PWD=/root PID=[22714] CMD="clear" Exit=[0] CONNECTION=

```

This is the log on the old server (different timestamps):

```
Jul 21 13:02:53 SERVER-DEV-NEW-167 root: USER=root PWD=/root PID=[10638] CMD="echo 2" Exit=[0] CONNECTION=1.2.3.4
Jul 21 13:02:54 SERVER-DEV-NEW-167 root: USER=root PWD=/root PID=[10638] CMD="echo 3" Exit=[0] CONNECTION=1.2.3.4
Jul 21 13:02:56 SERVER-DEV-NEW-167 root: USER=root PWD=/root PID=[10638] CMD="echo 4" Exit=[0] CONNECTION=1.2.3.4

```

Since these are the syntax of logs I have, I think it's best to have an 'if' statement that says - if grok failed to parse, try parsing it with this grok code. Thing is, even though they're very similar, I wasn't able to make grok parse that data. I'm trying to get it working with the grok debugger but I just can't get it to work.

This is my current .conf in logstash: [https://pastebin.com/QZv7zM1x](https://pastebin.com/QZv7zM1x)

Does anyone know how to parse the second block of code into fields? And how to make it parse only if the first one failed? Thanks ahead!

TL;DR: need help parsing the second block of logs and have it parsed by grok only on failure

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 21, 2020, 1:38pm UTC](https://discuss.elastic.co/t/logstash-parse-to-fields-with-grok/242033/2 "2020-07-21T13:38:05Z")

</div>

You can have grok match against an array of patterns. For example

```
grok{
    match => {
        "message" => [
            "^\[(%{TIMESTAMP_ISO8601:sys_timestamp})\]\s(?<Hostname>[0-9a-zA-Z_-]+)\s(?<Logged as>[0-9a-zA-Z_-]+)\:USER=(?<User>[0-9a-zA-Z_-]+)\sPWD=(?<Directory>[0-9a-zA-Z_/-]+)\sPID=\[(?<PID>[0-9]+)\]\sCMD=\"(?<Command>.*)\"\sExit=\[(?<Exit>[0-9]+)\]\sCONNECTION=(?<Connetion>.*)",
            "^%{SYSLOG_TIMESTAMP:sys_timestamp}\s(?<Hostname>[0-9a-zA-Z_-]+)\s(?<Logged as>[0-9a-zA-Z_-]+)\:USER=(?<User>[0-9a-zA-Z_-]+)\sPWD=(?<Directory>[0-9a-zA-Z_/-]+)\sPID=\[(?<PID>[0-9]+)\]\sCMD=\"(?<Command>.*)\"\sExit=\[(?<Exit>[0-9]+)\]\sCONNECTION=(?<Connetion>.*)"
        ]
    }
}

```

Note that I anchored the patterns with ^ to make them fail more quickly.

You might want to try parsing the timestamp, hostname and user using grok then parse the rest of the line with a kv filter.

---

<div class="post-metadata">

**Author:** ![elk6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/elk6/32/72282_2.png) [@elk6](https://discuss.elastic.co/u/elk6)\
**Post date:** [July 21, 2020, 7:24pm UTC](https://discuss.elastic.co/t/logstash-parse-to-fields-with-grok/242033/3 "2020-07-21T19:24:32Z")

</div>

Thank you so much for the response.

[I'm getting an error in the grok debugger](http://puu.sh/G9u8k/190f24aa04.png). Also, may I ask about the '^'?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 21, 2020, 7:35pm UTC](https://discuss.elastic.co/t/logstash-parse-to-fields-with-grok/242033/4 "2020-07-21T19:35:27Z")

</div>

Sorry, there is no underscore in SYSLOGTIMESTAMP.

The reason to anchor the patterns is discussed in [this](https://www.elastic.co/blog/do-you-grok-grok) blog post.

---

<div class="post-metadata">

**Author:** ![elk6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/elk6/32/72282_2.png) [@elk6](https://discuss.elastic.co/u/elk6)\
**Post date:** [July 21, 2020, 7:44pm UTC](https://discuss.elastic.co/t/logstash-parse-to-fields-with-grok/242033/5 "2020-07-21T19:44:45Z")

</div>

My bad, [what I tried at first was without the underscore and I just get "no match"](http://puu.sh/G9urr/2759746cbd.png). It does get results [if I only have the first field](http://puu.sh/G9usB/f1f6d00795.png). Maybe the custom fields don't play well with the pre-made ones?

Also, thanks for the anchor patterns article!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 21, 2020, 8:32pm UTC](https://discuss.elastic.co/t/logstash-parse-to-fields-with-grok/242033/6 "2020-07-21T20:32:24Z")

</div>

I think you need a \s before USER=

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 18, 2020, 8:32pm UTC](https://discuss.elastic.co/t/logstash-parse-to-fields-with-grok/242033/7 "2020-08-18T20:32:37Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.