# What is the best way to write grok pattern for below Log

**URL:** <https://discuss.elastic.co/t/what-is-the-best-way-to-write-grok-pattern-for-below-log/223646>\
**Category:** Logstash\
**Created:** [March 15, 2020, 6:31am UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-write-grok-pattern-for-below-log/223646 "2020-03-15T06:31:54Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [March 15, 2020, 6:31am UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-write-grok-pattern-for-below-log/223646/1 "2020-03-15T06:31:55Z")

</div>

Hi Folks,

Below is the line that I am trying to write a GROK pattern however later part starts with JSON and my codec =\> json is not matching due to the first part.

Any idea or clue how do I match the rules or do I need to write complete grok pattern for below log entries?

```auto
honeysap.events.logfeed - 2020-03-15 05:13:50,979 - EVENT - {"timestamp": "2020-03-15 05:13:50.978092", "request": "AAAAQE5JX1JPVVRFAAIoAgAAAAEAAAAoAAAAFDE5Mi4xNjguNS4xMjcAMzI5OQAAMTAuMC4wLjEwMABzYXBkcDAxAAA=", "session": "4ba57f1d-b5ad-4ebf-9ef9-35f9e8860723", "target_port": 3299, "target_ip": "0.0.0.0", "data": "", "event": "Received packet", "service": "saprouter", "source_ip": "192.168.5.76", "response": "", "source_port": 52183}

```

---

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [March 15, 2020, 6:37am UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-write-grok-pattern-for-below-log/223646/2 "2020-03-15T06:37:24Z")

</div>

My KV is working fine when only those inside braces entries are parased

```auto
input {
        stdin {
        codec => json
        }
}

filter {
        kv { }
}
output {
stdout {}
}

```

---

<div class="post-metadata">

**Author:** ![rcowart](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rcowart/32/88091_2.png) [@rcowart](https://discuss.elastic.co/u/rcowart)\
**Post date:** [March 15, 2020, 9:02am UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-write-grok-pattern-for-below-log/223646/3 "2020-03-15T09:02:03Z")

</div>

I used a combination of `grok` to grab the timestamp from the message and the main payload, which is JSON formatted.

```auto
input {
  file {
    path => "PATH/TO/honeysap.log"
    codec => plain {
      charset => "ISO-8859-1"
    }
    type => "honeysap"
    start_position => beginning
    sincedb_path => "/dev/null"
  }
}

filter {
  # Get rid of the logfile path if it isn't needed.
  mutate {
    remove_field => ["path"]
  }

  # Extract the timestamp and JSON payload. If successful remove the original message.
  grok {
    match => {
      "[message]" => "honeysap\.events\.logfeed%{SPACE}-%{SPACE}(?<datetime>20[0-3][0-9]-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-6][0-9]:[0-6][0-9],[0-9][0-9][0-9])%{SPACE}-%{SPACE}EVENT%{SPACE}-[^\{]+%{GREEDYDATA:payload}"
    }
    remove_field => ["[message]" ]
  }

  # If the grok was successful process the timestamp and JSON payload.
  if "_grokparsefailure" not in [tags] {
    # Transform the JSON payload into event fields. If successful remove the payload field.
    json {
      skip_on_invalid_json => true
      source => "[payload]"
      remove_field => ["[payload]" ]
    }
    
    # Set @timestamp based on datetime (the timestamp from the message). If successful remove the datetime field. Set the timezone as needed.
    date {
      match => ["[datetime]", "YYYY-MM-dd HH:mm:ss,SSS" ]
      remove_field => ["[datetime]" ]
      timezone => "UTC"
    }
  }
}

output {
  stdout {
    codec => rubydebug { }
  }
}

```

The resulting output is...

```auto
{
     "@timestamp" => 2020-03-15T05:13:50.979Z,
           "type" => "honeysap",
    "target_port" => 3299,
        "service" => "saprouter",
    "source_port" => 52183,
      "source_ip" => "192.168.5.76",
       "response" => "",
      "target_ip" => "0.0.0.0",
      "timestamp" => "2020-03-15 05:13:50.978092",
           "host" => "ws5",
        "request" => "AAAAQE5JX1JPVVRFAAIoAgAAAAEAAAAoAAAAFDE5Mi4xNjguNS4xMjcAMzI5OQAAMTAuMC4wLjEwMABzYXBkcDAxAAA=",
       "@version" => "1",
          "event" => "Received packet",
           "data" => "",
        "session" => "4ba57f1d-b5ad-4ebf-9ef9-35f9e8860723"
}

```

Since the JSON payload also includes a `timestamp` field you could also do something more simple like...

```auto
filter {
  # Get rid of the logfile path if it isn't needed. And remove the non-JSON log prefix.
  mutate {
    remove_field => ["path"]
    gsub => ["[message]", "^[^\{]+", "" ]
  }

  json {
    skip_on_invalid_json => true
    source => "[message]"
    remove_field => ["[message]" ]
  }
  
  # Set @timestamp based on the timestamp field from the JSON payload. If successful remove the datetime field. Set the timezone as needed.
  date {
    match => ["[timestamp]", "ISO8601" ]
    remove_field => ["[timestamp]" ]
    timezone => "UTC"
  }
}

```

This produces a similar output, without the extra timestamp...

```auto
{
           "type" => "honeysap",
       "@version" => "1",
      "target_ip" => "0.0.0.0",
          "event" => "Received packet",
           "data" => "",
       "response" => "",
        "session" => "4ba57f1d-b5ad-4ebf-9ef9-35f9e8860723",
     "@timestamp" => 2020-03-15T05:13:50.979Z,
        "service" => "saprouter",
        "request" => "AAAAQE5JX1JPVVRFAAIoAgAAAAEAAAAoAAAAFDE5Mi4xNjguNS4xMjcAMzI5OQAAMTAuMC4wLjEwMABzYXBkcDAxAAA=",
    "source_port" => 52183,
           "host" => "ws5",
      "timestamp" => "2020-03-15 05:13:50.978092",
      "source_ip" => "192.168.5.76",
    "target_port" => 3299
}

```

In some environments multiple timestamps are desired in order to record the time the event occurred, the time it was logged by the observing system, and the time it was received by the central log system. In ECS these are `@timestamp`, `event.created`, and `event.ingested` respectively.

You could further process the event data from this point in order to make it ECS compliant, which would allow use to use the SIEM app in Kibana.

Rob

[![GitHub](https://us1.discourse-cdn.com/elastic/original/3X/6/f/6f8ae834f16b1a02d31607317669716807844d84.png)](https://github.com/robcowart) [![YouTube](https://us1.discourse-cdn.com/elastic/original/3X/4/3/43b9b81a8c93786219985aeb5335c1c323449053.png)](https://www.youtube.com/channel/UCivWvTx1DwrWNcDLV58kmOg) [![LinkedIn](https://us1.discourse-cdn.com/elastic/original/3X/6/7/674f3370d0f0542ddc5e408516beb1b7edd6c1bf.png)](https://www.linkedin.com/in/robertcowart/)  
**[How to install Elasticsearch & Kibana on Ubuntu - incl. hardware recommendations](https://www.youtube.com/watch?v=gZb7HpVOges)**  
**[What is the best storage technology for Elasticsearch?](https://www.youtube.com/watch?v=nKUpfJCBiS4)**

---

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [March 15, 2020, 12:11pm UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-write-grok-pattern-for-below-log/223646/4 "2020-03-15T12:11:21Z")

</div>

Awesome man thanks a lot and that did work!!

Thanks again.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 15, 2020, 4:06pm UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-write-grok-pattern-for-below-log/223646/5 "2020-03-15T16:06:46Z")

</div>

Another option would be to use dissect rather than grok. Like [this](https://discuss.elastic.co/t/how-to-parse-mix-json-logs/167594/2).

---

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [March 18, 2020, 1:28pm UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-write-grok-pattern-for-below-log/223646/6 "2020-03-18T13:28:18Z")

</div>

Hi,

Somehow using those parsers I am unable to index my data in elastic search any idea why?

`````auto
[WARN] 2020-03-18 18:48:35.675 [[main]>worker3] elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"problox-2020.03.18", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x2da962da>], :response=>{"index"=>{"_index"=>"problox-2020.03.18", "_type"=>"_doc", "_id"=>"-tPL7XABjMrTL_9AtrF2", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [timestamp] of type [date] in document with id '-tPL7XABjMrTL_9AtrF2'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [2020-03-18 07:11:22.911641] with format [strict_date_optional_time||epoch_millis]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"Failed to parse with all enclosed parsers"}}}}}}
````
here is my final file

````
input {
# SAP Trap Internal
# file {
# path => ["/var/log/data/saphp/honeysap-internal.log"]
# codec => json
# type => "saptrapinternal"
# }

# SAP Trap External
# file {
# path => ["/var/log/data/saphp/honeysap-external.log"]
# codec => json
# type => "saptrapexternal"
# }
#}
        stdin {
        codec => json
        type => "saptrapexternal"
        }
}
filter {
#SAP Internal
        if [type] == "saptrapinternal"{
          mutate {
    remove_field => ["path"]
    gsub => ["[message]", "^[^\{]+", "" ]
                rename => {
                "source_ip" => "src_ip"
                "target_port" => "dest_port"
                "source_port" => "src_port"
                }
  }
  json {
    skip_on_invalid_json => true
    source => "[message]"
    remove_field => ["[message]" ]
  }
  # Set @timestamp based on the timestamp field from the JSON payload. If successful remove the datetime field. Set the timezone as needed.
  date {
    match => ["[timestamp]", "ISO8601" ]
    remove_field => ["[timestamp]" ]
    timezone => "UTC"
  }
}
############
        if [type] == "saptrapexternal" {
                  mutate {
    remove_field => ["path"]
    gsub => ["[message]", "^[^\{]+", "" ]
                        rename => {
                "source_ip" => "src_ip"
                "target_port" => "dest_port"
                "source_port" => "src_port"
                }
  }
  json {
    skip_on_invalid_json => "true"
    source => "[message]"
    remove_field => ["[message]" ]
  }
# mutate {
# remove_field => ["path"]
# gsub => ["[message]", "^[^\{]+", "" ]
# rename => {
# "source_ip" => "src_ip"
# "target_port" => "dest_port"
# "source_port" => "src_port"
# }
# }
  # Set @timestamp based on the timestamp field from the JSON payload. If successful remove the datetime field. Set the timezone as needed.
  date {
    match => ["[timestamp]", "ISO8601" ]
    remove_field => ["[timestamp]" ]
    timezone => "UTC"
        }
}
#### SAP Trap
if [type] == "saptrapinternal" {
        mutate {
                add_field => {
                        trap_type => "SAP-iNTERNAL"
                        }
                }
                        }

if [type] == "saptrapexternal" {
        mutate {
               add_field => {
                        trap_type => "SAP-Router"
                        }
                }
                        }
}
output {
  elasticsearch {
    hosts => ["https://127.0.0.1:16577"]
    user => xxxx
    password => XXXX
    ssl => true
    ssl_certificate_verification => false
    template => "/etc/logstash/elasticsearch-template-es7x.json"
    ilm_enabled => false
    index => "problox-%{+YYYY.MM.dd}"
# document_type => "doc"
  }

  #if [type] == "Suricata" {
  # file {
  # file_mode => 0770
  # path => "/data/suricata/log/suricata_ews.log"
  # }
  #}
  # Debug output
  #if [type] == "XYZ" {
  # stdout {
  # codec => rubydebug
  # }
  #}
  # Debug output
  #stdout {
  # codec => rubydebug
  #}

}

#output {stdout {}}

`````

And my mutate filter is now working as well

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 18, 2020, 3:40pm UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-write-grok-pattern-for-below-log/223646/7 "2020-03-18T15:40:22Z")

</div>

> [@Blason](#):
>
> "reason"=\>"failed to parse date field [2020-03-18 07:11:22.911641] with format [strict\_date\_optional\_time||epoch\_millis]"

The strict\_date\_optional\_time format only allows millisecond precision. If you need to retain microsecond precision you could use an index template with a [custom format](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html#custom-date-formats), or if milliseconds is OK use a date filter to parse that field before feeding it to elasticsearch (or mutate+gsub to throw away the last three characters of the field).

---

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [March 18, 2020, 3:54pm UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-write-grok-pattern-for-below-log/223646/8 "2020-03-18T15:54:29Z")

</div>

hmmm ..any example would be really appreciated. And do you mean cut out this 911641?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 18, 2020, 4:04pm UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-write-grok-pattern-for-below-log/223646/9 "2020-03-18T16:04:25Z")

</div>

If you want an example of a custom format I suggest you ask in the elasticsearch forum. I don't run elasticsearch.

If you want to discard the last three characters you can use

```
mutate { gsub => ["timestamp", "...$", ""] }
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 15, 2020, 4:04pm UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-write-grok-pattern-for-below-log/223646/10 "2020-04-15T16:04:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.