# Logstash GROK

**URL:** <https://discuss.elastic.co/t/logstash-grok/248485>\
**Category:** Logstash\
**Created:** [September 14, 2020, 8:13am UTC](https://discuss.elastic.co/t/logstash-grok/248485 "2020-09-14T08:13:51Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jokie74](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jokie74/32/75504_2.png) [@Jokie74](https://discuss.elastic.co/u/Jokie74)\
**Post date:** [September 14, 2020, 8:13am UTC](https://discuss.elastic.co/t/logstash-grok/248485/1 "2020-09-14T08:13:51Z")

</div>

Hello Community,

I'm a bit lost with creating a grok filter and need some help...  
I am using filebeat for AIX to send errpt messages with syslog to logstash and Kibana to visualize...  
The Log coming from AIX error\_deamon is collected by syslog and looks that way:

```auto
Sep 11 08:32:27 svrseng3-0 local4:warn|warning root: IDENTIFIER: AA8AB241 Sequence Number: 36 Machine Id: 00C0C5504B00 Node Id: svrseng3-0 Class: O Type: TEMP WPAR: Global Resource Name: OPERATOR Description OPERATOR NOTIFICATION User Causes ERRLOGGER COMMAND Recommended Actions REVIEW DETAILED DATA Detail Data MESSAGE FROM ERRLOGGER COMMAND test for logstash active filters and output config

```

The filter should looks like:

```auto
filter {
        if [type] == "aix-beat" {
              grok {
                match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{SYSLOGFACILITY} %{USERNAME} %{GREEDYDATA:syslog_message}" }
                add_field => ["received_at", "%{@timestamp}"]
                add_field => ["received_from", "%{host}"]
                }
                date {
                  match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]
                }
        }
}

```

Unfortunately, it is not working...  
When I try to start logstash with this filter I get an error and logstash is shutting down again...

```auto
[2020-09-14T10:04:37,705][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Duplicate keys found in your configuration: [add_field]\nAt line: 4, column 22 (byte 78)\nafter filter {\n if [type] == \"aix-beat\" {\n grok {\n match => {", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler/lscl.rb:183:in `validate!'"

```

Can anybody help me out? I'm trying to get that work since a week...

Thanks a lot in advance your your support.

Regards  
Joerg

---

<div class="post-metadata">

**Author:** ![Wolfram\_Haussig](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wolfram_haussig/32/70528_2.png) [@Wolfram\_Haussig](https://discuss.elastic.co/u/Wolfram_Haussig)\
**Post date:** [September 14, 2020, 8:32am UTC](https://discuss.elastic.co/t/logstash-grok/248485/2 "2020-09-14T08:32:58Z")

</div>

Hello Jörg,

Welcome to this forum! The error message is very clear about the root cause:

> Duplicate keys found in your configuration: [add\_field]

In your grok filter, you have the property add\_field twice and this is the error. You can add multiple fields like this:

```auto
filter {
        if [type] == "aix-beat" {
              grok {
                match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{SYSLOGFACILITY} %{USERNAME} %{GREEDYDATA:syslog_message}" }
                add_field => {
                          "received_at" => "%{@timestamp}"
                          "received_from" => "%{host}" 
                    }
                }
                date {
                  match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]
                }
        }
}

```

See here for details: [Grok filter plugin | Logstash Reference [master] | Elastic](https://www.elastic.co/guide/en/logstash/master/plugins-filters-grok.html#plugins-filters-grok-add_field)

Best regards  
Wolfram

---

<div class="post-metadata">

**Author:** ![Jokie74](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jokie74/32/75504_2.png) [@Jokie74](https://discuss.elastic.co/u/Jokie74)\
**Post date:** [September 14, 2020, 8:49am UTC](https://discuss.elastic.co/t/logstash-grok/248485/3 "2020-09-14T08:49:56Z")

</div>

Found it, thanks a lot... it was a wrongly set "}".

Now it works so far... but all entries are doubled in Kibana...  
Any Idea where that comes from?

best regards  
Joerg

---

<div class="post-metadata">

**Author:** ![Wolfram\_Haussig](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wolfram_haussig/32/70528_2.png) [@Wolfram\_Haussig](https://discuss.elastic.co/u/Wolfram_Haussig)\
**Post date:** [September 14, 2020, 8:52am UTC](https://discuss.elastic.co/t/logstash-grok/248485/4 "2020-09-14T08:52:03Z")

</div>

I don't think LogStash does the duplication - maybe the syslog message is sent twice? We once had the problem that the message was written to 2 different loggers which then forwarded it to syslog...

---

<div class="post-metadata">

**Author:** ![Jokie74](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jokie74/32/75504_2.png) [@Jokie74](https://discuss.elastic.co/u/Jokie74)\
**Post date:** [September 14, 2020, 9:08am UTC](https://discuss.elastic.co/t/logstash-grok/248485/5 "2020-09-14T09:08:42Z")

</div>

The message is written only one time in the logfile on the sending host...  
So you mean filebeat could be the problem??

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 12, 2020, 9:08am UTC](https://discuss.elastic.co/t/logstash-grok/248485/6 "2020-10-12T09:08:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.