Custom add_field storing as an array with value duplicated in Elasticsearch

[nodesocket (Justin)]

Hello. I am trying to add a custom field msgid and while the groc and field is working, the result is an array with the value duplicated twice.

Here is an example raw message:

Jan 26 22:58:01 ip-172-31-23-201 mailqueued: [1822bd7c-02e6-11e8-b20a-b566ad59a605] [90] Retry message in 120 minutes (retry: 3 of 30, increment_retry)

And the Logstash groc match:

%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: \[%{UUID:msgid}\] %{GREEDYDATA:syslog_message}

The Logstash field:

add_field => [ "msgid", "%{msgid}" ]

In Elasticsearch it is saving as:

"msgid": [
    "1822bd7c-02e6-11e8-b20a-b566ad59a605",
    "1822bd7c-02e6-11e8-b20a-b566ad59a605"
]

I want to store as a flat value, no array, and no duplicates. Ideas?

[Vladi (Vladimir Smorodinov)]

@nodesocket

The Logstash field:

add_field => [ "msgid", "%{msgid}" ]

Justin, your output is exactly correct.
The code add_field => [ "msgid", "%{msgid}" ] just dupicates the value from grok.
"..(?:[%{POSINT:syslog_pid}])?: [%{UUID:msgid}] %{GREEDYDATA:syslog_message}.."
Just comment the line #add_field => [ "msgid", "%{msgid}" ] and it will be solved. Or rename you custome field add_field => [ "msgid_renamed", "%{msgid}" ]

[nodesocket (Justin)]

@Vladi thanks for the reply. My misunderstanding. I thought I had to setup the groc to capture, and then add_field separately. Thanks so much.