Problem with duplicate data in field

[ahongyun (Aylwinns)]

Hi all

I have an issue with the data that is passed into elasticsearch from logstash
below is my logstash config.

if [type] == "coreservices-fs" or "coreservices-net" {
grok {
match => { "message" => [
"^%{TIMESTAMP_ISO8601:Time} [%{DATA:Thread}] %{GREEDYDATA:Logger} : %{LOGLEVEL:Severity} - %{GREEDYDATA:KEYMessage}",
"Process with pid=%{NOTSPACE:PID} %{GREEDYDATA:Info}\nService stopped at %{GREEDYDATA:StopTime} by",
"Start time %{GREEDYDATA:StartTime} by" ] } }
grok {
match => { "source" => "/appdir/%{NOTSPACE:Env_Name}/logs/%{WORD:Server}.%{WORD}.%{WORD:NPID}" } } }

the data being seen on kibana are as follows:

as you can see, for several fields, the data is repeated multiple times.

I cant seem to figure out why.

Any help would be greatly appreciated.

Thank you

[magnusbaeck (Magnus Bäck)]

Please show the full configuration. All files in /etc/logstash/conf.d.

if [type] == "coreservices-fs" or "coreservices-mxnet" {

This conditional expression doesn't mean what you think it means. See Logstash sending same message to multiple indexes in ES - #2 by magnusbaeck.

[ahongyun (Aylwinns)]

Perfect.

Thanks

This is great.