Split Logfiles

[A.Klos]

Hi,

we have following logformat:

2018-03-15 11:20:46|Field1|Field2|Field3|Field4|Field5|Field6

2018-03-15 11:30:46|Field1| |Field3| |Field5|Field6

How could I split this into fields?

currently my grok pattern (not all fields ):

match => [ "message" , "%{TIMESTAMP_ISO8601:timestamp}|%{WORD:field1}|%{WORD:field2}|%{WORD:field3}|%{WORD:field4}|%{GREEDYDATA:msg}" ]

Some Fields also could be empty.

How do I split those logs into seperate fields?

Regards

[magnusbaeck (Magnus Bäck)]

You may find the dissect filter easier to use in this case.

[A.Klos]

I think it would be possible by split:

mutate{
split => {"message" => "|"}
}

Output is now:

[0] Field1
[1] Field2
[2] Field3

and so on

If I try to add [1] into a field I got grokparsefailure
add_field => [ "received_from", "%{message}[3]" ]

What's wrong?

[A.Klos]

Not it looks working:

if [type] == "FilePluginInput" {
grok {

      match => [ "message" , "%{TIMESTAMP_ISO8601:timestamp}\|%{GREEDYDATA:nachricht}" ]
  }
  mutate{
    split => {"nachricht" => "|"}
     add_field => ["severity", "%{[nachricht][7]}" ]
     add_field => ["msg", "%{[nachricht][9]}" ]

}
date {
match => [ "timestamp", "yyyy-MM-dd HH:mm:ss" ]
timezone => [ "Europe/Berlin" ]
target => "@timestamp"
}
}
}

Because whe have 2 differnt seperators (; and |), would it be possible using if then else statement?

If seperator = "|" then
if [type] == "FilePluginInput" {
grok {

      match => [ "message" , "%{TIMESTAMP_ISO8601:timestamp}\|%{GREEDYDATA:nachricht}" ]
  }
  mutate{
    split => {"nachricht" => "|"}
     add_field => ["severity", "%{[nachricht][7]}" ]
     add_field => ["msg", "%{[nachricht][9]}" ]

}
date {
match => [ "timestamp", "yyyy-MM-dd HH:mm:ss" ]
timezone => [ "Europe/Berlin" ]
target => "@timestamp"
}
}
}
else
if [type] == "FilePluginInput" {
grok {

      match => [ "message" , "%{TIMESTAMP_ISO8601:timestamp}\|%{GREEDYDATA:nachricht}" ]
  }
  mutate{
    split => {"nachricht" => ";"}
     add_field => ["severity", "%{[nachricht][7]}" ]
     add_field => ["msg", "%{[nachricht][9]}" ]

}
date {
match => [ "timestamp", "yyyy-MM-dd HH:mm:ss" ]
timezone => [ "Europe/Berlin" ]
target => "@timestamp"
}
}
}