Auto Field Detection in Logs

gringo · February 24, 2016, 1:13am

Is there a way to write grok pattern such that it will automatically detect fields and parse it into elasticsearch. I am hoping to eliminate the need to constantly create grok patterns to recognize new log patterns

For example all the fields start with field_name = fieldvalue
src_ip = 10.1.1.223 src_port = 8080

magnusbaeck · February 24, 2016, 6:45am

Look at the kv filter.

gringo · February 24, 2016, 7:24am

I can see that within Logstash there is already predefined grok patterns for firewall like Juniper.

So under what kind of situation should we use customized grok pattern and what kind of situation should we use the KV filter?

magnusbaeck · February 24, 2016, 8:44am

If it's key/value data I see no reason to use grok unless the keys are very static (in which case the maintenance burden is low).

gringo · February 25, 2016, 12:50am

So why would the Logstash contain a grok pattern for Juniper while the same can be achieved with kv filter? I am trying understand the rational of why it is there.

Is there a way to modify the KV filter such that its can process key/value in the form of "mykey:myvalue" instead of the existing form "mykey=myvalue"?

magnusbaeck · February 25, 2016, 3:26pm

So why would the Logstash contain a grok pattern for Juniper while the same can be achieved with kv filter?

I don't know.

Is there a way to modify the KV filter such that its can process key/value in the form of "mykey:myvalue" instead of the existing form "mykey=myvalue"?

Yes. Please explore the various configuration options listed in the kv filter documentation.

gringo · March 1, 2016, 3:08am

Thank you for your help! I am now able to do what I need