Problem with kv filter

Elastic Stack Logstash
1

Hello everyone, I have the following situation, I am reading the logs from a file and these come as follows:

  • Jun 12 23:29:08 172.17.6.10 field1="value1" field2=value2 field3=value3 ... etc

My input, filter and output in my logstash config are like this:

input {
    	file {
        	path => [ "/var/log/fortigate/file.log" ]
        	start_position => "beginning"
        	type =>"firewall"
    	}
}

filter {
    if [type] == "firewall" {

    	mutate {
        	gsub => ["message","\"\"","NULL",
			 "message","\"","",
			 "message"," "," "]
    	}

  	grok {
    		match => { "message" => "%{DATA:month} %{NUMBER:day} %{TIME:timestamp} %{IP:ip} %{GREEDYDATA:rest}" }
  	} 

	kv {
  		source => "rest"
  		value_split => "=" 
		field_split => " "
  		remove_field => [ "rest" ]
	}

    }
}

output {

  if [type] == "firewall" {
     elasticsearch {
       hosts => ["x.x.x.20:9200","x.x.x.21:9200"]
       index => "fortinet-%{+YYYY.MM.dd}" 
     }
  }
}

When I go to kibana to generate the index it does not appear to me, however when I remove the kv filter the index appears to be generated. There are plenty of fields that have the log. What I want is to separate the fields from the log.

I noticed for example that i have the next field: devtype = Android Phone, that can cause the kv filter to fall because of the space?

There are several fields that must be disaggregated, that can generate some problem? the fields are these:

  • logver=56
  • timestamp=1560396270
  • tz=UTC-4
  • devname=FGT-HA
  • devid=TEXT
  • vd=TEXT
  • date=2019-06-13
  • time=00:24:30
  • logid=NUMBER
  • type=traffic
  • subtype=forward
  • level=notice
  • eventtime=1560396270
  • srcip=IP
  • srcport=PORT
  • srcintf=TEXT
  • srcintfrole=TEXT
  • dstip=IP
  • dstport=PORT
  • dstintf=TEXT
  • dstintfrole=wan
  • poluuid=TEXT
  • sessionid=NUMBER
  • proto=6
  • action=client-rst
  • policyid=50
  • policytype=policy
  • service=HTTPS
  • dstcountry=Chile
  • srccountry=Reserved
  • trandisp=noop
  • appcat=unknown
  • applist=TEXT
  • duration=106
  • sentbyte=NUMBER
  • rcvdbyte=NUMBER
  • sentpkt=14
  • shaperperipname=TEXT
  • shaperperipdropbyte=0
  • devtype=Android Phone
  • osname=Android
  • osversion=4.4.2
  • mastersrcmac=MAC
  • srcmac=MAC
  • srcserver=0

If you can guide me on how I can disaggregate the fields I would be very grateful

2

In modern Logstashes (or older ones with updated KV Filter Plugin), you can set whitespace => strict, which will not allow spaces around the field separator. This can greatly improve the performance of the KV filter's parser when given inputs that have unquoted values including spaces, since it eliminates ambiguity and reduces backtracking in the parser.

Newer versions of the KV filter also introduce a timeout enforcer, which gives it the ability to interrupt stuck parsers.

If you have control of the shape of your logs, I would also recommend adding quotes around your values that contain spaces.

To update the plugin, see: Logstash: Working With Plugins

3

Thank you very much for the response, but I still can not generate the index. When I restart the logstash service to run the new configuration, is there a place where I can verify which part of my configuration file failed? I have reviewed all the publications regarding this topic and I still do not find a solution

4

What do the Logstash logs tell you? For help finding the logs, see: https://www.elastic.co/guide/en/logstash/current/dir-layout.html

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.