Remove { and " from field name?

[Sjaak01 (Sjaak)]

Hi,

I'm outputting some log data to disk where and logstash is marking the first field name with { and the last field value with }. For whatever reason I also have a field name called "date so that field isn't parsed by the KV filter properly.

I'm trying to use gsub to remove " and }, which works on field values but not on field names. That in itself is strange because the KV filter comes after gsub so at that point it should only be text.

Is there any way I can remove { } and "? Especially the "date field is affected as it prevents me from creating a new @timestamp.

[magnusbaeck (Magnus Bäck)]

Please show example instead of describing what thing look like.

[Sjaak01 (Sjaak)]

On the remote client

filter {
 if [type] == "fortilog" {

 mutate {
    gsub => [
      "message", "[\=]", ":"
    ]
  }

		grok {
			match => ["message", "%{SYSLOG5424PRI:syslog_index}%{GREEDYDATA:message}"]
			overwrite => [ "message" ]
			tag_on_failure => [ "failure_grok_fortigate" ]
		}

 mutate {
    add_field => { "location_field" => "Location_A" }
    gsub => [
      "date", "[\"]", ""
	]
}
}
}

The second gsub doesn't appear to be doing anything.

The above results in the below file (.txt) output on the remote client.

{"@timestamp":"2017-11-10T00:46:12.687Z","syslog_index":"<188>","syslog5424_pri":"188","@version":"1","host":"10.0.0.111","message":"date:2017-11-10,
time:00:46:09,type:traffic,subtype:other,msg:\"iprope_in_check() check failed, drop\"","type":"fortilog","location_field":"Location_A"}

The above output causes several problems when I try to ingest the file on the server side logstash.

1. @timestamp becomes {@timestamp. Location_A becomes Location_A}
2. In the message field, date is shown as "date which causes issues later on.
3. The Fortigate logs include a type field. As logstash is adding a type field as well that is causing issues. I tried renaming the type field to log_type if the value is not fortilog (the logstash type value) but I can't get it to work.

Filter on the logstash server

filter {

mutate {
    gsub => [
      "message", "[\\"\}]", ""
    ]
  }

        kv {

    value_split => ":"
    field_split => ","
}
}



output {
  stdout {
    codec => rubydebug
  }
}

Results in

               "msg" => "iprope_in_check() check failed",
           "message" => "date:2017-11-10",
              "type" => [
        [0] "traffic",
        [1] "fortilog"
    ],
              "path" => "/home/test/Desktop/test/test1.txt",
        "@timestamp" => 2017-11-10T00:57:01.483Z,
      "syslog_index" => "188",
           "subtype" => "other",
       "{@timestamp" => "2017-11-10T00:46:12.687Z",
    "syslog5424_pri" => "188",
          "@version" => "1",
              "host" => "10.0.0.111",
              "time" => "00:46:09",
    "location_field" => "Location_A"

The "date is probably breaking the date field and the { I cannot remove by adding { to the gsub filter.

[magnusbaeck (Magnus Bäck)]

Don't use gsub to parse JSON. Remove all those filteres. Use a json codec in your input plugin, then a kv filter to parse the message field.

[Sjaak01 (Sjaak)]

Thanks, I will give it a try.

You mean the config on the server, correct? The one the remote client is OK like this? If I remove the gsub from that one the KV filter on the server side becomes problematic.

[magnusbaeck (Magnus Bäck)]

I don't think you need any of your gsub filters. But please post an example of an original and unprocessed log message. Reverse engineering what it looks like based on the end results and filters is tiresome.

[Sjaak01 (Sjaak)]

Removing the gsub and adding the json codec to my server side config was all that was needed to make the data look pretty. Thanks.

The only issue I have left is that of the type field. The Fortigate logs have a type field that, in my current config, gets used by elastic. As the type field in the Fortigate logs is changing based on the data I'm not sure if that is going to be an issue. I'd rather rename that field and use something like fortigate_logs as the type field to make it easier to search.

What would to best way to rename the type field in the message field?

Raw data

{"@version":"1","host":"10.0.0.111","@timestamp":"2017-11-14T01:14:36.195Z","message":"<188>date=2017-11-14,time=01:14:33,devname=name,device_id=FGT60C3,log_id=0038000006,type=traffic,subtype=other,pri=warning,vd=root,src=10.0.0.149,src_port=0,src_int=\"root\",dst=10.0.0.111,dst_port=771,dst_int=\"wan1\",SN=57612,status=deny,policyid=0,dst_country=\"Reserved\",src_country=\"Reserved\",service=3/3/icmp,proto=1,duration=241309,sent=0,rcvd=0,msg=\"check fail on allow error, drop.\"","type":"fortilog"}

I do need to apply the gsub in the remote logstash config otherwise data won't look pretty. Config is the same as in the first post.

Server config

input {
  file {
    path => "/home/test/Desktop/test/comma2.txt"
    sincedb_path => "/dev/null"
    start_position => "beginning"
	codec => json {
}
}
}


filter {

        kv {

    value_split => ":"
    field_split => ","
}

    mutate {
    add_field => { "temp_time" => "%{date} %{time}" }
}

date {
    match => [ "temp_time", "yyyy-MM-dd HH:mm:ss" ]
    timezone => "UTC"
    target => "@timestamp"
    }

    mutate {
    remove_field => ["syslog_index","syslog5424_pri"]
}
}




output {
  stdout {
    codec => rubydebug
  }
}

[magnusbaeck (Magnus Bäck)]

Surely you want value_split to be = and not :? Once you've extraced the fields inside message into fields of their own you can just use a mutate filter to update the type value.

[Sjaak01 (Sjaak)]

Because the Fortigate logs are separated by =.

 mutate {
    gsub => [ "message", "[\=]", ":" ]
  }

Is what I use on the remote client to solve that problem.

On the server I can then use value_split :. I'm sure there are better/prettier ways but a lack of skill is holding me back.

[Sjaak01 (Sjaak)]

And again the KV filter refuses to parse data if I put it in my remote client config.... I'm trying to make things a bit prettier and have the remote client do all the filtering so I could put the data directly in Elastic if I wanted to.

Just as per a previous topic I opened, the KV filter refuses to work when value_split matches the messages. Logstash is running, no errors, its just not generating any data. As soon as I change the value_split to something that doesn't match the logs, suddenly it starts saving (unfiltered) data.

input {
   udp {
     port => 9910
    type => "fortilog"
  }
}


filter {
 if [type] == "fortilog" {

 mutate {
    gsub => [
      "message", "[\\"]", ""
    ]
  }

	grok {
			match => ["message", "%{SYSLOG5424PRI:syslog_index}%{GREEDYDATA:message}"]
			overwrite => [ "message" ]
			tag_on_failure => [ "failure_grok_fortigate" ]
		}

 mutate {
    gsub => [
      "message", "[\=]", ":"
    ]
  }

        kv {

    value_split => ":"
    field_split => ","
}
}
}


output {
 if [type] == "fortilog" {
 file {
   path => "/home/test/Desktop/test/forti/test-%{+YYYY-MM-dd.HH}.gz"
   gzip => true
 }
}
}

[magnusbaeck (Magnus Bäck)]

The raw data you posted earlier looks fine. I can't spot any "=" characters except the key/value separators.

Logstash is running, no errors, its just not generating any data.

I don't believe that's true.

As soon as I change the value_split to something that doesn't match the logs, suddenly it starts saving (unfiltered) data.

Use a stdout { codec => rubydebug } output while you're debugging this. With your current file output you won't see anything but the message field.