# Remove { and " from field name? **URL:** \ **Category:** Logstash\ **Created:** [November 9, 2017, 6:56am UTC](https://discuss.elastic.co/t/remove-and-from-field-name/106977 "2017-11-09T06:56:43Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Sjaak01](https://avatars.discourse-cdn.com/v4/letter/s/73ab20/32.png) [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Post date:** [November 9, 2017, 6:56am UTC](https://discuss.elastic.co/t/remove-and-from-field-name/106977/1 "2017-11-09T06:56:43Z")
Hi, I'm outputting some log data to disk where and logstash is marking the first field name with { and the last field value with }. For whatever reason I also have a field name called "date so that field isn't parsed by the KV filter properly. I'm trying to use gsub to remove " and }, which works on field values but not on field names. That in itself is strange because the KV filter comes after gsub so at that point it should only be text. Is there any way I can remove { } and "? Especially the "date field is affected as it prevents me from creating a new @timestamp. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 9, 2017, 7:00am UTC](https://discuss.elastic.co/t/remove-and-from-field-name/106977/2 "2017-11-09T07:00:40Z")
Please show example instead of describing what thing look like. ---
**Author:** ![Sjaak01](https://avatars.discourse-cdn.com/v4/letter/s/73ab20/32.png) [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Post date:** [November 10, 2017, 1:02am UTC](https://discuss.elastic.co/t/remove-and-from-field-name/106977/3 "2017-11-10T01:02:43Z")
**On the remote client** ```auto filter { if [type] == "fortilog" { mutate { gsub => [ "message", "[\=]", ":" ] } grok { match => ["message", "%{SYSLOG5424PRI:syslog_index}%{GREEDYDATA:message}"] overwrite => ["message"] tag_on_failure => ["failure_grok_fortigate"] } mutate { add_field => { "location_field" => "Location_A" } gsub => [ "date", "[\"]", "" ] } } } ``` The second gsub doesn't appear to be doing anything. The above results in the below file (.txt) output on the remote client. ```auto {"@timestamp":"2017-11-10T00:46:12.687Z","syslog_index":"<188>","syslog5424_pri":"188","@version":"1","host":"10.0.0.111","message":"date:2017-11-10, time:00:46:09,type:traffic,subtype:other,msg:\"iprope_in_check() check failed, drop\"","type":"fortilog","location_field":"Location_A"} ``` The above output causes several problems when I try to ingest the file on the server side logstash. 1. @timestamp becomes {@timestamp. Location\_A becomes Location\_A} 2. In the message field, date is shown as "date which causes issues later on. 3. The Fortigate logs include a type field. As logstash is adding a type field as well that is causing issues. I tried renaming the type field to log\_type if the value is not fortilog (the logstash type value) but I can't get it to work. **Filter on the logstash server** ```auto filter { mutate { gsub => [ "message", "[\\"\}]", "" ] } kv { value_split => ":" field_split => "," } } output { stdout { codec => rubydebug } } ``` Results in ```auto "msg" => "iprope_in_check() check failed", "message" => "date:2017-11-10", "type" => [ [0] "traffic", [1] "fortilog" ], "path" => "/home/test/Desktop/test/test1.txt", "@timestamp" => 2017-11-10T00:57:01.483Z, "syslog_index" => "188", "subtype" => "other", "{@timestamp" => "2017-11-10T00:46:12.687Z", "syslog5424_pri" => "188", "@version" => "1", "host" => "10.0.0.111", "time" => "00:46:09", "location_field" => "Location_A" ``` The "date is probably breaking the date field and the { I cannot remove by adding { to the gsub filter. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 10, 2017, 6:06am UTC](https://discuss.elastic.co/t/remove-and-from-field-name/106977/4 "2017-11-10T06:06:14Z")
Don't use gsub to parse JSON. Remove all those filteres. Use a json codec in your input plugin, then a kv filter to parse the `message` field. ---
**Author:** ![Sjaak01](https://avatars.discourse-cdn.com/v4/letter/s/73ab20/32.png) [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Post date:** [November 10, 2017, 8:31am UTC](https://discuss.elastic.co/t/remove-and-from-field-name/106977/5 "2017-11-10T08:31:36Z")
Thanks, I will give it a try. You mean the config on the server, correct? The one the remote client is OK like this? If I remove the gsub from that one the KV filter on the server side becomes problematic. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 13, 2017, 6:17am UTC](https://discuss.elastic.co/t/remove-and-from-field-name/106977/6 "2017-11-13T06:17:43Z")
I don't think you need any of your gsub filters. But please post an example of an original and unprocessed log message. Reverse engineering what it looks like based on the end results and filters is tiresome. ---
**Author:** ![Sjaak01](https://avatars.discourse-cdn.com/v4/letter/s/73ab20/32.png) [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Post date:** [November 14, 2017, 1:53am UTC](https://discuss.elastic.co/t/remove-and-from-field-name/106977/7 "2017-11-14T01:53:11Z")
Removing the gsub and adding the json codec to my server side config was all that was needed to make the data look pretty. Thanks. The only issue I have left is that of the type field. The Fortigate logs have a type field that, in my current config, gets used by elastic. As the type field in the Fortigate logs is changing based on the data I'm not sure if that is going to be an issue. I'd rather rename that field and use something like fortigate\_logs as the type field to make it easier to search. What would to best way to rename the type field in the message field? Raw data ```auto {"@version":"1","host":"10.0.0.111","@timestamp":"2017-11-14T01:14:36.195Z","message":"<188>date=2017-11-14,time=01:14:33,devname=name,device_id=FGT60C3,log_id=0038000006,type=traffic,subtype=other,pri=warning,vd=root,src=10.0.0.149,src_port=0,src_int=\"root\",dst=10.0.0.111,dst_port=771,dst_int=\"wan1\",SN=57612,status=deny,policyid=0,dst_country=\"Reserved\",src_country=\"Reserved\",service=3/3/icmp,proto=1,duration=241309,sent=0,rcvd=0,msg=\"check fail on allow error, drop.\"","type":"fortilog"} ``` I do need to apply the gsub in the remote logstash config otherwise data won't look pretty. Config is the same as in the first post. Server config ```auto input { file { path => "/home/test/Desktop/test/comma2.txt" sincedb_path => "/dev/null" start_position => "beginning" codec => json { } } } filter { kv { value_split => ":" field_split => "," } mutate { add_field => { "temp_time" => "%{date} %{time}" } } date { match => ["temp_time", "yyyy-MM-dd HH:mm:ss"] timezone => "UTC" target => "@timestamp" } mutate { remove_field => ["syslog_index","syslog5424_pri"] } } output { stdout { codec => rubydebug } } ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 14, 2017, 6:05am UTC](https://discuss.elastic.co/t/remove-and-from-field-name/106977/8 "2017-11-14T06:05:37Z")
Surely you want `value_split` to be `=` and not `:`? Once you've extraced the fields inside `message` into fields of their own you can just use a mutate filter to update the `type` value. ---
**Author:** ![Sjaak01](https://avatars.discourse-cdn.com/v4/letter/s/73ab20/32.png) [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Post date:** [November 15, 2017, 2:07am UTC](https://discuss.elastic.co/t/remove-and-from-field-name/106977/9 "2017-11-15T02:07:56Z")
Because the Fortigate logs are separated by `=`. ```auto mutate { gsub => ["message", "[\=]", ":" ] } ``` Is what I use on the remote client to solve that problem. On the server I can then use `value_split` `:`. I'm sure there are better/prettier ways but a lack of skill is holding me back. ---
**Author:** ![Sjaak01](https://avatars.discourse-cdn.com/v4/letter/s/73ab20/32.png) [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Post date:** [November 16, 2017, 1:30am UTC](https://discuss.elastic.co/t/remove-and-from-field-name/106977/10 "2017-11-16T01:30:49Z")
And again the KV filter refuses to parse data if I put it in my remote client config.... I'm trying to make things a bit prettier and have the remote client do all the filtering so I could put the data directly in Elastic if I wanted to. Just as per a previous topic I opened, the KV filter refuses to work when value\_split matches the messages. Logstash is running, no errors, its just not generating any data. As soon as I change the value\_split to something that doesn't match the logs, suddenly it starts saving (unfiltered) data. ```auto input { udp { port => 9910 type => "fortilog" } } filter { if [type] == "fortilog" { mutate { gsub => [ "message", "[\\"]", "" ] } grok { match => ["message", "%{SYSLOG5424PRI:syslog_index}%{GREEDYDATA:message}"] overwrite => ["message"] tag_on_failure => ["failure_grok_fortigate"] } mutate { gsub => [ "message", "[\=]", ":" ] } kv { value_split => ":" field_split => "," } } } output { if [type] == "fortilog" { file { path => "/home/test/Desktop/test/forti/test-%{+YYYY-MM-dd.HH}.gz" gzip => true } } } ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 16, 2017, 6:29am UTC](https://discuss.elastic.co/t/remove-and-from-field-name/106977/11 "2017-11-16T06:29:59Z")
The raw data you posted earlier looks fine. I can't spot any "=" characters except the key/value separators. > Logstash is running, no errors, its just not generating any data. I don't believe that's true. > As soon as I change the value\_split to something that doesn't match the logs, suddenly it starts saving (unfiltered) data. Use a `stdout { codec => rubydebug }` output while you're debugging this. With your current file output you won't see anything but the `message` field. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 14, 2017, 6:30am UTC](https://discuss.elastic.co/t/remove-and-from-field-name/106977/12 "2017-12-14T06:30:05Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.