Json object field with name "tags" get converted to array after logstash mutate

Elastic Stack Logstash
1

I have a json in the following format. I am reading this from kafka in logstash.
{"id":"8eba168be707e85d","kind":"PRODUCER","name":"scheduler","timestamp":1560526848185066,"duration":23645,"tags":{"class":"TestJob","method":"execute"}}

If I use the mutate filter plugin, the data inserted into elasticsearch will convert the "tags" field from object to array.

This is my current config.

input
 { 
   kafka { 
         bootstrap_servers => "localhost:9092"
         topics => ["log-topic-new"] 
         codec => json
      } 
} 
filter {	
	mutate {
		add_field => {"timestamp_millis"=>"%{timestamp}"}
	}
	mutate {
		gsub => ["timestamp_millis", ".{3}$", ""]
  }
  mutate {
    convert => {
      "timestamp_millis" => "integer"
    }
  }
}
  output {
	 #--------Debugging in console -------------------
	  stdout { codec => rubydebug } 
	 #--------Sending logs to es index  ------------------- 
	  elasticsearch { 
			hosts =>"192.168.45.43:9200"
			manage_template => false
			document_type => span
			index => "span-%{+YYYY-MM-dd}"
	   }
	 
}

The data in ES

_source": {
	"timestamp_millis": 1560526848185,
    "kind": "PRODUCER",
    "tags": [
        [
            "class",
            "TestJob"
        ],
        [
            "method",
            "execute"
        ]
    ],
    "duration": 23645,
    "name": "scheduler",
    "id": "477811021df4dadc",
    "timestamp": 1560526848185066
}

But if the mutate filter plugin is removed, the tags data is inserted as object itself.

2

In Logstash, the top-level tags field has long been special-cased as an array and typically treated in plugins as an unordered set.

The following untested ruby filter, inserted as your first filter, should move anytags with a Hash (map) value out of the way and save it as _tags.

filter {
  ruby {
    code => "
      tags = event.get('tags')
      if tags.kind_of?(Hash)
        event.set('_tags', tags)
        event.remove('tags')
      end
    "
  }
}
3

This will change my "tags" field to "_tags" right. I don't want to rename my field to "_tags" as the data retrieval from elasticsearch is not done by me.
So there is no way to write the tags field as it is to elasticsearch?

I see an open issue already present in github.

4

Found that this is fixed in the latest version in this issue description. I was using 6.5.1. Now I switched to the latest 7.1.1 and it is working as expected. The tags field is preserved as it is.
It would be better if was available in the older versions as well.

Thanks @yaauie for the reply.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.