Convert a field datatype to object or nested in logstash

Elastic Stack Logstash
1

I am unable to convert datatype of field to object or nested datatype. I get following error. Any suggestions appreciated.

error.PNG1351×537 24.7 KB

2

Please don't post pictures of text, they are difficult to read and some people may not be even able to see them.

3

Apart from what Mark said, post your configuration. The stacktrace alone isn't enough to debug this.

4

Use case is, there are two fields in my logs say "field1" and "field2". In elasticsearch I want to access field2 as field1.field2 . So at logstash level, I am trying to make field1 as array object.
This is my logstash config.

input{
tcp {

port => xxxx
codec=>"json"

}
}
filter {
csv {
columns => ["field1","field2"]
separator => ","
}
mutate {
convert => ["field1","object"]
rename => {"field2" => "[field1][field2]"}
}
}
output{
elasticsearch{
hosts => ["x.x.x.x:9200"]
index => "logstash-%{+YYYY.MM.dd}"
document_type =>"test"
manage_template => "false"
}
}

5

convert => ["field1","object"]

"object" isn't a valid value here.

I don't understand what you want the end result to be. Please give an example of the JSON document you want to get from your filters.

6

It is just like collect_set query in hive. I want to group by on one field and want to get array of unique values from other field.
Below is the JSON I got from elasticsearch after performing group by on 2 fields, city and hotels. Instead of that I want to group by on city and get distinct hotel names as array.

{ "key" : "Mumbai",
"doc_count" : 31348,
"hotels in mumbai" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 250,
"buckets" : [ {
"key" : "The Lalit",
"doc_count" : 20927
}, {
"key" : "Three Star",
"doc_count" : 5421
}, {
"key" : "Holiday Inn",
"doc_count" : 2295
}]
}
}

7

I meant the end result you want to get from Logstash. Please show

  • the event as it look when it enters Logstash and
  • the event as you want it to look when it's sent to Elasticsearch.