I am unable to convert datatype of field to object or nested datatype. I get following error. Any suggestions appreciated.
Please don't post pictures of text, they are difficult to read and some people may not be even able to see them.
Apart from what Mark said, post your configuration. The stacktrace alone isn't enough to debug this.
Use case is, there are two fields in my logs say "field1" and "field2". In elasticsearch I want to access field2 as field1.field2 . So at logstash level, I am trying to make field1 as array object.
This is my logstash config.
input{
tcp {
port => xxxx
codec=>"json"
}
}
filter {
csv {
columns => ["field1","field2"]
separator => ","
}
mutate {
convert => ["field1","object"]
rename => {"field2" => "[field1][field2]"}
}
}
output{
elasticsearch{
hosts => ["x.x.x.x:9200"]
index => "logstash-%{+YYYY.MM.dd}"
document_type =>"test"
manage_template => "false"
}
}
convert => ["field1","object"]
"object" isn't a valid value here.
I don't understand what you want the end result to be. Please give an example of the JSON document you want to get from your filters.
It is just like collect_set query in hive. I want to group by on one field and want to get array of unique values from other field.
Below is the JSON I got from elasticsearch after performing group by on 2 fields, city and hotels. Instead of that I want to group by on city and get distinct hotel names as array.
{ "key" : "Mumbai",
"doc_count" : 31348,
"hotels in mumbai" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 250,
"buckets" : [ {
"key" : "The Lalit",
"doc_count" : 20927
}, {
"key" : "Three Star",
"doc_count" : 5421
}, {
"key" : "Holiday Inn",
"doc_count" : 2295
}]
}
}
I meant the end result you want to get from Logstash. Please show
- the event as it look when it enters Logstash and
- the event as you want it to look when it's sent to Elasticsearch.