I have my_index with a double nested property (nested of nested) and i'm looking for the right logshash configuration to correctly index documents with the following mappings:
"mappings": {
"properties": {
"id": {"type": "long"},
"name": {"type": "text"},
"age": {"type": "integer"},
"children": {
"type": "nested",
"properties": {
"id": {"type": "keyword"},
"name": {"type": "text"},
"age": {"type": "integer"},
"grandchildren": {
"type": "nested",
"properties": {
"id": {"type": "keyword"},
"name": {"type": "text"},
"age": {"type": "integer"}
}
}
}
}
}
}
Database structure:
Parent
id | age | name
----------------------------------------------------------------------------------------
1 | 58 | James
2 | 62 | Carl
Child
id | age | name | parent_id
-----------------------------------------------------------------------------------------------------------------
1 | 39 | David | 1
2 | 37 | Emma | 1
3 | 41 | Ned | 2
GrandChild
id | age | name | child_id
-----------------------------------------------------------------------------------------------------------------
1 | 7 | Matt | 1
2 | 3 | Zoe | 1
3 | 1 | Vittoria | 1
4 | 4 | Loki | 3
5 | 6 | Oscar | 3
Logstash configuration:
input {
jdbc {
...
statement => "
select
p.id, p.age, p.name,
c.id c_id, c.age c_age, c.name c_name,
gc.id gc_id, gc.age gc_age, gc.name gc_name
from parent p
left outer join child c on c.parent_id = p.id
left outer join grandchild gc on gc.child_id = c.id"
...
}
}
filter {
mutate {
...
rename => { "c_id" => "[children][id]"}
rename => { "c_age" => "[children][age]"}
rename => { "c_name" => "[children][name]"}
rename => { "gc_id" => "[children][grandchildren][id]"}
rename => { "gc_age" => "[children][grandchildren][age]"}
rename => { "gc_name" => "[children][grandchildren][name]"}
}
aggregate {
task_id => "%{id}"
code => "
...
map['children'] ||= []
map['children'] <<= event.get('children')
event.cancel()
"
push_previous_map_as_event => true
timeout => 5
timeout_tags => ["aggregate"]
}
ruby {
code => "
event.set('valutazioni.importi', event.get('valutazioni.importi').uniq)
"
}
}
output {
elasticsearch {
index => "my_index"
hosts => [".."]
document_id => "%{id}"
}
stdout { codec => dots }
}
What I GET (wrong) querying my_index with doc_id = 1:
{
"_index" : "my_index",
"_type" : "_doc",
"_id" : "1",
"_source" : {
"id" : 1,
"name" : "James",
"age" : 58,
"children" : [
{
"id" : 1,
"name" : "David",
"age" : 39,
"grandchildren" : [
{
"id" : 1,
"name" : "Matt",
"age" : 7
}
]
},
{
"id" : 1,
"name" : "David",
"age" : 39,
"grandchildren" : [
{
"id" : 2,
"name" : "Zoe",
"age" : 3
}
]
},
{
"id" : 1,
"name" : "David",
"age" : 39,
"grandchildren" : [
{
"id" : 3,
"name" : "Vittoria",
"age" : 1
}
]
},
{
"id" : 2,
"name" : "Emma",
"age" : 37,
"grandchildren" : []
}
]
}
}
What I WANT:
{
"_index" : "my_index",
"_type" : "_doc",
"_id" : "1",
"_source" : {
"id" : 1,
"name" : "James",
"age" : 58,
"children" : [
{
"id" : 1,
"name" : "David",
"age" : 39,
"grandchildren" : [
{
"id" : 1,
"name" : "Matt",
"age" : 7
},
{
"id" : 2,
"name" : "Zoe",
"age" : 3
},
{
"id" : 3,
"name" : "Vittoria",
"age" : 1
}
]
},
{
"id" : 2,
"name" : "Emma",
"age" : 37,
"grandchildren" : []
}
]
}
}
I am able to get my goal in Java with Spring-elasticsearch (therefore i know the mapping is correct), but I NEED to do it via Logstash!
Any suggestion is very appreciated, thanks.