Ingest Relational DB data with outer join relationships to Nested objects in Elasticsearch through Logstash

[NSK]

Hello All - I am trying to ingest Oracle Data into Elasticsearch. Here is the data structure

number number_type Comments
01 S abc
01 F cbd
02 A xyz

and wanted to see the output as nested objects
"mappings": {
"test": {
"properties": {
"NUMBER" : {"type" : "integer"},
"COMMENTS" :{
"type": "nested",
"properties" : {
"NUMBER_TYPE" : {"type" : "integer"},
"COMMENTS" : {"type" : "string"}
}
}
}

How do I parse the NUMBER_TYPE and COMMENTS column in Nested objects through logstash filters.

I read in a blog, where it talked about mutate filter option but really not sure how to use them in the logstash conf file.

Any pointers or examples on handling outer joins in logstash are greatly appreciated.

[magnusbaeck (Magnus Bäck)]

How do I parse the NUMBER_TYPE and COMMENTS column in Nested objects through logstash filters.

I read in a blog, where it talked about mutate filter option but really not sure how to use them in the logstash conf file.

Use a mutate filter and its rename option. There's an example in the filter's documentation. To created a nested field, use the [field][subfield] syntax described here: Accessing event data and fields | Logstash Reference [8.11] | Elastic

[NSK]

Thanks for the response.

I did try the rename option, however the hlog_comments fields is not getting displayed.

filter {
mutate {
rename => {"SEQUENCE_NUMBER" => "[HLOG_COMMENTS][SEQUENCE_NUMBER]" }
rename => {"USERID" => "[HLOG_COMMENTS][USERID]" }
rename => {"ACTIVITY" => "[HLOG_COMMENTS][ACTIVITY]" }
rename => {"HEADERLOG_STATE" => "[HLOG_COMMENTS][HLOG_STATE]" }
rename => {"HEADERLOG_STATUS" => "[HOG_COMMENTS][HLOG_STATUS]" }
......
-......
........

[NSK]

hlog_comment is the nested object.

[magnusbaeck (Magnus Bäck)]

Please show what you get from a stdout { codec => rubydebug } output so we can see exactly what your events look like.