Ingest Relational DB data with outer join relationships to Nested objects in Elasticsearch through Logstash

Elastic Stack Logstash
1

Hello All - I am trying to ingest Oracle Data into Elasticsearch. Here is the data structure

number number_type Comments
01 S abc
01 F cbd
02 A xyz

and wanted to see the output as nested objects
"mappings": {
"test": {
"properties": {
"NUMBER" : {"type" : "integer"},
"COMMENTS" :{
"type": "nested",
"properties" : {
"NUMBER_TYPE" : {"type" : "integer"},
"COMMENTS" : {"type" : "string"}
}
}
}

How do I parse the NUMBER_TYPE and COMMENTS column in Nested objects through logstash filters.

I read in a blog, where it talked about mutate filter option but really not sure how to use them in the logstash conf file.

Any pointers or examples on handling outer joins in logstash are greatly appreciated.

2

How do I parse the NUMBER_TYPE and COMMENTS column in Nested objects through logstash filters.

I read in a blog, where it talked about mutate filter option but really not sure how to use them in the logstash conf file.

Use a mutate filter and its rename option. There's an example in the filter's documentation. To created a nested field, use the [field][subfield] syntax described here: Accessing event data and fields | Logstash Reference [8.11] | Elastic

3

Thanks for the response.

I did try the rename option, however the hlog_comments fields is not getting displayed.

filter {
mutate {
rename => {"SEQUENCE_NUMBER" => "[HLOG_COMMENTS][SEQUENCE_NUMBER]" }
rename => {"USERID" => "[HLOG_COMMENTS][USERID]" }
rename => {"ACTIVITY" => "[HLOG_COMMENTS][ACTIVITY]" }
rename => {"HEADERLOG_STATE" => "[HLOG_COMMENTS][HLOG_STATE]" }
rename => {"HEADERLOG_STATUS" => "[HOG_COMMENTS][HLOG_STATUS]" }
......
-......
........

4

hlog_comment is the nested object.

5

Please show what you get from a stdout { codec => rubydebug } output so we can see exactly what your events look like.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.