How to construct JSON using filter{} and input data?

[AshwinMS (Ash)]

Hi , im new to logstash and found it to be super useful to loadup data from different sources to an es cluster ! But ive run into one problem which ive been stuck with for a while now and would really love some help.

Im trying to construct and store a json object in using the data from input and transformed in filter before sending them to my es output. I was following along this tutorial How to keep Elasticsearch synced with a RDBMS using Logstash and JDBC | Elastic Blog .

Heres the gist of the transformation i want to perform. I would like to have a json object constructed using the input.
Assuming the above mentioned tutorial is followed, and we have some more columns in rdbms table such as DoB ,data i want to load to es is as follows:

{
    "id": "123",
    "name": "John",
    "details": {
        "DoB": "01/01/2000",
    }
}

as i have defined my index mapping as below:

{
    "mappings": {
        "properties": {
            "id": {
                "type": "keyword"
            },
            "name": {
                "type": "search_as_you_type"
            },
            "details": {
                "properties": {
                    "DoB": {
                        "type": "date",
                    }
                }
            }
        }
    }
}

But when i tried to run my own logstash pipeline by adding this filter:

mutate {
        replace => {
            "DoB" => 
                { "DoB" => "%{DoB}" }
            
        }
    }

, i keep running into the following error:

"error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [details] tried to parse field [details] as object, but found a concrete value"}}}}

Is this not a desirable way to do index data on Elasticsearch ? Even so , how would one populate object type mapping on elastic using logstash? Would be super helpful if anyone could help me understand how to go about this.

[Tomo_M (Tomohiro Mitani)]

The error looks about [details] field of output and nothing to do with the DoB replace pipline. Maybe you need to explain more about your pipeline.

[AshwinMS (Ash)]

yes , i agree. Just having a root level DoB field works just fine. The main problem i encounter is when i try to construct a object type using the input fields. I'm unable to achieve this.

[aaron-nimocks]

When I see this issue it usually means 1 thing. You are trying to send data on a root level field that your mapping is not setup to receive.

So in your pipeline I think it's creating data that looks like this.

{
  "details": "some value", <--- this is the issue
  "details" : {
    "DoB": "some value"
  }
}

The error is saying you can't have a key/value for the field details. If you change your output to stdout and post a sample of that data and complete conf here it would help troubleshooting.