Problem overwriting field using json filter plugin

[tstrul (Tzachi)]

Hey,
I have a log that i want to parse as json. field is "log"
I want to decode json but use the same field as the orig message.
when i try:

      json {
        source => "log"
        target => "log"
      }

Im getting this error:

[2022-03-08T18:24:57,514][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"XXXX-2022.03.08", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x41fe54e5>], :response=>{"index"=>{"_index"=>"XXXX-2022.03.08", "_type"=>"_doc", "_id"=>"Uz3Gan8Bm8PSZeTG8hCn", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [log] of type [text] in document with id 'Uz3Gan8Bm8PSZeTG8hCn'.  "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:54"}}}}}

If i map target to a new field, everything works.
any idea why that happens?
the documentation mentions that:
" if the target field already exists, it will be overwritten!"

Thank you!

[Badger]

@tstrul

any idea why that happens?

Read this thread. You have indexed documents where the [log] field is a string, and now you are trying to index documents where [log] is an object in the same index. That does not work.

[tstrul (Tzachi)]

Thank you very much that was very helpful!
I understand the problem but im not sure i understand the solution though...

Currently, "log" field mostly contains json but from time to time it gets regular strings from legacy components.

index template defines this as a string.
I can change that but according to the thread you mentioned that will cause trouble as well.

I know that json filter can handle parsing errors and leave data untouched (_jsonparsefailure) but how should i define index template for this field ?

so is there another option besides changing the target field for json?

Thank you

[Badger]

@tstrul

Currently, "log" field mostly contains json but from time to time it gets regular strings from legacy components.

It cannot be an object on some events and a string on others. Pick one or the other and set it with the template (I think you would set the mapping to "type": "object", but that is an elasticsearch question and I do not run elasticsearch).

If you want to convert the strings to objects you would use

ruby {
     code => '
        host = event.get("log")
        if ! host.is_a? Hash
            event.set("[log][log]", event.remove("log"))
        end
    '
}