org.elasticsearch.index.mapper.MapperParsingException after a pipeline

Hi,
this is my scenario: I receive messages by Fluentbit through an elastic ingestion pipeline.
This is my ingestion pipeline:

{
    "jsonize_account_backend":{
        "processors":[
            {
                "grok":{
                    "field":"log",
                    "patterns":[
                        "%{TIMESTAMP_ISO8601:log_timestamp}%{SPACE}%{LOGLEVEL:log_level}%{SPACE}%{GREEDYDATA:log}\\n"
                    ],
                    "ignore_failure":true
                }
            },
            {
                "json":{
                    "field":"log",
                    "target_field":"json_log",
                    "on_failure":[
                        {
                            "set":{
                                "field":"json_log_text",
                                "value":{
                                    "error":"Log is not a valid json"
                                }
                            }
                        }
                    ]
                }
            }
        ]
    }
}

So, I expect that, at the second processor, if the "log" field is a valid one it parse the data and set the json data in "json_log" (flattened field in mapping), otherwise it set the error message into "json_log_text".

But, in some cases, I found this error in the elastic logs:

{"type": "server", "timestamp": "2020-11-16T09:08:35,039Z", "level": "DEBUG", "component": "o.e.a.b.TransportShardBulkAction", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "message": "[log-account-2020.11.12][0] failed to execute bulk item (index) index {[log-account-2020.11.12][_doc][MLZQ0HUBM1HcsZbaJtRJ], source[n/a, actual length: [18.6kb], max length: 2kb]}", "cluster.uuid": "D84-LczLRFWqtU30fjfeCg", "node.id": "g6I5hbZ_Q8-EakdU5NbBMw" ,
"stacktrace": ["org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [json_log] of type [flattened] in document with id 'MLZQ0HUBM1HcsZbaJtRJ'. Preview of field's value: ':null,'",

Why this error is thrown?
I expect that the json_log is empty if the parsing didn't work with success.

You can see that it's reporting the value of json_log as :null,, which looks a little weird to me. Can you see the output that the pipeline is creating?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.