org.elasticsearch.index.mapper.MapperParsingException after a pipeline

Elastic Stack Elasticsearch
1

Hi,
this is my scenario: I receive messages by Fluentbit through an elastic ingestion pipeline.
This is my ingestion pipeline:

{
    "jsonize_account_backend":{
        "processors":[
            {
                "grok":{
                    "field":"log",
                    "patterns":[
                        "%{TIMESTAMP_ISO8601:log_timestamp}%{SPACE}%{LOGLEVEL:log_level}%{SPACE}%{GREEDYDATA:log}\\n"
                    ],
                    "ignore_failure":true
                }
            },
            {
                "json":{
                    "field":"log",
                    "target_field":"json_log",
                    "on_failure":[
                        {
                            "set":{
                                "field":"json_log_text",
                                "value":{
                                    "error":"Log is not a valid json"
                                }
                            }
                        }
                    ]
                }
            }
        ]
    }
}

So, I expect that, at the second processor, if the "log" field is a valid one it parse the data and set the json data in "json_log" (flattened field in mapping), otherwise it set the error message into "json_log_text".

But, in some cases, I found this error in the elastic logs:

{"type": "server", "timestamp": "2020-11-16T09:08:35,039Z", "level": "DEBUG", "component": "o.e.a.b.TransportShardBulkAction", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "message": "[log-account-2020.11.12][0] failed to execute bulk item (index) index {[log-account-2020.11.12][_doc][MLZQ0HUBM1HcsZbaJtRJ], source[n/a, actual length: [18.6kb], max length: 2kb]}", "cluster.uuid": "D84-LczLRFWqtU30fjfeCg", "node.id": "g6I5hbZ_Q8-EakdU5NbBMw" ,
"stacktrace": ["org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [json_log] of type [flattened] in document with id 'MLZQ0HUBM1HcsZbaJtRJ'. Preview of field's value: ':null,'",

Why this error is thrown?
I expect that the json_log is empty if the parsing didn't work with success.

2

You can see that it's reporting the value of json_log as :null,, which looks a little weird to me. Can you see the output that the pipeline is creating?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.