Beats pipeline.json not received correctly in elastisearch/kibana

[schultz (Tom Schultz)]

Versions:

• Filebeat v7.9.1
• Elastisearch v 7.9.1
• Kibana v7.9.1

On windows.

I have a pipeline.json file that describes the ingest part of my module

{
"description": "Pipeline for parsing aaa server logs.",
"processors": [
    {
        "grok": {
            "field": "message",
            "patterns": [
                "^%{NUMBER:aaa.server.processid}%{SPACE}%{TIMESTAMP_ISO8601:aaa.server.timestamp}%{SPACE}%{LOGLEVEL:aaa.server.severity}%{SPACE}:%{SPACE}(?:\\[%{IPORHOST:aaa.server.shortcategory}\\])?%{SPACE}(?:\\[%{DATA:aaa.server.threadid}\\])?%{GREEDYDATA:aaa.server.message}"
            ]
        },          
        "date": {
            "field": "aaa.server.timestamp",
            "formats": ["yyyy-MM-dd HH:mm:ss,SSS","ISO8601"],
            "on_failure": [{"append": {"field": "date.error.message", "value": "{{ _ingest.on_failure_message }}"}}]
        }
    }
],
"on_failure": [{
        "set": {
            "field" : "error.message",
            "value" : "{{ _ingest.on_failure_message }}"
        }
    }
]
}

When i start filebeat this is transferred to elastisearch/kibana and if i click the link filebeat-7.9.1-aaa-server-pipeline on "Ingest Node Pipelines" i see the correct json on the right handside of the screen. But when i press edit on the pipeline, the whole date part is gone. The data ingested is not handled correctly with regards to the date part. @timestamp is not updated as described in the json.

manually adding a date processor, as described in the json file fixes the parsing problem.

[mtojek (Marcin Tojek)]

Try to review a similar file: https://github.com/elastic/beats/blob/master/x-pack/filebeat/module/cisco/shared/ingest/asa-ftd-pipeline.yml

Most likely you have to do a similar transformation as for the raw_date field.

[schultz (Tom Schultz)]

Is not that what this does? ref: date-processor it states that the default for target_field is @timestamp.

    "date": {
       "field": "aaa.server.timestamp",
        "formats": ["yyyy-MM-dd HH:mm:ss,SSS","ISO8601"],
        "on_failure": [{"append": {"field": "date.error.message", "value": "{{ _ingest.on_failure_message }}"}}]
    }

This is how the pipeline looks, seems fine...

But when i edit it, this is what i get:

The date part is nowhere to be found.