# Beats pipeline.json not received correctly in elastisearch/kibana

**URL:** <https://discuss.elastic.co/t/beats-pipeline-json-not-received-correctly-in-elastisearch-kibana/248624>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [September 15, 2020, 8:00am UTC](https://discuss.elastic.co/t/beats-pipeline-json-not-received-correctly-in-elastisearch-kibana/248624 "2020-09-15T08:00:57Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![schultz](https://avatars.discourse-cdn.com/v4/letter/s/c0e974/32.png) [@schultz](https://discuss.elastic.co/u/schultz)\
**Post date:** [September 15, 2020, 8:00am UTC](https://discuss.elastic.co/t/beats-pipeline-json-not-received-correctly-in-elastisearch-kibana/248624/1 "2020-09-15T08:00:57Z")

</div>

Versions:

- Filebeat v7.9.1
- Elastisearch v 7.9.1
- Kibana v7.9.1

On windows.

I have a pipeline.json file that describes the ingest part of my module

```auto
{
"description": "Pipeline for parsing aaa server logs.",
"processors": [
    {
        "grok": {
            "field": "message",
            "patterns": [
                "^%{NUMBER:aaa.server.processid}%{SPACE}%{TIMESTAMP_ISO8601:aaa.server.timestamp}%{SPACE}%{LOGLEVEL:aaa.server.severity}%{SPACE}:%{SPACE}(?:\\[%{IPORHOST:aaa.server.shortcategory}\\])?%{SPACE}(?:\\[%{DATA:aaa.server.threadid}\\])?%{GREEDYDATA:aaa.server.message}"
            ]
        },          
        "date": {
            "field": "aaa.server.timestamp",
            "formats": ["yyyy-MM-dd HH:mm:ss,SSS","ISO8601"],
            "on_failure": [{"append": {"field": "date.error.message", "value": "{{ _ingest.on_failure_message }}"}}]
        }
    }
],
"on_failure": [{
        "set": {
            "field" : "error.message",
            "value" : "{{ _ingest.on_failure_message }}"
        }
    }
]
}

```

When i start filebeat this is transferred to elastisearch/kibana and if i click the link filebeat-7.9.1-aaa-server-pipeline on "Ingest Node Pipelines" i see the correct json on the right handside of the screen. But when i press edit on the pipeline, the whole date part is gone. The data ingested is not handled correctly with regards to the date part. @timestamp is not updated as described in the json.

manually adding a date processor, as described in the json file fixes the parsing problem.

---

<div class="post-metadata">

**Author:** ![mtojek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mtojek/32/63863_2.png) [@mtojek](https://discuss.elastic.co/u/mtojek)\
**Post date:** [September 15, 2020, 11:26am UTC](https://discuss.elastic.co/t/beats-pipeline-json-not-received-correctly-in-elastisearch-kibana/248624/2 "2020-09-15T11:26:02Z")

</div>

Try to review a similar file: [https://github.com/elastic/beats/blob/master/x-pack/filebeat/module/cisco/shared/ingest/asa-ftd-pipeline.yml](https://github.com/elastic/beats/blob/master/x-pack/filebeat/module/cisco/shared/ingest/asa-ftd-pipeline.yml)

Most likely you have to do a similar transformation as for the `raw_date` field.

---

<div class="post-metadata">

**Author:** ![schultz](https://avatars.discourse-cdn.com/v4/letter/s/c0e974/32.png) [@schultz](https://discuss.elastic.co/u/schultz)\
**Post date:** [September 15, 2020, 12:45pm UTC](https://discuss.elastic.co/t/beats-pipeline-json-not-received-correctly-in-elastisearch-kibana/248624/3 "2020-09-15T12:45:14Z")

</div>

Is not that what this does? ref: [date-processor](https://www.elastic.co/guide/en/elasticsearch/reference/master/date-processor.html) it states that the default for target\_field is @timestamp.

```auto
    "date": {
       "field": "aaa.server.timestamp",
        "formats": ["yyyy-MM-dd HH:mm:ss,SSS","ISO8601"],
        "on_failure": [{"append": {"field": "date.error.message", "value": "{{ _ingest.on_failure_message }}"}}]
    }

```

This is how the pipeline looks, seems fine...

 ![ingest-node-pipeline](https://us1.discourse-cdn.com/elastic/original/3X/8/1/813c9f6770f3cad8ff945cb6d09b6de9f81726f4.jpeg)

But when i edit it, this is what i get:

 ![edit-pipeline](https://us1.discourse-cdn.com/elastic/original/3X/0/0/00be2a9ebf80a8c3fe6c897b77930951107d14f3.jpeg)

The date part is nowhere to be found.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 13, 2020, 2:45pm UTC](https://discuss.elastic.co/t/beats-pipeline-json-not-received-correctly-in-elastisearch-kibana/248624/4 "2020-10-13T14:45:23Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.