# Pipeline fails to match date field from JSON

**URL:** <https://discuss.elastic.co/t/pipeline-fails-to-match-date-field-from-json/191091>\
**Category:** Elasticsearch\
**Created:** [July 17, 2019, 9:18pm UTC](https://discuss.elastic.co/t/pipeline-fails-to-match-date-field-from-json/191091 "2019-07-17T21:18:24Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dimitris\_S](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dimitris_s/32/50172_2.png) [@Dimitris\_S](https://discuss.elastic.co/u/Dimitris_S)\
**Post date:** [July 17, 2019, 9:18pm UTC](https://discuss.elastic.co/t/pipeline-fails-to-match-date-field-from-json/191091/1 "2019-07-17T21:18:25Z")

</div>

Greetings. Some baseline info first. With a python script I'm sending simple messages formatted as JSON via **UDP**. The messages are of the form:

```
{"timestamp": "2019-07-17T23:10:17.940Z", "hw_stats": {"CPU_usage": 28.5, "RAM_used": 6561.5}}
{"timestamp": "2019-07-17T23:10:19.213Z", "hw_stats": {"CPU_usage": 28.5, "RAM_used": 6562.3}}
{"timestamp": "2019-07-17T23:10:20.346Z", "hw_stats": {"CPU_usage": 28.5, "RAM_used": 6562.7}}
{"timestamp": "2019-07-17T23:10:21.521Z", "hw_stats": {"CPU_usage": 28.5, "RAM_used": 6563.4}}

```

I tried to start from scratch with Filebeat, so for now I have the following configuration:

```
filebeat.inputs:   
- type: udp
  enabled: true
  max_message_size: 10KiB
  host: "localhost:8080"
  json.overwrites.keys: true
  processors:
  - decode_json_fields:
      fields: ['message']
  fields:
    idx_name: "test_idx"
  
  tags: ["test"]
    
output.elasticsearch:
  hosts: ["localhost:9200"]
  pipeline: "test"
  output.console:
    pretty: true

logging.level: debug
  
setup.kibana:
  host: "localhost:5601"

```

I use the _decode\_json\_fields_ to extract the nested measurements for CPU and RAM and I define a pipeline called "_test_" for use in ElasticSearch. My goal is to replace the default _@timestamp_ with the timestamps at the beginning of each JSON message. Back to ElasticSearch, I open Dev Tools and I enter the following code:

```
PUT /_ingest/pipeline/test
{
    "description": "Takes the eventTime field and turns it into a date field",
    "processors": [
        {
            "date": {
                "field": "timestamp",
                "target_field": "@timestamp",
                "formats": [
                    "YYYY-MM-DDTHH:mm:ss.SSSZ"
                ]
            }
        }
    ],
    "on_failure": [
        {
            "set": {
                "field": "_index",
                "value": "failed-{{_index}}"
            }
        },
        {
            "set": {
                "field": "error",
                "value": "{{_ingest.on_failure_message}}"
            }
        }
    ]
}

```

The formats should match the timestamp date format of the JSON but this always branches to the failed case and creates an index starting with "failed-....". Any suggestions?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 14, 2019, 9:18pm UTC](https://discuss.elastic.co/t/pipeline-fails-to-match-date-field-from-json/191091/2 "2019-08-14T21:18:27Z")

</div>

Closed on August 14, 2019, 9:18pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.