Pipeline fails to match date field from JSON

Greetings. Some baseline info first. With a python script I'm sending simple messages formatted as JSON via UDP. The messages are of the form:

{"timestamp": "2019-07-17T23:10:17.940Z", "hw_stats": {"CPU_usage": 28.5, "RAM_used": 6561.5}}
{"timestamp": "2019-07-17T23:10:19.213Z", "hw_stats": {"CPU_usage": 28.5, "RAM_used": 6562.3}}
{"timestamp": "2019-07-17T23:10:20.346Z", "hw_stats": {"CPU_usage": 28.5, "RAM_used": 6562.7}}
{"timestamp": "2019-07-17T23:10:21.521Z", "hw_stats": {"CPU_usage": 28.5, "RAM_used": 6563.4}}

I tried to start from scratch with Filebeat, so for now I have the following configuration:

filebeat.inputs:   
- type: udp
  enabled: true
  max_message_size: 10KiB
  host: "localhost:8080"
  json.overwrites.keys: true
  processors:
  - decode_json_fields:
      fields: ['message']
  fields:
    idx_name: "test_idx"
  
  tags: ["test"]
    
output.elasticsearch:
  hosts: ["localhost:9200"]
  pipeline: "test"
  output.console:
    pretty: true

logging.level: debug
  
setup.kibana:
  host: "localhost:5601"

I use the decode_json_fields to extract the nested measurements for CPU and RAM and I define a pipeline called "test" for use in ElasticSearch. My goal is to replace the default @timestamp with the timestamps at the beginning of each JSON message. Back to ElasticSearch, I open Dev Tools and I enter the following code:

PUT /_ingest/pipeline/test
{
    "description": "Takes the eventTime field and turns it into a date field",
    "processors": [
        {
            "date": {
                "field": "timestamp",
                "target_field": "@timestamp",
                "formats": [
                    "YYYY-MM-DDTHH:mm:ss.SSSZ"
                ]
            }
        }
    ],
    "on_failure": [
        {
            "set": {
                "field": "_index",
                "value": "failed-{{_index}}"
            }
        },
        {
            "set": {
                "field": "error",
                "value": "{{_ingest.on_failure_message}}"
            }
        }
    ]
}

The formats should match the timestamp date format of the JSON but this always branches to the failed case and creates an index starting with "failed-....". Any suggestions?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.