Greetings. Some baseline info first. With a python script I'm sending simple messages formatted as JSON via UDP. The messages are of the form:
{"timestamp": "2019-07-17T23:10:17.940Z", "hw_stats": {"CPU_usage": 28.5, "RAM_used": 6561.5}}
{"timestamp": "2019-07-17T23:10:19.213Z", "hw_stats": {"CPU_usage": 28.5, "RAM_used": 6562.3}}
{"timestamp": "2019-07-17T23:10:20.346Z", "hw_stats": {"CPU_usage": 28.5, "RAM_used": 6562.7}}
{"timestamp": "2019-07-17T23:10:21.521Z", "hw_stats": {"CPU_usage": 28.5, "RAM_used": 6563.4}}
I tried to start from scratch with Filebeat, so for now I have the following configuration:
filebeat.inputs:
- type: udp
enabled: true
max_message_size: 10KiB
host: "localhost:8080"
json.overwrites.keys: true
processors:
- decode_json_fields:
fields: ['message']
fields:
idx_name: "test_idx"
tags: ["test"]
output.elasticsearch:
hosts: ["localhost:9200"]
pipeline: "test"
output.console:
pretty: true
logging.level: debug
setup.kibana:
host: "localhost:5601"
I use the decode_json_fields to extract the nested measurements for CPU and RAM and I define a pipeline called "test" for use in ElasticSearch. My goal is to replace the default @timestamp with the timestamps at the beginning of each JSON message. Back to ElasticSearch, I open Dev Tools and I enter the following code:
PUT /_ingest/pipeline/test
{
"description": "Takes the eventTime field and turns it into a date field",
"processors": [
{
"date": {
"field": "timestamp",
"target_field": "@timestamp",
"formats": [
"YYYY-MM-DDTHH:mm:ss.SSSZ"
]
}
}
],
"on_failure": [
{
"set": {
"field": "_index",
"value": "failed-{{_index}}"
}
},
{
"set": {
"field": "error",
"value": "{{_ingest.on_failure_message}}"
}
}
]
}
The formats should match the timestamp date format of the JSON but this always branches to the failed case and creates an index starting with "failed-....". Any suggestions?