Filebeat - Ingest Node - Parsing Date

Elastic Stack Beats
1

My applications publishes logs in the following format:

[2018-06-17 21:39:14.779] [root] [info] The rest of message

I would like to extract the @timestamp from the log message. So I created the following pipeline in elasticsearch:

    PUT  _ingest/pipeline/test-pipeline 
    {
        "description": "log_parsing",
        "processors": [
          {
            "date": {
              "field": "message",
              "target_field": "@timestamp",
              "formats": [
                "yyyy-MM-dd hh:mm:ss.ms"
              ]
            }
          }
        ],
        "on_failure": [
        {
          "set": {
            "field": "error.message",
            "value": "{{ _ingest.on_failure_message }}"
          }
        }
        ]
      }

However, I get the "Invalid format error". Could you please let me know what is the wrong with the pipeline definition?

2

First, your pattern is wrong in a couple of places, it should be yyyy-MM-dd HH:mm:ss.SSS, that is, HH for the hour in 24-hour format, and SSS for the millisecond part.

However that is not enough to make it work because the date processor wants to parse the whole message field and it will still fail after the date:

Invalid format: \"[2018-06-17 21:39:14.779] [root] [info] The rest of message\" is malformed at \" [root] [info] The rest of message\"

I recommend you use a Grok Processor.

In the next version 6.4 you will also have the possibility to use a Dissect processor in filebeat to dissect the message into separate fields so you can parse the date individually.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.