Timestamp processor - parsing milliseconds with comma

[dcamozzato (Daniel Camozzato)]

Hi,

I am trying to use Filebeat 7.5.1 to parse a timestamp with the following format: '2020-01-17 06:37:17,849'

My layout is this: '2006-01-02 15:04:05,999'

Unfortunately, the above does not work. According to Golang docs, I need to define milliseconds with a period (e.g., ".999"). Indeed, if I change both the sample and the layout to use a period, it works. I am linking here the Go Playground for this.

Is there a way to get the timestamp processor to do what I need it to do? If not, is there some other way to get around this? For example, I couldn't find a way to concatenate fields, otherwise I could possibly use a dissect processor to split the comma out, and then join the fields with a period in the middle.

Any help is appreciated.

[andrewkroh]

The processor is pretty strict about the format of the fractional seconds. I've seen a few workarounds. What you mention with dissect would work. You could using the script processor in Beats to replace the comma with a period before running the timestamp processor.
You could defer the timestamp parsing to LS or ES Ingest Node, both have a more flexible date processor.

- dissect:
    tokenizer: '%{ts} %{+ts} %{message}'
- script:
    lang: javascript
    source: >
      function process(evt) {
        var ts = evt.Get('dissect.ts').replace(',', '.');
        evt.Put('dissect.ts', ts);
      }
- timestamp:
    field: dissect.ts
    layouts:
      - 2006-01-02 15:04:05.999

which gets you

  {
    "@timestamp": "2020-01-02T16:13:14.123Z",
    "dissect": {
      "message": "my log",
      "ts": "2020-01-02 16:13:14.123"
    },
    "message": "2020-01-02 16:13:14,123 my log"
  }

[dcamozzato (Daniel Camozzato)]

Andrew, that worked perfectly. Thank you so much for your assistance. I am not sure how I didn't see that "script" processor when I was looking for a solution, I will definitely keep it in mind from now on!