Get timestamp from log lines

Elastic Stack Beats
1

I'm using Filebeat to read logs in two different formats. Each portion of information is prefixed with timestamp in either "YYYYMMDD-hhmmss" or "MM.DD.YY hh:mm:ss" format. I use 2 different prospectors to merge lines for those 2 formats.

Sometimes the files are locked and Filebeat reads them only hours later. Is there a way to use these timestamps instead of the current time when data is added to Elasticsearch?

2

@tkzv Yes, we always recommend to correctly parse the date from the original string so the event is accurate. There are a few options to do that.

  1. Check if your log format (or application) that you are parsing is one of our supported modules, module is preconfigured prospectors that also include logic to parse the date correctly.

  2. If you are sending your events directly to Elasticsearch, you can use the ingest pipeline to configure some logic on the ES side to parse the event correctly. You may want to look at these processors: grok and date, You will also need to configure the filebeat output to send to the correct pipeline.

  3. If you send your events through Logstash, you can also use the grok filter and the date filter

3

Check if your log format (or application) that you are parsing is one of our supported modules,

It isn't.

If you are sending your events directly to Elasticsearch,
If you send your events through Logstash,

I'm not using Logstash. Does it have any advantages for this task?

You may want to look at these processors: grok and date, You will also need to configure the filebeat output to send to the correct pipeline.

I tried date earlier, when I was using only one prospector for "YYYYMMDD-hhmmss" format. For some reason it didn't work, and I reverted to defaults. What changes do I need to make to the example here Date processor | Elasticsearch Guide [8.11] | Elastic ? Would it be enough to change the "formats" line to "formats" : ["YYYYMMDD-hhmmss"], ?

P.S. Note to self: Joda format specification says it should be "yyyyMMdd-HHmmss".

4

@tkzv For what you are trying to achieve, the ingest pipeline would be enough. Logstash offers more inputs/processor than the ingest pipeline.

The patterns are using the Joda syntax, I believe in your case you want yyyyMMdd-hhmmss, the date processor accepts a list of patterns, it will return the value of the first one that correctly parses the date.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.