Does Filebeat's timestamp processor need the source field to only contain the time?

Elastic Stack Beats
1

What my subject says. I'm currently telling timestamp to use the message field as the source, and I'm not sure it's actually working. Do I need to parse out the time from the message before using the timestamp processor?

This is all via Elastic Agent 8.2.3.

2

Anyone?

3

Yes The filebeat timestamp processor expects only the timestamp in the field so No, you can't just use the message field.

Also you have to pay special attention to the formats. It can be a bit confusing.

But yes, you need just the time data, not the whole message.

1 Like
4

Ok, so, I've been experimenting with the dissect processor and I'm definitely missing something.

  - dissect:
      tokenizer: '(\d{4}-\d{2}-\d{2}T\d{2}\:\d{2}\:\d{2}\.\d+-\d{2}\:\d{2})'
      field: "message"
      target_prefix: "temptime"

Is supposed to pull out the timestamp from a message that looks a bit like:

2022-07-19T08:37:17.936485-07:00
  log message here
  more info
  blah

That config causes filebeat to spit out errors:

2022-07-20T09:43:57-07:00 - message: Application: filebeat--8.2.3[185c4974-815d-42ed-b3df-388b6aa2d2b0]: State changed to FAILED: 1 error occurred:
	* 1 error: Error creating runner from config: invalid dissect tokenizer accessing 'processors.0.dissect.tokenizer'

 - type: 'ERROR' - sub_type: 'FAILED'

I also tried setting the tokenizer field to this:

%{YEAR}-%{MONTHNUM2}-%{MONTHDAY}T%{HOUR}:%{MINUTE}:%{SECOND}%{ISO8601_TIMEZONE}

But that gave me YEAR, MONTH, DAY, etc fields instead of putting the entire timestamp into the temptime field.

That leaves me wondering how the tokenizer is even supposed to work. Is it even a regex/grok pattern field? Or is it something completely different?

I've read the docs multiple times. They mention the tokenizer field should container a " dissection pattern", but even after going over the examples a few times, I don't get what that pattern actually is. I thought it was a regex pattern, but my regex isn't working, so...

What am I missing?

Is there another processor that would do the job? Maybe something in the ingest pipeline?

Thanks in advance!

5

So... yes your are definitely missing some things :)... which we can help with ... but first can I ask you something? are you sure you want to do this in the filebeat processor instead of an ingest pipeline which is a lot more flexible / easier to use in my opionin?

Also you have multi-line which always makes things a bit more complicated.

Are you open to using an ingest pipeline?

6

I'm fine with using ingest pipelines. :slight_smile: Thanks.

7

Ok just so that we have a good thread.

Open an new thread something along the "Parse message and set timestamp with ingest Processor"

In please include the a full sample document as it appears in elasticsearch in json..

Also include what kind of filebeat input you are using / your filebeat config.

Then we can go from there.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.