# Does Filebeat's timestamp processor need the source field to only contain the time?

**URL:** <https://discuss.elastic.co/t/does-filebeats-timestamp-processor-need-the-source-field-to-only-contain-the-time/309130>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [July 7, 2022, 4:28pm UTC](https://discuss.elastic.co/t/does-filebeats-timestamp-processor-need-the-source-field-to-only-contain-the-time/309130 "2022-07-07T16:28:31Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\
**Post date:** [July 7, 2022, 4:28pm UTC](https://discuss.elastic.co/t/does-filebeats-timestamp-processor-need-the-source-field-to-only-contain-the-time/309130/1 "2022-07-07T16:28:31Z")

</div>

What my subject says. I'm currently telling timestamp to use the message field as the source, and I'm not sure it's actually working. Do I need to parse out the time from the message before using the timestamp processor?

This is all via Elastic Agent 8.2.3.

---

<div class="post-metadata">

**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\
**Post date:** [July 14, 2022, 5:15pm UTC](https://discuss.elastic.co/t/does-filebeats-timestamp-processor-need-the-source-field-to-only-contain-the-time/309130/2 "2022-07-14T17:15:05Z")

</div>

Anyone?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 14, 2022, 5:36pm UTC](https://discuss.elastic.co/t/does-filebeats-timestamp-processor-need-the-source-field-to-only-contain-the-time/309130/3 "2022-07-14T17:36:15Z")

</div>

Yes The filebeat timestamp processor expects only the timestamp in the field so No, you can't just use the `message` field.

Also you have to pay special attention to the formats. It can be a bit confusing.

But yes, you need just the time data, not the whole message.

---

<div class="post-metadata">

**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\
**Post date:** [July 20, 2022, 6:03pm UTC](https://discuss.elastic.co/t/does-filebeats-timestamp-processor-need-the-source-field-to-only-contain-the-time/309130/4 "2022-07-20T18:03:12Z")

</div>

Ok, so, I've been experimenting with the dissect processor and I'm definitely missing something.

```nohighlight
  - dissect:
      tokenizer: '(\d{4}-\d{2}-\d{2}T\d{2}\:\d{2}\:\d{2}\.\d+-\d{2}\:\d{2})'
      field: "message"
      target_prefix: "temptime"

```

Is supposed to pull out the timestamp from a message that looks a bit like:

```auto
2022-07-19T08:37:17.936485-07:00
  log message here
  more info
  blah

```

That config causes filebeat to spit out errors:

```auto
2022-07-20T09:43:57-07:00 - message: Application: filebeat--8.2.3[185c4974-815d-42ed-b3df-388b6aa2d2b0]: State changed to FAILED: 1 error occurred:
	* 1 error: Error creating runner from config: invalid dissect tokenizer accessing 'processors.0.dissect.tokenizer'

 - type: 'ERROR' - sub_type: 'FAILED'

```

I also tried setting the `tokenizer` field to this:

```auto
%{YEAR}-%{MONTHNUM2}-%{MONTHDAY}T%{HOUR}:%{MINUTE}:%{SECOND}%{ISO8601_TIMEZONE}

```

But that gave me YEAR, MONTH, DAY, etc fields instead of putting the entire timestamp into the temptime field.

That leaves me wondering how the tokenizer is even supposed to work. Is it even a regex/grok pattern field? Or is it something completely different?

I've read the [docs](https://www.elastic.co/guide/en/beats/filebeat/current/dissect.html) multiple times. They mention the tokenizer field should container a " **dissection** pattern", but even after going over the examples a few times, I don't get what that pattern actually is. I thought it was a regex pattern, but my regex isn't working, so...

What am I missing?

Is there another processor that would do the job? Maybe something in the ingest pipeline?

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 20, 2022, 7:30pm UTC](https://discuss.elastic.co/t/does-filebeats-timestamp-processor-need-the-source-field-to-only-contain-the-time/309130/5 "2022-07-20T19:30:43Z")

</div>

So... yes your are definitely missing some things :)... which we can help with ... but first can I ask you something? are you sure you want to do this in the filebeat processor instead of an ingest pipeline which is a lot more flexible / easier to use in my opionin?

Also you have multi-line which always makes things a bit more complicated.

Are you open to using an ingest pipeline?

---

<div class="post-metadata">

**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\
**Post date:** [July 20, 2022, 8:07pm UTC](https://discuss.elastic.co/t/does-filebeats-timestamp-processor-need-the-source-field-to-only-contain-the-time/309130/6 "2022-07-20T20:07:13Z")

</div>

I'm fine with using ingest pipelines. 🙂 Thanks.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 20, 2022, 8:20pm UTC](https://discuss.elastic.co/t/does-filebeats-timestamp-processor-need-the-source-field-to-only-contain-the-time/309130/7 "2022-07-20T20:20:01Z")

</div>

Ok just so that we have a good thread.

Open an new thread something along the "Parse message and set timestamp with ingest Processor"

In please include the a full sample document as it appears in elasticsearch in json..

Also include what kind of filebeat input you are using / your filebeat config.

Then we can go from there.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 17, 2022, 10:20pm UTC](https://discuss.elastic.co/t/does-filebeats-timestamp-processor-need-the-source-field-to-only-contain-the-time/309130/8 "2022-08-17T22:20:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.