Filebeat now processor seams to be buggy

Toony · August 19, 2026, 7:38am

Hi,

It seems the now processor behaves strangely

If I set a now processor in filebeat at the top yaml level like this:

processors:
  - now:
      field: metadata.timeline.t1

and configure Logstash like this:

input {
  beats {
    port => 5044
    add_field => { 
      "[metadata][timeline][t1]" => "%{[@timestamp]}"
      "[metadata][timeline][t2]" => "%{+yyyy-MM-dd'T'HH:mm:ss.SSS'Z'}" 
    }
  }
}

I sometimes get correct timestamping like this one:

          "metadata": {
            "timeline": {
              "t1": [
                "2026-08-18T16:58:59.561Z",
                "2026-08-18T16:58:59.561Z"
              ],
              "t2": "2026-08-18T16:58:59.561Z"
            }

where t2 is equal or after t1 (given by the @timestamp field comming from filebeat and by the filebeat "now" processor).

But I also have some messages like this:

          "metadata": {
            "timeline": {
              "t1": [
                "2026-08-18T16:59:04.568Z",
                "2026-08-18T16:58:59.565Z"
              ],
              "t2": "2026-08-18T16:58:59.565Z"
            }
          }

where the t1 = "2026-08-18T16:59:04.568Z" is given by the now processor and is"after" the filebeat given event @timestamp and after t2 wich is computed later, on the logsatsh side... I have checked that every time i get this behavior there is 5 seconds (plus 2 or 3 milliseconds ) in more than the @timestamp given for the event by filebeat.

Is the now processor a real "now" or just a cached timestamp which is computed roughly every 5 seconds ?

Thanks for your help,

Best regards

PS: My filebeat is in 9.4.0 version

TiagoQueiroz · August 19, 2026, 2:34pm

Hi @Toony, this might be related to the input you're using or input configuration. One example that could cause this 5s delay is the multiline timeout, if the multiline is waiting for a match and times out, the @timestamp will be about the configured timeout earlier than the now processor that runs on the final event.

Could you share your whole config (remember to redact any sensitive information)?

leandrojmp · August 19, 2026, 2:42pm

How is this being tested? Both Filebeat and Logstash are on the same machine?

Are you testing with a single event or with multiple events at the same time?

Filebeat would send a batch of events, and in this batch of events you may have events with a different metadata.timeline.t1 generated on Filebeat side.

I'm not sure if the @timestamp in the add_field on the input is from Filebeat, the message is not being parsed to extract the field, I think it may come from the @timestamp generated by Logstash, which makes sense in the second example because it matches the other field added by logstash.

Toony · August 19, 2026, 3:47pm

Hi Tiago,

Thanks for this first answer.

I do not think it is a timeout because, the t2 parameter is built with the logstash date format taking its value at the moment it reaches the logstash input... But t1 is sometime after that computed moment for some logs...

Toony · August 19, 2026, 4:15pm

Hi Leandro,

Thanks for your help too !

All my tests are done through a docker-compose list of services:

A filebeat listening to access.log and an application.log files
A logstash
An Elasticsearch cluster

I have a test log generator wich emits 3 logs on each message I send. 1 access log and two application logs.

When I send one message from my generator, three messages are produced roughly at the same time. Two of them written in application.log and one of them in the access.log file.

You can see in my post that working message is roughly treated by logstash at the same time as the non working one (few milliseconds between them)

in my tests, i found that @timestamp and t2 are always the same... So coming from logstash or filebeat is not the pain point here.

This is the now pocessor which fails.

:PS for you and Tiago, i have changed the now processor target field name to tnow to show you it is the real problem:

filebeat.modules:
- module: auditd
  log:
    enabled: true


filebeat.inputs:

#------------------------------ Log input --------------------------------
- type: filestream
  id: acc-id
  period: 1s
  enabled: true
  paths:
    - /var/log/collection-point/access*.log
  fields:
    log:
      type: access
  fields_under_root: true
  
- type: filestream
  id: app-id
  period: 1s
  enabled: true
  paths:
    - /var/log/collection-point/application*.log
  fields:
    log:
      type: application
  fields_under_root: true
  parsers:
    - multiline:
        type: pattern
        pattern: '^[0-9]{4}'
        negate: true
        match: after
processors:
  - now:
      field: metadata.timeline.tnow
  
output.logstash:
  hosts: ["stack-logstash:5044"]

The logstash conf is not really important as all is done in the beats input. I have not changed it since this morning...

With the new configuration in filebeat I have this strange timestamping:

"_source": {
          "metadata": {
            "timeline": {
              "tnow": "2026-08-19T16:05:41.551Z",
              "t1": "2026-08-19T16:05:36.550Z",
              "t2": "2026-08-19T16:05:36.550Z"
            }
          }, 
          "log": {
            "file": {
              "inode": "26879015",
              "path": "/var/log/collection-point/application.log",
              "device_id": "64513",
              "fingerprint": "aea217e63a46bac55e8560246aa23b00fc24347bc74726da2b116e4b22bd932b"
            },
            "offset": 24601,
            "type": "application"
          },[...]
        }

and this working message:

"_source": {
          "metadata": {
            "timeline": {
              "tnow": "2026-08-19T16:05:36.550Z",
              "t1": "2026-08-19T16:05:36.550Z",
              "t2": "2026-08-19T16:05:36.550Z"
            }
          },
          "log": {
            "file": {
              "inode": "26879015",
              "path": "/var/log/collection-point/application.log",
              "device_id": "64513",
              "fingerprint": "aea217e63a46bac55e8560246aa23b00fc24347bc74726da2b116e4b22bd932b"
            },
            "offset": 24506,
            "type": "application"
          },
          [...]
        }

Both are application logs so using multiline parser ...

Thanks again, I still searching for a solution

leandrojmp · August 19, 2026, 4:34pm

I'm not sure exactly what you are trying to achieve and what is the issue here, what could be the bug.

You are comparing time fields from Filebeat and Logstash, so the logstash configuration cannot be ignored.

Also, as far as I know the now processor is per event in Filebeat side and you have a multipline parser, this has a default 5s as the parser has to wait for a pattern to match, the now processor will run after that if I'm not wrong.

This is also not something that you can test with just a couple of messages, you would need to produce hundreds of messages to get close of how this would work on real usage.

I would suggest that you remove logstash entirely on theses tests and use the console ouput, if the time from logstash should be ignored, than there is no reason to add them to the event is this can be confusing in the troubleshoot.

For the multiline, you have this in the documentation:

multiline.timeout
After the specified timeout, Filebeat sends the multiline event even if no new pattern is found to start a new event. The default is 5s.

The last event will probably always reach this timeout as there is no further pattern to match before sending it.

Topic		Replies	Views
FileBeat Processing @timestamp instead of log @timestamp Beats filebeat	2	2708	January 29, 2020
Filebeat process timestamp from JSON Beats filebeat	1	2571	May 19, 2020
Filebeat uses process time instead event time Beats filebeat	1	546	May 21, 2021
Timestamps precision in beats Beats filebeat	10	1506	November 22, 2018
Filebeat timestamp processor no year Beats filebeat	0	361	March 28, 2022

Filebeat now processor seams to be buggy

Related topics