@timestamp not working in filesystem input

Elastic Stack Beats
1

Hello! I have same issue as described in Timestamp not being overwriten (ECS)

Solution is use deprecated input log, but what can we do with filesystem input?

I have such setup

- type: filestream
  id: some-log
  paths:
    - /path/to/logstash.log
  parsers:
    - ndjson:
      overwrite_keys: true
      message_key: "message"
      add_error_key: true

And such logs:

{"@timestamp":"2024-08-14T16:02:34.024439+00:00","@version":1,"host":"php","message":"Test message","type":"app","channel":"sms","level":"INFO","ctxt_foo":"bar","ctxt_bar":"baz"}
{"@timestamp":"2024-08-14T16:02:35.112766+00:00","@version":1,"host":"php","message":"Test message","type":"app","channel":"sms","level":"INFO","ctxt_foo":"bar","ctxt_bar":"baz"}
{"@timestamp":"2024-08-14T16:02:36.065564+00:00","@version":1,"host":"php","message":"Test message","type":"app","channel":"sms","level":"INFO","ctxt_foo":"bar","ctxt_bar":"baz"}

@timestamp value in elastic differ. If i use log input with json.overwrite_keys: true everything start working correctly, but i dont want to use deprecated component. What can i do?

2

Hi @wiistriker , welcome to our community.

Have you checked this migration guide? Migrate log input configurations to filestream | Filebeat Reference [8.15] | Elastic

You will probably need to create an ingest pipeline and then adjust the .yml configuration.

Example:

PUT _ingest/pipeline/update_timestamp_pipeline
{
  "description": "Pipeline to overwrite @timestamp with the value from logs",
  "processors": [
    {
      "date": {
        "field": "@timestamp",
        "target_field": "@timestamp",
        "formats": ["ISO8601"],
        "timezone": "UTC"
      }
    }
  ]
}

And, configure Filebeat to Use the Pipeline

filebeat.inputs:
  - type: filestream
    id: my-filestream-input
    paths:
      - /var/log/myapp/*.log
    parsers:
      - ndjson:
          target: ""
          overwrite_keys: true
          add_error_key: true
    pipeline: "update_timestamp_pipeline"
3

Hello @Alex_Salgado-Elastic. Thank you for the reply.

I didn't see that migration guide, because i start with filestream and dont have log input at my first try. so i have nothing to migrate from. But i am not really see where is information about ingest pipeline in this guide? How can this migration guilde can help me?

In my opinion is not intuitive to add pipelines for such simple use-case. Especially with arguments with the same name @timestamp. It looks little hacky as "put @timestamp field to @timestamp field".

Is there any reasons to have such behaviour? I expect that overwrite_keys: true will really override keys just like it described in documentation. Also in Step 3: Use new option names there is information that json option moved to parsers.n.ndjson so i expect overwrite_keys should work just like in log input