Filebeat timestamp processsor doesn't parse microseconds part

Elastic Stack Beats
1

Hi,

this is the first part of my filebeat.xml

filebeat.inputs:
  - type: tcp
    enabled: true
    host: "localhost:9000"

processors:
  - decode_json_fields:
      fields: ["message"]
      process_array: true
      target: ""
      add_error_key: true
#      overwrite_keys: true
      max_depth: 2
  - timestamp:
      field: xxtimestamp
      layouts:
        - '2020-01-02T12:34:56.123456Z'
  - drop_fields:
      fields: [xxtimestamp]

(nothing fancy in the second part)

My application sends JSON messages to Filebeat through TCP. Filebeat parsing the messages add some Filebeat specific fields and forwards it to Elasticsearch/Kibana.

I have an xxtimestamp field in the JSON message which contains an ISO8601 timestamp, like '2020-01-02T12:34:56.123456Z'
I'd like to replace @timestamp's field content with xxtimestamp's content.

Its working with the config above but this method cuts off the microseconds part

e.g.: '2020-01-02T12:34:56.123456Z' -> '2020-01-02T12:34:56.123000Z

How could I increase this date precision?

2

I'm not convinced if it's not a bug. Filebeat should support up to nanoseconds precision for timestamps (for reference: https://github.com/elastic/beats/pull/9818).

Which filebeat version are you using?

3

Filebeat:
(apt show filebeat)

Package: filebeat
Version: 7.9.1
Priority: extra
Section: default
Maintainer: <@5834351aa1d6>
...
APT-Sources: https://artifacts.elastic.co/packages/oss-7.x/apt stable/main amd64 Packages

Kibana:
(apt show kibana-oss)

Package: kibana-oss
Version: 7.9.1
Priority: optional
Section: default
Maintainer: Kibana Team <info@elastic.co>
...
APT-Sources: https://artifacts.elastic.co/packages/oss-7.x/apt stable/main amd64 Packages

Elasticsearch:
(apt show elasticsearch-oss)

Package: elasticsearch-oss
Version: 7.9.1
Priority: optional
Section: web
Source: elasticsearch-oss
Maintainer: Elasticsearch Team <info@elastic.co>
...
Depends: bash (>= 4.1), lsb-base (>= 4), libc6, adduser, coreutils (>= 8.4)
...
APT-Sources: https://artifacts.elastic.co/packages/oss-7.x/apt stable/main amd64 Packages
...
4

Do you think the issue which you posted related to 7.9.1?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.