Problem with @timestamp time in indexes

Elastic Stack Beats
1

Hello, we have filebeat sending messages directly to elasticsearch index. And we found such problem, @timestamp in kibana view ( when we search) is different from timestamp we have in file on server( which use filebeat). Looks like elastic change timestamp - to time when document was indexed, not timestamp from original document( which could be found in file on server) How to fix that ? Because it's important for us to have correct messages order in time

2

Are you using a specific filebeat module?
How your logs are parsed?
Are you using an ingest pipeline?

3

I use filebeat 7.4.2, logs are text files with json ojects on each row. I do not use igest pipline

4

So elasticsearch just index the json content as you are sending it.
You need to define an ingest pipeline to rename the field which contains the event date to @timestamp.

5

So if we have such field named @timestamp in logs on server, will it be replaced? Beacause our event date fiels called @timestamp - default name ...

6

Can you show a document?

7

Here is docuent in index https://take.ms/Ybmfy

And here is document in log on server
{"@timestamp":"2020-03-06T20:18:23.894670031Z","exportable":true,"level":"info","message":"xxx","report_uuid":"bf0f8a56-4a46-4cf1-8706-491b3607eaf3","username":"_eva....."}

8

Could you run

GET INDEXNAME/_doc/ID

where

  • INDEXNAME is the filebeat index name
  • ID is the _id of the document

And share the output here.

Please don't post images of text as they are hard to read, may not display correctly for everyone, and are not searchable.

Instead, paste the text and format it with </> icon or pairs of triple backticks (```), and check the preview window to make sure it's properly formatted before posting it. This makes it more likely that your question will receive a useful answer.

9

{"_index":"xxx-2020.02.21-000001","_type":"_doc","_id":"GG5_sXABULG93gUD3XMn","_version":1,"_seq_no":846086,"_primary_term":1,"found":true,"_source":{"@timestamp":"2020-03-06T20:18:29.973Z","log":{"file":{"path":"xxx.log"},"offset":14310474576},"report_uuid":"bf0f8a56-4a46-4cf1-8706-491b3607eaf3","input":{"type":"log"},"ecs":{"version":"1.1.0"},"exportable":true,"level":"info","message":"xxx","host":{"name":"xxx"},"agent":{"hostname":"xxx","id":"448f24c9-d971-4558-8651-b57f65cc4de2","version":"7.4.2","type":"filebeat","ephemeral_id":"9bf7ff2d-3c39-4e04-8b8d-d2fe300d4f3e"},"username":"_eva"}}

10

I can't see any other timestamp in this document. So I don't understand what the problem is. Unless you removed important things in the example you shared?

11

Nothing removed. And problem is that time stamps in index and in file are different.
20:18:29 in index and 20:18:23 in file.

12

Can you share the full json document without removing anything?

13

I did not remove any fields, i just replace some text to xxx, is that important? ( there only index name, log filename, message, hostname and thats all )

14

Yes it is

15 
{"_index":"trendhero-2020.02.21-000001","_type":"_doc","_id":"GG5_sXABULG93gUD3XMn","_version":1,"_seq_no":846086,"_primary_term":1,"found":true,"_source":{"@timestamp":"2020-03-06T20:18:29.973Z","log":{"file":{"path":"/var/log/report-generator/report-generator.log"},"offset":14310474576},"report_uuid":"bf0f8a56-4a46-4cf1-8706-491b3607eaf3","input":{"type":"log"},"ecs":{"version":"1.1.0"},"exportable":true,"level":"info","message":"Get full report request","host":{"name":"1ps-api"},"agent":{"hostname":"1ps-api","id":"448f24c9-d971-4558-8651-b57f65cc4de2","version":"7.4.2","type":"filebeat","ephemeral_id":"9bf7ff2d-3c39-4e04-8b8d-d2fe300d4f3e"},"username":"_eva.vls_"}}
16

So which field elasticsearch is supposed to use as a timestamp of the event?

I don't see anything that has been collected by filebeat which looks like a timestamp.

17

Could you share the log line that has been collected by filebeat which corresponds to this Event in elasticsearch?

18 
{"@timestamp":"2020-03-06T20:18:23.894670031Z","exportable":true,"level":"info","message":"Get full report request","report_uuid":"bf0f8a56-4a46-4cf1-8706-491b3607eaf3","username":"_eva.vls_"}
19

What is the filebeat configuration?

20 
filebeat.inputs:

- type: log
  enabled: true
  json.keys_under_root: true
  paths:
    - /var/log/report-generator/report-generator.log

#================================ Outputs =====================================

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["elk.trendhero.io:9200"]
  index: "trendhero-%{+yyyy.MM.dd}"

  # Optional protocol and basic auth credentials.
  protocol: "https"
  username: "elastic"
  password: "REMOVED"

setup.template:
  name: 'trendhero'
  pattern: 'trendhero-*'
  enabled: false

setup.ilm.enabled: auto
setup.ilm.rollover_alias: "trendhero"
setup.ilm.pattern: "{now/d}-000001"

logging.level: info

processors:
 - drop_event:
       when:
           not:
               equals:
                   exportable: true