Replacing @timestamp with actual logtime

Elastic Stack Logstash
1

Hi everyone :slight_smile: This is my first post here - I'm a rookie in Elasticstack and would really appreciate your help since obviously what I'm doing isn't working.

I am trying to replace @timestamp with actual time from logs so it is more realistic to preview in Kibana.

Input line:

2019-05-10 15:06:52,667 [10] INFO SignController - SignDocumentRequest use preprocessing - True",
      " [10] INFO SignController - SignDocumentRequest use preprocessing - True"

Logstash filter:

input {
  beats {
   port => 5044
   type => "log"
  }
}
filter {

    grok {
    	match => {"message" => "%{TIMESTAMP_ISO8601:timestamp}%{GREEDYDATA:message}"}
    	}
	

	date {
        match => ["timestamp", "YYYY/MMM/dd:HH:mm:ss Z"]
        target => "@timestamp"
     }
}
output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{+dd.MM.YYYY}"
    document_type => "%{[@metadata][type]}"
  }
}

JSON:

 "_index": "filebeat-29.07.2019",
  "_type": "_doc",
  "_id": "ITlRPWwBt7PELLKOd0-p",
  "_version": 1,
  "_score": null,
  "_source": {
    "agent": {
      "hostname": "WINElastic",
      "ephemeral_id": "431f4e14-d858-42cb-8008-9358690f0c30",
      "type": "filebeat",
      "id": "fb8f3c03-8533-4f17-b7c2-30e5f1cb7707",
      "version": "7.1.1"
    },
    "log": {
      "file": {
        "path": "C:\\Filebeat Logs\\docs4ECMApi.log.2019-05-10"
      },
      "offset": 1523529
    },
    "input": {
      "type": "log"
    },
    "tags": [
      "beats_input_codec_plain_applied",
      "_dateparsefailure"
    ],
    "ecs": {
      "version": "1.0.0"
    },
    "@timestamp": "2019-07-29T10:40:43.839Z",
    "@version": "1",
    "type": "log",
    "host": {
      "hostname": "WINElastic",
      "architecture": "x86_64",
      "os": {
        "build": "14393.2068",
        "family": "windows",
        "platform": "windows",
        "version": "10.0",
        "name": "Windows Server 2016 Standard",
        "kernel": "10.0.14393.2068 (rs1_release.180209-1727)"
      },
      "id": "85b59899-cbd7-458e-bead-51834768c824",
      "name": "WINElastic"
    },
    "message": [
      "2019-05-10 15:06:52,667 [10] INFO SignController - SignDocumentRequest use preprocessing - True",
      " [10] INFO SignController - SignDocumentRequest use preprocessing - True"
    ],
    "timestamp": "2019-05-10 15:06:52,667"
  },
  "fields": {
    "@timestamp": [
      "2019-07-29T10:40:43.839Z"
    ]
  },
  "sort": [
    1564396843839
  ]
}

As you can seem I created "timestamp", but I am unable to put it's value under the "@timestamp" field.

Basically, I have two values:

@timestamp - shows import to Elasticstack date/time
timestamp - shows actual log date/time

I would like @timestamp to show actual log date/time.

What am I doing wrong?

Also, is this something I have to worry about?

tags": [
"beats_input_codec_plain_applied",
"_dateparsefailure"

Thank you in advance and best regards,

Mario

2

Your timestamp format is incorrect.
The format from the message is:

"timestamp": "2019-05-10 15:06:52,667"

And you are trying to say that it is:

        match => ["timestamp", "YYYY/MMM/dd:HH:mm:ss Z"]

Try with YYYY-MM-dd HH:mm:ss,SSS

3

Try

date { match => [ "timestamp", "YYYY-MM-dd HH:mm:ss,SSS" ] }

(lower case dd, not upper). The _dateparsefailure tag will then not get added. You can ignore the other tag.

4

Are you sure it shouldn't be "YYYY-MM-dd HH:mm:ss.SSS" (dot instead of comma between ss.SSS)?

When I put it with comma, all log entries that include timestamp are not displayed in Kibana. When I put dot, all log entries are displayed, but still with :

"beats_input_codec_plain_applied",
"_dateparsefailure"

Also, @timestamp wasn't replaced with timestamp

5

Yes, I am sure. When you use comma have you set the time picker in Kibana to include the first weeks of May, or just the last 15 minutes?

6

You are correct, I was expecting it to be on top of Discover page. Thanks a lot, it is working now!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.