Replacing @timestamp with timestamp of my log

Elastic Stack Logstash
1

I want to replace timestamp with the timestamp of my log.. In place of @timestamp, I want to replace with timestamp_match.

input {
beats {
port => "5044"
}
}
filter {
if [type] == "log" {
grok {
match => { "message" => "[%{TIMESTAMP_ISO8601:timestamp_match}] %{LOGLEVEL:log-level} [%{DATA:CONNECTION}]"}
}
mutate {
remove_field => [ "message" ]
}
}
}
output {
elasticsearch { hosts => ["localhost:9200"]
index => "changed"
}
}
In the index I have:

"hits": [
{
"_index": "changed",
"_type": "log",
"_id": "AVjddGVby6hU3wZv0i-y",
"_score": 1,
"_source": {
"offset": 15252,
"timestamp_match": "2016-10-1 8:38:58,928",
"input_type": "log",
"source": "C:\Users\Desktop\logc\lc.log",
"type": "log",
"tags": [
"beats_input_codec_plain_applied"
],
"CONNECTION": "Result Success ",
"@timestamp": "2016-12-08T07:55:32.645Z",
"log-level": "INFO",
"@version": "1",
"beat": {
"hostname": "GT5377",
"name": "GT5377",
"version": "5.0.0"
},
"host": "GT5377"
}
}
Can anyone help me with this?

Thanks in advance.

2

You need to use a date filter, that'll do it :slight_smile:

3

I tried using like this .. it doesnt work
date {
match => [ "@timestamp", "[%{TIMESTAMP_ISO8601}]"]
}

4

It's not the @timestamp field you want to parse, it's timestamp_match. This might work:

date {
  match => [ "timestamp_match", "ISO8601"]
}

Since your date components aren't zero-padded (2016-10-1 instead of 2016-10-01) it might be a little bit more complicated.

1 Like
5

yeah. getting _dateparsefailure ..

6

See the logs for details.

7

I changed the log for dateparsefailure.((2016-10-1 instead of 2016-10-01) Dateparsefailure is fixed.
I want to put this timestamp_match("timestamp_match": "2016-10-1 8:38:58,928",) value in place of @timestamp("@timestamp": "2016-12-08T07:55:32.645Z") ..
I tried with replace option. It doesnt work..
date {
match => ["timestamp_match", "ISO8601"]
target => "@timestamp"
}

8

Sorry, I don't understand what you mean. What replace option?

You probably have to specify multiple patterns that the date filter will try one by one:

date {
  match => [ "timestamp_match", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-d HH:mm:ss,SSS"]
}

(And similarly for the month numbers if they don't always have two digits.)

9

yeah that two digits for the date issue is fixed..

I just need to put the value of
"timestamp_match": "2016-10-1 8:38:58,928",
"@timestamp": "2016-12-08T07:55:32.645Z",

I am expecting something like :
"@timestamp":"2016-10-1 8:38:58,928"

10

Try the suggestion I gave. If it still doesn't work I need to see the error message from the log.

11

yeah that two digits for the date issue is fixed..

I just need to put the value of
"timestamp_match": "2016-10-1 8:38:58,928",
"@timestamp": "2016-12-08T07:55:32.645Z",

I am expecting something like :
"@timestamp":"2016-10-1 8:38:58,928"

12

Add a stdout { codec => rubydebug } output to your configuration and show the results.

13

{
"offset" => 22866,
"timestamp_match" => "2016-10-01 12:51:24,484",
"input_type" => "log",
"source" => "C:\Users\Desktop\logc\lc.log",
"type" => "log",
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"CONNECTION" => "Adding REST interface for Profile",
"@timestamp" => 2016-10-01T07:21:24.484Z,
"log-level" => "INFO",
"@version" => "1",
"beat" => {
"hostname" => "GT5377",
"name" => "GT5377",
"version" => "5.0.0"
},
"host" => "GT5377"
}

14

I want to put the value of timestamp_match - in the value of @timestamp.

here is my config :

input {
beats {
port => "5044"
}
}
filter {
if [type] == "log" {
grok {
match => { "message" => "[%{TIMESTAMP_ISO8601:timestamp_match}] %{LOGLEVEL:log-level} [%{DATA:CONNECTION}]"}
}
mutate {
remove_field => [ "message"]
}
date {
match => ["timestamp_match", "ISO8601"]
target => "@timestamp"
}
}
}
output {
elasticsearch { hosts => ["localhost:9200"]
index => "sswi"
}
stdout { codec => rubydebug }
}

15

date {
match => ["timestamp_match", "ISO8601"]
target => "@timestamp"
}

I mentioned target as @timestamp - but it doesnt replace the value of timestamp_match date value in the @timestamp.

16

I also tried to replace this way:
mutate {
replace => ["@timestamp", "%{timestamp_match}" ]
}
It doesnt work too :frowning:

17

The @timestamp field is UTC but timestamp_match is local time. If your timezone is UTC+5:30 then things are working fine.

18

We wanted to do the same thing, plus we use a slightly non-standard timestamp syntax.
Here's what works for us in our indexer:
grok { match = { "message" => "^(?\d{4}-\d{2}-\d{2}@\d{2}:\d{2}:\d{2}(.\d{1,4})?)OTHERSTUFF" } }
date { match => [ "logtimestamp", "yyyy-MM-dd@HH:mm:ss.SSS" ] }

19

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.