How To use date Field in RAW data to used as @timestamp in Grok pattern

Elastic Stack Logstash
1

Hi all ,

i need to use the date field in my Raw Data to used as the @timestamp in my index:

RAW record data in the Filebeat -->
20190402-000037,8120791274.50,7174800715.40,4.18,/run 11.00% / 77.00% /data 55.00% /boot 1.00% /home 32.00%,,0,30.16,30.16,6.12,13.01
Date field in FileBeat = "20190402-000037"

Grok Pattern in logstash -->

filter {
grok {
match => { "message" => "%{DATA:DATE},%{NUMBER:used_memory:float},%{NUMBER:free_memory:float},%{NUMBER:CPU_load:float},%{DATA:run} %{NUMBER:run_usage:float}% %{DATA:
root} %{NUMBER:root_usage:float}% %{DATA:data} %{NUMBER:data_usage:float}% %{DATA:boot} %{NUMBER:boot_usage:float}% %{DATA:home} %{NUMBER:home_usage:float}%,,%{NUMBER:MO_SMS
_PER_SEC:float},%{NUMBER:MT_SMS_PER_SEC:float},%{NUMBER:Global_sms_per_second:float},%{NUMBER:OTA_PER_SECOND:float},%{NUMBER:Hand_set_change_per_second:float}" }
}

-so i need my field in index "DATE" to be used as the @timestamp in my index , how can i do that ?

thanks ,

2

Use a date filter

date { match => [ "DATE", "YYYYMMdd-HHmmss" ] }

You may need to add the timezone option to that.

3

thanks for reply , but i still have a problem.

-actually the DATE field still "text" note "date" as below -->
"DATE" : {
"type" : "text",
"norms" : false,
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
-while the only date field is timestamp field as below -->
"properties" : {
"@timestamp" : {
"type" : "date"
},
-grok pattern after adding the "date" filter as below -->

filter {
grok {
match => { "message" => "%{DATA:DATE},%{NUMBER:used_memory:float},%{NUMBER:free_memory:float},%{NUMBER:CPU_load:float},%{DATA:run} %{NUMBER:ru
n_usage:float}% %{DATA:root} %{NUMBER:root_usage:float}% %{DATA:data} %{NUMBER:data_usage:float}% %{DATA:boot} %{NUMBER:boot_usage:float}% %{DATA:hom
e} %{NUMBER:home_usage:float}%,,%{NUMBER:MO_SMS_PER_SEC:float},%{NUMBER:MT_SMS_PER_SEC:float},%{NUMBER:Global_sms_per_second:float},%{NUMBER:OTA_PER_SEC
OND:float},%{NUMBER:Hand_set_change_per_second:float}" }
}
date {
match => [ "DATE", "YYYYMMdd-HHmmss" ]
}

Note: when the RAW data record had yesterday date = "20190402-043536" and using the above grok , the index not created. however when i use the current data of today = "20190403-043536" , the index was created. and i don't know the reason.
-still need some suggestions to solve the issue for this field and use it as my "timestamp" in my index .

4

That's working as expected. If you want to overwrite DATE with the parsed value then set the target option on the date filter.

1 Like
5

do you mean to be like this -->

date {
match => [ "DATE", "YYYYMMdd-HHmmss" ]
target => "DATE"
}

?

6

Yes.

1 Like
7

perfect !! it is working now.

thanks for your support and help.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.