Conversion to date format? Already wrote.
Yes, But how can i fit it in the existing configuration?
input {
beats {
port => 5044
}
}
filter {
if [3dxp_tag] == "3dxp_apache"
{
grok {
match => { "message" => ["\[%{HTTPDATE:date:date}\] \| %{NUMBER:response} \| (?<duration>%{NUMBER} %{WORD}) \| (?<bytes>%{NUMBER} %{WORD}|%{DATA}) \| %{IP:remoteip} \| (%{IP:clientip}|%{DATA:clientip}) \| (%{WORD:token}|%{DATA:token}) \| (?<tag1>%{NUMBER} %{WORD}|%{DATA}) \| \"(?<method>%{WORD}) (?<url>%{URIPATHPARAM}) (?:HTTP/%{NUMBER:http_version})\""] }
}
}
}
output {
elasticsearch {
hosts => "http://IP:9200"
index => "%{type}-%{+YYYY.MM.dd}"
user => "elastic"
password => "pwd"
}
}
I want to apply it on HTTPDATE:date field
Tried this
if [3dxp_tag] == "3dxp_apache"
{
grok {
match => { "message" => ["\[%{HTTPDATE:date:date}\] \| %{NUMBER:response} \| (?<duration>%{NUMBER} %{WORD}) \| (?<bytes>%{NUMBER} %{WORD}|%{DATA}) \| %{IP:remoteip} \| (%{IP:clientip}|%{DATA:clientip}) \| (%{WORD:token}|%{DATA:token}) \| (?<tag1>%{NUMBER} %{WORD}|%{DATA}) \| \"(?<method>%{WORD}) (?<url>%{URIPATHPARAM}) (?:HTTP/%{NUMBER:http_version})\""] }
}
mutate {
gsub => [
"date", "CET", "+0100"
]
}
date {
match => [ "date", "dd/MMM/yyyy:HH:mm:ss.SSS Z" ]
# timezone => "Asia/Dubai"
# target=> "@timestamp"
}
}
Commented target because i want it to be saved as date name only.
Still it is creating it as text
the date is in this format
08/Jun/2023:10:40:08 +0530
%{HTTPDATE:date:date}=>%{HTTPDATE:date} - without the double date
Still it is creating it as text
You have to delete your data mapping 3dxp_apatchetrace and recreate it again. Also if no important data, existing index should be deleted, or do reindexing.
I have deleted the index and data view. and removed the extra date. still no luck
Here is the time format and i have used httpdate in grok to filter it into date
08/Jun/2023:10:40:08 +0530
Here is my current config
filter {
if [3dxp_tag] == "3dxp_apache"
{
grok {
match => { "message" => ["\[%{HTTPDATE:date}\] \| %{NUMBER:response} \| (?<duration>%{NUMBER} %{WORD}) \| (?<bytes>%{NUMBER} %{WORD}|%{DATA}) \| %{IP:remoteip} \| (%{IP:clientip}|%{DATA:clientip}) \| (%{WORD:token}|%{DATA:token}) \| (?<tag1>%{NUMBER} %{WORD}|%{DATA}) \| \"(?<method>%{WORD}) (?<url>%{URIPATHPARAM}) (?:HTTP/%{NUMBER:http_version})\""] }
}
# mutate {
# gsub => [
# "date", "CET", "+0100"
# ]
# }
date {
match => [ "date", "dd/MMM/yyyy:HH:mm:ss.SSS Z" ]
# timezone => "Asia/Dubai"
#target=> "@timestamp"
}
}
The date plugin converts a field to @timestamp by default.
Add "date" as target.
date {
match => [ "date", "dd/MMM/yyyy:HH:mm:ss.SSS Z" ]
target=> "date"
}
Repeat delete for index and mappings again.
Done, doesn't work still
i changed the following
match => [ "date", "dd/MMM/yyyy:HH:mm:ss.SSS Z" ]
to
match => [ "date", "dd/MMM/yyyy:HH:mm:ss Z" ]
removed the .SSS
and now it is coming as date
But the timezone is not coming in the date
[08/Jun/2023:12:37:21 +0530]
after grok date is
2023-06-08T07:07:21.000Z
Sorry for the date format, my mistake.
2023-06-08T07:07:21.000Z - this is normal, ES saves dates as UTC. Based on your Kibana settings, you will see correct time depend on your time zone. If you are located in London, you will see as 07:07:21, if your are in Berlin, Kibana shows as 08:07:21 or New Delphi will be 12:37:21 etc.
Ok. but how can i get the same timezone as in the log file in the date field?
after converting it to date there is only one field date and date.keyword is missing.
When i am trying to use date in a visualisation it is not coming in the fields. it is showing date.keyword as emprty field
If the date type is visible in Data mapping, do F5-the browser refresh in Discover
Thank you so much for the Solution!
1 Like
as u can see in the image i am trying to add the date field in a table. i want to display the value as it is in the table yyyy/mm/dd:hh:mm:ss ? How can i do that? it was easily possible when it was a text field.
Can you check where your data ends? From the sample above:
"type": "3dxp_servicetrace",
I didn't understad your question.
I have three data view
| Data View | I ndex |
|---|---|
| 3dxp_servicetrace | 3dxp_servicetrace |
| 3dxp_apachetrace | 3dxp_apachetrace |
| 3dxp_passportservice | 3dxp_passportservice |
| 3dxp_traceability | 3dxp_servicetrace, 3dxp_apachetrace |
With timestamp for each index.
I have used the above grok for all these indexes with minor changes. but the date field is there in all the indices. Now the type for all date fields is date. But when i want to visualise it in a dashboard i am not able to display the complete value of date
You have:
- "type": "3dxp_servicetrace" as data sample
- index => "%{type}-%{+YYYY.MM.dd}" as logstash.conf
- 3dxp_traceability and 3dxp_passportservice on the picture
Your string must be converted to date on all 3 index. Also all 4 data view patterns should be created.
"date": "30/May/2023:15:28:16.620 +0000" - string
"date": 2023-05-30T10:29:42.191317300Z, - date, without quotes
I don't know why 3dxp_traceability have date per day and date per minute as separate. It's very hard to understand because your are jumping between problems, indices and data views.
Ok sorry for the misunderstanding. I'll try to clarify
Currently let us consider only one data view that is 3dxp_traceability which points to two different indexes that is 3dxp_servicetrace and 3dxp_apachetrace.
so 3dxp_servicetrace and 3dxp_apachetrace are the data samples and two indexes. I've created 3dxp_traceabilty as a data view in kibana.
In my logstash config file I have created a date field of type date for both 3dxp_servicetrace and
3dxp_apachetrace . So when i try to create a dashboard with the data view 3dxp_traceability
if i add the a column for date field in the table then its either displaying only date or time . Is there any way i can display both in the same column. That is why i have date per day and date per minute are separate. with date as string format the column is visible like the following image
with date as type date its like this
