# Apply grok pattern based on the log file path **URL:** \ **Category:** Logstash\ **Created:** [May 26, 2023, 6:10am UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-the-log-file-path/334395 "2023-05-26T06:10:46Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [June 7, 2023, 12:10pm UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-the-log-file-path/334395/22 "2023-06-07T12:10:14Z")
Conversion to date format? Already wrote. > [@Rios](#): > > ```auto > date { > match => ["date", "dd/MMM/yyyy:HH:mm:ss.SSS Z"] > # timezone => "Asia/Dubai" > target=> "@timestamp" > } > > ``` ---
**Author:** ![Neelam\_Zanvar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/neelam_zanvar/32/103116_2.png) [@Neelam\_Zanvar](https://discuss.elastic.co/u/Neelam_Zanvar)\ **Post date:** [June 8, 2023, 4:27am UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-the-log-file-path/334395/23 "2023-06-08T04:27:11Z")
Yes, But how can i fit it in the existing configuration? ```auto input { beats { port => 5044 } } filter { if [3dxp_tag] == "3dxp_apache" { grok { match => { "message" => ["\[%{HTTPDATE:date:date}\] \| %{NUMBER:response} \| (?%{NUMBER} %{WORD}) \| (?%{NUMBER} %{WORD}|%{DATA}) \| %{IP:remoteip} \| (%{IP:clientip}|%{DATA:clientip}) \| (%{WORD:token}|%{DATA:token}) \| (?%{NUMBER} %{WORD}|%{DATA}) \| \"(?%{WORD}) (?%{URIPATHPARAM}) (?:HTTP/%{NUMBER:http_version})\""] } } } } output { elasticsearch { hosts => "http://IP:9200" index => "%{type}-%{+YYYY.MM.dd}" user => "elastic" password => "pwd" } } ``` I want to apply it on HTTPDATE:date field ---
**Author:** ![Neelam\_Zanvar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/neelam_zanvar/32/103116_2.png) [@Neelam\_Zanvar](https://discuss.elastic.co/u/Neelam_Zanvar)\ **Post date:** [June 8, 2023, 4:56am UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-the-log-file-path/334395/24 "2023-06-08T04:56:47Z")
Tried this ```auto if [3dxp_tag] == "3dxp_apache" { grok { match => { "message" => ["\[%{HTTPDATE:date:date}\] \| %{NUMBER:response} \| (?%{NUMBER} %{WORD}) \| (?%{NUMBER} %{WORD}|%{DATA}) \| %{IP:remoteip} \| (%{IP:clientip}|%{DATA:clientip}) \| (%{WORD:token}|%{DATA:token}) \| (?%{NUMBER} %{WORD}|%{DATA}) \| \"(?%{WORD}) (?%{URIPATHPARAM}) (?:HTTP/%{NUMBER:http_version})\""] } } mutate { gsub => [ "date", "CET", "+0100" ] } date { match => ["date", "dd/MMM/yyyy:HH:mm:ss.SSS Z"] # timezone => "Asia/Dubai" # target=> "@timestamp" } } ``` Commented target because i want it to be saved as date name only. Still it is creating it as text ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/1/01c015dcf1b2f4d6960938b6ea56e461dd808ad4.png) ---
**Author:** ![Neelam\_Zanvar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/neelam_zanvar/32/103116_2.png) [@Neelam\_Zanvar](https://discuss.elastic.co/u/Neelam_Zanvar)\ **Post date:** [June 8, 2023, 5:11am UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-the-log-file-path/334395/25 "2023-06-08T05:11:12Z")
the date is in this format 08/Jun/2023:10:40:08 +0530 ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [June 8, 2023, 6:21am UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-the-log-file-path/334395/26 "2023-06-08T06:21:45Z")
%{HTTPDATE:date:date}=\>`%{HTTPDATE:date}` - without the double date > Still it is creating it as text You have to delete your data mapping 3dxp\_apatchetrace and recreate it again. Also if no important data, existing index should be deleted, or do reindexing. ---
**Author:** ![Neelam\_Zanvar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/neelam_zanvar/32/103116_2.png) [@Neelam\_Zanvar](https://discuss.elastic.co/u/Neelam_Zanvar)\ **Post date:** [June 8, 2023, 6:29am UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-the-log-file-path/334395/27 "2023-06-08T06:29:54Z")
I have deleted the index and data view. and removed the extra date. still no luck ---
**Author:** ![Neelam\_Zanvar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/neelam_zanvar/32/103116_2.png) [@Neelam\_Zanvar](https://discuss.elastic.co/u/Neelam_Zanvar)\ **Post date:** [June 8, 2023, 6:43am UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-the-log-file-path/334395/28 "2023-06-08T06:43:49Z")
Here is the time format and i have used httpdate in grok to filter it into date 08/Jun/2023:10:40:08 +0530 Here is my current config ```auto filter { if [3dxp_tag] == "3dxp_apache" { grok { match => { "message" => ["\[%{HTTPDATE:date}\] \| %{NUMBER:response} \| (?%{NUMBER} %{WORD}) \| (?%{NUMBER} %{WORD}|%{DATA}) \| %{IP:remoteip} \| (%{IP:clientip}|%{DATA:clientip}) \| (%{WORD:token}|%{DATA:token}) \| (?%{NUMBER} %{WORD}|%{DATA}) \| \"(?%{WORD}) (?%{URIPATHPARAM}) (?:HTTP/%{NUMBER:http_version})\""] } } # mutate { # gsub => [ # "date", "CET", "+0100" # ] # } date { match => ["date", "dd/MMM/yyyy:HH:mm:ss.SSS Z"] # timezone => "Asia/Dubai" #target=> "@timestamp" } } ``` ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [June 8, 2023, 6:52am UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-the-log-file-path/334395/29 "2023-06-08T06:52:18Z")
The date plugin converts a field to @timestamp by default. Add "date" as target. ```auto date { match => ["date", "dd/MMM/yyyy:HH:mm:ss.SSS Z"] target=> "date" } ``` Repeat delete for index and mappings again. ---
**Author:** ![Neelam\_Zanvar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/neelam_zanvar/32/103116_2.png) [@Neelam\_Zanvar](https://discuss.elastic.co/u/Neelam_Zanvar)\ **Post date:** [June 8, 2023, 6:59am UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-the-log-file-path/334395/30 "2023-06-08T06:59:10Z")
Done, doesn't work still ---
**Author:** ![Neelam\_Zanvar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/neelam_zanvar/32/103116_2.png) [@Neelam\_Zanvar](https://discuss.elastic.co/u/Neelam_Zanvar)\ **Post date:** [June 8, 2023, 7:10am UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-the-log-file-path/334395/31 "2023-06-08T07:10:04Z")
i changed the following match =\> ["date", "dd/MMM/yyyy:HH:mm:ss.SSS Z"] to match =\> ["date", "dd/MMM/yyyy:HH:mm:ss Z"] removed the .SSS and now it is coming as date But the timezone is not coming in the date [08/Jun/2023:12:37:21 +0530] after grok date is 2023-06-08T07:07:21.000Z ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [June 8, 2023, 7:20am UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-the-log-file-path/334395/32 "2023-06-08T07:20:48Z")
Sorry for the date format, my mistake. 2023-06-08T07:07:21.000Z - this is normal, ES saves dates as UTC. Based on your Kibana settings, you will see correct time depend on your time zone. If you are located in London, you will see as 07:07:21, if your are in Berlin, Kibana shows as 08:07:21 or New Delphi will be 12:37:21 etc. ---
**Author:** ![Neelam\_Zanvar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/neelam_zanvar/32/103116_2.png) [@Neelam\_Zanvar](https://discuss.elastic.co/u/Neelam_Zanvar)\ **Post date:** [June 8, 2023, 7:22am UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-the-log-file-path/334395/33 "2023-06-08T07:22:55Z")
Ok. but how can i get the same timezone as in the log file in the date field? ---
**Author:** ![Neelam\_Zanvar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/neelam_zanvar/32/103116_2.png) [@Neelam\_Zanvar](https://discuss.elastic.co/u/Neelam_Zanvar)\ **Post date:** [June 8, 2023, 7:49am UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-the-log-file-path/334395/34 "2023-06-08T07:49:27Z")
after converting it to date there is only one field date and date.keyword is missing. When i am trying to use date in a visualisation it is not coming in the fields. it is showing date.keyword as emprty field ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/7/b7559182b21f4ee19e649c08ab4c7abe46ba6ba6.png) ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [June 8, 2023, 8:02am UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-the-log-file-path/334395/35 "2023-06-08T08:02:42Z")
If the date type is visible in Data mapping, do F5-the browser refresh in Discover ---
**Author:** ![Neelam\_Zanvar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/neelam_zanvar/32/103116_2.png) [@Neelam\_Zanvar](https://discuss.elastic.co/u/Neelam_Zanvar)\ **Post date:** [June 8, 2023, 10:07am UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-the-log-file-path/334395/36 "2023-06-08T10:07:05Z")
Thank you so much for the Solution! ---
**Author:** ![Neelam\_Zanvar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/neelam_zanvar/32/103116_2.png) [@Neelam\_Zanvar](https://discuss.elastic.co/u/Neelam_Zanvar)\ **Post date:** [June 8, 2023, 10:19am UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-the-log-file-path/334395/37 "2023-06-08T10:19:08Z")
![image](https://us1.discourse-cdn.com/elastic/original/3X/a/4/a4f3393558ba69e46bfb48054f2369b9333c1e98.png) as u can see in the image i am trying to add the date field in a table. i want to display the value as it is in the table yyyy/mm/dd:hh:mm:ss ? How can i do that? it was easily possible when it was a text field. ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [June 8, 2023, 1:45pm UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-the-log-file-path/334395/38 "2023-06-08T13:45:59Z")
> [@Neelam\_Zanvar](#): > > `index => "%{type}-%{+YYYY.MM.dd}"` Can you check where your data ends? From the sample above: ```auto "type": "3dxp_servicetrace", ``` ---
**Author:** ![Neelam\_Zanvar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/neelam_zanvar/32/103116_2.png) [@Neelam\_Zanvar](https://discuss.elastic.co/u/Neelam_Zanvar)\ **Post date:** [June 9, 2023, 4:24am UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-the-log-file-path/334395/39 "2023-06-09T04:24:06Z")
I didn't understad your question. I have three data view | **Data View** | **I** **ndex** | | --- | --- | | 3dxp\_servicetrace | 3dxp\_servicetrace | | 3dxp\_apachetrace | 3dxp\_apachetrace | | 3dxp\_passportservice | 3dxp\_passportservice | | 3dxp\_traceability | 3dxp\_servicetrace, 3dxp\_apachetrace | With timestamp for each index. I have used the above grok for all these indexes with minor changes. but the date field is there in all the indices. Now the type for all date fields is date. But when i want to visualise it in a dashboard i am not able to display the complete value of date ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [June 9, 2023, 10:38am UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-the-log-file-path/334395/40 "2023-06-09T10:38:06Z")
You have: - "type": "3dxp\_servicetrace" as data sample - index =\> "%{type}-%{+YYYY.MM.dd}" as logstash.conf - 3dxp\_traceability and 3dxp\_passportservice on the picture Your string must be converted to date on all 3 index. Also all 4 data view patterns should be created. "date": "30/May/2023:15:28:16.620 +0000" - string "date": 2023-05-30T10:29:42.191317300Z, - date, without quotes I don't know why 3dxp\_traceability have date per day and date per minute as separate. It's very hard to understand because your are jumping between problems, indices and data views. ---
**Author:** ![Neelam\_Zanvar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/neelam_zanvar/32/103116_2.png) [@Neelam\_Zanvar](https://discuss.elastic.co/u/Neelam_Zanvar)\ **Post date:** [June 12, 2023, 4:41am UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-the-log-file-path/334395/41 "2023-06-12T04:41:40Z")
Ok sorry for the misunderstanding. I'll try to clarify Currently let us consider only one data view that is 3dxp\_traceability which points to two different indexes that is 3dxp\_servicetrace and 3dxp\_apachetrace. so 3dxp\_servicetrace and 3dxp\_apachetrace are the data samples and two indexes. I've created 3dxp\_traceabilty as a data view in kibana. In my logstash config file I have created a date field of type date for both 3dxp\_servicetrace and 3dxp\_apachetrace . So when i try to create a dashboard with the data view 3dxp\_traceability if i add the a column for date field in the table then its either displaying only date or time . Is there any way i can display both in the same column. That is why i have date per day and date per minute are separate. with date as string format the column is visible like the following image ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/e/ee7fbfe36085d38864c8cd98b03cd3dc3ad52104.png) with date as type date its like this ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/4/a4f3393558ba69e46bfb48054f2369b9333c1e98.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/5/65efdde630d8e121b8aa8a9a217e22990a5e0377.png) [Previous page](https://discuss.elastic.co/t/apply-grok-pattern-based-on-the-log-file-path/334395.md?page=1) [Next page](https://discuss.elastic.co/t/apply-grok-pattern-based-on-the-log-file-path/334395.md?page=3)