# Log fields are not parsing as expected using grok filter **URL:** \ **Category:** Logstash\ **Created:** [July 4, 2021, 8:17pm UTC](https://discuss.elastic.co/t/log-fields-are-not-parsing-as-expected-using-grok-filter/277746 "2021-07-04T20:17:02Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![prat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prat/32/79978_2.png) [@prat](https://discuss.elastic.co/u/prat)\ **Post date:** [July 4, 2021, 8:17pm UTC](https://discuss.elastic.co/t/log-fields-are-not-parsing-as-expected-using-grok-filter/277746/1 "2021-07-04T20:17:02Z")
Hi All, I am using below configuration and expecting it to get parse as per below fields. below is the `pipeline.conf` file. ```auto input { beats { port => "5044" } } filter { grok { match => { "message" => "%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T]%{HOUR}:%{MINUTE}(?::?%{SECOND})\| %{USERNAME:exchangeId}\| %{DATA:trackingId}\| %{NUMBER:RoundTrip:int}%{SPACE}ms\| %{NUMBER:ProxyRoundTrip:int}%{SPACE}ms\| %{NUMBER:UserInfoRoundTrip:int}%{SPACE}ms\| %{DATA:Resource}\| %{DATA:subject}\| %{DATA:authmech}\| %{DATA:scopes}\| %{IPV4:Client}\| %{WORD:method}\| %{DATA:Request_URI}\| %{INT:response_code}\| %{DATA:failedRuleType}\| %{DATA:failedRuleName}\| %{DATA:APP_Name}\| %{DATA:Resource_Name}\| %{DATA:Path_Prefix}"} } geoip { source => "Client" } } output { elasticsearch { hosts => ["localhost:9200"] } } ``` actual logs are like - ```auto 2021-06-25T08:51:38,788| ETxatokABfg2U2wVXx2ww| atid:1b9mgcaaCgpwrcgE1FLBAiF88mk| 270 ms| 212 ms| 0 ms| api.dev.only.bfco.io [] / /*:443| | OAuth| | 156.50.222.27| POST| /piie-aiip/v5/aiip/account-success-constant| 201| | | Ba API| Root Resource| /* 2021-06-25T13:02:41,254| 3rURHHJEh936dQEBMx-6yA| atid:x6UY50zGPx2L_qZmFm251FkQDiU| 160 ms| 8 ms| 0 ms| api.dev.only.bfco.io [] / /*:443| | OAuth| | 156.50.222.27| GET| /piie-aiip/v5/aiip/account-success-constant/97e7a7b9-3e60-4508-a35b-d0a01ba902bb| 200| | | Ba API| Root Resource| /* 2021-06-25T13:03:51,257| P0nH46kGVFnhZZ5iC6ZU1g| atid:y7UX49zBPy1P_wXnFm251FkQDiU| 39 ms| 2 ms| 0 ms| api.dev.only.bfco.io [] / /*:443| | OAuth| | 156.50.222.27| GET| /piie-aiip/v5/aiip/account-success-constant-asu| 400| | | Ba API| Root Resource| /* ``` but when checked on kibana, i am seeing message field as one field (below) and not separated into above individual fields as mentioned in `pipeline.conf`. below from Expanded document - Table. ```auto message 2021-06-25T13:01:03,478| XQvIx-qYtp2lP0tLcr53pQ| atid:y7UX99zGPx2L_qZmFm101EkQBiU| 180 ms| 10 ms| 0 ms| api.dev.only.bfco.io [] / /*:443| | OAuth| | 156.50.222.27| POST| /piie-aiip/v5/aiip/account-success- constant| 201| | | Ba API| Root Resource| /* ``` I wanted to run query on one of the field (NUMBER:RoundTrip:int) When checked above in `Grok Debugger` it is showing the expected results. i.e put `sample data` as above one of the log line, Used above grok Patten, it is giving below output. ```auto { "response_code": "201", "method": "POST", "subject": "", "Request_URI": "/piie-aiip/v5/aiip/account-success-constant", "Resource": "api.dev.only.bfco.io [] / /*:443", "UserInfoRoundTrip": 0, "APP_Name": "Ba API", "authmech": "OAuth", "Resource_Name": "Root Resource", "failedRuleName": "", "exchangeId": "XEyIx-yYtp2lP0kLcr08kP", "RoundTrip": 180, "ProxyRoundTrip": 10, "scopes": "", "Client": "156.50.222.27", "Path_Prefix": "", "failedRuleType": "", "trackingId": "atid:y8YX50zGPx3L_qZmg981FkEBiU" } ``` 1 Q. Can someone please advise what is wrong in `pipeline.conf` preventing message field to be individual fields. 2 Q. also not sure why below fields are showing as `long` in Index Patterns when it is passed as `int`. ![Screenshot 2021-07-05 at 1.37.39 AM](https://us1.discourse-cdn.com/elastic/original/3X/a/3/a3e1d55af5da7e3fe011fe47da14901f6b0031b4.png) Thanks in Advance. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 4, 2021, 8:57pm UTC](https://discuss.elastic.co/t/log-fields-are-not-parsing-as-expected-using-grok-filter/277746/2 "2021-07-04T20:57:42Z")
When I use that grok filter it does create all the fields. The [message] field will not be modified. grok creates the additional fields. ---
**Author:** ![prat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prat/32/79978_2.png) [@prat](https://discuss.elastic.co/u/prat)\ **Post date:** [July 4, 2021, 10:47pm UTC](https://discuss.elastic.co/t/log-fields-are-not-parsing-as-expected-using-grok-filter/277746/3 "2021-07-04T22:47:16Z")
Thank you for prompt response. I missed to see it. Any idea about the 2 Q. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 4, 2021, 10:54pm UTC](https://discuss.elastic.co/t/log-fields-are-not-parsing-as-expected-using-grok-filter/277746/4 "2021-07-04T22:54:15Z")
The [rules for dynamic mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html) turn an integer into a long. If you want a 32-bit integer rather than a 64-bit long you would have to set the mapping with an index template. ---
**Author:** ![prat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prat/32/79978_2.png) [@prat](https://discuss.elastic.co/u/prat)\ **Post date:** [July 4, 2021, 11:26pm UTC](https://discuss.elastic.co/t/log-fields-are-not-parsing-as-expected-using-grok-filter/277746/5 "2021-07-04T23:26:34Z")
Ok. Thanks. I thought what ever we are setting via above pipeline.conf is explicit setting for data types which will overwrite elasticsearch's dynamic mapping and hence asked why even after explicitly specifying data type as int, it turned into long. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 4, 2021, 11:35pm UTC](https://discuss.elastic.co/t/log-fields-are-not-parsing-as-expected-using-grok-filter/277746/6 "2021-07-04T23:35:22Z")
logstash has int and float. elasticsearch has long, integer, byte, float, double. So someone had to decide what to map an int to. Since a logstash int might not fit in an elasticsearch integer it makes more sense to map it to long. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 1, 2021, 11:35pm UTC](https://discuss.elastic.co/t/log-fields-are-not-parsing-as-expected-using-grok-filter/277746/7 "2021-08-01T23:35:48Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.