# Logstash applying grok to only specific log lines **URL:** \ **Category:** Logstash\ **Created:** [May 17, 2019, 7:40pm UTC](https://discuss.elastic.co/t/logstash-applying-grok-to-only-specific-log-lines/181645 "2019-05-17T19:40:45Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![chandukreddi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chandukreddi/32/29241_2.png) [@chandukreddi](https://discuss.elastic.co/u/chandukreddi)\ **Post date:** [May 17, 2019, 7:40pm UTC](https://discuss.elastic.co/t/logstash-applying-grok-to-only-specific-log-lines/181645/1 "2019-05-17T19:40:46Z")
Hi, why logstash applying grok for only only few log lines even though I have grok patterns for all log lines and I can run through same logs using grok debugger and its working fine, I think its behaving like unpredictable way or am I missing anyth > **Summary** > > This text will be hidden ing? this is my logstash debug log: ``` [2019-05-17T11:54:12,123][DEBUG][logstash.pipeline] filter received {"event"=>{"@timestamp"=>2019-05-17T18:54:12.013Z, "@version"=>"1", "tags"=>["swift_paco_logs"], "message"=>"May 14 03:36:15 ``` > **Summary** > > [spoiler]dllabsw1 account-server: 172.24.7.109 [/spoiler] - - [14/May/2019:03:36:15 +0000] "PUT /d3/26974/.expiring\_objects/1557791933" 201 - "-" "-" "container-updater 13974" 0.0007 "-" 35828 0", "host"=\>"0.0.0.0", "path"=\>"/opt/swiftlogs/test.log"}} [2019-05-17T11:54:12,124][DEBUG][logstash.pipeline] filter received {"event"=\>{"@timestamp"=\>2019-05-17T18:54:11.934Z, "@version"=\>"1", "tags"=\>["swift\_paco\_logs"], "message"=\>"May 7 03:15:09 > **Summary** > > mstpol13 object-server: 10.188.42.67 - - [07/May/2019:10:15:09 +0000] "DELETE /d345/11314/ACC\_439/be495727-fcc8-4f2b-a3f8-e1305175b5bc/1554307448596%3A27a29814-dd8f-43ee-b768-19af98bf1d07%3A40" 404 70 "DELETE [http://localhost/v1/ACC\_439/be495727-fcc8-4f2b-a3f8-e1305175b5bc/1554307448596%3A27a29814-dd8f-43ee-b768-19af98bf1d07%3A40](http://localhost/v1/ACC_439/be495727-fcc8-4f2b-a3f8-e1305175b5bc/1554307448596%3A27a29814-dd8f-43ee-b768-19af98bf1d07%3A40)" "txb694b5d6133045268cffc-005cd15aad" "proxy-server 20232" 0.0235 "-" 32623 0", "host"=\>"0.0.0.0", "path"=\>"/opt/swiftlogs/test.log"}} [2019-05-17T11:54:12,127][DEBUG][logstash.pipeline] filter received {"event"=\>{"@timestamp"=\>2019-05-17T18:54:12.009Z, "@version"=\>"1", "tags"=\>["swift\_paco\_logs"], "message"=\>"May 14 03:36:05 > **Summary** > > dllabsw1 account-server: 172.24.7.109 - - [14/May/2019:03:36:05 +0000] "HEAD /d3/33163/ACC\_VZ\_8" 204 - "HEAD [http://localhost/v1/ACC\_VZ\_8?format=json](http://localhost/v1/ACC_VZ_8?format=json)" "tx4dce18b99d8d40b582337-005cda37a5" "proxy-server 13994" 0.0018 "-" 35858 -", "host"=\>"0.0.0.0", "path"=\>"/opt/swiftlogs/test.log"}} [2019-05-17T11:54:12,127][DEBUG][logstash.pipeline] filter received {"event"=\>{"@timestamp"=\>2019-05-17T18:54:12.014Z, "@version"=\>"1", "tags"=\>["swift\_paco\_logs"], "message"=\>"May 14 03:36:16 > **Summary** > > dllabsw1 account-server: 172.24.7.109 - - [14/May/2019:03:36:16 +0000] "GET /d3/49561/.misplaced\_objects" 404 - "GET [http://localhost/v1/.misplaced\_objects?marker=&prefix=&end\_marker=&format=json](http://localhost/v1/.misplaced_objects?marker=&prefix=&end_marker=&format=json)" "txcf347ccd4b5247f1975e7-005cda37b0" "proxy-server 13978" 0.0004 "-" 35821 -", "host"=\>"0.0.0.0", "path"=\>"/opt/swiftlogs/test.log"}} [2019-05-17T11:54:12,127][DEBUG][logstash.pipeline] filter received {"event"=\>{"@timestamp"=\>2019-05-17T18:54:12.016Z, "@version"=\>"1", "tags"=\>["swift\_paco\_logs"], "message"=\>"May 14 03:36:14 > **Summary** > > dllabsw1 object-server: 172.24.7.108 - - [14/May/2019:03:36:14 +0000] "DELETE /d10/60128/ACC\_VZ\_2/CONT\_VZ\_5/10a29c70-6590-11e9-93da-632735cf0214/1557632120301%3ARGVmYXVsdC8zMzAwMDY5MDU4%3Af7b7b9d1-7466-11e9-bed6-679edf32393d" 204 - "DELETE [http://localhost/v1/ACC\_VZ\_2/CONT\_VZ\_5/10a29c70-6590-11e9-93da-632735cf0214/1557632120301%3ARGVmYXVsdC8zMzAwMDY5MDU4%3Af7b7b9d1-7466-11e9-bed6-679edf32393d](http://localhost/v1/ACC_VZ_2/CONT_VZ_5/10a29c70-6590-11e9-93da-632735cf0214/1557632120301%3ARGVmYXVsdC8zMzAwMDY5MDU4%3Af7b7b9d1-7466-11e9-bed6-679edf32393d)" "tx65fde37e031b428dadf00-005cda37ae" "proxy-server 35676" 0.0270 "-" 36158 0", "host"=\>"0.0.0.0", "path"=\>"/opt/swiftlogs/test.log"}} [2019-05-17T11:54:12,128][DEBUG][logstash.pipeline] filter received {"event"=\>{"@timestamp"=\>2019-05-17T18:54:12.018Z, "@version"=\>"1", "tags"=\>["swift\_paco\_logs"], "message"=\>"May 14 03:36:16 > **Summary** > > dllabsw1 container-server: 172.24.7.109 - - [14/May/2019:03:36:16 +0000] "GET /d3/38894/.misplaced\_objects/1557802800" 404 - "GET [http://localhost/v1/.misplaced\_objects/1557802800?marker=&states=listing&prefix=&end\_marker=&format=json](http://localhost/v1/.misplaced_objects/1557802800?marker=&states=listing&prefix=&end_marker=&format=json)" "tx916fef5c496149da8790d-005cda37b0" "proxy-server 13978" 0.0006 "-" 36135 0", "host"=\>"0.0.0.0", "path"=\>"/opt/swiftlogs/test.log"}} [2019-05-17T11:54:12,128][DEBUG][logstash.pipeline] filter received {"event"=\>{"@timestamp"=\>2019-05-17T18:54:12.020Z, "@version"=\>"1", "tags"=\>["swift\_paco\_logs"], "message"=\>"May 13 03:30:30 > **Summary** > > dllabsw1 proxy-server: 172.24.7.97 172.24.7.97 13/May/2019/03/30/30 PUT /v1/ACC\_VZ\_1/CONT\_VZ\_2/file/NzYwMDA2NzI2OHwwMDAwMDAwLTAwMDAtMDE3Ny0wMDA3LTYyOTc1fDIwMTktMDUtMTRUMDM6MzA6Mjla/0 HTTP/1.0 201 - - - 5243780 - - txe8bfda41221243dc8fa93-005cd8e4d5 - 0.1125 - x-delete-at:1557804629 1557718229.895745039 1557718230.008220911 0", "host"=\>"0.0.0.0", "path"=\>"/opt/swiftlogs/test.log"}} Thanks Chandra > **Summary** > > This text will be hidden ---
**Author:** ![chandukreddi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chandukreddi/32/29241_2.png) [@chandukreddi](https://discuss.elastic.co/u/chandukreddi)\ **Post date:** [May 17, 2019, 7:40pm UTC](https://discuss.elastic.co/t/logstash-applying-grok-to-only-specific-log-lines/181645/2 "2019-05-17T19:40:52Z")
**output received:** [2019-05-17T11:54:12,270][DEBUG][logstash.pipeline] output received {"event"=\>{"logsource"=\>"mstpol13", "host"=\>"0.0.0.0", "Object"=\>"[http://localhost/v1/ACC\_439/be495727-fcc8-4f2b-a3f8-e1305175b5bc/1554307448596%3A27a29814-dd8f-43ee-b768-19af98bf1d07%3A40](http://localhost/v1/ACC_439/be495727-fcc8-4f2b-a3f8-e1305175b5bc/1554307448596%3A27a29814-dd8f-43ee-b768-19af98bf1d07%3A40)", "response\_time"=\>0.0235, "txn"=\>"txb694b5d6133045268cffc-005cd15aad", "@version"=\>"1", "method"=\>"DELETE", "tags"=\>["swift\_paco\_logs", "swift\_object\_parsed"], "response"=\>"404", "@timestamp"=\>2019-05-17T18:54:11.934Z, "message"=\>["May 7 03:15:09 mstpol13 object-server: 10.188.42.67 - - [07/May/2019:10:15:09 +0000] "DELETE /d345/11314/ACC\_439/be495727-fcc8-4f2b-a3f8-e1305175b5bc/1554307448596%3A27a29814-dd8f-43ee-b768-19af98bf1d07%3A40" 404 70 "DELETE [http://localhost/v1/ACC\_439/be495727-fcc8-4f2b-a3f8-e1305175b5bc/1554307448596%3A27a29814-dd8f-43ee-b768-19af98bf1d07%3A40](http://localhost/v1/ACC_439/be495727-fcc8-4f2b-a3f8-e1305175b5bc/1554307448596%3A27a29814-dd8f-43ee-b768-19af98bf1d07%3A40)" "txb694b5d6133045268cffc-005cd15aad" "proxy-server 20232" 0.0235 "-" 32623 0", "07/May/2019:10:15:09 +0000", "/d345/11314/ACC\_439/be495727-fcc8-4f2b-a3f8-e1305175b5bc/1554307448596%3A27a29814-dd8f-43ee-b768-19af98bf1d07%3A40", "70", "DELETE", ""proxy-server 20232"", ""-" 32623 0"], "path"=\>"/opt/swiftlogs/test.log", "program"=\>["object-server", "object-server"], "logdate"=\>2019-05-07T10:15:09.000Z}} [2019-05-17T11:54:12,275][DEBUG][logstash.pipeline] output received {"event"=\>{"logsource"=\>"dllabsw1", "host"=\>"0.0.0.0", "Object"=\>"[http://localhost/v1/ACC\_VZ\_8?format=json](http://localhost/v1/ACC_VZ_8?format=json)", "response\_time"=\>0.0018, "txn"=\>"tx4dce18b99d8d40b582337-005cda37a5", "@version"=\>"1", "method"=\>"HEAD", "tags"=\>["swift\_paco\_logs", "swift\_account\_parsed"], "response"=\>"204", "@timestamp"=\>2019-05-17T18:54:12.009Z, "message"=\>["May 14 03:36:05 dllabsw1 account-server: 172.24.7.109 - - [14/May/2019:03:36:05 +0000] "HEAD /d3/33163/ACC\_VZ\_8" 204 - "HEAD [http://localhost/v1/ACC\_VZ\_8?format=json](http://localhost/v1/ACC_VZ_8?format=json)" "tx4dce18b99d8d40b582337-005cda37a5" "proxy-server 13994" 0.0018 "-" 35858 -", "14/May/2019:03:36:05 +0000", "/d3/33163/ACC\_VZ\_8", "HEAD", ""proxy-server 13994"", ""-" 35858 -"], "path"=\>"/opt/swiftlogs/test.log", "program"=\>["account-server", "account-server"], "logdate"=\>2019-05-14T10:36:05.000Z}} [2019-05-17T11:54:12,276][DEBUG][logstash.pipeline] output received {"event"=\>{"logsource"=\>"dllabsw1", "host"=\>"0.0.0.0", "Object"=\>"[http://localhost/v1/.misplaced\_objects?marker=&prefix=&end\_marker=&format=json](http://localhost/v1/.misplaced_objects?marker=&prefix=&end_marker=&format=json)", "response\_time"=\>0.0004, "txn"=\>"txcf347ccd4b5247f1975e7-005cda37b0", "@version"=\>"1", "method"=\>"GET", "tags"=\>["swift\_paco\_logs", "swift\_account\_parsed"], "response"=\>"404", "@timestamp"=\>2019-05-17T18:54:12.014Z, "message"=\>["May 14 03:36:16 dllabsw1 account-server: 172.24.7.109 - - [14/May/2019:03:36:16 +0000] "GET /d3/49561/.misplaced\_objects" 404 - "GET [http://localhost/v1/.misplaced\_objects?marker=&prefix=&end\_marker=&format=json](http://localhost/v1/.misplaced_objects?marker=&prefix=&end_marker=&format=json)" "txcf347ccd4b5247f1975e7-005cda37b0" "proxy-server 13978" 0.0004 "-" 35821 -", "14/May/2019:03:36:16 +0000", "/d3/49561/.misplaced\_objects", "GET", ""proxy-server 13978"", ""-" 35821 -"], "path"=\>"/opt/swiftlogs/test.log", "program"=\>["account-server", "account-server"], "logdate"=\>2019-05-14T10:36:16.000Z}} [2019-05-17T11:54:12,276][DEBUG][logstash.pipeline] output received {"event"=\>{"logsource"=\>"dllabsw1", "host"=\>"0.0.0.0", "Object"=\>"[http://localhost/v1/.misplaced\_objects/1557802800?marker=&states=listing&prefix=&end\_marker=&format=json](http://localhost/v1/.misplaced_objects/1557802800?marker=&states=listing&prefix=&end_marker=&format=json)", "response\_time"=\>0.0006, "txn"=\>"tx916fef5c496149da8790d-005cda37b0", "@version"=\>"1", "method"=\>"GET", "tags"=\>["swift\_paco\_logs", "swift\_container\_parsed"], "response"=\>"404", "@timestamp"=\>2019-05-17T18:54:12.018Z, "message"=\>["May 14 03:36:16 dllabsw1 container-server: 172.24.7.109 - - [14/May/2019:03:36:16 +0000] "GET /d3/38894/.misplaced\_objects/1557802800" 404 - "GET [http://localhost/v1/.misplaced\_objects/1557802800?marker=&states=listing&prefix=&end\_marker=&format=json](http://localhost/v1/.misplaced_objects/1557802800?marker=&states=listing&prefix=&end_marker=&format=json)" "tx916fef5c496149da8790d-005cda37b0" "proxy-server 13978" 0.0006 "-" 36135 0", "14/May/2019:03:36:16 +0000", "/d3/38894/.misplaced\_objects/1557802800", "GET", ""proxy-server 13978"", ""-" 36135 0"], "path"=\>"/opt/swiftlogs/test.log", "program"=\>["container-server", "container-server"], "logdate"=\>2019-05-14T10:36:16.000Z}} ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [May 17, 2019, 8:01pm UTC](https://discuss.elastic.co/t/logstash-applying-grok-to-only-specific-log-lines/181645/3 "2019-05-17T20:01:29Z")
What does your configuration, especially the grok filter, look like? ---
**Author:** ![chandukreddi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chandukreddi/32/29241_2.png) [@chandukreddi](https://discuss.elastic.co/u/chandukreddi)\ **Post date:** [May 17, 2019, 8:07pm UTC](https://discuss.elastic.co/t/logstash-applying-grok-to-only-specific-log-lines/181645/4 "2019-05-17T20:07:12Z")
@Badger, here is the grok patterns ``` SWIFT_PROXY_HEAD %{SYSLOGTIMESTAMP:logdate} %{SYSLOGHOST:logsource} %{SYSLOGPROG:program}: - - %{NOTSPACE:message} %{WORD:method} %{UNIXPATH:Object} %{WORD:message}/%{NUMBER:httpversion} %{NUMBER:response} - %{WORD:message} - - - - %{NOTSPACE:txn} - %{NUMBER:response_time} %{GREEDYDATA:message} SWIFT_PROXY_GET_POST %{SYSLOGTIMESTAMP:logdate} %{SYSLOGHOST:logsource} %{SYSLOGPROG:program}: %{HOSTNAME} %{HOSTNAME} %{NOTSPACE:message} %{WORD:method} %{UNIXPATH:Object} %{WORD:message}/%{NUMBER:httpversion} %{NUMBER:response} - - - - %{NUMBER:message} - %{NOTSPACE:txn} - %{NUMBER:response_time} %{GREEDYDATA:message} SWIFT_PROXY_PUT %{SYSLOGTIMESTAMP:logdate} %{SYSLOGHOST:logsource} %{SYSLOGPROG:program}: %{HOSTNAME} %{HOSTNAME} %{NOTSPACE:message} %{WORD:method} %{UNIXPATH:Object} %{WORD:message}/%{NUMBER:httpversion} %{NUMBER:response} - - - %{NUMBER:message} - - %{NOTSPACE:txn} - %{NUMBER:response_time} %{GREEDYDATA:message} SWIFT_PROXY_DELETE %{SYSLOGTIMESTAMP:logdate} %{SYSLOGHOST:logsource} %{SYSLOGPROG:program}: %{HOSTNAME} %{HOSTNAME} %{NOTSPACE:message} %{WORD:method} %{UNIXPATH:Object} %{WORD:message}/%{NUMBER:httpversion} %{NUMBER:response} - - - - - - %{NOTSPACE:txn} - %{NUMBER:response_time} - - %{GREEDYDATA:message} ####################### #OBJECT-SERVER: PROGRAM ####################### SWIFT_OBJECT_GET %{SYSLOGTIMESTAMP:logdate} %{SYSLOGHOST:logsource} %{SYSLOGPROG:program}: %{IPV4} - - \[%{HAPROXYDATE:message}\] "%{WORD:method} %{UNIXPATH:message}" %{NUMBER:response} %{NUMBER:message} "%{WORD:message} %{NOTSPACE:Object}" "%{NOTSPACE:txn}" %{QS:message} %{NUMBER:response_time} %{GREEDYDATA:message} SWIFT_OBJECT_PUT_DEL %{SYSLOGTIMESTAMP:logdate} %{SYSLOGHOST:logsource} %{SYSLOGPROG:program}: %{IPV4} - - \[%{HAPROXYDATE:message}\] "%{WORD:method} %{UNIXPATH:message}" %{NUMBER:response} (-|%{NUMBER}) "%{WORD:message} %{NOTSPACE:Object}" "%{NOTSPACE:txn}" %{QS:message} %{NUMBER:response_time} %{GREEDYDATA:message} SWIFT_OBJECT_POST %{SYSLOGTIMESTAMP:logdate} %{SYSLOGHOST:logsource} %{SYSLOGPROG:program}: %{IPV4} - - \[%{HAPROXYDATE:message}\] "%{WORD:method} %{UNIXPATH:message}" %{NUMBER:response} %{NUMBER:message} "%{WORD:message} %{NOTSPACE:Object}" "%{NOTSPACE:txn}" %{QS:message} %{NUMBER:response_time} %{GREEDYDATA:message} ########################## #CONTAINER-SERVER: PROGRAM ########################## SWIFT_CONTAINER_HEAD_GET_PUT_DELETE %{SYSLOGTIMESTAMP:logdate} %{SYSLOGHOST:logsource} %{SYSLOGPROG:program}: %{IPV4} - - \[%{HAPROXYDATE:message}\] "%{WORD:method} %{UNIXPATH:message}" %{NUMBER:response} - "%{WORD:message} %{NOTSPACE:Object}" "%{NOTSPACE:txn}" %{QS:message} %{NUMBER:response_time} %{GREEDYDATA:message} ######################### #ACCOUNT-SERVER: PROGRAM ######################### SWIFT_ACCOUNT_HEAD_GET_PUT_DELETE_0 %{SYSLOGTIMESTAMP:logdate} %{SYSLOGHOST:logsource} %{SYSLOGPROG:program}: %{IPV4} - - \[%{HAPROXYDATE:message}\] "%{WORD:method} %{UNIXPATH:message}" %{NUMBER:response} - "%{WORD:message} %{NOTSPACE:Object}" "%{NOTSPACE:txn}" %{QS:message} %{NUMBER:response_time} %{GREEDYDATA:message} SWIFT_ACCOUNT_HEAD_GET_PUT_DELETE_1 %{SYSLOGTIMESTAMP:logdate} %{SYSLOGHOST:logsource} %{SYSLOGPROG:program}: %{IPV4} - - \[%{HAPROXYDATE:message}\] "%{WORD:method} %{UNIXPATH:Object}" %{NUMBER:response} - "-" "-" %{QS:message} %{NUMBER:response_time} %{GREEDYDATA:message} SWIFT_AP_REST_ALL %{SYSLOGTIMESTAMP:logdate} %{GREEDYDATA:msg} ``` ---
**Author:** ![chandukreddi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chandukreddi/32/29241_2.png) [@chandukreddi](https://discuss.elastic.co/u/chandukreddi)\ **Post date:** [May 17, 2019, 8:08pm UTC](https://discuss.elastic.co/t/logstash-applying-grok-to-only-specific-log-lines/181645/5 "2019-05-17T20:08:50Z")
these are my sample log lines: May 7 03:15:09 mstpol13 object-server: 10.188.42.67 - - [07/May/2019:10:15:09 +0000] "DELETE /d345/11314/ACC\_439/be495727-fcc8-4f2b-a3f8-e1305175b5bc/1554307448596%3A27a29814-dd8f-43ee-b768-19af98bf1d07%3A40" 404 70 "DELETE [http://localhost/v1/ACC\_439/be495727-fcc8-4f2b-a3f8-e1305175b5bc/1554307448596%3A27a29814-dd8f-43ee-b768-19af98bf1d07%3A40](http://localhost/v1/ACC_439/be495727-fcc8-4f2b-a3f8-e1305175b5bc/1554307448596%3A27a29814-dd8f-43ee-b768-19af98bf1d07%3A40)" "txb694b5d6133045268cffc-005cd15aad" "proxy-server 20232" 0.0235 "-" 32623 0 May 14 03:36:05 dllabsw1 account-server: 172.24.7.109 - - [14/May/2019:03:36:05 +0000] "HEAD /d3/33163/ACC\_VZ\_8" 204 - "HEAD [http://localhost/v1/ACC\_VZ\_8?format=json](http://localhost/v1/ACC_VZ_8?format=json)" "tx4dce18b99d8d40b582337-005cda37a5" "proxy-server 13994" 0.0018 "-" 35858 - May 14 03:36:15 dllabsw1 account-server: 172.24.7.109 - - [14/May/2019:03:36:15 +0000] "PUT /d3/26974/.expiring\_objects/1557791933" 201 - "-" "-" "container-updater 13974" 0.0007 "-" 35828 0 May 14 03:36:16 dllabsw1 account-server: 172.24.7.109 - - [14/May/2019:03:36:16 +0000] "GET /d3/49561/.misplaced\_objects" 404 - "GET [http://localhost/v1/.misplaced\_objects?marker=&prefix=&end\_marker=&format=json](http://localhost/v1/.misplaced_objects?marker=&prefix=&end_marker=&format=json)" "txcf347ccd4b5247f1975e7-005cda37b0" "proxy-server 13978" 0.0004 "-" 35821 - May 14 03:36:14 dllabsw1 object-server: 172.24.7.108 - - [14/May/2019:03:36:14 +0000] "DELETE /d10/60128/ACC\_VZ\_2/CONT\_VZ\_5/10a29c70-6590-11e9-93da-632735cf0214/1557632120301%3ARGVmYXVsdC8zMzAwMDY5MDU4%3Af7b7b9d1-7466-11e9-bed6-679edf32393d" 204 - "DELETE [http://localhost/v1/ACC\_VZ\_2/CONT\_VZ\_5/10a29c70-6590-11e9-93da-632735cf0214/1557632120301%3ARGVmYXVsdC8zMzAwMDY5MDU4%3Af7b7b9d1-7466-11e9-bed6-679edf32393d](http://localhost/v1/ACC_VZ_2/CONT_VZ_5/10a29c70-6590-11e9-93da-632735cf0214/1557632120301%3ARGVmYXVsdC8zMzAwMDY5MDU4%3Af7b7b9d1-7466-11e9-bed6-679edf32393d)" "tx65fde37e031b428dadf00-005cda37ae" "proxy-server 35676" 0.0270 "-" 36158 0 May 14 03:36:16 dllabsw1 container-server: 172.24.7.109 - - [14/May/2019:03:36:16 +0000] "GET /d3/38894/.misplaced\_objects/1557802800" 404 - "GET [http://localhost/v1/.misplaced\_objects/1557802800?marker=&states=listing&prefix=&end\_marker=&format=json](http://localhost/v1/.misplaced_objects/1557802800?marker=&states=listing&prefix=&end_marker=&format=json)" "tx916fef5c496149da8790d-005cda37b0" "proxy-server 13978" 0.0006 "-" 36135 0 May 13 03:30:30 dllabsw1 proxy-server: 172.24.7.97 172.24.7.97 13/May/2019/03/30/30 PUT /v1/ACC\_VZ\_1/CONT\_VZ\_2/file/NzYwMDA2NzI2OHwwMDAwMDAwLTAwMDAtMDE3Ny0wMDA3LTYyOTc1fDIwMTktMDUtMTRUMDM6MzA6Mjla/0 HTTP/1.0 201 - - - 5243780 - - txe8bfda41221243dc8fa93-005cd8e4d5 - 0.1125 - x-delete-at:1557804629 1557718229.895745039 1557718230.008220911 0 ---
**Author:** ![chandukreddi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chandukreddi/32/29241_2.png) [@chandukreddi](https://discuss.elastic.co/u/chandukreddi)\ **Post date:** [May 17, 2019, 8:10pm UTC](https://discuss.elastic.co/t/logstash-applying-grok-to-only-specific-log-lines/181645/6 "2019-05-17T20:10:28Z")
Please let me know if you want my config file. when I was testing above log lines I noticed inconsistent outputs, some times it parsing 2 log lines and some times 3/4 log lines. Thanks Chandra ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [May 17, 2019, 8:11pm UTC](https://discuss.elastic.co/t/logstash-applying-grok-to-only-specific-log-lines/181645/7 "2019-05-17T20:11:48Z")
I want to see the grok filter at least. ---
**Author:** ![chandukreddi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chandukreddi/32/29241_2.png) [@chandukreddi](https://discuss.elastic.co/u/chandukreddi)\ **Post date:** [May 17, 2019, 8:13pm UTC](https://discuss.elastic.co/t/logstash-applying-grok-to-only-specific-log-lines/181645/8 "2019-05-17T20:13:06Z")
conf file: ``` input { file { path => "/opt/swiftlogs/test.log" start_position => "beginning" sincedb_path => "/dev/null" #sincedb_path => "/opt/data/logstash/plugins/inputs/file/proxy-aco-logs.log" tags => ["swift_paco_logs"] } } #input { # beats { # port => 5044 # } #} filter { if "swift_paco_logs" in [tags] and "proxy-server:" in [message] { grok { patterns_dir => ["/opt/logstash/patterns"] break_on_match => true match => { "message" => ["%{SWIFT_PROXY_HEAD}", "%{SWIFT_PROXY_GET_POST}", "%{SWIFT_PROXY_PUT}", "%{SWIFT_PROXY_DELETE}" ] } add_tag => ["swift_proxy_parsed"] } } if "swift_paco_logs" in [tags] and "object-server:" in [message] { grok { patterns_dir => ["/opt/logstash/patterns"] match => { "message" => ["%{SWIFT_OBJECT_GET}", "%{SWIFT_OBJECT_PUT_DEL}", "%{SWIFT_OBJECT_POST}" ] } add_tag => ["swift_object_parsed"] } } if "swift_paco_logs" in [tags] and "container-server:" in [message] { grok { patterns_dir => ["/opt/logstash/patterns"] match => { "message" => ["%{SWIFT_CONTAINER_HEAD_GET_PUT_DELETE}"] } add_tag => ["swift_container_parsed"] } } if "swift_paco_logs" in [tags] and "account-server:" in [message] { grok { patterns_dir => ["/opt/logstash/patterns"] match => { "message" => ["%{SWIFT_ACCOUNT_HEAD_GET_PUT_DELETE_0}", "%{SWIFT_ACCOUNT_HEAD_GET_PUT_DELETE_1}" ] } add_tag => ["swift_account_parsed"] } } if "swift_proxy_parsed" not in [tags] and "swift_object_parsed" not in [tags] and "swift_container_parsed" not in [tags] and "swift_account_parsed" not in [tags] { grok { patterns_dir => ["/opt/logstash/patterns"] match => { "message" => ["%{SWIFT_AP_REST_ALL}"] } add_tag => ["swift_rest_all"] } } date { match => ["logdate", "MMM dd HH:mm:ss", "MMM d HH:mm:ss"] target => "logdate" timezone => "PST8PDT" } #date { #match => ["logdate", "MMM d HH:mm:ss"] #target => "logdate" #timezone => "PST8PDT" #} mutate { convert => { "httpversion" => "float" "status_code" => "integer" "response_time" => "float" } remove_field => ["httpversion","haproxy_hour", "haproxy_milliseconds", "haproxy_minute", "haproxy_month", "haproxy_monthday", "haproxy_second", "haproxy_time", "haproxy_year"] } } output { #if "swift_paco_logs" in [tags] { elasticsearch { hosts => ["http://IP:9200"] index => "swift_paco" #index => "swift_ap_log_write_alias" template => "/opt/logstash/config/templates/swift_ap_log_sizing.json" template_name => "swift_ap_log_template" template_overwrite => true #user => '***' #password => '***' #ssl => true #cacert => "/opt/logstash-6.1.3/config/ssl/logstash.pem" #ssl_certificate_verification => false } stdout {} #} } ``` ---
**Author:** ![chandukreddi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chandukreddi/32/29241_2.png) [@chandukreddi](https://discuss.elastic.co/u/chandukreddi)\ **Post date:** [May 17, 2019, 8:13pm UTC](https://discuss.elastic.co/t/logstash-applying-grok-to-only-specific-log-lines/181645/9 "2019-05-17T20:13:52Z")
using pipeline as below. - pipeline.id: swiftaplogs pipeline.workers: 2 path.config: "/opt/logstash/config/all\_config/swift\_aco\_proxy\_log.conf" ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [May 17, 2019, 8:42pm UTC](https://discuss.elastic.co/t/logstash-applying-grok-to-only-specific-log-lines/181645/10 "2019-05-17T20:42:21Z")
I am amazed it parses anything at all! I let it run for several minutes and then killed it without anything being parsed. Firstly, you should read [Do you grok Grok?](https://www.elastic.co/blog/do-you-grok-grok) on the elastic blog. Seriously, go and read that. My post will still be here when you get back. Next, anchor all of your patterns with ^, to match start of line. Your patterns match the entire line, so tell grok that. Change ``` SWIFT_PROXY_DELETE %{SYSLOGTIMESTAMP:logdate} [...] ``` to ``` SWIFT_PROXY_DELETE ^%{SYSLOGTIMESTAMP:logdate} [...] ``` At that point I still have logstash hung, burning 100% of the CPU and get no events. The next problem ... [do not use UNIXPATH!](https://github.com/logstash-plugins/logstash-patterns-core/issues/159) It is insanely expensive when it does not match. I replaced UNIXPATH with NOTSPACE and logstash processed the 7 test lines in your first post in a fraction of a second. Now UNIX paths can have spaces in them, so the two are not equivalent, but the point is use a different regexp that matches your data, do not use UNIXPATH. ---
**Author:** ![chandukreddi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chandukreddi/32/29241_2.png) [@chandukreddi](https://discuss.elastic.co/u/chandukreddi)\ **Post date:** [May 17, 2019, 8:46pm UTC](https://discuss.elastic.co/t/logstash-applying-grok-to-only-specific-log-lines/181645/11 "2019-05-17T20:46:50Z")
I really appreciate your comments @Badger.. I will read and change as you suggested and update you. but one qq.. Did you notices same behavior what I have seen before changing **^** and **UNIXPATH**? Thanks Chandra ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [May 17, 2019, 8:54pm UTC](https://discuss.elastic.co/t/logstash-applying-grok-to-only-specific-log-lines/181645/12 "2019-05-17T20:54:37Z")
> [@chandukreddi](#): > > but one qq.. Did you notices same behavior what I have seen before changing **^** and **UNIXPATH**? I am on a single CPU machine, if it is stuck back-tracking on one pattern, no other patterns can be processed. If you had a multi-CPU machine then it would be possible for events to be processed on one CPU which another CPU were stuck back-tracking on PROXY patterns. It would depend on the order of lines and how quickly they failed to match the PROXY patterns. I would not be at all surprised for it to be data dependent. PS. You have posted a commented-out password in your config. You might want to change that to ``` password => "*" ``` or something like that. ---
**Author:** ![chandukreddi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chandukreddi/32/29241_2.png) [@chandukreddi](https://discuss.elastic.co/u/chandukreddi)\ **Post date:** [May 21, 2019, 4:20pm UTC](https://discuss.elastic.co/t/logstash-applying-grok-to-only-specific-log-lines/181645/13 "2019-05-21T16:20:20Z")
Thanks @Badger for your help.. after applying you suggested changes its working perfect. Thanks Chandra ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 18, 2019, 4:25pm UTC](https://discuss.elastic.co/t/logstash-applying-grok-to-only-specific-log-lines/181645/14 "2019-06-18T16:25:30Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.