# Logstash filter is not working ...kindly help **URL:** \ **Category:** Logstash\ **Created:** [June 3, 2022, 9:01am UTC](https://discuss.elastic.co/t/logstash-filter-is-not-working-kindly-help/306306 "2022-06-03T09:01:10Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![aniketdatir](https://avatars.discourse-cdn.com/v4/letter/a/8e8cbc/32.png) [@aniketdatir](https://discuss.elastic.co/u/aniketdatir)\ **Post date:** [June 3, 2022, 9:01am UTC](https://discuss.elastic.co/t/logstash-filter-is-not-working-kindly-help/306306/1 "2022-06-03T09:01:10Z")
Hi, Need your help I've bunch of messages if kafka. few samples are given below. 1. {"processId":40,"parentProcessId":40,"type":"Info", **"service":"a-b-c"** ,"object":"common\_factory","method":"listUsers","log":"Start listUsers\nlistUsers-Success\n","startTime":1654164165036,"endTime":1654164165179,"createdBy":"Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpbWFnZVVybCI6IiIsImlkIjoiMTYzMDkwMjk3NDc1MyIsIkxBTl9pZCI6bnVsbCwidHlwZSI6InBhcnRuZXIiLCJ1c2VyIjoibmVlcmFqYSIsIm5hbWUiOiJOZWVyYWphIC0gMSIsImVtYWlsIjoibmVlcmFqYS5yZWRkeUBhZ2lsaXp0ZWNoLmNvbSIsIk5BRF9yb2xlIjoxLCJiYW5kIjpudWxsLCJqaXJhX3VzZXJuYW1lIjoibmVlcmFqYSIsImdpdGxhYl91c2VybmFtZSI6Ik5BIiwiZGVzaWduYXRpb24iOiJTb2Z0d2FyZSBEZXZlbG9wZXIiLCJleHRlcm5hbCI6dHJ1ZSwicGFydG5lcl9pZCI6IjYxMzU5OTM1ZThhMTE5MDAyMDY4NTUxNiIsInNlc3Npb25fa2V5IjoiYm1WbGNtRnFZUzV5WldSa2VVQmhaMmxzYVhwMFpXTm9MbU52YlRveE5qVTBNVFkwTVRZME9EYzUiLCJwYXNzd29yZCI6IlUyRnNkR1ZrWDE5VFJDcEd2d0I0L1RxSlF1b2M4Z24xcndsNzZRMGd5NFE9IiwidGltZXN0YW1wIjoxNjU0MTY0MTY0ODc5LCJvcmdfaWQiOiJhZ2lsaXp0ZWNoIiwiaWF0IjoxNjU0MTY0MTY0LCJleHAiOjE2NTQxNjUzNjR9.PzaM4jodwXCsX-W0iSEb3sVKqHD0-CMceacFv66F1rw","data":null} 2. {"processId":40,"parentProcessId":40,"type":"Info", **"service":"p-q-r"** ,"object":"common\_factory","method":"listUsers","log":"Start listUsers\nlistUsers-Success\n","startTime":1654164165036,"endTime":1654164165179,"createdBy":"Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpbWFnZVVybCI6IiIsImlkIjoiMTYzMDkwMjk3NDc1MyIsIkxBTl9pZCI6bnVsbCwidHlwZSI6InBhcnRuZXIiLCJ1c2VyIjoibmVlcmFqYSIsIm5hbWUiOiJOZWVyYWphIC0gMSIsImVtYWlsIjoibmVlcmFqYS5yZWRkeUBhZ2lsaXp0ZWNoLmNvbSIsIk5BRF9yb2xlIjoxLCJiYW5kIjpudWxsLCJqaXJhX3VzZXJuYW1lIjoibmVlcmFqYSIsImdpdGxhYl91c2VybmFtZSI6Ik5BIiwiZGVzaWduYXRpb24iOiJTb2Z0d2FyZSBEZXZlbG9wZXIiLCJleHRlcm5hbCI6dHJ1ZSwicGFydG5lcl9pZCI6IjYxMzU5OTM1ZThhMTE5MDAyMDY4NTUxNiIsInNlc3Npb25fa2V5IjoiYm1WbGNtRnFZUzV5WldSa2VVQmhaMmxzYVhwMFpXTm9MbU52YlRveE5qVTBNVFkwTVRZME9EYzUiLCJwYXNzd29yZCI6IlUyRnNkR1ZrWDE5VFJDcEd2d0I0L1RxSlF1b2M4Z24xcndsNzZRMGd5NFE9IiwidGltZXN0YW1wIjoxNjU0MTY0MTY0ODc5LCJvcmdfaWQiOiJhZ2lsaXp0ZWNoIiwiaWF0IjoxNjU0MTY0MTY0LCJleHAiOjE2NTQxNjUzNjR9.PzaM4jodwXCsX-W0iSEb3sVKqHD0-CMceacFv66F1rw","data":null} But I want to filter messages only having "service":"a-b-c" from bunch of messages I tried below filters 1. filter { grok { match =\> { "message" =\> "%{GREEDYDATA:nad-UaaS-ms}" } } 1. filter { grok { match =\> { "service" =\> "nad-UaaS-ms"} } } But its not working because it is inserting all messages into elastisearch. Kindly correct me ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [June 3, 2022, 12:32pm UTC](https://discuss.elastic.co/t/logstash-filter-is-not-working-kindly-help/306306/2 "2022-06-03T12:32:44Z")
Can you provide more context of what you want to do? It is not clear. Also, your message is a `json` message, there is no reason to use `grok` to parse a json message, you should use the `json` filter. ---
**Author:** ![aniketdatir](https://avatars.discourse-cdn.com/v4/letter/a/8e8cbc/32.png) [@aniketdatir](https://discuss.elastic.co/u/aniketdatir)\ **Post date:** [June 3, 2022, 12:41pm UTC](https://discuss.elastic.co/t/logstash-filter-is-not-working-kindly-help/306306/3 "2022-06-03T12:41:41Z")
Thanks for replying. Let me explain, I have bunch of error logs in which there is a key "service" which describes the error log for particular service or application. For ex. **"service":"a-b-c"** describes that this error log is of application "a-b-c". Likewise, there are many application wise error logs but I want to insert logs of application having name **"service":"a-b-c"** into elastisearch index. I tried few filters as given in above question but it is inserting all applications log instead specific logs in elastisearch. I hope this explains. If not let me know what part is confusing! ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [June 3, 2022, 1:27pm UTC](https://discuss.elastic.co/t/logstash-filter-is-not-working-kindly-help/306306/4 "2022-06-03T13:27:36Z")
First you need to parse your message, you can do that using the `json` filter. ```auto filter { json { source => "message" } } ``` This will create the fields in your document with the keys name, like `service`, `type`, `object` etc. Then after that you could have a conditional in your filter section to drop messages where the `service` is not `a-b-c`. ```auto if [service] != "a-b-c" { drop {} } ``` ```auto ``` ---
**Author:** ![aniketdatir](https://avatars.discourse-cdn.com/v4/letter/a/8e8cbc/32.png) [@aniketdatir](https://discuss.elastic.co/u/aniketdatir)\ **Post date:** [June 3, 2022, 2:19pm UTC](https://discuss.elastic.co/t/logstash-filter-is-not-working-kindly-help/306306/5 "2022-06-03T14:19:29Z")
Thanks leandrojmp sir ! It's a big help.....I appreciate it. Its running succesfully as expected ---
**Author:** ![aniketdatir](https://avatars.discourse-cdn.com/v4/letter/a/8e8cbc/32.png) [@aniketdatir](https://discuss.elastic.co/u/aniketdatir)\ **Post date:** [June 21, 2022, 10:53am UTC](https://discuss.elastic.co/t/logstash-filter-is-not-working-kindly-help/306306/6 "2022-06-21T10:53:42Z")
issue resolved ---
**Author:** ![aniketdatir](https://avatars.discourse-cdn.com/v4/letter/a/8e8cbc/32.png) [@aniketdatir](https://discuss.elastic.co/u/aniketdatir)\ **Post date:** [June 29, 2022, 11:14am UTC](https://discuss.elastic.co/t/logstash-filter-is-not-working-kindly-help/306306/7 "2022-06-29T11:14:18Z")
Hi, I am facing below issue. I want to extract full dump that is all available documents from index "X" using python code and save it to json file. But currently M able to extract few documents only using below command. **For this I am using elasticsearch python librarry** res = es.search(index="X",body={"query":{"match\_all":{}}}) kindly suggest ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [June 29, 2022, 12:58pm UTC](https://discuss.elastic.co/t/logstash-filter-is-not-working-kindly-help/306306/8 "2022-06-29T12:58:23Z")
I would suggest that you open a new topic as this question is unrelated to your problem. Also, a simple search won't return everything, you are limited to 10000 hits, if you want everything you need to use the [Scroll API](https://www.elastic.co/guide/en/elasticsearch/reference/current/scroll-api.html#scroll-api), with the python library you can use the [scan helper](https://elasticsearch-py.readthedocs.io/en/v7.17.5/helpers.html#scan). ---
**Author:** ![aniketdatir](https://avatars.discourse-cdn.com/v4/letter/a/8e8cbc/32.png) [@aniketdatir](https://discuss.elastic.co/u/aniketdatir)\ **Post date:** [June 29, 2022, 1:05pm UTC](https://discuss.elastic.co/t/logstash-filter-is-not-working-kindly-help/306306/9 "2022-06-29T13:05:55Z")
Thanks for the quick help @leandrojmp [leandrojmp]. I opened new ticket as title " [Issue in extracting all available data in index in elastic using python library elasticsearch](https://discuss.elastic.co/t/issue-in-extracting-all-available-data-in-index-in-elastic-using-python-library-elasticsearch/308470)" ---
**Author:** ![aniketdatir](https://avatars.discourse-cdn.com/v4/letter/a/8e8cbc/32.png) [@aniketdatir](https://discuss.elastic.co/u/aniketdatir)\ **Post date:** [July 6, 2022, 8:06am UTC](https://discuss.elastic.co/t/logstash-filter-is-not-working-kindly-help/306306/10 "2022-07-06T08:06:06Z")
facing issue in order to apply multiple if conditions. _I want to extract logs of name `service` :`a-b-c` whose type = "error"_ But its not working as expected, basically it is extracting all a-b-c service logs **below is my filter condition** filter { json { source =\> "message" } if [service] != "a-b-c" { drop {} } if [type] != "error" { drop() } } ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [July 6, 2022, 1:28pm UTC](https://discuss.elastic.co/t/logstash-filter-is-not-working-kindly-help/306306/11 "2022-07-06T13:28:20Z")
If you want to test two conditions at the same time, you need to use the two conditions on the same `if`. Your first conditional tests the field `service` and your second condition test the field `type`, the first `if` will drop everything where the value of the `service` field is not `a-b-c` and the second `if` will drop everything where the value of field `type` is not `error`, but those conditionals are independent from each other. You need to use something like this: `if [service] != "a-b-c" and [type] != "error"` I also recommend that you check this [part of the documentation](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html). ---
**Author:** ![aniketdatir](https://avatars.discourse-cdn.com/v4/letter/a/8e8cbc/32.png) [@aniketdatir](https://discuss.elastic.co/u/aniketdatir)\ **Post date:** [July 7, 2022, 10:20am UTC](https://discuss.elastic.co/t/logstash-filter-is-not-working-kindly-help/306306/12 "2022-07-07T10:20:18Z")
Thanks for reply! I modified the condition as below if [service] != "a-b-c" or [type] != "error" Now its working ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 4, 2022, 10:20am UTC](https://discuss.elastic.co/t/logstash-filter-is-not-working-kindly-help/306306/13 "2022-08-04T10:20:41Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.