Logstash filter is not working ...kindly help

Hi, Need your help
I've bunch of messages if kafka. few samples are given below.

  1. {"processId":40,"parentProcessId":40,"type":"Info","service":"a-b-c","object":"common_factory","method":"listUsers","log":"Start listUsers\nlistUsers-Success\n","startTime":1654164165036,"endTime":1654164165179,"createdBy":"Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpbWFnZVVybCI6IiIsImlkIjoiMTYzMDkwMjk3NDc1MyIsIkxBTl9pZCI6bnVsbCwidHlwZSI6InBhcnRuZXIiLCJ1c2VyIjoibmVlcmFqYSIsIm5hbWUiOiJOZWVyYWphIC0gMSIsImVtYWlsIjoibmVlcmFqYS5yZWRkeUBhZ2lsaXp0ZWNoLmNvbSIsIk5BRF9yb2xlIjoxLCJiYW5kIjpudWxsLCJqaXJhX3VzZXJuYW1lIjoibmVlcmFqYSIsImdpdGxhYl91c2VybmFtZSI6Ik5BIiwiZGVzaWduYXRpb24iOiJTb2Z0d2FyZSBEZXZlbG9wZXIiLCJleHRlcm5hbCI6dHJ1ZSwicGFydG5lcl9pZCI6IjYxMzU5OTM1ZThhMTE5MDAyMDY4NTUxNiIsInNlc3Npb25fa2V5IjoiYm1WbGNtRnFZUzV5WldSa2VVQmhaMmxzYVhwMFpXTm9MbU52YlRveE5qVTBNVFkwTVRZME9EYzUiLCJwYXNzd29yZCI6IlUyRnNkR1ZrWDE5VFJDcEd2d0I0L1RxSlF1b2M4Z24xcndsNzZRMGd5NFE9IiwidGltZXN0YW1wIjoxNjU0MTY0MTY0ODc5LCJvcmdfaWQiOiJhZ2lsaXp0ZWNoIiwiaWF0IjoxNjU0MTY0MTY0LCJleHAiOjE2NTQxNjUzNjR9.PzaM4jodwXCsX-W0iSEb3sVKqHD0-CMceacFv66F1rw","data":null}

  2. {"processId":40,"parentProcessId":40,"type":"Info","service":"p-q-r","object":"common_factory","method":"listUsers","log":"Start listUsers\nlistUsers-Success\n","startTime":1654164165036,"endTime":1654164165179,"createdBy":"Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpbWFnZVVybCI6IiIsImlkIjoiMTYzMDkwMjk3NDc1MyIsIkxBTl9pZCI6bnVsbCwidHlwZSI6InBhcnRuZXIiLCJ1c2VyIjoibmVlcmFqYSIsIm5hbWUiOiJOZWVyYWphIC0gMSIsImVtYWlsIjoibmVlcmFqYS5yZWRkeUBhZ2lsaXp0ZWNoLmNvbSIsIk5BRF9yb2xlIjoxLCJiYW5kIjpudWxsLCJqaXJhX3VzZXJuYW1lIjoibmVlcmFqYSIsImdpdGxhYl91c2VybmFtZSI6Ik5BIiwiZGVzaWduYXRpb24iOiJTb2Z0d2FyZSBEZXZlbG9wZXIiLCJleHRlcm5hbCI6dHJ1ZSwicGFydG5lcl9pZCI6IjYxMzU5OTM1ZThhMTE5MDAyMDY4NTUxNiIsInNlc3Npb25fa2V5IjoiYm1WbGNtRnFZUzV5WldSa2VVQmhaMmxzYVhwMFpXTm9MbU52YlRveE5qVTBNVFkwTVRZME9EYzUiLCJwYXNzd29yZCI6IlUyRnNkR1ZrWDE5VFJDcEd2d0I0L1RxSlF1b2M4Z24xcndsNzZRMGd5NFE9IiwidGltZXN0YW1wIjoxNjU0MTY0MTY0ODc5LCJvcmdfaWQiOiJhZ2lsaXp0ZWNoIiwiaWF0IjoxNjU0MTY0MTY0LCJleHAiOjE2NTQxNjUzNjR9.PzaM4jodwXCsX-W0iSEb3sVKqHD0-CMceacFv66F1rw","data":null}

But I want to filter messages only having "service":"a-b-c" from bunch of messages

I tried below filters
1.
filter {
grok {
match => {
"message" => "%{GREEDYDATA:nad-UaaS-ms}" }
}

filter {
grok {
match => { "service" => "nad-UaaS-ms"}
}
}

But its not working because it is inserting all messages into elastisearch. Kindly correct me

Can you provide more context of what you want to do? It is not clear.

Also, your message is a json message, there is no reason to use grok to parse a json message, you should use the json filter.

1 Like

Thanks for replying.
Let me explain, I have bunch of error logs in which there is a key "service" which describes the error log for particular service or application. For ex. "service":"a-b-c" describes that this error log is of application "a-b-c".

Likewise, there are many application wise error logs but I want to insert logs of application having name "service":"a-b-c" into elastisearch index.

I tried few filters as given in above question but it is inserting all applications log instead specific logs in elastisearch.

I hope this explains. If not let me know what part is confusing!

First you need to parse your message, you can do that using the json filter.

filter {
    json {
        source => "message"
    }
}

This will create the fields in your document with the keys name, like service, type, object etc.

Then after that you could have a conditional in your filter section to drop messages where the service is not a-b-c.

if [service] != "a-b-c" {
    drop {}
}

Thanks leandrojmp sir ! It's a big help.....I appreciate it.
Its running succesfully as expected

issue resolved

Hi, I am facing below issue.

I want to extract full dump that is all available documents from index "X" using python code and save it to json file. But currently M able to extract few documents only using below command. For this I am using elasticsearch python librarry

res = es.search(index="X",body={"query":{"match_all":{}}})

kindly suggest

I would suggest that you open a new topic as this question is unrelated to your problem.

Also, a simple search won't return everything, you are limited to 10000 hits, if you want everything you need to use the Scroll API, with the python library you can use the scan helper.

Thanks for the quick help @leandrojmp [leandrojmp].

I opened new ticket as title " Issue in extracting all available data in index in elastic using python library elasticsearch"