Logstash filter is not working ...kindly help

Elastic Stack Logstash
1

Hi, Need your help
I've bunch of messages if kafka. few samples are given below.

  1. {"processId":40,"parentProcessId":40,"type":"Info","service":"a-b-c","object":"common_factory","method":"listUsers","log":"Start listUsers\nlistUsers-Success\n","startTime":1654164165036,"endTime":1654164165179,"createdBy":"Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpbWFnZVVybCI6IiIsImlkIjoiMTYzMDkwMjk3NDc1MyIsIkxBTl9pZCI6bnVsbCwidHlwZSI6InBhcnRuZXIiLCJ1c2VyIjoibmVlcmFqYSIsIm5hbWUiOiJOZWVyYWphIC0gMSIsImVtYWlsIjoibmVlcmFqYS5yZWRkeUBhZ2lsaXp0ZWNoLmNvbSIsIk5BRF9yb2xlIjoxLCJiYW5kIjpudWxsLCJqaXJhX3VzZXJuYW1lIjoibmVlcmFqYSIsImdpdGxhYl91c2VybmFtZSI6Ik5BIiwiZGVzaWduYXRpb24iOiJTb2Z0d2FyZSBEZXZlbG9wZXIiLCJleHRlcm5hbCI6dHJ1ZSwicGFydG5lcl9pZCI6IjYxMzU5OTM1ZThhMTE5MDAyMDY4NTUxNiIsInNlc3Npb25fa2V5IjoiYm1WbGNtRnFZUzV5WldSa2VVQmhaMmxzYVhwMFpXTm9MbU52YlRveE5qVTBNVFkwTVRZME9EYzUiLCJwYXNzd29yZCI6IlUyRnNkR1ZrWDE5VFJDcEd2d0I0L1RxSlF1b2M4Z24xcndsNzZRMGd5NFE9IiwidGltZXN0YW1wIjoxNjU0MTY0MTY0ODc5LCJvcmdfaWQiOiJhZ2lsaXp0ZWNoIiwiaWF0IjoxNjU0MTY0MTY0LCJleHAiOjE2NTQxNjUzNjR9.PzaM4jodwXCsX-W0iSEb3sVKqHD0-CMceacFv66F1rw","data":null}

  2. {"processId":40,"parentProcessId":40,"type":"Info","service":"p-q-r","object":"common_factory","method":"listUsers","log":"Start listUsers\nlistUsers-Success\n","startTime":1654164165036,"endTime":1654164165179,"createdBy":"Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpbWFnZVVybCI6IiIsImlkIjoiMTYzMDkwMjk3NDc1MyIsIkxBTl9pZCI6bnVsbCwidHlwZSI6InBhcnRuZXIiLCJ1c2VyIjoibmVlcmFqYSIsIm5hbWUiOiJOZWVyYWphIC0gMSIsImVtYWlsIjoibmVlcmFqYS5yZWRkeUBhZ2lsaXp0ZWNoLmNvbSIsIk5BRF9yb2xlIjoxLCJiYW5kIjpudWxsLCJqaXJhX3VzZXJuYW1lIjoibmVlcmFqYSIsImdpdGxhYl91c2VybmFtZSI6Ik5BIiwiZGVzaWduYXRpb24iOiJTb2Z0d2FyZSBEZXZlbG9wZXIiLCJleHRlcm5hbCI6dHJ1ZSwicGFydG5lcl9pZCI6IjYxMzU5OTM1ZThhMTE5MDAyMDY4NTUxNiIsInNlc3Npb25fa2V5IjoiYm1WbGNtRnFZUzV5WldSa2VVQmhaMmxzYVhwMFpXTm9MbU52YlRveE5qVTBNVFkwTVRZME9EYzUiLCJwYXNzd29yZCI6IlUyRnNkR1ZrWDE5VFJDcEd2d0I0L1RxSlF1b2M4Z24xcndsNzZRMGd5NFE9IiwidGltZXN0YW1wIjoxNjU0MTY0MTY0ODc5LCJvcmdfaWQiOiJhZ2lsaXp0ZWNoIiwiaWF0IjoxNjU0MTY0MTY0LCJleHAiOjE2NTQxNjUzNjR9.PzaM4jodwXCsX-W0iSEb3sVKqHD0-CMceacFv66F1rw","data":null}

But I want to filter messages only having "service":"a-b-c" from bunch of messages

I tried below filters
1.
filter {
grok {
match => {
"message" => "%{GREEDYDATA:nad-UaaS-ms}" }
}

filter {
grok {
match => { "service" => "nad-UaaS-ms"}
}
}

But its not working because it is inserting all messages into elastisearch. Kindly correct me

2

Can you provide more context of what you want to do? It is not clear.

Also, your message is a json message, there is no reason to use grok to parse a json message, you should use the json filter.

1 Like
3

Thanks for replying.
Let me explain, I have bunch of error logs in which there is a key "service" which describes the error log for particular service or application. For ex. "service":"a-b-c" describes that this error log is of application "a-b-c".

Likewise, there are many application wise error logs but I want to insert logs of application having name "service":"a-b-c" into elastisearch index.

I tried few filters as given in above question but it is inserting all applications log instead specific logs in elastisearch.

I hope this explains. If not let me know what part is confusing!

4

First you need to parse your message, you can do that using the json filter.

filter {
    json {
        source => "message"
    }
}

This will create the fields in your document with the keys name, like service, type, object etc.

Then after that you could have a conditional in your filter section to drop messages where the service is not a-b-c.

if [service] != "a-b-c" {
    drop {}
}
5

Thanks leandrojmp sir ! It's a big help.....I appreciate it.
Its running succesfully as expected

6

issue resolved

7

Hi, I am facing below issue.

I want to extract full dump that is all available documents from index "X" using python code and save it to json file. But currently M able to extract few documents only using below command. For this I am using elasticsearch python librarry

res = es.search(index="X",body={"query":{"match_all":{}}})

kindly suggest

8

I would suggest that you open a new topic as this question is unrelated to your problem.

Also, a simple search won't return everything, you are limited to 10000 hits, if you want everything you need to use the Scroll API, with the python library you can use the scan helper.

9

Thanks for the quick help @leandrojmp [leandrojmp].

I opened new ticket as title " Issue in extracting all available data in index in elastic using python library elasticsearch"

10

facing issue in order to apply multiple if conditions.
I want to extract logs of name service :a-b-c whose type = "error"
But its not working as expected, basically it is extracting all a-b-c service logs

below is my filter condition

filter {
json {
source => "message"
}
if [service] != "a-b-c"
{
drop {}
}
if [type] != "error"
{
drop()
}
}

11

If you want to test two conditions at the same time, you need to use the two conditions on the same if.

Your first conditional tests the field service and your second condition test the field type, the first if will drop everything where the value of the service field is not a-b-c and the second if will drop everything where the value of field type is not error, but those conditionals are independent from each other.

You need to use something like this:

if [service] != "a-b-c" and [type] != "error"

I also recommend that you check this part of the documentation.

12

Thanks for reply!

I modified the condition as below
if [service] != "a-b-c" or [type] != "error"

Now its working

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.