Logstash not parsing eventhub logs

Elastic Stack Logstash
1

Hi Everyone ,
I am working on parsing the logs from Azure eventhub to Elasticsearch and trying to use logstash filter for this but getting _jsonparsing error for the logs.
the logs structure is someting like below:

{"records":[{"key1":"value1","key2":"value2","key3":{"key31":"value31","key32":"value32","key33:{"Message: "timestamp, level { "event1":"string", "eventlog":"string2", "eventlog1":"string2",}]}

I have tried the filter as below :

filter {
 json {
      source => "message"
    }

 mutate {
     split => [ "records" , "," ]

 }
}

experts please help me, Thanks in advance.

2

You are missing two double quotes there, that will result in a _jsonparsefailure with the message

<LogStash::Json::ParserError: Unexpected character ('M' (code 77)): was expecting a colon to separate field name and value

If that's not the message you are getting then tell us exactly what error message you are getting.

3

What is the source of these logs? Entra ID? Activity Logs? Some internal Azure service?

Normally Azure tools that send logs to Event Hub will add the events inside a top-level field named records, and you need to use a split filter in this field to get one event per message.

Note that this is the split filter, not the split action of the mutate filter as you used.

You need something like this:

filter {
    json {
        source => "message"        
    }
    split {
        field => "records"
    }
}

It is not exactly clear what is your issue and what is failing, but your split action inside the mutate filter is probably not what you want to do.

4

Thanks @Badger for your reply ,
you observation is correct but that is my typo , I have missed the quotes while sanitizing the log.

5

Thanks @leandrojmp for your suggestion and I am trying to parse the Activity Logs. I the error I am getting is as follows :

  :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): was expecting double-quote to start field name

[2024-09-20T07:48:05,225][WARN ][logstash.filters.split   ][movetest][ae5383c6896d462e6367ce3cfdf59ed1d97d8f31b59c110b8155711c7917b9e8] Only String and Array types are splittable. field:records is of type = NilClass
6

Hi @leandrojmp , @Badger ,
Same issue is discussed here in Logstash split filter issue ,and my logs are like same.

7

The filter in that thread successfully processes the example data in that thread. It certainly does not produce the error message shown in that thread. If you want us to help you need to provide a reproducible example, which you have not done so yet.

8

Yeah, it is what I thought.

My suggestion would be that if is possible you drop Logstash and use the Elastic Agent integration to get those logs.

The logs from Activity Logs and Entra ID are pretty bad to parse entirely in logstash, they can break in multiple places and you will need to keep creating workarounds with mutate to make it work.

Not sure why the split failed because you didn't shared the full error log, but when I tried to parse it with logstash this is what I was using in the beginning of the pipeline.

filter {
    json {
        source => "message"
        remove_field => ["message"]
    }
    split {
        field => "records"
    }
}

This should work without any issue, but my recommendation is to not use Logstash in this case, but the Elastic Agent Integration.

1 Like
9

Yes @Badger !! you are correct I will try to give an example which can give a fair idea. Thanks for your help and assistance. I really appreciate all the help support from community.

10

Thanks @leandrojmp for your suggestion. It is really wise suggestion and I appreciate your help. Thanks again I will try to follow your suggestion on Monday as soon as I can get the hold of it.

11

Hi @leandrojmp ,
Finally I was able to implement the filter you suggested and we were able to achieve the expected results.
I really appreciate your and @Badger's help and support from Community.

Thanks
Vaibhav