Hi Everyone ,
I am working on parsing the logs from Azure eventhub to Elasticsearch and trying to use logstash filter for this but getting _jsonparsing error for the logs.
the logs structure is someting like below:
What is the source of these logs? Entra ID? Activity Logs? Some internal Azure service?
Normally Azure tools that send logs to Event Hub will add the events inside a top-level field named records, and you need to use a split filter in this field to get one event per message.
Note that this is the split filter, not the split action of the mutate filter as you used.
Thanks @leandrojmp for your suggestion and I am trying to parse the Activity Logs. I the error I am getting is as follows :
:exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): was expecting double-quote to start field name
[2024-09-20T07:48:05,225][WARN ][logstash.filters.split ][movetest][ae5383c6896d462e6367ce3cfdf59ed1d97d8f31b59c110b8155711c7917b9e8] Only String and Array types are splittable. field:records is of type = NilClass
The filter in that thread successfully processes the example data in that thread. It certainly does not produce the error message shown in that thread. If you want us to help you need to provide a reproducible example, which you have not done so yet.
My suggestion would be that if is possible you drop Logstash and use the Elastic Agent integration to get those logs.
The logs from Activity Logs and Entra ID are pretty bad to parse entirely in logstash, they can break in multiple places and you will need to keep creating workarounds with mutate to make it work.
Not sure why the split failed because you didn't shared the full error log, but when I tried to parse it with logstash this is what I was using in the beginning of the pipeline.
Yes @Badger !! you are correct I will try to give an example which can give a fair idea. Thanks for your help and assistance. I really appreciate all the help support from community.
Thanks @leandrojmp for your suggestion. It is really wise suggestion and I appreciate your help. Thanks again I will try to follow your suggestion on Monday as soon as I can get the hold of it.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.