# Logstash not parsing eventhub logs

**URL:** <https://discuss.elastic.co/t/logstash-not-parsing-eventhub-logs/366816>\
**Category:** Logstash\
**Created:** [September 19, 2024, 5:34pm UTC](https://discuss.elastic.co/t/logstash-not-parsing-eventhub-logs/366816 "2024-09-19T17:34:22Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![vaibhavu](https://avatars.discourse-cdn.com/v4/letter/v/d07c76/32.png) [@vaibhavu](https://discuss.elastic.co/u/vaibhavu)\
**Post date:** [September 19, 2024, 5:34pm UTC](https://discuss.elastic.co/t/logstash-not-parsing-eventhub-logs/366816/1 "2024-09-19T17:34:22Z")

</div>

Hi Everyone ,  
I am working on parsing the logs from Azure eventhub to Elasticsearch and trying to use logstash filter for this but getting \_jsonparsing error for the logs.  
the logs structure is someting like below:

```auto
{"records":[{"key1":"value1","key2":"value2","key3":{"key31":"value31","key32":"value32","key33:{"Message: "timestamp, level { "event1":"string", "eventlog":"string2", "eventlog1":"string2",}]}  

```

I have tried the filter as below :

```auto
filter {
 json {
      source => "message"
    }

 mutate {
     split => ["records" , ","]

 }
}

```

experts please help me, Thanks in advance.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 19, 2024, 5:54pm UTC](https://discuss.elastic.co/t/logstash-not-parsing-eventhub-logs/366816/2 "2024-09-19T17:54:15Z")

</div>

> [@vaibhavu](#):
>
> `"key33:{"Message: "timestamp,...`

You are missing two double quotes there, that will result in a \_jsonparsefailure with the message

```
<LogStash::Json::ParserError: Unexpected character ('M' (code 77)): was expecting a colon to separate field name and value

```

If that's not the message you are getting then tell us exactly what error message you _are_ getting.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [September 19, 2024, 5:58pm UTC](https://discuss.elastic.co/t/logstash-not-parsing-eventhub-logs/366816/3 "2024-09-19T17:58:51Z")

</div>

What is the source of these logs? Entra ID? Activity Logs? Some internal Azure service?

Normally Azure tools that send logs to Event Hub will add the events inside a top-level field named `records`, and you need to use a `split` filter in this field to get one event per message.

Note that this is the `split` filter, not the `split` action of the `mutate` filter as you used.

You need something like this:

```auto
filter {
    json {
        source => "message"        
    }
    split {
        field => "records"
    }
}

```

It is not exactly clear what is your issue and what is failing, but your `split` action inside the `mutate` filter is probably not what you want to do.

---

<div class="post-metadata">

**Author:** ![vaibhavu](https://avatars.discourse-cdn.com/v4/letter/v/d07c76/32.png) [@vaibhavu](https://discuss.elastic.co/u/vaibhavu)\
**Post date:** [September 20, 2024, 7:55am UTC](https://discuss.elastic.co/t/logstash-not-parsing-eventhub-logs/366816/4 "2024-09-20T07:55:38Z")

</div>

Thanks @Badger for your reply ,  
you observation is correct but that is my typo , I have missed the quotes while sanitizing the log.

---

<div class="post-metadata">

**Author:** ![vaibhavu](https://avatars.discourse-cdn.com/v4/letter/v/d07c76/32.png) [@vaibhavu](https://discuss.elastic.co/u/vaibhavu)\
**Post date:** [September 20, 2024, 8:02am UTC](https://discuss.elastic.co/t/logstash-not-parsing-eventhub-logs/366816/5 "2024-09-20T08:02:15Z")

</div>

Thanks @leandrojmp for your suggestion and I am trying to parse the Activity Logs. I the error I am getting is as follows :

```auto
  :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): was expecting double-quote to start field name

[2024-09-20T07:48:05,225][WARN][logstash.filters.split][movetest][ae5383c6896d462e6367ce3cfdf59ed1d97d8f31b59c110b8155711c7917b9e8] Only String and Array types are splittable. field:records is of type = NilClass

```

---

<div class="post-metadata">

**Author:** ![vaibhavu](https://avatars.discourse-cdn.com/v4/letter/v/d07c76/32.png) [@vaibhavu](https://discuss.elastic.co/u/vaibhavu)\
**Post date:** [September 20, 2024, 10:58am UTC](https://discuss.elastic.co/t/logstash-not-parsing-eventhub-logs/366816/6 "2024-09-20T10:58:21Z")

</div>

Hi @leandrojmp , @Badger ,  
Same issue is discussed here in [Logstash split filter issue](https://discuss.elastic.co/t/logstash-split-filter-issue/119291) ,and my logs are like same.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 20, 2024, 2:05pm UTC](https://discuss.elastic.co/t/logstash-not-parsing-eventhub-logs/366816/7 "2024-09-20T14:05:42Z")

</div>

> [@vaibhavu](#):
>
> Same issue is discussed here in [Logstash split filter issue](https://discuss.elastic.co/t/logstash-split-filter-issue/119291) ,and my logs are like same.

The filter in that thread successfully processes the example data in that thread. It certainly does not produce the error message shown in that thread. If you want us to help you need to provide a reproducible example, which you have not done so yet.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [September 20, 2024, 2:51pm UTC](https://discuss.elastic.co/t/logstash-not-parsing-eventhub-logs/366816/8 "2024-09-20T14:51:58Z")

</div>

> [@vaibhavu](#):
>
> I am trying to parse the Activity Logs

Yeah, it is what I thought.

My suggestion would be that if is possible you drop Logstash and use the Elastic Agent integration to get those logs.

The logs from Activity Logs and Entra ID are pretty bad to parse entirely in logstash, they can break in multiple places and you will need to keep creating workarounds with mutate to make it work.

Not sure why the split failed because you didn't shared the full error log, but when I tried to parse it with logstash this is what I was using in the beginning of the pipeline.

```auto
filter {
    json {
        source => "message"
        remove_field => ["message"]
    }
    split {
        field => "records"
    }
}

```

This should work without any issue, but my recommendation is to not use Logstash in this case, but the Elastic Agent Integration.

---

<div class="post-metadata">

**Author:** ![vaibhavu](https://avatars.discourse-cdn.com/v4/letter/v/d07c76/32.png) [@vaibhavu](https://discuss.elastic.co/u/vaibhavu)\
**Post date:** [September 22, 2024, 4:27am UTC](https://discuss.elastic.co/t/logstash-not-parsing-eventhub-logs/366816/9 "2024-09-22T04:27:34Z")

</div>

Yes @Badger !! you are correct I will try to give an example which can give a fair idea. Thanks for your help and assistance. I really appreciate all the help support from community.

---

<div class="post-metadata">

**Author:** ![vaibhavu](https://avatars.discourse-cdn.com/v4/letter/v/d07c76/32.png) [@vaibhavu](https://discuss.elastic.co/u/vaibhavu)\
**Post date:** [September 22, 2024, 4:31am UTC](https://discuss.elastic.co/t/logstash-not-parsing-eventhub-logs/366816/10 "2024-09-22T04:31:46Z")

</div>

Thanks @leandrojmp for your suggestion. It is really wise suggestion and I appreciate your help. Thanks again I will try to follow your suggestion on Monday as soon as I can get the hold of it.

---

<div class="post-metadata">

**Author:** ![vaibhavu](https://avatars.discourse-cdn.com/v4/letter/v/d07c76/32.png) [@vaibhavu](https://discuss.elastic.co/u/vaibhavu)\
**Post date:** [October 9, 2024, 6:16am UTC](https://discuss.elastic.co/t/logstash-not-parsing-eventhub-logs/366816/11 "2024-10-09T06:16:59Z")

</div>

Hi @leandrojmp ,  
Finally I was able to implement the filter you suggested and we were able to achieve the expected results.  
I really appreciate your and @Badger's help and support from Community.

Thanks  
Vaibhav