# Filebeat configurations for multiline json input

**URL:** <https://discuss.elastic.co/t/filebeat-configurations-for-multiline-json-input/156992>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [November 16, 2018, 7:18am UTC](https://discuss.elastic.co/t/filebeat-configurations-for-multiline-json-input/156992 "2018-11-16T07:18:21Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Susmitha\_Devathi](https://avatars.discourse-cdn.com/v4/letter/s/b77776/32.png) [@Susmitha\_Devathi](https://discuss.elastic.co/u/Susmitha_Devathi)\
**Post date:** [November 16, 2018, 7:18am UTC](https://discuss.elastic.co/t/filebeat-configurations-for-multiline-json-input/156992/1 "2018-11-16T07:18:21Z")

</div>

Hi Team,

could you please let me know how to write json filter and filebeat configurations for below data .

```auto
{
agentId: "TMS",
apiVersion: "v2",
entities: [
{
agentId: "susmitha",
name: "EFGH",
cacheManagerName: "Article_",
attributes: {
Size: 799625,
NonStopTimeoutRate: 0,
LocalOffHeapSizeInBytes: 0,
LocalDiskSizeInBytes: 0,
CacheSearchRate: 0,
CacheRemoveRate: 0,
CacheOffHeapMissRate: 0,
CacheOnDiskHitRate: 0,
WriterQueueLength: 0,
CacheOffHeapHitRate: 0,
CacheExpirationRate: 0,
LocalHeapSize: 0,
NonStopFailureRate: 0,
CacheOnDiskMissRate: 0,
CacheInMemoryMissRate: 0,
TransactionCommitRate: 0,
LocalHeapSizeInBytes: 0,
NonStopRejoinTimeoutRate: 0,
TransactionRollbackRate: 0,
CacheHitRate: 0,
CacheEvictionRate: 0,
NonStopSuccessRate: 0,
LocalOffHeapSize: 0,
CacheInMemoryHitRate: 0,
LocalDiskSize: 0,
CacheUpdateRate: 0
}
},
{
agentId: "susmitha",
name: "ABCD",
cacheManagerName: "Article",
attributes: {
Size: 984362,
NonStopTimeoutRate: 0,
LocalOffHeapSizeInBytes: 0,
LocalDiskSizeInBytes: 0,
CacheSearchRate: 0,
CacheRemoveRate: 0,
CacheOffHeapMissRate: 0,
CacheOnDiskHitRate: 0,
WriterQueueLength: 0,
CacheOffHeapHitRate: 0,
CacheExpirationRate: 0,
LocalHeapSize: 0,
NonStopFailureRate: 0,
CacheOnDiskMissRate: 0,
CacheInMemoryMissRate: 0,
TransactionCommitRate: 0,
LocalHeapSizeInBytes: 0,
NonStopRejoinTimeoutRate: 0,
TransactionRollbackRate: 0,
CacheHitRate: 0,
CacheEvictionRate: 0,
NonStopSuccessRate: 0,
LocalOffHeapSize: 0,
CacheInMemoryHitRate: 0,
LocalDiskSize: 0,
CacheUpdateRate: 0
}
},

```

I need to filter each attribute individually to create data table similarly as below.  
Let me know all the changes that need to be done create similar data table.

 ![datatable](https://us1.discourse-cdn.com/elastic/original/3X/b/6/b6ee0430a70091f188c84f09ad1802b49fb85bf7.jpeg)

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [November 16, 2018, 1:49pm UTC](https://discuss.elastic.co/t/filebeat-configurations-for-multiline-json-input/156992/2 "2018-11-16T13:49:13Z")

</div>

Is the json file indented? If not I'm afraid this is not possible with filebeat.

---

<div class="post-metadata">

**Author:** ![Susmitha\_Devathi](https://avatars.discourse-cdn.com/v4/letter/s/b77776/32.png) [@Susmitha\_Devathi](https://discuss.elastic.co/u/Susmitha_Devathi)\
**Post date:** [November 20, 2018, 6:03am UTC](https://discuss.elastic.co/t/filebeat-configurations-for-multiline-json-input/156992/4 "2018-11-20T06:03:29Z")

</div>

Hi Team,

Could you please let me know if there is any possible way to convert below data to json format and then JSON filter to parse data.Because my data is not in exact JSON format,only few lines of data are in JSON format.

I want to fetch each attribute to create datatable.  
 ![JSON](https://us1.discourse-cdn.com/elastic/original/3X/2/3/230ed7955316e8c5745d212bf38e4c11e2521420.jpeg)

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [November 20, 2018, 10:24am UTC](https://discuss.elastic.co/t/filebeat-configurations-for-multiline-json-input/156992/5 "2018-11-20T10:24:13Z")

</div>

Is this the complete event?

Do you have one event per file or multiple events per file?

This is no valid json, you are at least missing `]}`.

---

<div class="post-metadata">

**Author:** ![Susmitha\_Devathi](https://avatars.discourse-cdn.com/v4/letter/s/b77776/32.png) [@Susmitha\_Devathi](https://discuss.elastic.co/u/Susmitha_Devathi)\
**Post date:** [November 21, 2018, 10:15am UTC](https://discuss.elastic.co/t/filebeat-configurations-for-multiline-json-input/156992/6 "2018-11-21T10:15:52Z")

</div>

Hi Steffen,

There are multiple events in file.  
This is not valid json, but could you please let me know how to parse this data, suggest any filter that converts this data to JSON format,so that we can parse data using json filter.

Regards,  
D Susmitha

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [November 21, 2018, 2:49pm UTC](https://discuss.elastic.co/t/filebeat-configurations-for-multiline-json-input/156992/7 "2018-11-21T14:49:29Z")

</div>

Trying to convert it to json and then parse would require some custom code. Sometimes the format is using commas when not allowed, sometimes it's correct with comma usage. And it looks like we're missing closing symbols. You can't do this with beats.

Depending on what an event should be here, maybe using multiline one can combine some fields between `agendID` and the first occurrence of `{` or `}`.But then you will have to do some more custom processing in logstash.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 19, 2018, 2:49pm UTC](https://discuss.elastic.co/t/filebeat-configurations-for-multiline-json-input/156992/8 "2018-12-19T14:49:35Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.