Filebeat configurations for multiline json input

[Susmitha_Devathi (Devathi Susmitha)]

Hi Team,

could you please let me know how to write json filter and filebeat configurations for below data .

{
agentId: "TMS",
apiVersion: "v2",
entities: [
{
agentId: "susmitha",
name: "EFGH",
cacheManagerName: "Article_",
attributes: {
Size: 799625,
NonStopTimeoutRate: 0,
LocalOffHeapSizeInBytes: 0,
LocalDiskSizeInBytes: 0,
CacheSearchRate: 0,
CacheRemoveRate: 0,
CacheOffHeapMissRate: 0,
CacheOnDiskHitRate: 0,
WriterQueueLength: 0,
CacheOffHeapHitRate: 0,
CacheExpirationRate: 0,
LocalHeapSize: 0,
NonStopFailureRate: 0,
CacheOnDiskMissRate: 0,
CacheInMemoryMissRate: 0,
TransactionCommitRate: 0,
LocalHeapSizeInBytes: 0,
NonStopRejoinTimeoutRate: 0,
TransactionRollbackRate: 0,
CacheHitRate: 0,
CacheEvictionRate: 0,
NonStopSuccessRate: 0,
LocalOffHeapSize: 0,
CacheInMemoryHitRate: 0,
LocalDiskSize: 0,
CacheUpdateRate: 0
}
},
{
agentId: "susmitha",
name: "ABCD",
cacheManagerName: "Article",
attributes: {
Size: 984362,
NonStopTimeoutRate: 0,
LocalOffHeapSizeInBytes: 0,
LocalDiskSizeInBytes: 0,
CacheSearchRate: 0,
CacheRemoveRate: 0,
CacheOffHeapMissRate: 0,
CacheOnDiskHitRate: 0,
WriterQueueLength: 0,
CacheOffHeapHitRate: 0,
CacheExpirationRate: 0,
LocalHeapSize: 0,
NonStopFailureRate: 0,
CacheOnDiskMissRate: 0,
CacheInMemoryMissRate: 0,
TransactionCommitRate: 0,
LocalHeapSizeInBytes: 0,
NonStopRejoinTimeoutRate: 0,
TransactionRollbackRate: 0,
CacheHitRate: 0,
CacheEvictionRate: 0,
NonStopSuccessRate: 0,
LocalOffHeapSize: 0,
CacheInMemoryHitRate: 0,
LocalDiskSize: 0,
CacheUpdateRate: 0
}
},

I need to filter each attribute individually to create data table similarly as below.
Let me know all the changes that need to be done create similar data table.

[steffens (Steffen Siering)]

Is the json file indented? If not I'm afraid this is not possible with filebeat.

[Susmitha_Devathi (Devathi Susmitha)]

Hi Team,

Could you please let me know if there is any possible way to convert below data to json format and then JSON filter to parse data.Because my data is not in exact JSON format,only few lines of data are in JSON format.

I want to fetch each attribute to create datatable.

[steffens (Steffen Siering)]

Is this the complete event?

Do you have one event per file or multiple events per file?

This is no valid json, you are at least missing ]}.

[Susmitha_Devathi (Devathi Susmitha)]

Hi Steffen,

There are multiple events in file.
This is not valid json, but could you please let me know how to parse this data, suggest any filter that converts this data to JSON format,so that we can parse data using json filter.

Regards,
D Susmitha

[steffens (Steffen Siering)]

Trying to convert it to json and then parse would require some custom code. Sometimes the format is using commas when not allowed, sometimes it's correct with comma usage. And it looks like we're missing closing symbols. You can't do this with beats.

Depending on what an event should be here, maybe using multiline one can combine some fields between agendID and the first occurrence of { or }.But then you will have to do some more custom processing in logstash.