Add Double quotes for json data in logstash

Elastic Stack Logstash
1

Hi Team,

I'm trying to parse a JSON data through logstash. I'm using JSON and split plugins to parse the data.

Below is my format of my JSON message.

{
agentId: "TMS",
apiVersion: "v2",
entities: [
{
agentId: "Server1",
name: "HouseBankID",
cacheManagerName: "RFT_HouseBank_CacheMgr",
attributes: {
Size: 21,
LocalHeapSize: 0,
CacheInMemoryMissRate: 0,
LocalHeapSizeInBytes: 0,
AverageGetTime: 0,
CacheHitRate: 0,
CacheOffHeapMissRate: 0,
LocalOffHeapSize: 0,
CacheInMemoryHitRate: 0,
CacheMissRate: 0,
CacheHitRatio: 0,
CacheOffHeapHitRate: 0
}
},
{
agentId: "Server2",
name: "HouseBankID",
cacheManagerName: "RFT_HouseBank_CacheMgr",
attributes: {
Size: 21,
LocalHeapSize: 0,
CacheInMemoryMissRate: 0,
LocalHeapSizeInBytes: 0,
AverageGetTime: 0,
CacheHitRate: 0,
CacheOffHeapMissRate: 0,
LocalOffHeapSize: 0,
CacheInMemoryHitRate: 0,
CacheMissRate: 0,
CacheHitRatio: 0,
CacheOffHeapHitRate: 0
}
}
],
exceptionEntities:
}

But I'm getting a parsing exception

exception=>#<LogStash::Json::ParserError: Unexpected character ('' (code 92)): was expecting double-quote to start field name

To my knowledge, A JSON data should be enclosed with in double quote like "field" : "value"

So is there any work around so that I can add double quotes to each and every line of JSON data. So that It's in a valid format to be parsed using a Json filter in logstash.

Below is my logstash configuration.

input {
beats {
port => 5044
}
}

filter{

mutate {
	gsub => [
		'message' , '},' , '}'
	]
}

json {
	source => "message"
}

split {
	field => "entities"
}

}

output{
stdout {
codec => rubydebug
}
}

It'd be of great help if anyone guide me with parsing this JSON data as expected.

Thank you in advance :slight_smile:

2

What is the source log file?

As in, where is the source read from? Is it possible to change the source to include the correct format?

3

Hi Lewis Barclay,

The source of log file is an Rest API. They read the data from that REST API and write them in a file format from which I'm accessing it through beats.

I'm afraid that the source cannot be modified , That's the reason I'm trying to figure out a work around through logstash. :confused:

Is there any way to overcome this scenario?

Thnak you taking your time and replying back.

4

How do you access the API? Could you use the logstash-http-poller input plugin to read the API directly rather than saving it to a file?

5

I'm afraid I cannot do that too. I've came across this while working on this. But since my employer is following this way of reading the data (from a file), there nothing much I can do about this input. :expressionless:

6

The issue is that the JSON is not JSON because it isn't in the correct format. So we would have to work around this issue. You could create a pattern using grok instead?

7

Yes Exactly, This is not a Valid JSON since they aren't enclosed with double quotes. I manually inserted double quotes and I'm able to parse them using the same logstash configuration. So the only thing missing are the quotes. If i can edit them I'll be able to parse them successfully.

I haven't tried grok for this data since there is already a JSON plugin which does the data parsing with ease. If grok can do it better I'll give it a try. :slight_smile:

8

It isn't that the grok is better, JSON would be better, except your file isn't JSON technically. Your other option would be to create a script that goes through the file and does the work for you. This isn't a logstash issue since the source file is incorrect. If you cannot correct the source then grok would be another option but you will need to create the pattern yourself.

9

Yeah, I'll look what I can do about the source file for inserting the quotes. If not I'll go ahead and try GROK filter for parsing the data.

Anyway, Thank you so much for your time mate, I appreciate it :slight_smile:

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.