Error with parsing Azure json multiline files

Elastic Stack Logstash
1

Hello,
Please advise how to solve this.

image.png1919×146 18.4 KB

Here is my logstash config file:
config.PNG1000×267 3.88 KB

2

Added this part to logstash config.

But still no success. Logstash can not parse json.
any ideas?

3

Please don't paste images of text as they can be very hard to read. Start troubleshooting this by removing the elasticsearch output and instead use the stdout plugin with a rubydebug codec, so you can see what the events Logstash processes look like.

4

Ok, thanks for your advice. but i do not know how to use Rubydebug.
Should i install ruby for this? i am not a programmer but only devops.

5

Create the following output block:

output {
  stdout {
    codec => rubydebug
  }
}

This will log messages to stdout so that you can inspect them.

6

Added but i can not see any messages in powershell console after logstash has been started. (and in logstash files as well)

[2017-03-03T13:33:25,634][INFO ][logstash.inputs.azureblob] Using version 0.9.x input plugin 'azureblob'. This plugin should work but would benefit from use by folks like you. Please let us know if you find bugs or have suggestions on how to improve this plugin.
[2017-03-03T13:33:26,068][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>250}
[2017-03-03T13:33:26,099][INFO ][logstash.pipeline ] Pipeline main started
[2017-03-03T13:33:26,240][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

7

Managed to see an error only after deleting existing index (and reindexing existing data).

at [Source: [B@7f4b9de9; line: 1, column: 16]>}
[2017-03-03T14:05:55,888][WARN ][logstash.filters.json ] Error parsing json {:source=>"message", :raw=>"\t\t\t "level": "Error",\r", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array,
ject, 'true', 'false' or 'null')
at [Source: [B@5b9388c5; line: 1, column: 13]>}
[2017-03-03T14:05:55,888][WARN ][logstash.filters.json ] Error parsing json {:source=>"message", :raw=>"\t\t\t "location": "global",\r", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, arr
, object, 'true', 'false' or 'null')
at [Source: [B@7aaacd26; line: 1, column: 16]>}
[2017-03-03T14:05:55,888][WARN ][logstash.filters.json ] Error parsing json {:source=>"message", :raw=>"\t\t\t "properties": {"statusCode":"BadRequest","serviceRequestId":null,"statusMessage":"{\"error\":{\"code\":\"InvalidTemplate\",\"mess
e\":\"Deployment template validation failed: 'The template reference 'PublicIP-LB-FE-0' is not valid: could not find template resource or resource copy with this name. Please see https://aka.ms/arm-template-expressions/#reference for usage details.'.\"}}"}\r",
xception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
at [Source: [B@3fc7bc5e; line: 1, column: 18]>}
[2017-03-03T14:05:55,888][WARN ][logstash.filters.json ] Error parsing json {:source=>"message", :raw=>"\t\t}\r", :exception=>#<LogStash::Json::ParserError: Unexpected close marker '}': expected ']' (for ROOT starting at [Source: [B@3e5a2208; line: 1, column: 0])
at [Source: [B@3e5a2208; line: 1, column: 4]>}
[2017-03-03T14:05:55,904][WARN ][logstash.filters.json ] Error parsing json {:source=>"message", :raw=>"\t]\r", :exception=>#<LogStash::Json::ParserError: Unexpected close marker ']': expected '}' (for ROOT starting at [Source: [B@1fbf7190; line: 1, column: 0])
at [Source: [B@1fbf7190; line: 1, column: 3]>}
[2017-03-03T14:05:55,919][WARN ][logstash.filters.json ] Error parsing json {:source=>"message", :raw=>"}\r", :exception=>#<LogStash::Json::ParserError: Unexpected close marker '}': expected ']' (for ROOT starting at [Source: [B@6f43f4e1; line: 1, column: 0])
at [Source: [B@6f43f4e1; line: 1, column: 2]>}
{
"@timestamp" => 2017-03-03T14:05:55.466Z,
"@version" => "1",
"message" => "{\r",
"tags" => [
[0] "_jsonparsefailure"
]
}
{
"@timestamp" => 2017-03-03T14:05:55.466Z,
"@version" => "1",
"message" => "\t"records": \r",
"tags" => [
[0] "_jsonparsefailure"
]
}
{
"@timestamp" => 2017-03-03T14:05:55.466Z,
"@version" => "1",
"message" => "\t[\r",
"tags" => [
[0] "_jsonparsefailure"
]
}
{
"@timestamp" => 2017-03-03T14:05:55.466Z,
"@version" => "1",
"message" => "\t\t{\r",
"tags" => [
[0] "_jsonparsefailure"
]
}

8

It looks like your JSON documents pan multiple lines, so unless you can change the format of the files and get one JSOn object per line you will probably need to use a multiline codec in order to assemble the JSON objects before parsing then with a json filter.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.