Hello, I'm using JSON filter to parse JSON files, the field title contains characters [ and ]
I got this error:
exception=>#<RuntimeError: Invalid FieldReference: [case] index abc...
As the whole value of this field should be indexed in Elasticsearch, I can't remove [ and ]
The value is valid in JSON but is not parsable by logstash.
How to correct it?
I found a similar topic:
opened 04:58PM - 19 Feb 20 UTC
enhancement
discuss
int-shortlist
Many issues have been reported, mainly related to JSON decoding either in a code… c or filter, where a *valid* JSON document contains keys that starts with a `[` which is interpreted as a [logstash field reference](https://www.elastic.co/guide/en/logstash/7.4/field-references-deepdive.html) and results in an `LogStash::Json::ParserError: Invalid FieldReference` error.
To reproduce:
```sh
echo '{"[foo":"bar"}' | bin/logstash -e 'input{stdin{codec=>json_lines}} output{stdout{codec=>rubydebug}}'
...
[2020-02-19T11:46:58,786][WARN ][logstash.codecs.jsonlines][main][ee68f56b1186b09c0ebc08387e2d8df11ff00788d3a22c61eeda228a073bb104] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Invalid FieldReference: `[foo`>, :data=>"{\"[foo\":\"bar\"}"}
{
"@timestamp" => 2020-02-19T16:46:58.803Z,
"message" => "{\"[foo\":\"bar\"}",
"host" => "mbp15r",
"@version" => "1",
"tags" => [
[0] "_jsonparsefailure"
]
}
```
The problem we have is that the keys are in fact valid JSON but are not parsable by logstash and result in a bad user experience.
I believe we should offer some way to mitigate that, maybe by allowing the user to specify some replacement character for the brackets that denote a field reference? Open to suggestions.
This relates to the FieldReference strict mode introduced in #9543
WDYT?
I solved the problem by modifing my Logstash conf.
Before:
filter {
json {
source => "message"
add_field => {
new_title => "%{[title][html]}"
}
remove_field => ["message", "%{[title]}"]
}
}
After:
filter {
json {
source => "message"
add_field => {
new_title => "%{[title][html]}"
}
remove_field => ["message", "title"]
}
}
But I don't know why this modification correct the parse JSON error.
system
(system)
Closed
November 25, 2021, 9:55am
3
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.