Hi, I wrote a Logstash pipeline to modify some fields of json input files. The keys introduction.text, introduction.html and body in json are optional, I don't want to create empty fields in json output files. How to add checks in my pipeline?
input {
file {
mode => "read"
path => "/documents/*"
start_position => "beginning"
sincedb_path => "/dev/null"
codec => multiline { pattern => "^Spalanzani" negate => true what => "previous" auto_flush_interval => 2 }
file_chunk_size => 409600
}
}
filter {
json {
source => "message"
id => "%{[message][id]}"
add_field => {
title => "%{[title][text]}"
date => "%{[editorial_date]}"
topic => "%{[topic][title]}"
text => "%{[introduction][text]} %{[body]}"
text_html => "%{[introduction][html]} %{[body]}"
}
}
...
As I couldn't write a valid pipeline with if condition, I tried gsub, but it didn't work. %{[introduction][text]} are in the output json file.
filter {
json {
source => "message"
id => "%{[message][id]}"
add_field => {
title => "%{[title][text]}"
date => "%{[editorial_date]}"
text => "%{[introduction][text]} %{[body]}"
text_html => "%{[introduction][html]} %{[body]}"
}
}
mutate {
gsub => [
"texte", "<.*?>", " ",
"texte", "(%{[introduction][text]}|%{[body]})", " ",
"texte_html", "(%{[introduction][html]}|%{[body]})", " ",
"date", "T.*$", ""
]
}
}
If I escape [ with \ in gsub:
"texte", "(%{\[introduction]\[text]}|%{\[body]})", " ",
I get error:
Exception caught while applying mutate filter {:exception=>"Invalid FieldReference: \\[introduction][\\[text]
](file://[text]%60)"}
here is a source json file:
{
"id": "1111",
"editorial_date": "2015-10-19T00:00:00Z",
"title": {
"text": "test title",
"html": "<h1>test title</h1>"
},
"body": "<video data-els-src-id=\"\"..."
}