Ok thanks @Joe_Fleming, you were right on checking for the value, I found another thread explaining it
Now the issue is that my scripted fields have empty values, even though the doc fields are not empty
Scripted field 1: _msg painless String
if (doc.err_msg.value != null)
return doc.err_msg.value + '--' + doc.err_stack.value;
return '';
Scripted field 2: _req painless String
if (doc['req.path'].value != null)
return doc['req.method'].value +' '+ doc['req.path'].value +' '+ doc['res.status'].value;
return '';
^ You can see above doc['req.path'], doc['req.method'] are not empty
We would thinkdoc['req.path'].value != null is evaluated to false, but if I change the script to just:
return doc['req.method'].value +' '+ doc['req.path'].value +' '+ doc['res.status'].value;
it's still the same
note I took care to refresh the kibana filebeat- index after editing those fields
note: on the 5 of 15 shards failed error
the error with shards I has before, is probably due to err_stack being a Text, and not a Keyword
I had force those types preconfigured in my filebeat.yml config:
setup.template.append_fields:
- name: err_msg
type: keyword
- name: err_stack
type: text
But I removed this, it's not necessary as kibana will take string type by default, which works