Updates indentation:
filebeat.prospectors:
- input_type: log
paths:
- /var/log/app-traces/trace-*.log
json.keys_under_root: true
fields:
document_type: doc
processors:
- rename:
fields:
- from: host
to: _host
processors:
- rename:
fields:
- from: _host
to: host
to add more context,
I try to migrate the old setup from filebeat-5.4.
but I'm ready to rewrite every filters/mappings if a proper solution apply.
Beats is moving towards Elastic Common Schems (ECS). The host.name field was introduced with Beats 6.4 I think, for ECS compatibility. Having a common schema will allow other products in and around the stack to provide additional functionality.
As you are in the process of changing things right now, I'd recommend to have a look at it as well.
The mapping comes from the template:
Right, this is your template, but the host field definition in it is not compatible with Beats its own event schema. Normally Beats uses fields.yml files for generating the template dynamically (with slight nuances per Elasticsearch version used), and the index patterns in Kibana.
Your mappings misses the beats namespace for example. In this case Elasticsearch will try to automatically derive a schema for missing fields.
Some more details here: Configure Elasticsearch index template loading | Filebeat Reference [8.11] | Elastic
If you comment out your setup.template settings, then you can run filebeat export template, so to print the default template filebeat would have installed.
using decode_json_files, I have numerous errors like:
2019-03-14T17:20:45.643Z ERROR jsontransform/jsonhelper.go:53 JSON: Won't overwrite @timestamp because of parsing error: parsing time "2019-03-14T17:20:37.590326+0000" as "2006-01-02T15:04:05Z07:00": cannot parse "+0000" as "Z07:00"
Oh, I see. The @timestamp in your JSON is not copmatible to the timestamp format in Beats 
Can you open a bug report for this on github?
an example of the data to parse from the log file:
Hm..., can host in your JSON document differ from the host filebeat runs on? If not then we don't need the initial rename or decode_json processors. Just migrate to host.name (remember, ECS), or use the global rename process to move host.name to host.