Hi,
I'm trying to use filebeat to parse my docker logs, and dissect them to send to Elasticsearch.
My filebeat.yml looks like this
filebeat.prospectors:
- type: docker
document_type: docker
containers:
ids: "*"
path: '/var/lib/docker/containers'
combine_partial: true
processors:
- add_cloud_metadata: ~
- add_docker_metadata: ~
- dissect:
field: log
tokenizer: "[%{log_stream}] [%{log_level}] [%{log_thread}] [%{@timestamp}] - "%{log_message}"\n"
A sample log line from my docker logs looks like this
{"@timestamp":"2019-03-29T16:01:19.316Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.6.2"},"offset":409357,"log":{"file":{"path":"/var/lib/docker/containers/d2ada679e60d1f88be90689eaf8be3aaf062dd76b42fc8af951b9b5c52a9b7b6/d2ada679e60d1f88be90689eaf8be3aaf062dd76b42fc8af951b9b5c52a9b7b6-json.log"}},"stream":"stdout","prospector":{"type":"docker"},"input":{"type":"docker"},"meta":{"cloud":{"region":"us-east-1","availability_zone":"us-east-1c","instance_id":"i-0ddd0540e506d573e","machine_type":"t3.large","provider":"ec2"}},"message":"[logger1] [ERROR] [tomcat-http--48] [29/03/2019 12:01:19,315] - " (writeError) [some_data]Some 'error message. [somepackage/someclass(someObject.java:1354)]","source":"/var/lib/docker/containers/d2ada679e60d1f88be90689eaf8be3aaf062dd76b42fc8af951b9b5c52a9b7b6/d2ada679e60d1f88be90689eaf8be3aaf062dd76b42fc8af951b9b5c52a9b7b6-json.log","docker":{"container":{"id":"d2ada679e60d1f88be90689eaf8be3aaf062dd76b42fc8af951b9b5c52a9b7b6","labels":{"com":{"docker":{"compose":{"project":"master","service":"web","version":"1.22.0","config-hash":"f02330e2e031990958d061bd36dbc1e1b5f59ca5f356bfb76e4134010e277f96","container-number":"1","oneoff":"False"}}},"PROJECT":"web","PROJECT_NAME":"Web"},"image":"some-repo/web:master","name":"master_web_1"}},"beat":{"name":"filebeat","hostname":"filebeat","version":"6.6.2"},"host":{"name":"filebeat"}}
Nothing seems broken out, I would've thought I would've had some keys created in the JSON output to match the keys from dissect? Instead I get standard message key with everything in it.
Any help would be appricated. Thanks