# Docker logs to ES with FileBeat **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [December 26, 2016, 1:26pm UTC](https://discuss.elastic.co/t/docker-logs-to-es-with-filebeat/70006 "2016-12-26T13:26:39Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![gilcoh](https://avatars.discourse-cdn.com/v4/letter/g/7feea3/32.png) [@gilcoh](https://discuss.elastic.co/u/gilcoh)\ **Post date:** [December 26, 2016, 1:26pm UTC](https://discuss.elastic.co/t/docker-logs-to-es-with-filebeat/70006/1 "2016-12-26T13:26:39Z")
Hi, I need help to **parse** my docker logs to ES with FileBeat - without using logstash. The main issue is that my "log" line transfers to the ES as a string instead of being parsed. Versions: FileBeat - 5.1.1 ES - 2.3 My filebeat.yml configured as below: ```auto filebeat: prospectors: - paths: ["/tmp/**/*-json.log"] json.message_key: log json.keys_under_root: true json.add_error_key: true output: elasticsearch: hosts: ["ES_URL:PORT"] index: "docker-swarm" template.name: "docker-swarm" ``` Every new line in the docker json logs looks like that: ```auto {"log":"{\"name\":\"test\",\"hostname\":\"4e7c4d8ef9ce\",\"pid\":16,\"level\":30,\"msg\":\"got health request\",\"time\":\"2016-12-26T10:58:05.221Z\",\"src\":{\"file\":\"/usr/src/app/src/index.js\",\"line\":42,\"func\":\"health\"},\"v\":0}\n","stream":"stdout","time":"2016-12-26T10:58:05.222365772Z"} ``` The template that I uploaded to the ES is: ```auto { "template": "docker-swarm", "settings": {}, "mappings": { "docker-swarm": { "properties": { "name": { "index": "not_analyzed", "type": "string" }, "hostname": { "index": "not_analyzed", "type": "string" }, "msg": { "index": "not_analyzed", "type": "string" }, "time": { "type": "date", "format": "yyyy-MM-dd'T'HH:mm:ss.SSSZZ" } } } } } ``` What I'm getting in the ES is that the "log" key is a string and not parsed: ```auto { "took" : 14, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 1.0, "hits" : [ { "_index" : "docker-swarm", "_type" : "log", "_id" : "AVk7Su1ZT58pSSDcWEiy", "_score" : 1.0, "_source" : { "@timestamp" : "2016-12-26T13:21:20.309Z", "beat" : { "hostname" : "b22567b66c53", "name" : "b22567b66c53", "version" : "5.1.1" }, "input_type" : "log", **"log" : "{\"name\":\"test\",\"hostname\":\"4e7c4d8ef9ce\",\"pid\":16,\"level\":30,\"msg\":\"got health request\",\"time\":\"2016-12-26T10:58:05.221Z\",\"src\":{\"file\":\"/usr/src/app/src/index.js\",\"line\":42,\"func\":\"health\"},\"v\":0}",** "offset" : 327, "source" : "/tmp/log/gil-json.log", "stream" : "stdout", "time" : "2016-12-26T10:58:05.222365772Z", "type" : "log" } } ] } } ``` Thanks in advanced, Gil ---
**Author:** ![tudor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tudor/32/3753_2.png) [@tudor](https://discuss.elastic.co/u/tudor)\ **Post date:** [December 26, 2016, 2:46pm UTC](https://discuss.elastic.co/t/docker-logs-to-es-with-filebeat/70006/2 "2016-12-26T14:46:26Z")
You have several options for parsing the second level JSON: - Configure the [decode\_json\_fields](https://www.elastic.co/guide/en/beats/filebeat/5.1/decode-json-fields.html) processor in Filebeat - Use the [Ingest Node](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html) of Elasticsearch, which also has a JSON decoder processor - Use Logstash, which can do that and much more I suspect the easiest for you would be the first option. Let us know if you have issues with the processor (currently marked experimental). ---
**Author:** ![gilcoh](https://avatars.discourse-cdn.com/v4/letter/g/7feea3/32.png) [@gilcoh](https://discuss.elastic.co/u/gilcoh)\ **Post date:** [December 27, 2016, 12:04pm UTC](https://discuss.elastic.co/t/docker-logs-to-es-with-filebeat/70006/3 "2016-12-27T12:04:05Z")
Exactly what I've been looking for! Many thanks!!! ---
**Author:** ![gilcoh](https://avatars.discourse-cdn.com/v4/letter/g/7feea3/32.png) [@gilcoh](https://discuss.elastic.co/u/gilcoh)\ **Post date:** [December 28, 2016, 4:51pm UTC](https://discuss.elastic.co/t/docker-logs-to-es-with-filebeat/70006/4 "2016-12-28T16:51:41Z")
Another question related to this, maybe you can help me here also Let's say that my "log" key can be either an object (like in my first example) or a string. 1. How can I setup my filebeat.yml file to support both cases? 2. If I can't - how can I use the drop\_event or any other processor to ignore the {"log": "string"} lines Thanks! ---
**Author:** ![odedpr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/odedpr/32/14192_2.png) [@odedpr](https://discuss.elastic.co/u/odedpr)\ **Post date:** [December 28, 2016, 6:05pm UTC](https://discuss.elastic.co/t/docker-logs-to-es-with-filebeat/70006/5 "2016-12-28T18:05:09Z")
I got the problem. The application logs are being exported as objects ( with the 'log' as key, this is done by docker's [file](https://docs.docker.com/engine/admin/logging/overview/#/json-file-options) logging driver together with [bunyan](https://github.com/trentm/node-bunyan) as logging node module ), while other logs, that are not directly from my application are being exported as String. to add 2 more questions to Gil's questions, 1. Is there any way I could drop all events that have String as 'log' value ? 2. does the 'decode\_json\_fields' support the contains condition ? could you please share a code snippet. 10x ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 25, 2017, 6:05pm UTC](https://discuss.elastic.co/t/docker-logs-to-es-with-filebeat/70006/6 "2017-01-25T18:05:19Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.