# Autodiscover AND extract field defined in module

**URL:** <https://discuss.elastic.co/t/autodiscover-and-extract-field-defined-in-module/157422>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [November 19, 2018, 6:42pm UTC](https://discuss.elastic.co/t/autodiscover-and-extract-field-defined-in-module/157422 "2018-11-19T18:42:04Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kaj\_Noppen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaj_noppen/32/68731_2.png) [@Kaj\_Noppen](https://discuss.elastic.co/u/Kaj_Noppen)\
**Post date:** [November 19, 2018, 6:42pm UTC](https://discuss.elastic.co/t/autodiscover-and-extract-field-defined-in-module/157422/1 "2018-11-19T18:42:05Z")

</div>

Hi all,

I have been playing around with filebeat on my CentOS machine. I am trying to get filebeat on the host OS to send logs of my Apache)containers to Elasticsearch, whilst also extracting the fields by using the Apache2 module. For some reason I am not able to get this to work. I am able to extract the raw Docker json messages, or nothing at all.

So I'd like to achieve that this message is nicely split in to fields, as defined in Apache2 module.

> "message": "{"log":"1.1.1.1 - - [19/Nov/2018:16:00:38 +0000] \"GET /somepage HTTP/1.1\" 200 427 \"[https://URL.php](https://URL.php)\" \"Mozilla/5.0 (X11; Ubuntu; Linux x86\_64; rv:63.0) Gecko/20100101 Firefox/63.0\"\n","stream":"stdout","time":"2018-11-19T16:00:38.6229692Z"}",

I have tried various configs, but this seems to be the one I distilled from other topics:

```
filebeat.autodiscover:
  providers:
   - type: docker
     templates:
       - condition.contains:
           docker.container.image: website
         config:
          - module: apache2
            error:
              paths:
                - "/docker/containers/${data.docker.container.id}/*.log"

```

I am running filebeat 6.5.0 with docker 18.06.1-ce. Docker is running under /docker/, not in /var/lib/docker. Filebeat is not running in a container itself.

---

<div class="post-metadata">

**Author:** ![Kaj\_Noppen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaj_noppen/32/68731_2.png) [@Kaj\_Noppen](https://discuss.elastic.co/u/Kaj_Noppen)\
**Post date:** [November 19, 2018, 8:26pm UTC](https://discuss.elastic.co/t/autodiscover-and-extract-field-defined-in-module/157422/2 "2018-11-19T20:26:21Z")

</div>

Also tried an alternative setup:

> filebeat.autodiscover:  
> providers:  
> - type: docker  
> templates:  
> - condition.contains:  
> docker.container.image: website  
> config:  
> - module: apache2  
> error:  
> type: docker  
> paths:  
> - /docker/containers/${data.docker.container.id}/\*.log  
> containers.ids:  
> - ${data.kubernetes.container.id}

gives the error:

> ERROR [autodiscover] cfgfile/list.go:96 Error creating runner from config: No paths were defined for input accessing config

---

<div class="post-metadata">

**Author:** ![Kaj\_Noppen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaj_noppen/32/68731_2.png) [@Kaj\_Noppen](https://discuss.elastic.co/u/Kaj_Noppen)\
**Post date:** [November 20, 2018, 9:15pm UTC](https://discuss.elastic.co/t/autodiscover-and-extract-field-defined-in-module/157422/3 "2018-11-20T21:15:23Z")

</div>

In the end found another blog post which had a working config:

```
filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains:
              docker.container.image: website
          config:
            - module: apache2
              access:
                enabled: true
                input:
                  type: docker
                  containers.ids:
                    - "${data.docker.container.id}"
              error:
                enabled: true
                var.hosts: ["${data.host}:${data.port}"]

```

Also, I needed to refresh my mappings in Kibana. This because the Apache fields had not been loaded in there yet.. (doh!)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 18, 2018, 9:15pm UTC](https://discuss.elastic.co/t/autodiscover-and-extract-field-defined-in-module/157422/4 "2018-12-18T21:15:31Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.