# Filebeat with multiple different index type

**URL:** <https://discuss.elastic.co/t/filebeat-with-multiple-different-index-type/178847>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [April 29, 2019, 6:02am UTC](https://discuss.elastic.co/t/filebeat-with-multiple-different-index-type/178847 "2019-04-29T06:02:08Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![sahinguler](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sahinguler/32/99832_2.png) [@sahinguler](https://discuss.elastic.co/u/sahinguler)\
**Post date:** [April 29, 2019, 6:02am UTC](https://discuss.elastic.co/t/filebeat-with-multiple-different-index-type/178847/1 "2019-04-29T06:02:08Z")

</div>

Hi all.

I want to use filebeat with different input in a single yml file. Such as input type docker and input type log in same file. Does anybody help?

My config yml for filebeat as following:

# Autodiscover allows you to detect changes in the system and spawn new modules or inputs as they happen.

```
filebeat.autodiscover:
# Autodiscover docker containers and parse logs
  providers:
    - type: docker
      templates:
          config:
            - type: docker
              containers.ids:
                - "${data.docker.container.id}"
                
filebeat.inputs:
#------------------------------ Docker input --------------------------------
- type: docker
  enabled: true
  encoding: "utf-8"
  containers.ids:
    - "*"
  paths:
    - /var/lib/docker/containers/${data.docker.container.id}/*.log
  # json.message_key: log
  # json.add_error_key: true
  # json.keys_under_root: true
  exclude_lines: ["^\\s+[\\-`('.|_]"] # drop asciiart lines
  processors:
    - add_host_metadata: ~
    - add_docker_metadata: ~  
#------------------------------ File input ----------------------------------
- type: log
  enabled: true
  paths:
    - /tests/test/*/*/worker.log
  fields: {log_type: test-logs}
#=========================== Filebeat inputs ==============================
output.logstash:
  hosts: ["localhost:5044"]
```

---

<div class="post-metadata">

**Author:** ![sahinguler](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sahinguler/32/99832_2.png) [@sahinguler](https://discuss.elastic.co/u/sahinguler)\
**Post date:** [April 29, 2019, 6:14am UTC](https://discuss.elastic.co/t/filebeat-with-multiple-different-index-type/178847/2 "2019-04-29T06:14:42Z")

</div>

My logstash config file as following :

```
input {
    beats {
        port => 5044
        codec => plain {
            charset => "UTF-8" 
        }
    }
}
filter{
    if [fields][log_type] == "test-log"
    {
        grok{ 
                match => {
                        "message" => "\[%{TIMESTAMP_ISO8601:timestamp}\] %{DATA:logLevel} %{DATA:class} %{GREEDYDATA:message}" 
                    }
                }
    }else{
        if[docker][container][name]=~"k8s_xxxxxxxx-node" or 
        [docker][container][name]=~"k8s_yyyyyyyy-node" or
        [fields][log_type] == "test-log"
        {
            grok{ 
                    match => {
                        "message" => "\[%{TIMESTAMP_ISO8601:timestamp}\] %{DATA:logLevel} %{DATA:class} %{GREEDYDATA:message}" 
                    }
                }
        }
    }
} 
output {
    if [fields][log_type] == "test-log"{
        elasticsearch {
            hosts => "localhost:9200"
                manage_template => false
            index => "test-logs"
        }
    }else{
        if[docker][container][name]=~"k8s_xxxxxxxx-node"{
            elasticsearch {
                hosts => "localhost:9200"
                    manage_template => false
                index => "xxxxxxxx-service"
            }
        }else if[docker][container][name]=~"k8s_yyyyyyyy-node"{
            elasticsearch {
                hosts => "localhost:9200"
                    manage_template => false
                index => "yyyyyyyy-service"
          }
        }
    }     
}
```

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [April 29, 2019, 9:18am UTC](https://discuss.elastic.co/t/filebeat-with-multiple-different-index-type/178847/3 "2019-04-29T09:18:27Z")

</div>

Filebeat cannot read from the same file from different inputs, as it leads to unexpected behaviour.  
What is that you are trying to achieve? Which files do you want to read and how?

---

<div class="post-metadata">

**Author:** ![sahinguler](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sahinguler/32/99832_2.png) [@sahinguler](https://discuss.elastic.co/u/sahinguler)\
**Post date:** [April 29, 2019, 9:45am UTC](https://discuss.elastic.co/t/filebeat-with-multiple-different-index-type/178847/4 "2019-04-29T09:45:51Z")

</div>

Thanks for reply.

In our scenario, we have docker logs that we should store. On the other hands, we have some \*.log file also for archiving. We want to save these logs by using a single filebeat config instance.

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [April 30, 2019, 1:27pm UTC](https://discuss.elastic.co/t/filebeat-with-multiple-different-index-type/178847/5 "2019-04-30T13:27:14Z")

</div>

In this case you need to configure `autodiscovery` and one `log` input. The second, `docker` input is not needed as it monitors the same files as it is configured in `autodiscovery`.

```auto
filebeat.autodiscover:
# Autodiscover docker containers and parse logs
  providers:
    - type: docker
      templates:
          config:
            - type: docker
              containers.ids:
                - "${data.docker.container.id}"
              exclude_lines: ["^\\s+[\\-`('.|_]"] # drop asciiart lines
              processors:
              - add_host_metadata: ~
              - add_docker_metadata: ~  
                
filebeat.inputs:
#------------------------------ File input ----------------------------------
- type: log
  enabled: true
  paths:
    - /tests/test/*/*/worker.log
  fields: {log_type: test-logs}
#=========================== Filebeat inputs ==============================
output.logstash:
  hosts: ["localhost:5044"]

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 28, 2019, 1:29pm UTC](https://discuss.elastic.co/t/filebeat-with-multiple-different-index-type/178847/6 "2019-05-28T13:29:10Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.