Filebeat with multiple different index type

[sahinguler]

Hi all.

I want to use filebeat with different input in a single yml file. Such as input type docker and input type log in same file. Does anybody help?

My config yml for filebeat as following:

Autodiscover allows you to detect changes in the system and spawn new modules or inputs as they happen.

filebeat.autodiscover:
# Autodiscover docker containers and parse logs
  providers:
    - type: docker
      templates:
          config:
            - type: docker
              containers.ids:
                - "${data.docker.container.id}"
                
filebeat.inputs:
#------------------------------ Docker input --------------------------------
- type: docker
  enabled: true
  encoding: "utf-8"
  containers.ids:
    - "*"
  paths:
    - /var/lib/docker/containers/${data.docker.container.id}/*.log
  # json.message_key: log
  # json.add_error_key: true
  # json.keys_under_root: true
  exclude_lines: ["^\\s+[\\-`('.|_]"]  # drop asciiart lines
  processors:
    - add_host_metadata: ~
    - add_docker_metadata: ~  
#------------------------------ File input ----------------------------------
- type: log
  enabled: true
  paths:
    - /tests/test/*/*/worker.log
  fields: {log_type: test-logs}
#=========================== Filebeat inputs ==============================
output.logstash:
  hosts: ["localhost:5044"]

[sahinguler]

My logstash config file as following :

input {
    beats {
        port => 5044
        codec => plain {
            charset => "UTF-8" 
        }
    }
}
filter{
    if [fields][log_type] == "test-log"
    {
        grok{ 
                match => {
                        "message" => "\[%{TIMESTAMP_ISO8601:timestamp}\] %{DATA:logLevel} %{DATA:class} %{GREEDYDATA:message}" 
                    }
                }
    }else{
        if[docker][container][name]=~"k8s_xxxxxxxx-node" or 
        [docker][container][name]=~"k8s_yyyyyyyy-node" or
        [fields][log_type] == "test-log"
        {
            grok{ 
                    match => {
                        "message" => "\[%{TIMESTAMP_ISO8601:timestamp}\] %{DATA:logLevel} %{DATA:class} %{GREEDYDATA:message}" 
                    }
                }
        }
    }
} 
output {
    if [fields][log_type] == "test-log"{
        elasticsearch {
            hosts => "localhost:9200"
                manage_template => false
            index => "test-logs"
        }
    }else{
        if[docker][container][name]=~"k8s_xxxxxxxx-node"{
            elasticsearch {
                hosts => "localhost:9200"
                    manage_template => false
                index => "xxxxxxxx-service"
            }
        }else if[docker][container][name]=~"k8s_yyyyyyyy-node"{
            elasticsearch {
                hosts => "localhost:9200"
                    manage_template => false
                index => "yyyyyyyy-service"
          }
        }
    }     
}

[kvch (Noémi Ványi)]

Filebeat cannot read from the same file from different inputs, as it leads to unexpected behaviour.
What is that you are trying to achieve? Which files do you want to read and how?

[sahinguler]

Thanks for reply.

In our scenario, we have docker logs that we should store. On the other hands, we have some *.log file also for archiving. We want to save these logs by using a single filebeat config instance.

[kvch (Noémi Ványi)]

In this case you need to configure autodiscovery and one log input. The second, docker input is not needed as it monitors the same files as it is configured in autodiscovery.

filebeat.autodiscover:
# Autodiscover docker containers and parse logs
  providers:
    - type: docker
      templates:
          config:
            - type: docker
              containers.ids:
                - "${data.docker.container.id}"
              exclude_lines: ["^\\s+[\\-`('.|_]"]  # drop asciiart lines
              processors:
              - add_host_metadata: ~
              - add_docker_metadata: ~  
                
filebeat.inputs:
#------------------------------ File input ----------------------------------
- type: log
  enabled: true
  paths:
    - /tests/test/*/*/worker.log
  fields: {log_type: test-logs}
#=========================== Filebeat inputs ==============================
output.logstash:
  hosts: ["localhost:5044"]