Data stream in filebeat 8.1.0

Elastic Stack Beats
,
1

Hi I'm using filebeat in 8.1.0 on docker below is my filebeat.yml
But I'm able to reach data stream option how i should enable it?
The second thing how to under docker image force enable dedicated module for example apache or tomcat for input file?

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

# filestream is an input for collecting log messages from files.
- type: filestream
  id: my-filestream

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/localhost_access_log.txt
  fields:
    tomcat: true

#---------------------------- Apache Tomcat Module ----------------------------
#- module: tomcat
#  log:
#    enabled: true

    # Set which input to use between udp (default), tcp or file.
    # var.input: udp
    # var.syslog_host: localhost
    # var.syslog_port: 9501

    # Set paths for the log files when file input is used.
#    var.paths:
#     - /var/log/localhost_access_log.txt



filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: true

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

setup.template.name: "filebeat-tomcat"
setup.template.pattern: "filebeat-tomcat*"
setup.ilm.enabled: true
setup.template.settings:
  index.number_of_shards: 6


#setup.dashboards.enabled: true


setup.kibana.host: ""
setup.kibana.ssl.enabled: false
setup.kibana.ssl.certificate_authorities: ["/usr/share/elasticsearch/config/certificates2/ca.crt"]
setup.kibana.ssl.certificate: "/usr/share/elasticsearch/config/certificates2/private.key"
setup.kibana.ssl.key: "/usr/share/elasticsearch/config/certificates2/private.key"

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
#  - add_cloud_metadata: ~
  - add_docker_metadata: ~
#  - add_kubernetes_metadata: ~

logging.level: debug



output.elasticsearch:
  hosts: []
  username: ''
  password: ''
  index: "filebeat-_t"
2

Hi @INS ,

From you configuration - It seems you are using hints based autodiscovery:

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

note that in this case you should add labels to the docker containers you want get logs from: The hints system looks for hints in Kubernetes Pod annotations or Docker labels that have the prefix co.elastic.logs.
To enable some specific module you can add co.elastic.logs/module: apache label to docker labels.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.