Trying to get a filebeat running in docker-compose

Elastic Stack Beats
1

Hi All

I'm a bit unsure what I've done wrong.
I have a docker-compose that places dmarc logs in a folder.
I then want to have another docker image running with a filebeat, that pushes it into logstash.

But for some reason it doesn't work.
As far as I can see, I've done what the docs say, but apparently not :slight_smile:

The docker-compose lines looks like this:

  dmarcfilebeat:
    image: docker.elastic.co/beats/filebeat:6.5.1
    container_name: dmarcfilebeat
    volumes:
      - ./dmarc/dmarclogs:/usr/share/filebeat/data
      - ./dmarc/filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro
      - ./dmarc/logs:/logs

The filebeat config looks like this:

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.registry_file:  /tmp/filebeat_registry

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /usr/share/filebeat/data/
  json.keys_under_root: true
  json.add_error_key: true
  fields_under_root: true
  fields:
    source_type: json-logs
    logtype: dmarc

output.logstash:
      hosts: ["logstash:5000"]

logging.level: debug
logging.to_files: true
logging.files:
  path: /logs
  name: filebeat
  keepfiles: 7
  permissions: 0644

All I see in the docker-compose logs is 'Exit 1', and I don't see anything in the logs folder.
What did I do wrong?

1 Like
2

/usr/share/filebeat/data is where filebeat puts its own data files, for example the registry file. I would not set it as a path for an input. Also, do not configure a directory in the path, as Filebeat skips them. I suggest you use the following format: /path/to/your/logs/*.log when setting the options paths.

3

Ok, I've tried updating this a bit, still no luck.
I see the logfiles I want it to get in the folder,

So the docker-compose shows this:

  dmarcfilebeat:
    image: docker.elastic.co/beats/filebeat:6.5.1
    container_name: dmarcfilebeat
    volumes:
      - ./dmarc/dmarclogs:/usr/share/filebeat/dmarclogs
      - ./dmarc/filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro
      - ./dmarc/logs:/logs

In my head thigs gives me to folders and a file mapped to the image.

The config is then:

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.registry_file:  /tmp/filebeat_registry

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

filebeat.inputs:
    - type: log
      enabled: true
      paths:
        - /usr/share/filebeat/dmarclogs
      json.keys_under_root: true
      json.add_error_key: true
      fields_under_root: true
      fields:
        source_type: json-logs
        logtype: dmarc

    output.logstash:
          hosts: ["logstash:5000"]

    logging.level: debug
    logging.to_files: true
    logging.files:
      path: /logs
      name: filebeat
      keepfiles: 7
      permissions: 0644

The logfile shows:

dmarcfilebeat     | 2019-03-06T12:32:26.820Z    ERROR   instance/beat.go:800    Exiting: error in autodiscover provider settings: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
dmarcfilebeat     | Exiting: error in autodiscover provider settings: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
dmarcfilebeat exited with code 1

So I've done something wrong, but what? :slight_smile:

4

Ok, got it working, had to add the .log as you said, and also removed the .autodiscover

1 Like
5

Weehee, got some very nice results in a dashboard now, maps, graphs circles and lists :laughing:

Thankyou for your help in getting the data in to elasticseach.

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.