I am evaluating the 6.0.0-beta2 release of Filebeat as I am interested in the docker metadata feature.
In my filebeat.yml I have added the following in my prospector:
processors:
add_docker_metadata:~
Filebeat is sending the output to a Redis instance which in turn makes it available for Kibana. However, I am not seeing any of the new docker metadata fields in the output.
I am running filebeat in a container and I am mounting both the docker.sock and /var/lib/docker/containers into it - I am seeing logs in Kibana but just without the new fields.
What user is filebeat running as? Most likely it's a permission issue and filebeat cannot read/write to the docker socket. One way to address the issue is to run filebeat as root inside the container (docker run -u root ...).
I created a filebeat:6.0.0-beta2 image from a Dockerfile based on the prima/filebeat:5.6.0 one (https://hub.docker.com/r/prima/filebeat/~/dockerfile/) - I updated the Dockerfile to pull the 6.0.0-beta2 Filebeat binary and amended the SHA check etc
Then I created another Dockerfile to reference my filebeat:6.0.0-beta2 image and in that I have the following:
FROM filebeat:6.0.0-beta2
COPY filebeat.yml /filebeat.yml
RUN chmod go-w /filebeat.yml
USER root
If I then attach to the running container I can see that filebeat is being run as root. Like I said, I can see data in Kibana but just not the new metadata fields.
Ok, if you were using the official Filebeat container images then by default Filebeat would not be running as root. This is why I made that suggestion before asking for logs.
Can you please check your Filebeat logs and share the config you are using.
####### Filebeat ####################################
filebeat:
# List of prospectors to fetch data.
prospectors:
-
type: log
paths:
- '/var/lib/docker/containers/*/*.log'
json.message_key: log
json.keys_under_root: true
processors:
add_docker_metadata: ~
fields:
sys: docker
env: dev
fields_under_root: true
ignore_older: 48h
close_older: 48h
multiline:
pattern: '^20|^[a-zA-Z]{3,4}'
negate: true
match: after
max_lines: 5000
tail_files: false
force_close_files: true
#registry_file: "/usr/share/filebeat/"
############################# Output ##########################################
# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.
output:
redis:
# Set the host and port where to find Redis.
hosts: *********
port: ****
# Optional index name. The default is filebeat and generates filebeat keys.
key: "v3docker"
# Optional Redis database number where the events are stored
# The default is 0.
db: 0
# Optional Redis database number where the topology is stored
# The default is 1. It must have a different value than db.
db_topology: 1
# Optional password to authenticate with. By default, no
# password is set.
password: "***********"
# Optional Redis initial connection timeout in seconds.
# The default is 5 seconds.
timeout: 5
# Optional interval for reconnecting to failed Redis connections.
# The default is 1 second.
reconnect_interval: 1
############################# Logging #########################################
# There are three options for the log ouput: syslog, file, stderr.
# Under Windos systems, the log files are per default sent to the file output,
# under all other system per default to syslog.
logging:
# Send all logging output to syslog. On Windows default is false, otherwise
# default is true.
#to_syslog: true
# Write all logging output to files. Beats automatically rotate files if rotateeveryb
# limit is reached.
to_files: true
# To enable logging to files, to_files option has to be set to true
files:
# The directory where the log files will written to.
path: "/"
# The name of the files where the logs are written to.
name: mybeat
# Configure log file size limit. If limit is reached, log file will be
# automatically rotated
rotateeverybytes: 10485760 # = 10MB
# Number of rotated log files to keep. Oldest files will be deleted first.
keepfiles: 5
# Enable debug output for selected components. To enable all selectors use ["*"]
# Other available selectors are beat, publish, service
# Multiple selectors can be chained.
#selectors: [ ]
# Sets log level. The default log level is error.
# Available log levels are: critical, error, warning, info, debug
level: error
Cannot see any issues in the logs - I also set the level to debug in the config and no problems but should I be looking for something specific?
Hi @stevejr, thank you for taking the time to test our beta and report back!
Recently we fixed a bug that may be related to your issue: https://github.com/elastic/beats/pull/5133, will be out soon with filebeat 6.0.0-RC1. Anyway we can do some tests to confirm that's the case, so we discard any other possible issue:
Please run filebeat with this debugging flag: -d 'docker'
Could you detail how you start the containers you are watching?
Could you try to run this and see if it get's correctly annotated?: docker run busybox echo hello world
Ok, I have downloaded rc1 and am not not seeing any logs being sent to my Redis output for new containers. Obviously something wrong with my setup but I can't find it.
One question, is the SHA1 file correct for RC1 (Linux 64bit) as when I try to build my Filebeat docker image I am getting a SHA mismatch. The SHA1 file contains value 89cb8a510ecf876d707b803cacef3bc6fd4f536f706d2ef7280d65413a7c9cc65cd0941d2c4eb85a5ba3c01a4718da99bb3c911357a59086819ddb750eb25e2a but when I do a sha1sum on the downloaded tar file I get dbdb7be9a45c47171dc9c06bf55f541a5dbb6d49
I think it's a sha512, hashes were changed to it recently, we will have to update link names
I'm wondering, could you try official images? From your Dockerfile at least I see at least one issue: fields.yml is not in place. In any case that should be in the logs as an error, perhaps you can share the log output for the first 1-2 minutes?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
