Hi guys,
During filebeat 5.x, I built a very tiny weight docker image for filebeat with alpine.
But for filebeat 6.x later, seems not support running based OS alpine. Anyone advise? Thanks.
With 6.0 cgo is enabled in the filebeat build. Filebeat 5.0 was fully statically linked. That is, 6.0 might dynamically link against libc I think. You will need to install missing dependencies in your image, or build filebeat with alpine + go.
What's the error message you're seeing?
Always pop up below exception said "no such file or directory".
main.main()
/go/src/github.com/opencontainers/runc/main.go:137 +0xe24
panic: standard_init_linux.go:175: exec user process caused "no such file or directory" [recovered]
panic: standard_init_linux.go:175: exec user process caused "no such file or directory"
goroutine 1 [running, locked to thread]:
panic(0x7e9de0, 0xc8201209a0)
/usr/local/go/src/runtime/panic.go:481 +0x3e6
github.com/urfave/cli.HandleAction.func1(0xc8200932f8)
/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/app.go:478 +0x38e
panic(0x7e9de0, 0xc8201209a0)
/usr/local/go/src/runtime/panic.go:443 +0x4e9
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization.func1(0xc820092c08, 0xc82001a0c8, 0xc820092d18)
/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:259 +0x136
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization(0xc820051630, 0x7ff29b96c6b0, 0xc8201209a0)
/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:277 +0x5b1
main.glob.func8(0xc82006ea00, 0x0, 0x0)
/go/src/github.com/opencontainers/runc/main_unix.go:26 +0x68
reflect.Value.call(0x74e5e0, 0x8ffda0, 0x13, 0x846308, 0x4, 0xc820093278, 0x1, 0x1, 0x0, 0x0, ...)
/usr/local/go/src/reflect/value.go:435 +0x120d
reflect.Value.Call(0x74e5e0, 0x8ffda0, 0x13, 0xc820093278, 0x1, 0x1, 0x0, 0x0, 0x0)
/usr/local/go/src/reflect/value.go:303 +0xb1
github.com/urfave/cli.HandleAction(0x74e5e0, 0x8ffda0, 0xc82006ea00, 0x0, 0x0)
/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/app.go:487 +0x2ee
github.com/urfave/cli.Command.Run(0x8491b8, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x8df0e0, 0x51, 0x0, ...)
/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/command.go:191 +0xfec
github.com/urfave/cli.(*App).Run(0xc820001500, 0xc82000a100, 0x2, 0x2, 0x0, 0x0)
/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/app.go:240 +0xaa4[spoiler]
[details="Summary"]
This text will be blurred
[/details]
[/spoiler]
main.main()
/go/src/github.com/opencontainers/runc/main.go:137 +0xe24
Here is my docker-compose.yml.
# From base OS image
FROM alpine:3.6
MAINTAINER Eason Lau <eason.lau02@hotmail.com>
ENV FILEBEAT_VERSION=6.1.1
COPY ./config/filebeat.yml /
RUN apk add --update-cache curl bash go && \
rm -rf /var/cache/apk/* && \
curl https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-${FILEBEAT_VERSION}-linux-x86_64.tar.gz -o /filebeat.tar.gz && \
tar xzvf filebeat.tar.gz && \
rm filebeat.tar.gz && \
mv filebeat-${FILEBEAT_VERSION}-linux-x86_64 filebeat && \
cd filebeat && \
cp filebeat /usr/bin && \
rm -rf /filebeat/filebeat.yml && \
cp /filebeat.yml /filebeat/ && \
ls -ltr /filebeat && \
cat /filebeat/filebeat.yml
VOLUME /filebeat/data
WORKDIR /filebeat/
CMD ["filebeat","-e","filebeat.yml"]
What I am sure are:
- Right working directory at container
- Try to run manually at container, also said "no such file or directory"
The error message is not by beats, but by docker itself. It can not find the filebeat binary. With workdir filebeat try to run with:
CMD ["./filebeat", "-e", "-c", "filebeat.yml"].
Using docker run with a custom command, you can startup your container and check everything is correctly in place.
No. Actually, all path/command is right. Just try your suggestion. not work.
I doubt that lack of lib. If any other advise. 
Once you have your image, run bash and try to run directly.
docker run -rm -i -t <image name> /bin/bash. This gives you a shell, running in a new container (will be deleted afterwards), based on your image. Now you can investigate filebeat setup being correct.
By the way, if I run filebeat directly inside container, it shows another error. like:
filebeat: cannot execute binary file
Other issue inside alpine container:
/filebeat # /bin/sh filebeat
filebeat: line 1: syntax error: unexpected "("
/filebeat # /bin/bash filebeat
filebeat: filebeat: cannot execute binary file
Hmm.... what do you get if you run which filebeat and file $(which filebeat). Why did you run /bin/sh filebeat?
For reference checkout the official docker images repo: https://github.com/elastic/beats-docker
I have added filebeat binary file into PATH so that which filebeat shows correct location.
If really could not support based on Linux Alpine for Filebeat 6.x? For 5.x, I can build image successfully based on Linux Alpine.
For your reference : https://hub.docker.com/r/eason02/filebeat-alpine/
Compared with Centos base, Linux alpine is more tiny which more helpful for fast auto-scale-out.
Looks like the correct libc version is missing.
File output:
bash-4.3# file /usr/bin/filebeat
/usr/bin/filebeat: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.18, BuildID[sha1]=c0f19e6605040a74953d9e08abf380b345b2be53, with debug_info, not stripped
for comparison:
bash-4.3# file /usr/bin/file
/usr/bin/file: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-x86_64.so.1, stripped
Checking for lib dependencies with ldd:
bash-4.3# ldd /usr/bin/filebeat
/lib64/ld-linux-x86-64.so.2 (0x7f947c01c000)
libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7f947c01c000)
libdl.so.2 => /lib64/ld-linux-x86-64.so.2 (0x7f947c01c000)
libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f947c01c000)
bash-4.3# ls /lib64/ld-linux-x86-64.so.2
ls: /lib64/ld-linux-x86-64.so.2: No such file or directory
Installing missing dependencies:
bash-4.3# apk add libc6-compat
(1/1) Installing libc6-compat (1.1.16-r14)
OK: 281 MiB in 36 packages
bash-4.3# ls /lib64/ld-linux-x86-64*
/lib64/ld-linux-x86-64.so.2
bash-4.3# /usr/bin/filebeat -h
Usage:
filebeat [flags]
filebeat [command]
Available Commands:
export Export current config or index template
help Help about any command
modules Manage configured modules
run Run filebeat
setup Setup index template, dashboards and ML jobs
test Test config
version Show current version info
Flags:
-E, --E setting=value Configuration overwrite
-M, --M setting=value Module configuration overwrite
-N, --N Disable actual publishing for testing
-c, --c string Configuration file, relative to path.config (default "filebeat.yml")
--cpuprofile string Write cpu profile to file
-d, --d string Enable certain debug selectors
-e, --e Log to stderr and disable syslog/file output
-h, --help help for filebeat
--httpprof string Start pprof http server
--memprofile string Write memory profile to this file
--modules string List of enabled modules (comma separated)
--once Run filebeat only once until all harvesters reach EOF
--path.config string Configuration path
--path.data string Data path
--path.home string Home path
--path.logs string Logs path
--plugin pluginList Load additional plugins
--setup Load the sample Kibana dashboards
--strict.perms Strict permission checking on config files (default true)
-v, --v Log at INFO level
Use "filebeat [command] --help" for more information about a command.
As I already mentioned, beats 6.x are build with cgo enabled by default. This way beats will be dynamically linked against gnu libc. But Alpine uses musl libc. The libc6-compat package provides a libc6 compatibility layer, based on musl. That is, I have no idea how stable beats with libc6-compat will run.
Beats 5.x did not use CGO -> statically compiled -> no libc required. That's why build a 5.x image did work well for you.
Compiling filebeat with an alpine image, or compile/link with cgo disabled, might give you some better results.
As you care about images sizes, when compilining filebeat with cgo disabled, you won't need alpine or any other linux distribution as base image. A scratch image with filebeat only will do. There are a many tutorials how to do this. From skimming the contents, this blog post seems fine. Uses a golang based container for building + builds a final 'scratch' image just containing the binary.
1 Like
That's a perfect research. Thanks @steffens . Just completed image based on alpine.
https://github.com/easonlau02/filebeat-alpine/tree/master/6.1.1
https://hub.docker.com/r/eason02/filebeat-alpine/
[user@elk 6.1.1]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6c3998d151f5 eason02/filebeat-alpine:6.1.1 "./filebeat -e -c fil" 2 seconds ago Up 1 seconds filebeat-alpine-6.1.1
[user@elk 6.1.1]$ docker logs -f 6c
2018/01/24 16:10:55.201148 beat.go:436: INFO Home path: [/filebeat] Config path: [/filebeat] Data path: [/filebeat/data] Logs path: [/filebeat/logs]
2018/01/24 16:10:55.201435 beat.go:443: INFO Beat UUID: 061ab011-2cce-4e36-89b5-e9d5182cf341
2018/01/24 16:10:55.201447 beat.go:203: INFO Setup Beat: filebeat; Version: 6.1.1
2018/01/24 16:10:55.201786 metrics.go:23: INFO Metrics logging every 30s
2018/01/24 16:10:55.201832 module.go:76: INFO Beat name: elk
2018/01/24 16:10:55.210044 beat.go:276: INFO filebeat start running.
2018/01/24 16:10:55.210091 registrar.go:71: INFO No registry file found under: /filebeat/data/registry. Creating a new registry file.
2018/01/24 16:10:55.229984 registrar.go:108: INFO Loading registrar data from /filebeat/data/registry
2018/01/24 16:10:55.230024 registrar.go:119: INFO States Loaded from registrar: 0
2018/01/24 16:10:55.230043 filebeat.go:261: WARN Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2018/01/24 16:10:55.230053 crawler.go:48: INFO Loading Prospectors: 2
2018/01/24 16:10:55.230316 prospector.go:87: INFO Starting prospector of type: log; ID: 17882058445014555430
2018/01/24 16:10:55.230482 prospector.go:87: INFO Starting prospector of type: log; ID: 9014056516912101426
2018/01/24 16:10:55.230490 crawler.go:82: INFO Loading and starting Prospectors completed. Enabled prospectors: 2
2018/01/24 16:10:55.230530 registrar.go:150: INFO Starting Registrar
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.