Filebeat 6.x could not support running under OS alpine


(Eason Lau) #1

Hi guys,

During filebeat 5.x, I built a very tiny weight docker image for filebeat with alpine.

But for filebeat 6.x later, seems not support running based OS alpine. Anyone advise? Thanks.


(Steffen Siering) #2

With 6.0 cgo is enabled in the filebeat build. Filebeat 5.0 was fully statically linked. That is, 6.0 might dynamically link against libc I think. You will need to install missing dependencies in your image, or build filebeat with alpine + go.

What's the error message you're seeing?


(Eason Lau) #3

@steffens

Always pop up below exception said "no such file or directory".

main.main()
        /go/src/github.com/opencontainers/runc/main.go:137 +0xe24
panic: standard_init_linux.go:175: exec user process caused "no such file or directory" [recovered]
        panic: standard_init_linux.go:175: exec user process caused "no such file or directory"

goroutine 1 [running, locked to thread]:
panic(0x7e9de0, 0xc8201209a0)
        /usr/local/go/src/runtime/panic.go:481 +0x3e6
github.com/urfave/cli.HandleAction.func1(0xc8200932f8)
        /go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/app.go:478 +0x38e
panic(0x7e9de0, 0xc8201209a0)
        /usr/local/go/src/runtime/panic.go:443 +0x4e9
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization.func1(0xc820092c08, 0xc82001a0c8, 0xc820092d18)
        /go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:259 +0x136
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization(0xc820051630, 0x7ff29b96c6b0, 0xc8201209a0)
        /go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:277 +0x5b1
main.glob.func8(0xc82006ea00, 0x0, 0x0)
        /go/src/github.com/opencontainers/runc/main_unix.go:26 +0x68
reflect.Value.call(0x74e5e0, 0x8ffda0, 0x13, 0x846308, 0x4, 0xc820093278, 0x1, 0x1, 0x0, 0x0, ...)
        /usr/local/go/src/reflect/value.go:435 +0x120d
reflect.Value.Call(0x74e5e0, 0x8ffda0, 0x13, 0xc820093278, 0x1, 0x1, 0x0, 0x0, 0x0)
        /usr/local/go/src/reflect/value.go:303 +0xb1
github.com/urfave/cli.HandleAction(0x74e5e0, 0x8ffda0, 0xc82006ea00, 0x0, 0x0)
        /go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/app.go:487 +0x2ee
github.com/urfave/cli.Command.Run(0x8491b8, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x8df0e0, 0x51, 0x0, ...)
        /go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/command.go:191 +0xfec
github.com/urfave/cli.(*App).Run(0xc820001500, 0xc82000a100, 0x2, 0x2, 0x0, 0x0)
        /go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/app.go:240 +0xaa4[spoiler]
[details="Summary"]
This text will be blurred
[/details]
[/spoiler]
main.main()
        /go/src/github.com/opencontainers/runc/main.go:137 +0xe24

Here is my docker-compose.yml.

# From base OS image
FROM alpine:3.6

MAINTAINER Eason Lau <eason.lau02@hotmail.com>

ENV FILEBEAT_VERSION=6.1.1

COPY ./config/filebeat.yml /

RUN apk add --update-cache curl bash go && \
    rm -rf /var/cache/apk/* && \
    curl https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-${FILEBEAT_VERSION}-linux-x86_64.tar.gz -o /filebeat.tar.gz && \
    tar xzvf filebeat.tar.gz && \
    rm filebeat.tar.gz && \
    mv filebeat-${FILEBEAT_VERSION}-linux-x86_64 filebeat && \
    cd filebeat && \
    cp filebeat /usr/bin && \
    rm -rf /filebeat/filebeat.yml && \
    cp /filebeat.yml /filebeat/ && \
    ls -ltr /filebeat && \
    cat /filebeat/filebeat.yml

VOLUME /filebeat/data

WORKDIR /filebeat/
CMD ["filebeat","-e","filebeat.yml"]

What I am sure are:

  1. Right working directory at container
  2. Try to run manually at container, also said "no such file or directory"

(Steffen Siering) #4

The error message is not by beats, but by docker itself. It can not find the filebeat binary. With workdir filebeat try to run with:

CMD ["./filebeat", "-e", "-c", "filebeat.yml"].

Using docker run with a custom command, you can startup your container and check everything is correctly in place.


(Eason Lau) #5

No. Actually, all path/command is right. Just try your suggestion. not work.
I doubt that lack of lib. If any other advise. :joy:


(Steffen Siering) #6

Once you have your image, run bash and try to run directly.

docker run -rm -i -t <image name> /bin/bash. This gives you a shell, running in a new container (will be deleted afterwards), based on your image. Now you can investigate filebeat setup being correct.


(Eason Lau) #7

By the way, if I run filebeat directly inside container, it shows another error. like:

filebeat: cannot execute binary file

Other issue inside alpine container:

/filebeat # /bin/sh filebeat
filebeat: line 1: syntax error: unexpected "("
/filebeat # /bin/bash filebeat
filebeat: filebeat: cannot execute binary file


(Steffen Siering) #8

Hmm.... what do you get if you run which filebeat and file $(which filebeat). Why did you run /bin/sh filebeat?


(Steffen Siering) #9

For reference checkout the official docker images repo: https://github.com/elastic/beats-docker


(Eason Lau) #10

I have added filebeat binary file into PATH so that which filebeat shows correct location.

If really could not support based on Linux Alpine for Filebeat 6.x? For 5.x, I can build image successfully based on Linux Alpine.

For your reference : https://hub.docker.com/r/eason02/filebeat-alpine/

Compared with Centos base, Linux alpine is more tiny which more helpful for fast auto-scale-out.


(Steffen Siering) #11

Looks like the correct libc version is missing.

File output:

bash-4.3# file /usr/bin/filebeat
/usr/bin/filebeat: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.18, BuildID[sha1]=c0f19e6605040a74953d9e08abf380b345b2be53, with debug_info, not stripped

for comparison:

bash-4.3# file /usr/bin/file
/usr/bin/file: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-x86_64.so.1, stripped

Checking for lib dependencies with ldd:

bash-4.3# ldd /usr/bin/filebeat
	/lib64/ld-linux-x86-64.so.2 (0x7f947c01c000)
	libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7f947c01c000)
	libdl.so.2 => /lib64/ld-linux-x86-64.so.2 (0x7f947c01c000)
	libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f947c01c000)
bash-4.3# ls /lib64/ld-linux-x86-64.so.2
ls: /lib64/ld-linux-x86-64.so.2: No such file or directory

Installing missing dependencies:

bash-4.3# apk add libc6-compat
(1/1) Installing libc6-compat (1.1.16-r14)
OK: 281 MiB in 36 packages
bash-4.3# ls /lib64/ld-linux-x86-64*
/lib64/ld-linux-x86-64.so.2
bash-4.3# /usr/bin/filebeat -h
Usage:
  filebeat [flags]
  filebeat [command]

Available Commands:
  export      Export current config or index template
  help        Help about any command
  modules     Manage configured modules
  run         Run filebeat
  setup       Setup index template, dashboards and ML jobs
  test        Test config
  version     Show current version info

Flags:
  -E, --E setting=value      Configuration overwrite
  -M, --M setting=value      Module configuration overwrite
  -N, --N                    Disable actual publishing for testing
  -c, --c string             Configuration file, relative to path.config (default "filebeat.yml")
      --cpuprofile string    Write cpu profile to file
  -d, --d string             Enable certain debug selectors
  -e, --e                    Log to stderr and disable syslog/file output
  -h, --help                 help for filebeat
      --httpprof string      Start pprof http server
      --memprofile string    Write memory profile to this file
      --modules string       List of enabled modules (comma separated)
      --once                 Run filebeat only once until all harvesters reach EOF
      --path.config string   Configuration path
      --path.data string     Data path
      --path.home string     Home path
      --path.logs string     Logs path
      --plugin pluginList    Load additional plugins
      --setup                Load the sample Kibana dashboards
      --strict.perms         Strict permission checking on config files (default true)
  -v, --v                    Log at INFO level

Use "filebeat [command] --help" for more information about a command.

As I already mentioned, beats 6.x are build with cgo enabled by default. This way beats will be dynamically linked against gnu libc. But Alpine uses musl libc. The libc6-compat package provides a libc6 compatibility layer, based on musl. That is, I have no idea how stable beats with libc6-compat will run.

Beats 5.x did not use CGO -> statically compiled -> no libc required. That's why build a 5.x image did work well for you.

Compiling filebeat with an alpine image, or compile/link with cgo disabled, might give you some better results.

As you care about images sizes, when compilining filebeat with cgo disabled, you won't need alpine or any other linux distribution as base image. A scratch image with filebeat only will do. There are a many tutorials how to do this. From skimming the contents, this blog post seems fine. Uses a golang based container for building + builds a final 'scratch' image just containing the binary.


(Eason Lau) #12

That's a perfect research. Thanks @steffens . Just completed image based on alpine.
https://github.com/easonlau02/filebeat-alpine/tree/master/6.1.1
https://hub.docker.com/r/eason02/filebeat-alpine/
:grinning:

[user@elk 6.1.1]$ docker ps
CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS              PORTS               NAMES
6c3998d151f5        eason02/filebeat-alpine:6.1.1   "./filebeat -e -c fil"   2 seconds ago       Up 1 seconds                            filebeat-alpine-6.1.1
[user@elk 6.1.1]$ docker logs -f 6c
2018/01/24 16:10:55.201148 beat.go:436: INFO Home path: [/filebeat] Config path: [/filebeat] Data path: [/filebeat/data] Logs path: [/filebeat/logs]
2018/01/24 16:10:55.201435 beat.go:443: INFO Beat UUID: 061ab011-2cce-4e36-89b5-e9d5182cf341
2018/01/24 16:10:55.201447 beat.go:203: INFO Setup Beat: filebeat; Version: 6.1.1
2018/01/24 16:10:55.201786 metrics.go:23: INFO Metrics logging every 30s
2018/01/24 16:10:55.201832 module.go:76: INFO Beat name: elk
2018/01/24 16:10:55.210044 beat.go:276: INFO filebeat start running.
2018/01/24 16:10:55.210091 registrar.go:71: INFO No registry file found under: /filebeat/data/registry. Creating a new registry file.
2018/01/24 16:10:55.229984 registrar.go:108: INFO Loading registrar data from /filebeat/data/registry
2018/01/24 16:10:55.230024 registrar.go:119: INFO States Loaded from registrar: 0
2018/01/24 16:10:55.230043 filebeat.go:261: WARN Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2018/01/24 16:10:55.230053 crawler.go:48: INFO Loading Prospectors: 2
2018/01/24 16:10:55.230316 prospector.go:87: INFO Starting prospector of type: log; ID: 17882058445014555430
2018/01/24 16:10:55.230482 prospector.go:87: INFO Starting prospector of type: log; ID: 9014056516912101426
2018/01/24 16:10:55.230490 crawler.go:82: INFO Loading and starting Prospectors completed. Enabled prospectors: 2
2018/01/24 16:10:55.230530 registrar.go:150: INFO Starting Registrar


(system) #13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.