Hi there,
I'am trying to setup a demo scenario, to demonstrate how to collect Tomcat log data from different containers and send it to an elastic stack for log aggregation and log analysis.
I try to describe my setup, but as this is a complex scenario it becomes a long description (i already had to shorten it). Hope you don't mind and also that it covers information you need...
I'm using a windows 10 with docker desktop and WSL 2 as host . All docker-cli commands are executed in an Ubuntu Terminal (WSL).
I'm using https://github.com/deviantony/docker-elk as my elastic stack with permission checking configured off.
Then I build the following docker-compose to spin up to two Tomcat containers each with a war file deyployed. There is a mounted volume named logs where the tomcat containers write their log files. There is also a Filebeat container observing the files in the shared log volume. The external network docker-elk_elk comes from the already running elk-stack.
version: "3"
volumes:
logs:
services:
filebeat:
container_name: filebeat
# image: docker.elastic.co/beats/filebeat:8.8.2
image: myfilebeat:latest
user: root
volumes:
- ./filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro
- /var/lib/docker/containers:/var/lib/docker/containers:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
- logs:/app/logs:ro
command: ["-e", "--strict.perms=false"]
# command: ["setup", "--strict.perms=false"]
environment:
- output.elasticsearch.hosts=["elasticsearch:9200"]
# - setup.kibana.host=kibana:5601
restart: always
networks:
- docker-elk_elk
tomcat1:
depends_on:
- filebeat
environment:
- node=app1
image: resttest:latest
container_name: tomcat1
ports:
- "8090:8080"
volumes:
- logs:/usr/local/tomcat/logs
restart: always
networks:
- docker-elk_elk
tomcat2:
depends_on:
- filebeat
environment:
- node=app2
image: resttest:latest
container_name: tomcat2
ports:
- "8091:8080"
volumes:
- logs:/usr/local/tomcat/logs
restart: always
networks:
- docker-elk_elk
networks:
docker-elk_elk:
external: true
I extended the default Filebeat container with some tools for debugging purposes:
FROM docker.elastic.co/beats/filebeat:8.8.2
USER root:root
RUN apt-get update -y \
&& apt-get install -y --no-install-recommends \
netcat \
tzdata \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
ENV TZ=Europe/Berlin
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
USER filebeat:root
That is my filebeat.yml that is mounted into the Filebeat container when I run docker-compose up:
# ============================== Filebeat inputs ===============================
filebeat.inputs:
# filestream is an input for collecting log messages from files.
- type: filestream
# Unique ID among all inputs, an ID is required.
id: tomcat-json
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /app/logs/app1_resttestapp.*.log.jsonl
- /app/logs/app2_resttestapp.*.log.jsonl
- /app/logs/resttestapp.*.log.jsonl
parsers:
- ndjson:
keys_under_root: true
add_error_key: false
processors:
- timestamp:
field: time
target_field: '@timestamp'
layouts:
- '2006-01-02 15:04:05.999'
test:
- '2023-07-26 12:32:08.789'
- drop_fields:
fields:
- time
# ======================= Elasticsearch template setting =======================
setup.template.settings:
index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: false
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
# ================================== Outputs ===================================
#output.console:
# enabled: true
# pretty: true
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["elasticsearch:9200"]
indices:
- index: "filebeat-%{[agent.version]}-tomcat-%{+yyyy.MM.dd}"
#output.file:
# enabled: true
# path: /tmp
# filename: filebeat2.log
logging.level: debug
logging.to_stderr: true
logging.to_files: true
#logging.files:
# path: /var/log
# name: filebeat2.log
What is working and what not?
Running the Elastic Stack works fine. I already successfully imported my tomcat logfile data into elasticsearch by running Tomcat and filebeat locally on my windows host and only the elastic stack in containers.
Running Tomcat in containers is also successful. Logfiles are generated. The log volume contains logfiles and are updated by the application (war). The Filebeat container spins up and finds the logfiles as configured in filebeat.yml. Filebeat also reads the logfiles as you can see by checking /usr/share/filebeat/data/registry/filebeat :
{"op":"set","id":1}
{"k":"filestream::tomcat-json::native::124222-2112","v":{"cursor":null,"meta":{"source":"/app/logs/app1_resttestapp.2023-08-15.log.jsonl","identifier_name":"native"},"ttl":0,"updated":[281470681743360,18446744011573954816]}}
{"op":"set","id":2}
{"k":"filestream::tomcat-json::native::118228-2112","v":{"updated":[281470681743360,18446744011573954816],"cursor":null,"meta":{"source":"/app/logs/app1_resttestapp.2023-08-16.log.jsonl","identifier_name":"native"},"ttl":0}}
{"op":"set","id":3}
{"k":"filestream::tomcat-json::native::124223-2112","v":{"updated":[281470681743360,18446744011573954816],"cursor":null,"meta":{"source":"/app/logs/app2_resttestapp.2023-08-15.log.jsonl","identifier_name":"native"},"ttl":0}}
{"op":"set","id":4}
{"k":"filestream::tomcat-json::native::118229-2112","v":{"ttl":0,"updated":[281470681743360,18446744011573954816],"cursor":null,"meta":{"source":"/app/logs/app2_resttestapp.2023-08-16.log.jsonl","identifier_name":"native"}}}
{"op":"set","id":5}
{"k":"filestream::tomcat-json::native::118229-2112","v":{"updated":[281470681743360,18446744011573954816],"cursor":null,"meta":{"identifier_name":"native","source":"/app/logs/app2_resttestapp.2023-08-16.log.jsonl"},"ttl":0}}
{"op":"set","id":6}
{"k":"filestream::tomcat-json::native::124222-2112","v":{"ttl":0,"updated":[281470681743360,18446744011573954816],"cursor":null,"meta":{"source":"/app/logs/app1_resttestapp.2023-08-15.log.jsonl","identifier_name":"native"}}}
{"op":"set","id":7}
{"k":"filestream::tomcat-json::native::118228-2112","v":{"meta":{"source":"/app/logs/app1_resttestapp.2023-08-16.log.jsonl","identifier_name":"native"},"ttl":0,"updated":[281470681743360,18446744011573954816],"cursor":null}}
{"op":"set","id":8}
{"k":"filestream::tomcat-json::native::124223-2112","v":{"ttl":0,"updated":[281470681743360,18446744011573954816],"cursor":null,"meta":{"source":"/app/logs/app2_resttestapp.2023-08-15.log.jsonl","identifier_name":"native"}}}
{"op":"set","id":9}
{"k":"filestream::tomcat-json::native::118229-2112","v":{"meta":{"source":"/app/logs/app2_resttestapp.2023-08-16.log.jsonl","identifier_name":"native"},"ttl":0,"updated":[281470681743360,18446744011573954816],"cursor":null}}
{"op":"set","id":10}
{"k":"filestream::tomcat-json::native::124222-2112","v":{"ttl":0,"updated":[281470681743360,18446744011573954816],"cursor":null,"meta":{"source":"/app/logs/app1_resttestapp.2023-08-15.log.jsonl","identifier_name":"native"}}}
{"op":"set","id":11}
{"k":"filestream::tomcat-json::native::118228-2112","v":{"cursor":null,"meta":{"source":"/app/logs/app1_resttestapp.2023-08-16.log.jsonl","identifier_name":"native"},"ttl":0,"updated":[281470681743360,18446744011573954816]}}
{"op":"set","id":12}
{"k":"filestream::tomcat-json::native::124223-2112","v":{"cursor":null,"meta":{"source":"/app/logs/app2_resttestapp.2023-08-15.log.jsonl","identifier_name":"native"},"ttl":0,"updated":[281470681743360,18446744011573954816]}}
But there is none of this data transferred to elasticsearch! Here are 2 excerpts from my filebeat log:
There is this error I don't understand:
filebeat | {"log.level":"error","@timestamp":"2023-08-16T11:59:25.681Z","log.logger":"input.filestream","log.origin":{"file.name":"task/group.go","file.line":69},"message":"harvester:: error while connecting to output with pipeline: each processor must have exactly one action, but found 5 actions (target_field,layouts,test,timestamp,field)","service.name":"filebeat","id":"tomcat-json","ecs.version":"1.6.0"}
filebeat | {"log.level":"error","@timestamp":"2023-08-16T11:59:25.681Z","log.logger":"input.filestream","log.origin":{"file.name":"task/group.go","file.line":69},"message":"harvester:: error while connecting to output with pipeline: each processor must have exactly one action, but found 5 actions (target_field,layouts,test,timestamp,field)","service.name":"filebeat","id":"tomcat-json","ecs.version":"1.6.0"}
filebeat | {"log.level":"error","@timestamp":"2023-08-16T11:59:25.681Z","log.logger":"input.filestream","log.origin":{"file.name":"task/group.go","file.line":69},"message":"harvester:: error while connecting to output with pipeline: each processor must have exactly one action, but found 5 actions (timestamp,field,target_field,layouts,test)","service.name":"filebeat","id":"tomcat-json","ecs.version":"1.6.0"}
filebeat | {"log.level":"error","@timestamp":"2023-08-16T11:59:25.681Z","log.logger":"input.filestream","log.origin":{"file.name":"task/group.go","file.line":69},"message":"harvester:: error while connecting to output with pipeline: each processor must have exactly one action, but found 5 actions (layouts,test,timestamp,field,target_field)","service.name":"filebeat","id":"tomcat-json","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-16T10:44:10.226+0200","log.origin":{"file.name":"instance/beat.go","file.line":528},"message":"filebeat stopped.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-16T10:44:26.454+0200","log.origin":{"file.name":"instance/beat.go","file.line":779},"message":"Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.454+0200","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":870},"message":"Beat metadata path: /usr/share/filebeat/data/meta.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-16T10:44:26.454+0200","log.origin":{"file.name":"instance/beat.go","file.line":787},"message":"Beat ID: 536418a5-8fce-49eb-a242-fbf9b483275e","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.455+0200","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":117},"message":"Loading syscall filter","service.name":"filebeat","seccomp_filter":{"no_new_privs":true,"flag":"tsync","policy":{"default_action":"errno","syscalls":[{"names":["accept","accept4","access","arch_prctl","bind","brk","chmod","chown","clock_gettime","clock_nanosleep","clone","clone3","close","connect","dup","dup2","epoll_create","epoll_create1","epoll_ctl","epoll_pwait","epoll_wait","execve","exit","exit_group","fchdir","fchmod","fchmodat","fchown","fchownat","fcntl","fdatasync","flock","fstat","fstatfs","fsync","ftruncate","futex","getcwd","getdents","getdents64","geteuid","getgid","getpeername","getpid","getppid","getrandom","getrlimit","getrusage","getsockname","getsockopt","gettid","gettimeofday","getuid","inotify_add_watch","inotify_init1","inotify_rm_watch","ioctl","kill","listen","lseek","lstat","madvise","mincore","mkdirat","mmap","mprotect","munmap","nanosleep","newfstatat","open","openat","pipe","pipe2","poll","ppoll","pread64","pselect6","pwrite64","read","readlink","readlinkat","recvfrom","recvmmsg","recvmsg","rename","renameat","rseq","rt_sigaction","rt_sigprocmask","rt_sigreturn","sched_getaffinity","sched_yield","sendfile","sendmmsg","sendmsg","sendto","set_robust_list","setitimer","setsockopt","shutdown","sigaltstack","socket","splice","stat","statfs","sysinfo","tgkill","time","tkill","uname","unlink","unlinkat","wait4","waitid","write","writev"],"action":"allow"}]}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-16T10:44:26.456+0200","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":124},"message":"Syscall filter successfully installed","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-16T10:44:26.456+0200","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1299},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"/usr/share/filebeat","data":"/usr/share/filebeat/data","home":"/usr/share/filebeat","logs":"/usr/share/filebeat/logs"},"type":"filebeat","uuid":"536418a5-8fce-49eb-a242-fbf9b483275e"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-08-16T10:44:26.456+0200","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1308},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"92c6b2370e46e549acda91b396f665a7e51e249c","libbeat":"8.8.2","time":"2023-06-23T19:04:44.000Z","version":"8.8.2"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-08-16T10:44:26.456+0200","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1311},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":20,"version":"go1.19.10"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-08-16T10:44:26.456+0200","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1317},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2023-08-16T08:49:07+02:00","containerized":true,"name":"67cff593a29b","ip":["127.0.0.1","172.18.0.5"],"kernel_version":"5.15.90.1-microsoft-standard-WSL2","mac":["02:42:ac:12:00:05"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.6 LTS (Focal Fossa)","major":20,"minor":4,"patch":6,"codename":"focal"},"timezone":"CEST","timezone_offset_sec":7200},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-08-16T10:44:26.456+0200","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1346},"message":"Process info","service.name":"filebeat","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null},"cwd":"/usr/share/filebeat","exe":"/usr/share/filebeat/filebeat","name":"filebeat","pid":7,"ppid":1,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2023-08-16T10:44:25.990+0200"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-08-16T10:44:26.456+0200","log.origin":{"file.name":"instance/beat.go","file.line":330},"message":"Setup Beat: filebeat; Version: 8.8.2","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.457+0200","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":358},"message":"Initializing output plugins","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-16T10:44:26.459+0200","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: http://elasticsearch:9200","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.459+0200","log.logger":"publisher","log.origin":{"file.name":"pipeline/consumer.go","file.line":110},"message":"start pipeline event consumer","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.459+0200","log.logger":"publisher","log.origin":{"file.name":"pipeline/queue_reader.go","file.line":49},"message":"pipeline event consumer queue reader: start","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-16T10:44:26.459+0200","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":105},"message":"Beat name: 67cff593a29b","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-16T10:44:26.459+0200","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":120},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-16T10:44:26.459+0200","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":145},"message":"Starting metrics logging every 30s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-16T10:44:26.459+0200","log.origin":{"file.name":"instance/beat.go","file.line":516},"message":"filebeat start running.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.459+0200","log.logger":"test","log.origin":{"file.name":"registrar/migrate.go","file.line":287},"message":"isFile(/usr/share/filebeat/data/registry) -> false","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.459+0200","log.logger":"test","log.origin":{"file.name":"registrar/migrate.go","file.line":287},"message":"isFile() -> false","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.459+0200","log.logger":"test","log.origin":{"file.name":"registrar/migrate.go","file.line":280},"message":"isDir(/usr/share/filebeat/data/registry/filebeat) -> true","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.459+0200","log.logger":"test","log.origin":{"file.name":"registrar/migrate.go","file.line":287},"message":"isFile(/usr/share/filebeat/data/registry/filebeat/meta.json) -> true","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.459+0200","log.logger":"registrar","log.origin":{"file.name":"registrar/migrate.go","file.line":82},"message":"Registry type '1' found","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-16T10:44:26.460+0200","log.origin":{"file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=8","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-16T10:44:26.460+0200","log.origin":{"file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=8","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.460+0200","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform request:append","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.460+0200","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform request:delete","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.460+0200","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform request:set","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.460+0200","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform response:append","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.460+0200","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform response:delete","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.460+0200","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform response:set","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.460+0200","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform pagination:append","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.460+0200","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform pagination:delete","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.460+0200","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform pagination:set","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.460+0200","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":84},"message":"registering encoder 'application/json': returned error: <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.460+0200","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":87},"message":"registering encoder 'application/x-www-form-urlencoded': returned error: <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.460+0200","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":93},"message":"registering decoder 'application/json': returned error: <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.460+0200","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":96},"message":"registering decoder 'application/x-ndjson': returned error: <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.460+0200","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":99},"message":"registering decoder 'text/csv': returned error: <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.460+0200","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":102},"message":"registering decoder 'application/zip': returned error: <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-16T10:44:26.460+0200","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":109},"message":"States Loaded from registrar: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-16T10:44:26.460+0200","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":71},"message":"Loading Inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-16T10:44:26.460+0200","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":117},"message":"starting input, keys present on the config: [filebeat.inputs.0.enabled filebeat.inputs.0.id filebeat.inputs.0.parsers.0.ndjson.add_error_key filebeat.inputs.0.parsers.0.ndjson.keys_under_root filebeat.inputs.0.paths.0 filebeat.inputs.0.paths.1 filebeat.inputs.0.paths.2 filebeat.inputs.0.processors.0.field filebeat.inputs.0.processors.0.layouts.0 filebeat.inputs.0.processors.0.target_field filebeat.inputs.0.processors.0.test.0 filebeat.inputs.0.processors.1.fields.0 filebeat.inputs.0.type]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.461+0200","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":140},"message":"Starting Registrar","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.461+0200","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":286},"message":"recursive glob enabled","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-16T10:44:26.461+0200","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":148},"message":"Starting input (ID: 8616917715378447078)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-16T10:44:26.461+0200","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":106},"message":"Loading and starting Inputs completed. Enabled inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-16T10:44:26.461+0200","log.logger":"input.filestream","log.origin":{"file.name":"compat/compat.go","file.line":120},"message":"Input 'filestream' starting","service.name":"filebeat","id":"tomcat-json","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.461+0200","log.logger":"input.filestream","log.origin":{"file.name":"filestream/prospector.go","file.line":131},"message":"Starting prospector","service.name":"filebeat","id":"tomcat-json","prospector":"file_prospector","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.461+0200","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":139},"message":"Start next scan","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.461+0200","log.logger":"input.filestream","log.origin":{"file.name":"filestream/prospector.go","file.line":177},"message":"A new file /app/logs/app2_resttestapp.2023-08-16.log.jsonl has been found","service.name":"filebeat","id":"tomcat-json","prospector":"file_prospector","operation":"create","source_name":"native::118229-2112","os_id":"118229-2112","new_path":"/app/logs/app2_resttestapp.2023-08-16.log.jsonl","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.461+0200","log.logger":"input.filestream","log.origin":{"file.name":"input-logfile/harvester.go","file.line":138},"message":"Starting harvester for file","service.name":"filebeat","id":"tomcat-json","source_file":"filestream::tomcat-json::native::118229-2112","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.461+0200","log.logger":"input.filestream","log.origin":{"file.name":"filestream/prospector.go","file.line":177},"message":"A new file /app/logs/app1_resttestapp.2023-08-15.log.jsonl has been found","service.name":"filebeat","id":"tomcat-json","prospector":"file_prospector","operation":"create","source_name":"native::124222-2112","os_id":"124222-2112","new_path":"/app/logs/app1_resttestapp.2023-08-15.log.jsonl","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.461+0200","log.logger":"input.filestream","log.origin":{"file.name":"input-logfile/harvester.go","file.line":138},"message":"Starting harvester for file","service.name":"filebeat","id":"tomcat-json","source_file":"filestream::tomcat-json::native::124222-2112","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.461+0200","log.logger":"input.filestream","log.origin":{"file.name":"input-logfile/harvester.go","file.line":224},"message":"Stopped harvester for file","service.name":"filebeat","id":"tomcat-json","source_file":"filestream::tomcat-json::native::124222-2112","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-08-16T10:44:26.461+0200","log.logger":"input.filestream","log.origin":{"file.name":"task/group.go","file.line":69},"message":"harvester:: error while connecting to output with pipeline: each processor must have exactly one action, but found 5 actions (timestamp,field,target_field,layouts,test)","service.name":"filebeat","id":"tomcat-json","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.461+0200","log.logger":"input.filestream","log.origin":{"file.name":"filestream/prospector.go","file.line":177},"message":"A new file /app/logs/app1_resttestapp.2023-08-16.log.jsonl has been found","service.name":"filebeat","id":"tomcat-json","prospector":"file_prospector","operation":"create","source_name":"native::118228-2112","os_id":"118228-2112","new_path":"/app/logs/app1_resttestapp.2023-08-16.log.jsonl","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.461+0200","log.logger":"input.filestream","log.origin":{"file.name":"input-logfile/harvester.go","file.line":224},"message":"Stopped harvester for file","service.name":"filebeat","id":"tomcat-json","source_file":"filestream::tomcat-json::native::118229-2112","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-08-16T10:44:26.461+0200","log.logger":"input.filestream","log.origin":{"file.name":"task/group.go","file.line":69},"message":"harvester:: error while connecting to output with pipeline: each processor must have exactly one action, but found 5 actions (timestamp,field,target_field,layouts,test)","service.name":"filebeat","id":"tomcat-json","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.461+0200","log.logger":"input.filestream","log.origin":{"file.name":"input-logfile/harvester.go","file.line":138},"message":"Starting harvester for file","service.name":"filebeat","id":"tomcat-json","source_file":"filestream::tomcat-json::native::118228-2112","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.461+0200","log.logger":"input.filestream","log.origin":{"file.name":"filestream/prospector.go","file.line":177},"message":"A new file /app/logs/app2_resttestapp.2023-08-15.log.jsonl has been found","service.name":"filebeat","id":"tomcat-json","prospector":"file_prospector","operation":"create","source_name":"native::124223-2112","os_id":"124223-2112","new_path":"/app/logs/app2_resttestapp.2023-08-15.log.jsonl","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.461+0200","log.logger":"input.filestream","log.origin":{"file.name":"input-logfile/harvester.go","file.line":138},"message":"Starting harvester for file","service.name":"filebeat","id":"tomcat-json","source_file":"filestream::tomcat-json::native::124223-2112","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.461+0200","log.logger":"input.filestream","log.origin":{"file.name":"input-logfile/harvester.go","file.line":224},"message":"Stopped harvester for file","service.name":"filebeat","id":"tomcat-json","source_file":"filestream::tomcat-json::native::124223-2112","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.461+0200","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":213},"message":"Found 4 paths","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-08-16T10:44:26.461+0200","log.logger":"input.filestream","log.origin":{"file.name":"task/group.go","file.line":69},"message":"harvester:: error while connecting to output with pipeline: each processor must have exactly one action, but found 5 actions (target_field,layouts,test,timestamp,field)","service.name":"filebeat","id":"tomcat-json","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-08-16T10:44:26.461+0200","log.logger":"input.filestream","log.origin":{"file.name":"input-logfile/harvester.go","file.line":224},"message":"Stopped harvester for file","service.name":"filebeat","id":"tomcat-json","source_file":"filestream::tomcat-json::native::118228-2112","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-08-16T10:44:26.461+0200","log.logger":"input.filestream","log.origin":{"file.name":"task/group.go","file.line":69},"message":"harvester:: error while connecting to output with pipeline: each processor must have exactly one action, but found 5 actions (layouts,test,timestamp,field,target_field)","service.name":"filebeat","id":"tomcat-json","ecs.version":"1.6.0"}
I tried debugging by setting output to console or to file so I can check whether there is any output generated but nothing is written neither to console, nor to the file I specified.
I successfully check whether the tcp network connection could be established between the Filebeat container and the elasticsearch container:
# nc -zv elasticsearch 9200
Connection to elasticsearch 9200 port [tcp/*] succeeded!
# nc -zv logstash 5044
Connection to logstash 5044 port [tcp/*] succeeded!
# pwd
/usr/share/filebeat
I've tried so many things, but find no reason why filebeat does not send any data to elasticsearch or can't write any output to console or a file. I have no idea where to look next.
Any help for further debugging is appreciated.
Thanks in advance