Starting ELK-stack from docker-compose but nothing happen

zebulyon · February 27, 2020, 11:34am

Hello beginner with the ELK-stack here and apologize in advance regarding the long post ,

So I'm using a docker-compose file to start the entire elastic stack from the official docker images ver 7.6.0. Seemingly all of the containers starts up correctly but after that nothing happens, I have spent a lot of time going through the logs they generate but am so far unable to find the problem.
What I mean when I say nothing happens is that I can not see any new indices appearing in KIbana nor is logstash reacting to the input.

Here we can see that all containers has been able to start up and continue running:

version: '3'
services:
  elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.6.0
container_name: elasticsearch
environment:
  - cluster.name=elastic-cluster
  - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
  - "ES_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
  - "discovery.type=single-node"
ulimits:
  nofile:
    soft: 65536
    hard: 65536
  memlock:
    soft: -1
    hard: -1
volumes:
  - elastic-data:/usr/share/elasticsearch/data
ports:
  - 9200:9200
  - 9600:9600 # required for Performance Analyzer
expose:
  - "9200"
networks:
  - elastic-net


  kibana:
depends_on:      
  - "elasticsearch"
image: docker.elastic.co/kibana/kibana:7.6.0
container_name: kibana
ports:
  - 5601:5601
expose:
  - "5601"
environment:
  ELASTICSEARCH_URL: http://elasticsearch:9200
networks:
  - elastic-net


  filebeat:
depends_on:
  - "elasticsearch"
  - "logstash"
image: docker.elastic.co/beats/filebeat:7.6.0
container_name: filebeat
user: root
volumes:
  - "./filebeat_log.yml:/usr/share/filebeat/filebeat.yml"
  - "./pl_agent.log:/usr/share/filebeat/sample.log"
  - "/var/lib/docker/containers:/usr/share/filebeat/dockerlogs:ro"
  - "/var/run/docker.sock:/var/run/docker.sock"
networks:
  - elastic-net


  logstash:
depends_on:
  - "elasticsearch"
  - "kibana"
image: docker.elastic.co/logstash/logstash:7.6.0
container_name: logstash
ports:
  - 5044:5044
expose: 
  - "5044"
volumes:
- "./logstash3.conf:/usr/share/logstash/logstash.conf"
- "./logstash.yml:/usr/share/logstash/config/logstash.yml"
networks:
  - elastic-net


volumes:
  elastic-data:

networks:
  elastic-net:
driver: bridge

When starting up the process I use the command docker-compose up -d

My filebeat.yml file look like this:

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.inputs:
    type: log
    enabled: true
    paths: 
      - ./sample.log

processors:
- add_cloud_metadata: ~
- add_docker_metadata: ~

output.logstash:
  hosts: ["logstash:5044"]
  

setup.kibana:
  host: "http://kibana:5601"

setup.template.name: "filebeat"
setup.template.pattern: "filebeat-*"
setup.ilm.overwrite: true

Can see here that sample.log is in the same folder as filebeat.yml

While inside the filebeat container I have tested to ping both logstash and kibana and it is able to establish a connection between them.

Pinging Logstash

Pinging Kibana

When using the filebeat test config command I get an OK as the response and when using the filebeat test output command I get

logstash: logstash:5044...
  connection...
parse host... OK
dns lookup... OK
addresses: 172.27.0.4
dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK

On the logstash side I am able to reach the point of
[INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

Though I am a bit confused about why it says port=>9600 as I specify port 9200 in the config file

output {
	elasticsearch { hosts => ["http://elasticsearch:9200"] index => "logstash_pl_agent_log_testing_from_filebeat" }
	stdout{ codec => rubydebug }
}

Can also give how I take in input in the config file:

input {
	beats {
		port => "5044"
	}
}

My logstash.yml file look like this:

http.host: "0.0.0.0"
xpack.monitoring.elasticsearch.hosts: [ "http://elasticsearch:9200" ]
xpack.monitoring.enabled: true

Do not know if this is useful information but this is also repeatedly printed in my filebeat log:

2020-02-27T11:15:41.587Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":940,"time":{"ms":17}},"total":{"ticks":1490,"time":{"ms":30},"value":1490},"user":{"ticks":550,"time":{"ms":13}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"ff6a5e00-92d5-4186-89e5-67763fc191a9","uptime":{"ms":2310184}},"memstats":{"gc_next":9025584,"memory_alloc":4520200,"memory_total":35361736},"runtime":{"goroutines":29}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":1.31,"15":0.83,"5":0.91,"norm":{"1":0.3275,"15":0.2075,"5":0.2275}}}}}}

Can mention that when I do not use filebeat, and instead give logstash the input as stdin{} then everything seem to be functioning as intended which leads me to believe that filebeat is the failing link in the process.

ChrsMark · February 27, 2020, 12:56pm

Hi!

It seems that filebeat is not collecting anything. One thing you could do is to keep out Logstash for now and set output to console so as to see if Filebeat is able to collect logs.

See: https://www.elastic.co/guide/en/beats/filebeat/current/console-output.html

zebulyon · February 27, 2020, 3:47pm

You assumption seem to be correct.
Because when I change the input command to

filebeat.inputs:
- type: stdin

And output to

output.console:
  pretty: true

Then I am able to send a message to my self:

However whenever I try to make it read my log file, using this setup:
Version 1

filebeat.inputs:
- type: log
- enabled: true
- paths: \logs\*

I get this error message.

And when using this setup:
Version 2

filebeat.inputs:
  type: log
  enabled: true
  paths: \logs\*

Nothing happens:
2020-2-27-StartRun

After 1 min I shut it down

2020-2-27-endRun

So now I know that the problems lies in how it reads the input but I do not know how to fix it when using the command filebeat test config I get an Config OK message for all version of the .yml setup

ChrsMark · February 27, 2020, 4:45pm

I would say that the problem is that paths setting is not not valid. Could you try with sth like:

  paths:
    - "/var/log/apache2/*"

ref: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html

zebulyon · February 27, 2020, 6:26pm

Your solution sadly does not seem to do anything.

Do not know if it changes anything, but for the sake of quick problem searching I have created a new container of filebeat that is completely stand alone.
This is the start up command it uses the same image as the one in the docker-compose

docker run \
  --name=filebeat_test \
  --user=root \
  --rm \
  -it --entrypoint /bin/bash \
  --volume="$(pwd)/filebeat_bash.yml:/usr/share/filebeat/filebeat.yml:ro" \
  --volume="$(pwd)/pl_agent3.log:/usr/share/filebeat/var/log/apache2/sample.log:ro" \
  --volume="/var/lib/docker/containers:/var/lib/docker/containers:ro" \
  --volume="/var/run/docker.sock:/var/run/docker.sock:ro" \
  docker.elastic.co/beats/filebeat:7.6.0 \

And this is my current filebeat.yml file:

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: true

filebeat.inputs:
  type: log
  enabled: true
  paths:
    - "/var/log/apache2/*"
  
output.console:
  pretty: true

Here how it look in the container

Here we can see the path to the log file
2020-2-27-ShowLog

Here we can see that the YAML file was properly copied over to the container

Here we can see the list of modules enabled in the case this is of importance
2020-2-27-modules

Here we start the program and wait for 2 min

Here we stop the run of the program after 2min
2020-2-27-EndRun

The log file in question is a very small file containing only a single line of test.

ChrsMark · February 28, 2020, 9:07am

Could you on your tests start filebeat in debug mode please and share the logs? It is filebeat -e -d "*".

Thanks!

zebulyon · February 28, 2020, 9:35am

Of course here they are

2020-02-28T09:30:10.608Z	INFO	instance/beat.go:622	Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2020-02-28T09:30:10.608Z	DEBUG	[beat]	instance/beat.go:674	Beat metadata path: /usr/share/filebeat/data/meta.json
2020-02-28T09:30:10.608Z	INFO	instance/beat.go:630	Beat ID: 1f471e55-a35d-421c-8127-4777cddf69c9
2020-02-28T09:30:10.608Z	DEBUG	[seccomp]	seccomp/seccomp.go:117	Loading syscall filter	{"seccomp_filter": {"no_new_privs":true,"flag":"tsync","policy":{"default_action":"errno","syscalls":[{"names":["accept","accept4","access","arch_prctl","bind","brk","chmod","clock_gettime","clone","close","connect","dup","dup2","epoll_create","epoll_create1","epoll_ctl","epoll_pwait","epoll_wait","exit","exit_group","fchdir","fchmod","fchown","fcntl","fdatasync","flock","fstat","fstatfs","fsync","ftruncate","futex","getcwd","getdents","getdents64","geteuid","getgid","getpeername","getpid","getppid","getrandom","getrlimit","getrusage","getsockname","getsockopt","gettid","gettimeofday","getuid","inotify_add_watch","inotify_init1","inotify_rm_watch","ioctl","kill","listen","lseek","lstat","madvise","mincore","mkdirat","mmap","mprotect","munmap","nanosleep","newfstatat","open","openat","pipe","pipe2","poll","ppoll","pread64","pselect6","pwrite64","read","readlink","readlinkat","recvfrom","recvmmsg","recvmsg","rename","renameat","rt_sigaction","rt_sigprocmask","rt_sigreturn","sched_getaffinity","sched_yield","sendfile","sendmmsg","sendmsg","sendto","set_robust_list","setitimer","setsockopt","shutdown","sigaltstack","socket","splice","stat","statfs","sysinfo","tgkill","time","tkill","uname","unlink","unlinkat","wait4","waitid","write","writev"],"action":"allow"}]}}}
2020-02-28T09:30:10.609Z	INFO	[seccomp]	seccomp/seccomp.go:124	Syscall filter successfully installed
2020-02-28T09:30:10.609Z	INFO	[beat]	instance/beat.go:958	Beat info	{"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "1f471e55-a35d-421c-8127-4777cddf69c9"}}}
2020-02-28T09:30:10.609Z	INFO	[beat]	instance/beat.go:967	Build info	{"system_info": {"build": {"commit": "6a23e8f8f30f5001ba344e4e54d8d9cb82cb107c", "libbeat": "7.6.0", "time": "2020-02-05T23:06:45.000Z", "version": "7.6.0"}}}
2020-02-28T09:30:10.609Z	INFO	[beat]	instance/beat.go:970	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.13.7"}}}
2020-02-28T09:30:10.610Z	INFO	[beat]	instance/beat.go:974	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-02-24T06:56:55Z","containerized":true,"name":"abe48f1ef867","ip":["127.0.0.1/8","172.17.0.2/16"],"kernel_version":"4.19.0-8-amd64","mac":["02:42:ac:11:00:02"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":7,"patch":1908,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2020-02-28T09:30:10.610Z	INFO	[beat]	instance/beat.go:1003	Process info	{"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 26, "ppid": 1, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-02-28T09:30:09.590Z"}}}
2020-02-28T09:30:10.610Z	INFO	instance/beat.go:298	Setup Beat: filebeat; Version: 7.6.0
2020-02-28T09:30:10.611Z	DEBUG	[beat]	instance/beat.go:324	Initializing output plugins
2020-02-28T09:30:10.611Z	DEBUG	[publisher]	pipeline/consumer.go:137	start pipeline event consumer
2020-02-28T09:30:10.611Z	INFO	[publisher]	pipeline/module.go:110	Beat name: abe48f1ef867
2020-02-28T09:30:10.612Z	WARN	beater/filebeat.go:152	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-02-28T09:30:10.612Z	INFO	instance/beat.go:439	filebeat start running.
2020-02-28T09:30:10.612Z	DEBUG	[test]	registrar/migrate.go:159	isFile(/usr/share/filebeat/data/registry) -> false
2020-02-28T09:30:10.612Z	DEBUG	[test]	registrar/migrate.go:159	isFile() -> false
2020-02-28T09:30:10.612Z	DEBUG	[test]	registrar/migrate.go:152	isDir(/usr/share/filebeat/data/registry/filebeat) -> false
2020-02-28T09:30:10.612Z	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
2020-02-28T09:30:10.612Z	DEBUG	[registrar]	registrar/migrate.go:51	Registry type '' found
2020-02-28T09:30:10.612Z	DEBUG	[test]	registrar/migrate.go:159	isFile(.bak) -> false
2020-02-28T09:30:10.612Z	DEBUG	[test]	registrar/migrate.go:152	isDir(/usr/share/filebeat/data/registry/filebeat) -> false
2020-02-28T09:30:10.612Z	INFO	registrar/migrate.go:104	No registry home found. Create: /usr/share/filebeat/data/registry/filebeat
2020-02-28T09:30:10.613Z	DEBUG	[test]	registrar/migrate.go:159	isFile(/usr/share/filebeat/data/registry/filebeat/meta.json) -> false
2020-02-28T09:30:10.613Z	INFO	registrar/migrate.go:112	Initialize registry meta file
2020-02-28T09:30:10.615Z	INFO	registrar/registrar.go:108	No registry file found under: /usr/share/filebeat/data/registry/filebeat/data.json. Creating a new registry file.
2020-02-28T09:30:10.615Z	DEBUG	[registrar]	registrar/registrar.go:411	Write registry file: /usr/share/filebeat/data/registry/filebeat/data.json (0)
2020-02-28T09:30:10.620Z	DEBUG	[registrar]	registrar/registrar.go:404	Registry file updated. 0 states written.
2020-02-28T09:30:10.620Z	INFO	registrar/registrar.go:145	Loading registrar data from /usr/share/filebeat/data/registry/filebeat/data.json
2020-02-28T09:30:10.620Z	INFO	registrar/registrar.go:152	States Loaded from registrar: 0
2020-02-28T09:30:10.620Z	WARN	beater/filebeat.go:368	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-02-28T09:30:10.621Z	INFO	crawler/crawler.go:72	Loading Inputs: 0
2020-02-28T09:30:10.621Z	INFO	crawler/crawler.go:106	Loading and starting Inputs completed. Enabled inputs: 0
2020-02-28T09:30:10.621Z	DEBUG	[registrar]	registrar/registrar.go:278	Starting Registrar
2020-02-28T09:30:10.621Z	INFO	cfgfile/reload.go:171	Config reloader started
2020-02-28T09:30:20.621Z	DEBUG	[cfgfile]	cfgfile/reload.go:197	Scan for new config files
2020-02-28T09:30:20.622Z	DEBUG	[cfgfile]	cfgfile/cfgfile.go:193	Load config from file: /usr/share/filebeat/modules.d/system.yml
2020-02-28T09:30:20.624Z	DEBUG	[cfgfile]	cfgfile/reload.go:216	Number of module configs found: 1
2020-02-28T09:30:20.624Z	DEBUG	[reload]	cfgfile/list.go:62	Starting reload procedure, current runners: 0
2020-02-28T09:30:20.624Z	DEBUG	[reload]	cfgfile/list.go:80	Start list: 1, Stop list: 0
2020-02-28T09:30:20.629Z	DEBUG	[processors]	processors/processor.go:101	Generated new processors: add_locale=[format=offset]
2020-02-28T09:30:20.630Z	DEBUG	[input]	log/config.go:204	recursive glob enabled
2020-02-28T09:30:20.631Z	DEBUG	[input]	log/input.go:164	exclude_files: [(?-s:.)gz(?-m:$)]. Number of stats: 0
2020-02-28T09:30:20.631Z	DEBUG	[input]	log/input.go:185	input with previous states loaded: 0
2020-02-28T09:30:20.631Z	INFO	log/input.go:152	Configured paths: [/var/log/auth.log* /var/log/secure*]
2020-02-28T09:30:20.631Z	DEBUG	[processors]	processors/processor.go:101	Generated new processors: add_locale=[format=offset]
2020-02-28T09:30:20.631Z	DEBUG	[input]	log/config.go:204	recursive glob enabled
2020-02-28T09:30:20.631Z	DEBUG	[input]	log/input.go:164	exclude_files: [(?-s:.)gz(?-m:$)]. Number of stats: 0
2020-02-28T09:30:20.631Z	DEBUG	[input]	log/input.go:185	input with previous states loaded: 0
2020-02-28T09:30:20.632Z	INFO	log/input.go:152	Configured paths: [/var/log/messages* /var/log/syslog*]
2020-02-28T09:30:20.632Z	DEBUG	[reload]	cfgfile/list.go:101	Starting runner: system (auth, syslog)
2020-02-28T09:30:20.632Z	INFO	input/input.go:114	Starting input of type: log; ID: 10062196400887264802 
2020-02-28T09:30:20.632Z	INFO	input/input.go:114	Starting input of type: log; ID: 8252974517244581286 
2020-02-28T09:30:20.632Z	DEBUG	[input]	log/input.go:191	Start next scan
2020-02-28T09:30:20.632Z	DEBUG	[input]	log/input.go:191	Start next scan
2020-02-28T09:30:20.632Z	DEBUG	[input]	log/input.go:212	input states cleaned up. Before: 0, After: 0, Pending: 0
2020-02-28T09:30:20.632Z	DEBUG	[input]	log/input.go:212	input states cleaned up. Before: 0, After: 0, Pending: 0
2020-02-28T09:30:30.632Z	DEBUG	[cfgfile]	cfgfile/reload.go:197	Scan for new config files
2020-02-28T09:30:30.633Z	DEBUG	[input]	input/input.go:152	Run input
2020-02-28T09:30:30.633Z	DEBUG	[input]	log/input.go:191	Start next scan
2020-02-28T09:30:30.633Z	DEBUG	[input]	input/input.go:152	Run input
2020-02-28T09:30:30.633Z	DEBUG	[input]	log/input.go:191	Start next scan
2020-02-28T09:30:30.633Z	DEBUG	[cfgfile]	cfgfile/cfgfile.go:193	Load config from file: /usr/share/filebeat/modules.d/system.yml
2020-02-28T09:30:30.633Z	DEBUG	[input]	log/input.go:212	input states cleaned up. Before: 0, After: 0, Pending: 0
2020-02-28T09:30:30.634Z	DEBUG	[input]	log/input.go:212	input states cleaned up. Before: 0, After: 0, Pending: 0
2020-02-28T09:30:30.634Z	DEBUG	[cfgfile]	cfgfile/reload.go:216	Number of module configs found: 1
2020-02-28T09:30:30.634Z	DEBUG	[reload]	cfgfile/list.go:62	Starting reload procedure, current runners: 1
2020-02-28T09:30:30.634Z	DEBUG	[reload]	cfgfile/list.go:80	Start list: 0, Stop list: 0

This is the output for the this YAML file.

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: true

filebeat.inputs:
  type: log
  enabled: true
  paths:
    - "/var/log/apache2/*"
  
output.console:
  pretty: true

x

zebulyon · February 28, 2020, 9:36am

And this is the output:

2020-02-28T09:27:44.919Z	INFO	instance/beat.go:622	Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2020-02-28T09:27:44.919Z	DEBUG	[beat]	instance/beat.go:674	Beat metadata path: /usr/share/filebeat/data/meta.json
2020-02-28T09:27:44.919Z	INFO	instance/beat.go:630	Beat ID: 7e96049e-e5be-4347-8175-ee2d30c8ae0e
2020-02-28T09:27:44.919Z	DEBUG	[seccomp]	seccomp/seccomp.go:117	Loading syscall filter	{"seccomp_filter": {"no_new_privs":true,"flag":"tsync","policy":{"default_action":"errno","syscalls":[{"names":["accept","accept4","access","arch_prctl","bind","brk","chmod","clock_gettime","clone","close","connect","dup","dup2","epoll_create","epoll_create1","epoll_ctl","epoll_pwait","epoll_wait","exit","exit_group","fchdir","fchmod","fchown","fcntl","fdatasync","flock","fstat","fstatfs","fsync","ftruncate","futex","getcwd","getdents","getdents64","geteuid","getgid","getpeername","getpid","getppid","getrandom","getrlimit","getrusage","getsockname","getsockopt","gettid","gettimeofday","getuid","inotify_add_watch","inotify_init1","inotify_rm_watch","ioctl","kill","listen","lseek","lstat","madvise","mincore","mkdirat","mmap","mprotect","munmap","nanosleep","newfstatat","open","openat","pipe","pipe2","poll","ppoll","pread64","pselect6","pwrite64","read","readlink","readlinkat","recvfrom","recvmmsg","recvmsg","rename","renameat","rt_sigaction","rt_sigprocmask","rt_sigreturn","sched_getaffinity","sched_yield","sendfile","sendmmsg","sendmsg","sendto","set_robust_list","setitimer","setsockopt","shutdown","sigaltstack","socket","splice","stat","statfs","sysinfo","tgkill","time","tkill","uname","unlink","unlinkat","wait4","waitid","write","writev"],"action":"allow"}]}}}
2020-02-28T09:27:44.919Z	INFO	[seccomp]	seccomp/seccomp.go:124	Syscall filter successfully installed
2020-02-28T09:27:44.919Z	INFO	[beat]	instance/beat.go:958	Beat info	{"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "7e96049e-e5be-4347-8175-ee2d30c8ae0e"}}}
2020-02-28T09:27:44.919Z	INFO	[beat]	instance/beat.go:967	Build info	{"system_info": {"build": {"commit": "6a23e8f8f30f5001ba344e4e54d8d9cb82cb107c", "libbeat": "7.6.0", "time": "2020-02-05T23:06:45.000Z", "version": "7.6.0"}}}
2020-02-28T09:27:44.919Z	INFO	[beat]	instance/beat.go:970	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.13.7"}}}
2020-02-28T09:27:44.920Z	INFO	[beat]	instance/beat.go:974	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-02-24T06:56:55Z","containerized":true,"name":"effad391e392","ip":["127.0.0.1/8","172.17.0.2/16"],"kernel_version":"4.19.0-8-amd64","mac":["02:42:ac:11:00:02"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":7,"patch":1908,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2020-02-28T09:27:44.921Z	INFO	[beat]	instance/beat.go:1003	Process info	{"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 27, "ppid": 1, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-02-28T09:27:43.900Z"}}}
2020-02-28T09:27:44.921Z	INFO	instance/beat.go:298	Setup Beat: filebeat; Version: 7.6.0
2020-02-28T09:27:44.921Z	DEBUG	[beat]	instance/beat.go:324	Initializing output plugins
2020-02-28T09:27:44.921Z	DEBUG	[publisher]	pipeline/consumer.go:137	start pipeline event consumer
2020-02-28T09:27:44.921Z	INFO	[publisher]	pipeline/module.go:110	Beat name: effad391e392
2020-02-28T09:27:44.922Z	WARN	beater/filebeat.go:152	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-02-28T09:27:44.922Z	INFO	instance/beat.go:439	filebeat start running.
2020-02-28T09:27:44.922Z	DEBUG	[test]	registrar/migrate.go:159	isFile(/usr/share/filebeat/data/registry) -> false
2020-02-28T09:27:44.923Z	DEBUG	[test]	registrar/migrate.go:159	isFile() -> false
2020-02-28T09:27:44.923Z	DEBUG	[test]	registrar/migrate.go:152	isDir(/usr/share/filebeat/data/registry/filebeat) -> false
2020-02-28T09:27:44.923Z	DEBUG	[registrar]	registrar/migrate.go:51	Registry type '' found
2020-02-28T09:27:44.923Z	DEBUG	[test]	registrar/migrate.go:159	isFile(.bak) -> false
2020-02-28T09:27:44.923Z	DEBUG	[test]	registrar/migrate.go:152	isDir(/usr/share/filebeat/data/registry/filebeat) -> false
2020-02-28T09:27:44.923Z	INFO	registrar/migrate.go:104	No registry home found. Create: /usr/share/filebeat/data/registry/filebeat
2020-02-28T09:27:44.922Z	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
2020-02-28T09:27:44.923Z	DEBUG	[test]	registrar/migrate.go:159	isFile(/usr/share/filebeat/data/registry/filebeat/meta.json) -> false
2020-02-28T09:27:44.923Z	INFO	registrar/migrate.go:112	Initialize registry meta file
2020-02-28T09:27:44.925Z	INFO	registrar/registrar.go:108	No registry file found under: /usr/share/filebeat/data/registry/filebeat/data.json. Creating a new registry file.
2020-02-28T09:27:44.925Z	DEBUG	[registrar]	registrar/registrar.go:411	Write registry file: /usr/share/filebeat/data/registry/filebeat/data.json (0)
2020-02-28T09:27:44.930Z	DEBUG	[registrar]	registrar/registrar.go:404	Registry file updated. 0 states written.
2020-02-28T09:27:44.930Z	INFO	registrar/registrar.go:145	Loading registrar data from /usr/share/filebeat/data/registry/filebeat/data.json
2020-02-28T09:27:44.930Z	INFO	registrar/registrar.go:152	States Loaded from registrar: 0
2020-02-28T09:27:44.931Z	WARN	beater/filebeat.go:368	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-02-28T09:27:44.931Z	INFO	crawler/crawler.go:72	Loading Inputs: 3
2020-02-28T09:27:44.931Z	DEBUG	[registrar]	registrar/registrar.go:278	Starting Registrar
2020-02-28T09:27:44.931Z	DEBUG	[publisher]	pipeline/client.go:162	client: closing acker
2020-02-28T09:27:44.931Z	DEBUG	[publisher]	pipeline/client.go:173	client: done closing acker
2020-02-28T09:27:44.931Z	DEBUG	[publisher]	pipeline/client.go:176	client: cancelled 0 events
2020-02-28T09:27:44.931Z	DEBUG	[publisher]	pipeline/client.go:147	client: wait for acker to finish
2020-02-28T09:27:44.931Z	DEBUG	[publisher]	pipeline/client.go:149	client: acker shut down
2020-02-28T09:27:44.931Z	INFO	crawler/crawler.go:139	Stopping Crawler
2020-02-28T09:27:44.931Z	INFO	crawler/crawler.go:149	Stopping 0 inputs
2020-02-28T09:27:44.931Z	INFO	crawler/crawler.go:165	Crawler stopped
2020-02-28T09:27:44.931Z	INFO	registrar/registrar.go:367	Stopping Registrar
2020-02-28T09:27:44.931Z	INFO	registrar/registrar.go:293	Ending Registrar
2020-02-28T09:27:44.931Z	DEBUG	[registrar]	registrar/registrar.go:411	Write registry file: /usr/share/filebeat/data/registry/filebeat/data.json (0)
2020-02-28T09:27:44.936Z	DEBUG	[registrar]	registrar/registrar.go:404	Registry file updated. 0 states written.
2020-02-28T09:27:44.938Z	INFO	[monitoring]	log/log.go:153	Total non-zero metrics	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":20,"time":{"ms":20}},"total":{"ticks":50,"time":{"ms":57},"value":50},"user":{"ticks":30,"time":{"ms":37}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":6},"info":{"ephemeral_id":"564e56cd-5057-4e45-b586-774f79f01ede","uptime":{"ms":42}},"memstats":{"gc_next":9573456,"memory_alloc":7900760,"memory_total":12933696,"rss":52154368},"runtime":{"goroutines":12}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"console"},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0},"writes":{"success":2,"total":2}},"system":{"cpu":{"cores":4},"load":{"1":0.95,"15":1.6,"5":1.14,"norm":{"1":0.2375,"15":0.4,"5":0.285}}}}}}
2020-02-28T09:27:44.938Z	INFO	[monitoring]	log/log.go:154	Uptime: 43.046076ms
2020-02-28T09:27:44.938Z	INFO	[monitoring]	log/log.go:131	Stopping metrics logging.
2020-02-28T09:27:44.939Z	INFO	instance/beat.go:445	filebeat stopped.
2020-02-28T09:27:44.939Z	ERROR	instance/beat.go:933	Exiting: Error while initializing input: No paths were defined for input accessing 'filebeat.inputs.0' (source:'filebeat.yml')
Exiting: Error while initializing input: No paths were defined for input accessing 'filebeat.inputs.0' (source:'filebeat.yml')

That corresponds to this YAML file:

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: true

filebeat.inputs:
- type: log
- enabled: true
- paths:
    - "/var/log/apache2/*"
  
output.console:
  pretty: true

ChrsMark · February 28, 2020, 2:41pm

The last configuration is not valid as you can see in the logs:

2020-02-28T09:27:44.939Z	ERROR	instance/beat.go:933	Exiting: Error while initializing input: No paths were defined for input accessing 'filebeat.inputs.0' (source:'filebeat.yml')
Exiting: Error while initializing input: No paths were defined for input accessing 'filebeat.inputs.0' (source:'filebeat.yml')

The first one is valid but seems to not collecting logs from /var/log/apache2/.

In my case with:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/apache_logs

I'm able to collect logs from /tmp/apache and the output looks like this:

2020-02-28T16:33:49.641+0200	DEBUG	[harvester]	log/harvester.go:478	Update state: /tmp/apache_logs, offset: 0
2020-02-28T16:33:49.641+0200	DEBUG	[acker]	beater/acker.go:64	stateful ack	{"count": 1}
2020-02-28T16:33:49.641+0200	DEBUG	[input]	log/input.go:212	input states cleaned up. Before: 1, After: 1, Pending: 0
2020-02-28T16:33:49.641+0200	DEBUG	[cfgfile]	cfgfile/reload.go:216	Number of module configs found: 0
2020-02-28T16:33:49.641+0200	INFO	log/harvester.go:297	Harvester started for file: /tmp/apache_logs

You can try with a specific file and not all files under /var/log/apache2/.

C.

zebulyon · February 28, 2020, 3:30pm

Still the same situation response messages as before

YAML file:

filebeat.inputs:
  type: log
  enabled: true
  paths:
    - "/var/log/apache2/sample.log

I have a question

When the YAML file looks for the path does it look for the whole path, or is it based one what directory the file is in when executing.

I'm asking because my var directory is a directory I create in the filebeat directory

Absolute filepath to log file is thus:

/usr/share/filebeat/var/log/apache2/sample.log

Selection_012

I thus tried to do another YAML configuration like this

filebeat.inputs:
  type: log
  enabled: true
  paths:
    - "/usr/share/filebeat/var/log/apache2/sample.log"

Now it does not look like it changed anything I seemingly get the same response regardless of YAML configuration with the filebeat -e -d "*" command but it might still have something to do with it.

ChrsMark · February 28, 2020, 4:02pm

Hey, yes you have to define full paths.

zebulyon · February 28, 2020, 4:14pm

But is the not what I have done here

filebeat.inputs:
  type: log
  enabled: true
  paths:
    - "/usr/share/filebeat/var/log/apache2/sample.log"

And this does not seem to be working either

zebulyon · February 28, 2020, 4:29pm

I have now also tried using the the absolute path of

--volume="$(pwd)/pl_agent_test.log:/var/log/apache2/sample.log" \

Selection_013

Selection_014

But I still get the same response when I run the filebeat -e -d "*" command

zebulyon · February 28, 2020, 4:30pm

2020-02-28T16:26:53.547Z	INFO	instance/beat.go:622	Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2020-02-28T16:26:53.547Z	DEBUG	[beat]	instance/beat.go:674	Beat metadata path: /usr/share/filebeat/data/meta.json
2020-02-28T16:26:53.547Z	INFO	instance/beat.go:630	Beat ID: 403b8cce-9415-4b28-8d89-0a8c7c66bab9
2020-02-28T16:26:53.547Z	DEBUG	[seccomp]	seccomp/seccomp.go:117	Loading syscall filter	{"seccomp_filter": {"no_new_privs":true,"flag":"tsync","policy":{"default_action":"errno","syscalls":[{"names":["accept","accept4","access","arch_prctl","bind","brk","chmod","clock_gettime","clone","close","connect","dup","dup2","epoll_create","epoll_create1","epoll_ctl","epoll_pwait","epoll_wait","exit","exit_group","fchdir","fchmod","fchown","fcntl","fdatasync","flock","fstat","fstatfs","fsync","ftruncate","futex","getcwd","getdents","getdents64","geteuid","getgid","getpeername","getpid","getppid","getrandom","getrlimit","getrusage","getsockname","getsockopt","gettid","gettimeofday","getuid","inotify_add_watch","inotify_init1","inotify_rm_watch","ioctl","kill","listen","lseek","lstat","madvise","mincore","mkdirat","mmap","mprotect","munmap","nanosleep","newfstatat","open","openat","pipe","pipe2","poll","ppoll","pread64","pselect6","pwrite64","read","readlink","readlinkat","recvfrom","recvmmsg","recvmsg","rename","renameat","rt_sigaction","rt_sigprocmask","rt_sigreturn","sched_getaffinity","sched_yield","sendfile","sendmmsg","sendmsg","sendto","set_robust_list","setitimer","setsockopt","shutdown","sigaltstack","socket","splice","stat","statfs","sysinfo","tgkill","time","tkill","uname","unlink","unlinkat","wait4","waitid","write","writev"],"action":"allow"}]}}}
2020-02-28T16:26:53.548Z	INFO	[seccomp]	seccomp/seccomp.go:124	Syscall filter successfully installed
2020-02-28T16:26:53.548Z	INFO	[beat]	instance/beat.go:958	Beat info	{"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "403b8cce-9415-4b28-8d89-0a8c7c66bab9"}}}
2020-02-28T16:26:53.548Z	INFO	[beat]	instance/beat.go:967	Build info	{"system_info": {"build": {"commit": "6a23e8f8f30f5001ba344e4e54d8d9cb82cb107c", "libbeat": "7.6.0", "time": "2020-02-05T23:06:45.000Z", "version": "7.6.0"}}}
2020-02-28T16:26:53.548Z	INFO	[beat]	instance/beat.go:970	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.13.7"}}}
2020-02-28T16:26:53.549Z	INFO	[beat]	instance/beat.go:974	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-02-24T06:56:56Z","containerized":true,"name":"5d5c3c4d34a9","ip":["127.0.0.1/8","172.17.0.2/16"],"kernel_version":"4.19.0-8-amd64","mac":["02:42:ac:11:00:02"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":7,"patch":1908,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2020-02-28T16:26:53.549Z	INFO	[beat]	instance/beat.go:1003	Process info	{"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 25, "ppid": 1, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-02-28T16:26:53.020Z"}}}
2020-02-28T16:26:53.549Z	INFO	instance/beat.go:298	Setup Beat: filebeat; Version: 7.6.0
2020-02-28T16:26:53.549Z	DEBUG	[beat]	instance/beat.go:324	Initializing output plugins
2020-02-28T16:26:53.550Z	DEBUG	[publisher]	pipeline/consumer.go:137	start pipeline event consumer
2020-02-28T16:26:53.550Z	INFO	[publisher]	pipeline/module.go:110	Beat name: 5d5c3c4d34a9
2020-02-28T16:26:53.551Z	WARN	beater/filebeat.go:152	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-02-28T16:26:53.551Z	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
2020-02-28T16:26:53.551Z	INFO	instance/beat.go:439	filebeat start running.
2020-02-28T16:26:53.551Z	DEBUG	[test]	registrar/migrate.go:159	isFile(/usr/share/filebeat/data/registry) -> false
2020-02-28T16:26:53.551Z	DEBUG	[test]	registrar/migrate.go:159	isFile() -> false
2020-02-28T16:26:53.551Z	DEBUG	[test]	registrar/migrate.go:152	isDir(/usr/share/filebeat/data/registry/filebeat) -> false
2020-02-28T16:26:53.551Z	DEBUG	[registrar]	registrar/migrate.go:51	Registry type '' found
2020-02-28T16:26:53.551Z	DEBUG	[test]	registrar/migrate.go:159	isFile(.bak) -> false
2020-02-28T16:26:53.551Z	DEBUG	[test]	registrar/migrate.go:152	isDir(/usr/share/filebeat/data/registry/filebeat) -> false
2020-02-28T16:26:53.551Z	INFO	registrar/migrate.go:104	No registry home found. Create: /usr/share/filebeat/data/registry/filebeat
2020-02-28T16:26:53.551Z	DEBUG	[test]	registrar/migrate.go:159	isFile(/usr/share/filebeat/data/registry/filebeat/meta.json) -> false
2020-02-28T16:26:53.551Z	INFO	registrar/migrate.go:112	Initialize registry meta file
2020-02-28T16:26:53.554Z	INFO	registrar/registrar.go:108	No registry file found under: /usr/share/filebeat/data/registry/filebeat/data.json. Creating a new registry file.
2020-02-28T16:26:53.554Z	DEBUG	[registrar]	registrar/registrar.go:411	Write registry file: /usr/share/filebeat/data/registry/filebeat/data.json (0)
2020-02-28T16:26:53.559Z	DEBUG	[registrar]	registrar/registrar.go:404	Registry file updated. 0 states written.
2020-02-28T16:26:53.559Z	INFO	registrar/registrar.go:145	Loading registrar data from /usr/share/filebeat/data/registry/filebeat/data.json
2020-02-28T16:26:53.559Z	INFO	registrar/registrar.go:152	States Loaded from registrar: 0
2020-02-28T16:26:53.559Z	WARN	beater/filebeat.go:368	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-02-28T16:26:53.559Z	INFO	crawler/crawler.go:72	Loading Inputs: 0
2020-02-28T16:26:53.559Z	INFO	crawler/crawler.go:106	Loading and starting Inputs completed. Enabled inputs: 0
2020-02-28T16:26:53.559Z	DEBUG	[registrar]	registrar/registrar.go:278	Starting Registrar
2020-02-28T16:26:53.559Z	INFO	cfgfile/reload.go:171	Config reloader started
2020-02-28T16:27:03.560Z	DEBUG	[cfgfile]	cfgfile/reload.go:197	Scan for new config files
2020-02-28T16:27:03.561Z	DEBUG	[cfgfile]	cfgfile/cfgfile.go:193	Load config from file: /usr/share/filebeat/modules.d/system.yml
2020-02-28T16:27:03.561Z	DEBUG	[cfgfile]	cfgfile/reload.go:216	Number of module configs found: 1
2020-02-28T16:27:03.561Z	DEBUG	[reload]	cfgfile/list.go:62	Starting reload procedure, current runners: 0
2020-02-28T16:27:03.561Z	DEBUG	[reload]	cfgfile/list.go:80	Start list: 1, Stop list: 0
2020-02-28T16:27:03.570Z	DEBUG	[processors]	processors/processor.go:101	Generated new processors: add_locale=[format=offset]
2020-02-28T16:27:03.571Z	DEBUG	[input]	log/config.go:204	recursive glob enabled
2020-02-28T16:27:03.571Z	DEBUG	[input]	log/input.go:164	exclude_files: [(?-s:.)gz(?-m:$)]. Number of stats: 0
2020-02-28T16:27:03.571Z	DEBUG	[input]	log/input.go:185	input with previous states loaded: 0
2020-02-28T16:27:03.571Z	INFO	log/input.go:152	Configured paths: [/var/log/auth.log* /var/log/secure*]
2020-02-28T16:27:03.571Z	DEBUG	[processors]	processors/processor.go:101	Generated new processors: add_locale=[format=offset]
2020-02-28T16:27:03.572Z	DEBUG	[input]	log/config.go:204	recursive glob enabled
2020-02-28T16:27:03.572Z	DEBUG	[input]	log/input.go:164	exclude_files: [(?-s:.)gz(?-m:$)]. Number of stats: 0
2020-02-28T16:27:03.572Z	DEBUG	[input]	log/input.go:185	input with previous states loaded: 0
2020-02-28T16:27:03.572Z	INFO	log/input.go:152	Configured paths: [/var/log/messages* /var/log/syslog*]
2020-02-28T16:27:03.572Z	DEBUG	[reload]	cfgfile/list.go:101	Starting runner: system (auth, syslog)
2020-02-28T16:27:03.572Z	INFO	input/input.go:114	Starting input of type: log; ID: 10062196400887264802 
2020-02-28T16:27:03.572Z	INFO	input/input.go:114	Starting input of type: log; ID: 8252974517244581286 
2020-02-28T16:27:03.572Z	DEBUG	[input]	log/input.go:191	Start next scan
2020-02-28T16:27:03.572Z	DEBUG	[input]	log/input.go:212	input states cleaned up. Before: 0, After: 0, Pending: 0
2020-02-28T16:27:03.572Z	DEBUG	[input]	log/input.go:191	Start next scan
2020-02-28T16:27:03.572Z	DEBUG	[input]	log/input.go:212	input states cleaned up. Before: 0, After: 0, Pending: 0
2020-02-28T16:27:13.572Z	DEBUG	[cfgfile]	cfgfile/reload.go:197	Scan for new config files
2020-02-28T16:27:13.572Z	DEBUG	[input]	input/input.go:152	Run input
2020-02-28T16:27:13.573Z	DEBUG	[input]	log/input.go:191	Start next scan
2020-02-28T16:27:13.573Z	DEBUG	[cfgfile]	cfgfile/cfgfile.go:193	Load config from file: /usr/share/filebeat/modules.d/system.yml
2020-02-28T16:27:13.573Z	DEBUG	[input]	input/input.go:152	Run input
2020-02-28T16:27:13.573Z	DEBUG	[input]	log/input.go:191	Start next scan
2020-02-28T16:27:13.573Z	DEBUG	[input]	log/input.go:212	input states cleaned up. Before: 0, After: 0, Pending: 0
2020-02-28T16:27:13.573Z	DEBUG	[cfgfile]	cfgfile/reload.go:216	Number of module configs found: 1
2020-02-28T16:27:13.573Z	DEBUG	[reload]	cfgfile/list.go:62	Starting reload procedure, current runners: 1
2020-02-28T16:27:13.573Z	DEBUG	[reload]	cfgfile/list.go:80	Start list: 0, Stop list: 0
2020-02-28T16:27:13.573Z	DEBUG	[input]	log/input.go:212	input states cleaned up. Before: 0, After: 0, Pending: 0
2020-02-28T16:27:23.556Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":50,"time":{"ms":52}},"total":{"ticks":100,"time":{"ms":111},"value":100},"user":{"ticks":50,"time":{"ms":59}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":6},"info":{"ephemeral_id":"e7357f99-c6c7-47f6-8671-abeaf604f0a9","uptime":{"ms":30026}},"memstats":{"gc_next":9023808,"memory_alloc":4746624,"memory_total":13428448,"rss":55914496},"runtime":{"goroutines":27}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":2},"output":{"type":"console"},"pipeline":{"clients":2,"events":{"active":0}}},"registrar":{"states":{"current":0},"writes":{"success":1,"total":1}},"system":{"cpu":{"cores":4},"load":{"1":0.31,"15":0.61,"5":0.47,"norm":{"1":0.0775,"15":0.1525,"5":0.1175}}}}}}

zebulyon · February 28, 2020, 4:30pm

2020-02-28T16:27:23.573Z	DEBUG	[input]	input/input.go:152	Run input
2020-02-28T16:27:23.573Z	DEBUG	[input]	log/input.go:191	Start next scan
2020-02-28T16:27:23.574Z	DEBUG	[input]	input/input.go:152	Run input
2020-02-28T16:27:23.574Z	DEBUG	[input]	log/input.go:191	Start next scan
2020-02-28T16:27:23.574Z	DEBUG	[input]	log/input.go:212	input states cleaned up. Before: 0, After: 0, Pending: 0
2020-02-28T16:27:23.574Z	DEBUG	[cfgfile]	cfgfile/reload.go:197	Scan for new config files
2020-02-28T16:27:23.574Z	DEBUG	[input]	log/input.go:212	input states cleaned up. Before: 0, After: 0, Pending: 0
2020-02-28T16:27:23.575Z	DEBUG	[cfgfile]	cfgfile/cfgfile.go:193	Load config from file: /usr/share/filebeat/modules.d/system.yml
2020-02-28T16:27:23.575Z	DEBUG	[cfgfile]	cfgfile/reload.go:216	Number of module configs found: 1
2020-02-28T16:27:23.575Z	DEBUG	[reload]	cfgfile/list.go:62	Starting reload procedure, current runners: 1
2020-02-28T16:27:23.575Z	DEBUG	[reload]	cfgfile/list.go:80	Start list: 0, Stop list: 0
^C2020-02-28T16:27:29.971Z	DEBUG	[service]	service/service.go:53	Received sigterm/sigint, stopping
2020-02-28T16:27:29.971Z	INFO	beater/filebeat.go:443	Stopping filebeat
2020-02-28T16:27:29.971Z	INFO	crawler/crawler.go:139	Stopping Crawler
2020-02-28T16:27:29.971Z	INFO	crawler/crawler.go:149	Stopping 0 inputs
2020-02-28T16:27:29.971Z	INFO	cfgfile/reload.go:193	Dynamic config reloader stopped
2020-02-28T16:27:29.971Z	INFO	[reload]	cfgfile/list.go:118	Stopping 1 runners ...
2020-02-28T16:27:29.971Z	DEBUG	[reload]	cfgfile/list.go:129	Stopping runner: system (auth, syslog)
2020-02-28T16:27:29.971Z	DEBUG	[publisher]	pipeline/client.go:162	client: closing acker
2020-02-28T16:27:29.971Z	INFO	input/input.go:149	input ticker stopped
2020-02-28T16:27:29.971Z	INFO	input/input.go:167	Stopping Input: 10062196400887264802
2020-02-28T16:27:29.971Z	DEBUG	[publisher]	pipeline/client.go:173	client: done closing acker
2020-02-28T16:27:29.972Z	DEBUG	[publisher]	pipeline/client.go:176	client: cancelled 0 events
2020-02-28T16:27:29.971Z	DEBUG	[publisher]	pipeline/client.go:162	client: closing acker
2020-02-28T16:27:29.972Z	DEBUG	[publisher]	pipeline/client.go:173	client: done closing acker
2020-02-28T16:27:29.972Z	DEBUG	[publisher]	pipeline/client.go:176	client: cancelled 0 events
2020-02-28T16:27:29.972Z	DEBUG	[publisher]	pipeline/client.go:147	client: wait for acker to finish
2020-02-28T16:27:29.972Z	DEBUG	[publisher]	pipeline/client.go:149	client: acker shut down
2020-02-28T16:27:29.972Z	INFO	input/input.go:149	input ticker stopped
2020-02-28T16:27:29.972Z	INFO	input/input.go:167	Stopping Input: 8252974517244581286
2020-02-28T16:27:29.972Z	DEBUG	[publisher]	pipeline/client.go:147	client: wait for acker to finish
2020-02-28T16:27:29.972Z	DEBUG	[publisher]	pipeline/client.go:149	client: acker shut down
2020-02-28T16:27:29.972Z	DEBUG	[reload]	cfgfile/list.go:131	Stopped runner: system (auth, syslog)
2020-02-28T16:27:29.972Z	INFO	crawler/crawler.go:165	Crawler stopped
2020-02-28T16:27:29.972Z	INFO	registrar/registrar.go:367	Stopping Registrar
2020-02-28T16:27:29.973Z	INFO	registrar/registrar.go:293	Ending Registrar
2020-02-28T16:27:29.973Z	DEBUG	[registrar]	registrar/registrar.go:411	Write registry file: /usr/share/filebeat/data/registry/filebeat/data.json (0)
2020-02-28T16:27:29.978Z	DEBUG	[registrar]	registrar/registrar.go:404	Registry file updated. 0 states written.
2020-02-28T16:27:29.980Z	INFO	[monitoring]	log/log.go:153	Total non-zero metrics	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":60,"time":{"ms":64}},"total":{"ticks":120,"time":{"ms":124},"value":120},"user":{"ticks":60,"time":{"ms":60}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":6},"info":{"ephemeral_id":"e7357f99-c6c7-47f6-8671-abeaf604f0a9","uptime":{"ms":36455}},"memstats":{"gc_next":9023808,"memory_alloc":5141840,"memory_total":13823664,"rss":55914496},"runtime":{"goroutines":11}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":3},"output":{"type":"console"},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0},"writes":{"success":2,"total":2}},"system":{"cpu":{"cores":4},"load":{"1":0.26,"15":0.6,"5":0.46,"norm":{"1":0.065,"15":0.15,"5":0.115}}}}}}

zebulyon · March 2, 2020, 10:48am

I think I just solved the issue. In the .YAML file I did not put a - in-front of the type declaration in the input field.

This is what I mean.

Old and Wrong of doing things

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: true

filebeat.inputs:
  type: log
  enabled: true
  paths:
    - "/var/log/sample.log"
      
output.console:
  pretty: true

Correct setup:

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: true

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - "/var/log/sample.log"
      
output.console:
  pretty: true

So correct solution is to declare type tag with a - followed by a space " " then type
- type: while still adhereing to the indentation of the filebeat.input tag

system · March 30, 2020, 10:49am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.