Help me in configure filebeat index

Rajesh119 · May 20, 2022, 5:55am

i use this command, it will not shown my filebeat index

curl http://localhost:9200/_cat/indices?v

And i use below command to know the status

filebeat -e -c /etc/filebeat/filebeat.yml
output:

{"log.level":"info","@timestamp":"2022-05-20T06:46:04.062Z","log.origin":{"file.name":"instance/beat.go","file.line":685},"message":"Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T06:46:04.063Z","log.origin":{"file.name":"instance/beat.go","file.line":693},"message":"Beat ID: a985c9e2-fd39-42dd-9b1d-6c45a4629a2b","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T06:46:04.069Z","log.origin":{"file.name":"instance/beat.go","file.line":424},"message":"filebeat stopped.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-05-20T06:46:04.069Z","log.origin":{"file.name":"instance/beat.go","file.line":1038},"message":"Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).","service.name":"filebeat","ecs.version":"1.6.0"}
Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).

Please anyone help me to fix this.

Thanks in advance.

ibra_013 · May 20, 2022, 10:49am

Hi @Rajesh119

when you tried to run the below command, the filebeat service was still running , this is why you have this error.

Rajesh119:

{"log.level":"error","@timestamp":"2022-05-20T06:46:04.069Z","log.origin":{"file.name":"instance/beat.go","file.line":1038},"message":"Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).","service.name":"filebeat","ecs.version":"1.6.0"}
Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).

Rajesh119 · May 20, 2022, 10:57am

@ibra_013 Thank you
It shows like this

sudo service filebeat status

Output:

filebeat-god (pid  3695) is running...

ibra_013 · May 20, 2022, 11:01am

Hi @Rajesh119

So i you want to test the filebeat output on the console (filebeat -e ), you have to make sure that the filebeat service is stopped.

Rajesh119 · May 20, 2022, 11:07am

Hi @ibra_013 once i stop the filebeat service and check status

filebeat -e

{"log.level":"info","@timestamp":"2022-05-20T11:06:30.604Z","log.origin":{"file.name":"instance/beat.go","file.line":685},"message":"Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:06:30.605Z","log.origin":{"file.name":"instance/beat.go","file.line":693},"message":"Beat ID: a985c9e2-fd39-42dd-9b1d-6c45a4629a2b","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:06:30.609Z","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":124},"message":"Syscall filter successfully installed","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:06:30.609Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1063},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"/etc/filebeat","data":"/var/lib/filebeat","home":"/usr/share/filebeat","logs":"/var/log/filebeat"},"type":"filebeat","uuid":"a985c9e2-fd39-42dd-9b1d-6c45a4629a2b"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-20T11:06:30.609Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1072},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"045da3a1bb89944373c33332c18ca99ef6192df2","libbeat":"8.2.0","time":"2022-04-19T23:31:06.000Z","version":"8.2.0"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-20T11:06:30.609Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1075},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":1,"version":"go1.17.8"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-20T11:06:30.610Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1079},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-05-16T12:47:00Z","containerized":false,"name":"mail.thapos.net","ip":["127.0.0.1/8","::1/128","172.31.19.214/20","fe80::8bf:52ff:febd:8a4/64"],"kernel_version":"4.14.214-118.339.amzn1.x86_64","mac":["0a:bf:52:bd:08:a4"],"os":{"type":"linux","family":"redhat","platform":"amzn","name":"Amazon Linux AMI","version":"2018.03","major":2018,"minor":3,"patch":0},"timezone":"UTC","timezone_offset_sec":0,"id":"ef32a8ed995e86063ec99c115b0b9d78"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-20T11:06:30.610Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1108},"message":"Process info","service.name":"filebeat","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null},"cwd":"/root","exe":"/usr/share/filebeat/bin/filebeat","name":"filebeat","pid":4253,"ppid":2146,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2022-05-20T11:06:30.330Z"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-20T11:06:30.610Z","log.origin":{"file.name":"instance/beat.go","file.line":325},"message":"Setup Beat: filebeat; Version: 8.2.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:06:30.611Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":105},"message":"elasticsearch url: http://34.205.74.243:9200","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:06:30.611Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: mail.thapos.net","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:06:30.611Z","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":108},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:06:30.612Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":142},"message":"Starting metrics logging every 30s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:06:30.612Z","log.origin":{"file.name":"instance/beat.go","file.line":505},"message":"filebeat start running.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:06:30.612Z","log.origin":{"file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for '/var/lib/filebeat/filebeat'. Active transaction id=8","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:06:30.612Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":109},"message":"States Loaded from registrar: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:06:30.612Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":71},"message":"Loading Inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:06:30.612Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":117},"message":"starting input, keys present on the config: [filebeat.inputs.0.enabled filebeat.inputs.0.paths.0 filebeat.inputs.0.type]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-20T11:06:30.612Z","log.logger":"cfgwarn","log.origin":{"file.name":"log/input.go","file.line":89},"message":"DEPRECATED: Log input. Use Filestream input instead.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:06:30.613Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":171},"message":"Configured paths: [/home/thapos/tomcat/tomcat8/logs/catalina.out]","service.name":"filebeat","input_id":"f0e80e3a-2020-407d-a26b-8f669d0d9eca","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:06:30.613Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":148},"message":"Starting input (ID: 8813593274292630345)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:06:30.614Z","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":108},"message":"Enabled modules/filesets: tomcat (log)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:06:30.615Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":106},"message":"Loading and starting Inputs completed. Enabled inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:06:30.616Z","log.origin":{"file.name":"cfgfile/reload.go","file.line":164},"message":"Config reloader started","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:06:30.616Z","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":108},"message":"Enabled modules/filesets: tomcat (log)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-20T11:06:30.643Z","log.logger":"cfgwarn","log.origin":{"file.name":"registered_domain/registered_domain.go","file.line":61},"message":"BETA: The registered_domain processor is beta.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-20T11:06:30.644Z","log.logger":"cfgwarn","log.origin":{"file.name":"registered_domain/registered_domain.go","file.line":61},"message":"BETA: The registered_domain processor is beta.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-20T11:06:30.644Z","log.logger":"cfgwarn","log.origin":{"file.name":"registered_domain/registered_domain.go","file.line":61},"message":"BETA: The registered_domain processor is beta.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-20T11:06:30.644Z","log.logger":"cfgwarn","log.origin":{"file.name":"registered_domain/registered_domain.go","file.line":61},"message":"BETA: The registered_domain processor is beta.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-20T11:06:30.644Z","log.logger":"cfgwarn","log.origin":{"file.name":"registered_domain/registered_domain.go","file.line":61},"message":"BETA: The registered_domain processor is beta.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-20T11:06:30.644Z","log.logger":"cfgwarn","log.origin":{"file.name":"registered_domain/registered_domain.go","file.line":61},"message":"BETA: The registered_domain processor is beta.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:06:30.644Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":105},"message":"elasticsearch url: http://34.205.74.243:9200","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:06:30.651Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":105},"message":"add_cloud_metadata: hosting provider type detected as aws, metadata={\"cloud\":{\"account\":{\"id\":\"632467875476\"},\"availability_zone\":\"us-east-1c\",\"image\":{\"id\":\"ami-14c5486b\"},\"instance\":{\"id\":\"i-0e4a37307f16515d3\"},\"machine\":{\"type\":\"t2.small\"},\"provider\":\"aws\",\"region\":\"us-east-1\",\"service\":{\"name\":\"EC2\"}}}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:06:30.654Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":287},"message":"Attempting to connect to Elasticsearch version 8.2.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:06:30.658Z","log.origin":{"file.name":"cfgfile/reload.go","file.line":224},"message":"Loading of config files completed.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:06:30.658Z","log.origin":{"file.name":"udp/input.go","file.line":98},"message":"Starting UDP input","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:06:30.658Z","log.logger":"UDP","log.origin":{"file.name":"dgram/server.go","file.line":99},"message":"Started listening for UDP connection","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:07:00.614Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":20,"time":{"ms":20}},"total":{"ticks":170,"time":{"ms":170},"value":0},"user":{"ticks":150,"time":{"ms":150}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":12},"info":{"ephemeral_id":"9166e7ea-1600-434b-bdd2-00976b2f1c12","uptime":{"ms":30084},"version":"8.2.0"},"memstats":{"gc_next":26460864,"memory_alloc":13813096,"memory_sys":41501704,"memory_total":60948384,"rss":133533696},"runtime":{"goroutines":34}},"filebeat":{"events":{"added":1,"done":1},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1,"starts":1},"reloads":1,"scans":1},"output":{"events":{"active":0},"type":"elasticsearch"},"pipeline":{"clients":2,"events":{"active":0,"filtered":1,"total":1},"queue":{"max_events":4096}}},"registrar":{"states":{"current":1,"update":1},"writes":{"success":1,"total":1}},"system":{"cpu":{"cores":1},"load":{"1":0,"15":0,"5":0,"norm":{"1":0,"15":0,"5":0}}}},"ecs.version":"1.6.0"}}
^C{"log.level":"info","@timestamp":"2022-05-20T11:07:01.423Z","log.origin":{"file.name":"beater/filebeat.go","file.line":425},"message":"Stopping filebeat","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:07:01.424Z","log.origin":{"file.name":"beater/crawler.go","file.line":155},"message":"Stopping Crawler","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:07:01.424Z","log.origin":{"file.name":"beater/crawler.go","file.line":165},"message":"Stopping 1 inputs","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:07:01.428Z","log.origin":{"file.name":"cfgfile/reload.go","file.line":227},"message":"Dynamic config reloader stopped","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:07:01.428Z","log.logger":"reload","log.origin":{"file.name":"cfgfile/list.go","file.line":129},"message":"Stopping 1 runners ...","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:07:01.428Z","log.origin":{"file.name":"input/input.go","file.line":134},"message":"input ticker stopped","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:07:01.428Z","log.origin":{"file.name":"udp/input.go","file.line":113},"message":"Stopping UDP input","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:07:01.429Z","log.logger":"udp","log.origin":{"file.name":"dgram/handler.go","file.line":73},"message":"Connection has been closed","service.name":"filebeat","address":"localhost:9523","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:07:01.429Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":170},"message":"Stopping input: 8813593274292630345","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:07:01.429Z","log.origin":{"file.name":"input/input.go","file.line":134},"message":"input ticker stopped","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:07:01.429Z","log.origin":{"file.name":"beater/crawler.go","file.line":185},"message":"Crawler stopped","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:07:01.429Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":132},"message":"Stopping Registrar","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:07:01.429Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":166},"message":"Ending Registrar","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:07:01.429Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":137},"message":"Registrar stopped","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:07:01.442Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":192},"message":"Total metrics","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":20,"time":{"ms":20}},"total":{"ticks":180,"time":{"ms":180},"value":0},"user":{"ticks":160,"time":{"ms":160}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":10},"info":{"ephemeral_id":"9166e7ea-1600-434b-bdd2-00976b2f1c12","uptime":{"ms":30912},"version":"8.2.0"},"memstats":{"gc_next":26460864,"memory_alloc":14114672,"memory_sys":41501704,"memory_total":61249960,"rss":133533696},"runtime":{"goroutines":15}},"filebeat":{"events":{"active":0,"added":1,"done":1},"harvester":{"closed":0,"open_files":0,"running":0,"skipped":0,"started":0},"input":{"log":{"files":{"renamed":0,"truncated":0}},"netflow":{"flows":0,"packets":{"dropped":0,"received":0}}}},"libbeat":{"config":{"module":{"running":1,"starts":1,"stops":0},"reloads":1,"scans":1},"output":{"events":{"acked":0,"active":0,"batches":0,"dropped":0,"duplicates":0,"failed":0,"toomany":0,"total":0},"read":{"bytes":0,"errors":0},"type":"elasticsearch","write":{"bytes":0,"errors":0}},"pipeline":{"clients":0,"events":{"active":0,"dropped":0,"failed":0,"filtered":1,"published":0,"retry":0,"total":1},"queue":{"acked":0,"max_events":4096}}},"registrar":{"states":{"cleanup":0,"current":1,"update":1},"writes":{"fail":0,"success":1,"total":1}},"system":{"cpu":{"cores":1},"load":{"1":0,"15":0,"5":0,"norm":{"1":0,"15":0,"5":0}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-20T11:07:01.442Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":193},"message":"Uptime: 30.913237556s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:07:01.442Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":160},"message":"Stopping metrics logging.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-20T11:07:01.442Z","log.origin":{"file.name":"instance/beat.go","file.line":510},"message":"filebeat stopped.","service.name":"filebeat","ecs.version":"1.6.0"}

ibra_013 · May 20, 2022, 11:10am

Hi @Rajesh119

We need some context, what are you trying to achieve?, please share the filebeat.yml file.

Rajesh119 · May 20, 2022, 11:16am

Hi @ibra_013
This is my .yml file

vim /etc/filebeat/filebeat.yml

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

# For more available modules and options, please see the filebeat.reference.yml sample
# configuration file.

# ============================== Filebeat inputs ===============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

# filestream is an input for collecting log messages from files.
- type: log

  # Unique ID among all inputs, an ID is required.
  #id: my-filestream-id

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /home/thapos/tomcat/tomcat8/logs/catalina.out
    #- c:\programdata\elasticsearch\logs\*

  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  #exclude_lines: ['^DBG']

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #prospector.scanner.exclude_files: ['.gz$']

  # Optional additional fields. These fields can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  #  level: debug
  #  review: 1

# ============================== Filebeat modules ==============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false


# ================================== General ===================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:
filebeat.registry.path: ${path.data}/.
# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]
setup.ilm.overwrite: true
# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
  host: "34.205.74.243:5601"

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  #host: "localhost:5601"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

# =============================== Elastic Cloud ================================

# These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["34.205.74.243:9200"]
  #index: "elastic"
  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  #username: "elastic"
  #password: "changeme"

# ------------------------------ Logstash Output -------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

# ================================== Logging ===================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publisher", "service".
#logging.selectors: ["*"]

# ============================= X-Pack Monitoring ==============================
# Filebeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#monitoring.enabled: false

# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Filebeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
#monitoring.cluster_uuid:

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:

# ============================== Instrumentation ===============================

# Instrumentation support for the filebeat.
#instrumentation:
    # Set to true to enable instrumentation of filebeat.
    #enabled: false

    # Environment in which filebeat is running on (eg: staging, production, etc.)
    #environment: ""

    # APM Server hosts to report instrumentation results to.
    #hosts:
    #  - http://localhost:8200

    # API Key for the APM Server(s).
    # If api_key is set then secret_token will be ignored.
    #api_key:

    # Secret token for the APM Server(s).
    #secret_token:


# ================================= Migration ==================================

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true

ibra_013 · May 20, 2022, 11:24am

Hi @Rajesh119

can you share the output of the below commands:

filebeat modules list | head

cat /etc/filebeat/modules.d/tomcat.yml

Rajesh119 · May 20, 2022, 11:41am

Hi @ibra_013
output:

filebeat modules list | head

Enabled:
tomcat

Disabled:
activemq
apache
auditd
aws
awsfargate
azure

cat /etc/filebeat/modules.d/tomcat.yml

# Module: tomcat
# Docs: https://www.elastic.co/guide/en/beats/filebeat/8.2/filebeat-module-tomcat.html

- module: tomcat
  log:
    enabled: true

    # Set which input to use between udp (default), tcp or file.
    # var.input: udp
    # var.syslog_host: localhost
    # var.syslog_port: 9501

    # Set paths for the log files when file input is used.
    # var.paths:
    #   - /var/log/tomcat/*.log

    # Toggle output of non-ECS fields (default true).
    # var.rsa_fields: true

    # Set custom timezone offset.
    # "local" (default) for system timezone.
    # "+02:00" for GMT+02:00
    # var.tz_offset: local

Rajesh119 · May 23, 2022, 7:39am

Hi @ibra_013
Please Help

ibra_013 · May 23, 2022, 9:36am

Hi @Rajesh119

Add these logging parameters to your filebeat and restart the filebeat service to see the logs.

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0640

and set this to false

Rajesh119 · May 23, 2022, 11:14am

Hi @ibra_013 First of all Thank you for your response.

filebeat -e -c /etc/filebeat/filebeat.yml

Output:

{"log.level":"info","@timestamp":"2022-05-23T11:12:06.718Z","log.origin":{"file.name":"instance/beat.go","file.line":685},"message":"Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T11:12:06.719Z","log.origin":{"file.name":"instance/beat.go","file.line":693},"message":"Beat ID: a985c9e2-fd39-42dd-9b1d-6c45a4629a2b","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T11:12:06.725Z","log.origin":{"file.name":"instance/beat.go","file.line":424},"message":"filebeat stopped.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-05-23T11:12:06.725Z","log.origin":{"file.name":"instance/beat.go","file.line":1038},"message":"Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).","service.name":"filebeat","ecs.version":"1.6.0"}
Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).

ibra_013 · May 23, 2022, 11:18am

Hi @Rajesh119
in order to ran this command you have to stop filebeat service

Rajesh119 · May 23, 2022, 11:21am

Hi @ibra_013 I stopped the filebeat and run the command

filebeat -e -c /etc/filebeat/filebeat.yml

Output:

{"log.level":"info","@timestamp":"2022-05-23T11:26:11.385Z","log.origin":{"file.name":"instance/beat.go","file.line":685},"message":"Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T11:26:11.385Z","log.origin":{"file.name":"instance/beat.go","file.line":693},"message":"Beat ID: a985c9e2-fd39-42dd-9b1d-6c45a4629a2b","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T11:26:11.390Z","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":124},"message":"Syscall filter successfully installed","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T11:26:11.390Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1063},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"/etc/filebeat","data":"/var/lib/filebeat","home":"/usr/share/filebeat","logs":"/var/log/filebeat"},"type":"filebeat","uuid":"a985c9e2-fd39-42dd-9b1d-6c45a4629a2b"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-23T11:26:11.390Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1072},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"045da3a1bb89944373c33332c18ca99ef6192df2","libbeat":"8.2.0","time":"2022-04-19T23:31:06.000Z","version":"8.2.0"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-23T11:26:11.391Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1075},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":1,"version":"go1.17.8"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-23T11:26:11.391Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1079},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-05-16T12:47:00Z","containerized":false,"name":"mail.thapos.net","ip":["127.0.0.1/8","::1/128","172.31.19.214/20","fe80::8bf:52ff:febd:8a4/64"],"kernel_version":"4.14.214-118.339.amzn1.x86_64","mac":["0a:bf:52:bd:08:a4"],"os":{"type":"linux","family":"redhat","platform":"amzn","name":"Amazon Linux AMI","version":"2018.03","major":2018,"minor":3,"patch":0},"timezone":"UTC","timezone_offset_sec":0,"id":"ef32a8ed995e86063ec99c115b0b9d78"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-23T11:26:11.392Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1108},"message":"Process info","service.name":"filebeat","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null},"cwd":"/root","exe":"/usr/share/filebeat/bin/filebeat","name":"filebeat","pid":8646,"ppid":8177,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2022-05-23T11:26:11.120Z"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-23T11:26:11.392Z","log.origin":{"file.name":"instance/beat.go","file.line":325},"message":"Setup Beat: filebeat; Version: 8.2.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T11:26:11.393Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":105},"message":"elasticsearch url: http://34.205.74.243:9200","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T11:26:11.393Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: mail.thapos.net","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T11:26:11.393Z","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":108},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T11:26:11.394Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":142},"message":"Starting metrics logging every 30s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T11:26:11.394Z","log.origin":{"file.name":"instance/beat.go","file.line":505},"message":"filebeat start running.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T11:26:11.394Z","log.origin":{"file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for '/var/lib/filebeat/filebeat'. Active transaction id=16","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T11:26:11.395Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":109},"message":"States Loaded from registrar: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T11:26:11.395Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":71},"message":"Loading Inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T11:26:11.395Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":117},"message":"starting input, keys present on the config: [filebeat.inputs.0.enabled filebeat.inputs.0.paths.0 filebeat.inputs.0.type]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-23T11:26:11.395Z","log.logger":"cfgwarn","log.origin":{"file.name":"log/input.go","file.line":89},"message":"DEPRECATED: Log input. Use Filestream input instead.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T11:26:11.395Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":171},"message":"Configured paths: [/home/thapos/tomcat/tomcat8/logs/catalina.out]","service.name":"filebeat","input_id":"c6698066-efc9-4dae-a774-e88b4af6ece2","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T11:26:11.395Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":148},"message":"Starting input (ID: 8813593274292630345)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T11:26:11.396Z","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":108},"message":"Enabled modules/filesets: tomcat (log)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T11:26:11.397Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":106},"message":"Loading and starting Inputs completed. Enabled inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T11:26:11.398Z","log.origin":{"file.name":"cfgfile/reload.go","file.line":164},"message":"Config reloader started","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T11:26:11.399Z","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":108},"message":"Enabled modules/filesets: tomcat (log)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-23T11:26:11.428Z","log.logger":"cfgwarn","log.origin":{"file.name":"registered_domain/registered_domain.go","file.line":61},"message":"BETA: The registered_domain processor is beta.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-23T11:26:11.428Z","log.logger":"cfgwarn","log.origin":{"file.name":"registered_domain/registered_domain.go","file.line":61},"message":"BETA: The registered_domain processor is beta.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-23T11:26:11.428Z","log.logger":"cfgwarn","log.origin":{"file.name":"registered_domain/registered_domain.go","file.line":61},"message":"BETA: The registered_domain processor is beta.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-23T11:26:11.428Z","log.logger":"cfgwarn","log.origin":{"file.name":"registered_domain/registered_domain.go","file.line":61},"message":"BETA: The registered_domain processor is beta.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-23T11:26:11.428Z","log.logger":"cfgwarn","log.origin":{"file.name":"registered_domain/registered_domain.go","file.line":61},"message":"BETA: The registered_domain processor is beta.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-23T11:26:11.428Z","log.logger":"cfgwarn","log.origin":{"file.name":"registered_domain/registered_domain.go","file.line":61},"message":"BETA: The registered_domain processor is beta.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T11:26:11.428Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":105},"message":"elasticsearch url: http://34.205.74.243:9200","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T11:26:11.435Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":105},"message":"add_cloud_metadata: hosting provider type detected as aws, metadata={\"cloud\":{\"account\":{\"id\":\"632467875476\"},\"availability_zone\":\"us-east-1c\",\"image\":{\"id\":\"ami-14c5486b\"},\"instance\":{\"id\":\"i-0e4a37307f16515d3\"},\"machine\":{\"type\":\"t2.small\"},\"provider\":\"aws\",\"region\":\"us-east-1\",\"service\":{\"name\":\"EC2\"}}}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T11:26:11.437Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":287},"message":"Attempting to connect to Elasticsearch version 8.2.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T11:26:11.441Z","log.origin":{"file.name":"cfgfile/reload.go","file.line":224},"message":"Loading of config files completed.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T11:26:11.441Z","log.origin":{"file.name":"udp/input.go","file.line":98},"message":"Starting UDP input","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T11:26:11.441Z","log.logger":"UDP","log.origin":{"file.name":"dgram/server.go","file.line":99},"message":"Started listening for UDP connection","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T11:26:41.395Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":30,"time":{"ms":30}},"total":{"ticks":170,"time":{"ms":170},"value":0},"user":{"ticks":140,"time":{"ms":140}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":12},"info":{"ephemeral_id":"24291d39-e901-4296-bb74-1a8ebf4a2310","uptime":{"ms":30084},"version":"8.2.0"},"memstats":{"gc_next":26556240,"memory_alloc":13811192,"memory_sys":37307400,"memory_total":60991840,"rss":132804608},"runtime":{"goroutines":34}},"filebeat":{"events":{"added":1,"done":1},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1,"starts":1},"reloads":1,"scans":1},"output":{"events":{"active":0},"type":"elasticsearch"},"pipeline":{"clients":2,"events":{"active":0,"filtered":1,"total":1},"queue":{"max_events":4096}}},"registrar":{"states":{"current":1,"update":1},"writes":{"success":1,"total":1}},"system":{"cpu":{"cores":1},"load":{"1":0,"15":0,"5":0,"norm":{"1":0,"15":0,"5":0}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-23T11:27:11.395Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":30},"total":{"ticks":170,"value":0},"user":{"ticks":140}},"handles":{"limit":{"hard":4096,"soft":1024},"open":12},"info":{"ephemeral_id":"24291d39-e901-4296-bb74-1a8ebf4a2310","uptime":{"ms":60084},"version":"8.2.0"},"memstats":{"gc_next":26556240,"memory_alloc":14048640,"memory_total":61229288,"rss":132804608},"runtime":{"goroutines":34}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"active":0}},"pipeline":{"clients":2,"events":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0,"15":0,"5":0,"norm":{"1":0,"15":0,"5":0}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-23T11:27:41.395Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":30},"total":{"ticks":170,"value":0},"user":{"ticks":140}},"handles":{"limit":{"hard":4096,"soft":1024},"open":11},"info":{"ephemeral_id":"24291d39-e901-4296-bb74-1a8ebf4a2310","uptime":{"ms":90084},"version":"8.2.0"},"memstats":{"gc_next":26556240,"memory_alloc":14296016,"memory_total":61476664,"rss":132804608},"runtime":{"goroutines":32}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"active":0}},"pipeline":{"clients":2,"events":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0,"15":0,"5":0,"norm":{"1":0,"15":0,"5":0}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-23T11:28:11.395Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":30},"total":{"ticks":180,"time":{"ms":10},"value":0},"user":{"ticks":150,"time":{"ms":10}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":11},"info":{"ephemeral_id":"24291d39-e901-4296-bb74-1a8ebf4a2310","uptime":{"ms":120084},"version":"8.2.0"},"memstats":{"gc_next":26556240,"memory_alloc":14522304,"memory_total":61702952,"rss":132804608},"runtime":{"goroutines":32}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"active":0}},"pipeline":{"clients":2,"events":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0,"15":0,"5":0,"norm":{"1":0,"15":0,"5":0}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-23T11:28:41.395Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":40,"time":{"ms":10}},"total":{"ticks":210,"time":{"ms":30},"value":0},"user":{"ticks":170,"time":{"ms":20}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":11},"info":{"ephemeral_id":"24291d39-e901-4296-bb74-1a8ebf4a2310","uptime":{"ms":150084},"version":"8.2.0"},"memstats":{"gc_next":23205408,"memory_alloc":11635952,"memory_total":61869104,"rss":123580416},"runtime":{"goroutines":32}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"active":0}},"pipeline":{"clients":2,"events":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0,"15":0,"5":0,"norm":{"1":0,"15":0,"5":0}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-23T11:29:11.395Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":40},"total":{"ticks":220,"time":{"ms":10},"value":0},"user":{"ticks":180,"time":{"ms":10}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":11},"info":{"ephemeral_id":"24291d39-e901-4296-bb74-1a8ebf4a2310","uptime":{"ms":180084},"version":"8.2.0"},"memstats":{"gc_next":23205408,"memory_alloc":11980504,"memory_total":62213656,"rss":123580416},"runtime":{"goroutines":32}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"active":0}},"pipeline":{"clients":2,"events":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0,"15":0,"5":0,"norm":{"1":0,"15":0,"5":0}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-23T11:29:41.395Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":40},"total":{"ticks":220,"value":0},"user":{"ticks":180}},"handles":{"limit":{"hard":4096,"soft":1024},"open":11},"info":{"ephemeral_id":"24291d39-e901-4296-bb74-1a8ebf4a2310","uptime":{"ms":210083},"version":"8.2.0"},"memstats":{"gc_next":23205408,"memory_alloc":12093976,"memory_total":62327128,"rss":123580416},"runtime":{"goroutines":32}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"active":0}},"pipeline":{"clients":2,"events":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.07,"15":0,"5":0.02,"norm":{"1":0.07,"15":0,"5":0.02}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-23T11:30:11.395Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":50,"time":{"ms":10}},"total":{"ticks":230,"time":{"ms":10},"value":0},"user":{"ticks":180}},"handles":{"limit":{"hard":4096,"soft":1024},"open":11},"info":{"ephemeral_id":"24291d39-e901-4296-bb74-1a8ebf4a2310","uptime":{"ms":240083},"version":"8.2.0"},"memstats":{"gc_next":23205408,"memory_alloc":12305776,"memory_total":62538928,"rss":123580416},"runtime":{"goroutines":32}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"active":0}},"pipeline":{"clients":2,"events":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.04,"15":0,"5":0.01,"norm":{"1":0.04,"15":0,"5":0.01}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-23T11:30:41.395Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":50},"total":{"ticks":250,"time":{"ms":20},"value":0},"user":{"ticks":200,"time":{"ms":20}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":11},"info":{"ephemeral_id":"24291d39-e901-4296-bb74-1a8ebf4a2310","uptime":{"ms":270084},"version":"8.2.0"},"memstats":{"gc_next":23112592,"memory_alloc":11619488,"memory_total":62813720,"rss":123580416},"runtime":{"goroutines":32}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"active":0}},"pipeline":{"clients":2,"events":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.02,"15":0,"5":0.01,"norm":{"1":0.02,"15":0,"5":0.01}}}},"ecs.version":"1.6.0"}}

ibra_013 · May 23, 2022, 12:38pm

hi @Rajesh119

on the tomcat module configure the path and comment the part of filebeat inputs

Rajesh119:

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

# filestream is an input for collecting log messages from files.
- type: log

  # Unique ID among all inputs, an ID is required.
  #id: my-filestream-id

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /home/thapos/tomcat/tomcat8/logs/catalina.out

Rajesh119 · May 23, 2022, 1:02pm

Hi @ibra_013
After comment the filebeat inputs it's shows like this:

filebeat -e -c /etc/filebeat/filebeat.yml

{"log.level":"info","@timestamp":"2022-05-23T12:58:11.675Z","log.origin":{"file.name":"instance/beat.go","file.line":685},"message":"Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T12:58:11.676Z","log.origin":{"file.name":"instance/beat.go","file.line":693},"message":"Beat ID: a985c9e2-fd39-42dd-9b1d-6c45a4629a2b","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T12:58:11.679Z","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":124},"message":"Syscall filter successfully installed","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T12:58:11.680Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1063},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"/etc/filebeat","data":"/var/lib/filebeat","home":"/usr/share/filebeat","logs":"/var/log/filebeat"},"type":"filebeat","uuid":"a985c9e2-fd39-42dd-9b1d-6c45a4629a2b"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-23T12:58:11.680Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1072},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"045da3a1bb89944373c33332c18ca99ef6192df2","libbeat":"8.2.0","time":"2022-04-19T23:31:06.000Z","version":"8.2.0"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-23T12:58:11.680Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1075},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":1,"version":"go1.17.8"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-23T12:58:11.680Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1079},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-05-16T12:47:00Z","containerized":false,"name":"mail.thapos.net","ip":["127.0.0.1/8","::1/128","172.31.19.214/20","fe80::8bf:52ff:febd:8a4/64"],"kernel_version":"4.14.214-118.339.amzn1.x86_64","mac":["0a:bf:52:bd:08:a4"],"os":{"type":"linux","family":"redhat","platform":"amzn","name":"Amazon Linux AMI","version":"2018.03","major":2018,"minor":3,"patch":0},"timezone":"UTC","timezone_offset_sec":0,"id":"ef32a8ed995e86063ec99c115b0b9d78"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-23T12:58:11.681Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1108},"message":"Process info","service.name":"filebeat","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null},"cwd":"/home/thapos/tomcat/tomcat8/logs","exe":"/usr/share/filebeat/bin/filebeat","name":"filebeat","pid":10266,"ppid":8177,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2022-05-23T12:58:11.400Z"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-23T12:58:11.681Z","log.origin":{"file.name":"instance/beat.go","file.line":325},"message":"Setup Beat: filebeat; Version: 8.2.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T12:58:11.682Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":105},"message":"elasticsearch url: http://34.205.74.243:9200","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T12:58:11.682Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: mail.thapos.net","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T12:58:11.682Z","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":108},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T12:58:11.683Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":142},"message":"Starting metrics logging every 30s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T12:58:11.683Z","log.origin":{"file.name":"instance/beat.go","file.line":505},"message":"filebeat start running.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T12:58:11.683Z","log.origin":{"file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for '/var/lib/filebeat/filebeat'. Active transaction id=20","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T12:58:11.684Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":109},"message":"States Loaded from registrar: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T12:58:11.684Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":71},"message":"Loading Inputs: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-05-23T12:58:11.684Z","log.origin":{"file.name":"cfgfile/reload.go","file.line":273},"message":"Error loading config from file '/etc/filebeat/modules.d/tomcat.yml', error invalid config: yaml: line 13: did not find expected key","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T12:58:11.684Z","log.origin":{"file.name":"beater/crawler.go","file.line":155},"message":"Stopping Crawler","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T12:58:11.684Z","log.origin":{"file.name":"beater/crawler.go","file.line":165},"message":"Stopping 0 inputs","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T12:58:11.684Z","log.origin":{"file.name":"beater/crawler.go","file.line":185},"message":"Crawler stopped","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T12:58:11.684Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":132},"message":"Stopping Registrar","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T12:58:11.685Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":166},"message":"Ending Registrar","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T12:58:11.685Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":137},"message":"Registrar stopped","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T12:58:11.688Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":192},"message":"Total metrics","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":20,"time":{"ms":20}},"total":{"ticks":130,"time":{"ms":130},"value":0},"user":{"ticks":110,"time":{"ms":110}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":16},"info":{"ephemeral_id":"16e301f9-8c16-425d-9554-1dc069e85ae5","uptime":{"ms":91},"version":"8.2.0"},"memstats":{"gc_next":22180720,"memory_alloc":13348432,"memory_sys":32850952,"memory_total":52250360,"rss":127102976},"runtime":{"goroutines":36}},"filebeat":{"events":{"active":0,"added":0,"done":0},"harvester":{"closed":0,"open_files":0,"running":0,"skipped":0,"started":0},"input":{"log":{"files":{"renamed":0,"truncated":0}},"netflow":{"flows":0,"packets":{"dropped":0,"received":0}}}},"libbeat":{"config":{"module":{"running":0,"starts":0,"stops":0},"reloads":0,"scans":0},"output":{"events":{"acked":0,"active":0,"batches":0,"dropped":0,"duplicates":0,"failed":0,"toomany":0,"total":0},"read":{"bytes":0,"errors":0},"type":"elasticsearch","write":{"bytes":0,"errors":0}},"pipeline":{"clients":0,"events":{"active":0,"dropped":0,"failed":0,"filtered":0,"published":0,"retry":0,"total":0},"queue":{"acked":0,"max_events":4096}}},"registrar":{"states":{"cleanup":0,"current":0,"update":0},"writes":{"fail":0,"success":0,"total":0}},"system":{"cpu":{"cores":1},"load":{"1":0,"15":0,"5":0,"norm":{"1":0,"15":0,"5":0}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-23T12:58:11.689Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":193},"message":"Uptime: 91.804282ms","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T12:58:11.689Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":160},"message":"Stopping metrics logging.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-23T12:58:11.689Z","log.origin":{"file.name":"instance/beat.go","file.line":510},"message":"filebeat stopped.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-05-23T12:58:11.689Z","log.origin":{"file.name":"instance/beat.go","file.line":1038},"message":"Exiting: Failed to start crawler: creating module reloader failed: loading configs: 1 error: invalid config: yaml: line 13: did not find expected key","service.name":"filebeat","ecs.version":"1.6.0"}
Exiting: Failed to start crawler: creating module reloader failed: loading configs: 1 error: invalid config: yaml: line 13: did not find expected key

cat /etc/filebeat/modules.d/tomcat.yml

# Module: tomcat
# Docs: https://www.elastic.co/guide/en/beats/filebeat/8.2/filebeat-module-tomcat.html

- module: tomcat
  log:
    enabled: true

    # Set which input to use between udp (default), tcp or file.
    # var.input: udp
    # var.syslog_host: localhost
    # var.syslog_port: 9501

    # Set paths for the log files when file input is used.
     var.paths:
       - /home/thapos/tomcat/tomcat8/logs/catalina.out

    # Toggle output of non-ECS fields (default true).
    # var.rsa_fields: true

    # Set custom timezone offset.
    # "local" (default) for system timezone.
    # "+02:00" for GMT+02:00
    # var.tz_offset: local

ibra_013 · May 23, 2022, 2:08pm

hi @Rajesh119

you have an issue on the filebeat yml file on line 13

Rajesh119 · May 23, 2022, 2:13pm

Hi @ibra_013
This is my filebeat.yml file. i didn't get why the error coming. that 13th line is filebeat inputs.

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

# For more available modules and options, please see the filebeat.reference.yml sample
# configuration file.

# ============================== Filebeat inputs ===============================

#filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

# filestream is an input for collecting log messages from files.
#- type: log

  # Unique ID among all inputs, an ID is required.
  #id: my-filestream-id

  # Change to true to enable this input configuration.
  #enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  #paths:
    #- /home/thapos/tomcat/tomcat8/logs/catalina.out
    #- c:\programdata\elasticsearch\logs\*

  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  #exclude_lines: ['^DBG']

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #prospector.scanner.exclude_files: ['.gz$']

  # Optional additional fields. These fields can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  #  level: debug
  #  review: 1

# ============================== Filebeat modules ==============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false


# ================================== General ===================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:
filebeat.registry.path: ${path.data}/.
# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]
#setup.ilm.overwrite: true
# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
  host: "34.205.74.243:5601"

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  #host: "localhost:5601"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

# =============================== Elastic Cloud ================================

# These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["34.205.74.243:9200"]
  #index: "elastic"
  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  #username: "elastic"
  #password: "changeme"

# ------------------------------ Logstash Output -------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

# ================================== Logging ===================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug
logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0640
# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publisher", "service".
#logging.selectors: ["*"]

# ============================= X-Pack Monitoring ==============================
# Filebeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#monitoring.enabled: false

# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Filebeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
#monitoring.cluster_uuid:

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:

# ============================== Instrumentation ===============================

# Instrumentation support for the filebeat.
#instrumentation:
    # Set to true to enable instrumentation of filebeat.
    #enabled: false

    # Environment in which filebeat is running on (eg: staging, production, etc.)
    #environment: ""

    # APM Server hosts to report instrumentation results to.
    #hosts:
    #  - http://localhost:8200

    # API Key for the APM Server(s).
    # If api_key is set then secret_token will be ignored.
    #api_key:

    # Secret token for the APM Server(s).
    #secret_token:


# ================================= Migration ==================================

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true

Rajesh119 · May 24, 2022, 3:43am

Hi @ibra_013
Please solve this issue i faced.

Rajesh119 · May 24, 2022, 5:21am

Please anyone help me regarding this issue. Iam new to this elk tool. Iam stucked here from last few days.