Tomcat logs in filebeat

Prasanth_V · June 29, 2021, 8:05am

If I add a tomcat log paths in tomcat.yml..why is it not showing in discover section..could you pls give the solution on this case??

legoguy1000 · June 30, 2021, 12:55am

Have you enabled the tomcat module?

Prasanth_V · June 30, 2021, 5:04am

Yeah, I have been enabled. After that paste the log path in yaml file. It's not showing in discover tab...this is exact issue on this case..could you pls suggest it??

legoguy1000 · June 30, 2021, 10:54am

Can you please post the tomcat.yml and filebeat.yml files? Also can you post the filebeat logs?

Prasanth_V · July 1, 2021, 5:30am

'''
tomcat.yml:

module: tomcat

Access logs

access:
enabled: true

Set custom paths for the log files. If left empty,

Filebeat will choose the paths depending on your OS.

tomcat.log.var.paths:
- /home/tomcat/logs/*.txt

Error logs

error:
enabled: true

Set custom paths for the log files. If left empty,

Filebeat will choose the paths depending on your OS.

tomcat.log.var.paths:
- /home/tomcat/logs/*.out

Filebeat.yml:

###################### Filebeat Configuration Example #########################

This file is an example configuration file highlighting only the most common

options. The filebeat.reference.yml file from the same directory contains all the

supported options with more comments. You can use it as a reference.

You can find the full configuration reference here:

Filebeat Reference | Elastic

For more available modules and options, please see the filebeat.reference.yml sample

configuration file.

============================== Filebeat inputs ===============================

filebeat.inputs:

Each - is an input. Most options can be set at the input level, so

you can use different inputs for various configurations.

Below are the input specific configurations.

type: log

Change to true to enable this input configuration.

enabled: true

Paths that should be crawled and fetched. Glob based paths.

paths:
- /home/tomcat/logs/*.txt
- /home/tomcat/logs/*.out
- /var/log/*.log
  #- c:\programdata\elasticsearch\logs*
Exclude lines. A list of regular expressions to match. It drops the lines that are

matching any regular expression from the list.

#exclude_lines: ['^DBG']

Include lines. A list of regular expressions to match. It exports the lines that are

matching any regular expression from the list.

#include_lines: ['^ERR', '^WARN']

Exclude files. A list of regular expressions to match. Filebeat drops the files that

are matching any regular expression from the list. By default, no files are dropped.

#exclude_files: ['.gz$']

Optional additional fields. These fields can be freely picked

to add additional information to the crawled log files for filtering

#fields:

level: debug

review: 1

Multiline options

Multiline can be used for log messages spanning multiple lines. This is common

for Java Stack Traces or C-Line Continuation

The regexp Pattern that has to be matched. The example pattern matches all lines starting with [

#multiline.pattern: ^[

Defines if the pattern set under pattern should be negated or not. Default is false.

#multiline.negate: false

Match can be set to "after" or "before". It is used to define if lines should be append to a pattern

that was (not) matched before or after or as long as a pattern is not matched based on negate.

Note: After is the equivalent to previous and before is the equivalent to to next in Logstash

#multiline.match: after

filestream is an experimental input. It is going to replace log input in the future.

type: filestream

Change to true to enable this input configuration.

enabled: false

Paths that should be crawled and fetched. Glob based paths.

paths:
- /var/log/*.log
  #- c:\programdata\elasticsearch\logs*
Exclude lines. A list of regular expressions to match. It drops the lines that are

matching any regular expression from the list.

#exclude_lines: ['^DBG']

Include lines. A list of regular expressions to match. It exports the lines that are

matching any regular expression from the list.

#include_lines: ['^ERR', '^WARN']

Exclude files. A list of regular expressions to match. Filebeat drops the files that

are matching any regular expression from the list. By default, no files are dropped.

#prospector.scanner.exclude_files: ['.gz$']

Optional additional fields. These fields can be freely picked

to add additional information to the crawled log files for filtering

#fields:

level: debug

review: 1

============================== Filebeat modules ==============================

filebeat.config.modules:

Glob pattern for configuration loading

path: ${path.config}/modules.d/*.yml

Set to true to enable config reloading

reload.enabled: true

Period on which files under path should be checked for changes

#reload.period: 10s

======================= Elasticsearch template setting =======================

setup.template.settings:
index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: false

================================== General ===================================

The name of the shipper that publishes the network data. It can be used to group

all the transactions sent by a single shipper in the web interface.

#name:

The tags of the shipper are included in their own field with each

transaction published.

tags: ["filebeat"]

Optional fields that you can specify to add additional information to the

output.

#fields:

env: staging

================================= Dashboards =================================

These settings control loading the sample dashboards to the Kibana index. Loading

the dashboards is disabled by default and can be enabled either by setting the

options here or by using the `setup` command.

#setup.dashboards.enabled: false

The URL from where to download the dashboards archive. By default this URL

has a value which is computed based on the Beat name and version. For released

versions, this URL points to the dashboard archive on the artifacts.elastic.co

website.

#setup.dashboards.url:

=================================== Kibana ===================================

Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.

This requires a Kibana endpoint configuration.

#setup.kibana:

Kibana Host

Scheme and port can be left out and will be set to the default (http and 5601)

In case you specify and additional path, the scheme is required: http://localhost:5601/path

IPv6 addresses should always be defined as: https://[2001:db8::1]:5601

#host: "localhost:5601"

Kibana Space ID

ID of the Kibana Space into which the dashboards should be loaded. By default,

the Default Space will be used.

#space.id:

=============================== Elastic Cloud ================================

These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).

The cloud.id setting overwrites the `output.elasticsearch.hosts` and

`setup.kibana.host` options.

You can find the `cloud.id` in the Elastic Cloud web UI.

#cloud.id:

The cloud.auth setting overwrites the `output.elasticsearch.username` and

`output.elasticsearch.password` settings. The format is `<user>:<pass>`.

#cloud.auth:

================================== Outputs ===================================

Configure what output to use when sending the data collected by the beat.

---------------------------- Elasticsearch Output ----------------------------

#output.elasticsearch:

Array of hosts to connect to.

#hosts: ["localhost:9200"]

Protocol - either `http` (default) or `https`.

#protocol: "https"

Authentication credentials - either API key or username/password.

#api_key: "id:api_key"
#username: "elastic"
#password: "changeme"

------------------------------ Logstash Output -------------------------------

output.logstash:

The Logstash hosts

hosts: ["IP Address:5044"]

Optional SSL. By default is off.

List of root certificates for HTTPS server verifications

#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

Certificate for SSL client authentication

#ssl.certificate: "/etc/pki/client/cert.pem"

Client Certificate Key

#ssl.key: "/etc/pki/client/cert.key"

Filebeat Logs:

Jun 30 16:38:52 tomcat filebeat[1093]: 2021-06-30T16:38:52.809+0530 Jun 30 16:38:52 tomcat filebeat[1093]: 2021-06-30T16:38:52.810+0530 Jun 30 16:38:52 tomcat filebeat[1093]: 2021-06-30T16:38:52.810+0530 Jun 30 16:38:52 tomcat filebeat[1093]: 2021-06-30T16:38:52.811+0530 Jun 30 16:38:52 tomcat filebeat[1093]: 2021-06-30T16:38:52.811+0530 Jun 30 16:38:52 tomcat filebeat[1093]: 2021-06-30T16:38:52.811+0530 Jun 30 16:38:52 tomcat filebeat[1093]: 2021-06-30T16:38:52.813+0530 Jun 30 16:38:52 tomcat filebeat[1093]: 2021-06-30T16:38:52.853+0530 Jun 30 16:38:52 tomcat filebeat[1093]: 2021-06-30T16:38:52.853+0530 Jun 30 16:38:52 tomcat filebeat[1093]: 2021-06-30T16:38:52.854+0530 Jun 30 16:38:52 tomcat filebeat[1093]: 2021-06-30T16:38:52.854+0530 Jun 30 16:38:53 tomcat filebeat[1093]: 2021-06-30T16:38:53.100+0530 Jun 30 16:38:53 tomcat filebeat[1093]: 2021-06-30T16:38:53.101+0530 Jun 30 16:38:53 tomcat filebeat[1093]: 2021-06-30T16:38:53.101+0530 Jun 30 16:38:53 tomcat filebeat[1093]: 2021-06-30T16:38:53.101+0530 Jun 30 16:38:55 tomcat filebeat[1093]: 2021-06-30T16:38:55.807+0530 Jun 30 16:39:03 tomcat filebeat[1093]: 2021-06-30T16:39:03.104+0530 Jun 30 16:39:03 tomcat filebeat[1093]: 2021-06-30T16:39:03.106+0530 Jun 30 16:39:03 tomcat filebeat[1093]: 2021-06-30T16:39:03.335+0530 Jun 30 16:39:03 tomcat filebeat[1093]: 2021-06-30T16:39:03.384+0530 Jun 30 16:39:03 tomcat filebeat[1093]: 2021-06-30T16:39:03.387+0530 Jun 30 16:39:03 tomcat filebeat[1093]: 2021-06-30T16:39:03.389+0530 Jun 30 16:39:04 tomcat filebeat[1093]: 2021-06-30T16:39:04.390+0530 Jun 30 16:39:04 tomcat filebeat[1093]: 2021-06-30T16:39:04.390+0530 Jun 30 16:39:04 tomcat filebeat[1093]: 2021-06-30T16:39:04.390+0530 Jun 30 16:39:04 tomcat filebeat[1093]: 2021-06-30T16:39:04.391+0530 INFO [beat] instance/beat.go:1041 Process info {"system_info": {"process"
INFO instance/beat.go:304 Setup Beat: filebeat; Version: 7.12.1
INFO [publisher] pipeline/module.go:113 Beat name: tomcat
WARN beater/filebeat.go:178 Filebeat is unable to load the Ingest Node pipelines for th
INFO [monitoring] log/log.go:117 Starting metrics logging every 30s
INFO instance/beat.go:468 filebeat start running.
INFO memlog/store.go:119 Loading data file of '/var/lib/filebeat/registry/filebeat' suc
INFO memlog/store.go:124 Finished loading transaction log file for '/var/lib/filebeat/
WARN beater/filebeat.go:381 Filebeat is unable to load the Ingest Node pipelines for th
INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 127
INFO [crawler] beater/crawler.go:71 Loading Inputs: 2
INFO log/input.go:157 Configured paths: [/home/tomcat/logs/localhost_access.txt
INFO [crawler] beater/crawler.go:141 Starting input (ID: 11010157828768882092)
INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enab
INFO cfgfile/reload.go:164 Config reloader started
INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:101 add_cloud
INFO log/input.go:157 Configured paths: [/]
INFO log/input.go:157 Configured paths: [/home/tomcat/logs/catalina.out]
INFO log/input.go:157 Configured paths: [/home/tomcat/logs/localhost_access.txt]
INFO log/input.go:157 Configured paths: [/var/log/auth.log* /var/log/secure*]
INFO log/input.go:157 Configured paths: [/home/tomcat/logs/catalina.out]
INFO log/harvester.go:302 Harvester started for file: /var/log/secure
INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(as
INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
INFO [publisher] pipeline/retry.go:223 done
INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(as

'''

legoguy1000 · July 1, 2021, 11:03am

Can you please repost the configs using the code blocks. Its very hard to read without it.
3x <Code> 3x

system · July 29, 2021, 11:03am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Tomcat logs in filebeat

Access logs

Set custom paths for the log files. If left empty,

Filebeat will choose the paths depending on your OS.

Error logs

Set custom paths for the log files. If left empty,

Filebeat will choose the paths depending on your OS.

This file is an example configuration file highlighting only the most common

options. The filebeat.reference.yml file from the same directory contains all the

supported options with more comments. You can use it as a reference.

You can find the full configuration reference here:

Filebeat Reference | Elastic

For more available modules and options, please see the filebeat.reference.yml sample

configuration file.

============================== Filebeat inputs ===============================

Each - is an input. Most options can be set at the input level, so

you can use different inputs for various configurations.

Below are the input specific configurations.

Change to true to enable this input configuration.

Paths that should be crawled and fetched. Glob based paths.

Exclude lines. A list of regular expressions to match. It drops the lines that are

matching any regular expression from the list.

Include lines. A list of regular expressions to match. It exports the lines that are

matching any regular expression from the list.

Exclude files. A list of regular expressions to match. Filebeat drops the files that

are matching any regular expression from the list. By default, no files are dropped.

Optional additional fields. These fields can be freely picked

to add additional information to the crawled log files for filtering

level: debug

review: 1

Multiline options

Multiline can be used for log messages spanning multiple lines. This is common

for Java Stack Traces or C-Line Continuation

The regexp Pattern that has to be matched. The example pattern matches all lines starting with [

Defines if the pattern set under pattern should be negated or not. Default is false.

Match can be set to "after" or "before". It is used to define if lines should be append to a pattern

that was (not) matched before or after or as long as a pattern is not matched based on negate.

Note: After is the equivalent to previous and before is the equivalent to to next in Logstash

filestream is an experimental input. It is going to replace log input in the future.

Change to true to enable this input configuration.

Paths that should be crawled and fetched. Glob based paths.

Exclude lines. A list of regular expressions to match. It drops the lines that are

matching any regular expression from the list.

Include lines. A list of regular expressions to match. It exports the lines that are

matching any regular expression from the list.

Exclude files. A list of regular expressions to match. Filebeat drops the files that

are matching any regular expression from the list. By default, no files are dropped.

Optional additional fields. These fields can be freely picked

to add additional information to the crawled log files for filtering

level: debug

review: 1

============================== Filebeat modules ==============================

Glob pattern for configuration loading

Set to true to enable config reloading

Period on which files under path should be checked for changes

======================= Elasticsearch template setting =======================

================================== General ===================================

The name of the shipper that publishes the network data. It can be used to group

all the transactions sent by a single shipper in the web interface.

The tags of the shipper are included in their own field with each

transaction published.

Optional fields that you can specify to add additional information to the

output.

env: staging

================================= Dashboards =================================

These settings control loading the sample dashboards to the Kibana index. Loading

the dashboards is disabled by default and can be enabled either by setting the

options here or by using the setup command.

The URL from where to download the dashboards archive. By default this URL

has a value which is computed based on the Beat name and version. For released

versions, this URL points to the dashboard archive on the artifacts.elastic.co

website.

=================================== Kibana ===================================

Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.

This requires a Kibana endpoint configuration.

Kibana Host

Scheme and port can be left out and will be set to the default (http and 5601)

In case you specify and additional path, the scheme is required: http://localhost:5601/path

IPv6 addresses should always be defined as: https://[2001:db8::1]:5601

Kibana Space ID

options here or by using the `setup` command.

The cloud.id setting overwrites the `output.elasticsearch.hosts` and

`setup.kibana.host` options.

You can find the `cloud.id` in the Elastic Cloud web UI.

The cloud.auth setting overwrites the `output.elasticsearch.username` and

`output.elasticsearch.password` settings. The format is `<user>:<pass>`.

Protocol - either `http` (default) or `https`.