Filebeat Startup problem

Hello
Env description :
I have a cluster of two nodes with a Jetty app.
The application generate log on the file /var/log/jetty/jetty.log
The log rotation is handle by logrotate avery day at 6:30.
On both I run filebeat with the same configuration:

filebeat.inputs:

- type: log
enabled: true
multiline:
pattern: '^[?[0-9]{4}-[0-9]{2}-[0-9]{2}|^[0-9]{4}/[0-9]{2}/[0-9]{2}|^ts=[0-9]{4}-[0-9]{2}-[0-9]{2}|^[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3} - - [[0-9]{1,2}/\w{3}/[0-9]{4}:'
negate: true
match: after
paths:
- /var/logs/jetty/*.log
fields_under_root: true
fields:
application: "jetty"
function: "str"
environment: "prod"

filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false

setup.template.settings:
index.number_of_shards: 1

setup.kibana:
host: "supervision.XXXXXXXX.fr:XXXX"

output:
logstash:
hosts: ["supervision.XXXXXXXX.fr:XXXX"]
username: "XXXXXXXXXXXXXXXXXXX"
password: "XXXX"

processors:
- add_host_metadata: ~
# - dissect:
# when:
# message: "^[?[0-9]{4}-[0-9]{2}-[0-9]{2}"
# tokenizer: "%{timestamp->} : %{level}"
# field: "message"
# target_prefix: "dissect"
# %{TIMESTAMP_ISO8601:timestamp} : %{LOGLEVEL:level} - %{WORD:service} | %{GREEDYDATA:logmessage}

and with the logstash module configuration /etc/filebeat/modules.d/logstash.yml :

- module: logstash
# logs
log:
enabled: true

# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:

# Slow logs
slowlog:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:

But only on one of them I have a consumption problem.

First this morning I observe that the logs stop to be sent to logstash since 6:30, and the Filebeat process consume lot of memory and CPU, so I decide to restart it, but it does not restart to log file processing.

Any ideas ?
Do you want more detail ?

Could you provide the output from the Filebeat process and/or it's log file?

2019-12-27T09:59:16.933+0100 INFO crawler/crawler.go:72 Loading Inputs: 1
2019-12-27T09:59:16.933+0100 INFO log/input.go:152 Configured paths: [/var/log/jetty/logs/*.log]
2019-12-27T09:59:16.933+0100 INFO input/input.go:114 Starting input of type: log; ID: 6813183259199911499
2019-12-27T09:59:16.935+0100 INFO log/harvester.go:251 Harvester started for file: /var/log/jetty/logs/jetty.log
2019-12-27T09:59:16.964+0100 INFO log/input.go:152 Configured paths: [/var/log/logstash/logstash-plain*.log]
2019-12-27T09:59:16.964+0100 INFO log/input.go:152 Configured paths: [/var/log/logstash/logstash-slowlog-plain*.log]
2019-12-27T09:59:16.964+0100 INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1
2019-12-27T09:59:16.964+0100 INFO cfgfile/reload.go:171 Config reloader started
2019-12-27T09:59:16.965+0100 INFO log/input.go:152 Configured paths: [/var/log/logstash/logstash-plain*.log]
2019-12-27T09:59:16.966+0100 INFO log/input.go:152 Configured paths: [/var/log/logstash/logstash-slowlog-plain*.log]
2019-12-27T09:59:16.966+0100 INFO input/input.go:114 Starting input of type: log; ID: 10399321948351278122
2019-12-27T09:59:16.966+0100 INFO input/input.go:114 Starting input of type: log; ID: 12720006207850947031
2019-12-27T09:59:16.966+0100 INFO cfgfile/reload.go:226 Loading of config files completed.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.