Metricbeats logging into /var/log/messages

Elastic Observability Metrics
1

Hello! I'm trying to redirect my metricbeats logs to /var/log/metricbeats instead of /var/log/messages.

I'm using metricbeat 8.0.0 and I've configured the metricbeat.yml logging section to this:

#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/metricbeat
  name: metricbeat
  keepfiles: 7
  permissions: 0640
logging.to_stderr: false
logging.to_syslog: false

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]

At this point, there are still logs being recorded in /var/log/messages. Does anyone knows how to completely separate these logs from syslog?

Sep 27 22:50:35 adlg3625 selogrd[14026]: Unable to communicate with [adlg3804.na.pg.com: RPC: Success#012]
Sep 27 22:51:03 adlg3625 metricbeat: 2022-09-27T22:51:03.012-0400#011INFO#011[monitoring]#011log/log.go:145#011Non-zero metrics in the last 30s#011{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":12040,"time":{"ms":1158}},"total":{"ticks":195270,"time":{"ms":1158},"value":195270},"user":{"ticks":183230}},"handles":{"limit":{"hard":16384,"soft":16384},"open":11},"info":{"ephemeral_id":"8468961d-3bae-458b-9ab3-d078a8bd5a98","uptime":{"ms":570062}},"memstats":{"gc_next":28366128,"memory_alloc":24996480,"memory_total":25250714968,"rss":229376},"runtime":{"goroutines":265}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":38,"batches":15,"total":38}},"outputs":{"kafka":{"bytes_read":1622,"bytes_write":22189}},"pipeline":{"clients":56,"events":{"active":0,"published":38,"total":38},"queue":{"acked":38}}},"metricbeat":{"vsphere":{"virtualmachinereplication":{"events":38,"success":38}}},"system":{"load":{"1":0.12,"15":0.58,"5":0.53,"norm":{"1":0.015,"15":0.0725,"5":0.0663}}}}}}
Sep 27 22:51:25 adlg3625 selogrd[14026]: Unable to communicate with [adlg3804.na.pg.com: RPC: Success#012]
Sep 27 22:51:33 adlg3625 metricbeat: 2022-09-27T22:51:33.011-0400#011INFO#011[monitoring]#011log/log.go:145#011Non-zero metrics in the last 30s#011{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":12180,"time":{"ms":131}},"total":{"ticks":195410,"time":{"ms":131},"value":195410},"user":{"ticks":183230}},"handles":{"limit":{"hard":16384,"soft":16384},"open":10},"info":{"ephemeral_id":"8468961d-3bae-458b-9ab3-d078a8bd5a98","uptime":{"ms":600064}},"memstats":{"gc_next":26454000,"memory_alloc":22434592,"memory_total":25261672056},"runtime":{"goroutines":262}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":16,"batches":4,"total":16}},"outputs":{"kafka":{"bytes_read":848,"bytes_write":10889}},"pipeline":{"clients":56,"events":{"active":0,"published":16,"total":16},"queue":{"acked":16}}},"metricbeat":{"vsphere":{"virtualmachinereplication":{"events":16,"success":16}}},"system":{"load":{"1":0.07,"15":0.56,"5":0.48,"norm":{"1":0.0088,"15":0.07,"5":0.06}}}}}}
Sep 27 22:52:03 adlg3625 metricbeat: 2022-09-27T22:52:03.011-0400#011INFO#011[monitoring]#011log/log.go:145#011Non-zero metrics in the last 30s#011{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":12180,"time":{"ms":6}},"total":{"ticks":195410,"time":{"ms":6},"value":195410},"user":{"ticks":183230}},"handles":{"limit":{"hard":16384,"soft":16384},"open":10},"info":{"ephemeral_id":"8468961d-3bae-458b-9ab3-d078a8bd5a98","uptime":{"ms":630063}},"memstats":{"gc_next":26454000,"memory_alloc":22816344,"memory_total":25262053808},"runtime":{"goroutines":262}},"libbeat":{"config":{"module":{"running":0}},"outputs":{"kafka":{"bytes_read":520,"bytes_write":41}},"pipeline":{"clients":56,"events":{"active":0}}},"system":{"load":{"1":0.12,"15":0.55,"5":0.45,"norm":{"1":0.015,"15":0.0688,"5":0.0563}}}}}}
Sep 27 22:52:15 adlg3625 selogrd[14026]: Unable to communicate with [adlg3804.na.pg.com: RPC: Success#012]
Sep 27 22:52:33 adlg3625 metricbeat: 2022-09-27T22:52:33.011-0400#011INFO#011[monitoring]#011log/log.go:145#011Non-zero metrics in the last 30s#011{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":12180,"time":{"ms":4}},"total":{"ticks":195410,"time":{"ms":4},"value":195410},"user":{"ticks":183230}},"handles":{"limit":{"hard":16384,"soft":16384},"open":10},"info":{"ephemeral_id":"8468961d-3bae-458b-9ab3-d078a8bd5a98","uptime":{"ms":660062}},"memstats":{"gc_next":26454000,"memory_alloc":23028696,"memory_total":25262266160},"runtime":{"goroutines":262}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":56,"events":{"active":0}}},"system":{"load":{"1":0.14,"15":0.54,"5":0.42,"norm":{"1":0.0175,"15":0.0675,"5":0.0525}}}}}}
Sep 27 22:52:36 adlg3625 selogrd[14026]: Unable to communicate with [adlg3804.na.pg.com: RPC: Success#012]```
2

Hi @justinepaulpadayao

How are you starting metricbeat with systemctl / systemd

Did you see this?

When Filebeat is running on a Linux system with systemd, it uses by default the -e command line option, that makes it write all the logging output to stderr so it can be captured by journald. Other outputs are disabled. See Filebeat and systemd to know more and learn how to change this.

I think you just need to follow the example and create a log.conf that looks like this (I think this gets rid the -e but I don't have a filebeat installed on linux with a package handy... but this should point you in the right direction, then I think it will honor your logging settings.

[Service]
Environment="BEAT_LOG_OPTS=
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.