Using Metricbeat to send Filebeat logs to ES

[hjazz6]

Hi,

I'm running both Filebeat 8.3.3 and Metricbeat 8.3.3 on my RHEL 7.9 server, and sending the logs to another server which is hosting Elasticsearch and Kibana.

My filebeat is sending syslog to the ES (I'm simply using this as a connectivity test, I'll be sending Netflow to ES later), and Metricbeat is sending the server's system stats to ES. I want to also send Filebeat's logs to ES using Metricbeat, i.e. the logs written to /var/log/filebeat.

I have configured beat-xpack.yml in Metricbeat, and everything seems to be fine. My question is how do I check if the filebeat logs are in the ES? The default Kibana dashboards that come with Metricbeat seem to only show the server system stats.

I don't see any index related to the filebeat logs under the Discover app. In fact, I don't think any indices are created other than the reserved indices for monitoring and the filebeat-*, metricbeat-* data views.

Thank you.

[Wave (Andrew Sullivan)]

Hi @hjazz6,

Could you run GET _cat/indices from dev tools and share the results? Also would be good to see the configuration as text of filebeat and metricbeat. Please make sure to remove any confidential or sensitive data in what you post (passwords, etc).

[hjazz6]

I'm manually typing in the info you requested, so it's kind of a summarized version.

My results for GET _cat/indices are a listing of the following indices:

.monitoring-es-7-<date>
.monitoring-kibana-7-<date>
.monitoring-beats-7-<date>
.ds-.monitoring-es-8-mb-<date>-000001
.ds-.monitoring-kibana-8-mb-<date>-000001
.ds-.monitoring-beats-8-mb-<date>-000001
.ds-.filebeat-8.3.3-<date>-000001
.ds-.metricbeat-8.3.3-<date>-000001

My config files are as follows (mostly unchanged from default/commented values):

filebeat.yml

queue.mem.events: 64000
queue.mem.flush.min_events: 4000

setup.kibana:
    host: "http://x.x.x.x:5601"

output.elasticsearch:
    hosts: ["x.x.x.x:9200"]
    username: "elastic"
    password: "xxxxx"
    bulk_max_size: 4000
    worker: 8

path.logs: /var/log/filebeat

monitoring.enabled: true
monitoring.elasticsearch:
    username: beats_system
    password: xxxxx

http.enabled: true
http.port: 5066

metricbeat.yml

setup.dashboards.enabled: true

setup.kibana:
    host: "http://x.x.x.x:5601"

output.elasticsearch:
    hosts: ["x.x.x.x:9200"]
    username: "elastic"
    password: "xxxxx"

path.logs: /var/log/metricbeat

monitoring.enabled: true
monitoring.elasticsearch:
    username: remote_monitoring_user
    password: xxxxx

beat-xpack.yml in /etc/metricbeat/modules.d
I'm running 2 instances of filebeat in the same host machine, hence, the 2 hosts below.

- module: beat
  xpack.enabled: true
  period: 10s
  hosts: ["http://localhost:5066","http://localhost:5067"]
  username: beats_system
  password: "xxxxx"

[Wave (Andrew Sullivan)]

Thanks for that. Do you see anything in Observability > Streams when you are in Kibana?
Also, do you see the document count of

@hjazz6

.ds-.filebeat-8.3.3-<date>-000001

for instance increase over time (when you run the _cat/indices)?

[hjazz6]

I do see a lot of records in Stream (under Logs), and all event.dataset are system.syslog.

And yes, the document count of .ds-.filebeat-8.3.3-<date>-000001 increases over time, slowly.

[stephenb (Stephen Brown)]

@hjazz6

I want to also send Filebeat's logs to ES using Metricbeat,

Why do you want to use Metricbeat to send Filebeat Logs? That is not a normal pattern? What are you trying to accomplish? Metricbeat is not purpose-built to send logs, Filebeat is.

You can use Filebeat to send its own logs.

In the filebeat.yml just setup up

# filestream is an input for collecting log messages from files.
- type: filestream
  id: my-filebeat-logs-stream-id
  enabled: true
  # Path to filebeat logs
  paths:
    - /var/log/filebeat/*.log

You can also just configure filebeat to write it's logs to syslog then your system logs will pick them up. See here

logging.to_syslog

When true, writes all logging output to the syslog.

[hjazz6]

Thank you! that's exactly what I was looking for!

I guess in my head, I was thinking filebeat was used to ingest external logs into ES, while metricbeat was used for internal logs. Thanks for clearing that up for me!