I'm running both Filebeat 8.3.3 and Metricbeat 8.3.3 on my RHEL 7.9 server, and sending the logs to another server which is hosting Elasticsearch and Kibana.
My filebeat is sending syslog to the ES (I'm simply using this as a connectivity test, I'll be sending Netflow to ES later), and Metricbeat is sending the server's system stats to ES. I want to also send Filebeat's logs to ES using Metricbeat, i.e. the logs written to
I have configured
beat-xpack.yml in Metricbeat, and everything seems to be fine. My question is how do I check if the filebeat logs are in the ES? The default Kibana dashboards that come with Metricbeat seem to only show the server system stats.
I don't see any index related to the filebeat logs under the Discover app. In fact, I don't think any indices are created other than the reserved indices for monitoring and the
metricbeat-* data views.
Could you run
GET _cat/indices from dev tools and share the results? Also would be good to see the configuration as text of filebeat and metricbeat. Please make sure to remove any confidential or sensitive data in what you post (passwords, etc).
I'm manually typing in the info you requested, so it's kind of a summarized version.
My results for
GET _cat/indices are a listing of the following indices:
My config files are as follows (mostly unchanged from default/commented values):
beat-xpack.yml in /etc/metricbeat/modules.d
I'm running 2 instances of filebeat in the same host machine, hence, the 2 hosts below.
- module: beat
Thanks for that. Do you see anything in Observability > Streams when you are in Kibana?
Also, do you see the document count of
for instance increase over time (when you run the _cat/indices)?
I do see a lot of records in Stream (under Logs), and all
And yes, the document count of
.ds-.filebeat-8.3.3-<date>-000001 increases over time, slowly.
Why do you want to use Metricbeat to send Filebeat Logs? That is not a normal pattern? What are you trying to accomplish? Metricbeat is not purpose-built to send logs, Filebeat is.
You can use Filebeat to send its own logs.
In the filebeat.yml just setup up
# filestream is an input for collecting log messages from files.
- type: filestream
# Path to filebeat logs
You can also just configure filebeat to write it's logs to
syslog then your system logs will pick them up. See here
When true, writes all logging output to the syslog.
Thank you! that's exactly what I was looking for!
I guess in my head, I was thinking filebeat was used to ingest external logs into ES, while metricbeat was used for internal logs. Thanks for clearing that up for me!
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.