Hi, I'm brand new to the whole ELK stack, I've setup one in my company. I just wanted to know how best to clean up the logs, I'm getting a lot of system logs from the server and I don't want them. I just want the logs from our application. Is it possible to set headings and subheadings that point to those app logs, because at the moment, I've added logs into filebeat.yml and I've just added in tags and then saved those searches, would be nice to have the app logs subheadings down the side of Elasticsearch. Apologies for my question, I know I'm asking quite a lot and not sure if its even possible in Elasticsearch. Thanks.
Hi @eprop-marc,
glad to hear you're trying it out. The logs app at this point doesn't have a way to store sets of filters, but it's something we're thinking about. In the meantime a workaround could be to bookmark the logs app with the filters applied. They are stored in the URL and would be restored when you open the bookmark.
I just found the enhancement issue [Metrics & Logs UI] Saved Queries and Universal Kuery Bar · Issue #48290 · elastic/kibana · GitHub, which seem to be related to your question.
Hi Felix, thanks for your quick reply. With regards to the system logs Im getting from the server Elastic is hosted on, is there a way to disable those metrics please? My Kibana output gets spammed with them.
Yes, you should be able to tune the filebeat configuration to only pick up the logs you want. Could you provide more details about the filebeat configuration and the enabled modules? Which paths are you harvesting?
This is what I have so far in the filebeats.yml file. Our logs are mounted on AWS EFS, so we get them from /mnt/general which is mounted to EFS. It would be nice if these logs could be headings in Elastic instead of me tagging them, and then saving those searches. I haven't enabled any modules yet, however, I am looking to enabling the AWS module. Is it in the filebeats.yml file where I can take a lot of the system metrics out? Thanks
# ============================== Filebeat inputs ===============================
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
####Worker Logs####
#
- type: log
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /mnt/general/logs/worker/laravel*.log
tags: ["Laravel Worker Logs"]
- type: log
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /mnt/general/logs/worker/horizon*.log
tags: ["Horizon Worker Logs"]
- type: log
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /mnt/general/logs/worker/feeds*.log
tags: ["Feeds Worker Logs"]
####Member Logs####
#
- type: log
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /mnt/general/logs/member/laravel*.log
tags: ["Horizon Member Logs"]
####Web Logs#####
#
- type: log
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /mnt/general/logs/web/laravel*.log
tags: ["Horizon Web Logs"]
####Cron Logs####
#
- type: log
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /mnt/general/logs/cron/cron.log
tags: ["Cron Logs"]
#- c:\programdata\elasticsearch\logs\*
Thanks for providing these details. The system log might be enabled by default. Can you check what metricbeat modules list shows as "enabled"? Alternatively, could you check in the modules.d directory in the metricbeat directory which modules' configs don't have a .disabled suffix?
Ahh I have disabled the system module, thanks. Just having some issues with enabling the Nginx module. Do I configure the nginx module in the modules.d direcotry, or in the filebeat.yml file?
Also, is there a way to just add log path and the message instead of Document at the top near Time please?
That's up to you, but it would probably be cleaner to configure it in its separate file.
The screenshot looks like it's taken in Kibana's "Discover" app? If so, you can add the specific columns via the sidebar on the left or by expanding a document and clicking the corresponding icon in the matching row of the field table: Discover | Kibana Guide [8.11] | Elastic
Ahh its the available fields on the left, thanks. Is there a way to clean up and delete the other available fields?
I don't think there is. You can only collapse the sidebar, I think. But you can also prepare your set of desired columns and store it as a "saved search" in Discover if that helps.
Ahh great thanks
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.