How to write queries to pull a specific data in ELK

[Kumbum]

Hi Team,

How to write queries to pull specific information in the custom dashboards. I want to pull specific information in the drop-down list in the custom dashboard.

I was able to create a drop-down list(control type) in the custom dashboard but was unable to see accurate details in the drop-down list. I have through some articles. They say queries may give accurate data but I am new to ELK and don't know how to write them. I am looking for some useful material related this.

[Priscilla_Parodi]

Hello @Kumbum,

Could you please share which version you are using? (you can check in Stack Management). Additionally, more details on what you mean by accurate data would be helpful. What would be considered specific information in this case?

[Kumbum]

@Priscilla_Parodi
I use Filebeat to stream the logs to ELK. The version is 7.17.11
Yes, a specific information.

[Priscilla_Parodi]

@Kumbum

They say queries may give accurate data

You mean by filtering, correct?

You can use the Kibana Query Language (KQL) for filtering data:

Or you can edit as a query DSL:

[Kumbum]

@Priscilla_Parodi

Let me give a brief background about this. I have a program that creates a log file once a day. It contains the following information. And, the filebeat has been configured to push these logs as follows to ELK. Now, I want to create a custom dashboard to display the following metrics on it.

Log_01052024.log - Every day a new log gets generated by the program. The following are the sample details.

LOG_LEVELS, TIMESTAMP, USER, DESCRIPTION, Otherinformation
INFO, 01/05/2024 23:00:00, USER1, "Task Started", ""
INFO, 01/05/2024 23:00:10, USER1, "Deployment is successful", "CHG123"
INFO, 01/05/2024 23:00:20, USER1, "Task Ended", ""
INFO, 01/05/2024 23:00:30, USER1, "Task Started", ""
INFO, 01/05/2024 23:00:40, USER1, "Deployment is successful", "CHG456"
INFO, 01/05/2024 23:00:50, USER1, "Task Ended", ""
INFO, 01/05/2024 23:01:00, USER1, "Task Started", ""
INFO, 01/05/2024 23:01:10, USER1, "Deployment is unsuccessful", "CHG789"
INFO, 01/05/2024 23:01:20, USER1, "Task Ended", ""
.
.
.
so on

This is how the data shows in ELK. Because I have added a script in filebeat.yaml file to split the log details.

Entry 1
LOG_LEVELS : INFO
TIMESTAMP : 01/05/2024 23:10:00
USER. : USER1
DESCRIPTION : "Task Started"
Otherinformation : ""

Entry 2
LOG_LEVELS : INFO
TIMESTAMP : 01/05/2024 23:00:00
USER. : USER1
DESCRIPTION : "Deployment is successful"
Otherinformation : "CHG123"

Entry 3
LOG_LEVELS : INFO
TIMESTAMP : 01/05/2024 23:20:00
USER. : USER1
DESCRIPTION : "Task Ended"
Otherinformation : ""
.
.
So on

I want to pull the following details on the dashboard.

---

|Metric 1 | Metric 2|
|Metric 3 | |

Metric 1: Total number deployments : 3
Metric 2: Total number of success and fail deployment :
Deployment is successful: 2
Deployment is unsuccessful: 1
Metric 3:
CHG123 : Deployment is successful
CHG456 : Deployment is successful
CHG789 : Deployment is unsuccessful
.
.
so on.

I hope this is very clear to you now. I was creating visualization for these but was unable to pull metrics how I wanted to show. Please help me with how this can accomplished.

[Priscilla_Parodi]

Thanks for the additional info. Are you already able to visualize these logs in Kibana?

Filebeat comes packaged with example dashboards (and visualizations) for visualizing Filebeat data. You need to config the dashboard loading. And before you can use the dashboards, you need to create an index pattern, filebeat-*.

Now, if you've already done that and your question is only about different types of visualization, you probably want to check TSVB and aggregation-based visualizations, 'metric' is a good one here, if you’re unsure about the visualization type Lens is a good option, it will provide some suggestions based on the data.