Please help this poor guy. Another day of upping this. hoping someone can help me.
This is what it looks like in my visualization. I cannot create a pattern in my logstash cause as you know error logs have so many. So I want to have like that pattern that can identify it and count them as a category.
Here is the pattern in discover that I want to like imatate in my visualization.
I also tried a job at machine learning but nothing is showing and I cannot use it in my visualization
What exactly are you trying to do?
To have like a pattern in my visualization or dashboard. Something that it can identify the pattern of error message so it can count them as one category or group cause as you can see they are being count as individual which should be count as a group or category.
What do you mean when you say pattern? It is confusing.
You mean to parse your message and have the information in different fields, like one field with /usr/sbin/mysqld, another with mysqld other with the version 8.2.0 and other with the process id?
If so you need to parse your message before indexing it, in Logstash or using an Elasticsearch ingest pipeline.
If you have different log patterns, you need to create one parse for each log type.
Thank you for replying @leandrojmp
This is the pattern that I mean that I want to do in my visualization also.
and yes if I know that I can parse them but there are too many logs that I need to parse.Like what if there is new kind of logs that can't be parse by what I declare in logstash. Cause error message have many different structures so it was hard to declare them one by one.
But this is how this works, if you have a message that does not match your current patterns to parse it, you will need to create a new pattern that will match this new message format.
If you want to use part of a message as a field in visualization or queries, then you need to parse the message and create the field.
Elastic provides hundreds of integrations for the Elastic Agent with ingest pipelines that have patterns to parse a lot of different kinds of messages as you can check here.
But if you are using Logstash, you will need to build your own pipelines with the patterns to parse your messages.
The Kibana Pattern in Discover is used to help you understand your unstructured messages and then use this information to build a pattern to parse it.
Thank you very much @leandrojmp for replying it is now clear that I cannot use pattern in my visualization aside from specifying the pattern in logstash. Yes I am not using a modules or built-in as it was also have more restriction in our desired visualization. But lastly how about using a categorization in anomaly detection at machine learning, can't I use that alternatively?
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.
