Visualize List files from logs

[Oniriel]

Hi,

I have an index that I try to create visualization on for monitoring. Some of the logs have the following structure

TEXT - path of a file - TEXT

the same file can appear multiple time in the logs

I can successfully create a search to get all the logs with this pattern, but I am trying to create a data table visualization to display the list of unique file in them

I couldn't figure out how to do that, can somebody point me to the proper resource ?

thanks

[Andrew_Tate (Drew Tate)]

Hi @Oniriel ! Welcome to the community.

So, I take it that your documents store that pattern in a field something like this?

{
    "_source": {
        ....
        "pattern": "TEXT - path of a file - TEXT"
    }
}

And you want a table where each row displays one of these patterns with no duplicates.

Is all this correct?

[Oniriel]

yes that is correct, I want to provide the list of file (without duplication) to the people using the dashboard to monitors the servers (all the logs are sent to Elasticsearch and then vizualized in Kibana)

So I was trying to find a way to extract the file path from the logs and then remove duplicates to create a table

[Andrew_Tate (Drew Tate)]

Are you on version 8.6 or higher? If so, you can do this using our "technical preview" SQL support.

First, you have to enable the discover:enableSql setting in advanced settings:

Then, select SQL from the data view dropdown in Lens

Finally, write your SQL query. Let's say the field containing the pattern field is called "pattern". You could write

SELECT pattern, count(pattern) FROM 'my_index_pattern' GROUP BY pattern

Here is an example of this strategy, but the field in my example is called geo.dest:

If you don't want to see the count column, you can always hide it

Does this help?

[Oniriel]

Hi,

I just came back from vacation, sorry for the delay. It seems we are using Kibana version 7.10

I will see what is available from what you proposed