Make a field "less specific"

Elastic Stack Logstash
1

Hi,

I am new in Logstash, I could perform my first log analysis without any problem but I'm facing a situation I have not find a solution for.

So in my logs I have a path that I manage to isolate easily, but here is the thing, this path may include some ID's that I don't need.
For example I have this line :
/Manager/Playlists/48d38243-1def-45a/Entries/0f875fde-37ea-410c/Clip/Export

So I would like to make those ID in bold all the same, as I care more about the kind of path, more than the specific element I deal with.
So far I managed to get all the fields seperately and my idea was to reconstruct a simplified field, by replacing the IDs to a static string in a mutate, and recreating a new field that would become for the previous example :
/Manager/Playlists/idString/Entries/idString/Clip/Export

But at this moment the problem is that the paths I deal with don't all have the same pattern, so the IDs are not at always the same positions.
And unless I'm mistaken (?) we can't create a new field including optional fields.

So I was wondering if it's possible to change the field on the fly while we parse (within the grok). Or maybe there is another solution I haven't think about.

Thanks to all of you, and hope I have been clear enough in my explanation

2

Would the mutate filter's gsub option help?

3

thanks for your input, I indeed didn't check this possibility
can we apply it in the whole field like this for example :
"fieldname", "*", "idString"

4

You mean replace the field value completely? Sure, but the mutate filter's replace option is more efficient for that.

5

oh yes but actually the same problem will occur I guess, it will change for the field taken individually but not within the path right?
say I have those fields :
/Manager/Playlists/48d38243-1def-45a/Entries/0f875fde-37ea-410c/Clip/Export => myPath
Manager => part1
Playlist => part2
48d38243-1def-45a => id
Entries => part4
0f875fde-37ea-410c => id
Clip => part6
Export => part7

If I use replace or gsub on id it won't replace within myPath, is there a solution to "update" myPath with the new value?

6

Yes, use gsub.

7

ok I will try.

thank you very much

8

after trying the problem remain.
I would like the field to be recursively changed in myPath
or maybe I miss something

I might have to write a ruby piece of code

9

ok I managed to do what I wanted :
I added all the new_field (part1, part2 etc) beforehand with empty string as value.
then I parsed my path as I wanted (I also managed to write a general match with optional fields), overwriting the fields previously created, when a new value is found.

Then I wrote a Ruby code to change all the IDs that are not empty string to "idString"
then I reconstruct all the paths with all the fields (they all exists though some are just an empty string so no need to deal with optional)

Thanks Magnus for your inputs anyway :slight_smile:

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.