Some newbie questions file path and message replacement

Elastic Stack Logstash
1

Hi
extracting part of the path from source

I found this

And it seems to say what I want to do, but I am not sure how to implement

I have filebeat setup to send to a logstash input

I'm using grok, i've got
grok {
match => [ "message" , "%{TIMESTAMP_ISO8601:timestamp} (?[.*]) %{LOGLEVEL:loglevel} %{GREEDYDATA:message}"]

}

all my source files names should be like

/path1/path2/path3/path4/

I am interested in getting path4 into a file name

this is what i saw from the message
grok { match => [ "message", "^/[^/]+/[^/]+/[^/]+/(?[^/]+)" ] }
grok { match => [ "message", "/(?[^/]+)/[^/]+/[^/]+$" ] }

what do i need to add to my code to extract path4. even to add to that lets say all the paths are like

/var/log/abc/<path i'm interested in>/
so I'm guessing
grok { match => [ "message", "^/var/log+/(?[^/]+)" ] }

does this mean I get a variable called dir1 which has the info I am after ?

also

match => [ "message" , "%{TIMESTAMP_ISO8601:timestamp} (?<thread>\[.*\]) %{LOGLEVEL:loglevel} %{GREEDYDATA:message}"]

the last bit adds the "rest" of the message as message[1]. doesn't save it in message[0]

how can i get {GREEDYDATA:message} to replace the message variable ?

thanks

edit

I'm thinking
grok { match => [ "message", "^/var/log+/(?[^/]+)" ] } should be
grok { match => [ "source", "^/var/log+/(?[^/]+)" ] }

2

My recommendation is to use the dissect filter instead of grok for parsing your file path.

3

could you maybe give an example ??

also any hint on my my %{GREEDYDATA:message} is added to message[1] and not message[0]

4

Dissect info here

5

Hi

Thanks, not sure I understand - could you provide an example of inside and outside ?

6

Nevermind...I just thought of why it's like that. You're reusing the field name "message". Use another name and then either remove the original "message" field or overwrite it with your new field name. i.e. {GREEDYDATA:new_message}

You can use the common option "remove_field" in all filters and the mutate filter to update/replace/rename your message field.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.