Some newbie questions file path and message replacement

[alexsamad]

Hi
extracting part of the path from source

I found this

Extracting particular folder from the path and adding that to a field

Hi Guys, I need help to fetch particular folder from the path and assign the same to a different field. path ==> /var/mqm/qmgrs/FOLDER_NEEDED/errors/*.log. I need to extract FOLDER_NEEDED and assign it to a field for every message. Thanks.

And it seems to say what I want to do, but I am not sure how to implement

I have filebeat setup to send to a logstash input

I'm using grok, i've got
grok {
match => [ "message" , "%{TIMESTAMP_ISO8601:timestamp} (?[.*]) %{LOGLEVEL:loglevel} %{GREEDYDATA:message}"]

}

all my source files names should be like

/path1/path2/path3/path4/

I am interested in getting path4 into a file name

this is what i saw from the message
grok { match => [ "message", "^/[^/]+/[^/]+/[^/]+/(?[^/]+)" ] }
grok { match => [ "message", "/(?[^/]+)/[^/]+/[^/]+$" ] }

what do i need to add to my code to extract path4. even to add to that lets say all the paths are like

/var/log/abc/<path i'm interested in>/
so I'm guessing
grok { match => [ "message", "^/var/log+/(?[^/]+)" ] }

does this mean I get a variable called dir1 which has the info I am after ?

also

match => [ "message" , "%{TIMESTAMP_ISO8601:timestamp} (?<thread>\[.*\]) %{LOGLEVEL:loglevel} %{GREEDYDATA:message}"]

the last bit adds the "rest" of the message as message[1]. doesn't save it in message[0]

how can i get {GREEDYDATA:message} to replace the message variable ?

thanks

edit

I'm thinking
grok { match => [ "message", "^/var/log+/(?[^/]+)" ] } should be
grok { match => [ "source", "^/var/log+/(?[^/]+)" ] }

[bigphil (Philip Nunn)]

My recommendation is to use the dissect filter instead of grok for parsing your file path.

[alexsamad]

could you maybe give an example ??

also any hint on my my %{GREEDYDATA:message} is added to message[1] and not message[0]

[bigphil (Philip Nunn)]

Dissect info here

[alexsamad]

Hi

Thanks, not sure I understand - could you provide an example of inside and outside ?

[bigphil (Philip Nunn)]

Nevermind...I just thought of why it's like that. You're reusing the field name "message". Use another name and then either remove the original "message" field or overwrite it with your new field name. i.e. {GREEDYDATA:new_message}

You can use the common option "remove_field" in all filters and the mutate filter to update/replace/rename your message field.