Trim the file name remove last 2 characters and extension using grok

Elastic Stack Logstash
1

HI Everyone,

I am trying to push data from s3 to Es using logstash on Ec2.

this is my config file for logstash

input {
s3 {
bucket => ""
access_key_id => ""
secret_access_key => "*"
region => "us-east-1"
codec => "json"
}
}

filter{
mutate{
add_field =>{
"file" => "%{[@metadata][s3][key]}"
}
}
}
output {
amazon_es {
hosts => *
region => "us-east-1"
aws_access_key_id => ''
aws_secret_access_key => ''
index => "%{file}"
template_name => "sqe_template"
template_overwrite => "true"
codec => "json"
}

Now because of "file" => "%{[@metadata][s3][key]}"
i get my index as "filenameins3"

Eg: if I have a file name as log-ecoa-production-index-2019-04-16-17.txt on s3

index comes as log-ecoa-production-index-2019-04-16-17.txt

I want to remove the extension and if required "-17" too and put as index, is there anyway using grok I can achieve this >
then index should be log-ecoa-production-index-2019-04-16-17
or log-ecoa-production-index-2019-04-16 as per requirement

Kindly provide your suggestions.

Thank You,
Mohit Ruke

2

I would do it with mutate+gsub.

3

Yes I researched about it but cant seem to have the right syntax for it
Can you help ?

4 
mutate { gsub => [ "message", "\.\w{3}", "", "message", "([0-9]{4}-[0-9]{2}-[0-9]{2})-[0-9]+", "\1" ] }
5

Thanks Badger,
I am trying it. Could tell me what script is this line from ```
mutate { gsub => [ "message", ".\w{3}", "", "message", "([0-9]{4}-[0-9]{2}-[0-9]{2})-[0-9]+", "\1" ] } ?

like what language does gsub or kibana use ? so for future adjustments I could do it

6

gsub uses ruby regexps.

7

Thank You, worked!

8

HI badger !

Now I get the file name as filename-2019-01-32
How can I get get of the '-32' and set it as index name?

Please help!!

9 
    mutate { add_field => { "someField" => "log-ecoa-production-index-2019-04-16-17.txt" } }
    mutate { gsub => [ "someField", "\.\w{3}", "", "message", "([0-9]{4}-[0-9]{2}-[0-9]{2})-[0-9]+", "\1" ] }
    grok { match => [ "someField", "-%{INT:anotherField}$" ] }

will result in

"anotherField" => "17",
   "someField" => "log-ecoa-production-index-2019-04-16-17"
10

Hi Badger thank you so much for the help. I was able to using this config

filter{

mutate{
add_field =>{
"file" => "%{[@metadata][s3][key]}"
}
}
mutate {
gsub => [ "file", ".{3}$", "", "file", "([0-9]{4}-[0-9]{2})-[0-9]+", "\1" ] }
}

I am getting the required result! But is this the correct way? I dont know the syntax for regex! managed to put this together

Thank You,
Mohit Ruke

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.