Trim the file name remove last 2 characters and extension using grok

Elastic Stack Logstash
1

HI Everyone,

I am trying to push data from s3 to Es using logstash on Ec2.

this is my config file for logstash

input {
s3 {
bucket => ""
access_key_id => ""
secret_access_key => "*"
region => "us-east-1"
codec => "json"
}
}

filter{
mutate{
add_field =>{
"file" => "%{[@metadata][s3][key]}"
}
}
}
output {
amazon_es {
hosts => *
region => "us-east-1"
aws_access_key_id => ''
aws_secret_access_key => ''
index => "%{file}"
template_name => "sqe_template"
template_overwrite => "true"
codec => "json"
}

Now because of "file" => "%{[@metadata][s3][key]}"
i get my index as "filenameins3"

Eg: if I have a file name as log-ecoa-production-index-2019-04-16-17.txt on s3

index comes as log-ecoa-production-index-2019-04-16-17.txt

I want to remove the extension and if required "-17" too and put as index, is there anyway using grok I can achieve this >
then index should be log-ecoa-production-index-2019-04-16-17
or log-ecoa-production-index-2019-04-16 as per requirement

Kindly provide your suggestions.

Thank You,
Mohit Ruke

2

I would do it with mutate+gsub.

3

Yes I researched about it but cant seem to have the right syntax for it
Can you help ?

4 
mutate { gsub => [ "message", "\.\w{3}", "", "message", "([0-9]{4}-[0-9]{2}-[0-9]{2})-[0-9]+", "\1" ] }
5

Thanks Badger,
I am trying it. Could tell me what script is this line from ```
mutate { gsub => [ "message", ".\w{3}", "", "message", "([0-9]{4}-[0-9]{2}-[0-9]{2})-[0-9]+", "\1" ] } ?

like what language does gsub or kibana use ? so for future adjustments I could do it

6

gsub uses ruby regexps.

7

Thank You, worked!

8

HI badger !

Now I get the file name as filename-2019-01-32
How can I get get of the '-32' and set it as index name?

Please help!!

9 
    mutate { add_field => { "someField" => "log-ecoa-production-index-2019-04-16-17.txt" } }
    mutate { gsub => [ "someField", "\.\w{3}", "", "message", "([0-9]{4}-[0-9]{2}-[0-9]{2})-[0-9]+", "\1" ] }
    grok { match => [ "someField", "-%{INT:anotherField}$" ] }

will result in

"anotherField" => "17",
   "someField" => "log-ecoa-production-index-2019-04-16-17"
10

Hi Badger thank you so much for the help. I was able to using this config

filter{

mutate{
add_field =>{
"file" => "%{[@metadata][s3][key]}"
}
}
mutate {
gsub => [ "file", ".{3}$", "", "file", "([0-9]{4}-[0-9]{2})-[0-9]+", "\1" ] }
}

I am getting the required result! But is this the correct way? I dont know the syntax for regex! managed to put this together

Thank You,
Mohit Ruke