Pattern: grok/mutate dont work?

Elastic Stack Logstash
1

hi all! i ran into a problem.
i have a pipeline:

input {
    tcp {
        host => "10.10.10.10"
        port => "5959"
        codec => "json"
        type => "my_type"
        mode => "server"
    }
}

filter {
 if [type] == "my_type" {
    grok {
        match => {"message" => ['\#%{GREEDYDATA:who}%{SPACE}#%{GREEDYDATA:what}\n\n%{GREEDYDATA}: %{GREEDYDATA:contact}%{SPACE} #%{GREEDYDATA:ident}\n%{GREEDYDATA}Cost:%{SPACE}%{DATA:cost}\n%{GREEDYDATA}Date:%{SPACE}%{DATE_EU:when}\n%{GREEDYDATA}Vacant:%{SPACE}%{DATA:vacant}\n(?<info>(.|\r|\n)*)\n\[%{GREEDYDATA}']
           }
       }

    mutate {
        gsub => [
            "what", "milkshake", "Milk 🥛",
            "what", "icecream", "Ice 🍦"
            ]  
        }
        
    mutate {
        gsub => [
            "cost", "-1", "?",
            "contact", "9", "+9"
            ]  
        }
    
    mutate {
        gsub => [
            "contact", "\+9", "tel:+9"
            ]  
        }
        
      if [date] {
            date {
                match => [ "date", "ISO8601", "YYYY-MM-dd'T'HH:mm:ss.ZZZ" ]
                target => "@timestamp"
                } }

    if "_grokparsefailure" in [tags] {
    drop {}
            }
    }
}

output {
    if [type] == "my_type" {
        elasticsearch {
            hosts => ["https://localhost:9200"]
            index => "my_type-%{+xxxx.ww}"
            ilm_rollover_alias => "my_type"
            ilm_policy => "my_type"
            ilm_enabled => "true"
            cacert => ["/etc/logstash/ca.crt"]
            user => "elastic"
            password => "password"
        }
    }
}

in the mutate section after the GROK i replace one word with another which will contain emoji and be capitalized

mutate {
        gsub => [
            "what", "milkshake", "Milk 🥛",
            "what", "icecream", "Ice 🍦"
            ]  
        }

but in elasticsearch in the "what" field I see that the word is substituted with a non-capital letter "milk :milk_glass:" instead of "Milk :milk_glass:"

I tried to force mutate for the capital letter, but that doesn't help either ("mutate" for the capital letter was placed after the "mutate" for replacement)

mutate {
        capitalize => [ "what" ]
    }

What can be wrong?

2

Do you have something special in your index pattern mapping?

1 Like
3

no, as far as i know, nothing special

4

Maybe instead of gsub you could use the translate plugin?

5

sorry what plugin are you talking about?

6

First verify that the problem is not in logstash by looking at what is produced by

output { stdout { codec => rubydebug } }

(or a file output with that codec). If the problem is on the elasticsearch side it could be an analyzer filter or a processor in an ingestion pipeline.

7

This plugin

8

with Translate filter pluginedit the same - the word is inserted in lowercase :frowning:

9

no, i dont have analyzer or a processor in an ingestion pipeline

10

Does the rubydebug codec show it as uppercase or lowercase?

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.