Convert uppercase in lowercase

Elastic Stack Logstash
1

Hi, i've tried to convert field.keyword in lowrcase by using this mapping:
filter {
mutate {
lowercase => {
match => [ "request"]
}
}
grok {
match => {
"message" => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:respon$
}
}

this is my logstash.conf
My input field is like:
//fileadmin/Material/mSite/logo_curv.GIF?r=90.
i'd like convert all string in lowercase.
//fileadmin/material/msite/curv.gif?r=90.

2

mutate {
lowercase => {
match => [ "request"]
}
}

That's not how the lowercase option should be used. The documentation contains a correct example.

3

@magnusbaeck when i upload data i recived this error message:
][ERROR][logstash.pipeline ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {"exception"=>#<TypeError: can't convert Array into String>,

4

Without seeing your configuration file the only thing I can say is that the error message should tell you roughly which line in your configuration file that contains the error.

5

yes sure, i'm sorry @magnusbaeck

logstash.conf
input {
stdin { }
}
filter {
mutate {
lowercase => {
match => [ "request"]
}
}
grok {
match => {
"message" => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "%
{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}'

     }

}

date {
match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ]
target => "@timestamp"
locale => en
}

geoip {
source => "clientip"
}

useragent {
source => "agent"
target => "useragent"
}
}

output {
stdout {
codec => dots {}
}

elasticsearch {
hosts => ["localhost:9200"]
index => "apache_elastic"
template => "./apache_template.json"
template_name => "apache_elastic"
template_overwrite => true
}
}

for kibana.json i use this on git https://github.com/elastic/examples/blob/master/ElasticStack_apache/apache_kibana.json

for template.json
{
"template": "apache_elastic",
"settings": {
"index.refresh_interval": "5s"
}
}

6

mutate {
lowercase => {
match => [ "request"]
}
}

Why haven't you fixed this?

7

@magnusbaeck i've tried to change the filter like documentation but in kibana show request value with uppercase and lowercase. nothing change. and when i use this filter in exclude pattern or in a search bar "..(png|jpg|gif|js|css)(?.)?" match only lowercase.

8

At this point I don't know what you want help with.

If you're getting the "Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash" error message then you're not getting any data into Kibana because Logstash isn't starting.

9

I'm sorry @magnusbaeck i didn't explain weel. Now Logstash upload data in kibana and not give me any error but the request.keyword values aren't lowercase. I fixed the mutate filter, i restart all machines, and uploaded data but i still have request.keyword value like this: //fileadmin/Material/mSite/logo_.GIF?r=90. GIF in uppercase and other characters same. i don't have idea what i can do for transform this url in lowercase. I need this because i want to exclud uppercase jpg css and other of my result query. In the exclude pattern in Kibana you can't specify the ignorecase so i have t upload data already lowercase.

10

Please show an example event, e.g. copied from the JSON tab of Kibana's Discover panel. Also, what does your configuration look like now?

11

Immagine.png1366×768 73.2 KB
This is JSONobject and for about configuration are the same before copied but i fixed filter. Now:
filter {
mutate {
lowercase => ["request"]
}
}

12

The mutate filter must come after the grok filter. Filters are evaluated in order and until the grok filter has run there is no request field.

1 Like
13

Thank you very much @magnusbaeck it go.
This is the solution:
filter {
grok {
match => {
"message" => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:resp$
}
}
mutate {
lowercase => ["request"]
}

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.