Logstash Stopping Pipeline

Elastic Stack Logstash
1

Hi Guys ,

two days ago one of my logstash servers crash with Logstash service.
the error recieved is :message=>"stopping pipeline", :id=>"main"

The configuration file is the following:

input {
    udp {
      port => 9933
        codec => netflow {
        versions => [9]
      }
    }
}

filter {

if [host] == "xxxxxx" {
  grok {
    match => { "host" => "xxxxxxxxx" }
  }


geoip {
 add_tag => [ "geoip" ]
 database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat" ### Change me to location of GeoLiteCity.dat file
 source => "ipv4_dst_addr"
 }

 if [geoip][city_name] == "" { mutate { remove_field => "[geoip][city_name]" } }
 if [geoip][continent_code] == "" { mutate { remove_field => "[geoip][continent_code]" } }
 if [geoip][country_code2] == "" { mutate { remove_field => "[geoip][country_code2]" } }
 if [geoip][country_code3] == "" { mutate { remove_field => "[geoip][country_code3]" } }
 if [geoip][country_name] == "" { mutate { remove_field => "[geoip][country_name]" } }
 if [geoip][latitude] == "" { mutate { remove_field => "[geoip][latitude]" } }
 if [geoip][longitude] == "" { mutate { remove_field => "[geoip][longitude]" } }
 if [geoip][postal_code] == "" { mutate { remove_field => "[geoip][postal_code]" } }
 if [geoip][region_name] == "" { mutate { remove_field => "[geoip][region_name]" } }
 if [geoip][time_zone] == "" { mutate { remove_field => "[geoip][time_zone]" } }

}
}

output {
 if [host] == "xxxxxxxx" {
   stdout { codec => rubydebug }
   elasticsearch {
         manage_template => false
        index => "xxxxxxxa%{+YYYY.MM.dd}"
        hosts => "xxxxxxxxxxl:9200"
}
   }
}

I detected that if i take out Filter section in this conf logstash work ok. for that reason I checked the plugins installed and i saw that " grok " & " geoip " are installed.

I`m not a programmer, I googled the error but i not saw nothing about this specific plugins/filters.

Can someone help me with this ??

Thanks in advance.

2

DId you figure this out? Having the same issue. Thanks.

3

Any eror logs available? Could you provide the message format to grok and regular expression used in grok？

4

Hi Tim, good morning.

Yes I did figure out this issue.
In this particular case I found more logs about plugin erros, like grok, mutate, among others, all errors related about incompatibilities. So, I updated some plugins using rubygems.org. was difficult for me because I didn't know how to update the gems and never work with ruby. But after some shoots logstash came back from the dead.

5

Great. My logstash install somehow was missing a config folder and I finally figured it out. Like you, these tools are new to me so it was a process for sure. Best of luck. TIm

6

I'm glad to know that you can resolve the issue Tim.
If i can help you with something more, please let me know. :slight_smile: