Logstash stopped processing because of an error: (SystemExit) exit

We are trying to index Nginx access and error log separately in Elasticsearch. for that we have created Filbeat and Logstash config as below.

Below is our /etc/filebeat/filebeat.yml configuration

    filebeat.inputs:
    - type: log
      paths:
        - /var/log/nginx/*access*.log
      exclude_files: ['\.gz$']
      exclude_lines: ['*ELB-HealthChecker*']
      fields:
        log_type: type1 
    - type: log
      paths:
        - /var/log/nginx/*error*.log
      exclude_files: ['\.gz$']
      exclude_lines: ['*ELB-HealthChecker*']
      fields:
        log_type: type2
    
    output.logstash:
      hosts: ["10.227.XXX.XXX:5400"]

Our logstash file /etc/logstash/conf.d/logstash-nginx-es.conf config is as below

    input {
        beats {
            port => 5400
        }
    }
    
    filter {
      if ([fields][log_type] == "type1") {
        grok {
          match => [ "message" , "%{NGINXACCESS}+%{GREEDYDATA:extra_fields}"]
          overwrite => [ "message" ]
        }
        mutate {
          convert => ["response", "integer"]
          convert => ["bytes", "integer"]
          convert => ["responsetime", "float"]
        }
        geoip {
          source => "clientip"
          target => "geoip"
          add_tag => [ "nginx-geoip" ]
        }
        date {
          match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
          remove_field => [ "timestamp" ]
        }
        useragent {
          source => "user_agent"
        }
      } else {
          grok {
            match => [ "message" , "(?<timestamp>%{YEAR}[./]%{MONTHNUM}[./]%{MONTHDAY} %{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER:threadid}\: \*%{NUMBER:connectionid} %{GREEDYDATA:message}, client: %{IP:client}, server: %{GREEDYDATA:server}, request: "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion}))"(, upstream: "%{GREEDYDATA:upstream}")?, host: "%{DATA:host}"(, referrer: "%{GREEDYDATA:referrer}")?"]
            overwrite => [ "message" ]
          }
          mutate {
            convert => ["response", "integer"]
            convert => ["bytes", "integer"]
            convert => ["responsetime", "float"]
          }
          geoip {
            source => "clientip"
            target => "geoip"
            add_tag => [ "nginx-geoip" ]
          }
          date {
            match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
            remove_field => [ "timestamp" ]
          }
          useragent {
            source => "user_agent"
          }
        }
    }
    
    output {
      if ([fields][log_type] == "type1") {
        amazon_es {
          hosts => ["vpc-XXXX-XXXX.ap-southeast-1.es.amazonaws.com"]
          region => "ap-southeast-1"
          aws_access_key_id => 'XXXX'
          aws_secret_access_key => 'XXXX'
          index => "nginx-access-logs-%{+YYYY.MM.dd}"
        }
    } else {
        amazon_es {
          hosts => ["vpc-XXXX-XXXX.ap-southeast-1.es.amazonaws.com"]
          region => "ap-southeast-1"
          aws_access_key_id => 'XXXX'
          aws_secret_access_key => 'XXXX'
          index => "nginx-error-logs-%{+YYYY.MM.dd}"
        }
      }
        stdout { 
          codec => rubydebug 
        }
    }

And we are receiving below error while starting logstash.

[2020-10-12T06:05:39,183][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.9.2", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 25.265-b01 on 1.8.0_265-b01 +indy +jit [linux-x86_64]"} [2020-10-12T06:05:39,861][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2020-10-12T06:05:41,454][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"{\", \",\", \"]\" at line 32, column 263 (byte 918) after filter {\n if ([fields][log_type] == \"type1\") {\n grok {\n match => [ \"message\" , \"%{NGINXACCESS}+%{GREEDYDATA:extra_fields}\"]\n overwrite => [ \"message\" ]\n }\n mutate {\n convert => [\"response\", \"integer\"]\n convert => [\"bytes\", \"integer\"]\n convert => [\"responsetime\", \"float\"]\n }\n geoip {\n source => \"clientip\"\n target => \"geoip\"\n add_tag => [ \"nginx-geoip\" ]\n }\n date {\n match => [ \"timestamp\" , \"dd/MMM/YYYY:HH:mm:ss Z\" ]\n remove_field => [ \"timestamp\" ]\n }\n useragent {\n source => \"user_agent\"\n }\n } else {\n grok {\n match => [ \"message\" , \"(?<timestamp>%{YEAR}[./]%{MONTHNUM}[./]%{MONTHDAY} %{TIME}) \\[%{LOGLEVEL:severity}\\] %{POSINT:pid}#%{NUMBER:threadid}\\: \\*%{NUMBER:connectionid} %{GREEDYDATA:message}, client: %{IP:client}, server: %{GREEDYDATA:server}, request: \"", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:183:in initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:44:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:357:in block in converge_state'"]} [2020-10-12T06:05:41,795][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} [2020-10-12T06:05:46,685][INFO ][logstash.runner ] Logstash shut down. [2020-10-12T06:05:46,706][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

There seems to be some formatting issue. Please help what is the problem

Hi,

You configuration is false you're using quotes in your grok patterns that break the string after the word

request: " <--
(?:%{WORD:verb}

Alternatively you can use tags in filebeat with ( add_tag ) to easily retrieve log and use default nginx patterns.

Hi @grumo35

Thank you for the response. As I am new to this filebeat and logstash. I am not too sure how to use the tags in filebeat. I know that we have predefined pattern filter for the Nginx Access logs which I have used but to parse the error logs we used formatted pattern. So now we corrected the format as per your suggestion. which looks like below.

grok {
        match => [ "message" , "(?<timestamp>%{YEAR}[./]%{MONTHNUM}[./]%{MONTHDAY} %{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER:threadid}\: \*%{NUMBER:connectionid} %{GREEDYDATA:message}, client: %{IP:client}, server: %{GREEDYDATA:server}, request: (?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion}))(, upstream: %{GREEDYDATA:upstream})?, host: %{DATA:host}(, referrer: %{GREEDYDATA:referrer})?" ]
        overwrite => [ "message" ]
      }

But now we are facing below error in the logs. Please suggest.

[2020-10-12T07:10:57,518][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2020-10-12T07:10:57,532][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.8.12"} [2020-10-12T07:11:07,116][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50} [2020-10-12T07:11:07,676][INFO ][logstash.outputs.amazonelasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://xxx-xxx-xxx-elasticsearch.ap-southeast-1.es.amazonaws.com:443/]}} [2020-10-12T07:11:07,686][INFO ][logstash.outputs.amazonelasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://xxx-xxx-xxx-elasticsearch.ap-southeast-1.es.amazonaws.com:443/, :path=>"/"} [2020-10-12T07:11:08,375][WARN ][logstash.outputs.amazonelasticsearch] Restored connection to ES instance {:url=>"https://xxx-xxx-xxx-elasticsearch.ap-southeast-1.es.amazonaws.com:443/"} [2020-10-12T07:11:08,447][INFO ][logstash.outputs.amazonelasticsearch] ES Output version determined {:es_version=>7} [2020-10-12T07:11:08,450][WARN ][logstash.outputs.amazonelasticsearch] Detected a 6.x and above cluster: the typeevent field won't be used to determine the document _type {:es_version=>7} [2020-10-12T07:11:08,487][INFO ][logstash.outputs.amazonelasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::AmazonElasticSearch", :hosts=>["//xxx-xxx-xxx-elasticsearch.ap-southeast-1.es.amazonaws.com"]} [2020-10-12T07:11:08,504][INFO ][logstash.outputs.amazonelasticsearch] Using mapping template from {:path=>nil} [2020-10-12T07:11:08,519][INFO ][logstash.outputs.amazonelasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://xxx-xxx-xxx-elasticsearch.ap-southeast-1.es.amazonaws.com:443/]}} [2020-10-12T07:11:08,525][INFO ][logstash.outputs.amazonelasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://xxx-xxx-xxx-elasticsearch.ap-southeast-1.es.amazonaws.com:443/, :path=>"/"} [2020-10-12T07:11:08,539][INFO ][logstash.outputs.amazonelasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60002, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}} [2020-10-12T07:11:08,586][WARN ][logstash.outputs.amazonelasticsearch] Restored connection to ES instance {:url=>"https://xxx-xxx-xxx-elasticsearch.ap-southeast-1.es.amazonaws.com:443/"} [2020-10-12T07:11:08,603][INFO ][logstash.outputs.amazonelasticsearch] ES Output version determined {:es_version=>7} [2020-10-12T07:11:08,604][WARN ][logstash.outputs.amazonelasticsearch] Detected a 6.x and above cluster: thetypeevent field won't be used to determine the document _type {:es_version=>7} [2020-10-12T07:11:08,612][INFO ][logstash.outputs.amazonelasticsearch] Using mapping template from {:path=>nil} [2020-10-12T07:11:08,614][INFO ][logstash.outputs.amazonelasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::AmazonElasticSearch", :hosts=>["//xxx-xxx-xxx-elasticsearch.ap-southeast-1.es.amazonaws.com"]} [2020-10-12T07:11:08,636][INFO ][logstash.outputs.amazonelasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60002, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}} [2020-10-12T07:11:08,777][ERROR][logstash.pipeline ] Error registering plugin {:pipeline_id=>"main", :plugin=>"#<LogStash::FilterDelegator:0x48f466ac>", :error=>"pattern %{NGINXACCESS} not defined", :thread=>"#<Thread:0x35c6ad13 run>"} [2020-10-12T07:11:08,782][ERROR][logstash.pipeline ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<Grok::PatternError: pattern %{NGINXACCESS} not defined>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:123:inblock in compile'", "org/jruby/RubyKernel.java:1425:in loop'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:93:in compile'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.2.0/lib/logstash/filters/grok.rb:284:in block in register'", "org/jruby/RubyArray.java:1792:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.2.0/lib/logstash/filters/grok.rb:278:in block in register'", "org/jruby/RubyHash.java:1419:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.2.0/lib/logstash/filters/grok.rb:273:in register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:56:in register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:259:in register_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:270:in block in register_plugins'", "org/jruby/RubyArray.java:1792:in each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:270:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:612:in maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:280:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:217:in run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:176:in block in start'"], :thread=>"#<Thread:0x35c6ad13 run>"}
[2020-10-12T07:11:08,803][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil}
[2020-10-12T07:11:09,125][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2020-10-12T07:11:14,011][INFO ][logstash.runner ] Logstash shut down.`

Here you have your error :wink:

You should use the grok debugger in : kibana.https://www.elastic.co/guide/en/kibana/current/xpack-grokdebugger.html

As of the use of tags in filebeat just do :

- type: log   
paths:     - "/var/log/log.log"   
encoding: utf-8   
tags: ["test","log","tag3"]

Hey @grumo35

You are a life saver. That made it work and running fine. But now that I have updated the filebeat yaml to below, the service is failing after start. Is there something wrong with the config? Please suggest.

filebeat.inputs:
- type: log
  paths:
    - /var/log/nginx/*access*.log
  exclude_files: ['\.gz$']
  exclude_lines: ['*ELB-HealthChecker*']
  fields:
    log_type: type1
- type: log
  paths:
    - /var/log/nginx/*error*.log
  exclude_files: ['\.gz$']
  exclude_lines: ['*ELB-HealthChecker*']
  fields:
    log_type: type2

output.logstash:
  hosts: ["10.227.XXX.XXX:5400"]

Be careful when using yaml to not use "TABS" only spaces.

Wheck your configuratin file for tabs or missing parts.

Hey @grumo35

Thanks for the tip, however this time the problem was with the REGEXP so I had to change the formatting to exclude_lines: ['.*ELB-HealthChecker.*'] instead.

Well now the whole pipeline is working fine and both the indices have been created separately. But the only thing is whatever the GROK pattern we defined doesn't seem to work to bind different values inside the log message unlike the access log one. It is still showing whole message as single unit. What do you suggest?

It's not very important, your solution works too but since log type are deprecated and you're struggling with patterns, adding a field is not ideal.

This is how i'll do it in filebeat:

filebeat.inputs:
- type: log
  paths:
    - /var/log/nginx/*access*.log
  exclude_files: ['\.gz$']
  exclude_lines: ['*ELB-HealthChecker*']
  tags: ["access","nginx"]
- type: log
  paths:
    - /var/log/nginx/*error*.log
  exclude_files: ['\.gz$']
  exclude_lines: ['*ELB-HealthChecker*']
  tags: ["error","nginx"]

output.logstash:
  hosts: ["10.227.XXX.XXX:5400"]

Then you can test in logstash

if "nginx" in tags {
     if "access" in tags {}
     if "error" in tags {}
}

Can you send me messages log samples ?

Did you try to use your pattern in the grok debugger to see if it was matching the message ?

Woow ... you are too fast ... well thanks again for such bang on response ... Sure let me try that pattern with the GROK pattern debugger. Also, how can I share you the log message of nginx log and the kibana view personally ?

Sorry but I could not locate Grok debugger inside Dev tools of Kibana UI. My version is 7.7 is there any other way?

https://XXX:5601/app/kibana#/dev_tools/grokdebugger

Thanks for the tip @grumo35 but that URL change is still not redirecting me to the grokedebugger. May AWS ES kibana does not support it?

Well that's very strange you can try this one https://grokdebug.herokuapp.com/

The input is your messege content.

Thank you @grumo35 , that link really helped to troubleshoot the grok filter :slight_smile:

Hey @grumo35, hope you are doing well ... we have a fresh issue with the logstash wherein it is not able to push the access logs to kibana because we had enabled the request and response body in the nginx logs. This was working since last one week but just now it broke and no access logs are being sent. Please help

[2020-10-22T18:28:42,054][WARN ][logstash.filters.grok ][main][75cf125bb85824cfe019239fc09ae1cbd6720de4350077c150c98a9c4db7efc9] Timeout executing grok '(?<timestamp>%{YEAR}[./]%{MONTHNUM}[./]%{MONTHDAY} %{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER:threadid}\: \*%{NUMBER:connectionid} %{GREEDYDATA:errormessage}, client: %{IP:client}, server: %{GREEDYDATA:server}, request: \"(?<httprequest>%{WORD:httpcommand} %{UNIXPATH:httpfile} HTTP/(?<httpversion>[0-9.]*))\"(, )?(upstream: \"(?<upstream>[^,]*)\")?(, )?(host: \"(?<host>[^,]*)\")?, referrer: %{GREEDYDATA:referrer}?' against field 'message' with value 'Value too large to output (290 bytes)! First 255 chars are: 2020/10/22 17:35:56 [error] 21124#21124: *2912000 open() "/etc/nginx/html/eRetailWebsel/CompanyLocation.action" failed (2: No such file or directory), client: 114.143.221.54, server: 10.227.11.20, request: "GET /eRetailWebsel/CompanyLocation.action HTTP/1.'!

My grok filter is this

grok {
      match => [ "message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields}"]
      overwrite => [ "message" ]
    }

No, it is not. The grok filter that is timing out has multiple occurrences of GREEDYDATA inside the pattern (not at the end). COMBINEDAPACHELOG does not. You should try to make the GREEDYDATA patterns more specific. NOTSPACE or even DATA may speed things up enough to avoid timeouts.

Hello Badger, thank you for the response. Yes that indeed cleared the issue.

Hello Team,

We are again facing issues with the grok filter timing out. Below is the error log

[2020-11-13T16:41:35,379][WARN ][logstash.filters.grok ][main][58c88f4471fa659dd678b65812683f51eea291b14f022ff42d6e99a655baaa7c] Timeout executing grok '(?<timestamp>%{YEAR}[./]%{MONTHNUM}[./]%{MONTHDAY} %{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER:threadid}\: \*%{NUMBER:connectionid} %{DATA:errormessage}, client: %{IP:client}, server: %{DATA:server}, request: \"(?<httprequest>%{WORD:httpcommand} %{UNIXPATH:httpfile} HTTP/(?<httpversion>[0-9.]*))\"(, )?(upstream: \"(?<upstream>[^,]*)\")?(, )?(host: \"(?<host>[^,]*)\")?, referrer: %{DATA:referrer}?' against field 'message' with value 'Value too large to output (283 bytes)! First 255 chars are: 2020/11/13 14:02:50 [crit] 17022#17022: *3829654 open() "/var/cache/nginx/client_temp/0005303216" failed (24: Too many open files), client: 116.203.129.7, server: 10.227.11.20, request: "POST /RestWS/api/sellerPanel/v3/inventoryPriceUpdate HTTP/1.1", host:'!

Also, our grok filter pattern is like this

(?<timestamp>%{YEAR}[./]%{MONTHNUM}[./]%{MONTHDAY} %{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER:threadid}\: \*%{NUMBER:connectionid} %{DATA:errormessage}, client: %{IP:client}, server: %{DATA:server}, request: \"(?<httprequest>%{WORD:httpcommand} %{UNIXPATH:httpfile} HTTP/(?<httpversion>[0-9.]*))\"(, )?(upstream: \"(?<upstream>[^,]*)\")?(, )?(host: \"(?<host>[^,]*)\")?, referrer: %{DATA:referrer}?

Please help to resolve this ... actually this error is repeatedly getting logged into logstash logs due to which access logs are not getting pushed to ES but the error logs are getting pushed somehow.

Don't use UNIXPATH. It is extremely expensive when it fails to match.