Http input plugins is not behaving as expected

Elastic Stack Logstash
1

Hi, I am using http input plugins, which should be capable to expect json events, and the amount of data is huge (around 7,00,000 in 30 mins). after processing some data it stopped working and throwing error:

[13004ae0f9329a0b799a2bd79c4e1e9b537e35dcd25c826c7b2239b8a605faa6] uncaught error (in thread http-input-handler-executor[T#2])

My config file is simple:


input{
  http{
      port => "10008"
      codec => "json"
}
}

filter {
json {source => "message"}
date {
      match => ["export_time", "UNIX_MS"]
      target => "export_Time"
      timezone => "UTC"
     }
date {
      match => ["usage_end_time", "UNIX_MS" ]
      target => "usage_end_Time"
      timezone => "UTC"
     }
date {match => ["usage_start_time", "UNIX_MS" ]
      target => "usage_start_Time"
      timezone => "UTC"
     }
if [process][exit_code] { drop {}}

mutate {remove_field => ["http", "@version", "url", "user_agent", "export_time", "usage_end_time", "usage_start_time", "message", "event"] }

ruby { code => "event.set('[@metadata][prevmonth]', DateTime.now.prev_month.strftime('%Y.%m'))" }
mutate { add_field => { "[@metadata][indextime]" => "gcp-monthly%{[@metadata][prevmonth]}" } }

}
output {

if [billFrequency] == "monthly" {
      elasticsearch {
         hosts => "https://elkstack.domain.com:9200"
         index => "%{[@metadata][indextime]}"
         manage_template => true
         user => "myusername@domain.com"
         password => "mypassword"
     }
  }
  }

stdout { codec => rubydebug {metadata => true}}
}

please help to find the root cause of the error/ is there anything missing in config.

2

What is the complete error message? Also, please edit your post, select the configuration and click on </> in the toolbar above the edit pane to make the configuration more readable.

3

Hi @Badger,

Thanks for your reply!

I checked the logs thoroughly. Since it is processing lakhs of data in a less time span, that is the reason it had java heap size issue, which solved by increasing jvm.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.