Logstash 2.3.4 getting stuck while attempting to install template in elasticsearch

Elastic Stack Logstash
1

Cross-posting from stackoverflow:-

I am trying to upgradelogstash 1.5.3 to logstash 2.3.4

logstash config for 1.5.3 looks like below:-

input {
    kafka {
        zk_connect => "kafka:2181"
        group_id => "logstash"
        topic_id => "logstash_logs2"
        reset_beginning => false
        consumer_threads => 3
#       auto_offset_reset => "smallest"
    }
}

filter {
  if [app] == "walle_slowquery" or [app] == "walle_slowindex" {
    ruby {
        code => "event['timestamp'] = event['@timestamp']"
    }
  }
  grok {
    match => [
     "timestamp", "^(?<app_log_time>%{YEAR}-%{MONTHNUM}-%{MONTHDAY})"
    ]
  }
  mutate {
    rename => {
      "app_log_time" => "[@metadata][app_log_time]"
    }
  }
}

output {
  if [env] == "prod" or [env] == "common" {
    elasticsearch {
      index => "jabong-%{env}-%{app}-%{iver}-%{[@metadata][app_log_time]}"
      cluster => "elasticsearch"
      host => ["172.16.84.230:9300"]

      protocol => "transport"
    }
    file {
      path => "/var/log/shop/%{env}/%{app}/%{app}_%{host}_%{[@metadata][app_log_time]}.log"
    }
    stdout { codec => rubydebug }
  }
}

I modifed the logstash config as below for 2.3.4:-

input {
    kafka {
        zk_connect => "kafka:2181"
        group_id => "logstash"
        topic_id => "logstash_logs2"
        reset_beginning => false
        consumer_threads => 3
    }
}

filter {
  if [app] == "walle_slowquery" or [app] == "walle_slowindex" {
    ruby {
        code => "event['timestamp'] = event['@timestamp']"
    }
  }
  grok {
    match => [
     "timestamp", "^(?<app_log_time>%{YEAR}-%{MONTHNUM}-%{MONTHDAY})"
    ]
  }
  mutate {
    rename => {
      "app_log_time" => "[@metadata][app_log_time]"
    }
  }
}

output {
  if [env] == "prod" or [env] == "common" {
    elasticsearch_java {
      #For daily index creation used the time notation, Remove if not required.
      index => "jabong-%{env}-%{app}-%{iver}-%{[@metadata][app_log_time]}"
      cluster => "elasticsearch"
      network_host => "172.16.84.230"
      hosts => ["172.16.84.230:9300"]
      protocol => "transport"
    }
    file {
      path => "/var/log/shop/%{env}/%{app}/%{app}_%{host}_%{[@metadata][app_log_time]}.log"
    }
    stdout { codec => rubydebug }
  }
}

In elasticsearch I am using custom template for indices jabong-*

I am using Kafka 0.8 and logstash-input-kafka plugin is at 2.0.8 and Elasticsearch version 1.7.1

If I start the logstash in a verbose mode, logstash seems to be stuck with the below message:-

Using mapping template from {:path=>nil, :level=>:info, :file=>"logstash/outputs/elasticsearch/template_manager.rb", :line=>"6", :method=>"install_template"}
Attempting to install template {:manage_template=>{"template"=>"logstash-*", "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "omit_norms"=>true}, "dynamic_templates"=>[{"message_field"=>{"match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}, "fields"=>{"raw"=>{"type"=>"string", "index"=>"not_analyzed", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"string", "index"=>"not_analyzed"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"float"}, "longitude"=>{"type"=>"float"}}}}}}}, :level=>:info, :file=>"logstash/outputs/elasticsearch/template_manager.rb", :line=>"8", :method=>"install_template"}
2

How do you know the input is working okay, i.e. that Logstash actually has data to process? What happens if you disable the elasticsearch_java output? Any particular reason to use elasticsearch_java over the standard elasticsearch output?

3

I modified my output plugin as follows:-

output {

  if [env] == "prod" or [env] == "common" {
    stdout { codec => rubydebug }
  }
}

Logstash log said:-
Pipeline main started {:file=>"logstash/agent.rb",:line=>"473", :method=>"start_pipeline"}

Also I started getting data on the stdout.

We are using transport protocol so we are using elasticsearch_java output.

4

Also I started getting data on the stdout.

Have you tried increasing the log level by starting Logstash with --verbose or even --debug? That might give extra clues.

We are using transport protocol so we are using elasticsearch_java output.

Yes, but why are you insisting on using the transport protocol?

5

Only after starting the logstash with --debug I got the template message that I mentioned in my first post. Because of space issue here I could not post the entire log here. But I have placed full debug log here.

It was legacy when the cluster was step-up couple of years back that time we observed some performance issues while using http protocol over the light transport protocol. This is working fine with logstash-1.5.3.

6

If it works without the elasticsearch_java output I'd assume that's where the problem lies. If you sniff the traffic, are any connections being made to the cluster? What if you replace it with a (HTTP based) elasticsearch output?

7

If I switch to elasticsearch output then it is working fine:-

output {
  if [env] == "prod" or [env] == "common" {
    elasticsearch {
      index => "jabong-%{env}-%{app}-%{iver}-%{[@metadata][app_log_time]}"
      hosts => ["es-master1","es-master2","es-master3"]
    }
    file {
      path => "/var/log/shop/%{env}/%{app}/%{app}_%{host}_%{[@metadata][app_log_time]}.log"
    }
    stdout { codec => rubydebug }
  }
}

The issue seems to be only with elasticsearch_java output plugin. Is that an issue on elasticsearch_java? Should I file an issue?

8

It certainly sounds like a bug in the elasticsearch_java plugin, if nothing else that it doesn't seem to log much to help out in situations like this. I wouldn't wait on this to be fixed though as the plugin isn't particularly actively maintained.