Logstash on AWS Elasticsearch Service - UNEXPECTED POOL ERROR | Cant install template

regic · September 9, 2020, 12:12am

Hi,

I have an Elasticsearch service running on an EC2 instance (Linux/Redhat) and I am trying to setup a Logstash service but getting a template error. Anyone have any ideas?

Setup

• AWS Elasticsearch 7.7
• Logstash 7.9.0 on Red Hat Linux (EC2)

Startup (batch file)
CD logstash-7.9.0
./logstash -f /opt/elk/logstash-7.9.0/config/logstash.conf

My configuration file: logstash.conf)

  beats {
    ssl  => false 
    port  => 5044
  }
}
output {
  amazon_es {
    hosts => ["myinstance.es.amazonaws.com"]
    region => "us-east-1"
    aws_access_key_id => ''
    aws_secret_access_key => ''
    index => "%{[@metadata][beat]}-%{[@metadata][version]}"
    manage_template => true 
    template => "/opt/elk/logstash-7.9.0/plugins/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/elasticsearch-template-es7x.json"
  }
}

Logs (I renamed the domain name to “myinstance”)

Sending Logstash logs to /opt/elk/logstash-7.9.0/logs which is now configured via log4j2.properties
[2020-09-08T20:05:35,544][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.9.0", "jruby.version"=>"jruby 9.2.12.0 (2.5.7) 2020-07-01 db01a49ba6 OpenJDK 64-Bit Server VM 25.252-b09 on 1.8.0_252-b09 +indy +jit [linux-x86_64]"}
[2020-09-08T20:05:36,298][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-09-08T20:05:38,832][INFO ][org.reflections.Reflections] Reflections took 79 ms to scan 1 urls, producing 22 keys and 45 values
/opt/elk/logstash-7.9.0/plugins/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/http_client/pool.rb:33: warning: already initialized constant ROOT_URI_PATH
/opt/elk/logstash-7.9.0/plugins/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/http_client/pool.rb:36: warning: already initialized constant DEFAULT_OPTIONS
/opt/elk/logstash-7.9.0/plugins/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/http_client/pool.rb:160: warning: already initialized constant ES1_SNIFF_RE_URL
/opt/elk/logstash-7.9.0/plugins/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/http_client/manticore_adapter.rb:7: warning: already initialized constant DEFAULT_HEADERS
/opt/elk/logstash-7.9.0/plugins/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/http_client.rb:24: warning: already initialized constant TARGET_BULK_BYTES
/opt/elk/logstash-7.9.0/plugins/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/common.rb:8: warning: already initialized constant DOC_DLQ_CODES
/opt/elk/logstash-7.9.0/plugins/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/common.rb:9: warning: already initialized constant DOC_SUCCESS_CODES
/opt/elk/logstash-7.9.0/plugins/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/common.rb:10: warning: already initialized constant DOC_CONFLICT_CODE
/opt/elk/logstash-7.9.0/plugins/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/common.rb:16: warning: already initialized constant VERSION_TYPES_PERMITTING_CONFLICT
/opt/elk/logstash-7.9.0/plugins/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/common.rb:133: warning: already initialized constant VALID_HTTP_ACTIONS
/opt/elk/logstash-7.9.0/plugins/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/common.rb:247: warning: already initialized constant DEFAULT_EVENT_TYPE_ES6
/opt/elk/logstash-7.9.0/plugins/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/common.rb:248: warning: already initialized constant DEFAULT_EVENT_TYPE_ES7
url template
{:scheme=>nil, :user=>nil, :password=>nil, :host=>"URLTEMPLATE", :port=>443, :path=>nil}
[2020-09-08T20:05:40,910][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://myinstance.es.amazonaws.com:443/]}}
[2020-09-08T20:05:40,949][INFO ][logstash.outputs.elasticsearch][main] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://myinstance.es.amazonaws.com:443/, :path=>"/"}
[2020-09-08T20:05:41,974][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"https://myinstance.es.amazonaws.com:443/"}
[2020-09-08T20:05:42,102][INFO ][logstash.outputs.elasticsearch][main] ES Output version determined {:es_version=>7}
[2020-09-08T20:05:42,107][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2020-09-08T20:05:42,160][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//myinstance.es.amazonaws.com"]}
[2020-09-08T20:05:42,186][INFO ][logstash.outputs.elasticsearch][main] Using mapping template from {:path=>"/opt/elk/logstash-7.9.0/plugins/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/elasticsearch-template-es7x.json"}
[2020-09-08T20:05:42,242][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"_doc"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2020-09-08T20:05:42,294][WARN ][logstash.outputs.elasticsearch][main] UNEXPECTED POOL ERROR {:e=>#<URI::InvalidComponentError: bad component(expected absolute path component): _template/logstash>}
[2020-09-08T20:05:42,324][ERROR][logstash.outputs.elasticsearch][main] Failed to install template. {:message=>"bad component(expected absolute path component): _template/logstash", :class=>"URI::InvalidComponentError", :backtrace=>["uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/uri/generic.rb:771:in `check_path'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/uri/generic.rb:819:in `path='", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/uri/generic.rb:193:in `initialize'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/uri/generic.rb:138:in `build'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/uri/http.rb:62:in `build'", "/opt/elk/logstash-7.9.0/plugins/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/http_client/manticore_adapter.rb:99:in `perform_request'", "/opt/elk/logstash-7.9.0/plugins/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/http_client/pool.rb:291:in `perform_request_to_url'", "/opt/elk/logstash-7.9.0/plugins/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/http_client/pool.rb:278:in `block in perform_request'", "/opt/elk/logstash-7.9.0/plugins/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/http_client/pool.rb:373:in `with_connection'", "/opt/elk/logstash-7.9.0/plugins/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/http_client/pool.rb:277:in `perform_request'", "/opt/elk/logstash-7.9.0/plugins/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/http_client/pool.rb:285:in `block in Pool'", "/opt/elk/logstash-7.9.0/plugins/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/http_client.rb:358:in `template_exists?'", "/opt/elk/logstash-7.9.0/plugins/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/http_client.rb:84:in `template_install'", "/opt/elk/logstash-7.9.0/plugins/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/template_manager.rb:21:in `install'", "/opt/elk/logstash-7.9.0/plugins/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/template_manager.rb:9:in `install_template'", "/opt/elk/logstash-7.9.0/plugins/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/common.rb:118:in `install_template'", "/opt/elk/logstash-7.9.0/plugins/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/common.rb:49:in `block in install_template_after_successful_connection'"]}
[2020-09-08T20:05:42,344][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/opt/elk/logstash-7.9.0/config/logstash.conf"], :thread=>"#<Thread:0x37107e5f run>"}
[2020-09-08T20:05:43,583][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.23}
[2020-09-08T20:05:43,606][INFO ][logstash.inputs.beats    ][main] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2020-09-08T20:05:43,647][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-09-08T20:05:43,701][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-09-08T20:05:43,961][INFO ][org.logstash.beats.Server][main][cea700e0ff1e89f79b6a62e834d11f558ab095839616688d76cfd6eaedfb3d2b] Starting server on port: 5044
[2020-09-08T20:05:44,314][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

warkolm · September 9, 2020, 12:21am

Welcome to our community!
Please format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you

What does the template look like?

regic · September 9, 2020, 12:35am

Hi @warkolm

Thanks for the tip.

I am using the default template that comes with the amazon_es_output plugin, here's what the elasticsearch-template-es7x.json looks like:

{
  "template" : "logstash-*",
  "version" : 60001,
  "settings" : {
    "index.refresh_interval" : "5s",
    "number_of_shards": 1
  },
  "mappings" : {
    "_doc" : {
      "dynamic_templates" : [ {
        "message_field" : {
          "path_match" : "message",
          "match_mapping_type" : "string",
          "mapping" : {
            "type" : "text",
            "norms" : false
          }
        }
      }, {
        "string_fields" : {
          "match" : "*",
          "match_mapping_type" : "string",
          "mapping" : {
            "type" : "text", "norms" : false,
            "fields" : {
              "keyword" : { "type": "keyword", "ignore_above": 256 }
            }
          }
        }
      } ],
      "properties" : {
        "@timestamp": { "type": "date"},
        "@version": { "type": "keyword"},
        "geoip"  : {
          "dynamic": true,
          "properties" : {
            "ip": { "type": "ip" },
            "location" : { "type" : "geo_point" },
            "latitude" : { "type" : "half_float" },
            "longitude" : { "type" : "half_float" }
          }
        }
      }
    }
  }
}

warkolm · September 9, 2020, 12:36am

If you're using the default one, why are you adding it to the config?

regic · September 9, 2020, 12:51am

@warkolm i tried seeing if putting the exact location of the template in the conf file would solve the issue but it didn't.

Even if i comment out the template location, i still get the same error.

warkolm · September 9, 2020, 1:19am

Ok that's odd, it may be related to the output plugin, in which case we aren't able to help I am sorry, as it's a 3rd party plugin.

system · October 7, 2020, 1:19am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Logstash failed to create template on target server Logstash	4	420	December 21, 2020
Logstash -> Elasticsearch index template Elasticsearch	10	3530	August 28, 2020
Error while connecting to Amazon Elasticsearch Logstash	1	803	June 10, 2020
Elasticsearch output Logstash	3	551	March 3, 2020
Logstash crashing Logstash	3	620	August 11, 2021

Logstash on AWS Elasticsearch Service - UNEXPECTED POOL ERROR | Cant install template

Related topics