I'm trying to log to an elastic cloud cluster but my logstash server keeps throwing this error whenever it receives logs:
[2017-07-26T06:42:27,934][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 500 ({"type"=>"illegal_state_exception", "reason"=>"There are no external requests known to support wildcards that don't support replacing their indices"})
My stack is Rails with logstash-logger -> logstash -> elasticsearch (cloud cluster).
Previously I had sucessfully got my apps to log to elasticsearch on AWS, however I decided to switch to elasticsearch cloud so I could use Security. Config as follows:
Rails & logstash-logger:
# application.rb (just testing on localhost for now)
# logstash
config.logstash.host = ENV.fetch('LOGSTASH_HOST', 'localhost')
config.logstash.port = ENV.fetch('LOGSTASH_PORT', 5228)
config.logstash.type = :tcp
Logstash:
# pipeline/logstash-listener.conf (local docker instance)
input {
tcp {
port => 5228
codec => json_lines
}
}
output {
elasticsearch {
hosts => "https://xxxx:9243"
user => "xxxx"
password => "xxxx"
index => "logstash-rib"
codec => json_lines
}
}
Kibana:
I've enabled Security/Shield and added a logstash user and a role with access to logstash-* indice and read/write/create index priveledges.
I've also tried connecting with the default user. Both accounts seem to connect just fine, but the error still occurs when logs are sent from my rails app.
This is the startup output from when I start my Logstash server:
docker run -p 5228:5228 -p 9600:9600 --rm 4dd045e77027
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
Sending Logstash's logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2017-07-26T06:54:30,528][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[2017-07-26T06:54:30,535][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[2017-07-26T06:54:30,570][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"b64d3028-18b1-4d41-bd5f-bceac607f0d3", :path=>"/usr/share/logstash/data/uuid"}
[2017-07-26T06:54:31,280][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash:xxxxxx@141fc805ec376afcd5b9b638849e7655.ap-southeast-2.aws.found.io:9243/]}}
[2017-07-26T06:54:31,282][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://logstash:xxxxxx@141fc805ec376afcd5b9b638849e7655.ap-southeast-2.aws.found.io:9243/, :path=>"/"}
[2017-07-26T06:54:31,877][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<Java::JavaNet::URI:0x1e3a9809>}
[2017-07-26T06:54:31,878][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2017-07-26T06:54:32,034][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2017-07-26T06:54:32,060][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>[#<Java::JavaNet::URI:0x16c246af>]}
[2017-07-26T06:54:32,061][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2017-07-26T06:54:32,079][INFO ][logstash.inputs.tcp ] Starting tcp input listener {:address=>"0.0.0.0:5228"}
[2017-07-26T06:54:32,086][INFO ][logstash.pipeline ] Pipeline main started
[2017-07-26T06:54:32,156][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
and then the error floods the screen:
[2017-07-26T06:58:27,257][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 500 ({"type"=>"illegal_state_exception", "reason"=>"There are no external requests known to support wildcards that don't support replacing their indices"})
[2017-07-26T06:58:27,258][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 500 ({"type"=>"illegal_state_exception", "reason"=>"There are no external requests known to support wildcards that don't support replacing their indices"})
[2017-07-26T06:58:27,259][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 500 ({"type"=>"illegal_state_exception", "reason"=>"There are no external requests known to support wildcards that don't support replacing their indices"})
[2017-07-26T06:58:27,261][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 500 ({"type"=>"illegal_state_exception", "reason"=>"There are no external requests known to support wildcards that don't support replacing their indices"})
[2017-07-26T06:58:27,262][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 500 ({"type"=>"illegal_state_exception", "reason"=>"There are no external requests known to support wildcards that don't support replacing their indices"})
[2017-07-26T06:58:27,262][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 500 ({"type"=>"illegal_state_exception", "reason"=>"There are no external requests known to support wildcards that don't support replacing their indices"})
[2017-07-26T06:58:27,263][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 500 ({"type"=>"illegal_state_exception", "reason"=>"There are no external requests known to support wildcards that don't support replacing their indices"})
Can anyone please shed some light on what that error means? I don't know where to begin...