Logstash crashing

Elastic Stack Logstash
1

I'm using OSS version 7.10 of both filebeat and logstash on RHEL 7.3

Elastisearch "destination" is a Elastisearch 7.10 AWS Elastisearch domain

I can run a curl and insert data into the AWS ES however logstash keeps crashing with following

Jul 13 14:03:55 pb-mp01 logstash: [2021-07-13T14:03:55,535][ERROR][logstash.outputs.elasticsearch][main] Failed to install template. {:message=>"Got response code '401' contacting Elasticsearch at URL 'https://XXXXXXXXXXXXXXXXX.us-east-1.es.amazonaws.com:443/_xpack'", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:332:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:319:in `block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:414:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:318:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:326:in `block in Pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:162:in `get'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:378:in `get_xpack_info'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:57:in `ilm_ready?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elas
Jul 13 14:03:55 pb-mp01 logstash: ticsearch/ilm.rb:28:in `ilm_in_use?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:15:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/common.rb:218:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/common.rb:49:in `block in setup_after_successful_connection'"]}

filebeat.yml - contains

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["XXXXXXXXXXXXXXXXXXXXxx.us-east-1.es.amazonaws.com:443"]

  # Protocol - either `http` (default) or `https`.
  protocol: "https"

  # Authentication credentials - either API key or username/password.
  aws_access_key_id: 'XXXXXXXXXXXXXXXXXXXX'
  aws_secret_access_key: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXx'
  setup.ilm.enabled: false 
  ilm.enabled: false

logstash-apache conf file in /etc/logstash/conf.d

input {
  file {
    path => "/var/log/httpd/access_log"
    start_position => "beginning"
  }
}

filter {
  if [path] =~ "access" {
    mutate { replace => { "type" => "apache_access" } }
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
  }
  date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}

output {
  elasticsearch {
    hosts => ["https://XXXXXXXXXXXXXXXXXXXXXXX.us-east-1.es.amazonaws.com:443"]
    #ssl = true
    #region => "us-east-1"
    index => "pb_apache-%{+YYYY.MM.dd}"
  }
 stdout { codec => rubydebug }
}

xpack has also been disabled

 grep xpack /usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb
      #load_xpack unless LogStash::OSS
    def load_xpack

any suggestion would be much appreciated

2

Your error message is returning a 401 error code

Got response code '401' contacting Elasticsearch at URL

You need to supply the credentials for your elasticsearch cluster in your logstash output.

Also... your filebeat config is outputting to elasticsearch and your logstash is outputting to elasticsearch... do you want the filebeat to send to logstash first and then have logstash output to elasticsearch instead?

3

@AquaX thansk for the response. As seems to be normal for me I resolved this after I made this post AND got a decent nights sleep LOL

my solution was to redo / update my logstash and filebeats config based on this article with changes made for our environment

credentials are covered in my solution by the fact that the aws cli is installed and configured on the server with the correct IAM creds so creds not needed in config files

1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.