LS can't connect to ES with Shield

Elastic Stack Logstash
1

Hey all,

I've recently installed a new ES cluster with version 2.0, and Logstash at 1.5.5. This was done on 3 Centos 7.1 hosts. I'm running shield on this setup. Kibana is running behind nginx on the 1st node.

I then created a logstash user on each node using this command:

esusers useradd ls_user -r logstash

When I restart logstash, I'm seeing this output in kibana:

plugin:elasticsearch	 Authentication Exception

I'm using this as my output filter:

 output {
  elasticsearch {
    protocol => http
    host => ["xx.xx.xx.xx, "xx.xx.xx.xx", "xx.xx.xx.xxx"]
    index => 'logstash-%{+YYYY.MM.dd}'
    user => "ls_user"
    password => "secret"
  }

}

What am I doing wrong?

2

I'm not sure what the KB log has to do with this.... What does LS or ES say?

3

I'm getting these exceptions in my ES logs:

ES Exceptions

It's too many lines, so I had to pastebin it.

In Logstash I'm able to see this:

LS Logs

This is my entire input conf:

#cat 10-logstash-input.conf
input {


   lumberjack {
       # The port to listen on
       port => 2541

       # The paths to your ssl cert and key
       ssl_certificate => "/etc/pki/tls/certs/logstash.crt"
       ssl_key => "/etc/pki/tls/private/logstash.key"

         # Set this to whatever you want.
         type => "logstash"
         codec => "json"
       }


     syslog {
        type => "syslog"
        port => "5514"

    }
}

Also if I disable shield, everything works fine!

How can I get this working?

  [1]: http://pastebin.ca/3258836
4

Can you auth using that user directly to ES?

And I presume you've read the docs on integrating LS with Shield?

5

Yes I can auth with that user:

#curl --user ls_user http://localhost:9200
Enter host password for user 'ls_user':
{
  "name" : "JF_ES1",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "2.0.0",
    "build_hash" : "de54438d6af8f9340d50c5c786151783ce7d6be5",
    "build_timestamp" : "2015-10-22T08:09:48Z",
    "build_snapshot" : false,
    "lucene_version" : "5.2.1"
  },
  "tagline" : "You Know, for Search"
}

And I have read the docs about using LS with shield.

LS with shield

That's the doc I used to set this up! Any ideas why this isn't working?

7

from the pastebin, it looks like your logstash user might not be granted the proper roles. What are the roles that you've defined for it?

edit: Also, do any indices exist in your cluster?

8

Hi jaymode,

The user looks like he has the correct role assigned to me, unless I'm mistaken:

esusers list ls_user
ls_user        : logstash

And yeah I do have some logstash indices created.

[root@logs:~] #curl --user admin:$ES_PASS  -XGET http://localhost:9200/_cat/indices
green open logstash-2015.11.19 5 1  7011724 0  6.4gb  3.2gb
green open .kibana             1 1        2 0 10.5kb  5.2kb
green open logstash-2015.11.17 5 1 20507975 0 17.9gb  8.9gb
green open logstash-2015.11.18 5 1 26372939 0 23.9gb 11.9gb

It seems that Logstash is functioning normally more or less. The problem is still that i can't get the ls_user to auth to elasticsearch from the LS config. Although as I demonstrated earlier he can auth to the ES cluster on the command line:

[root@logs:~] #curl --user ls_user:$ES_PASS localhost:9200
{
  "name" : "JF_ES1",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "2.0.0",
    "build_hash" : "de54438d6af8f9340d50c5c786151783ce7d6be5",
    "build_timestamp" : "2015-10-22T08:09:48Z",
    "build_snapshot" : false,
    "lucene_version" : "5.2.1"
  },
  "tagline" : "You Know, for Search"
}

I'm still seeing this output when I restart LS with the user login info:

plugin:elasticsearch	 Authentication Exception

Still open to suggestions!

9

I'm facing the same issue.
When I have my systems working without Shield, everything goes smooth. However, if Shield is enabled, LS can't connect to with ES. The log I get is from it is:
{:timestamp=>"2015-11-23T14:23:01.906000+0100", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://X.X.X.X:9200/\"]', but an error occurred and it failed! Are you sure you can reach elasticsearch from this machine using the configuration provided?", :client_config=>{:hosts=>["http://X.X.X.X:9200/"], :ssl=>nil, :transport_options=>{:socket_timeout=>0, :request_timeout=>0, :proxy=>nil, :ssl=>{}}, :transport_class=>Elasticsearch::Transport::Transport::HTTP::Manticore, :logger=>nil, :tracer=>nil, :reload_connections=>false, :retry_on_failure=>false, :reload_on_failure=>false, :randomize_hosts=>false}, :error_message=>"X.X.X.X:9200 failed to respond", :error_class=>"Manticore::ClientProtocolException", :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.4.4-java/lib/manticore/response.rb:35:in `initialize'", "org/jruby/RubyProc.java:271:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.4.4-java/lib/manticore/response.rb:70:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.4.4-java/lib/manticore/response.rb:245:in `call_once'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.4.4-java/lib/manticore/response.rb:148:in `code'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/http/manticore.rb:71:in `perform_request'", "org/jruby/RubyProc.java:271:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/base.rb:191:in `perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/http/manticore.rb:54:in `perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/client.rb:119:in `perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.0.14/lib/elasticsearch/api/actions/bulk.rb:87:in `bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.1.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:56:in `bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.1.2-java/lib/logstash/outputs/elasticsearch.rb:353:in `submit'", "org/jruby/ext/thread/Mutex.java:149:in `synchronize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.1.2-java/lib/logstash/outputs/elasticsearch.rb:350:in `submit'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.1.2-java/lib/logstash/outputs/elasticsearch.rb:382:in `flush'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:219:in `buffer_flush'", "org/jruby/RubyHash.java:1342:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:216:in `buffer_flush'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:193:in `buffer_flush'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:159:in `buffer_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.1.2-java/lib/logstash/outputs/elasticsearch.rb:343:in `receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/outputs/base.rb:80:in `handle'", "(eval):793:in `output_func'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/pipeline.rb:252:in `outputworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/pipeline.rb:169:in `start_outputs'"], :level=>:error}

I'm using Logstash 2.0, Elasticsearch 2.0 and Shield 2.0. And I checked that Logstash user is allowed to do what is has to do.

10

The problem I was having was actually with my kibana configuration. Because my kibana setup didn't have any authentication parameters in the config, it was actually kibana that couldn't reach ElasticSearch. Logstash had been talking to it the whole time, without issue.

You should check out this guide:

Kibana + Shield

Make sure you have a kibana user created so that you can log into ES with it. Use esusers to do that.

Then make sure that username and password is setup in your kibana.yml file:

elasticsearch.username: 'kibana4-server'
elasticsearch.password: 'secret'

That will probably solve your issue as it did mine!

11

That was already configured on my system... :frowning:

12

Then please start your own thread :slight_smile:

13

I was going to start my own thread on this very topic, when I came across this thread.

I ran into the same exact issue last night. I enabled Shield V2.0.2 on my machine. I then followed the documentation here to enable Shield for Logstash.

I have created lstash user for logstah and have granted the 'logstash' role but kept getting 403 authorization errors when LS tried to connect to ES. I even added another role (transport_client) for user lstash to no avail.

Finally, I just configured the Elastic output to use one of the Power_user ids that I've created and that seems to work, but it makes me thing that there is a problem in there somewhere - according to the documentation this should work and it doesn't.

Suggestions to try next, plz?

My env:
ES 2.0.x LS 2.0.x and latest Shield version=2.0.0.

14

hi
had the same problem
i found out that in the output section you need to write the user and password before you write any other features
like that :
elasticsearch {
hosts => ["es_host:9200"]
user => "logstash"
password => "password"
index => "index"
document_type => "type"

}

this way worked for me
i guss logstash thinks you didn't enterd any credentials cuse it dosent get to them .. even though you did

hope that works for you too :slight_smile: