Logstash with Shield : Failed action with response of 404

Elastic Stack Logstash
1

Hello,

I have installed Elasticsearsh 1.4.4, Shield 1.1.1, Logstash 1.5.1.
I put this configuration for Logstash in /etc/logstash/conf.d/log.conf

input
{ 
    stdin {}
}
output {
    stdout { codec => rubydebug }
    elasticsearch {
	    user => "admin"
	    password => "password"
	    protocol => "http"    	
    }
}

But when I test it with the following command :

 /opt/logstash/bin/logstash -f /etc/logstash/conf.d/log.conf

Logstash start correctly

Logstash startup completed

So I put a string "test", and this is my result

failed action with response of 404, dropping action: ["index", {:_id=>nil, :_index=>"logstash-2015.06.22", :_type=>"logs", :_routing=>nil}, #<LogStash::Event:0x7ac7c9cc @metadata={"retry_count"=>0}, @accessors=#<LogStash::Util::Accessors:0x5a7ba8d4 @store={"message"=>"test", "@version"=>"1", "@timestamp"=>"2015-06-22T15:53:13.376Z", "host"=>"slnxelknode01.simat.ntes.ext.sopra"}, @lut={"host"=>[{"message"=>"test", "@version"=>"1", "@timestamp"=>"2015-06-22T15:53:13.376Z", "host"=>"slnxelknode01.simat.ntes.ext.sopra"}, "host"], "type"=>[{"message"=>"test", "@version"=>"1", "@timestamp"=>"2015-06-22T15:53:13.376Z", "host"=>"slnxelknode01.simat.ntes.ext.sopra"}, "type"]}>, @data={"message"=>"test", "@version"=>"1", "@timestamp"=>"2015-06-22T15:53:13.376Z", "host"=>"slnxelknode01.simat.ntes.ext.sopra"}, @metadata_accessors=#<LogStash::Util::Accessors:0x3397a6be @store={"retry_count"=>0}, @lut={}>, @cancelled=false>] {:level=>:warn, :file=>"logstash/outputs/elasticsearch.rb", :line=>"478", :method=>"submit"}

No index created in Elasticsearch

I tried with --debug, but it doesn't help me to solve the problem

Any idea to help me ?

Thanks

2

are you able to connect to the ES cluster using curl and basic auth? I would check that first to rule out any setup issues in ES Shield. If everything is fine with curl we can debug the LS side

3

I make the test with

curl -XGET 'http://localhost:9200/*?pretty' --user admin:password

I have no problem, all my index appear.
So I haven't problem with Elasticsearch. It really Logstash which have problem

4

When I put --verbose I have those messages which appear.

Using version 0.1.x input plugin 'stdin'. This plugin isn't well supported by the community and likely has no maintainer. {:level=>:info}
    Using version 0.1.x codec plugin 'line'. This plugin isn't well supported by the community and likely has no maintainer. {:level=>:info}
    Using version 0.1.x output plugin 'stdout'. This plugin isn't well supported by the community and likely has no maintainer. {:level=>:info}
    Using version 0.1.x codec plugin 'rubydebug'. This plugin isn't well supported by the community and likely has no maintainer. {:level=>:info}
    Using version 0.1.x output plugin 'elasticsearch'. This plugin isn't well supported by the community and likely has no maintainer. {:level=>:info}
    Using version 0.1.x codec plugin 'plain'. This plugin isn't well supported by the community and likely has no maintainer. {:level=>:info}

Before I was in version 1.4.2 of logstash and I it was working, I install the version 1.5.1 with the rpm to be compatible with shield, but apparently the versions of logstash plugins are not updated.

Can it explain why I have the message "failed action with response of 404" ? How can I update those plugin (my server is not connect to internet)?

5

You should ignore the "This plugin isn't well supported by the community" messages. It's unrelated to your 404 problem.

6

I find why I have this message.
In my config file of ES I have :

http.port: 9200

And in my config file of Logstash don't say the port in output elasticsearch. I used the default port. But in Logstash 1.5.1, the default port is not 9200. So I add in Logstash config :

elasticsearch {
    ...
    port => 9200
    ...
}

And now it works !

7

That's odd. According to the documentation, port 9200 is the default with protocol => http. It sounds like either the documentation or the code is buggy.

8

The default is indeed 9200 if you use http protocol. Can you stick your full ES config here (without the username/password)

9

Ah never mind, I see your config on top. This is weird. So adding the port fixed the issue?

10

This is my complete ES configuration: elasticsearch.yml :

marvel.agent.exporter.es.hosts: [ "username:password@slnxelknode01.simat.ntes.ext.sopra:9200" ]

action.auto_create_index: .marvel-*

shield:
  authc:
    realms:
      realm_name:
        type: esusers
        files:
          users: /usr/share/elasticsearch/config/shield/users
          users_roles: /usr/share/elasticsearch/config/shield/users_roles

cluster.name: simat    

bootstrap.mlockall: true

http.port: 9200

I know it's weird, but when I add the port in my logastash conf, it works !