Unable to create index after x-pack

Elastic Stack Logstash
1

Hello, Filebeat, Logstash, Elasticsearch were working sucessfully until I installed x-pack.
Now when I try to index data, I get this:
[2017-02-26T16:10:45,438][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>404, :action=>["index", {:_id=>nil, :_index=>"logstash-2017.02.26", :_type=>"log", :_routing=>nil}, 2017-02-26T21:10:39.878Z student 86.1.76.62 - - [04/Jan/2015:05:30:37 +0000] "GET /style2.css HTTP/1.1" 200 4877 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0"], :response=>{"index"=>{"_index"=>"logstash-2017.02.26", "_type"=>"log", "_id"=>nil, "status"=>404, "error"=>{"type"=>"index_not_found_exception", "reason"=>"no such index", "resource.type"=>"index_expression", "resource.id"=>"logstash-2017.02.26", "index_uuid"=>"na", "index"=>"logstash-2017.02.26"}}}}

I have setup new user & role for logstash per documentation
I also updated .conf to include credentials:
output {
elasticsearch {
hosts => [ "localhost:9200" ]
user => logstash_internal
password => changeme
}

Can you you help why Logstash is not able to create new index?

Thanks,
AC

2

Did you update the index name before running the post ? If not, you only have permission to create an index starting with logstash-*.

POST _xpack/security/role/logstash_reader
{
  "indices": [
    {
      "names": [ "logstash-*" ], 
      "privileges": ["read","view_index_metadata"]
    }
  ]
}
3

Jimmy,

I'm not sure which index to rename. So logstash is supposed to create a new index from the data coming from filebeat. It should have created index logstash-2017.02.26 but I don't think it's able to create the index.

Here's the .conf:
input {
beats {
port => "5077"
}
}

filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
geoip {
source => "clientip"
}
}
output {
elasticsearch {
hosts => [ "localhost:9200" ]
user => logstash_internal
password => changeme
}

}

I also added logstash_reader role as you suggested and assigned to logstash_internal?

Ankit

4

Since the data is coming in from beat, the index that will be created will be something "beat" .e.g. packetbeat-*.

If you followed the directions on the doc, the post api only created the role with permission to logstash-* indices . The problem is your user does not have access to create the "beat" indice e.g. packbeat-* .

You will need to go into Kibana > Management > roles > and put " * " for the indices field or specify the exact beat e.g. packetbeat-* and not logstash-*.

5

I changed the indices AND privileges field to * for both logstash_writer and logstash_reader but no luck.
Also, before x-pack, when everything was running okay, filebeat was pushing data into logstash and indices were getting created with logstash- prefix. And I think that's was logstash is still trying to create the index but not able to.

Any other ideas?

6

Yes, you are right . From beat > logstash > indices will have logstash-prefix. When going from beat > ES > indices will have beat-prefix e.g. packetbeat-

You can't change privileges to * . You have to select the drop down option "all". For the indices field you can set to * .

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.