After installing x-pack on ELK, Logstash not create index to elasticsearch

Elastic Stack Logstash
1

Before x-pack installed on ELK, There was no any issue

This my logstash.conf file

input{
file{
path => "E:\ELK\data\actor.csv"
start_position => "beginning"
}
}
filter {
csv {
separator => ","
columns => ["actor_id","first_name","last_name","last_update"]
}
}
output{
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash-actor"
user => elastic
password => elastic5277
}
}

2

And what is the issue now? What is the error message in the logs, what is the behavior you are seeing/not seeing ?

  • Did you install X-Pack on Elasticsearch and Logstash
  • Did you set the passwords for the internal users with setup-passwords ?

It will be much easier for you if you start from our documentation and follow all the necessary steps. Then we can refer to the things you have done or not and find gaps or focus on specific steps that cause you any issues.

user => elastic
password => elastic

You really should not use the elastic super user for the Elasticsearch output plugin of Logstash. We have detailed instructions in our documentation for how to setup a dedicated user for that purpose.

3

I am using ELK 6.2.4 version
Yes i followed your documentation its very easy steps

according to your documentation
i installed x-pack on Elasticsearch ,even set the password for internal user
and installed x-pack on logstash and also add following lines in logstash.yml file

> xpack.monitoring.elasticsearch.url: http://localhost:9200
> xpack.monitoring.elasticsearch.username: elastic
> xpack.monitoring.elasticsearch.password: elastic5277

but i am seeing one error

Attempted to resurrect connection to dead ES instance, but got an error.

4

You need to set credentials so that the Elasticsearch output plugin of Logstash can authenticate to Elasticsearch. I shared the link to the documentation for that above

5

ok
once i configured credentials
i will reply whats happening

for now thanks for supporting me

6

Sorry for saying this
according to documentation I configured everything like

add _xpack/security/role/logstash_writer
add _xpack/security/user/logstash_internal
add _xpack/security/role/logstash_reader

and also update logstash.conf file like this

> input{
> 	elasticsearch {
> 		file{
> 			path => "E:\ELK\data\actor.csv"
> 			start_position => "beginning"
> 		}
> 
> 		user => logstash_internal
>     	password => "xpack5277"
> 	}
> }
> 
> filter {
> 	elasticsearch {
> 		csv {
> 	    	separator => ","
> 	    	columns => ["actor_id","first_name","last_name","last_update"]
> 	    }
> 	user => logstash_internal
> 	password => "xpack5277"
>   }
> }
> 
> output{
> 	elasticsearch {
> 		hosts => ["localhost:9200"]
> 		index => "logstash-actor"
> 		
> 		user => logstash_internal
> 		password => "xpack5277"
> 	}
> 
> 	stdout {
> 		codec => rubydebug
> 	}
> }

but still i getting some error in logstash like:

[ERROR][logstash.agent ] Failed to execute action

Some Warning

Detected a 6.x and above cluster: the type event field won't be used to determine the document _type
Restored connection to ES instance

7

You only need to update the config for your Elasticsearch output, not your csv filter or your file input ( these are not Elasticserarch related) Restore those two back to how you had them before.

For future reference, please share a much larger portion of your log file,

[ERROR][logstash.agent ] Failed to execute action

is not enough.

8

I restored both input and csv filter and run it on my terminal
there no any issue and also i am not getting any error

but when i refreshed elasticsearch in my browser i am not getting index what i written in my
conf file

I uploaded losgstash logs file
You can see there

log
log1372×823 468 KB

9

Please don't share screenshots. These are really diffcult to read and cannot be searched as text so other people can't take advantage of the discussion and possible solutions to your issue.

I don't see any errors for the last time you started Logstash. What is your issue? What goes wrong?

10

Next time i will care about this

Please don't share screenshots. These are really diffcult to read and cannot be searched as text so other people can't take advantage of the discussion and possible solutions to your issue.

My question is according to my logstash.conf file there must be a index named "logstash-actor" in elastisearch

But i don't know why such index not created
i am totally upset
i am trying this from yesterday and today itself
i don't know what's wrong going with me

11

Can you post your logstash.conf file as it is now and the output of running logstash in your terminal with that file ?

12

Yes, ofcource

but i have only permission to upload image not any file
when i try to upload any file
it gives error that you can only upload png/jpg extensions files

Can you give me idea to upload file ?

13

You don't need to upload the file. Copy the contents and paste them here (using the </> button so that it gets formatted correctly.

14

Conf File:

input{
	file{
		path => "E:\ELK\data\actor.csv"
		start_position => "beginning"
	}
}
filter {
  csv {
    	separator => ","
    	columns => ["actor_id","first_name","last_name","last_update"]
    }
}
output{
	elasticsearch {
		hosts => ["localhost:9200"]
		index => "logstash-actor"
		user => logstash_internal
		password => "xpack5277"
	}

	stdout {
		codec => rubydebug
	}
}

This is current lines seeingin my Terminal :

[2018-05-10T11:09:37,028][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elastic:xxxxxx@localhost:9200/, :path=>"/"}
[2018-05-10T11:09:37,049][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elastic:xxxxxx@localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://localhost:9200/'"}
[2018-05-10T11:09:37,090][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elastic:xxxxxx@localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://localhost:9200/'"}
[2018-05-10T11:09:37,603][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_internal:xxxxxx@localhost:9200/, :path=>"/"}
[2018-05-10T11:09:37,655][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_internal:xxxxxx@localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://localhost:9200/'"}
[2018-05-10T11:09:42,062][INFO ][logstash.licensechecker.licensereader] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elastic:xxxxxx@localhost:9200/, :path=>"/"}
[2018-05-10T11:09:42,104][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elastic:xxxxxx@localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://localhost:9200/'"}
[2018-05-10T11:09:42,104][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elastic:xxxxxx@localhost:9200/, :path=>"/"}
[2018-05-10T11:09:42,152][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elastic:xxxxxx@localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://localhost:9200/'"}
[2018-05-10T11:09:42,663][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_internal:xxxxxx@localhost:9200/, :path=>"/"}
[2018-05-10T11:09:42,719][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_internal:xxxxxx@localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://localhost:9200/'"}
[2018-05-10T11:09:47,112][INFO ][logstash.licensechecker.licensereader] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elastic:xxxxxx@localhost:9200/, :path=>"/"}
[2018-05-10T11:09:47,160][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elastic:xxxxxx@localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://localhost:9200/'"}
[2018-05-10T11:09:47,165][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elastic:xxxxxx@localhost:9200/, :path=>"/"}
[2018-05-10T11:09:47,208][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elastic:xxxxxx@localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://localhost:9200/'"}
[2018-05-10T11:09:47,726][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_internal:xxxxxx@localhost:9200/, :path=>"/"}
[2018-05-10T11:09:47,783][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_internal:xxxxxx@localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://localhost:9200/'"}
[2018-05-10T11:09:52,166][INFO ][logstash.licensechecker.licensereader] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elastic:xxxxxx@localhost:9200/, :path=>"/"}
[2018-05-10T11:09:52,211][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elastic:xxxxxx@localhost:9200/, :path=>"/"}
[2018-05-10T11:09:52,217][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elastic:xxxxxx@localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://localhost:9200/'"}

Yesterday there was no any error showing in terminal
but today when i started my terminal I getting same error what said previous conversations

15 
[2018-05-10T11:09:37,028][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elastic:xxxxxx@localhost:9200/, :path=>"/"}
[2018-05-10T11:09:37,049][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elastic:xxxxxx@localhost:9200/",

As I mentioned a couple of times yesterday, you need to set the configuration correctly for logstash monitoring. You need to stop using the elastic user and use the logstash_system user as described in our documentation in the following settings :

xpack.monitoring.elasticsearch.username: 
xpack.monitoring.elasticsearch.password: 

[2018-05-10T11:09:47,783][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_internal:xxxxxx@localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://localhost:9200/'"}

This also means that the password you are using for logstash_internal user is not correct. I know you went through the documentation yesterday and created the logstash_internal user. Did you follow the documentation in setting the password properly? You can verify that the password you are using in your config is correct by running

curl -u logstash_system 'http://localhost:9200/_xpack/security/_authenticate?pretty'

from your terminal

16

Let me try x-pack installation process again through documentation
After that I will tell you what I did step by step

Then you tell me which step I missed.

I know it well take some time but i will post defensively.

1 Like
17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.