Logstash don't create index

Elastic Stack Logstash
1

Hi,
I run ELK services on ubuntu 16.04.
I have a csv file and I want to add it to elasticsearch, I use logstash for that:
First this is my conf file:

input {
file {
path => "/home/ahmed/Desktop/PopulationGenre.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
separator => ","
columns => ["Year", "Population", "Male", "Female", "Density (km²)"]
}
}

output {
elasticsearch {
hosts => "http://localhost:9200"
index => "popu"
document_type => "populationDensity"
}
stdout {}
}

I run this command for the logststash:

sudo /usr/share/logstash/bin/logstash --path.settings=/etc/logstash/ -f /home/ahmed/Desktop/population.conf

This is the output of the command line:

Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties

Any help please !

2

Are there any errors in /var/log/logstash/logstash-plain.log? I am thinking of a 'multiple instances' error, for example.

3 
[2017-11-13T15:45:09,275][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/mod$
[2017-11-13T15:45:09,284][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/m$
[2017-11-13T15:45:09,294][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/var/lib/logstash/que$
[2017-11-13T15:45:09,295][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/var/lib/$
[2017-11-13T15:45:09,321][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"bea36a1f-5a5a-41e2-b88$
[2017-11-13T15:45:09,461][ERROR][logstash.agent           ] Cannot create pipeline {:reason=>"Expected one of #, {, ,, ] at line 14, column 32$
[2017-11-13T15:47:37,131][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/mod$
[2017-11-13T15:47:37,138][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/m$
[2017-11-13T15:47:37,351][ERROR][logstash.agent           ] Cannot create pipeline {:reason=>"Expected one of #, {, ,, ] at line 14, column 32$
[2017-11-13T15:51:32,594][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/mod$
[2017-11-13T15:51:32,608][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/m$
[2017-11-13T15:51:33,023][ERROR][logstash.agent           ] Cannot create pipeline {:reason=>"Expected one of #, {, ,, ] at line 14, column 32$
[2017-11-13T16:28:46,335][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/mod$
[2017-11-13T16:28:46,364][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/m$
[2017-11-13T16:28:46,660][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/ve$
[2017-11-13T16:28:47,099][INFO ][logstash.agent           ] No config files found in path {:path=>"/etc/logstash/conf.d/*"}

I think the import log is:

  • No config files found in path {:path=>"/etc/logstash/conf.d/*"}

The folder /etc/logstash/conf.d is empty.

4 
[2017-11-13T19:59:44,330][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>401, :url=>"http://127.0.0.1:9200/_bulk"}
[2017-11-13T19:59:46,834][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://127.0.0.1:9200/, :path=>"/"}
[2017-11-13T19:59:46,838][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://127.0.0.1:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://127.0.0.1:9200/'"}

Also I found error to attempt elasticsearch from logstash.

5

You pass the path to the conf file on the command line, so it should not matter that there is not one in the default directory. What is the complete text of the "Cannot create pipeline" ERROR?

6

I have just change the path of hosts in elasticsearch field in the config file.

hosts => ["http://elastic:changeme@127.0.0.1:9200"]

and it works fine.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.