Cannot get indexes made by logstash

Elastic Stack Logstash
1

I think my logstash is not working,

Here is the configuration of logstash,

input {
  file {
     type => nginx
     path => "/var/log/nginx/kibana_access.log"
    }
 }
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    manage_template => false
    index => "%{type}-index"
    document_type => "[type]"
  }
}

But I cannot see any kind of new index made in elasticsearch made by this,
This is the command I use to get all indexes,

curl -XGET 'localhost:9200/_cat/indices?v&pretty'

Why so ? Both services are up and running.

2

I would first check the Logstash logs to see what's going on.

3

@pablosan
Looking at logs, I think the logstash is doing fine, Here have a look,
I have shortened them a bit, as the body was exceeding 7000 characters,

[2017-04-21T13:27:30,211][DEBUG][logstash.runner          ] 
The following is the merged configuration
[2017-04-21T13:27:30,211][DEBUG][logstash.runner          ] 
input {
  file {
    type => nginx
    path => "/var/log/nginx/kibana_access.log"
   }
 }

output {
elasticsearch {
   hosts => ["localhost:9200"]
   manage_template => false
   index => "%{type}-index"
   document_type => "[type]"
 }
}


 [2017-04-21T13:27:30,235][DEBUG][logstash.basepipeline    ] Compiled pipeline code {:code=>"        @inputs = []\n        @filters = []\n        @outputs = []\n        @periodic_flushers = []\n        @shutdown_flushers = []\n        @generated_objects = {}\n\n          @generated_objects[:input_file_1] = plugin(\"input\", \"file\", LogStash::Util.hash_merge_many({ \"type\" => (\"nginx\") }, { \"path\" => (\"/var/log/nginx/kibana_access.log\") }))\n\n          @inputs << @generated_objects[:input_file_1]\n\n          @generated_objects[:output_elasticsearch_2] = plugin(\"output\", \"elasticsearch\", LogStash::Util.hash_merge_many({ \"hosts\" => [(\"localhost:9200\")] }, { \"manage_template\" => (\"false\") }, { \"index\" => (\"%{type}-index\") }, { \"document_type\" => (\"[type]\") }))\n\n          @outputs << @generated_objects[:output_elasticsearch_2]\n\n  define_singleton_method :filter_func do |event|\n    events = [event]\n    @logger.debug? && @logger.debug(\"filter received\", \"event\" => event.to_hash)\n    events\n  end\n  define_singleton_method :output_func do |event|\n    targeted_outputs = []\n    @logger.debug? && @logger.debug(\"output received\", \"event\" => event.to_hash)\n    targeted_outputs << @generated_objects[:output_elasticsearch_2]\n    \n    targeted_outputs\n  end"}
[2017-04-21T13:27:30,261][DEBUG][logstash.plugins.registry] On demand adding plugin to the registry {:name=>"file", :type=>"input", :class=>LogStash::Inputs::File}
[2017-04-21T13:27:30,270][DEBUG][logstash.plugins.registry] On demand adding plugin to the registry {:name=>"plain", :type=>"codec", :class=>LogStash::Codecs::Plain}
[2017-04-21T13:27:30,273][DEBUG][logstash.codecs.plain    ] config LogStash::Codecs::Plain/@id = "plain_61376031-3058-4698-969e-e611013fa608"
[2017-04-21T13:27:30,274][DEBUG][logstash.codecs.plain    ] config LogStash::Codecs::Plain/@enable_metric = true
[2017-04-21T13:27:30,274][DEBUG][logstash.codecs.plain    ] config LogStash::Codecs::Plain/@charset = "UTF-8"
[2017-04-21T13:27:30,275][DEBUG][logstash.inputs.file     ] config LogStash::Inputs::File/@type = "nginx"
[2017-04-21T13:27:30,276][DEBUG][logstash.inputs.file     ] config LogStash::Inputs::File/@path = ["/var/log/nginx/kibana_access.log"]
   [2017-04-21T13:27:30,277][DEBUG][logstash.inputs.file     ] config LogStash::Inputs::File/@stat_interval = 1
[2017-04-21T13:27:30,277][DEBUG][logstash.inputs.file     ] config LogStash::Inputs::File/@discover_interval = 15
[2017-04-21T13:27:30,277][DEBUG][logstash.inputs.file     ] config LogStash::Inputs::File/@sincedb_write_interval = 15
[2017-04-21T13:27:30,279][DEBUG][logstash.inputs.file     ] config LogStash::Inputs::File/@start_position = "end"
[2017-04-21T13:27:30,279][DEBUG][logstash.inputs.file     ] config LogStash::Inputs::File/@delimiter = "\n"
[2017-04-21T13:27:30,279][DEBUG][logstash.inputs.file     ] config LogStash::Inputs::File/@close_older = 3600
[2017-04-21T13:27:30,484][DEBUG][logstash.plugins.registry] On demand adding plugin to the registry {:name=>"elasticsearch", :type=>"output", :class=>LogStash::Outputs::ElasticSearch}
[2017-04-21T13:27:30,495][DEBUG][logstash.codecs.plain    ] config LogStash::Codecs::Plain/@id = "plain_96e60119-2cc8-4dc5-9ba7-26d0feb37ed6"
[2017-04-21T13:27:30,496][DEBUG][logstash.codecs.plain    ] config LogStash::Codecs::Plain/@enable_metric = true
[2017-04-21T13:27:30,497][DEBUG][logstash.codecs.plain    ] config LogStash::Codecs::Plain/@charset = "UTF-8"
[2017-04-21T13:27:30,504][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@hosts = [//localhost:9200]
[2017-04-21T13:27:30,504][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@manage_template = false
[2017-04-21T13:27:30,504][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@index = "%{type}-index"
[2017-04-21T13:27:30,504][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@document_type = "[type]"
[2017-04-21T13:27:30,505][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@id = "fd36a650089d7d9fbc7505a2c84eaae8af276ba6-2"
[2017-04-21T13:27:30,505][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@enable_metric = true
[2017-04-21T13:27:30,505][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@codec = <LogStash::Codecs::Plain id=>"plain_96e60119-2cc8-4dc5-9ba7-26d0feb37ed6", enable_metric=>true, charset=>"UTF-8">
[2017-04-21T13:27:30,505][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@workers = 1
[2017-04-21T13:27:30,505][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@template_name = "logstash"
[2017-04-21T13:27:30,505][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@template_overwrite = false
[2017-04-21T13:27:30,506][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@parent = nil
[2017-04-21T13:27:30,506][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@idle_flush_time = 1
[2017-04-21T13:27:30,506][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@upsert = ""
[2017-04-21T13:27:30,506][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@doc_as_upsert = false
[2017-04-21T13:27:30,506][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@script = ""
[2017-04-21T13:27:30,506][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@script_type = "inline"
[2017-04-21T13:27:30,507][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@retry_on_conflict = 1
[2017-04-21T13:27:30,507][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@pipeline = nil
[2017-04-21T13:27:30,520][INFO ][logstash.runner          ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
4

You can use the stdout debug to see if there are any logs being processed, and probably set the log level to trace in logstash.yml to try to understand why the logs are not picked up.

stdout {
  codec => rubydebug
}

You were testing the config there, right?

One thing that could be happening is that the log files are old and logstash won't use them.

5

@pablosan it gives this ,

[2017-04-21T14:35:50,474][FATAL][logstash.runner          ] The given configuration is invalid. Reason: Couldn't find any input plugin named 'stdout'. Are you sure this is correct? Trying to load the stdout input plugin resulted in this error: Problems loading the requested plugin named stdout of type input. Error: NameError NameError
6

Sorry I probably didn't explain where to put it, it goes in the output tag

https://www.elastic.co/guide/en/logstash/current/plugins-outputs-stdout.html

7

One thing is that ,

[2017-04-21T14:47:03,712][INFO ][logstash.runner          ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

It is using config_test_and_exit mode, I have not told it to do so. Can this be the problem ?

8

Most likely, it's not really running, just checking the config, at least you know your config si good :smile:

Please check how are you starting it.

9

The biggest thing I am not getting is that, port 5044 is opened.

I can check that by sudo netstat -ntlp

10

ps aux | grep logstash?

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.