I have been trying to follow this guide in order to try to understand this system:
https://www.elastic.co/guide/en/logstash/current/advanced-pipeline.html
In the middle of the page, it suggests that you issue a test query to Elasticsearch with the following command:
curl -XGET 'localhost:9200/logstash-$DATE/_search?q=response=200'
So I entered curl -XGET 'localhost:9200/logstash-2015.01.04/_search?q=response=200' since the logstash-tutorial.log starts with that date.
But when I do that, I get the following error:
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","index":"logstash-2015.01.04","resource.type":"index_or_alias","resource.id":"logstash-2015.01.04"}],"type":"index_not_found_exception","reason":"no such index","index":"logstash-2015.01.04","resource.type":"index_or_alias","resource.id":"logstash-2015.01.04"},"status":404}
My logstash version is 2.3.1 and my elasticsearch version is 2.3.2. It would seem my file is not being indexed by elasticsearch, but I do not know why or how to troubleshoot this. Please advise. Thank you.
As this question isn't related to the deprecated logstash-forwarder product you may get better replies if you edit your post and move it to the Logstash category.
Do you actually have a logstash-2015.01.04 index? If not, what indexes do you have? Which tutorial have you been following?
This is what I have been following:
Oh, sorry. I wasn't aware there was a complete example in there. AFAICT that example doesn't contain a date filter so the @timestamp field in your events (which is what's used to name the ES indexes) will contain the time when Logstash processed the event. Therefore, you should probably be using logstash-2016.05.17 or similar in your curl command.
Same result:
curl -XGET 'localhost:9200/logstash-2016.05.17/_search?q=response=200'
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","index":"logstash-2016.05.17","resource.type":"index_or_alias","resource.id":"logstash-2016.05.17"}],"type":"index_not_found_exception","reason":"no such index","index":"logstash-2016.05.17","resource.type":"index_or_alias","resource.id":"logstash-2016.05.17"},"status":404}
Again, what indexes do you have? Using the ES APIs to list indexes would be a good exercise.
Magnus - I'm following the exercises on the elasticsearch.co and by your responses I'm basically doing it wrong because I have no idea about the indexes - this is my first attempt at this and I've provided all of the information I have been given and/or know how to locate. If there is some other resource that is more current as far as learning how to set up these I am open to it. Please advise. Thank you.
Use Elasticsearch's cat indices API to list which Elasticsearch indexes you have. Then we can see if you have any indexes at all and, if you do, what they're named. If you don't have any indexes then something's wrong on the Logstash end.
This command gives me no results: curl 'localhost:9200/_cat/indices/'
So there's an issue with Logstash, then? This is the command I used to start logstash:
bin/logstash -f /etc/logstash/conf.d/first-pipeline.conf --debug > /var/log/logstash/test.log
The debug has been loading a lot of data into my test.log file like this:
{:timestamp=>"2016-05-18T10:23:25.113000-0500", :message=>"Flushing buffer at interval", :instance=>"#<LogStash::Outputs::ElasticSearch::Buffer:0x2926bfd5 @stopping=#<Concurrent::AtomicBoolean:0x2efc4e0e>, @last_flush=2016-05-18 10:23:24 -0500, @flush_thread=#<Thread:0x2f533951 run>, @max_size=500, @operations_lock=#<Java::JavaUtilConcurrentLocks::ReentrantLock:0xd80d1d>, @submit_proc=#<Proc:0x4b7558a@/home/jamen/Software/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.5-java/lib/logstash/outputs/elasticsearch/common.rb:57>, @flush_interval=1, @logger=#<Cabin::Channel:0x2050d971 @subscriber_lock=#<Mutex:0x1bf39a4a>, @data={}, @metrics=#<Cabin::Metrics:0x16167e9c @channel=#<Cabin::Channel:0x2050d971 ...>, @metrics={}, @metrics_lock=#<Mutex:0x1ed23f18>>, @subscribers={18556=>#<Cabin::Subscriber:0x6856b32d @options={}, @output=#<Cabin::Outputs::IO:0x140d84df @lock=#<Mutex:0x38e56c2>, @io=#<IO:fd 1>>>}, @level=:debug>, @buffer=[], @operations_mutex=#<Mutex:0x75e1dc60>>", :interval=>1, :level=>:debug, :file=>"logstash/outputs/elasticsearch/buffer.rb", :line=>"90", :method=>"interval_flush"}
@Jamen_McGranahan
In your outputs section, do you have stdout {}?
If so then comment out the elasticsearch line (prefix with a #) and make the stdout look like this:
stdout {
codec => rubydebug
}
then re-run without --debug and don't redirect the output to a file.
what do you see in the console?
This is all that is displaying on my console - it's been about 5 minutes so far:
bin/logstash -f /etc/logstash/conf.d/first-pipeline.conf
Settings: Default pipeline workers: 1
Pipeline main started
and here is my configuration file:
#The # character at the beginning of a line indicates a comment. Use
# comments to describe your configuration.
input {
file {
path => "/var/log/logstash/logstash-tutorial.log"
start_position => beginning
ignore_older => 0
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
geoip {
source => "clientip"
}
}
output {
# elasticsearch { }
stdout {
codec => rubydebug
}
}
I'm pretty sure Logstash is tailing the input file. Delete the sincedb file or set the file input's sincedb_path option to /dev/null if you want Logstash to always ignore the file that stores Logstash's current state. The file input documentation contains more information about sincedb.
Eureka! Once I put this in in my config file (in the input / file section), I got output:
sincedb_path => "/dev/null"
And now curl 'localhost:9200/_cat/indices/' shows me this: yellow open logstash-2016.05.18 5 1 100 0 232.8kb 232.8kb, and curl -XGET 'localhost:9200/logstash-2016.05.18/_search?q=response=200' gives me a nice JSON response. Thank you for your patience as I try to understand how all of this works together!