Logstash pipeline example apache.log

I am doing my first example of logstash located in the Documents and I am not getting a proper curl output from querying "200" from the apache log tutorial.log file

I have configured my config file correctly(tripled checked). My --configtest ran fine. And so did the:
bin/logstash -f first-pipeline.conf
And when I issue:
curl -XGET 'localhost:9200/alerts/_search?q=response=200'
I get the following message:

{"took":3,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":
{"total":0,"max_score":null,"hits":}}

The only problem I could think of is I am querying the wrong index. But when you issue the -f first-pipeline.conf which index do we know it went too when we point the config file to a specific file location?

I have two indexes (.kibana and alerts). Any help on how to correctly curl the output of an apache log.

Here is my config file just in case you are wondering:

input {
file {
path => "/home/suricata/logstash-tutorial.log"
start_position => beginning
}
}

filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
geoip {
source=> "clientip"
}
}
output {
elasticsearch {}
stdout {}
}

Please don't post images of text like that, it's hard to read and may not show up for some people :slightly_smiling:

What does the output from _cat/indices show?

Mark,

Here is my out put states for my indices:

health status index pri rep docs.count docs.deleted store.size pri.store.size

yellow open .kibana 1 1 1 0 3.1kb 3.1kb

yellow open alerts 5 1 0 0 795b 795b

It looks like I get the following message after running logstash with my config file:

./logstash -f first-pipeline.conf
Settings: Default filter workers: 1
Logstash startup completed

And than I just hang there with a blinking cursor. So now I ran a simple config file with logstash using stdin and stdout and it works fine. But what I realized was that as I go through the steps of adding input..filter..output in the apache log example I do not once get an output I just get the same Logstash startup completed message at each step of the way. Let me know if you need anymore details.

Ok, your alerts index has no docs in it, that 6th column in the cat output tells me that.

Chances are you have run into a sincedb issue. See if you can find and then delete the file.

I had two .sincedb files at the same time. Deleted but I am still getting a blinking cursor when I issue the following:

HOME_NET/bin/logstash -f first-pipeline.conf

I would agree with you that the sincedb was part of the issue. Also, unless It takes more than a few minutes for logstash to log a apache log the problem still persists. (Imagine the cursor is blink on the next line)

Settings: Default filter workers: 1
Logstash startup completed

I will keep troubleshooting. Let me know if you need any more questions asked about details or others paths to confirm something.

UPDATE: Reading a log file into Logstash - #5 by gruszeckim2

There is more to this issue. I will update as I get more information after deciphering the debugging phase.