Logstash pipeline example apache.log

Elastic Stack Logstash
#1

I am doing my first example of logstash located in the Documents and I am not getting a proper curl output from querying "200" from the apache log tutorial.log file

I have configured my config file correctly(tripled checked). My --configtest ran fine. And so did the:
bin/logstash -f first-pipeline.conf
And when I issue:
curl -XGET 'localhost:9200/alerts/_search?q=response=200'
I get the following message:

{"took":3,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":
{"total":0,"max_score":null,"hits":[]}}

The only problem I could think of is I am querying the wrong index. But when you issue the -f first-pipeline.conf which index do we know it went too when we point the config file to a specific file location?

I have two indexes (.kibana and alerts). Any help on how to correctly curl the output of an apache log.

Here is my config file just in case you are wondering:

input {
file {
path => "/home/suricata/logstash-tutorial.log"
start_position => beginning
}
}

filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
geoip {
source=> "clientip"
}
}
output {
elasticsearch {}
stdout {}
}

#2

Please don't post images of text like that, it's hard to read and may not show up for some people :slightly_smiling:

What does the output from _cat/indices show?

#3

Mark,

Here is my out put states for my indices:

health status index pri rep docs.count docs.deleted store.size pri.store.size

yellow open .kibana 1 1 1 0 3.1kb 3.1kb

yellow open alerts 5 1 0 0 795b 795b

It looks like I get the following message after running logstash with my config file:

./logstash -f first-pipeline.conf
Settings: Default filter workers: 1
Logstash startup completed

And than I just hang there with a blinking cursor. So now I ran a simple config file with logstash using stdin and stdout and it works fine. But what I realized was that as I go through the steps of adding input..filter..output in the apache log example I do not once get an output I just get the same Logstash startup completed message at each step of the way. Let me know if you need anymore details.

#4

Ok, your alerts index has no docs in it, that 6th column in the cat output tells me that.

Chances are you have run into a sincedb issue. See if you can find and then delete the file.

#5

I had two .sincedb files at the same time. Deleted but I am still getting a blinking cursor when I issue the following:

HOME_NET/bin/logstash -f first-pipeline.conf

I would agree with you that the sincedb was part of the issue. Also, unless It takes more than a few minutes for logstash to log a apache log the problem still persists. (Imagine the cursor is blink on the next line)

Settings: Default filter workers: 1
Logstash startup completed

I will keep troubleshooting. Let me know if you need any more questions asked about details or others paths to confirm something.

UPDATE: Reading a log file into Logstash

There is more to this issue. I will update as I get more information after deciphering the debugging phase.

#6