I am trying to have logstash parse through the apache log example. I have randomly succeeded in creating a logstash-$DATE stamp with the example "here and there" and when I do get the logstash-$DATE index it says “no such index” in debug node.
The step are:
Apache log is downloaded to the following folder: /home/suricata/logs
My first-pipeline.conf is:
input {
file {
path => "/home/suricata/logs/logstash-tutorial.log"
start_position => beginning
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
}
output {
elasticsearch {}
stdout {}
}
I run the following command:
root@elk:/etc/logstash/conf.d# /opt/logstash/bin/logstash -f first-pipeline.conf
Settings: Default pipeline workers: 2
Logstash startup completed
^CSIGINT received. Shutting down the pipeline. {:level=>:warn}
Logstash shutdown completed
In debug mode: I get a bunch of "adding patterns" and "replacement_patterns" short example:
Grok compiled OK {:pattern=>"%{COMBINEDAPACHELOG}", :expanded_pattern=>"(?:(?:(?IPORHOST:clientip(?:(?:(?:(?:((([0-9A-Fa-f]{1,4}:
And before this example I get the following:
Indices status:
root@elk:/etc/logstash/conf.d# curl -XGET 'localhost:9200/_cat/indices?v'
health status index pri rep docs.count docs.deleted store.size pri.store.size
yellow open alerts 5 1 0 0 795b 795b
yellow open logstash-2016.02.17 5 1 2 0 8.4kb 8.4kb
yellow open logstash-2016.02.15 5 1 1 0 4.7kb 4.7kb
yellow open .kibana 1 1 2 0 8.2kb 8.2kb
The two logstash index are not from this output.
When I run the:
curl -XGET 'localhost:9200/logstash-2016.02.17/_search?q=response=200'
{"took":1,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":0,"max_score":null,"hits":}}
But I do have a losgstash-2016.02.17 index...
Any suggestions on what I am doing wrong? I pretty much tore this problem apart and I hit a dead end.
Troubleshooting:
-No extra .since238402394830989 file is located in the home directory.
-ES & logstash services is up and running
-Configuration file is right
-I have been able to do a stdin and stdout logstash example
-Nodes are yellow but are running(probably because I have one node)
Errors:
-Index logstash does not exist even though the indices says they do.
-I do not get a "pretty font" when I have an output.
-reponse=200 is wrong