Hi All,
Once again, please be kind. im 2 days into this and still getting round the learning curve.
ES 5.4.2 installed from tar.gz
filebeat-5.4.2-1.x86_64 Installed from RPM
logstash-5.4.3-1.noarch Installed from RPM
I have been trying to configure Filebeat -> Logstash -> Elasticsearch using this URL
https://www.elastic.co/guide/en/logstash/current/advanced-pipeline.html
Here is my logstash first-pipline,conf file
input {
beats {
port => "5043"
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
geoip {
source => "clientip"
}
}
output {
stdout { codec => rubydebug }
}
When i execute this against the tutorial data logstash fires up and I get a nice bit of json output in the rubydebug
{
"request" => "/style2.css",
"agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36\"",
"geoip" => {
"timezone" => "America/Denver",
"ip" => "71.212.224.97",
"latitude" => 39.8663,
"continent_code" => "NA",
"city_name" => "Denver",
"country_name" => "United States",
"country_code2" => "US",
"dma_code" => 751,
"country_code3" => "US",
"region_name" => "Colorado",
"location" => {
"lon" => -105.0061,
"lat" => 39.8663
},
"postal_code" => "80260",
"region_code" => "CO",
"longitude" => -105.0061
},
"offset" => 20664,
"auth" => "-",
"ident" => "-",
"input_type" => "log",
"verb" => "GET",
"source" => "/tmp/tutorial/logstash-tutorial.log",
"message" => "71.212.224.97 - - [04/Jan/2015:05:27:35 +0000] \"GET /style2.css HTTP/1.1\" 200 4877 \"http://www.semicomplete.com/projects/xdotool/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, li ke Gecko) Chrome/32.0.1700.107 Safari/537.36\"",
"type" => "log",
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"referrer" => "\"http://www.semicomplete.com/projects/xdotool/\"",
"@timestamp" => 2017-06-27T19:59:20.797Z,
"response" => "200",
"bytes" => "4877",
"clientip" => "71.212.224.97",
"@version" => "1",
"beat" => {
"hostname" => "bravo",
"name" => "bravo",
"version" => "5.4.2"
},
"host" => "bravo",
"httpversion" => "1.1",
"timestamp" => "04/Jan/2015:05:27:35 +0000"
}
My problem is when I move it across to output to elastic search and then try and query the data I get the following error message.
[log@bravo config]# curl --user USERNAME:PASSWORD -XGET 'localhost:9200/logstash-2017-06-27/_search?pretty&q=response=200'
{
"error" : {
"root_cause" : [
{
"type" : "index_not_found_exception",
"reason" : "no such index",
"resource.type" : "index_or_alias",
"resource.id" : "logstash-2017-06-27",
"index_uuid" : "_na_",
"index" : "logstash-2017-06-27"
}
],
"type" : "index_not_found_exception",
"reason" : "no such index",
"resource.type" : "index_or_alias",
"resource.id" : "logstash-2017-06-27",
"index_uuid" : "_na_",
"index" : "logstash-2017-06-27"
},
"status" : 404
}
I read the following page but this did not seem to work for me either
https://www.elastic.co/guide/en/x-pack/current/logstash.html
I think its an authentication problem but I just cant nail it down.