Good day to you!
I am pretty new to ELK stack and i am working on a issue since 3 days.
I got an apache-access.log but i can't send the logfile through kibana.
Using Kibana 5.1.1, elasticsearch 5.1.1, logstash 5.1.1
Tried to keep my config file very simple:
input {
file {
path => "/home/grzechca/logfolder/jobvector_de_access.*"
type => "apache-access"
start_position => "beginning"
ignore_older => 0
sincedb_path => "/dev/null"
}
}
output {
elasticsearch { hosts => ["localhost:9200"] }
stdout { }
}
I start logstash with: sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/apache_to_elastic_Test01.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs to console
13:35:29.103 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>["http://localhost:9200"]}}
13:35:29.107 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:url=>#<URI::HTTP:0x4084382a URL:http://localhost:9200>, :healthcheck_path=>"/"}
13:35:29.294 [[main]-pipeline-manager] WARN logstash.outputs.elasticsearch - Restored connection to ES instance {:url=>#<URI::HTTP:0x4084382a URL:http://localhost:9200>}
13:35:29.302 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Using mapping template from {:path=>nil}
13:35:29.413 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword"}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
13:35:29.425 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["localhost:9200"]}
13:35:29.433 [[main]-pipeline-manager] INFO logstash.pipeline - Starting pipeline {"id"=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>125}
13:35:29.434 [[main]-pipeline-manager] INFO logstash.pipeline - Pipeline main started
13:35:29.574 [Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
^C13:35:29.777 [SIGINT handler] WARN logstash.runner - SIGINT received. Shutting down the agent.
2017-01-05T12:35:29.060Z ELK 198.100.145.140 - - [03/Jan/2017:06:25:10 +0100] "GET /en/index.html?__ajaxMethod=ajax_vacancy_search&keywords=&locations=&locations_lat=&locations_lng=&locations_string=&country%5B%5D=44&country%5B%5D=s2&country%5B%5D=s10&country%5B%5D=s1&country%5B%5D=s7&country%5B%5D=s11&country%5B%5D=s9&country%5B%5D=s3&country%5B%5D=s6&country%5B%5D=s14&country%5B%5D=s15&country%5B%5D=s16&country%5B%5D=s13&country%5B%5D=s4&country%5B%5D=s8&country%5B%5D=s12&country%5B%5D=s5&country%5B%5D=8&country%5B%5D=s22&country%5B%5D=s24&country%5B%5D=s25&country%5B%5D=s19&country%5B%5D=s20&country%5B%5D=s23&country%5B%5D=s17&country%5B%5D=s18&country%5B%5D=s21&sort=_score&_pn=30 HTTP/1.1" 200 47545 "http://www.jobvector.de/en/search-jobs.html?keywords=&locations=&locations_lat=&locations_lng=&locations_string=&country%5B%5D=44&country%5B%5D=s2&country%5B%5D=s10&country%5B%5D=s1&country%5B%5D=s7&country%5B%5D=s11&country%5B%5D=s9&country%5B%5D=s3&country%5B%5D=s6&country%5B%5D=s14&country%5B%5D=s15&country%5B%5D=s16&country%5B%5D=s13&country%5B%5D=s4&country%5B%5D=s8&country%5B%5D=s12&country%5B%5D=s5&country%5B%5D=8&country%5B%5D=s22&country%5B%5D=s24&country%5B%5D=s25&country%5B%5D=s19&country%5B%5D=s20&country%5B%5D=s23&country%5B%5D=s17&country%5B%5D=s18&country%5B%5D=s21&sort=_score&_pn=30" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36"
13:35:29.795 [LogStash::Runner] WARN logstash.agent - stopping pipeline {:id=>"main"}
2017-01-05T12:35:29.445Z ELK 136.243.152.18 - - [03/Jan/2017:06:25:10 +0100] "GET /?pdf_print=1 HTTP/1.1" 200 19975 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/)"
2017-01-05T12:35:29.464Z ELK 207.46.13.119 - - [03/Jan/2017:06:25:13 +0100] "GET /en/jobs/biology-life-sciences/editorial-staff-public-relations/weiterbildung-fuer-hochschulabsolventen-seminar-online-redakteur-schwerpunkt-new-media-m-w-75374.html HTTP/1.1" 200 19593 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
(...terminal is scrolling the whole logfile...)
on Kibana i use filebeat-*
in addition i didn't get this warning message before. I am wondering about that, because the mentioned logstash.yml file is in this folder. Can't use the --verbose and --debug modus.
System is a virtualmachine on virtualbox with Ubuntu 16.04 Server.
What did i wrong?
If you start LS via the console (ie