Hello ! I'm trying to run filebeat and i have one question.
Below is what i have done so far.
- Install Elasticstack ( Elasticsearch & kibana & logstash & filebeat )
and i succeed in making a pipeline Elasticsearch, Kibana, logstash ( use sample log data )
- sample data looks like this.
212.109.27.134 - - [30/Apr/2017:04:32:36 +0000] "GET /wp-login.php HTTP/1.1" 200 5226 "-" "Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0"
54.210.20.202 - - [30/Apr/2017:04:32:38 +0000] "POST /wp-cron.php?doing_wp_cron=1493526758.2967031002044677734375 HTTP/1.1" 200 - "http://sundog-soft.com/wp-cron.php?doing_wp_cron=1493526758.2967031002044677734375" "WordPress/4.7.4; http://sundog-soft.com"
*logstash.conf
input{
file {
path => "/home/wonki/access_log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter{
grok {
match => {"message" => "%{COMBINEDAPACHELOG}"}
}
date {
match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
}
}
output{
elasticsearch{
hosts => ["127.0.0.1:9200"]
}
stdout{
codec => rubydebug
}
}
Simply, when i run the logstash, this sample log data is parsed by logstash filter and stored in elasticsearch and also i can find it in kibana.
After made a link among elk, i tried to add filebeat.
So what i expected :
my sample access_log ( combined apache log format ) -> filebeat -> logstash ( parsed by filter and changed json format ) -> elasticsearch ( make index and store my sample data in elasticsearch ) -> kibana.
And what i do:
- install filebeat.
filebeat.yml
change configurations
type: log
enabled: true
paths:
- /home/wonki/access_log
(commenting it out)
output elasticsearch
(uncommenting)
output logstash
hosts: ["localhost:5044"]
- create index
filebeat setup --template -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'
- logstash configuration
-> filebeat.conf
input{
beats {
port => 5044
}
}
filter{
grok {
match => {"message" => "%{COMBINEDAPACHELOG}"}
}
date {
match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
}
}
output{
elasticsearch{
hosts => ["127.0.0.1:9200"]
index => "filebeat_example"
}
stdout{
codec => rubydebug
}
}
-
dashboard
filebeat setup --dashboard -
start filebeat
sudo service filebeat start
Filebeat run well but i cannot find filebeat_example index in elasticsearch and kibana.
But in kibana dashboard, i can see my example data from filebeat.
So, if i can find my filebeat_example index in elasticsearch and kibana, what should i do?
Did i miss some configurations??
And i wonder that the differences between filebeat input configuration and filebeat modules configuration. In filebeat module configuration, do they have the ability to parse log data? ( for example,
combined apache log data -> json data format parsing -> apache2 modules can do these things? )