We have the following logstash configuration file which has been tested successfully using the configtest option,
input {
file {
path => "/etc/httpd/logs/clickstream.log"
}
}
filter {
grok { match => { "message" => "%{COMBINEDAPACHELOG} (?<cookie>[\w\W]*) (?<pageLoadTime>[\d]*)" } }
date { match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] }
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate { convert => [ "[geoip][coordinates]", "float"] }
mutate { add_field => { "map_location" => "%{[geoip][latitude]},%{[geoip][longitude]}" } }
urldecode { field => "referrer" }
useragent {
source => "agent"
target => "useragent"
}
mutate { add_field => { "browser_name" => "%{[useragent][name]} %{[useragent][major]}" } }
}
output {
elasticsearch {
hosts => ["http://10.0.0.163:9200"]
manage_template => false
index => "epicon-rp_wiki-%{+YYYY.MM.dd}"
document_type => "page_performance"
}
}
but when we started the logstash service, it does not seem to read any new content from the input file.
We can see the target file has been updated with new content and the logstash user has the read permission.
Somehow it works just fine if we start the logstash process manually as follow,
bin/logstash -f /etc/logstash/conf.d/rp_wiki.conf
We only have one configuration file as above.
We've also tried to remove the .sincedb* file but it still won't parse the input file.
There's no warn or error from the logstash.log, so not really sure what went wrong here.
Logstash 2.3.5 on redhat.
Appreciate any suggestion,