I'm analyzing nginx logs with ELK.
For the logs were too large (10GB per day), I don't want to save all the detail data into es, but only the aggregations.And I found metrics
filter is suitable for my occasion.
What I wanna record was the counts of different groups for every ten minutes.
- counts of groups (ip, user-agent)
- counts of groups (ip, session_id, platform)
- and so on
However, the document of metric
filter is too simple for me to understand.
This is the config file I've written, it seemed not work.
input {
file {
path => "/path/to/access.log"
}
}
filter {
some_filter {
# grok,geoip,useragent to parse log
}
metrics {
add_field => {
"session_id" => "%{id}"
"time" => "%{t}"
"landing_page" => "%{lh}"
"period" => "%{p}"
"platform" => "%{pf}"
}
timer => ["aggs", "%{id} %{t} %{lh} %{p} %{pf}"]
add_tag => "me"
}
}
output {
if "me" in [tags] {
elasticsearch {
index => "user_record"
document_type => "tjj_access"
hosts => ["10.10.20.143:9200"]
}
#stdout { codec => json }
}
}