Hi,
We are using Logstash 2.0 to move a couple of months of log data (3000 files, 10G total) from Amazon S3 into Elasticsearch (1.7.1), but it's taking a very long time - roughly 1 hour to get through a day of data (200-300M, 1M lines). Only a small part of the incoming data needs to be indexed into ES (~100 docs per day), while the rest is discarded by our LS filters.
Logstash is configured with an S3 input, various filters to transform entries into JSON, and an Elasticsearch output:
input { s3 { ... bucket => "${s3Bucket}" prefix => "${s3Prefix}" delete => true backup_to_bucket => "${s3Bucket}" backup_add_prefix => "archive/" ...filter {
grok {
match => {"message" => "%{IP:client}\t%{NUMBER:[@metadata][timestamp]}\t%{QS:[@metadata][payload]}\t%{QS:agent}\t%{QS:[@metadata][referrer]}"}
... plus several others ...output {
if "_noresults" in [tags] {
# ignore event
} else if "_jsonparsefailure" in [tags] or "_grokparsefailure" in [tags] {
# log parse error
stdout {
codec => rubydebug
}
} else if [tags] {
# log unknown tag(s)
stdout {
codec => rubydebug
}
} else if [@metadata][type] {
elasticsearch {
hosts => ["${esHost}:9200"]
action => "update"
doc_as_upsert => true
index => "analytics-%{[@metadata][type]}-%{+YYYY.MM.dd}"
document_id => "%{[@metadata][doc_id]}"
document_type => "impression"
template => "/etc/logstash/analytics-template-v0.1.json"
template_name => "analytics"
template_overwrite => true
}
} else {
# log uncaught error
stdout {
codec => rubydebug
}
}
Our S3 files consist of lines like below, except some will have JSON in the third field (those are the ones we index):
11.222.333.444 1446649267.395 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36" "http://referrer"
Since so little data ends up indexed into ES, I would assume the problem lies in the input or filters?
Are there any performance settings we should set? With default settings, CPU ranges from 20-300% and heap seems OK (eden 10M/32M, old 63M/114M).
Would appreciate any hints about how to debug and what settings we should play with.
Thanks!