I'm trying to query elasticsearch with logstash elasticsearch input plugin and save the query result to files, messages in the file appears disordered, although I've put in a sort clause in the query string, and not all messages are there for the duration. Not sure whether this is expected, or something wrong with my configuration.
Elasticsearch/logstash version: 2.4
Logstash configuration as below:
input{
elasticsearch {
hosts => ["serverA", "serverB", "serverC"]
query => '{ "query": {"match_all": {}}, "filter": {"range": {"@timestamp": { "gte": "now-5m", "lt": "now"} } },"sort": [{"@timestamp": {"order": "asc"}}]} '
index => "logstash-*"
scroll => "5m"
}
}
filter {
ruby {
code => "
tstamp_ticks = event['@timestamp'].to_i
tstamp_gmt = Time.at(tstamp_ticks)
tstamp_local = tstamp_gmt.getlocal
event['time_str'] = tstamp_local.strftime('%Y-%m-%d-%H')
event['min_index'] = tstamp_local.min/5 "
}
}
output {
file {
codec => line { format => "%{@timestamp},%{message}"}
path => "/tmp/data/logstash-%{time_str}-%{min_index}.txt"
}
}