# Missing and disordered when using logstash elasticsearch input and file output

**URL:** <https://discuss.elastic.co/t/missing-and-disordered-when-using-logstash-elasticsearch-input-and-file-output/75012>\
**Category:** Logstash\
**Created:** [February 14, 2017, 10:38am UTC](https://discuss.elastic.co/t/missing-and-disordered-when-using-logstash-elasticsearch-input-and-file-output/75012 "2017-02-14T10:38:01Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![zpp](https://avatars.discourse-cdn.com/v4/letter/z/54ee81/32.png) [@zpp](https://discuss.elastic.co/u/zpp)\
**Post date:** [February 14, 2017, 10:38am UTC](https://discuss.elastic.co/t/missing-and-disordered-when-using-logstash-elasticsearch-input-and-file-output/75012/1 "2017-02-14T10:38:01Z")

</div>

I'm trying to query elasticsearch with logstash elasticsearch input plugin and save the query result to files, messages in the file appears disordered, although I've put in a sort clause in the query string, and not all messages are there for the duration. Not sure whether this is expected, or something wrong with my configuration.  
Elasticsearch/logstash version: 2.4  
Logstash configuration as below:

input{  
elasticsearch {  
hosts =\> ["serverA", "serverB", "serverC"]  
query =\> '{ "query": {"match\_all": {}}, "filter": {"range": {"@timestamp": { "gte": "now-5m", "lt": "now"} } },"sort": [{"@timestamp": {"order": "asc"}}]} '  
index =\> "logstash-\*"  
scroll =\> "5m"  
}  
}

filter {  
ruby {  
code =\> "  
tstamp\_ticks = event['@timestamp'].to\_i  
tstamp\_gmt = [Time.at](http://Time.at)(tstamp\_ticks)  
tstamp\_local = tstamp\_gmt.getlocal  
event['time\_str'] = tstamp\_local.strftime('%Y-%m-%d-%H')  
event['min\_index'] = tstamp\_local.min/5 "  
}  
}

output {  
file {  
codec =\> line { format =\> "%{@timestamp},%{message}"}  
path =\> "/tmp/data/logstash-%{time\_str}-%{min\_index}.txt"  
}  
}

---

<div class="post-metadata">

**Author:** ![zpp](https://avatars.discourse-cdn.com/v4/letter/z/54ee81/32.png) [@zpp](https://discuss.elastic.co/u/zpp)\
**Post date:** [February 14, 2017, 10:39am UTC](https://discuss.elastic.co/t/missing-and-disordered-when-using-logstash-elasticsearch-input-and-file-output/75012/2 "2017-02-14T10:39:46Z")

</div>

add on, when I use exact time frame in the range filter (using epoch time), number of records returned are correct.

---

<div class="post-metadata">

**Author:** ![zpp](https://avatars.discourse-cdn.com/v4/letter/z/54ee81/32.png) [@zpp](https://discuss.elastic.co/u/zpp)\
**Post date:** [February 23, 2017, 9:26am UTC](https://discuss.elastic.co/t/missing-and-disordered-when-using-logstash-elasticsearch-input-and-file-output/75012/3 "2017-02-23T09:26:50Z")

</div>

Realised that if the record doesn't contain the field I'm trying to write to the file, the record will not be written to the file. I still have no clue why the order is incorrect, anyone have any ideas?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 23, 2017, 9:27am UTC](https://discuss.elastic.co/t/missing-and-disordered-when-using-logstash-elasticsearch-input-and-file-output/75012/4 "2017-03-23T09:27:00Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.