# File input and file output line count mismatch

**URL:** <https://discuss.elastic.co/t/file-input-and-file-output-line-count-mismatch/33112>\
**Category:** Logstash\
**Created:** [October 27, 2015, 10:13pm UTC](https://discuss.elastic.co/t/file-input-and-file-output-line-count-mismatch/33112 "2015-10-27T22:13:45Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![mihir\_ray](https://avatars.discourse-cdn.com/v4/letter/m/e95f7d/32.png) [@mihir\_ray](https://discuss.elastic.co/u/mihir_ray)\
**Post date:** [October 27, 2015, 10:13pm UTC](https://discuss.elastic.co/t/file-input-and-file-output-line-count-mismatch/33112/1 "2015-10-27T22:13:45Z")

</div>

Input data:  
Input data is a static file containing 10 lines of accesslogs.  
If there is a way to share a file, please let me know, i can share the input file.

Config:

input {  
file {  
sincedb\_path =\> "/opt/analytics/logstash/sincedb/da\_neat.sincedb"  
path =\> ["/opt/analytics/logstash/conf/neat\_test"]  
start\_position =\> "beginning"  
exclude =\> "\*.gz"  
type =\> "da\_neat"  
}  
}

output {  
file  
{  
codec =\> "plain"  
path =\> "/opt/analytics/logstash/data/da\_neat/da\_neat02-%{+YYYY-MM-dd-HH}.json"  
}  
}

#1: With the above input and config i get 9 lines in output instead of 10. This is happening with any file. Not sure why it skips the last line always.  
#2: After adding grok,urldecode and kv filters, i got 10 records which matches the input line count, which is good.  
grok {  
match =\> { "message" =\> "%{DATA:remote\_addr} %{DATA:attr1} %{DATA:remote\_user} [%{HTTPDATE:server\_timestamp}] "%{DATA:attr2}" %{NUMBER:status} (?:%{NUMBER:body\_bytes\_sent}|-) "%{DATA:http\_referer}" "%{DATA:http\_user\_agent}" "%{DATA:http\_x\_forwarded\_for}" "%{DATA:request\_body}"" }  
}

```
    kv { source => "request_body"
            field_split => "&"
    }

   urldecode {
            all_fields => "true"
    }

```

#3: Then i added the date filter to match a key in the log. This results in first and seventh record missing in the output(total 8 records in output).

```
    date {
            match => ["server_timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
    }

```

#4: Added mutate to remove unwanted fields:  
mutate {  
remove\_field =\> ["message","@version","type","host","path","request\_body","attr1", "attr2", "status", "body\_bytes\_sent", "http\_x\_forwarded\_for"]  
}  
This further decreased the output line count to 6.

Can someone please help me understand this behavior of logstash.

Thanks,  
Mihir Ray

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 28, 2015, 6:35am UTC](https://discuss.elastic.co/t/file-input-and-file-output-line-count-mismatch/33112/2 "2015-10-28T06:35:36Z")

</div>

Are you deleting the sincedb file between each test run, or how are you getting Logstash to reprocess the file?

---

<div class="post-metadata">

**Author:** ![mihir\_ray](https://avatars.discourse-cdn.com/v4/letter/m/e95f7d/32.png) [@mihir\_ray](https://discuss.elastic.co/u/mihir_ray)\
**Post date:** [October 28, 2015, 6:27pm UTC](https://discuss.elastic.co/t/file-input-and-file-output-line-count-mismatch/33112/3 "2015-10-28T18:27:05Z")

</div>

Yes, i am dropping the sincedb file between each run.

---

<div class="post-metadata">

**Author:** ![jordansissel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jordansissel/32/44957_2.png) [@jordansissel](https://discuss.elastic.co/u/jordansissel)\
**Post date:** [October 28, 2015, 8:48pm UTC](https://discuss.elastic.co/t/file-input-and-file-output-line-count-mismatch/33112/4 "2015-10-28T20:48:59Z")

</div>

The file output only flushes periodically, and only decides _to_ flush after each write, which means if it flushes, and 2 new events are received but the flush interval hasn't expired, those 2 events won't be flushed.

Try observing using the stdout output instead, which doesn't buffer.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:25am UTC](https://discuss.elastic.co/t/file-input-and-file-output-line-count-mismatch/33112/5 "2017-07-06T05:25:01Z")

</div>