# Logstash Filter CSV last row not reading

**URL:** <https://discuss.elastic.co/t/logstash-filter-csv-last-row-not-reading/128005>\
**Category:** Logstash\
**Created:** [April 13, 2018, 3:29pm UTC](https://discuss.elastic.co/t/logstash-filter-csv-last-row-not-reading/128005 "2018-04-13T15:29:10Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![sabdoul](https://avatars.discourse-cdn.com/v4/letter/s/74df32/32.png) [@sabdoul](https://discuss.elastic.co/u/sabdoul)\
**Post date:** [April 13, 2018, 3:29pm UTC](https://discuss.elastic.co/t/logstash-filter-csv-last-row-not-reading/128005/1 "2018-04-13T15:29:10Z")

</div>

Hello,  
I have my last line of csv data that I get with http\_poller plugin that is not imported. Some people have solved the problem by adding a blank newline at the end of the csv file. But me in my case my file is recovered directly through an http\_poller so I tried to add a new line with ruby code but it did not work and I'm looking for that normal because I read in codec line . and I tried in multiline but it does not work to parser the file in csv. So How to fix this problem??  
here is my code below:

```
input {
 #Import all data
 http_poller {
  urls => {
    data => "https://site/index.php?format=CSV"
  }
  request_timeout => 60
  socket_timeout => 30
  # Supports "cron", "every", "at" and "in" schedules by rufus scheduler
  schedule => { cron => "49 16 * * * Europe/Paris"}
  codec => line { charset => "UTF-16" }
  proxy => "http://xxxx
  }
 }

filter {
	mutate {gsub => ["message", "[^\d]$", ""] }
	ruby {code => "event.set('message', event.get('message') + ' ' + 10.chr)"}
        csv {
			skip_header => true
			separator => ","
			autodetect_column_names => true
		}
}
output {

	elasticsearch {
			hosts => ["localhost:9200"]
			index => "piwik-%{+YYYY.MM.dd}"
}  

```

Thank you for your help !

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [April 13, 2018, 7:32pm UTC](https://discuss.elastic.co/t/logstash-filter-csv-last-row-not-reading/128005/2 "2018-04-13T19:32:08Z")

</div>

> [@sabdoul](#):
>
> I have my last line of csv data that I get with http\_poller plugin that is not imported. Some people have solved the problem by adding a blank newline at the end of the csv file.

It looks like the HTTP Poller Input was failing to flush its codec at the end of its response; since the `line` codec is set to only emit events when it encounters a newline, it was holding onto the last bit and not releasing it. I've opened up a Pull-Request to fix it here: [logstash-plugins/logstash-input-http\_poller#97](https://github.com/logstash-plugins/logstash-input-http_poller/pull/97).

In the mean-time, you could get by with a combination of the `plain` codec and the `split` filter:

```auto
input {
 http_poller {
    # ...
    codec => plain { charset => "UTF-16" }
  }
}
# split the "batch" of events into individual events
filter {
  split {
    "field" => "message"
  }
}
# at this point, each event is a single line from the source,
# and the last item is included, whether or not it had a trailing newline.
filter {
  csv {
    # ...
  }
}

```

---

<div class="post-metadata">

**Author:** ![sabdoul](https://avatars.discourse-cdn.com/v4/letter/s/74df32/32.png) [@sabdoul](https://discuss.elastic.co/u/sabdoul)\
**Post date:** [April 14, 2018, 10:35am UTC](https://discuss.elastic.co/t/logstash-filter-csv-last-row-not-reading/128005/3 "2018-04-14T10:35:10Z")

</div>

Thank you for answering me,  
Ok I will test the solution that you propose on Monday because today being Saturday I do not work and therefore I have no access to the servers. Then I will see you confirm if the soluton has walked, waiting for you fix the bug.

Thanks for your help!

---

<div class="post-metadata">

**Author:** ![sabdoul](https://avatars.discourse-cdn.com/v4/letter/s/74df32/32.png) [@sabdoul](https://discuss.elastic.co/u/sabdoul)\
**Post date:** [April 17, 2018, 7:59am UTC](https://discuss.elastic.co/t/logstash-filter-csv-last-row-not-reading/128005/4 "2018-04-17T07:59:48Z")

</div>

Hello yaauie,

The solution with `codec => plain` and `split => "message"` works.

Thanks for your help!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 15, 2018, 8:00am UTC](https://discuss.elastic.co/t/logstash-filter-csv-last-row-not-reading/128005/5 "2018-05-15T08:00:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.