Logstash Filter CSV last row not reading

Elastic Stack Logstash
1

Hello,
I have my last line of csv data that I get with http_poller plugin that is not imported. Some people have solved the problem by adding a blank newline at the end of the csv file. But me in my case my file is recovered directly through an http_poller so I tried to add a new line with ruby code but it did not work and I'm looking for that normal because I read in codec line . and I tried in multiline but it does not work to parser the file in csv. So How to fix this problem??
here is my code below:

input {
 #Import all data
 http_poller {
  urls => {
    data => "https://site/index.php?format=CSV"
  }
  request_timeout => 60
  socket_timeout =>  30
  # Supports "cron", "every", "at" and "in" schedules by rufus scheduler
  schedule => { cron => "49 16 * * * Europe/Paris"}
  codec => line { charset => "UTF-16" }
  proxy => "http://xxxx
  }
 }

filter {
	mutate {gsub => [ "message", "[^\d]$", ""] }
	ruby {code => "event.set('message', event.get('message') + ' ' + 10.chr)"}
        csv {
			skip_header => true
			separator => ","
			autodetect_column_names => true
		}
}
output {

	elasticsearch {
			hosts => ["localhost:9200"]
			index => "piwik-%{+YYYY.MM.dd}"
}

Thank you for your help !

2

It looks like the HTTP Poller Input was failing to flush its codec at the end of its response; since the line codec is set to only emit events when it encounters a newline, it was holding onto the last bit and not releasing it. I've opened up a Pull-Request to fix it here: logstash-plugins/logstash-input-http_poller#97.

In the mean-time, you could get by with a combination of the plain codec and the split filter:

input {
 http_poller {
    # ...
    codec => plain { charset => "UTF-16" }
  }
}
# split the "batch" of events into individual events
filter {
  split {
    "field" => "message"
  }
}
# at this point, each event is a single line from the source,
# and the last item is included, whether or not it had a trailing newline.
filter {
  csv {
    # ...
  }
}
3

Thank you for answering me,
Ok I will test the solution that you propose on Monday because today being Saturday I do not work and therefore I have no access to the servers. Then I will see you confirm if the soluton has walked, waiting for you fix the bug.

Thanks for your help!

4

Hello yaauie,

The solution with codec => plain and split => "message" works.

Thanks for your help!

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.