Logstash 6.1 failing to parse CSV

Elastic Stack Logstash
1

While parsing csv file i am getting below error. Any guess or help to remove this?

[2018-01-15T12:33:39,047][WARN ][logstash.filters.csv ] Error parsing csv {:field=>"message", :source=>"\"ip_addr\",\"initiator_line\",\"bytesOut\",\"load_ms\",\"request_id\",\"image_save\",\"minify_total\",\"bytesIn\",\"cacheControl\",\"score_cdn\",\"id\",\"dns_end\",\"priority\",\"image_total\",\"minify_save\",\"score_cookies\",\"ttfb_ms\",\"score_cache\",\"protocol\",\"score_compress\",\"cache_time\",\"load_start\",\"server_port\",\"score_gzip\",\"type\",\"full_url\",\"ssl_end\",\"objectSize\",\"gzip_total\",\"socket_group\",\"contentType\",\"initiator_column\",\"expires\",\"score_combine\",\"score_etags\",\"frame_id\",\"host\",\"socket\",\"gzip_save\",\"score_keep-alive\",\"is_secure\",\"dns_start\",\"server_count\",\"score_minify\",\"initiator\",\"connect_ms\",\"connect_start\",\"url\",\"ssl_start\",\"client_port\",\"method\",\"dns_ms\",\"connect_end\",\"server_rtt\",\"responseCode\",\"ssl_ms\",\"contentEncoding\",\"cdn_provider\",\"load_end\",\"ttfb_start\",\"ttfb_end\",\"download_start\",\"download_end\",\"download_ms\",\"all_start\",\"all_end\",\"all_ms\",\"index\",\"number\",\"run\",\"cached\",\"final_base_page\",\"objectSizeUncompressed\",\"is_base_page\",\"score_progressive_jpeg\",\"jpeg_scan_count\",\"http2_stream_id\",\"http2_stream_weight\",\"http2_stream_exclusive\",\"http2_stream_dependency\",\r", :exception=>java.lang.ArrayIndexOutOfBoundsException: -1}

1 Like
2

I am using stdin input filter and csv plugin.

Parsing multiple csv files sequentially. Observed that for one of the csv file header was not catched by csv filter hence reported "_csvparsefailure".

How to handle this?

3

Attaching error log. Auto detect column getting value row instead column header. How to control stdin {} to send events sequentially in FIFO.

[2018-01-16T12:51:19,534][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"\"151.101.128.217\",\"343\",\"330\",\"34\",\"64026.21\",\"\",\"\",\"12544\",\"no-store, no-cache, must-revalidate, post-check=0, pre-check=0\",\"100\",\"180115_8N_0045870c7e89fe0d28c06403af9bde45\",\"294\",\"LOW\",\"\",\"\",\"-1\",\"32\",\"-1\",\"http/1.1\",\"-1\",\"\",\"908\",\"443\",\"100\",\"3\",\"https://player.vimeo.com/api/player.js\",\"370\",\"5275\",\"5275\",\"ssl/player.vimeo.com:443\",\"application/javascript\",\"\",\"Mon, 15 Jan 2018 13:39:16 GMT\",\"-1\",\"-1\",\"(8C687B1F7E876B42804F2A1D5DA9A45E)\",\"player.vimeo.com\",\"238\",\"0\",\"100\",\"1\",\"261\",\"\",\"-1\",\"https://www.dotdash.com/\",\"32\",\"294\",\"/api/player.js\",\"327\",\"49872\",\"GET\",\"33\",\"326\",\"\",\"200\",\"43\",\"gzip\",\"Fastly\",\"942\",\"908\",\"940\",\"940\",\"942\",\"2\",\"261\",\"942\",\"142\",\"6\",\"7\",\"1\",\"0\",\"\",\"16316\",\"\",\"\",\"\",\"\",\"\",\"\",\"\",\r", :exception=>java.lang.ArrayIndexOutOfBoundsException: -1}
[2018-01-16T12:51:19,537][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"\"ip_addr\",\"initiator_line\",\"bytesOut\",\"load_ms\",\"request_id\",\"image_save\",\"minify_total\",\"bytesIn\",\"cacheControl\",\"score_cdn\",\"id\",\"dns_end\",\"priority\",\"image_total\",\"minify_save\",\"score_cookies\",\"ttfb_ms\",\"score_cache\",\"protocol\",\"score_compress\",\"cache_time\",\"load_start\",\"server_port\",\"score_gzip\",\"type\",\"full_url\",\"ssl_end\",\"objectSize\",\"gzip_total\",\"socket_group\",\"contentType\",\"initiator_column\",\"expires\",\"score_combine\",\"score_etags\",\"frame_id\",\"host\",\"socket\",\"gzip_save\",\"score_keep-alive\",\"is_secure\",\"dns_start\",\"server_count\",\"score_minify\",\"initiator\",\"connect_ms\",\"connect_start\",\"url\",\"ssl_start\",\"client_port\",\"method\",\"dns_ms\",\"connect_end\",\"server_rtt\",\"responseCode\",\"ssl_ms\",\"contentEncoding\",\"cdn_provider\",\"load_end\",\"ttfb_start\",\"ttfb_end\",\"download_start\",\"download_end\",\"download_ms\",\"all_start\",\"all_end\",\"all_ms\",\"index\",\"number\",\"run\",\"cached\",\"final_base_page\",\"objectSizeUncompressed\",\"is_base_page\",\"score_progressive_jpeg\",\"jpeg_scan_count\",\"http2_stream_id\",\"http2_stream_weight\",\"http2_stream_exclusive\",\"http2_stream_dependency\",\r", :exception=>java.lang.ArrayIndexOutOfBoundsException: -1}
4

Please help if any workaround. I am badly stuck due to this.

5

I don't know what's wrong with CSV filter, but when I had problem with kv filter. It has a few bugs.
I wrote my own splitter using ruby plugin. This could be a way out if nothing else works.

6

Hey all there,

Same issue here but when I delete my sincedb file to index again the CSV, works some times. This isn't a workaround but maybe give us a clue about what is going on with this plug-in.

7

Another issue I have with this csv plugin is some times (random), doesn't read the first line like headers. It is the csv plugin very unstable or is just me?

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.