Logstash Error parsing csv java.lang.ArrayIndexOutOfBoundsException

when i use logstash to import data from csv file to elasticsearch, it has error like below and there are no data imported to es.

how to fix it?

 [2018-08-15T23:23:07,540][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"\"XXXXXXXXXXXXX\",\"XXXXXXXXXXXXX\",\"XXXXXXXXXXXXX\",\"XXXXXXXXXXXXX\",(null),\"XXXXXXXXXXXXX\",\"XXXXXXXXXXXXX\",\"XXXXXXXXXXXXX\",\"X\",100.00,20220602,\"XXXX\",\"XXXXXXXXXXXXX\",1,\"XXXXXXXXXXXXX\",\"XXXXXXXXXXXXX\",\"XXXXXXXXXXXXX\",\"XXXXXXXXXXXXX\",\"XXXXXXXXXXXXX\",\"XXXXXXXXXXXXX\",\"XXXXXXXXXXXXX\",(null),(null),\"XXXXXXXXXXXXX\",\"XXXXXXXXXXXXX\",\"XXXXXXXXXXXXX\",\"\",\"XXXXXXXXXXXXX\"\r", :exception=>java.lang.ArrayIndexOutOfBoundsException}

If you show us your Logstash configuration and an example line of input we might be able to help out.

thanks for reply.

my configuration is like below

input {
file {
path => "C:\Application\logstash\script\data\trx.csv"
start_position => "beginning"
sincedb_path => "C:\Application\logstash\trx.read"
}
}

filter {
csv {
separator => ","
autodetect_column_names => true
#remove_field => ["host","path"]
}

date{
match => [ "TRXDATE", "yyyyMMdd" ]
}

date{
match => [ "TRXTIME", "HHmmss" ]
}

mutate
{
remove_field => ["message","path","@version","@timestamp"]
}
}

output {

elasticsearch {
template => "C:\Application\logstash\script\template\trxdetail-template.json"
template_name => "trx-template"
template_overwrite => true

  	hosts => ["XXX.XXX.XXX:9200"]
  	user => "XXXXX"
  	password => "XXXXX"
  	
  	index => "trxdetail"
  	document_type=>"trx"
  	document_id => "%{TRXDATE}-%{TRXFLOW}-%{TRXFLOWNO}"
  	
  }

}

and example line is

ID,CUSNO,CUSACC,CUSNAME,OPPCUSNO,OPPACC,OPPNAME,OPPACCFLAG,CDFLAG,AMT,TRXDATE,TRXTIME,TRXFLOW,TRXFLOWNO,ACCCODE,ACCNAME,BANKNO,BANKNAME,BRANCHNO,BRANCHNAME,TELLER,TERMINAL,CUSWILL,BUSICODE,BUSINAME,BUSICMMCODE,BUSICMM,BUSITECHCMM
"2017XXXXXXXXXXXXXXXXXXXXX","0115XXXXXXXXXXX","62XXXXXXXXXXXXXXXXXX","userXXXX",(null),"X7380XXXXXXXXXXXXXX","XXXXXXXXXXXXXXXX","inner acct","credit",619.00,20100902,"092900","EXXXXXXXXXXXX",1,"2XXXXXXXXX","TEST ACCOUNT","0XXXXXXXXXXXX","XXXXXXXXXXXX","0XXXXXXXXXXXXXX","XXXXXXXXXXXXXXXXXX","ECXXXXXX",(null),(null),"2TTTTTTTT","POSTRX","AXXXX","","PDXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 1"

Works with an older Logstash and autodetect_column_names disabled. Have you tried disabling column autodetection? If that works I'm guessing it's either

  • a mismatch in the column count (more declared columns that actual columns, or vice versa) or
  • a csv filter bug.

thanks a lot, i try it. my es is version 6.3.0 and logstash is also 6.3.0

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.