Csv parsing error, like in an out of buffer error [solved]

Elastic Stack Logstash
1

Hy guys, i'm having a really hard time diagnosing this.

I'm getting tons of tons of parsing errors where it appears logstash is sending to ES completely out of bound fields. For example, it's supposed to send a number between the two "," in a csv but it's just rending random jibberish, some of which i can't find anywhere in the logs.

I think this could be a bug, the csv filter reading where it's not supposed to.

Example config:

input {
  file {
path => "/home/cdrs/*.CDR"
start_position => "beginning"
#    sincedb_path => "/home/cdrs/sincedb/sincedb"
sincedb_path => "/dev/null"

ignore_older => 86400
  }
}
filter
{
  mutate {
   gsub => [ "message", '\\"', '' ]
   gsub => ['message','\"','']
   gsub => ['message','"',""]
   remove_field => [ "null*"]
  }
  csv {
  separator => ";"
 columns => ["start-time","start-time_ts", "another 200 or so columns here..."]
 periodic_flush => true
 skip_empty_columns => true
  }
}
output {
   elasticsearch {
 hosts => "http://localhost:9200"
 index => "cdr"
  }
}

here is an error, from elasticsearch as an example, from a similar config:

[2017-07-02T17:24:49,290][DEBUG][o.e.a.b.TransportShardBulkAction] [OHshd_3] [callstat][3] failed to execute bulk item (index) BulkShardRequest [[callstat][3]] containing [22] requests
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [egress_peak_concurrent_calls]
	at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:298) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:450) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:576) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:396) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:373) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:93) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:66) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:277) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:536) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.index.shard.IndexShard.prepareIndexOnPrimary(IndexShard.java:513) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.prepareIndexOperationOnPrimary(TransportShardBulkAction.java:450) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:458) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:143) [elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:113) [elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:69) [elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:939) [elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:908) [elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:113) [elasticsearch-5.4.1.jar:5.4.1]


[...]

	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_73]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_73]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_73]
Caused by: java.lang.NumberFormatException: For input string: "ingress-reg-p;;;;;;;;;PCMA;PCMA;"
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) ~[?:1.8.0_73]
	at java.lang.Long.parseLong(Long.java:589) ~[?:1.8.0_73]
	at java.lang.Long.parseLong(Long.java:631) ~[?:1.8.0_73]
	at org.elasticsearch.common.xcontent.support.AbstractXContentParser.longValue(AbstractXContentParser.java:172) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.index.mapper.NumberFieldMapper$NumberType$7.parse(NumberFieldMapper.java:740) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.index.mapper.NumberFieldMapper$NumberType$7.parse(NumberFieldMapper.java:719) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.index.mapper.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:1058) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:287) ~[elasticsearch-5.4.1.jar:5.4.1]
	... 36 more

The funny thing? ingress-reg-p;;;;;;;;;PCMA;PCMA; should not be anywhere in that place, it's definetly a parsing bug, but it's parsing the wrong file. from another configuration.

What do you think?

2

Do you by any chance have multiple configuration files that Logstash picks up and concatenation, leading data from other inputs being processed by the csv filter?

3

yeah. that's the problem. it's a bit counterintuitive.

Thank you very much

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.