Logstash avro parse error

Elastic Stack Logstash
1

Hi

I am trying to process an avro file from logstash.

version

- jdk 1.8
- logstash 6.2.2
- https://github.com/logstash-plugins/logstash-codec-avro 3.2.3

logstash test.conf

input {

	file {

		path => ["test.avro"]
		codec => avro {
			schema_uri => 'test.avsc'
		}

		start_position => "beginning"
		sincedb_path => "/dev/null"
	}
}

output {

	stdout {  }
}

execute

logstash --log.level debug -f test.conf

error message

...
[DEBUG][logstash.inputs.file] _open_file: test.avro: opening
[DEBUG][logstash.inputs.file] test.avro: initial create, no sincedb, seeking to beginning of file

...

[DEBUG][logstash.inputs.file] observe_read_file: general error reading test.avro - error: #<ArgumentError: negative length -40 given>

If you read the avro file from java, it works fine.

How can I fix the error?

2

Can you please try giving the full path of avro file ?

3

I get an error even if I try to use the full path of the avro file.

input {
	file {
		path => ["/data/avro/test.avro"]
		codec => avro {
			schema_uri => '/data/avro/test.avsc'
		}

		start_position => "beginning"
		sincedb_path => "/dev/null"
	}
}
4

Kindly paste your error!

5

  • This is the log I ran.

    [INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
    [DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
    [DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
    [INFO ][logstash.pipeline ] Pipeline started succesfully {:pipeline_id=>"main", :thread=>"#<Thread:0x3d17437c run>"}
    [INFO ][logstash.agent ] Pipelines running {:count=>1, :pipelines=>["main"]}
    [DEBUG][logstash.inputs.file ] _globbed_files: /data/avro/test_avro.log: glob is: ["/data/avro/test_avro.log"]
    [DEBUG][logstash.inputs.file ] _discover_file: /data/avro/test_avro.log: new: /data/avro/test_avro.log (exclude is [])
    [DEBUG][logstash.inputs.file ] _open_file: /data/avro/test_avro.log: opening
    [DEBUG][logstash.inputs.file ] /data/avro/test_avro.log: initial create, no sincedb, seeking to beginning of file
    [DEBUG][logstash.inputs.file ] Received line {:path=>"/data/avro/test_avro.log", :text=>"Obj\x01\x04\x16avro.schema\xDCD{"type":"record","name" ..."}
    [DEBUG][logstash.inputs.file ] observe_read_file: general error reading /data/avro/test_avro.log - error: #<ArgumentError: negative length -40 given>
    [DEBUG][logstash.inputs.file ] writing sincedb (delta since last write = 1523940577)
    [DEBUG][logstash.inputs.file ] each: file grew: /data/avro/test_avro.log: old size 0, new size 6918
    [DEBUG][logstash.inputs.file ] each: file grew: /data/avro/test_avro.log: old size 0, new size 6918
    [DEBUG][logstash.inputs.file ] each: file grew: /data/avro/test_avro.log: old size 0, new size 6918

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.