Load avro export from BigQuery

I would like to load avro export from BigQuery. Ideally without making any changes to that file. File has the following content:

Objavro.codecnullavro.schemaØ {"type":"record","name":"Root","fields":[{"name":"id","type":["null","long"]},{"name":"info","type":["null",{"type":"record","namespace":"root","name":"Info","fields":[{"name":"first_name","type":["null","string"]},{"name":"last_name","type":["null","string"]},{"name":"address","type":["null",{"type":"record","namespace":"root.info","name":"Address","fields":[{"name":"address_street","type":["null","string"]},{"name":"address_street_num","type":["null","string"]},{"name":"address_zip_code","type":["null","long"]},{"name":"address_city","type":["null","string"]}]}]}]}]},{"name":"degree","type":["null","string"]}]} *í½€ðf£šxT[&دÌ/
ˆmikeasdstr22aÔÍchicagoPhDjonqwestr11öatlantaMSc
annsdfstr55c€Íorlando ronzxcstr33†öatlanta 
jennywerstr44bðÀ
tampa *í½€ðf£šxT[&دÌ/

I pointed fo this file in schema_uri parameter.
When lauched Logstash pipeline I received the following error:

[2019-09-24T13:29:53,589][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"MultiJson::ParseError", :message=>"invalid byte sequence in UTF-8", :backtrace=>["C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/jrjackson-0.4.9-java/lib/jrjackson/jrjackson.rb:91:in `is_time_string?'", "C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/jrjackson-0.4.9-java/lib/jrjackson/jrjackson.rb:36:in `load'", "C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/multi_json-1.13.1/lib/multi_json/adapters/jr_jackson.rb:11:in `load'", "C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/multi_json-1.13.1/lib/multi_json/adapter.rb:21:in `load'", "C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/multi_json-1.13.1/lib/multi_json.rb:122:in `load'", "C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:38:in `parse'", "C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/logstash-codec-avro-3.2.3-java/lib/logstash/codecs/avro.rb:69:in `register'", "C:/logstash-7.3.2/logstash-core/lib/logstash/codecs/base.rb:18:in `initialize'", "org/logstash/plugins/PluginFactoryExt.java:258:in `plugin'", "org/logstash/execution/JavaBasePipelineExt.java:50:in `initialize'", "C:/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb:24:in `initialize'", "C:/logstash-7.3.2/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "C:/logstash-7.3.2/logstash-core/lib/logstash/agent.rb:325:in `block in converge_state'"]}

Any help and/or advice on this topic appreciated.

Tried to get schema definition to another file, co copied following:

{"type":"record","name":"Root","fields":[{"name":"id","type":["null","long"]},{"name":"info","type":["null",{"type":"record","namespace":"root","name":"Info","fields":[{"name":"first_name","type":["null","string"]},{"name":"last_name","type":["null","string"]},{"name":"address","type":["null",{"type":"record","namespace":"root.info","name":"Address","fields":[{"name":"address_street","type":["null","string"]},{"name":"address_street_num","type":["null","string"]},{"name":"address_zip_code","type":["null","long"]},{"name":"address_city","type":["null","string"]}]}]}]}]},{"name":"degree","type":["null","string"]}]}

To another file and pointed it in schema_uri parameter.
After that I am facing the following errors:

[2019-09-24T15:05:27,120][ERROR][logstash.javapipeline ] A plugin had an unrecoverable error. Will restart this plugin.
Pipeline_id:main
Plugin: <LogStash::Inputs::Stdin codec=><LogStash::Codecs::Avro schema_uri=>"C:\logstash-7.3.2\data\test_struct_nulls.avsc", id=>"72952661-0853-4a91-861c-1c0723389b34", enable_metric=>true, tag_on_failure=>false>, id=>"9c35c795547d449c5f689ee72e72a5faae3bd71dadeb1203c31943ac0017f4e0", enable_metric=>true>
Error: undefined method type_sym' for nil:NilClass Exception: NoMethodError Stack: C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema_compatibility.rb:36:in match_schemas'
C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/io.rb:240:in match_schemas' C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/io.rb:257:in read_data'
C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/io.rb:355:in read_union' C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/io.rb:286:in read_data'
C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/io.rb:363:in block in read_record' org/jruby/RubyArray.java:1792:in each'
C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/io.rb:361:in read_record' C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/io.rb:287:in read_data'
C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/io.rb:252:in read' C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/logstash-codec-avro-3.2.3-java/lib/logstash/codecs/avro.rb:77:in decode'
C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/logstash-input-stdin-3.2.6/lib/logstash/inputs/stdin.rb:38:in run' C:/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb:309:in inputworker'
C:/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb:302:in `block in start_input'

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.