# Load avro export from BigQuery

**URL:** <https://discuss.elastic.co/t/load-avro-export-from-bigquery/200918>\
**Category:** Logstash\
**Created:** [September 24, 2019, 5:46pm UTC](https://discuss.elastic.co/t/load-avro-export-from-bigquery/200918 "2019-09-24T17:46:04Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![MMH](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mmh/32/54449_2.png) [@MMH](https://discuss.elastic.co/u/MMH)\
**Post date:** [September 24, 2019, 5:46pm UTC](https://discuss.elastic.co/t/load-avro-export-from-bigquery/200918/1 "2019-09-24T17:46:04Z")

</div>

I would like to load avro export from BigQuery. Ideally without making any changes to that file. File has the following content:

> Objavro.codecnullavro.schemaØ {"type":"record","name":"Root","fields":[{"name":"id","type":["null","long"]},{"name":"info","type":["null",{"type":"record","namespace":"root","name":"Info","fields":[{"name":"first\_name","type":["null","string"]},{"name":"last\_name","type":["null","string"]},{"name":"address","type":["null",{"type":"record","namespace":"root.info","name":"Address","fields":[{"name":"address\_street","type":["null","string"]},{"name":"address\_street\_num","type":["null","string"]},{"name":"address\_zip\_code","type":["null","long"]},{"name":"address\_city","type":["null","string"]}]}]}]}]},{"name":"degree","type":["null","string"]}]} \*í½€ðf£šxT[&Ø¯Ì/  
> ˆmikeasdstr22aÔÍchicagoPhDjonqwestr11öatlantaMSc  
> annsdfstr55c€Íorlando ronzxcstr33†öatlanta   
> jennywerstr44bðÀ  
> tampa \*í½€ðf£šxT[&Ø¯Ì/

I pointed fo this file in `schema_uri` parameter.  
When lauched Logstash pipeline I received the following error:

> [2019-09-24T13:29:53,589][ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"MultiJson::ParseError", :message=\>"invalid byte sequence in UTF-8", :backtrace=\>["C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/jrjackson-0.4.9-java/lib/jrjackson/jrjackson.rb:91:in `is_time_string?'", "C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/jrjackson-0.4.9-java/lib/jrjackson/jrjackson.rb:36:in `load'", "C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/multi\_json-1.13.1/lib/multi\_json/adapters/jr\_jackson.rb:11:in `load'", "C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/multi_json-1.13.1/lib/multi_json/adapter.rb:21:in `load'", "C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/multi\_json-1.13.1/lib/multi\_json.rb:122:in `load'", "C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:38:in `parse'", "C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/logstash-codec-avro-3.2.3-java/lib/logstash/codecs/avro.rb:69:in `register'", "C:/logstash-7.3.2/logstash-core/lib/logstash/codecs/base.rb:18:in `initialize'", "org/logstash/plugins/PluginFactoryExt.java:258:in `plugin'", "org/logstash/execution/JavaBasePipelineExt.java:50:in `initialize'", "C:/logstash-7.3.2/logstash-core/lib/logstash/java\_pipeline.rb:24:in `initialize'", "C:/logstash-7.3.2/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "C:/logstash-7.3.2/logstash-core/lib/logstash/agent.rb:325:in `block in converge\_state'"]}

Any help and/or advice on this topic appreciated.

---

<div class="post-metadata">

**Author:** ![MMH](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mmh/32/54449_2.png) [@MMH](https://discuss.elastic.co/u/MMH)\
**Post date:** [September 24, 2019, 7:10pm UTC](https://discuss.elastic.co/t/load-avro-export-from-bigquery/200918/2 "2019-09-24T19:10:05Z")

</div>

Tried to get schema definition to another file, co copied following:

> {"type":"record","name":"Root","fields":[{"name":"id","type":["null","long"]},{"name":"info","type":["null",{"type":"record","namespace":"root","name":"Info","fields":[{"name":"first\_name","type":["null","string"]},{"name":"last\_name","type":["null","string"]},{"name":"address","type":["null",{"type":"record","namespace":"root.info","name":"Address","fields":[{"name":"address\_street","type":["null","string"]},{"name":"address\_street\_num","type":["null","string"]},{"name":"address\_zip\_code","type":["null","long"]},{"name":"address\_city","type":["null","string"]}]}]}]}]},{"name":"degree","type":["null","string"]}]}

To another file and pointed it in `schema_uri` parameter.  
After that I am facing the following errors:

> [2019-09-24T15:05:27,120][ERROR][logstash.javapipeline] A plugin had an unrecoverable error. Will restart this plugin.  
> Pipeline\_id:main  
> Plugin: \<LogStash::Inputs::Stdin codec=\>\<LogStash::Codecs::Avro schema\_uri=\>"C:\logstash-7.3.2\data\test\_struct\_nulls.avsc", id=\>"72952661-0853-4a91-861c-1c0723389b34", enable\_metric=\>true, tag\_on\_failure=\>false\>, id=\>"9c35c795547d449c5f689ee72e72a5faae3bd71dadeb1203c31943ac0017f4e0", enable\_metric=\>true\>  
> Error: undefined method `type_sym' for nil:NilClass Exception: NoMethodError Stack: C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema_compatibility.rb:36:in `match\_schemas'  
> C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/io.rb:240:in `match_schemas' C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/io.rb:257:in `read\_data'  
> C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/io.rb:355:in `read_union' C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/io.rb:286:in `read\_data'  
> C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/io.rb:363:in `block in read_record' org/jruby/RubyArray.java:1792:in `each'  
> C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/io.rb:361:in `read_record' C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/io.rb:287:in `read\_data'  
> C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/io.rb:252:in `read' C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/logstash-codec-avro-3.2.3-java/lib/logstash/codecs/avro.rb:77:in `decode'  
> C:/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/logstash-input-stdin-3.2.6/lib/logstash/inputs/stdin.rb:38:in `run' C:/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb:309:in `inputworker'  
> C:/logstash-7.3.2/logstash-core/lib/logstash/java\_pipeline.rb:302:in `block in start\_input'

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 22, 2019, 7:10pm UTC](https://discuss.elastic.co/t/load-avro-export-from-bigquery/200918/3 "2019-10-22T19:10:06Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
