# Google BigQuery output plugin

**URL:** https://discuss.elastic.co/t/google-bigquery-output-plugin/111016
**Category:** Logstash
**Created:** [December 11, 2017, 6:49am UTC](https://discuss.elastic.co/t/google-bigquery-output-plugin/111016 "2017-12-11T06:49:51Z")
**Posts on this page:** 2
**Page:** 1

<div class="post-metadata">

### Author: ![Vasily\_Bronsky](https://avatars.discourse-cdn.com/v4/letter/v/f14d63/32.png) [@Vasily\_Bronsky](https://discuss.elastic.co/u/Vasily_Bronsky)
#### Post date: [December 11, 2017, 6:49am UTC](https://discuss.elastic.co/t/google-bigquery-output-plugin/111016/1 "2017-12-11T06:49:51Z")

</div>

Hello! I have logs with this structure:

{  
"syslog\_program" : bla bla bla,  
"message" : \<14\>Dec 8 18:45:50 bla bla bla ,  
"type" : syslog,  
"syslog\_message" : bla bla bla  
"@timestamp" : 2017-12-08T17:45:50.000Z,  
"syslog\_hostname" : [MOW-NB0036.horsgroup.com](http://MOW-NB0036.horsgroup.com),  
"port" : 51328,  
"syslog\_timestamp" : Dec 8 18:45:50,  
"@version" : 1,  
"host" : localhost,  
"ip\_address" : 0.0.0.0  
"@metdata" : { "ip\_address" : 127.0.0.1 }  
}

So i need to output this logs to bigquery. I used this code of config file:

input {  
tcp {  
host =\> "127.0.0.1"  
port =\> 5000  
type =\> syslog  
}  
}

filter {  
if [type] == "syslog" {  
grok {  
match =\> { "message" =\> "%{SYSLOGTIMESTAMP:syslog\_timestamp} %{SYSLOGHOST:syslog\_hostname} %{DATA:syslog\_program}(?:[%{POSINT:syslog\_pid}])?: %{GREEDYDATA:syslog\_message}" }  
}  
date {  
match =\> ["syslog\_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]  
}  
mutate { remove\_field =\> ["[metdata][ip\_address]", "metdata", "@metdata", "[@metdata][ip\_adress]" ] }  
}  
}

output {  
google\_bigquery {  
project\_id =\> "etl-to-bigquery"  
dataset =\> "LOGSTASH"  
csv\_schema =\> "syslog\_program:STRING,message:STRING,type:STRING,syslog\_message:STRING,timestamp:STRING,syslog\_hostname:STRING,port:INTEGER,syslog\_timestamp:STRING,version:STRING,host:STRING"  
key\_path =\> "/var/local/ETL-TO-BIGQUERY-64f6ab42e01e.p12"  
key\_password =\> "notasecret"  
service\_account =\> "google-cloud-sdk@etl-to-bigquery.iam.gserviceaccount.com"  
flush\_interval\_secs =\> 2  
uploader\_interval\_secs =\> 15  
deleter\_interval\_secs =\> 60  
}  
}

But all the time i have an error with @metdata field (i tried to remove it but doesn't work):

"message"=\>"Error while reading data, error message: JSON parsing error in row starting at position 0: No such field: metdata."}]}

How can i add RECORD field to my csv schema? Any help?

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 8, 2018, 6:49am UTC](https://discuss.elastic.co/t/google-bigquery-output-plugin/111016/2 "2018-01-08T06:49:57Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.