# A logstash to bigquery transmission problem
**URL:** https://discuss.elastic.co/t/a-logstash-to-bigquery-transmission-problem/320771
**Category:** Logstash
**Created:** [December 8, 2022, 11:01am UTC](https://discuss.elastic.co/t/a-logstash-to-bigquery-transmission-problem/320771 "2022-12-08T11:01:21Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@dalaopo](https://discuss.elastic.co/u/dalaopo)
#### Post date: [December 8, 2022, 11:01am UTC](https://discuss.elastic.co/t/a-logstash-to-bigquery-transmission-problem/320771/1 "2022-12-08T11:01:21Z")
Helloļ¼
I encountered an error when I used logstash to transfer kafka data to google bigquery
```auto
[ERROR][logstash.outputs.googlebigquery][main][30729410306cb4d98635a59cbf171cea3b769fd734b29b17e925785d4edac5ba] Error uploading data. {:exception=>com.google.cloud.bigquery.BigQueryException: Request payload size exceeds the limit: 10485760 bytes.}
```
this is my config
```auto
output {
36 if[database] == "t8891" {
37 google_bigquery {
38 project_id => "newcar8891"
39 dataset => "logstash"
40 csv_schema => "original:STRING,timestamp:TIMESTAMP,id:STRING,topic:STRING,consumer_group:STRING,partition:STRING,offset:STRING,key:STRING,database:STRING"
41 json_key_file => "/home/shurui/bin/newcar8891-maxwell.json"
42 error_directory => "/opt/module/bqerror"
43 table_prefix => "logstash_t8891"
44 date_pattern => "%Y-%m"
45 batch_size => 6000
46 flush_interval_secs => 10
47 batch_size_bytes => 6000000
48 }
49 }
```
---
### Author:  [@Rios](https://discuss.elastic.co/u/Rios)
#### Post date: [December 8, 2022, 11:13am UTC](https://discuss.elastic.co/t/a-logstash-to-bigquery-transmission-problem/320771/2 "2022-12-08T11:13:54Z")
It's not related to LS.
Check [here](https://cloud.google.com/bigquery/quotas#streaming_inserts)
_|HTTP request size limit|10 MB|Exceeding this value causes `invalid` errors._
_Internally the request is translated from HTTP JSON into an internal data structure. The translated data structure has its own enforced size limit. It's hard to predict the size of the resulting internal data structure, but if you keep your HTTP requests to 10 MB or less, the chance of hitting the internal limit is low._
---
### Author:  [@dalaopo](https://discuss.elastic.co/u/dalaopo)
#### Post date: [December 8, 2022, 11:22am UTC](https://discuss.elastic.co/t/a-logstash-to-bigquery-transmission-problem/320771/3 "2022-12-08T11:22:04Z")
Hello!
```auto
batch_size => 6000
flush_interval_secs => 10
batch_size_bytes => 6000000
```
Can I adjust these three parameters to help this error?
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [December 8, 2022, 12:41pm UTC](https://discuss.elastic.co/t/a-logstash-to-bigquery-transmission-problem/320771/4 "2022-12-08T12:41:59Z")
Your `batch_size_bytes` is already under 10 MB, but probably the `batch_size` has preference and a batch of `6000` events may be over `10 MB`.
Try to use the default `batch_size` of `128` and start increasing it until it stops working.
The [documentation](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-google_bigquery.html#plugins-outputs-google_bigquery-batch_size) says that the recommendation is about `500`.
---
### Author:  [@dalaopo](https://discuss.elastic.co/u/dalaopo)
#### Post date: [December 9, 2022, 2:01am UTC](https://discuss.elastic.co/t/a-logstash-to-bigquery-transmission-problem/320771/5 "2022-12-09T02:01:50Z")
Thanks!
My current practice is to transfer tables that may exceed the limit separately to ensure that other tables will not be affected
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 6, 2023, 2:02am UTC](https://discuss.elastic.co/t/a-logstash-to-bigquery-transmission-problem/320771/6 "2023-01-06T02:02:46Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.