# Http input and output google bigquery
**URL:** https://discuss.elastic.co/t/http-input-and-output-google-bigquery/304323
**Category:** Logstash
**Created:** [May 10, 2022, 8:40am UTC](https://discuss.elastic.co/t/http-input-and-output-google-bigquery/304323 "2022-05-10T08:40:17Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@dharminfadia](https://discuss.elastic.co/u/dharminfadia)
#### Post date: [May 10, 2022, 8:40am UTC](https://discuss.elastic.co/t/http-input-and-output-google-bigquery/304323/1 "2022-05-10T08:40:17Z")
input {
http {
host =\> "0.0.0.0" # default: 0.0.0.0
port =\> 0000 # default: 8080
}
}
filter {
urldecode {
all\_fields =\> true
}
}
```
filter {
mutate { gsub => ["message", "\]", "}", "message", "\[", "{"] }
kv { field_split => "&" allow_empty_values => true }
```
}
filter {
mutate {
```
split => { "contact{fields}{39}" => '+' }
split => { "unsubscribe{reason}" => '+' }
split => { "date_time" => '+' }
split => { "share{content}" => '+' }
split => { "message" => '+' }
split => { "bounce{description}" => '+' }
```
}
}
filter {
mutate {
join =\> { "contact{fields}{39}" =\> " " }
join =\> { "unsubscribe{reason}" =\> " " }
join =\> { "date\_time" =\> " " }
join =\> { "share{content}" =\> " " }
join =\> { "message" =\> " " }
join =\> { "bounce{description}" =\> " " }
}
}
output {
Elasticsearch {
hosts =\> ['es:9200']
ndex =\> "test"
}
stdout {
codec =\> "json"
}
}
output {
google\_bigquery {
project\_id =\> "project-273709"
dataset =\> "test"
csv\_schema =\> "campaign{recipients}:INTEGER,contact{ip}:INTEGER,contact{id}:FLOAT"
json\_key\_file =\> "/key/test.json"
error\_directory =\> "/tmp/bigquery-errors"
date\_pattern =\> "%Y-%m-%dT%H:00"
flush\_interval\_secs =\> 30
}
}
I am trying to get http input data in to google boig gquery but first I am facing Schema Error and
if I am using above config I am getting following error anyone pls help me
**"campaign{recipients}". Fields must contain only letters, numbers, and underscores, start with a letter or underscore, and be at most 300 characters long.}**
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 10, 2022, 4:08pm UTC](https://discuss.elastic.co/t/http-input-and-output-google-bigquery/304323/2 "2022-05-10T16:08:56Z")
> [@dharminfadia](#):
>
> "campaign{recipients}". Fields must contain only letters, numbers, and underscores, start with a letter or underscore, and be at most 300 characters long.}
The message is saying that {} are not allowed in the schema name.
---
### Author:  [@dharminfadia](https://discuss.elastic.co/u/dharminfadia)
#### Post date: [May 11, 2022, 3:58am UTC](https://discuss.elastic.co/t/http-input-and-output-google-bigquery/304323/3 "2022-05-11T03:58:22Z")
Hello @Badger
thank you for promp reply my Log is like
{"initiated\_by":"admin","bounce{code}":"5.1.1","contact\_fields\_39":"custom field value","campaign{id}":"98","contact{last\_name}":"Last","date\_time":"2013-01-01 12:00:00","bounce{type}":"hard","initiated\_from":"admin","version":"1","contact{email}":"test@test.com","contact{id}":"42","bounce{description}":"Address+not+found.","host":"10.10.4.151","type":"bounce","list":"1","contact\_first\_name":"First","url":"","timestamp":"2022-05-10T13:26:57.392924Z","headers":{"content\_length":"390","content\_type":"application/x-www-form-urlencoded","accept\_encoding":"gzip","http\_user\_agent":null,"http\_accept":"_/_","request\_method":"POST","http\_version":"HTTP/1.1","http\_host":"[9947-62-255-138-230.eu.ngrok.io](http://9947-62-255-138-230.eu.ngrok.io)","x\_forwarded\_proto":"https","request\_path":"/","x\_forwarded\_for":"34.233.145.248"},"contacti\_ip":"127.0.0.1","message":["url=&type=bounce&date\_time=2013-01-01","12:00:00&initiated\_by=admin&initiated\_from=admin&list=1&campaign{id}=98&contact{id}=42&contact{email}=test@test.com&contact{first\_name}=First&contact{last\_name}=Last&contact{ip}=127.0.0.1&contact{fields}{39}=custom","field","value&bounce{type}=hard&bounce{code}=5.1.1&bounce{description}=Address","not","found."]}
\*\*Comming in to http input now I want to output this Data into Bigquery As well \*\*
**so how can I achiev that I read thar mesasge {} is not allowed in to schema but I want to store data in to bigquery selective fileds Other files I want to drop How can I do that**
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 11, 2022, 4:47pm UTC](https://discuss.elastic.co/t/http-input-and-output-google-bigquery/304323/4 "2022-05-11T16:47:39Z")
I would do things rather differently
```
mutate { gsub => ["message", "(\w+){(\w+)}", "\1_\2"] }
json { source => "message" remove_field => ["message"] }
urldecode { all_fields => true }
mutate {
gsub => [
"contact_fields_39", "\+", " ",
"unsubscribe_reason", "\+", " ",
"date_time", "\+", " ",
"share_content", "\+", " ",
"message", "\+", " ",
"bounce_description", "\+", " "
]
}
# If you want headers at the top-level
ruby {
code => '
headers = event.remove("headers")
if headers headers.each { |k, v| event.set(k, v) }; end
'
}
```
---
### Author:  [@dharminfadia](https://discuss.elastic.co/u/dharminfadia)
#### Post date: [May 12, 2022, 4:27am UTC](https://discuss.elastic.co/t/http-input-and-output-google-bigquery/304323/5 "2022-05-12T04:27:22Z")
input {
http {
host =\> "0.0.0.0" # default: 0.0.0.0
port =\> 0000 # default: 8080
}
}
filter {
mutate { gsub =\> ["message", "(\w+){(\w+)}", "\1\_\2"] }
json { source =\> "message" remove\_field =\> ["message"] }
urldecode { all\_fields =\> true }
mutate {
gsub =\> [
"contact\_fields\_39", "+", " ",
"unsubscribe\_reason", "+", " ",
"date\_time", "+", " ",
"share\_content", "+", " ",
"message", "+", " ",
"bounce\_description", "+", " "
]
}
}
# If you want headers at the top-level
ruby {
code =\> '
headers = event.remove("headers")
if headers headers.each { |k, v| event.set(k, v) }; end
'
}
output {
Elasticsearch {
hosts =\> ['es:9200']
index =\> "ac\_test"
}
stdout {
codec =\> "json"
}
}
output {
google\_bigquery {
project\_id =\> "id-709"
dataset =\> "\_test"
table\_separator =\> ""
batch\_size =\> 1000
table\_prefix =\> "\_test"
csv\_schema =\> "campaign\_name:STRING,timestamp:TIMESTAMP,date\_time:DATETIME,campaign\_id:STRING,account\_id:STRING,type:STRING,campaign\_recipients:STRING"
json\_key\_file =\> "/key/key.json"
error\_directory =\> "/tmp/bigquery-errors"
date\_pattern =\> ""
ignore\_unknown\_values =\> true
flush\_interval\_secs =\> 30
}
}
**Hello @Badger Thank you for reply I tried above code in logstash but I am getting error in Logstash start I need All fileds in Elasticsearch and selective field in to bigquery**
**see the following error**
[2022-05-12T09:46:25,110][INFO][logstash.runner] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2022-05-12T09:46:25,136][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"8.1.2", "jruby.version"=\>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.14.1+1 on 11.0.14.1+1 +indy +jit [linux-x86\_64]"}
[2022-05-12T09:46:25,137][INFO][logstash.runner] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2022-05-12T09:46:27,661][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9601, :ssl\_enabled=\>false}
[2022-05-12T09:46:28,526][ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of [\t\r\n], "#", "input", "filter", "output" at line 25, column 5 (byte 607) after ", :backtrace=\>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:189:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline\_action/create.rb:50:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:376:in `block in converge\_state'"]}
[2022-05-12T09:46:28,626][INFO][logstash.runner] Logstash shut down.
[2022-05-12T09:46:28,652][FATAL][org.logstash.Logstash] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby.jar:?]
at usr.share.logstash.lib.bootstrap.environment.(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]
[2022-05-12T09:46:56,247][INFO][logstash.runner] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2022-05-12T09:46:56,310][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"8.1.2", "jruby.version"=\>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.14.1+1 on 11.0.14.1+1 +indy +jit [linux-x86\_64]"}
[2022-05-12T09:46:56,312][INFO][logstash.runner] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2022-05-12T09:46:58,626][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9601, :ssl\_enabled=\>false}
[2022-05-12T09:46:59,490][ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of [\t\r\n], "#", "input", "filter", "output" at line 25, column 5 (byte 607) after ", :backtrace=\>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:189:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline\_action/create.rb:50:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:376:in `block in converge\_state'"]}
[2022-05-12T09:46:59,623][INFO][logstash.runner] Logstash shut down.
[2022-05-12T09:46:59,643][FATAL][org.logstash.Logstash] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby.jar:?]
at usr.share.logstash.lib.bootstrap.environment.(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 9, 2022, 4:27am UTC](https://discuss.elastic.co/t/http-input-and-output-google-bigquery/304323/6 "2022-06-09T04:27:40Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.