Hello,
I am using logstash 7.1.1 to fetch data from elasticsearch cluster to BigQuery database. My logstash instance ran successfully but ran into some problems:
Logstash config file:
input {
elasticsearch {
hosts => myhost
index => "index-v1"
query => '{"query": { "type" : {"value" : "ObjectTerritory2AssignmentRuleItem"}}, "_source" : {}}'
}
}
filter {
mutate {
remove_field => ["@version","@DATE","DATE","path","host","type", "message", "version", "timestamp", "@timestamp"]
}
if [ "lastmodifieddate" ] != "null" { mutate { gsub => [ "lastmodifieddate", "000+00:00", "" ] }}
if [ "systemmodstamp" ] != "null" { mutate { gsub => [ "systemmodstamp", "000+00:00", "" ] }}
}
output {
google_bigquery {
project_id => "project"
dataset => "dataset"
table_prefix => "ObjectTerritory2AssignmentRuleItem_2016_01"
table_separator => ""
flush_interval_secs => 50
date_pattern => ""
error_directory => "/logstash/7.1.1/tmp/bigquery-errors/tmp"
json_key_file => "key.json"
json_schema => {
fields => [
{ name => "CreatedById" type => "STRING" mode => "NULLABLE"},
{ name => "CreatedDate" type => "TIMESTAMP" mode => "NULLABLE"},
{ name => "Field" type => "STRING" mode => "NULLABLE"},
{ name => "Id" type => "STRING" mode => "NULLABLE"},
{ name => "IsDeleted" type => "BOOLEAN" mode => "NULLABLE"},
{ name => "LastModifiedById" type => "STRING" mode => "NULLABLE"},
{ name => "LastModifiedDate" type => "TIMESTAMP" mode => "NULLABLE"},
{ name => "Operation" type => "STRING" mode => "NULLABLE"},
{ name => "RuleId" type => "STRING" mode => "NULLABLE"},
{ name => "SortOrder" type => "FLOAT" mode => "NULLABLE"},
{ name => "SystemModstamp" type => "TIMESTAMP" mode => "NULLABLE"},
{ name => "Value" type => "STRING" mode => "NULLABLE"},
{ name => "sfdc_type" type => "STRING" mode => "NULLABLE"}
]
}
}
}
Logfile Content:
[2019-07-11T17:05:57,080][INFO ][logstash.outputs.googlebigquery] Publishing 128 messages to ObjectTerritory2AssignmentRuleItem_2016_01
[2019-07-11T17:05:58,526][INFO ][logstash.outputs.googlebigquery] Publishing 128 messages to ObjectTerritory2AssignmentRuleItem_2016_01
[2019-07-11T17:05:58,757][ERROR][logstash.outputs.googlebigquery] Error creating table. {:exception=>java.lang.NullPointerException}
...
[2019-07-11T17:06:44,678][INFO ][logstash.outputs.googlebigquery] Publishing 128 messages to ObjectTerritory2AssignmentRuleItem_2016_01
[2019-07-11T17:06:45,169][INFO ][logstash.outputs.googlebigquery] Publishing 128 messages to ObjectTerritory2AssignmentRuleItem_2016_01
[2019-07-11T17:06:46,082][INFO ][logstash.runner ] Logstash shut down.
Issues:
-
Although my destination table exists, I get an error message - '[ERROR][logstash.outputs.googlebigquery] Error creating table. {:exception=>java.lang.NullPointerException}'. Why does logstash try to create a table if it exists? Can this feature disabled? I didn't find any parameter that can do this. Please suggest.
-
The number of documents in elasticsearch is 11437 whereas total number of records that got loaded in BigQuery is 11392. As an alternative, I tried deleting the table and reloading it but I saw the same behavior.
Please help.