Snowflake to Elasticsearch Using Logtsash

Elastic Stack Logstash
1

We are migrating data from Snowflake to Elasticsearch using the Logtsash driver.

I took the logstash conf file template from [Pull data from Snowflake with logstash | by Izek Chen | Medium].

Below is the Logstash configuration file template.

input {
  jdbc {
    jdbc_driver_library => "C:\ELKStack\logstash-8.8.1-windows-x86_64\logstash-8.8.1\logstash-core\lib\jars\snowflake-jdbc-3.9.2-javadoc.jar"
    jdbc_driver_class => "com.snowflake.client.jdbc.SnowflakeDriver"
    jdbc_connection_string => "jdbc:snowflake://*******.uae-north.azure.snowflakecomputing.com/?db=SNOWFLAKE_SAMPLE_DATA&warehouse=COMPUTE_WH"
    jdbc_user => "<username>"
    jdbc_password => "<password>"
    schedule => "* * * * *"
    statement => "SELECT * FROM TPCH_SF1.CUSTOMER LIMIT 10;"
  }
}
filter
{}
output
{
  elasticsearch
  {
    hosts => ["http://localhost:9200"]
    user => "<username>"
    password => "<password>"
    index =>"snowflakelog_index"
  }
}

When i am executing the logtsash using the command

logstash.bat -f .\config\test.conf

I am facing the below error "Unable to configure the plugin Cannot determine timezone from nil"

image
image1885×318 32.6 KB

2

Add a timezone. Check here

3

Thanks @Rios for help

Tried the same as mentioned using the link you shared regarding Logstash TimeZone

Here is the modified config file


input {
  jdbc {
    jdbc_driver_library => "C:\ELKStack\logstash-8.8.1-windows-x86_64\logstash-8.8.1\logstash-core\lib\jars\snowflake-jdbc-3.9.2-javadoc.jar"
    jdbc_driver_class => "com.snowflake.client.jdbc.SnowflakeDriver"
    jdbc_connection_string => "jdbc:snowflake://*******.uae-north.azure.snowflakecomputing.com/?db=SNOWFLAKE_SAMPLE_DATA&warehouse=COMPUTE_WH"
    jdbc_user => "username"
    jdbc_password => "password"
    jdbc_default_timezone => "Asia/Dubai[dst_enabled_on_overlap:true]"
    schedule => "* * * * *"
    statement => "SELECT * FROM TPCH_SF1.CUSTOMER LIMIT 10;"
  }
}
filter
{}
output
{
  elasticsearch
  {
    hosts => ["http://localhost:9200"]
    user => "username"
    password => "password"
    index =>"snowflakelog_index"
  }
}

And executed the conf file using the command

logstash.bat -f C:\ELKStack\logstash-8.8.1-windows-x86_64\logstash-8.8.1\config\conf.d\test.conf --path.settings C:\ELKStack\logstash-8.8.1-windows-x86_64\logstash-8.8.1\config

But still the error remains the same

image
image1918×529 52.4 KB

4

It worked as I changed my time zone closer to UTC. (UTC+1 or UTC-1)

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.