BigQuery to Elasticsearch Self-Managed configuration

Mandark1990 · October 20, 2021, 6:45pm

Hi All,

Am trying to test out the steps given here: https://www.elastic.co/blog/ingest-data-directly-from-google-bigquery-into-elastic-using-google-dataflow

As per the documentation it says the implementation flow remains the same for "The integration works for all users, regardless of whether you are using the Elastic Stack on Elastic Cloud, Elastic Cloud in the Google Cloud Marketplace, or a self-managed environment."

However when trying to use this dataflow template in big query dataflows it shows that the field Cloud API Key is mandatory. In that case how can we use this dataflow template for Self-Managed Instances with basic auth enabled. If anyone has been able to get this working please help.

leandrojmp · October 20, 2021, 7:48pm

This is the API Key, it is not related to Elastic Cloud.

You can create an API Key on self-managed without any problems, just go on Kibana, Stack Management, Security, API Keys.

The guide you shared has a link to the part of the documentation on how you can create an API key.

Fill in the required parameters, including your Cloud ID and Base64-encoded API Key for Elasticsearch. Cloud ID can be found from Elastic Cloud UI as shown below. API Key can be created using the Create API key API.

You just need to create an API Key and use it in that field.

Mandark1990 · October 20, 2021, 8:14pm

HI,

I even tried by creating a trial elastic could id and then applying the bigquery template but I get the following error now.

[RuntimeException: org.apache.beam.sdk.util.UserCodeException: java.lang.IllegalArgumentException: Cannot get Elasticsearch version]

Mandark1990 · October 20, 2021, 8:16pm

Also is there any option on the big query template to not use API Key and only use the basic auth creds.

Christian_Dahlqvist · October 21, 2021, 1:07am

What version of Elasticsearch are you using?

Mandark1990 · October 21, 2021, 11:36am

7.10.2

Christian_Dahlqvist · October 21, 2021, 11:55am

Are you using the default distribution?

Mandark1990 · October 21, 2021, 11:55am

Yes.

Mandark1990 · October 25, 2021, 7:53am

Hi,

In this template it has a mandatory parameter for API key. however in the code it looks like its either api key or username/password. Please help on how to process with just using username/password as I cannot use the API key approach.

github.com

GoogleCloudPlatform/DataflowTemplates/blob/master/v2/elasticsearch-common/src/main/java/com/google/cloud/teleport/v2/elasticsearch/transforms/WriteToElasticsearch.java

/*
 * Copyright (C) 2021 Google LLC
 *
 * Licensed under the Apache License, Version 2.0 (the "License"); you may not
 * use this file except in compliance with the License. You may obtain a copy of
 * the License at
 *
 *   http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
 * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
 * License for the specific language governing permissions and limitations under
 * the License.
 */
package com.google.cloud.teleport.v2.elasticsearch.transforms;

import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;

import com.google.auto.value.AutoValue;

This file has been truncated. show original

Christian_Dahlqvist · October 25, 2021, 8:05am

Out of curiosity, why can you not use the API key approach?

Mandark1990 · October 25, 2021, 8:52am

Am using the OSS build which is mapped behind our internal proxy using which we are creating the security layer and currently in our security layer we only support the Basic auth and don't yet have an API key mech implemented. Hence want to see if there is a way to get it working with Basic Auth.

system · November 22, 2021, 8:52am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.