Customized Connector Client running in Kubernetes

[sebastianboelling (Sebastian Bölling)]

Hi,

has somebody experiences or examples of how to run a customized or self developed connector client in Kubernetes?

There is a good documentation starting point for Docker at connectors/docs/DOCKER.md at main · elastic/connectors · GitHub

But for Kubernetes I've to create a complete new Container including the URL to Elasticsearch and the API keys, etc. and run it as Pod.

The samples only show how to start in an Docker environment and reference local config files.

Regards

Sebastian

[Jedr_Blaszyk]

Hi Sebastian,

ConfigMaps in Kubernetes can be used for managing configuration files like the one you have for Docker.

You can create a ConfigMap from your existing config.yml file with:

kubectl create configmap connectors-config --from-file=config.yml=path/to/your/config.yml

Here are reference k8s docs that explain how to configure ConfigMap in detail.

I tested the setup with a simple deployment file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: elastic-connector
spec:
  replicas: 1
  selector:
    matchLabels:
      app: elastic-connector
  template:
    metadata:
      labels:
        app: elastic-connector
    spec:
      containers:
        - name: connector
          image: docker.elastic.co/enterprise-search/elastic-connectors:8.11.1.0
          args: ["/app/bin/elastic-ingest", "-c", "/config/config.yml"]
          volumeMounts:
            - name: config-volume
              mountPath: /config
      volumes:
        - name: config-volume
          configMap:
            name: connectors-config

The pod running the connector service in a local k8s cluster was configured and able to ingest some data to ES running in the cloud.

I hope this helps. We will work to improve docs around the k8s configuration.

Have a great day,
Jedr