Unable to setup connector in docker environment

Eshwar_K · January 17, 2025, 7:21am

Hi Community,

I am setting up a connector on top of docker but while running docker with provided configuration from Kibana UI, I am getting below errors. I am not sure why docker is giving permission denied error. Can any one suggest please!

command:

docker run -v "/root/elastic-connectors:/config" --tty --rm docker.elastic.co/integrations/elastic-connectors:8.17.0 /app/bin/elastic-ingest -c /config/config.yml

Note: Running above command with root user.

Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
[FMWK][07:05:58][INFO] Running connector service version 8.17.0
[FMWK][07:05:58][INFO] Loading config from /config/config.yml
[FMWK][07:05:58][ERROR] Could not parse /config/config.yml. Check logs for more information.
[Errno 13] Permission denied: '/config/config.yml'
Traceback (most recent call last):
  File "/app/connectors/service_cli.py", line 116, in run
    config = load_config(config_file)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/connectors/config.py", line 21, in load_config
    yaml_config = EnvYAML(config_file, flatten=False).export()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/envyaml/envyaml.py", line 117, in __init__
    yaml_config = self.__read_yaml_file(
                  ^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/envyaml/envyaml.py", line 208, in __read_yaml_file
    with io.open(file_path, encoding="utf8") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/config/config.yml'
Error: Could not parse /config/config.yml. Check logs for more information

Eshwar_K · January 17, 2025, 7:22am

Configuration file.

connectors:
-
  connector_id: "NgMwcJQBJ82eM09mQReR"
  service_type: "microsoft_teams"
  api_key: "TkpyeWNwUUJkQ3Z6VE83REdKR3o6bE9xN3hTcTBTaks4Z3hFWWNvTzkyQQ=="
elasticsearch:
  host: "https://xx.xx.xx.xx:9200"
  api_key: "TkpyeWNwUUJkQ3Z6VE83REdKR3o6bE9xN3hTcTBTaks4Z3hFWWNvTzkyQQ=="

Artem_Shelkovnikov · January 17, 2025, 10:01am

Hi Eshwar_K!

It seems from the error message that configuration file you're mounting into the docker container is not accessible because of permissions in the OS.

Can you check the permissions on your config directory/file to verify that the user that's running the container can access the file?

Ideally don't store config file available only to root, if that's the case.

Adam_Kutrasinski · February 24, 2025, 7:28am

I have similar problem with dockerized connector:
Here is my config.yml file:

elasticsearch:
  host: "http://host.docker.internal:9200"
  api_key: "here is my pass"

connectors:
  - connector_id: "mysql-xxx"
    service_type: "mysql"
    database:
      host: mysql
      port: 3306
      username: myuser
      password: mypass
      database: test

and here is error from docker container logs:

es_connector  | [FMWK][23:27:27][ERROR] Could not parse /app/config.yml. Check logs for more information.
es_connector  | 'list' object has no attribute 'keys'
es_connector  | Traceback (most recent call last):
es_connector  |   File "/app/connectors/service_cli.py", line 116, in run
es_connector  |     config = load_config(config_file)
es_connector  |              ^^^^^^^^^^^^^^^^^^^^^^^^
es_connector  |   File "/app/connectors/config.py", line 24, in load_config
es_connector  |     _nest_configs(nested_yaml_config, key, value)
es_connector  |   File "/app/connectors/config.py", line 187, in _nest_configs
es_connector  |     current_leaf[last_key] = dict(_merge_dicts(current_leaf[last_key], value))
es_connector  |                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
es_connector  |   File "/app/connectors/config.py", line 193, in _merge_dicts
es_connector  |     for k in set(hsh1.keys()).union(hsh2.keys()):
es_connector  |                                     ^^^^^^^^^
es_connector  | AttributeError: 'list' object has no attribute 'keys'
es_connector  | Error: Could not parse /app/config.yml. Check logs for more information

Jedr_Blaszyk · February 24, 2025, 8:30am

Hey @Adam_Kutrasinski, config.yml doesn’t support source-specific config fields in connectors[], so removing database key should resolve your issue and let the framework start.

Once the framework starts up, you can configure your MySQL connector via the Kibana Connectors UI or via API.

Hope this helps!

Adam_Kutrasinski · February 27, 2025, 10:28am

Thanks for your answer.
Unfortunately it didn't change anyting.

I'm trying to launch self-hosted connector using docker on WSL2 Win11 Pro

I prepared docker-compose.yml with services

db: MySQL8 (works ok)
es: Elasticsearch 8.17.0 (works ok)
kb: Kibana (works ok)
connector: Elasticsearch Connector 8.17.0 (here is problem with parsing config.yml)

And still got parsing errors inside connector container:

es_connector  | [FMWK][10:04:58][INFO] Running connector service version 8.17.0
es_connector  | [FMWK][10:04:58][INFO] Loading config from /app/config.yml
es_connector  | [FMWK][10:04:58][WARNING] configured API key will be used over configured basic auth
es_connector  | [FMWK][10:04:58][INFO] Running preflight checks
es_connector  | [FMWK][10:04:58][INFO] Waiting for Elasticsearch at https://es:9200 (so far: 0 secs)
es_connector  | [FMWK][10:04:58][INFO] Elasticsearch 8.17.0 and Connectors 8.17.0 are compatible
es_connector  | [FMWK][10:04:58][INFO] Extraction service is not configured, skipping its preflight check.
es_connector  | [FMWK][10:04:58][INFO] Bye
es_connector  | Traceback (most recent call last):
es_connector  |   File "/app/bin/elastic-ingest", line 33, in <module>
es_connector  |     sys.exit(load_entry_point('elasticsearch-connectors', 'console_scripts', 'elastic-ingest')())
es_connector  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
es_connector  |   File "/app/.venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
es_connector  |     return self.main(*args, **kwargs)
es_connector  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
es_connector  |   File "/app/.venv/lib/python3.11/site-packages/click/core.py", line 1078, in main
es_connector  |     rv = self.invoke(ctx)
es_connector  |          ^^^^^^^^^^^^^^^^
es_connector  |   File "/app/.venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
es_connector  |     return ctx.invoke(self.callback, **ctx.params)
es_connector  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
es_connector  |   File "/app/.venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
es_connector  |     return __callback(*args, **kwargs)
es_connector  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
es_connector  |   File "/app/connectors/service_cli.py", line 233, in main
es_connector  |     return run(action, config_file, log_level, filebeat, service_type, uvloop)
es_connector  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
es_connector  |   File "/app/connectors/service_cli.py", line 172, in run
es_connector  |     return loop.run_until_complete(coro)
es_connector  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
es_connector  |   File "/usr/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
es_connector  |     return future.result()
es_connector  |            ^^^^^^^^^^^^^^^
es_connector  |   File "/app/connectors/service_cli.py", line 48, in _start_service
es_connector  |     success, is_serverless = await preflight.run()
es_connector  |                              ^^^^^^^^^^^^^^^^^^^^^
es_connector  |   File "/app/connectors/preflight_check.py", line 51, in run
es_connector  |     valid_configuration = self._validate_configuration()
es_connector  |                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
es_connector  |   File "/app/connectors/preflight_check.py", line 235, in _validate_configuration
es_connector  |     configured_connector_id = connector.get("connector_id", None)
es_connector  |                               ^^^^^^^^^^^^^
es_connector  | AttributeError: 'str' object has no attribute 'get'

My config.yml is

elasticsearch:
  host: "https://es:9200"
  ssl: true
  verify_certs: true
  ca_certs: /app/certs/ca.crt
  api_key: my_api_key_from_kibana

connectors:
  connector_id: "my_connector_id_form_kibana"
  service_type: "mysql"

When i tried different format (along with connectors/docs/CONFIG.md at main · elastic/connectors · GitHub)

elasticsearch:
  host: "https://es:9200"
  ssl: true
  verify_certs: true
  ca_certs: /app/certs/ca.crt
  api_key: my_api_key_from_kibana

connectors:
  - 
    connector_id: "my_connector_id_form_kibana"
    service_type: "mysql"

i got another parsing error

[FMWK][10:10:34][INFO] Running connector service version 8.17.0
es_connector  | [FMWK][10:10:34][INFO] Loading config from /app/config.yml
es_connector  | [FMWK][10:10:34][ERROR] Could not parse /app/config.yml. Check logs for more information.
es_connector  | 'list' object has no attribute 'keys'
es_connector  | Traceback (most recent call last):
es_connector  |   File "/app/connectors/service_cli.py", line 116, in run
es_connector  |     config = load_config(config_file)
es_connector  |              ^^^^^^^^^^^^^^^^^^^^^^^^
es_connector  |   File "/app/connectors/config.py", line 24, in load_config
es_connector  |     _nest_configs(nested_yaml_config, key, value)
es_connector  |   File "/app/connectors/config.py", line 186, in _nest_configs
es_connector  |     current_leaf[last_key] = dict(_merge_dicts(current_leaf[last_key], value))
es_connector  |                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
es_connector  |   File "/app/connectors/config.py", line 192, in _merge_dicts
es_connector  |     for k in set(hsh1.keys()).union(hsh2.keys()):
es_connector  |                                     ^^^^^^^^^
es_connector  | AttributeError: 'list' object has no attribute 'keys'
es_connector  | Error: Could not parse /app/config.yml. Check logs for more information

I checked also another one format (along with config.yml.example)

elasticsearch:
  host: "https://es:9200"
  ssl: true
  verify_certs: true
  ca_certs: /app/certs/ca.crt
  api_key: my_api_key_from_kibana

connectors:
  - connector_id: "my_connector_id_form_kibana"
    service_type: "mysql"

but still got parsing errors

es_connector  | [FMWK][10:17:39][INFO] Running connector service version 8.17.0
es_connector  | [FMWK][10:17:39][INFO] Loading config from /app/config.yml
es_connector  | [FMWK][10:17:39][ERROR] Could not parse /app/config.yml. Check logs for more information.
es_connector  | 'list' object has no attribute 'keys'
es_connector  | Traceback (most recent call last):
es_connector  |   File "/app/connectors/service_cli.py", line 116, in run
es_connector  |     config = load_config(config_file)
es_connector  |              ^^^^^^^^^^^^^^^^^^^^^^^^
es_connector  |   File "/app/connectors/config.py", line 24, in load_config
es_connector  |     _nest_configs(nested_yaml_config, key, value)
es_connector  |   File "/app/connectors/config.py", line 186, in _nest_configs
es_connector  |     current_leaf[last_key] = dict(_merge_dicts(current_leaf[last_key], value))
es_connector  |                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
es_connector  |   File "/app/connectors/config.py", line 192, in _merge_dicts
es_connector  |     for k in set(hsh1.keys()).union(hsh2.keys()):
es_connector  |                                     ^^^^^^^^^
es_connector  | AttributeError: 'list' object has no attribute 'keys'
es_connector  | Error: Could not parse /app/config.yml. Check logs for more information

I've tried also with the newest connector version (8.17.2) but still the same.

UPDATE:
Passing config in old deprecated format works propertly:

elasticsearch:
  host: "https://es:9200"
  ssl: true
  verify_certs: true
  ca_certs: /app/certs/ca.crt
  api_key: my_api_key_from_kibana

connector_id: "my_connector_id_form_kibana"
service_type: "mysql"

but it's deprcated and support only one connector.

Jedr_Blaszyk · February 28, 2025, 10:37am

Hey @Adam_Kutrasinski, unfortunately I couldn’t reproduce this issue in the 8.17.2 version of the connector framework using your config... the configs are being parsed correctly. I tried that one:

Adam_Kutrasinski:

elasticsearch:
  host: "https://es:9200"
  ssl: true
  verify_certs: true
  ca_certs: /app/certs/ca.crt
  api_key: my_api_key_from_kibana

connectors:
  - connector_id: "my_connector_id_form_kibana"
    service_type: "mysql"

I recommend creating a connector in Kibana. You’ll be taken to a page where you can copy and paste the config.

Then, on Mac, you can simply run:

pbpaste > config.yml

Adam_Kutrasinski · March 7, 2025, 9:10am

I'm using 8.17.0 (not 8.17.2) and docker on WSL2 Win11 Pro (not Mac). I think that your image for docker and mine are not the same.

Topic		Replies	Views
Your connector has not connected to Search Elasticsearch docker , connectors	6	125	August 7, 2024
Connection refused with Python connection in Docker Elasticsearch elastic-stack-security , docker	2	1981	November 16, 2022
Can't start docker-compose elasticsearch Kibana docker	1	22	June 4, 2025
Kibana on Docker connection refused Kibana	3	2115	May 12, 2017
Azure Blob Storage connector Kibana docker , connectors	2	149	September 2, 2024

Unable to setup connector in docker environment

Related topics