Using the Python Mysql Connector

Hello, I am running into issues when using GitHub - elastic/connectors-python: Official Connector Clients for Elastic Elasticsearch, Enterprise Search, App Search and Workplace Search CLI. I am able to run the connector client using elastic-ingest --debug -c config.yml, where my config looks like this:

elasticsearch:
  host: https://127.0.0.1:9200
  # API key generated in elastic search. Otherwise, use username and password.
  username: elastic
  password: el_password
  ssl: true
  bulk:
    queue_max_size: 1024
    queue_max_mem_size: 25
    display_every: 100
    chunk_size: 1000
    max_concurrency: 5
    chunk_max_mem_size: 5
  request_timeout: 120
  max_wait_duration: 120
  initial_backoff_duration: 1
  backoff_multiplier: 2
  log_level: info
  language_code: en
  analysis_icu: false
  

service:
  idling: 30
  heartbeat: 300
  max_errors: 20
  max_errors_span: 600
  trace_mem: false

native_service_types:
  - mysql
  # - network_drive
  # - s3
  # - gcs

# some id


sources:
  # mongodb: connectors.sources.mongo:MongoDataSource
  # s3: connectors.sources.s3:S3DataSource
  # dir: connectors.sources.directory:DirectoryDataSource
  mysql: connectors.sources.mysql:MySqlDataSource
  # network_drive: connectors.sources.network_drive:NASDataSource
  # gcs: connectors.sources.gcs:GoogleCloudStorageDataSource

and the output looks like this:

Desktop/connectors-python > elastic-ingest --debug -c config.yml
[FMWK][11:56:10][DEBUG] BYOIndex connecting to https://127.0.0.1:9200
[FMWK][11:56:10][DEBUG] Host is NodeConfig(scheme='https', host='127.0.0.1', port=9200, path_prefix='', headers={}, connections_per_node=10, request_timeout=10.0, http_compress=False, verify_certs=True, ca_certs=None, client_cert=None, client_key=None, ssl_assert_hostname=None, ssl_assert_fingerprint=None, ssl_version=None, ssl_context=None, ssl_show_warn=True, _extras={})
[FMWK][11:56:10][DEBUG] Connecting using Basic Auth (user: elastic, password: el_...)
[FMWK][11:56:10][DEBUG] Native support for mysql
[FMWK][11:56:10][DEBUG] Wait for Elasticsearch (max: 120)
[FMWK][11:56:10][INFO] Waiting for NodeConfig(scheme='https', host='127.0.0.1', port=9200, path_prefix='', headers={}, connections_per_node=10, request_timeout=10.0, http_compress=False, verify_certs=True, ca_certs=None, client_cert=None, client_key=None, ssl_assert_hostname=None, ssl_assert_fingerprint=None, ssl_version=None, ssl_context=None, ssl_show_warn=True, _extras={}) (so far: 0 secs)
[FMWK][11:56:10][INFO] Preflight checks...
[FMWK][11:56:10][DEBUG] Checking for index .elastic-connectors presence
[FMWK][11:56:10][DEBUG] Checking for index .elastic-connectors-sync-jobs presence
[FMWK][11:56:10][INFO] Service started, listening to events from https://127.0.0.1:9200
[FMWK][11:56:10][DEBUG] ElasticServer connecting to https://127.0.0.1:9200
[FMWK][11:56:10][DEBUG] Host is NodeConfig(scheme='https', host='127.0.0.1', port=9200, path_prefix='', headers={}, connections_per_node=10, request_timeout=10.0, http_compress=False, verify_certs=True, ca_certs=None, client_cert=None, client_key=None, ssl_assert_hostname=None, ssl_assert_fingerprint=None, ssl_version=None, ssl_context=None, ssl_show_warn=True, _extras={})
[FMWK][11:56:10][DEBUG] Connecting using Basic Auth (user: elastic, password: el_...)
[FMWK][11:56:10][DEBUG] Polling every 30 seconds

But I am still not able to use the connector, nor does it look like the CLI is trying to connect to my database. I also changed the mysql.py class config to use the correct values:

 @classmethod
    def get_default_configuration(cls):
        """Get the default configuration for MySQL server

        Returns:
            dictionary: Default configuration
        """
        return {
            "host": {
                "value": "127.0.0.1",
                "label": "Host",
                "type": "str",
            },
            "port": {
                "value": 3306,
                "label": "Port",
                "type": "int",
            },
            "user": {
                "value": "strapi",
                "label": "Username",
                "type": "str",
            },
            "password": {
                "value": "strapi",
                "label": "Password",
                "type": "str",
            },
            "database": {
                "value": ["strapi"],
                "label": "Databases",
                "type": "list",
            },
            "connector_name": {
                "value": "MySQL Connector",
                "label": "Friendly name for the connector",
                "type": "str",
            },
            "fetch_size": {
                "value": DEFAULT_FETCH_SIZE,
                "label": "How many rows to fetch on each call",
                "type": "int",
            },
            "retry_count": {
                "value": DEFAULT_RETRY_COUNT,
                "label": "How many retry count for fetching rows on each call",
                "type": "int",
            },
            "ssl_disabled": {
                "value": DEFAULT_SSL_DISABLED,
                "label": "SSL verification will be enabled or not",
                "type": "bool",
            },
            "ssl_ca": {
                "value": DEFAULT_SSL_CA,
                "label": "SSL certificate",
                "type": "str",
            },
        }

Even using an API key, and the connector ID I still do not see the connector becoming available as a source.

Screen Shot 2023-01-05 at 11.58.43 AM

Any help will be greatly appreciated!

Thanks for posting this here and in our community slack! since we've started a chat in the slack, let's discuss there and we can update this thread with the results of our conversation.

We chatted with Andrew to make it work against the latest 8.5, below is the relevant part in the config:

elasticsearch:
  host: https://localhost:9200
  api_key: "<my key>"
  ssl: true
  ca_certs: 'http_ca.crt'

service_type: mysql
connector_id: "<my connector id>"

sources:
  mysql: connectors.sources.mysql:MySqlDataSource

Notice that the ca_certs option is only needed when you run against a local Elasticsearch instance that has its own certificate and runs on https. This might be good to add in our documentation

There's also the service_type: mysql line that needs to be added, I don't think it's explicit in the Kibana UI.

Last, the Kibana form validation has been failing when we validated it, and one workaround is to type explicitly every value and not keep the default ones provided when you add a connector.
This is probably a bug in the backend, I will add an issue and we'll fix it asap.

Thanks Andrew for your willingness to hang on until we got it working :smile:

Added Default values breaks the Kibana UI · Issue #282 · elastic/connectors-python · GitHub