Hello, I am running into issues when using GitHub - elastic/connectors-python: Official Connector Clients for Elastic Elasticsearch, Enterprise Search, App Search and Workplace Search CLI. I am able to run the connector client using elastic-ingest --debug -c config.yml, where my config looks like this:
elasticsearch:
host: https://127.0.0.1:9200
# API key generated in elastic search. Otherwise, use username and password.
username: elastic
password: el_password
ssl: true
bulk:
queue_max_size: 1024
queue_max_mem_size: 25
display_every: 100
chunk_size: 1000
max_concurrency: 5
chunk_max_mem_size: 5
request_timeout: 120
max_wait_duration: 120
initial_backoff_duration: 1
backoff_multiplier: 2
log_level: info
language_code: en
analysis_icu: false
service:
idling: 30
heartbeat: 300
max_errors: 20
max_errors_span: 600
trace_mem: false
native_service_types:
- mysql
# - network_drive
# - s3
# - gcs
# some id
sources:
# mongodb: connectors.sources.mongo:MongoDataSource
# s3: connectors.sources.s3:S3DataSource
# dir: connectors.sources.directory:DirectoryDataSource
mysql: connectors.sources.mysql:MySqlDataSource
# network_drive: connectors.sources.network_drive:NASDataSource
# gcs: connectors.sources.gcs:GoogleCloudStorageDataSource
and the output looks like this:
Desktop/connectors-python > elastic-ingest --debug -c config.yml
[FMWK][11:56:10][DEBUG] BYOIndex connecting to https://127.0.0.1:9200
[FMWK][11:56:10][DEBUG] Host is NodeConfig(scheme='https', host='127.0.0.1', port=9200, path_prefix='', headers={}, connections_per_node=10, request_timeout=10.0, http_compress=False, verify_certs=True, ca_certs=None, client_cert=None, client_key=None, ssl_assert_hostname=None, ssl_assert_fingerprint=None, ssl_version=None, ssl_context=None, ssl_show_warn=True, _extras={})
[FMWK][11:56:10][DEBUG] Connecting using Basic Auth (user: elastic, password: el_...)
[FMWK][11:56:10][DEBUG] Native support for mysql
[FMWK][11:56:10][DEBUG] Wait for Elasticsearch (max: 120)
[FMWK][11:56:10][INFO] Waiting for NodeConfig(scheme='https', host='127.0.0.1', port=9200, path_prefix='', headers={}, connections_per_node=10, request_timeout=10.0, http_compress=False, verify_certs=True, ca_certs=None, client_cert=None, client_key=None, ssl_assert_hostname=None, ssl_assert_fingerprint=None, ssl_version=None, ssl_context=None, ssl_show_warn=True, _extras={}) (so far: 0 secs)
[FMWK][11:56:10][INFO] Preflight checks...
[FMWK][11:56:10][DEBUG] Checking for index .elastic-connectors presence
[FMWK][11:56:10][DEBUG] Checking for index .elastic-connectors-sync-jobs presence
[FMWK][11:56:10][INFO] Service started, listening to events from https://127.0.0.1:9200
[FMWK][11:56:10][DEBUG] ElasticServer connecting to https://127.0.0.1:9200
[FMWK][11:56:10][DEBUG] Host is NodeConfig(scheme='https', host='127.0.0.1', port=9200, path_prefix='', headers={}, connections_per_node=10, request_timeout=10.0, http_compress=False, verify_certs=True, ca_certs=None, client_cert=None, client_key=None, ssl_assert_hostname=None, ssl_assert_fingerprint=None, ssl_version=None, ssl_context=None, ssl_show_warn=True, _extras={})
[FMWK][11:56:10][DEBUG] Connecting using Basic Auth (user: elastic, password: el_...)
[FMWK][11:56:10][DEBUG] Polling every 30 seconds
But I am still not able to use the connector, nor does it look like the CLI is trying to connect to my database. I also changed the mysql.py class config to use the correct values:
@classmethod
def get_default_configuration(cls):
"""Get the default configuration for MySQL server
Returns:
dictionary: Default configuration
"""
return {
"host": {
"value": "127.0.0.1",
"label": "Host",
"type": "str",
},
"port": {
"value": 3306,
"label": "Port",
"type": "int",
},
"user": {
"value": "strapi",
"label": "Username",
"type": "str",
},
"password": {
"value": "strapi",
"label": "Password",
"type": "str",
},
"database": {
"value": ["strapi"],
"label": "Databases",
"type": "list",
},
"connector_name": {
"value": "MySQL Connector",
"label": "Friendly name for the connector",
"type": "str",
},
"fetch_size": {
"value": DEFAULT_FETCH_SIZE,
"label": "How many rows to fetch on each call",
"type": "int",
},
"retry_count": {
"value": DEFAULT_RETRY_COUNT,
"label": "How many retry count for fetching rows on each call",
"type": "int",
},
"ssl_disabled": {
"value": DEFAULT_SSL_DISABLED,
"label": "SSL verification will be enabled or not",
"type": "bool",
},
"ssl_ca": {
"value": DEFAULT_SSL_CA,
"label": "SSL certificate",
"type": "str",
},
}
Even using an API key, and the connector ID I still do not see the connector becoming available as a source.
Any help will be greatly appreciated!
