Hello,
I am showing you here a problem encountered in our environments with the indexing of a row in Elasticsearch which is not done as soon as the decimal value contained in the key exceeds 64 (side effect of a base 64 connector?).
Context: replication of data from a DB2 Z / os to a Kafka (confluence) and indexing in an Elasticsearch 7.6.2 via a connector (Kafka connect confluence).
Data: line containing a key composed of:
1 column CHAR format
1 column decimal format
1 column CHAR format
Example:
PAUL 1 DUBOIS ADDRESS 1 BOULANGER
PAUL 2 DUBOIS ADDRESS 2 BOUCHER
PAUL 3 DUBOIS ADDRESS 2 PLOMBIER
... / ...
PAUL 65 DUBOIS ADDRESS 65 JOINER
Attempt to insert more than 64 rows with the same key except the decimal column.
Description of the problem: from the 64th line on no longer finds the record in Elasticsearch.
The lines> to 64 are however extracted (via the capture of modified data) from the DB2 Z / os database, well written in the Kafka topic (verified) but they are not found in Elasticsearch after indexing via Kafka connect.
The problem therefore occurs in the chain at indexing. The Kafka connector does not display any rejects or warnings.
Elasticsearch isn't meaningful either.
Test done several times with different records but always with col1 and col3 of the same key and col2 incremented (1,2,3,4 etc ... up to more than 64).
Note:
The functional key used in the connector definition is the one indicated above.
But the internal ID generated by Elasticsearch still has a key technique.
Is this the source of the problem?
If so, how to get around it?
Other opinions?
Thanks for your help.