Problems with Java_transport client tribe

Elastic Stack Elasticsearch
1

I have a situation after updating to Elasticsearch to 2.3.5 (from 2.1)

We have two clusters federated via tribe.
To make things clear, one cluster we will call the tablet_cluster and the other we will call site_cluster.

Before the update we had no issues with a java_transport client getting documents using wildcards. (from either cluster via tribe transport node.)

The java_transport client jobs run out of talend and there are several java_transport client calls to indices that run w/o error.
There are six jobs that run nightly
1 process, the java_transport client was successfully pulling documents from multiple indices using slp_site_sm_* wildcard call (from the site_cluster)
on 5 other processes, the java_transport client was also pulling docs from the other tablet_cluster using aliases and wildcard calls. (sv_payment_info, an alias and sv_purchase_* a wildcard etc)

After the update:
The 5 jobs that run w/o error are all pointed to the tablet_cluster using both aliases and wildcards. (sv_payment_info and sv_purchase_*)

The 1 job that errors out is a java_transport client job that is pointed to slp_site_sm_* coming from the site_cluster and is using a wildcard.

If I change the call to an actual index name slp_site_sm_06603_2016218 it will pull data but obviously only from that index. We need a wildcard to pull data from all slp_site_sm_* matches.
If I use sense and the http rest call I can get all documents using the wildcard call slp_site_sm_*.
This is only an issue on transport for indices on the site_cluster side using wildcard calls. It appears aliases work and so do real index names.

I dont want to make a permenent bandaid by creating a alias for a bunch of indices. I really would like the * wildcard in the transport to work as the http calls do.

One more thing:
I do get a message from Talend stating that the _type tablets is not found. However using sense/REST commands on http I can retrieve over 6000 documents where type=tablets.
Not sure if the error returned via talend (type tablets not found) is because it couldn't find the index or it finds the index and then cannot find that type.

Hints and suggestions would be appreciated.

2

Is talend doing the integration? Was it updated along with your ES?

3

No talend change were made as far as I know.
The JRE on the talend machine was updated to 1.8.0 as was the ELK cluster.

There are many other jobs running out of Talend just this one process to get documents against the slp_site_sm_* wildcard is broken.

4

Figured this one out on my own.

We had a project to change naming convention of a series of indices in order to make the alias and index names different. (so if people reindex and forget to remove the old index it doesn't get included w/o an alias. )

example
we had indices with names similar to
slp_site_sm_06603_2016218
and were in the process of reindexing/renaming them to
sysmdl_site_06603_2016218 with an alias of slp_site_sm_06603

when talend called the wildcard slp_site_sm_* it not only matched the old index names we hadn't finished migrating to the new naming convention it also matched the new index's alias name slp_site_sm_06603.

I think this created a circular reference.
I also found one index with a different schema and so one field had a data type mismatch.

I re-aliased the indices I wanted talend to see with talend_slp_site_sm_06603 (or what numbers matched the other indices) to break up the match of wildcard slp_site_sm_* from the alias names.

This seems to have fixed my issues.

Clear as mud ?

Mark