Create view or limit fields returned by metadata

[paulcarey]

Hi

I have an ES index which contains about 3000 fields. A number of clients are interested in using Dremio to query this index, joining the results with queries against other data sources. Unfortunately, Dremio has a hardcoded limit that prevents it from being used to query an index with more than 800 fields.

The number of fields required by most clients will typically be less than 800, so in practical terms there shouldn't be an issue - this seems like an artificial limitation that could be disabled. However, I'd like to know if there's anything I can do within ES to present a "view" of the index such that the mapping returns a subset of the total fields within the index.

Quoting from a Dremio solution architect:

Dremio gets the metadata for all of the columns to know if a given column a user references in a query exists or not. So if the ES dataset has 3000 fields, Dremio has to get information about them initially before it can be used, which results in hitting the 800 field limit that you are. A workaround would be to create a structure in ES that is like an RDBMS view, so to clients such as Dremio it looks like the dataset only has the subset of fields.

Many thanks

Paul

[paulcarey]

If ES doesn't support any way of restricting the fields returned by the metadata, then I will likely do one of the following:

• create a proxy that Dremio connects to - this would transparently proxy all endpoints except the mapping endpoint, for which it would return the required subset of fields
• create a separate index which contains only the required subset of fields

Neither option is particularly attractive. What would be much better is if Dremio would allow this check / artificial limitation to be overridden. I will follow up with Dremio separately on this point.