I have an ES index which contains about 3000 fields. A number of clients are interested in using Dremio to query this index, joining the results with queries against other data sources. Unfortunately, Dremio has a hardcoded limit that prevents it from being used to query an index with more than 800 fields.
The number of fields required by most clients will typically be less than 800, so in practical terms there shouldn't be an issue - this seems like an artificial limitation that could be disabled. However, I'd like to know if there's anything I can do within ES to present a "view" of the index such that the mapping returns a subset of the total fields within the index.
Quoting from a Dremio solution architect:
Dremio gets the metadata for all of the columns to know if a given column a user references in a query exists or not. So if the ES dataset has 3000 fields, Dremio has to get information about them initially before it can be used, which results in hitting the 800 field limit that you are. A workaround would be to create a structure in ES that is like an RDBMS view, so to clients such as Dremio it looks like the dataset only has the subset of fields.