You mentioned 50 fields above, which is considerably lower than 1000....
50,This is just one example
I suspect the main problem with fetching non-indexed fields from source is that the format has not been validated and defined at index time. You could easily have field conflicts, e.g. string vs objects, that would cause problems. In a relational database the column format is controlled even if it is not indexed, which is what the indexing step in Elasticsearch does.
I normally only trust what I can measure.
You might see a difference indeed. The question is more about "how much", when the cluster has loaded most of the data structures in the file system cache? Is that nanoseconds, milliseconds or dozen of seconds we are speaking about?
Of course, 50 fields vs 1000 fields might be another story.
ES/SQL will make use of the new fields API in one of the next coming releases. That will lift some of the inconveniences of having to selecting fields from specific "sources".
That said, however:
- as it's been pointed out, doing actual performance analysis and point out problems that way is generally more productive than theoretical discussions.
- doing
SELECT *on hundred or more fields is rarely useful. Feel free to share your use case if you think we should improve anything here, though, we're happy to help!
My cluster has 500TB data， Do you think most data caches are possible？
Thankyou.
My cluster has a large amount of data(500TB), almost every index has hundreds of fields.
Most businesses use RestClient for queries. But now some businesses want to try SQL.
But I found,SQL can't retrieve unindexed fields（ I have to modify all index mapping to set these fields "index:true" or "store:true",It's inconvenient）.And some indexes have thousands of fields, I find it a little slow to retrieve these fields using SQL.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.