Within painless scripts I use for script queries (in a filter context), sort scripts and script fields, I need to refer to the JSON structure of the source document, which is fairly complex and has several arrays of objects within its structure (stored as nested fields).
The queries I write are analytics-heavy so performance isn't the most important thing.
To refer to the document source, I'm able to refer to params._source within painless when I'm writing a script field or a sort script. But this params._source field is missing when I'm writing a script query.
Is there a way around this? I cannot do doc['path.to.field'].value when path.to is a nested object, and I need to run a for loop through the objects in path.to.
Accessing source is very slow. I would think with analytics performance would be very important. Regardless, it was never intentional to allow access to source from search scripts in painless. It was simply a side effect of giving access for script fields, which we are now correcting with the separation of script contexts.
What kind of calculation are you trying to do over the nested docs? Have you tried a nested query? You can do sum, min or max currently.
I don't have that many documents for the speed to have that much of an
impact. I'm happy with the speed at which my sort filter is running and it
uses params._source.
The documents themselves are fat though. The JSON file of a single document
with indentation is around 32kb. My mapping was automatically generated by
observing the structure of a document - each time I have an array of
objects I treat it as a nested object whole everything else is a regular
object. All other fields are keywords, booleans or numerics.
The reason I need access is that I sometimes need to search an array of
objects and look for an entry which has a key containing a specific value
and then read another key's value in that entry and use it in some numeric
computations. Other times I need to selectively apply a different filter
depending on what value a specific field contains.
I suppose my biggest problem is that I am running queries on raw data that
hasn't been processed by hand for optimal query performance so I need the
queries to be very flexible.
Without examining the base document structure this would be really hard.
Is there some way of iterating through nested field objects with regular
queries?
I'm using a script filter within a nested query. The result is less readable but works for my specific use-case. If the use-case becomes more complex and I absolutely need _source, I'll start worrying about how to do it then. Right now my ideas are along the lines of implementing field double indexing (store the same field twice, once as nested object and another as a regular object) in some form when the need arises.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.