I want to run the above script runtime, that's why I'm using Injest Pipeline test-embedded-script isn't working in apiproxy-embedded pipeline.
Please help me to write this script as one of the processors of my pipeline.
Ingest pipline works literally while ingesting. If you want to run the script at query runtime, use runtime mappings in the query. "Define runtime fields in a search request" may help you.
@Tomo_M I tried this and as per my understanding this only works if my existing field is mapped or has to be mapped? As per my need I don't want to map my existing fields (responseBody.results[i].statusCode) but new field embedded_error_code.
Because Injest pipeline worked in my case that gives me new single field without mapping my existing field which might be a json array, I inclined towards it. Any other solution please help
In most cases, retrieve field values through doc_values whenever possible. Accessing doc_values with a runtime field is faster than retrieving values from _source because of how data is loaded from Lucene.
However, there are cases where retrieving fields from _source is necessary. For example, text fields do not have doc_values available by default, so you have to retrieve values from _source. In other instances, you might choose to disable doc_values on a specific field. Map a runtime field | Elasticsearch Guide [8.11] | Elastic
Accessing _source in runtime field is also possible.
Hi @Tomo_M, at runtime - I mean doing this all stuff while data takes defined mapping or default pipeline configuration from Index Template.
Now, Why runtime field didn't work for me - this field (responseBody.results[i].statusCode) has different data type in different document for eg. my logs are coming from different apiProxies so that's why I get different responseBody and sometimes it's a keyword and sometimes it's an object.
Injest pipeline worked for my other cases where I didn't have list in my responseBody but not here - (responseBody.results[i].statusCode) because result is a list.
Error above in screenshot- Object mapping for [responseBody] tried to parse field [responseBodt] as object, but foound a concrete value
That said, the error caused by the ' responseBody ' field type is not consistent with mappings. The error will occure even without the runtime mapping.
To answer above statement -
That's why I'm not indexing this field (responseBody) in my index pattern so that I don't get conflicted field issue (occurs when same field has different data type in the single index pattern)
Maybe you do not turn off dynamic mappings and the field type mapping for responseBody was created at the first document indexed. Then error occured when documents containing other type responseBody were to be indexed.
Than meant your processor does not remove the original fields.
Tested with this script although this time I didn't get any error but I didn't get result in embedded_error_code either. Is there any mistakes in below script ?
Note : I defined this script directly in Injest pipeline processor list
Using this script, not getting null pointer exception while reindexing but still not getting values ['responseBody']['results'][i]['statusCode']of in [embedded_error_code]
POST _scripts/embeddedcode_script
{
"script": {
"description": "testing responseBody script again",
"lang": "painless",
"source": """
if( ctx['responseBody'] != null && ctx.containsKey('results'))
{
for (int i = 0; i < ctx['responseBody']['results'].value.size(); ++i)
{
ctx['embedded_error_code'] = ctx['responseBody']['results'][i]['statusCode'];
}
}"""
}
}
I'm understanding a mistake here why I'm not getting values in embedded_error_code
When I tested a document in pipeline for testing pipeline, Script got executed but I guess this line is not working here in Script as it worked for _update_query -
is the way to set a value on that field. But your script overwrite the value in each iteration.
I'm not sure what you want to do, it is possible to create an array and add values in each iteration, then finally set the array to ctx['embedded_error_code'].
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.