I'm ingesting documents into an index (not a data stream). As I need to update specific documents, I created an ingest pipline which sets the _id field. However I can not see my documents being updated and I suspect it's because of the following according to the documentation of the Index API:
When using the POST /<target>/_doc/ request format, the op_type is automatically set to create and the index operation generates a unique ID for the document.
My intended _id is correctly set via the ingest pipeline, but I wonder if the above still applies (as at request time there was no _id yet) and If I can force the op_type to be set to "index" ? I'm using elastic-agent for document ingestion which is already writing to a normal index as I have adjusted the index template.
My overall goal is to override documents ingested by the elastic-agent (alternatively filebeat) based on the _id field set in the ingest pipeline.
As you suggested at the bottom of the other threat, I tried to set @metadata.op_type: "index" in my ingest pipeline, but I guess that does not do the trick and instead I need to add it to the processor?
Is that correct or am I misunderstanding something?
Sorry I got a little confused in the other threat, I'm using version 8.4
Edit:
I think adding it to the processor might do the trick (next I need to handle a permission issue regarding the agent which does by default not have permissions for op_type "index" requests.
Thanks again. Actually I configured Elastic-Agent to write to a "normal" index and not a data stream (by changing the index template). And adding the op_type to the processor instead of ingest pipeline seems to bring me one step forward.
However now I'm facing the problem that fleet managed elastic-agents do not have the privilege to use op_type "index" per default:
action [indices:data/write/index:op_type/index] is unauthorized for API key ... of user [elastic/fleet-server] on indices [logs-...], this action is granted by the index privileges [create,index,write,all]"}]}, dropping event!
Hmmm .... I would need to think about that ....keep digging... let us know what you find
Yes you will probably need to fix the from
An output API key
This API key is used to send data to Elasticsearch. It has the minimal permissions needed to ingest all the data specified by the agent policy. If the API key is invalid, the Elastic Agent stops ingesting data into Elasticsearch.
How to the that set... in Fleet Managed not sure....
Yep, the "problem" is the "minimal permissions" configuration used by Fleet to create the Elasticsearch Output API Key.
It seems it is not possible to specify custom requirements for that key when agents are managed by Fleet.
Alternatively maybe the API key could be replaced inside the elastic-agent.yml configuration file on the host (to prevent changing the API key for all agents), but as it should not be touched it might also not be a preferred solution. Edit: config file might only be an option for standalone agents
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
