Set op_type to "index" filebeat / elastic-agent

Hi All,

I'm ingesting documents into an index (not a data stream). As I need to update specific documents, I created an ingest pipline which sets the _id field. However I can not see my documents being updated and I suspect it's because of the following according to the documentation of the Index API:

When using the POST /<target>/_doc/ request format, the op_type is automatically set to create and the index operation generates a unique ID for the document.

My intended _id is correctly set via the ingest pipeline, but I wonder if the above still applies (as at request time there was no _id yet) and If I can force the op_type to be set to "index" ? I'm using elastic-agent for document ingestion which is already writing to a normal index as I have adjusted the index template.

My overall goal is to override documents ingested by the elastic-agent (alternatively filebeat) based on the _id field set in the ingest pipeline.

Thanks in advance!

Hi @beijo

See this rather detailed thread.

Hi @stephenb

thanks for your answer!

As you suggested at the bottom of the other threat, I tried to set @metadata.op_type: "index" in my ingest pipeline, but I guess that does not do the trick and instead I need to add it to the processor?
Is that correct or am I misunderstanding something?

Sorry I got a little confused in the other threat, I'm using version 8.4

I think adding it to the processor might do the trick (next I need to handle a permission issue regarding the agent which does by default not have permissions for op_type "index" requests.

Hi @beijo

In Short it will not work either way today... in ingest pipeline not processor.. the processor is what I tried.

And I think you missed the part about this not working at all in 8.x because they are data streams which are create / append only by definition.

So I think the only way to get this to work for you is to use indices and use the solution I showed with logstash.


Thanks again. Actually I configured Elastic-Agent to write to a "normal" index and not a data stream (by changing the index template). And adding the op_type to the processor instead of ingest pipeline seems to bring me one step forward.

However now I'm facing the problem that fleet managed elastic-agents do not have the privilege to use op_type "index" per default:

action [indices:data/write/index:op_type/index] is unauthorized for API key ... of user [elastic/fleet-server] on indices [logs-...], this action is granted by the index privileges [create,index,write,all]"}]}, dropping event!

Awesome I did not get to that yet...

Hmmm .... I would need to think about that ....keep digging... let us know what you find

Yes you will probably need to fix the from

An output API key

This API key is used to send data to Elasticsearch. It has the minimal permissions needed to ingest all the data specified by the agent policy. If the API key is invalid, the Elastic Agent stops ingesting data into Elasticsearch.

How to the that set... in Fleet Managed not sure....

Perhaps use the new API

Yep, the "problem" is the "minimal permissions" configuration used by Fleet to create the Elasticsearch Output API Key.
It seems it is not possible to specify custom requirements for that key when agents are managed by Fleet.

  • Maybe the "update API key" Update API key API | Elasticsearch Guide [master] | Elastic
    API can be used to adjust the permissions of that key, but I hesitate to modify the system generated key.

  • Alternatively maybe the API key could be replaced inside the elastic-agent.yml configuration file on the host (to prevent changing the API key for all agents), but as it should not be touched it might also not be a preferred solution. Edit: config file might only be an option for standalone agents

Related information: [Discuss] Indexing permissions as part of the Elastic Agent policy · Issue #101 · elastic/fleet-server · GitHub

Heya @beijo A thought...

Perhaps add a new output and create your own key and set it there....

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.