# Set op\_type to "index" filebeat / elastic-agent **URL:** \ **Category:** Elastic Agent\ **Tags:** filebeat\ **Created:** [December 19, 2022, 1:24pm UTC](https://discuss.elastic.co/t/set-op-type-to-index-filebeat-elastic-agent/321562 "2022-12-19T13:24:12Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![beijo](https://avatars.discourse-cdn.com/v4/letter/b/58f4c7/32.png) [@beijo](https://discuss.elastic.co/u/beijo)\ **Post date:** [December 19, 2022, 1:24pm UTC](https://discuss.elastic.co/t/set-op-type-to-index-filebeat-elastic-agent/321562/1 "2022-12-19T13:24:12Z")
Hi All, I'm ingesting documents into an index (not a data stream). As I need to update specific documents, I created an ingest pipline which sets the `_id` field. However I can not see my documents being updated and I suspect it's because of the following according to the documentation of the Index API: > When using the `POST //_doc/` request format, the `op_type` is automatically set to `create` and the index operation generates a unique ID for the document. My intended `_id` is correctly set via the ingest pipeline, but I wonder if the above still applies (as at request time there was no `_id` yet) and If I can force the `op_type` to be set to `"index"` ? I'm using elastic-agent for document ingestion which is already writing to a normal index as I have adjusted the index template. My overall goal is to override documents ingested by the elastic-agent (alternatively filebeat) based on the `_id` field set in the ingest pipeline. Thanks in advance! ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [December 19, 2022, 2:48pm UTC](https://discuss.elastic.co/t/set-op-type-to-index-filebeat-elastic-agent/321562/2 "2022-12-19T14:48:36Z")
Hi @beijo See this rather detailed thread. > [@Filebeat and updating existing docs](https://discuss.elastic.co/t/filebeat-and-updating-existing-docs/320781/13): > > Wow I spent a lot of time on this...I have a question into engineering... I have no problem getting the fingerprint to work on the initial index but I can not get it to update the document. I have a suspicion why, but I will wait. I tried everything I know including rarely used configurations I could not get it to work... Apologies about the difficulty I agree Filebeat / Elasticsearch is not working as documented described [here](https://www.elastic.co/guide/en/beats/filebeat/7.17/filebeat-deduplication.html) So in the meantime if you want to set your own \_id and update the… ---
**Author:** ![beijo](https://avatars.discourse-cdn.com/v4/letter/b/58f4c7/32.png) [@beijo](https://discuss.elastic.co/u/beijo)\ **Post date:** [December 19, 2022, 3:01pm UTC](https://discuss.elastic.co/t/set-op-type-to-index-filebeat-elastic-agent/321562/3 "2022-12-19T15:01:25Z")
Hi @stephenb thanks for your answer! As you suggested at the bottom of the other threat, I tried to set `@metadata.op_type: "index"` in my ingest pipeline, but I guess that does not do the trick and instead I need to add it to the processor? Is that correct or am I misunderstanding something? Sorry I got a little confused in the other threat, I'm using version 8.4 Edit: I think adding it to the processor might do the trick (next I need to handle a permission issue regarding the agent which does by default not have permissions for op\_type "index" requests. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [December 19, 2022, 3:57pm UTC](https://discuss.elastic.co/t/set-op-type-to-index-filebeat-elastic-agent/321562/4 "2022-12-19T15:57:42Z")
Hi @beijo In Short it will not work either way today... in ingest pipeline not processor.. the processor is what I tried. And I think you missed the part about this not working at all in 8.x because they are data streams which are create / append only by definition. So I think the only way to get this to work for you is to use indices and use the solution I showed with logstash. ---
**Author:** ![beijo](https://avatars.discourse-cdn.com/v4/letter/b/58f4c7/32.png) [@beijo](https://discuss.elastic.co/u/beijo)\ **Post date:** [December 19, 2022, 4:05pm UTC](https://discuss.elastic.co/t/set-op-type-to-index-filebeat-elastic-agent/321562/5 "2022-12-19T16:05:18Z")
@stephenb Thanks again. Actually I configured Elastic-Agent to write to a "normal" index and not a data stream (by changing the index template). And adding the op\_type to the processor instead of ingest pipeline seems to bring me one step forward. However now I'm facing the problem that fleet managed elastic-agents do not have the privilege to use `op_type "index"` per default: `action [indices:data/write/index:op_type/index] is unauthorized for API key ... of user [elastic/fleet-server] on indices [logs-...], this action is granted by the index privileges [create,index,write,all]"}]}, dropping event!` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [December 19, 2022, 4:29pm UTC](https://discuss.elastic.co/t/set-op-type-to-index-filebeat-elastic-agent/321562/6 "2022-12-19T16:29:11Z")
> [@beijo](#): > > Thanks again. Actually I configured Elastic-Agent to write to a "normal" index and not a data stream (by changing the index template). And adding the op\_type to the processor instead of ingest pipeline seems to bring me one step forward. Awesome I did not get to that yet... > [@beijo](#): > > action [indices:data/write/index:op\_type/index] is unauthorized for API key ... of user [elastic/fleet-server] on indices [logs-...], this action is granted by the index privileges [create,index,write,all]"}]}, dropping event! Hmmm .... I would need to think about that ....keep digging... let us know what you find Yes you will probably need to fix the from > An output API key > > This API key is used to send data to Elasticsearch. It has the minimal permissions needed to ingest all the data specified by the agent policy. If the API key is invalid, the Elastic Agent stops ingesting data into Elasticsearch. How to the that set... in Fleet Managed not sure.... > **[Configure the Elasticsearch output | Fleet and Elastic Agent Guide \[8.11\] |...](https://www.elastic.co/guide/en/fleet/current/elasticsearch-output.html#output-elasticsearch-apikey-authentication-settings)** Perhaps use the new API > **[Kibana Fleet APIs | Fleet and Elastic Agent Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/fleet/current/fleet-api-docs.html#create-agent-policy-api)** ---
**Author:** ![beijo](https://avatars.discourse-cdn.com/v4/letter/b/58f4c7/32.png) [@beijo](https://discuss.elastic.co/u/beijo)\ **Post date:** [December 19, 2022, 4:51pm UTC](https://discuss.elastic.co/t/set-op-type-to-index-filebeat-elastic-agent/321562/7 "2022-12-19T16:51:41Z")
Yep, the "problem" is the "minimal permissions" configuration used by Fleet to create the Elasticsearch Output API Key. It seems it is not possible to specify custom requirements for that key when agents are managed by Fleet. - Maybe the "update API key" [Update API key API | Elasticsearch Guide [master] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/master/security-api-update-api-key.html) API can be used to adjust the permissions of that key, but I hesitate to modify the system generated key. - Alternatively maybe the API key could be replaced inside the elastic-agent.yml configuration file on the host (to prevent changing the API key for all agents), but as it should not be touched it might also not be a preferred solution. Edit: config file might only be an option for standalone agents Related information: [[Discuss] Indexing permissions as part of the Elastic Agent policy · Issue #101 · elastic/fleet-server · GitHub](https://github.com/elastic/fleet-server/issues/101) ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [December 19, 2022, 5:06pm UTC](https://discuss.elastic.co/t/set-op-type-to-index-filebeat-elastic-agent/321562/8 "2022-12-19T17:06:52Z")
Heya @beijo A thought... Perhaps add a new output and create your own key and set it there.... ![Screen Shot 2022-12-19 at 9.04.54 AM](https://us1.discourse-cdn.com/elastic/original/3X/a/d/adc43a2dcd7598c8a63ec76a3104933fc2f9e72c.jpeg) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 16, 2023, 5:07pm UTC](https://discuss.elastic.co/t/set-op-type-to-index-filebeat-elastic-agent/321562/9 "2023-01-16T17:07:20Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.