# Logstash upsert across multiple indexes

**URL:** <https://discuss.elastic.co/t/logstash-upsert-across-multiple-indexes/279307>\
**Category:** Logstash\
**Created:** [July 21, 2021, 6:49pm UTC](https://discuss.elastic.co/t/logstash-upsert-across-multiple-indexes/279307 "2021-07-21T18:49:21Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![jpchev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpchev/32/47826_2.png) [@jpchev](https://discuss.elastic.co/u/jpchev)\
**Post date:** [July 21, 2021, 6:49pm UTC](https://discuss.elastic.co/t/logstash-upsert-across-multiple-indexes/279307/1 "2021-07-21T18:49:21Z")

</div>

hi there, I need to import data in elasticsearch with an upsert logic, for this I use the following logstash plugin

```auto
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "myindex-{+YYYY.MM.dd}"
    document_id => "%{message_id}"
    doc_as_upsert => true
    action => "update"
  }
}

```

which works on the daily index myindex-{+YYYY.MM.dd}

but this doesn't seem to work as the document to be update could belong to an index different from myindex-{+YYYY.MM.dd}

So what's the best approach to have the upsert search across multiple index, or an index pattern, and have it inserted on the latest index myindex-{+YYYY.MM.dd} ?

I tried this plugin

> **[Elasticsearch filter plugin | Logstash Reference \[7.13\] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html)**

which I used to lookup the document and get its exact index name, but I was wondering if there is an easier way.

This is the template I use for the elasticsearch filter plugin

```auto
{
  "size": 1,
  "sort" : [{ "@timestamp" : "desc" }],
  "query": {
    "query_string": {
      "query": "_id:Bp4nynoBuY_R2Bpz1Xku"
    }
  },
  "_source": []
}

```

which is referenced by the following filter in logstash

```auto
filter {
   elasticsearch {
      hosts => ["localhost:9200"]
      index => "test*"
      query_template => "template.json"
      docinfo_fields => { 
        "_index" => "retrieved_index"
      }
   }
}

```

Any recommendations?  
Thanks

---

<div class="post-metadata">

**Author:** ![AquaX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aquax/32/92006_2.png) [@AquaX](https://discuss.elastic.co/u/AquaX)\
**Post date:** [July 22, 2021, 1:39pm UTC](https://discuss.elastic.co/t/logstash-upsert-across-multiple-indexes/279307/2 "2021-07-22T13:39:51Z")

</div>

I initially thought you may be able to alias the index but then I remembered that you cannot specify multiple write indexes.

> **[Aliases API | Elasticsearch Guide \[7.13\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-aliases.html#write-index)**

You cannot write to multiple indices and update a whole bunch at the same time.

Updating an index is an expensive process as it takes 3x the I/O (read, delete, write) vs (write) and is generally not what Elasticsearch should be used for.

Your solution to using the Elasticsearch filter to find the index is probably the best solution, but now you're doing 4x the I/O (read, read, delete, write). So as long as you are OK with that then you probably have the only solution that I can think of.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 19, 2021, 1:40pm UTC](https://discuss.elastic.co/t/logstash-upsert-across-multiple-indexes/279307/3 "2021-08-19T13:40:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.