# Ensuring Document Ordering in Bulk Ingestion **URL:** \ **Category:** Elastic Agent\ **Created:** [January 12, 2024, 2:50pm UTC](https://discuss.elastic.co/t/ensuring-document-ordering-in-bulk-ingestion/350972 "2024-01-12T14:50:48Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Ivelin\_Yanev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ivelin_yanev/32/130754_2.png) [@Ivelin\_Yanev](https://discuss.elastic.co/u/Ivelin_Yanev)\ **Post date:** [January 12, 2024, 2:50pm UTC](https://discuss.elastic.co/t/ensuring-document-ordering-in-bulk-ingestion/350972/1 "2024-01-12T14:50:48Z")
Hi everyone, I'm currently working on implementing bulk operations for documents. In my scenario, I receive PubSub messages and generate corresponding Elasticsearch documents using the data from these PubSub messages. I have a concern regarding the ordering of these documents during bulk operations. For example: I receive messages through PubSub: `message_1, message_2, message_3, message_4` Subsequently, I create the following Elasticsearch documents: `document_1, document_2, document_3, document_4` My objective is to maintain the order of these documents within Elasticsearch. I'm wondering if the bulk operation guarantees the preservation of the order I've created. Any insights or guidance on whether the order of documents can be ensured during bulk ingestion would be greatly appreciated. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [January 12, 2024, 3:38pm UTC](https://discuss.elastic.co/t/ensuring-document-ordering-in-bulk-ingestion/350972/2 "2024-01-12T15:38:13Z")
Hi @Ivelin_Yanev Welcome to the community What version are you on... Just curious. Can you explain a bit more what you mean by ordering? Is that based on ingestion timestamp or some other field? ---
**Author:** ![yago82](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yago82/32/97755_2.png) [@yago82](https://discuss.elastic.co/u/yago82)\ **Post date:** [January 12, 2024, 4:43pm UTC](https://discuss.elastic.co/t/ensuring-document-ordering-in-bulk-ingestion/350972/3 "2024-01-12T16:43:49Z")
Hi, Elasticsearch does maintain the order of operations in a bulk request. When you send a bulk request to Elasticsearch, it processes the operations in the order they appear in the request. This means that if you have document\_1, document\_2, document\_3, and document\_4 in that order in your bulk request, they will be processed in that same order. However, it's important to note that this doesn't necessarily mean that the documents will be available for search in the order they were processed. Elasticsearch is a distributed system, and once documents are indexed, they go through a process called refresh to make them available for search. This process is asynchronous and doesn't guarantee that documents will become searchable in the order they were indexed. Regards ---
**Author:** ![Ivelin\_Yanev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ivelin_yanev/32/130754_2.png) [@Ivelin\_Yanev](https://discuss.elastic.co/u/Ivelin_Yanev)\ **Post date:** [January 13, 2024, 7:17am UTC](https://discuss.elastic.co/t/ensuring-document-ordering-in-bulk-ingestion/350972/4 "2024-01-13T07:17:34Z")
Hi @stephenb In case you're inquiring about the Elasticsearch version, I'm currently using the v8.11 Java client. Allow me to provide more context: I receive ordered PubSub messages, and I've taken steps to ensure their sequencing. Specifically, when I receive multiple messages of the same type, I can affirm that the latest message is indeed the most recent one. My question pertains to the ordering of messages from the broker. If I send these ordered message(as create the proper Elasticsearch documents) in bulk, is there an assurance that they will be stored in Elasticsearch exactly as I've arranged them? Your insights into this matter would be greatly appreciated. ---
**Author:** ![Ivelin\_Yanev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ivelin_yanev/32/130754_2.png) [@Ivelin\_Yanev](https://discuss.elastic.co/u/Ivelin_Yanev)\ **Post date:** [January 13, 2024, 7:28am UTC](https://discuss.elastic.co/t/ensuring-document-ordering-in-bulk-ingestion/350972/5 "2024-01-13T07:28:17Z")
Hi, Thank you for your response. Could you kindly share the documentation resource supporting your statement regarding the order of operations in a bulk request? To clarify, I'm not concerned about searching or retrieving based on the order of processing. My primary focus is on ensuring that if the bulk operation guarantees ordering, it will resolve my issue. Here's the specific challenge I'm addressing: I might receive multiple messages of the same type, and the order is guaranteed by the PubSub broker. Consequently, I expect the last message to represent the latest version of the Elasticsearch document. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 13, 2024, 7:43am UTC](https://discuss.elastic.co/t/ensuring-document-ordering-in-bulk-ingestion/350972/6 "2024-01-13T07:43:37Z")
> [@Ivelin\_Yanev](#): > > Could you kindly share the documentation resource supporting your statement regarding the order of operations in a bulk request? I am not sure this is explicitly called out in the documentation. > [@Ivelin\_Yanev](#): > > Here's the specific challenge I'm addressing: I might receive multiple messages of the same type, and the order is guaranteed by the PubSub broker. Consequently, I expect the last message to represent the latest version of the Elasticsearch document. If you are indexing a document with a single id multiple times in a single bulk request I believe the ordering is preserved and the last one will prevail. If you however are sending multiple bulk requests in parallel based on the data in the PubSub broker there is no guarantee across bulk requests. As described earlier, note that some of the documents in the bulk request may be made available for search before others. It is also worth noting that frequenty indexing/updating the same document can result in significant overhead and reduce indexing throughput. ---
**Author:** ![Ivelin\_Yanev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ivelin_yanev/32/130754_2.png) [@Ivelin\_Yanev](https://discuss.elastic.co/u/Ivelin_Yanev)\ **Post date:** [January 15, 2024, 9:29am UTC](https://discuss.elastic.co/t/ensuring-document-ordering-in-bulk-ingestion/350972/7 "2024-01-15T09:29:34Z")
Thanks @Christian_Dahlqvist I would like to inquire about the scenario where a build operation is executed, and if a specific document fails, will it lead to the failure of the entire build process or not? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 15, 2024, 9:49am UTC](https://discuss.elastic.co/t/ensuring-document-ordering-in-bulk-ingestion/350972/8 "2024-01-15T09:49:51Z")
All items of a bulk request are indexed individually, so failures will just affect parts of the bulk request. The success or failure of individual documents is reported in the response. You could therefore have e.g. 3 updates to the same document in a bulk request and have only the second one fail. There is no transactions or atomicity. ---
**Author:** ![Ivelin\_Yanev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ivelin_yanev/32/130754_2.png) [@Ivelin\_Yanev](https://discuss.elastic.co/u/Ivelin_Yanev)\ **Post date:** [January 15, 2024, 10:54am UTC](https://discuss.elastic.co/t/ensuring-document-ordering-in-bulk-ingestion/350972/9 "2024-01-15T10:54:22Z")
In this scenario, I have three documents (doc1, doc2, doc3) containing the same data. Ideally, in a perfect case, the last document (doc3) would represent the latest version, given that my ordering is guaranteed by PubSub. However, if the last document (doc3) encounters a failure, Elasticsearch would then have an outdated version (not the latest) available. Additionally, I need to confirm the acknowledgment message to PubSub once these documents are successfully stored in Elasticsearch. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 15, 2024, 11:14am UTC](https://discuss.elastic.co/t/ensuring-document-ordering-in-bulk-ingestion/350972/10 "2024-01-15T11:14:19Z")
> [@Ivelin\_Yanev](#): > > However, if the last document (doc3) encounters a failure, Elasticsearch would then have an outdated version (not the latest) available. True, but you can identify this from the bulk response and potentially retry (if needed). ---
**Author:** ![Ivelin\_Yanev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ivelin_yanev/32/130754_2.png) [@Ivelin\_Yanev](https://discuss.elastic.co/u/Ivelin_Yanev)\ **Post date:** [January 15, 2024, 2:27pm UTC](https://discuss.elastic.co/t/ensuring-document-ordering-in-bulk-ingestion/350972/11 "2024-01-15T14:27:27Z")
Hi @Christian_Dahlqvist I believe there is a perfect approach for my case. I will utilize **versioning** with `version_type` set to `external`. This way, my data will have versioning, and I can leverage it. Elasticsearch will ensure that the latest version is stored. Consequently, I won't need to be concerned about the ordering of the documents in this scenario. > **[Index API | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html#index-versioning)** ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 15, 2024, 2:41pm UTC](https://discuss.elastic.co/t/ensuring-document-ordering-in-bulk-ingestion/350972/12 "2024-01-15T14:41:04Z")
If you have a field in your data that indicate the version that may indeed be a good solution. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 12, 2024, 2:41pm UTC](https://discuss.elastic.co/t/ensuring-document-ordering-in-bulk-ingestion/350972/13 "2024-02-12T14:41:27Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.