# Insert and Update records in Elasticsearch **URL:** \ **Category:** Elasticsearch\ **Created:** [April 28, 2021, 4:35am UTC](https://discuss.elastic.co/t/insert-and-update-records-in-elasticsearch/271459 "2021-04-28T04:35:04Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Thamini\_NellaiKumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thamini_nellaikumar/32/62550_2.png) [@Thamini\_NellaiKumar](https://discuss.elastic.co/u/Thamini_NellaiKumar)\ **Post date:** [April 28, 2021, 4:35am UTC](https://discuss.elastic.co/t/insert-and-update-records-in-elasticsearch/271459/1 "2021-04-28T04:35:04Z")
Hi, I am looping through each record in elasticsearch and inserting the new records and updating the existing records. Say for example, i have a csv file which has 5000 records, only few are updated in the index and the rest is missed. And my script to insert and update runs for every 5 mins. Is there a way to insert all the records without missing it? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 28, 2021, 7:27am UTC](https://discuss.elastic.co/t/insert-and-update-records-in-elasticsearch/271459/2 "2021-04-28T07:27:51Z")
How do you know there are missing? Did you refresh the index before searching? How are you inserting data? ---
**Author:** ![Thamini\_NellaiKumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thamini_nellaikumar/32/62550_2.png) [@Thamini\_NellaiKumar](https://discuss.elastic.co/u/Thamini_NellaiKumar)\ **Post date:** [April 29, 2021, 11:29am UTC](https://discuss.elastic.co/t/insert-and-update-records-in-elasticsearch/271459/3 "2021-04-29T11:29:58Z")
I am inserting the data as a dataframe using python. The number of rows in the csv file is not the same in the index after the insertion. No, i dont refresh the index ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 29, 2021, 11:33am UTC](https://discuss.elastic.co/t/insert-and-update-records-in-elasticsearch/271459/4 "2021-04-29T11:33:10Z")
So 3 things are coming to my mind: - You are using the same `_id` for some documents so some documents are updated - There are errors for some documents but you are not looking at them - You are calling `_search` immediately after the last index operation and because you are not refreshing the index manually, the last batch of documents is not searchable yet. ---
**Author:** ![Thamini\_NellaiKumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thamini_nellaikumar/32/62550_2.png) [@Thamini\_NellaiKumar](https://discuss.elastic.co/u/Thamini_NellaiKumar)\ **Post date:** [May 4, 2021, 6:37am UTC](https://discuss.elastic.co/t/insert-and-update-records-in-elasticsearch/271459/5 "2021-05-04T06:37:12Z")
Thanks for the response. I have added the refresh part in my script. ---
**Author:** ![Thamini\_NellaiKumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thamini_nellaikumar/32/62550_2.png) [@Thamini\_NellaiKumar](https://discuss.elastic.co/u/Thamini_NellaiKumar)\ **Post date:** [May 13, 2021, 10:14am UTC](https://discuss.elastic.co/t/insert-and-update-records-in-elasticsearch/271459/6 "2021-05-13T10:14:03Z")
Hi, I added refresh. But what I found is only at a particular time of the day, the data is missed from insert. Index has 50,000 to 2 lakh records and each run will insert a minimum of 2000 records. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 14, 2021, 2:12am UTC](https://discuss.elastic.co/t/insert-and-update-records-in-elasticsearch/271459/7 "2021-05-14T02:12:22Z")
Then the other two options I mentioned are still valid. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 11, 2021, 2:12am UTC](https://discuss.elastic.co/t/insert-and-update-records-in-elasticsearch/271459/8 "2021-06-11T02:12:48Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.