# Replace the indexed data with new data

**URL:** <https://discuss.elastic.co/t/replace-the-indexed-data-with-new-data/25442>\
**Category:** Logstash\
**Created:** [July 13, 2015, 12:25pm UTC](https://discuss.elastic.co/t/replace-the-indexed-data-with-new-data/25442 "2015-07-13T12:25:26Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![sanju1323](https://avatars.discourse-cdn.com/v4/letter/s/5daacb/32.png) [@sanju1323](https://discuss.elastic.co/u/sanju1323)\
**Post date:** [July 13, 2015, 12:25pm UTC](https://discuss.elastic.co/t/replace-the-indexed-data-with-new-data/25442/1 "2015-07-13T12:25:26Z")

</div>

Hi,

I have an Index named "sanjay\_data" and I want to replace the already existing data in the index with the new data.

Instead of deleting the index and creating it again, is there any possibility?

Please help me in doing this.

Thanks & Regards,  
Sanjay Reddy.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 13, 2015, 1:14pm UTC](https://discuss.elastic.co/t/replace-the-indexed-data-with-new-data/25442/2 "2015-07-13T13:14:48Z")

</div>

You can update a single document atomically but for a multi-document index you should look into using index aliases as described in the [Changing Mapping with Zero Downtime](https://www.elastic.co/blog/changing-mapping-with-zero-downtime) blog post. Unfortunately I don't think Logstash has any built-in support for updating indexes to support this use case.

---

<div class="post-metadata">

**Author:** ![sanju1323](https://avatars.discourse-cdn.com/v4/letter/s/5daacb/32.png) [@sanju1323](https://discuss.elastic.co/u/sanju1323)\
**Post date:** [July 13, 2015, 1:20pm UTC](https://discuss.elastic.co/t/replace-the-indexed-data-with-new-data/25442/3 "2015-07-13T13:20:00Z")

</div>

@magnusbaeck

"sanjay\_data" is a single document index. If I update, the new data will be added up to the old data. But, I want to replace all the old data with the new one.

Can we do this?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 13, 2015, 1:24pm UTC](https://discuss.elastic.co/t/replace-the-indexed-data-with-new-data/25442/4 "2015-07-13T13:24:21Z")

</div>

If you set the elasticsearch output's [`document_id`](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-document_id) parameter to a fixed value, Logstash will update the existing document atomically updated instead of just piling on another document with an automatically chosen document id.

---

<div class="post-metadata">

**Author:** ![sanju1323](https://avatars.discourse-cdn.com/v4/letter/s/5daacb/32.png) [@sanju1323](https://discuss.elastic.co/u/sanju1323)\
**Post date:** [July 15, 2015, 6:43am UTC](https://discuss.elastic.co/t/replace-the-indexed-data-with-new-data/25442/5 "2015-07-15T06:43:43Z")

</div>

@magnusbaeck

When I tried giving the document\_id, only the last record in the document is indexing. Remaining data is not coming up

I have provided the screenshots of kibana and Head plugin to show only one record is indexed.

 ![](https://us1.discourse-cdn.com/elastic/original/2X/1/10f9f3adf6a16f7169677631c0a313fc4c6002dd.png)

![](https://us1.discourse-cdn.com/elastic/original/2X/3/39358dc811ba9f40c6b1e5a265b162d4a2c746b7.png)

This is the output of the config file that I used.  
output  
{  
elasticsearch\_http  
{  
host =\> "localhost"  
index =\> "sanjay\_data"  
index\_type =\> "sanjay\_data"  
document\_id =\> "%{[@metadata][\_id]}"  
template =\> "Q:/softwares/ElasticSearch/logstash-1.3.3-flatjar/elasticsearch-template-sanjay\_data.json"  
template\_name =\> "sanjay\_data"  
}  
stdout  
{  
codec =\> "json"  
debug =\> true  
}

}

Should I change anything?  
Please help.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 15, 2015, 7:26am UTC](https://discuss.elastic.co/t/replace-the-indexed-data-with-new-data/25442/6 "2015-07-15T07:26:29Z")

</div>

I thought that's what you wanted; update the existing document (singular).

> document\_id =\> "%{[@metadata][\_id]}"

As your Kibana screenshot shows, there is no `[@metadata][_id]` field so the id of each document is the literal string `[@metadata][_id]`, which obviously explains why there's only one document. What inputs do you have?

---

<div class="post-metadata">

**Author:** ![sanju1323](https://avatars.discourse-cdn.com/v4/letter/s/5daacb/32.png) [@sanju1323](https://discuss.elastic.co/u/sanju1323)\
**Post date:** [July 15, 2015, 8:51am UTC](https://discuss.elastic.co/t/replace-the-indexed-data-with-new-data/25442/7 "2015-07-15T08:51:26Z")

</div>

@magnusbaeck

I have an input file having 20 records which is aalready indexed. Now the 20 records in the file has changed. So, I want to replace all the records in that index.

Here is the config file that I'm using

input  
{  
file  
{  
path =\> "Q:/sanjay/sanjay-data.psv"  
type =\> "all"  
start\_position =\> "beginning"  
}  
}  
filter  
{

```
csv 
{
	columns =>["IPID","AdmissionNumber","PatientID","RegCode","FirstName","Middlename","LastName","FirstName2l","Middlename2l","LastName2l","PatientName","PatientName2l","Age","AgeUoM","AgeUoM2l","FullAge","FullAge2l","Gender","Gender2L","BedID","BedName","BedName2l","BedTypeId","BedType","Room","WardID","Ward","Ward2l","Status","AdmitDate","AgeUoMID","ConsultantID","Consultant","Consultant2l","GenderId","CompanyID","CompanyName","CompanyName2l","PatientType","TariffID","BillBedTypeID","ParentIPID","DOB","EpisodeID","DischargeDate","DischargeReason","DischargeReason2l","IsVIP","NameNoTitle","NameNoTitle2l","IsNewBorn","IsRefDocExternal","RefDocID","RefDoctorName","RefDoctorName2l","ExRefDocID","ExRefDoctorName","ExRefDoctorName2l","City","City2l","PhoneNo","Address","Address2l","HospitalID","SpecialiseID","Specialisation","Specialisation2L","LetterID","BillType","EligibleBedType","CityID","ExpiredDate","ENDDATE","Remarks","NationalityID","Clearence","ClearenceRemarks","TransferID","BLOCKED","GradeId","EmpNo","VisitID","VisitDate","VisitType","PassportNo","SSN","MrNo","WorkPermitID","AdmSourceID","AdmSourceName","RoomId","Title","DischargeReasonID","DischargeRemarks","CALAGE","CALUOMID","RefDocCode","ExRefDocCode","ConsultantCode","RefDocNo","ConsultantNo"]
    separator => "|"
}
grok    
{
    patterns_dir => "Q:/softwares/ElasticSearch/logstash-1.3.3-flatjar/patterns"
   
     match => ["AdmitDate", "%{YEAR:al_year}-%{MONTHNUM:al_month}-%{MONTHDAY:al_monthday} %{TIME:al_time}"]
    add_field => ["LogTime", "%{al_year}-%{al_month}-%{al_monthday} %{al_time}"]
}
date 
{
    match => ["LogTime", "YYYY-MM-dd HH:mm:ss.SSS"]
}
mutate 
{
	convert => ["PatientID", "integer"]
	convert => ["Age", "integer"]
}

```

}

output  
{  
elasticsearch\_http  
{  
host =\> "localhost"  
index =\> "sanjay\_data"  
index\_type =\> "sanjay\_data"  
document\_id =\> "%{[@metadata][\_id]}"  
template =\> "Q:/softwares/ElasticSearch/logstash-1.3.3-flatjar/elasticsearch-template-hcg-sanjay-data.json"  
template\_name =\> "sanjay\_data"  
}  
stdout  
{  
codec =\> "json"  
debug =\> true  
}

}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 15, 2015, 9:47am UTC](https://discuss.elastic.co/t/replace-the-indexed-data-with-new-data/25442/8 "2015-07-15T09:47:27Z")

</div>

Okay. The value you assign to `document_id` should be a value that's unique to that log entry. You're currently using `[@metadata][_id]` which doesn't work since there's no such field. Perhaps the admission number would be more appropriate? Or that patient id? Whatever is the primary key of each entry will do.

---

<div class="post-metadata">

**Author:** ![sanju1323](https://avatars.discourse-cdn.com/v4/letter/s/5daacb/32.png) [@sanju1323](https://discuss.elastic.co/u/sanju1323)\
**Post date:** [July 15, 2015, 11:22am UTC](https://discuss.elastic.co/t/replace-the-indexed-data-with-new-data/25442/9 "2015-07-15T11:22:25Z")

</div>

Is it like document\_id =\> "PatientID" or any other format?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 15, 2015, 11:29am UTC](https://discuss.elastic.co/t/replace-the-indexed-data-with-new-data/25442/10 "2015-07-15T11:29:00Z")

</div>

```
document_id => "%{name-of-field}"

```

See the [documentation](https://www.elastic.co/guide/en/logstash/current/configuration.html#logstash-config-field-references).

---

<div class="post-metadata">

**Author:** ![sanju1323](https://avatars.discourse-cdn.com/v4/letter/s/5daacb/32.png) [@sanju1323](https://discuss.elastic.co/u/sanju1323)\
**Post date:** [July 15, 2015, 11:50am UTC](https://discuss.elastic.co/t/replace-the-indexed-data-with-new-data/25442/11 "2015-07-15T11:50:29Z")

</div>

Thanks @magnusbaeck it worked 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:34am UTC](https://discuss.elastic.co/t/replace-the-indexed-data-with-new-data/25442/12 "2017-07-06T05:34:39Z")

</div>