# Importing file to an existing index

**URL:** <https://discuss.elastic.co/t/importing-file-to-an-existing-index/293932>\
**Category:** Logstash\
**Created:** [January 10, 2022, 6:25pm UTC](https://discuss.elastic.co/t/importing-file-to-an-existing-index/293932 "2022-01-10T18:25:32Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![kibanauser4](https://avatars.discourse-cdn.com/v4/letter/k/ecd19e/32.png) [@kibanauser4](https://discuss.elastic.co/u/kibanauser4)\
**Post date:** [January 10, 2022, 6:25pm UTC](https://discuss.elastic.co/t/importing-file-to-an-existing-index/293932/1 "2022-01-10T18:25:32Z")

</div>

I wanted to update my index by importing a file via Logstash. I used the same logstash file (the input file was updated with new records) cause I thought it's gonna overwrite the existing index. I tested this on a different index and everything worked as far as I could've seen. But now, somehow I got more documents than the count of records in the input file. But at the same time, it wasn't duplicated cause the difference in count is not that big.  
Do you have any idea what could be the issue? Thanks in advance.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [January 11, 2022, 12:25am UTC](https://discuss.elastic.co/t/importing-file-to-an-existing-index/293932/2 "2022-01-11T00:25:23Z")

</div>

> [@kibanauser4](#):
>
> cause I thought it's gonna overwrite the existing index

That might happen, but it's not a guarantee.

You'd need to share your config for us to comment more.

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [January 11, 2022, 12:38am UTC](https://discuss.elastic.co/t/importing-file-to-an-existing-index/293932/3 "2022-01-11T00:38:17Z")

</div>

that happens because there is \_id for each record. and that \_id is uniq.  
if you have not define it then ELK will create automatically

i.e next time if you import same record again it will be dulicated with different \_id.  
if you try third time your record count will increase again.

to avoid this on logstash output section you will have to define uniq document\_id

you will have to create it , the way you can do is by combine multiple field and create uniq id. it will depend on your input.

---

<div class="post-metadata">

**Author:** ![kibanauser4](https://avatars.discourse-cdn.com/v4/letter/k/ecd19e/32.png) [@kibanauser4](https://discuss.elastic.co/u/kibanauser4)\
**Post date:** [January 11, 2022, 9:10am UTC](https://discuss.elastic.co/t/importing-file-to-an-existing-index/293932/4 "2022-01-11T09:10:08Z")

</div>

I think I already am defining an unique document\_id in the config file. Here's how my config looks:

```auto
input {
	file {
		path => "input_sample.csv"
		start_position => "beginning"
		sincedb_path => "NUL"
	}
}

filter {
	csv {
		separator => ","
		autodetect_column_names => true
	}
	ruby {
	    code => "wanted_columns = ['License Plate','Brand','Expiry Date','Catalogue Price']
	    event.to_hash.keys.each { |k| event.remove(k) unless wanted_columns.include? k }"
	}
	mutate {
		rename => {
			"License Plate" => "licensePlate"
			"Brand" => "brand"
			"Expiry Date" => "expiryDate"
			"Catalogue Price" => "cataloguePrice"
		}
	}
	date {
		match => ["expiryDate", "yyyyMMdd"]
		target => "expiryDate"
		timezone => "UTC"
	}
}

output {
	elasticsearch {
		hosts => ["http://localhost:9200"]
		index => "car"
		document_id => "%{licensePlate}"
	}
}

```

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [January 11, 2022, 3:11pm UTC](https://discuss.elastic.co/t/importing-file-to-an-existing-index/293932/5 "2022-01-11T15:11:09Z")

</div>

by this config it will not create new entry but it will not update either.

for example licensePlate=ABC123.

in first run you had already created record.  
in second pass when it try to create this record with updated value it sees that \_id already exist and it won't do anything. if you want to update you have to use action =\> "update" in output section.

now to the original problem. you will have to find out which record are duplicated/new/unwanted and find out why they are there.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 8, 2022, 3:11pm UTC](https://discuss.elastic.co/t/importing-file-to-an-existing-index/293932/6 "2022-02-08T15:11:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.