# Missing CSV records in ElasticSearch

**URL:** <https://discuss.elastic.co/t/missing-csv-records-in-elasticsearch/102497>\
**Category:** Logstash\
**Created:** [October 3, 2017, 12:44am UTC](https://discuss.elastic.co/t/missing-csv-records-in-elasticsearch/102497 "2017-10-03T00:44:40Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![xnaud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xnaud/32/59294_2.png) [@xnaud](https://discuss.elastic.co/u/xnaud)\
**Post date:** [October 3, 2017, 12:44am UTC](https://discuss.elastic.co/t/missing-csv-records-in-elasticsearch/102497/1 "2017-10-03T00:44:40Z")

</div>

I am importing into Amazon Elasticsearch with logstash 5.6.2 a 120MB CSV file that contains 27,858 US cities with their polygons.  
I run logstash with stdin and not with a file so it stops at the end.  
when importing, I only have around 26,400 cities. The rest is MIA and no error in the output or any logs.

Here are some facts:

- The file has one geo json field for the geo shapes and all are well formed
- All document ids are uniq (and I get no deleted docs in the index).

I tried to use a persistent queue and set drain=true in the setting file but it does not change anything.  
The process ends without error but I am still missing some records.

The settings file:  
queue.type: persisted  
queue.drain: true

the configuration file:

> ```
> input {
> stdin {
> }
> }
> filter {
> csv {
> separator => "|"
> quote_char => "@"
> columns => ["gid", "community_id", "hj_id", "name", "state", "lon", "lat", "geo"]
> add_field => {
> "location" => "%{lat},%{lon}"
> "suggest" => "%{name}, %{state}"
> }
> remove_field => ["lat", "lon", "host", "message", "@version"]
> }
> mutate {
> convert => { "gid" => "integer" }
> convert => { "community_id" => "integer" }
> }
> json {
> source => "geo"
> target => "geo"
> }
> }
> output {
> stdout { codec => rubydebug }
> amazon_es {
> hosts => ["XXXX.es.amazonaws.com"]
> region => "XXX"
> aws_access_key_id => 'XXXX'
> aws_secret_access_key => 'XXXX'
> index => "cities-%{+YYYYMMdd}"
> template => "./cities-template.json"
> template_name => "cities"
> document_type => "city"
> document_id=>"%{hj_id}"
> }
> }
> 
> ```

Where shall I look at next?  
How can I check what is in the persistent queue?  
Do I need to restart logstash in a certain way to process records from the queue?

Thanks

Xavier

---

<div class="post-metadata">

**Author:** ![xnaud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xnaud/32/59294_2.png) [@xnaud](https://discuss.elastic.co/u/xnaud)\
**Post date:** [October 3, 2017, 11:35pm UTC](https://discuss.elastic.co/t/missing-csv-records-in-elasticsearch/102497/2 "2017-10-03T23:35:27Z")

</div>

I isolated some of the records that don't show up and tried to index them manually.  
Only then can I see the error:

> "caused\_by": {  
> "type": "invalid\_shape\_exception",  
> "reason": "Self-intersection at or near point (-75.21756305418718, 38.757002463054185, NaN)"  
> }

so the polygon is not valid.

Is there a way to get this error in some log file so I can get a list of invalid polygons?  
do I need to enable some log4j logger?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [October 6, 2017, 5:40am UTC](https://discuss.elastic.co/t/missing-csv-records-in-elasticsearch/102497/3 "2017-10-06T05:40:23Z")

</div>

Elasticsearch should respond with an error back to the client, is that happening?

---

<div class="post-metadata">

**Author:** ![xnaud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xnaud/32/59294_2.png) [@xnaud](https://discuss.elastic.co/u/xnaud)\
**Post date:** [October 6, 2017, 1:44pm UTC](https://discuss.elastic.co/t/missing-csv-records-in-elasticsearch/102497/4 "2017-10-06T13:44:15Z")

</div>

ElasticSearch probably respond with an error, at least when I send the request manually but logstash does not report anything.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 3, 2017, 1:44pm UTC](https://discuss.elastic.co/t/missing-csv-records-in-elasticsearch/102497/5 "2017-11-03T13:44:24Z")

</div>

Closed on November 3, 2017, 1:44pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.