# Unable to load CSV file into Elasticsearch

**URL:** <https://discuss.elastic.co/t/unable-to-load-csv-file-into-elasticsearch/198826>\
**Category:** Logstash\
**Created:** [September 10, 2019, 4:36am UTC](https://discuss.elastic.co/t/unable-to-load-csv-file-into-elasticsearch/198826 "2019-09-10T04:36:15Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ramya\_Tanikanti](https://avatars.discourse-cdn.com/v4/letter/r/9de053/32.png) [@Ramya\_Tanikanti](https://discuss.elastic.co/u/Ramya_Tanikanti)\
**Post date:** [September 10, 2019, 4:36am UTC](https://discuss.elastic.co/t/unable-to-load-csv-file-into-elasticsearch/198826/1 "2019-09-10T04:36:16Z")

</div>

Hi,

Logstash uploads the data into Kibana once and sometimes it doesn't even though I use the same same config file.

I use the same config file to load the same data again after deleting the index it created in Kibana only. I make some small changes and upload it again. When I do this I get the following output.

C:\Users\ramya.t\logstash-7.3.0\bin\>logstash -f C:\Users\ramya.t\Downloads\Colt\retail.conf  
Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.  
WARNING: An illegal reflective access operation has occurred  
WARNING: Illegal reflective access by org.jruby.runtime.encoding.EncodingService (file:/C:/Users/ramya.t/logstash-7.3.0/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar) to field java.io.Console.cs  
WARNING: Please consider reporting this to the maintainers of org.jruby.runtime.encoding.EncodingService  
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations  
WARNING: All illegal access operations will be denied in a future release  
Thread.exclusive is deprecated, use Thread::Mutex  
Sending Logstash logs to C:/Users/ramya.t/logstash-7.3.0/logs which is now configured via log4j2.properties  
[2019-09-10T09:59:15,782][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified  
[2019-09-10T09:59:15,792][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"7.3.0"}  
[2019-09-10T09:59:17,980][INFO][org.reflections.Reflections] Reflections took 29 ms to scan 1 urls, producing 19 keys and 39 values  
[2019-09-10T09:59:19,347][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=\>{:removed=\>, :added=\>[[http://localhost:9200/](http://localhost:9200/)]}}  
[2019-09-10T09:59:19,481][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>"[http://localhost:9200/](http://localhost:9200/)"}  
[2019-09-10T09:59:19,516][INFO][logstash.outputs.elasticsearch] ES Output version determined {:es\_version=\>7}  
[2019-09-10T09:59:19,519][WARN][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document \_type {:es\_version=\>7}  
[2019-09-10T09:59:19,538][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>["[//localhost](https://localhost)"]}  
[2019-09-10T09:59:19,589][INFO][logstash.outputs.elasticsearch] Using default mapping template  
[2019-09-10T09:59:19,631][INFO][logstash.outputs.elasticsearch] Attempting to install template {:manage\_template=\>{"index\_patterns"=\>"logstash-_", "version"=\>60001, "settings"=\>{"index.refresh\_interval"=\>"5s", "number\_of\_shards"=\>1}, "mappings"=\>{"dynamic\_templates"=\>[{"message\_field"=\>{"path\_match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false}}}, {"string\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false, "fields"=\>{"keyword"=\>{"type"=\>"keyword", "ignore\_above"=\>256}}}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date"}, "@version"=\>{"type"=\>"keyword"}, "geoip"=\>{"dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip"}, "location"=\>{"type"=\>"geo\_point"}, "latitude"=\>{"type"=\>"half\_float"}, "longitude"=\>{"type"=\>"half\_float"}}}}}}}  
[2019-09-10T09:59:19,636][WARN][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been create for key: cluster\_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.  
[2019-09-10T09:59:19,639][INFO][logstash.javapipeline] Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>4, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50, "pipeline.max\_inflight"=\>500, :thread=\>"#\<Thread:0x501ad2c2 run\>"}  
[2019-09-10T09:59:20,228][INFO][logstash.inputs.file] No sincedb\_path set, generating one based on the "path" setting {:sincedb\_path=\>"C:/Users/ramya.t/logstash-7.3.0/data/plugins/inputs/file/.sincedb\_8b2a1e7d1249525b6e487bf33b908668", :path=\>["C:/Users/ramya.t/Downloads/Colt/ELK/CC\_FINAL\_Master.csv"]}  
[2019-09-10T09:59:20,247][INFO][logstash.javapipeline] Pipeline started {"pipeline.id"=\>"main"}  
[2019-09-10T09:59:20,307][INFO][logstash.agent] Pipelines running {:count=\>1, :running\_pipelines=\>[:main], :non\_running\_pipelines=\>}  
[2019-09-10T09:59:20,312][INFO][filewatch.observingtail] START, creating Discoverer, Watch with file and sincedb collections  
[2019-09-10T09:59:20,881][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600}

The index does not created after this, and logstash does not read the data after this.

Kindly help!!

---

<div class="post-metadata">

**Author:** ![Ramya\_Tanikanti](https://avatars.discourse-cdn.com/v4/letter/r/9de053/32.png) [@Ramya\_Tanikanti](https://discuss.elastic.co/u/Ramya_Tanikanti)\
**Post date:** [September 10, 2019, 4:37am UTC](https://discuss.elastic.co/t/unable-to-load-csv-file-into-elasticsearch/198826/2 "2019-09-10T04:37:00Z")

</div>

**My Config file is,**

**My congfig file is:**

input {  
file {  
path =\> "C:/Users/ramya.t/Downloads/Colt/ELK/CC\_FINAL\_Master.csv"  
start\_position =\> "beginning"  
}  
}

filter {  
csv {  
separator =\> ","  
columns =\> [ "USERNAME", "SEQUENCEID", "REQUEST\_TIMESTAMP", "A\_FLOORSUITE",  
"A\_BUILDINGNAME", "A\_PREMISESNUMBER", "A\_STREETNAME", "A\_CITYTOWN",  
"A\_STATE", "A\_POSTALZIPCODE", "A\_LATITUDE", "A\_LONGITUDE",  
"A\_LONGADDRESS", "A\_SITETELEPHONENUMBER", "A\_RADIUS", "A\_ISHUB",  
"A\_COLTOPERATINGCOUNTRY", "A\_REQUIREDPRODUCT", "A\_BANDWIDTH",  
"A\_CONNECTIVITYTYPE1", "A\_CONNECTIVITYTYPE2", "A\_CONNECTIVITYTYPE3",  
"A\_CONNECTIVITYTYPE4", "B\_FLOORSUITE", "B\_BUILDINGNAME",  
"B\_PREMISESNUMBER", "B\_STREETNAME", "B\_CITYTOWN", "B\_STATE",  
"B\_POSTALZIPCODE", "B\_LATITUDE", "B\_LONGITUDE", "B\_LONGADDRESS",  
"B\_SITETELEPHONENUMBER", "B\_RADIUS", "B\_ISHUB",  
"B\_COLTOPERATINGCOUNTRY", "B\_REQUIREDPRODUCT", "B\_BANDWIDTH",  
"B\_CONNECTIVITYTYPE1", "B\_CONNECTIVITYTYPE2", "B\_CONNECTIVITYTYPE3",  
"B\_CONNECTIVITYTYPE4", "SCHEMAVERSION", "REQUESTTYPE", "RESPONSE",  
"RESP\_ERRORTYPE", "RESP\_ERRORCODE", "STATUS", "ONNET\_STATUS",  
"ONNETAEND\_STATUS", "ONNETBEND\_STATUS", "OFFNET\_STATUS",  
"OFFNETOPTION\_STATUS", "OFFNETOPTION\_AEND\_STATUS",  
"OFFNETOPTION\_BEND\_STATUS", "OLO\_STATUS", "OLOAEND\_STATUS",  
"OLOBEND\_STATUS", "OLOOPTION\_STATUS", "OLOOPTAENDRES\_STATUS",  
"OLOOPTBENDRES\_STATUS", "NEARNETSTATUS\_STATUS",  
"NEARNETSTATUS\_AENDRESULT\_STATUS", "NEARNETSTATUS\_BENDRESULT\_STATUS","is\_A\_country\_colt", "is\_A\_city\_colt", "is\_B\_country\_colt", "is\_B\_city\_colt"]  
}

```
   if [SCHEMAVERSION] == "5.0" {
      mutate {
               replace => ["SCHEMAVERSION", "5"]
                }
            }
    if [SCHEMAVERSION] == "4.0" {
      mutate {
               replace => ["SCHEMAVERSION", "4"]
                }
            }
     if [SCHEMAVERSION] == "3.0" {
      mutate {
               replace => ["SCHEMAVERSION", "3"]
                }
            }
       if [SCHEMAVERSION] == "2.0" {
      mutate {
               replace => ["SCHEMAVERSION", "2"]
                }
            }
   
    if [SCHEMAVERSION] == "6.0" {
      mutate {
               replace => ["SCHEMAVERSION", "6"]
                }
            }

    if [SCHEMAVERSION] == "1.0" {
      mutate {
               replace => ["SCHEMAVERSION", "1"]
                }
            }

    if [A_COLTOPERATINGCOUNTRY] == "" {
      mutate {
               replace => ["is_A_country_colt", ""]
                }
            }
    if [A_CITYTOWN] == "" {
      mutate {
               replace => ["is_A_city_colt", ""]
                }
            }
    if [B_COLTOPERATINGCOUNTRY] == "" {
      mutate {
               replace => ["is_B_country_colt", ""]
                }
            }
    if [B_CITYTOWN] == "" {
      mutate {
               replace => ["is_B_city_colt", ""]
                }
            }

   date {
   match => ["REQUEST_TIMESTAMP", "yyyy-MM-dd HH:mm:ss"]
  }

```

}

output {  
elasticsearch {  
hosts =\> "localhost"  
index =\> "ccfinal1"  
}  
stdout {}  
}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 10, 2019, 11:50am UTC](https://discuss.elastic.co/t/unable-to-load-csv-file-into-elasticsearch/198826/3 "2019-09-10T11:50:58Z")

</div>

What does retail.conf contain?

---

<div class="post-metadata">

**Author:** ![Ramya\_Tanikanti](https://avatars.discourse-cdn.com/v4/letter/r/9de053/32.png) [@Ramya\_Tanikanti](https://discuss.elastic.co/u/Ramya_Tanikanti)\
**Post date:** [September 10, 2019, 11:53am UTC](https://discuss.elastic.co/t/unable-to-load-csv-file-into-elasticsearch/198826/4 "2019-09-10T11:53:40Z")

</div>

input {  
file {  
path =\> "C:/Users/ramya.t/Downloads/Colt/CC\_FINAL\_Master.csv"  
start\_position =\> "beginning"  
}  
}

filter {  
csv {  
separator =\> ","  
columns =\> [ "USERNAME", "SEQUENCEID", "REQUEST\_TIMESTAMP", "A\_FLOORSUITE",  
"A\_BUILDINGNAME", "A\_PREMISESNUMBER", "A\_STREETNAME", "A\_CITYTOWN",  
"A\_STATE", "A\_POSTALZIPCODE", "A\_LATITUDE", "A\_LONGITUDE",  
"A\_LONGADDRESS", "A\_SITETELEPHONENUMBER", "A\_RADIUS", "A\_ISHUB",  
"A\_COLTOPERATINGCOUNTRY", "A\_REQUIREDPRODUCT", "A\_BANDWIDTH",  
"A\_CONNECTIVITYTYPE1", "A\_CONNECTIVITYTYPE2", "A\_CONNECTIVITYTYPE3",  
"A\_CONNECTIVITYTYPE4", "B\_FLOORSUITE", "B\_BUILDINGNAME",  
"B\_PREMISESNUMBER", "B\_STREETNAME", "B\_CITYTOWN", "B\_STATE",  
"B\_POSTALZIPCODE", "B\_LATITUDE", "B\_LONGITUDE", "B\_LONGADDRESS",  
"B\_SITETELEPHONENUMBER", "B\_RADIUS", "B\_ISHUB",  
"B\_COLTOPERATINGCOUNTRY", "B\_REQUIREDPRODUCT", "B\_BANDWIDTH",  
"B\_CONNECTIVITYTYPE1", "B\_CONNECTIVITYTYPE2", "B\_CONNECTIVITYTYPE3",  
"B\_CONNECTIVITYTYPE4", "SCHEMAVERSION", "REQUESTTYPE", "RESPONSE",  
"RESP\_ERRORTYPE", "RESP\_ERRORCODE", "STATUS", "ONNET\_STATUS",  
"ONNETAEND\_STATUS", "ONNETBEND\_STATUS", "OFFNET\_STATUS",  
"OFFNETOPTION\_STATUS", "OFFNETOPTION\_AEND\_STATUS",  
"OFFNETOPTION\_BEND\_STATUS", "OLO\_STATUS", "OLOAEND\_STATUS",  
"OLOBEND\_STATUS", "OLOOPTION\_STATUS", "OLOOPTAENDRES\_STATUS",  
"OLOOPTBENDRES\_STATUS", "NEARNETSTATUS\_STATUS",  
"NEARNETSTATUS\_AENDRESULT\_STATUS", "NEARNETSTATUS\_BENDRESULT\_STATUS","is\_A\_country\_colt", "is\_A\_city\_colt", "is\_B\_country\_colt", "is\_B\_city\_colt"]  
}

```
   if [SCHEMAVERSION] == "5.0" {
      mutate {
               replace => ["SCHEMAVERSION", "5"]
                }
            }
    if [SCHEMAVERSION] == "4.0" {
      mutate {
               replace => ["SCHEMAVERSION", "4"]
                }
            }
     if [SCHEMAVERSION] == "3.0" {
      mutate {
               replace => ["SCHEMAVERSION", "3"]
                }
            }
       if [SCHEMAVERSION] == "2.0" {
      mutate {
               replace => ["SCHEMAVERSION", "2"]
                }
            }
   
    if [SCHEMAVERSION] == "6.0" {
      mutate {
               replace => ["SCHEMAVERSION", "6"]
                }
            }

    if [SCHEMAVERSION] == "1.0" {
      mutate {
               replace => ["SCHEMAVERSION", "1"]
                }
            }

    if [A_COLTOPERATINGCOUNTRY] == "" {
      mutate {
               replace => ["is_A_country_colt", ""]
                }
            }
    if [A_CITYTOWN] == "" {
      mutate {
               replace => ["is_A_city_colt", ""]
                }
            }
    if [B_COLTOPERATINGCOUNTRY] == "" {
      mutate {
               replace => ["is_B_country_colt", ""]
                }
            }
    if [B_CITYTOWN] == "" {
      mutate {
               replace => ["is_B_city_colt", ""]
                }
            }

   date {
   match => ["REQUEST_TIMESTAMP", "yyyy-MM-dd HH:mm:ss"]
  }

```

}

output {  
elasticsearch {  
hosts =\> "localhost"  
index =\> "ccfinal1"  
}  
stdout {}  
}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 10, 2019, 12:50pm UTC](https://discuss.elastic.co/t/unable-to-load-csv-file-into-elasticsearch/198826/5 "2019-09-10T12:50:40Z")

</div>

> [@Ramya\_Tanikanti](#):
>
> Logstash uploads the data into Kibana once

That is the expected behaviour of a file input. It [tracks](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#_tracking_of_current_position_in_watched_files) how much of the file it has read in the sincedb. The start\_position option only affects what it reads the first time it sees a file. It has no effect after a restart.

It is possible you want to use

```
sincedb_path => "NUL"

```

---

<div class="post-metadata">

**Author:** ![Ramya\_Tanikanti](https://avatars.discourse-cdn.com/v4/letter/r/9de053/32.png) [@Ramya\_Tanikanti](https://discuss.elastic.co/u/Ramya_Tanikanti)\
**Post date:** [September 11, 2019, 4:10am UTC](https://discuss.elastic.co/t/unable-to-load-csv-file-into-elasticsearch/198826/6 "2019-09-11T04:10:53Z")

</div>

Thank you @Badger!! It got in now : )

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 9, 2019, 4:10am UTC](https://discuss.elastic.co/t/unable-to-load-csv-file-into-elasticsearch/198826/7 "2019-10-09T04:10:53Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.