# CSV data to Elasticsearch using Logstash

**URL:** <https://discuss.elastic.co/t/csv-data-to-elasticsearch-using-logstash/192055>\
**Category:** Logstash\
**Created:** [July 24, 2019, 2:27pm UTC](https://discuss.elastic.co/t/csv-data-to-elasticsearch-using-logstash/192055 "2019-07-24T14:27:42Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Fragan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fragan/32/45475_2.png) [@Fragan](https://discuss.elastic.co/u/Fragan)\
**Post date:** [July 24, 2019, 2:27pm UTC](https://discuss.elastic.co/t/csv-data-to-elasticsearch-using-logstash/192055/1 "2019-07-24T14:27:42Z")

</div>

Hello, im trying to push csv data to Elasticsearch using Logstash, here's my logstash conf file :

```
input{
	file{
		path => "C:\Users\zk3i\Desktop\ELK\kpassdemo_vue_qsse.CSV"
		start_position => "beginning"
		sincedb_path => "C:\Users\zk3i\Desktop\ELK\null.txt"
	}
}

filter{
	csv{
		separator => ","
		columns => ["Nom","Etape","Etape [id<:ver>]","Nature évènement","Nature évènement [id<:ver>]","Lieu" ,"Lieu [id<:ver>]","Détecté par", "Détecté par [id<:ver>]","Date de détection"]
	}	
}

output{
	elasticsearch{
		hosts => "http://localhost:9200"
		index => "qsse"
	}
	
	stdout{ }
}

```

config file is located at `C:\Users\zk3i\Desktop\ELK\logstach_qsse.conf`  
Elasticsearch is running as a service here's what i get when i go to localhost:9200  
{  
"name" : "elk\_local",  
"cluster\_name" : "elasticsearch",  
"cluster\_uuid" : "zzjDN3PKQOyvBcpi9\_I6Ng",  
"version" : {  
"number" : "7.2.0",  
"build\_flavor" : "unknown",  
"build\_type" : "unknown",  
"build\_hash" : "508c38a",  
"build\_date" : "2019-06-20T15:54:18.811730Z",  
"build\_snapshot" : false,  
"lucene\_version" : "8.0.0",  
"minimum\_wire\_compatibility\_version" : "6.8.0",  
"minimum\_index\_compatibility\_version" : "6.0.0-beta1"  
},  
"tagline" : "You Know, for Search"  
}  
I dont know what im doing wrong, im really new to the Elastic Stack.  
When i run `./bin/logstash -f C:\Users\zk3i\Desktop\ELK\logstach_qsse.conf` i got a tons of errors, you can find the error log here [https://pastebin.com/n7dMiiiF](https://pastebin.com/n7dMiiiF)

I couldn't paste it here because its limited to 7000 chars 😕

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 24, 2019, 2:46pm UTC](https://discuss.elastic.co/t/csv-data-to-elasticsearch-using-logstash/192055/2 "2019-07-24T14:46:20Z")

</div>

There are two basic errors here. The first is

```
java.io.IOException: Could not create directory C:\Program Files\Elastic\Logstach\7.2.0\logs

```

That in turn results in hundreds of lines of error from log4j RollingFileAppender as it cannot create log files in that directory. I suggest you create that directory and make sure it is writeable by the user that logstash is running as.

The next error is

```
[2019-07-24T16:15:50,074][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.

```

What is path.config set to?

Thirdly, you cannot use backslash in the path option of a file input. Change them to forward slash.

---

<div class="post-metadata">

**Author:** ![Fragan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fragan/32/45475_2.png) [@Fragan](https://discuss.elastic.co/u/Fragan)\
**Post date:** [July 24, 2019, 2:53pm UTC](https://discuss.elastic.co/t/csv-data-to-elasticsearch-using-logstash/192055/3 "2019-07-24T14:53:00Z")

</div>

Thanks for your answer , i created the logs folder and i still have that issue, path.config is set to `C:\Users\zk3i\Desktop\ELK\logstach_qsse.conf`

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 24, 2019, 3:00pm UTC](https://discuss.elastic.co/t/csv-data-to-elasticsearch-using-logstash/192055/4 "2019-07-24T15:00:41Z")

</div>

Run with '--log.level debug'. Check the line

```
[2019-07-24T14:58:07,125][DEBUG][logstash.runner] *path.config: "/home/user/logstash.conf"

```

and make sure that path.config contains what you expect.

---

<div class="post-metadata">

**Author:** ![Fragan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fragan/32/45475_2.png) [@Fragan](https://discuss.elastic.co/u/Fragan)\
**Post date:** [July 24, 2019, 3:21pm UTC](https://discuss.elastic.co/t/csv-data-to-elasticsearch-using-logstash/192055/5 "2019-07-24T15:21:24Z")

</div>

The path.config is alright, and i still have this error `[2019-07-24T17:37:55,467][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.`

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 24, 2019, 3:53pm UTC](https://discuss.elastic.co/t/csv-data-to-elasticsearch-using-logstash/192055/6 "2019-07-24T15:53:09Z")

</div>

That error means the configuration is [empty](https://github.com/elastic/logstash/blob/3e3a061a5e1724087961e746b4dde8e6ecee40eb/logstash-core/lib/logstash/config/source_loader.rb#L77). I cannot explain that.

---

<div class="post-metadata">

**Author:** ![Fragan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fragan/32/45475_2.png) [@Fragan](https://discuss.elastic.co/u/Fragan)\
**Post date:** [July 25, 2019, 8:03am UTC](https://discuss.elastic.co/t/csv-data-to-elasticsearch-using-logstash/192055/7 "2019-07-25T08:03:48Z")

</div>

I fixed it, i noticed that it was using the GIT folder as the home folder because im using git terminal x), now im having another error  
`[2019-07-25T10:00:17,757][DEBUG][logstash.instrument.periodicpoller.cgroup] One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu`  
`[2019-07-25T10:00:22,817][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}`  
`[2019-07-25T10:00:22,817][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}`  
`[2019-07-25T10:00:23,484][DEBUG][org.logstash.execution.PeriodicFlush] Pushing flush onto pipeline.`  
`[2019-07-25T10:00:27,765][DEBUG][logstash.instrument.periodicpoller.cgroup] One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu`

The index is created but its health isnt `Green` its `Yellow`

And im also having some CSVParseFailure

```
[2019-07-25T10:57:09,113][DEBUG][logstash.filters.csv] Running csv filter {:event=>#<LogStash::Event:0x5b9ae3f5>}
[2019-07-25T10:57:09,113][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"EVT-QSSE-2018-0034,4. Clôturé,116,Qualité,1,\"\",\"\",Nicolas Duval,1,\"\"\r", :exception=>#<RuntimeError: Invalid FieldReference: `[Etape [id<:ver>]]`>}
[2019-07-25T10:57:09,114][DEBUG][logstash.filters.csv] Running csv filter {:event=>#<LogStash::Event:0x78b13507>}
[2019-07-25T10:57:09,115][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"EVT-QSSE-2019-0004,2. En traitement,114,Santé,3,Atelier,1,Nicolas Duval,1,\"\"\r", :exception=>#<RuntimeError: Invalid FieldReference: `[Etape [id<:ver>]]`>}
[2019-07-25T10:57:09,116][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"EVT-QSSE-2018-0026,2. En traitement,114,Sécurité,2,Atelier,1,Sébastien Lopez,2,24/06/2018\r", :exception=>#<RuntimeError: Invalid FieldReference: `[Etape [id<:ver>]]`>}
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 25, 2019, 12:52pm UTC](https://discuss.elastic.co/t/csv-data-to-elasticsearch-using-logstash/192055/8 "2019-07-25T12:52:38Z")

</div>

Those DEBUG messages are normal.

If you have a single node your index health will normally be yellow because there is nowhere to assign the replica shards to.

Remove the square brackets from your column names.

---

<div class="post-metadata">

**Author:** ![Fragan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fragan/32/45475_2.png) [@Fragan](https://discuss.elastic.co/u/Fragan)\
**Post date:** [July 25, 2019, 1:59pm UTC](https://discuss.elastic.co/t/csv-data-to-elasticsearch-using-logstash/192055/9 "2019-07-25T13:59:27Z")

</div>

Working perfectly thanks, can you please suggest me a good website or youtube channel having Elasticsearch courses ?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 22, 2019, 1:59pm UTC](https://discuss.elastic.co/t/csv-data-to-elasticsearch-using-logstash/192055/10 "2019-08-22T13:59:29Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.