# Filter csv Logstash

**URL:** <https://discuss.elastic.co/t/filter-csv-logstash/174124>\
**Category:** Logstash\
**Created:** [March 27, 2019, 12:41pm UTC](https://discuss.elastic.co/t/filter-csv-logstash/174124 "2019-03-27T12:41:57Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Lebedev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lebedev/32/53579_2.png) [@Lebedev](https://discuss.elastic.co/u/Lebedev)\
**Post date:** [March 27, 2019, 12:41pm UTC](https://discuss.elastic.co/t/filter-csv-logstash/174124/1 "2019-03-27T12:41:57Z")

</div>

Hi, everyone.  
I am have same problem with add csv file to elastic. Could you kindly help me understand what is wrong.  
My logstash config:

input {

file {  
path =\> "C:\Saratov\_report\ \*.csv"  
start\_position =\> "beginning"  
sincedb\_path =\> "nul"  
tags =\> ["report"]  
}  
}

filter {

if "report" in [tags] {  
csv {  
separator =\> ";"  
columns =\> ["id", "availability", "address\_of\_the\_object", "start\_accident", "end\_accident", "reason\_of\_service\_affect", "alarm description", "responsibility", "provider\_name", "TT", "classificator", "duration(Seconds)"]  
}   
}  
}

output {  
if "report" in [tags] {

elasticsearch {  
hosts =\> "localhost:9200"  
index =\> "report-%{+YYYY.MM.dd}"

```
    }
  }

```

}

Well in Kibana i dont see index "report" /

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [March 27, 2019, 12:44pm UTC](https://discuss.elastic.co/t/filter-csv-logstash/174124/2 "2019-03-27T12:44:10Z")

</div>

Exactly what problem are you having?

---

<div class="post-metadata">

**Author:** ![Lebedev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lebedev/32/53579_2.png) [@Lebedev](https://discuss.elastic.co/u/Lebedev)\
**Post date:** [March 27, 2019, 12:44pm UTC](https://discuss.elastic.co/t/filter-csv-logstash/174124/3 "2019-03-27T12:44:24Z")

</div>

> [@Lebedev](#):
>
> Well in Kibana i dont see index "report" /

Well in Kibana i dont see index "report" /

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 27, 2019, 12:53pm UTC](https://discuss.elastic.co/t/filter-csv-logstash/174124/4 "2019-03-27T12:53:26Z")

</div>

> [@Lebedev](#):
>
> path =\> "C:\Saratov\_report\ \*.csv"

Use forward slash, not backslash, in Windows file paths.

---

<div class="post-metadata">

**Author:** ![Lebedev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lebedev/32/53579_2.png) [@Lebedev](https://discuss.elastic.co/u/Lebedev)\
**Post date:** [March 27, 2019, 12:55pm UTC](https://discuss.elastic.co/t/filter-csv-logstash/174124/5 "2019-03-27T12:55:26Z")

</div>

One second, i try to check

---

<div class="post-metadata">

**Author:** ![Lebedev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lebedev/32/53579_2.png) [@Lebedev](https://discuss.elastic.co/u/Lebedev)\
**Post date:** [March 27, 2019, 1:02pm UTC](https://discuss.elastic.co/t/filter-csv-logstash/174124/6 "2019-03-27T13:02:15Z")

</div>

Great, i see the index "report". You was right! Thanks!  
But now i have parser error, maybe somebody know, how is logstash work with russian languge (in the file, a lot of russian words).

[logstash.filters.csv] Error parsing csv {:field=\>"message", :source=\>"1877;\xE4\xEE\xF1\xF2\xF3\xEF\xE5\xED \xEF\xEE \xF0\xE5\xE7\xE5\xF0\xE2\xF3;\"\xCE\xCE \"\"\xD0\xFF\xE7\xE0\xED\xFC \xB9 3\"\" \xE3. \xD0\xFF\xE7\xE0\xED\xFC, \xF3\xEB. \xCA\xEE\xEB\xFC\xF6\xEE\xE2\xE0, \xE4. 8\_\";29.08.2017 1:08;29.08.2017 1:10;\xD1\xE1\xEE\xE9 \xF3 \xEF\xF0\xEE\xE2\xE0\xE9\xE4\xE5\xF0\xE0. \xCF\xF0\xEE\xE1\xEB\xE5\xEC\xFB \xED\xE0 \xF1\xE5\xF2\xE8 \xCE\xCF\xCC.;\xCF\xF0\xEE\xE1\xEB\xE5\xEC\xE0 \xED\xE0 \xEC\xE0\xF0\xF8\xF0\xF3\xF2\xE8\xE7\xE0\xF2\xEE\xF0\xE5 \xEF\xF0\xEE\xE2\xE0\xE9\xE4\xE5\xF0\xE0.;\xC1\xE8\xEB\xE0\xE9\xED;\xC1\xE8\xEB\xE0\xE9\xED;4051413;2;120;;\r", :exception=\>#\<CSV::MalformedCSVError: Illegal quoting in line 1.\>}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 27, 2019, 1:12pm UTC](https://discuss.elastic.co/t/filter-csv-logstash/174124/7 "2019-03-27T13:12:06Z")

</div>

You probably need to add a codec to your file input and set the charset option. The default codec for the file input is [plain](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-plain.html), which is OK, you just need to tell it what encoding to expect for your file.

---

<div class="post-metadata">

**Author:** ![Lebedev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lebedev/32/53579_2.png) [@Lebedev](https://discuss.elastic.co/u/Lebedev)\
**Post date:** [March 27, 2019, 1:18pm UTC](https://discuss.elastic.co/t/filter-csv-logstash/174124/8 "2019-03-27T13:18:23Z")

</div>

So if i correctly understad i need to add in area output iformation about codec, for example like this:

stdout { codec =\> rubydebug }

right?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 27, 2019, 1:32pm UTC](https://discuss.elastic.co/t/filter-csv-logstash/174124/9 "2019-03-27T13:32:37Z")

</div>

No, I am saying you need to replace

```auto
file {
path => "C:\Saratov_report\ *.csv"
start_position => "beginning"
sincedb_path => "nul"
tags => ["report"]
}

```

with

```auto
file {
path => "C:\Saratov_report\ *.csv"
start_position => "beginning"
sincedb_path => "nul"
tags => ["report"]
codec => plain { charset => "?" }
}

```

I do not know what value you should have for charset. The default is UTF-8 and your encoding is not UTF-8. "Windows-1251" perhaps?

That would make the first few characters горстпд. I cannot read Cyrllic, so I have no idea whether that makes any sense.

---

<div class="post-metadata">

**Author:** ![Lebedev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lebedev/32/53579_2.png) [@Lebedev](https://discuss.elastic.co/u/Lebedev)\
**Post date:** [March 27, 2019, 1:50pm UTC](https://discuss.elastic.co/t/filter-csv-logstash/174124/10 "2019-03-27T13:50:46Z")

</div>

Ou i understand, thank you for show me.

Information in the file looks like:  
1;полностью недоступен;Офис в г. Краснодар, ул. Северная, д. 288\_;20.11.2016 23:50;21.11.2016 0:30;Отсутствие электропитания на оборудовании банка.;"Оборудование Банка было обесточено.";БРС;БРС;3100;1;2400;;

2;полностью недоступен;Офис в г. Москва, ул. Фестивальная, д. 2А\_;21.11.2016 10:40;21.11.2016 11:55;Отсутствие электропитания на оборудовании банка.;электропитание отключено арендодателем.;БРС;БРС;3100;1;4500;;

The structure columns have the folowing structure:

"id",  
"availability",  
"address\_of\_the\_object",  
"start\_accident",  
"end\_accident",  
"reason\_of\_service\_affect",  
"alarm description",  
"responsibility",  
"provider\_name",  
"TT",  
"classificator",  
"duration(Seconds)"

I think you are right about windows 1251.  
Give me one moment i will check

---

<div class="post-metadata">

**Author:** ![Lebedev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lebedev/32/53579_2.png) [@Lebedev](https://discuss.elastic.co/u/Lebedev)\
**Post date:** [March 27, 2019, 2:10pm UTC](https://discuss.elastic.co/t/filter-csv-logstash/174124/11 "2019-03-27T14:10:50Z")

</div>

Windows - 1251 gave same results, but its not enought,

I cheched the information in es, and saw that all columns is empty.  
Is it mean that i need to add information about data type or no?

\_index": "report-2019.03.27",  
"\_type": "doc",  
"\_id": "G7F4v2kBy37sTD7xcEd6",  
"\_version": 1,  
"\_score": null,  
"\_source": {  
"path": "C:/Saratov\_report/List\_of\_Alarm.csv",  
"column2": null,  
"column4": null,  
"host": "srvocu01",  
"column12": null,  
"column1": null,  
"@version": "1",  
"column6": null,  
"column9": null,  
"message": ";;;;;;;;;;;;;\r",  
"column8": null,  
"column5": null,  
"column11": null,  
"@timestamp": "2019-03-27T14:05:34.820Z",  
"column10": null,  
"column3": null,  
"column13": null,  
"tags": [  
"report"  
],  
"column14": null,  
"column7": null  
},  
"fields": {  
"@timestamp": [  
"2019-03-27T14:05:34.820Z"

About codec i will try to find more information, now error looks like:

[WARN][logstash.filters.csv] Error parsing csv {:field=\>"message", :source=\>"";├Б├и├л├а├й├н;├Б├и├л├а├й├н;4942115;1;157;;\r", :exception=\>#\<CSV::MalformedCSVError: Unclosed quoted field on line 1.\>}  
[2019-03-27T17:05:31,054][WARN][logstash.filters.csv] Error parsing csv {:field=\>"message", :source=\>"5082;├д├о├▒├▓├│├п├е├н ├п├о ├░├е├з├е├░├в├│;├К├а├н├а├л ├в Call Center ├К├а├з├а├н├╝ - ├В├а├╡├и├▓├о├в├а, ├г. ├К├а├з├а├н├╝, ├│├л. ├В├а├╡├и├▓├о├в├а, ├д. 8\_(ID 4084089);03.12.2018 3:16;03.12.2018 3:27;├С├б├о├й ├│ ├п├░├о├в├а├й├д├е├░├а. ├П├░├о├б├л├е├м├╗ ├н├а ├▒├е├▓├и ├О├П├М.;"├А├в├а├░├и├┐ ├н├а ├▒├е├▓├и ├п├░├о├в├а├й├д├е├░├а тАЬ├Б├и├л├а├й├нтАЭ. ", :exception=\>#\<CSV::MalformedCSVError: Unclosed quoted field on line 1.\>}

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 24, 2019, 2:11pm UTC](https://discuss.elastic.co/t/filter-csv-logstash/174124/12 "2019-04-24T14:11:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.