# Unable to index csv data

**URL:** <https://discuss.elastic.co/t/unable-to-index-csv-data/156442>\
**Category:** Logstash\
**Created:** [November 13, 2018, 10:24am UTC](https://discuss.elastic.co/t/unable-to-index-csv-data/156442 "2018-11-13T10:24:05Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Gauti](https://avatars.discourse-cdn.com/v4/letter/g/cdc98d/32.png) [@Gauti](https://discuss.elastic.co/u/Gauti)\
**Post date:** [November 13, 2018, 10:24am UTC](https://discuss.elastic.co/t/unable-to-index-csv-data/156442/1 "2018-11-13T10:24:05Z")

</div>

Hi All,

I have been trying to upload csv data using logstash, but was unable to index the data and getting error like characterset miss match,

I was able to identify these warnings are occurring only for the field values where empty spaces are there (for example if there is a ticket is not resolved then those "Resolved\_Date" column will be empty)

Here is the error

```
[2018-11-13T14:13:25,886][WARN][logstash.codecs.plain] Received an event that has a different character encoding than you configured. {:text=>"INC0230207,UA-SAP-UK,10-25-18 15:38,SAP workflow in transaction ZMMBV duplicates purchaseorder workflow item,TCS-SAP-DC,Self-service,FALSE,Ren\\x82 Wijnenga,Low,,,Assigned,a306815,,,\\r", :expected_charset=>"UTF-8"}
[2018-11-13T14:13:26,062][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"INC0230914,UA-SAP-UK,10-26-18 14:39,SAP password reset / account unlock SAP-ID: B015060,SD-INCIDENTS,Email,FALSE,Simon Edwards,Support,10-26-18 14:42,,Resolved,a323921,,Password reset / Unlocked Account,\"Password reset done.", :exception=>#<CSV::MalformedCSVError: Unclosed quoted field on line 1.>}

```

Here is my config file:

```
input {
  file {
    path => "/opt/installables/csv/Book1.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null"
 }
}
filter {
  csv {
      separator => ","
      columns => ["Number","Configuration_item","Opened","Short_description","Assignment_group","Contact_type","Major_incident","Caller","Priority","Resolved","Closed","Status","Updated_by","Closed_by","Category","Close_notes"]
  }
  }
output {
# elasticsearch {
# hosts => "1.12.1.3:9200"
# index => "incanalysis"
# }
  stdout {
    codec => rubydebug
  }
}

```

Any suggestions where i'm doing wrong

Thanks  
Gauti

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [November 13, 2018, 5:11pm UTC](https://discuss.elastic.co/t/unable-to-index-csv-data/156442/2 "2018-11-13T17:11:18Z")

</div>

All looks correct.  
I assume you have following line uncommented when you run it

# elasticsearch {

# hosts =\> "1.12.1.3:9200"

# index =\> "incanalysis"

# }

can you post how is your csv file looks like. just one-two line from it

---

<div class="post-metadata">

**Author:** ![Gauti](https://avatars.discourse-cdn.com/v4/letter/g/cdc98d/32.png) [@Gauti](https://discuss.elastic.co/u/Gauti)\
**Post date:** [November 14, 2018, 6:48am UTC](https://discuss.elastic.co/t/unable-to-index-csv-data/156442/3 "2018-11-14T06:48:11Z")

</div>

@elasticforme ven if i enable to send to elasticsearch still i get the same warning message, other than the warning items remaining documents are getting indexed.

Here is the sample of my csv

| Number | Configuration\_item | Opened | Short\_description | Assignment\_group | Contact\_type | Major\_incident | Caller | Priority | Resolved | Closed | Status | Updated\_by | Closed\_by | Category | Close\_notes |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| INC0230928 | DOM-CE | 10-26-18 14:53 | Domain Password reset | SD-INCIDENTS | Phone | FALSE | Eggermont | Support | 10-26-18 14:55 | | Resolved | a044 | | Password reset / Unlocked Account | Reset the users password |
| INC0230927 | DOM-EMAIL | 10-26-18 14:51 | FUSE: Office 365 Teams | EMAIL | Phone | FALSE | Dorman | Low | | | Assigned | a015 | | | |
| INC0230926 | UA-SAP-UK | 10-26-18 14:45 | SAP password reset / account unlock SAP-ID: A018 | SD-INCIDENTS | Email | FALSE | ravi ravi | Support | 10-26-18 14:47 | | Resolved | a006 | | Password reset / Unlocked Account | Password reset done sent an email to the user. |

Thanks  
Gauti

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [November 14, 2018, 3:14pm UTC](https://discuss.elastic.co/t/unable-to-index-csv-data/156442/4 "2018-11-14T15:14:48Z")

</div>

try with skip\_empty\_columns on filter field.

[https://www.elastic.co/guide/en/logstash/current/plugins-filters-csv.html#plugins-filters-csv-skip\_empty\_columns](https://www.elastic.co/guide/en/logstash/current/plugins-filters-csv.html#plugins-filters-csv-skip_empty_columns)

---

<div class="post-metadata">

**Author:** ![Gauti](https://avatars.discourse-cdn.com/v4/letter/g/cdc98d/32.png) [@Gauti](https://discuss.elastic.co/u/Gauti)\
**Post date:** [November 15, 2018, 7:55am UTC](https://discuss.elastic.co/t/unable-to-index-csv-data/156442/5 "2018-11-15T07:55:15Z")

</div>

@elasticforme i havent done this skip\_empty\_rows before can you please share an example syntax on how to write it down in the config,

Also have identified there are few special characters which are also causing the issue.

Now two things "empty rows and special characters" is there any way by which we can ignore these two and proceed indexing,  
I also tried to write a mapping for all those fields in the csv and wrote an ignore malformed for the whole index and ended up with failure.

Any advice on this scenario

Thanks  
Gauti

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [November 15, 2018, 2:57pm UTC](https://discuss.elastic.co/t/unable-to-index-csv-data/156442/6 "2018-11-15T14:57:48Z")

</div>

sorry don't know much about how to ignore mailform character. this is all new to me as well.

---

<div class="post-metadata">

**Author:** ![Gauti](https://avatars.discourse-cdn.com/v4/letter/g/cdc98d/32.png) [@Gauti](https://discuss.elastic.co/u/Gauti)\
**Post date:** [November 15, 2018, 4:08pm UTC](https://discuss.elastic.co/t/unable-to-index-csv-data/156442/7 "2018-11-15T16:08:36Z")

</div>

@elasticforme Any advice on skipping special characters?

Thanks  
Gauti

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [November 15, 2018, 8:24pm UTC](https://discuss.elastic.co/t/unable-to-index-csv-data/156442/8 "2018-11-15T20:24:41Z")

</div>

I think you can do mutate. I show a example somewhere  
filter {  
if [message\_id] {  
mutate { add\_field =\>; { "[[@metadata][id]" =\> "%{message\_id}" } }  
} else {  
mutate { add\_field =\> { "[[@metadata][id]" =\> nil } }  
}  
}

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 13, 2018, 8:24pm UTC](https://discuss.elastic.co/t/unable-to-index-csv-data/156442/9 "2018-12-13T20:24:42Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.