# Logstash encoding issue for csv file

**URL:** <https://discuss.elastic.co/t/logstash-encoding-issue-for-csv-file/185426>\
**Category:** Logstash\
**Created:** [June 12, 2019, 1:34pm UTC](https://discuss.elastic.co/t/logstash-encoding-issue-for-csv-file/185426 "2019-06-12T13:34:07Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![kamaleshwar](https://avatars.discourse-cdn.com/v4/letter/k/ea5d25/32.png) [@kamaleshwar](https://discuss.elastic.co/u/kamaleshwar)\
**Post date:** [June 12, 2019, 1:34pm UTC](https://discuss.elastic.co/t/logstash-encoding-issue-for-csv-file/185426/1 "2019-06-12T13:34:08Z")

</div>

I am facing the encoding issue,when i run the logstash ,the fields are displaying in unreadable format

**Logstash.conf**

input {  
file {  
path =\>"C:/db2mon\_v105/v105/db2mon\_reports.csv"  
start\_position =\> "beginning"  
sincedb\_path =\> "null"  
}  
}  
filter {  
csv {  
separator =\> ","  
columns =\>["TIME\_STAMP","TS\_DELTA","MEMBER","ACT\_PER\_S","CMT\_PER\_S ","RB\_PER\_S","DDLCK\_PER\_S","SEL\_P\_S","UID\_P\_S","ROWS\_INS\_P\_S","ROWS\_UPD\_P\_S","ROWS\_RET\_P\_S","ROWS\_MOD\_P\_S","PKG\_CACHE\_INS\_P\_S","P\_RD\_PER\_S"]  
}  
}  
output {  
elasticsearch {  
hosts =\> "[http://172.31.55.33:9200](http://172.31.55.33:9200)"  
user =\> "yyyyyy"  
password =\> "xxxxxxx"  
index =\> "icd"  
}   
stdout {  
}

```
    }

```

Please do find the filebeats.yml file

```
###################### Filebeat Configuration Example #logstash 

#=========================== Filebeat inputs =============================

filebeat.inputs:

filebeat.prospectors:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true
  encoding: utf-16

  # Paths that should be crawled and fetched. Glob based paths.

  
  
    #- /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*
    #- C:\Program Files\IBM\SQLLIB\TPAEAutomation\DB2_10\DB2_install.log
    #- C:\db2mon_v105\v105\db2mon_report.txt
     - C:\db2mon_v105\v105\db2mon_reports.csv
    
  
#============================= Filebeat modules ===============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 5
  #index.codec: best_compression
  #_source.enabled: false

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
# env: staging

#============================== Dashboards =====================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here, or by using the `-setup` CLI flag or the `setup` command.
#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

   #Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "http://172.31.55.33:5601"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

#============================= Elastic Cloud ==================================

# These settings simplify using filebeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["172.31.55.33:9200"]

  # Enabled ilm (beta) to use index lifecycle management instead daily indices.
  #ilm.enabled: false

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  username: "vvvvv"
  password: "pppppp"

#----------------------------- Logstash output --------------------------------
#output.logstash:
  #The Logstash hosts
  #hosts: ["172.31.55.33:5044"]

```

**What i see when i run the logstash :**  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/4/84c5bf9cc9594592377234a66bbf87dd68404c61.png)

Kindly let me now what needs to be done to rectify this issue.

Thanks in advance.

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [June 12, 2019, 11:53pm UTC](https://discuss.elastic.co/t/logstash-encoding-issue-for-csv-file/185426/2 "2019-06-12T23:53:56Z")

</div>

What encoding are the source files that Filebeat is handling? Your Filebeat configuration indicates that you have explicitly set `UTF-16`, so Filebeat processes them _as if_ they were `UTF-16`.

Here is a similar issue, where the user found that their files were encoded with `ANSI_X3.4-1968` and were able to configure Filebeat to read them correctly: [Character encoding problems Filebeat & Logstash](https://discuss.elastic.co/t/character-encoding-problems-filebeat-logstash/150353/2)

---

<div class="post-metadata">

**Author:** ![kamaleshwar](https://avatars.discourse-cdn.com/v4/letter/k/ea5d25/32.png) [@kamaleshwar](https://discuss.elastic.co/u/kamaleshwar)\
**Post date:** [June 13, 2019, 5:38am UTC](https://discuss.elastic.co/t/logstash-encoding-issue-for-csv-file/185426/3 "2019-06-13T05:38:04Z")

</div>

@yaauie,

I haven't added the encoding as utf-16 in filebeat.yml earlier, just found that some one has used it for the similar issue which has fixed the encoding issue .  
But for me it does not seems to be working ☹

Do i need to set the below as it was mentioned from the same kinda issue.

```
filebeat.prospectors:
- type: log
  enabled: true
  encoding: ANSI_X3.4-1968

```

Let me know if i need to do anything else apart from this.

Thanks in Advance,

Kamalesh

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [June 13, 2019, 9:26pm UTC](https://discuss.elastic.co/t/logstash-encoding-issue-for-csv-file/185426/4 "2019-06-13T21:26:36Z")

</div>

You need to figure out the encoding of your source files, and specify what that encoding is explicitly into the filebeat configuration.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 11, 2019, 9:31pm UTC](https://discuss.elastic.co/t/logstash-encoding-issue-for-csv-file/185426/5 "2019-07-11T21:31:55Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.