# Upload csv fle in elasticsearch using filebeat

**URL:** <https://discuss.elastic.co/t/upload-csv-fle-in-elasticsearch-using-filebeat/306018>\
**Category:** Kibana\
**Created:** [May 31, 2022, 9:28am UTC](https://discuss.elastic.co/t/upload-csv-fle-in-elasticsearch-using-filebeat/306018 "2022-05-31T09:28:19Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![safaeaz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/safaeaz/32/105379_2.png) [@safaeaz](https://discuss.elastic.co/u/safaeaz)\
**Post date:** [May 31, 2022, 9:28am UTC](https://discuss.elastic.co/t/upload-csv-fle-in-elasticsearch-using-filebeat/306018/1 "2022-05-31T09:28:19Z")

</div>

i'm trying to load csv file in Elasticsearch using filebeat  
here is my filebeat.yml

```auto
filebeat.inputs:
#- type: log
- type: stdin
  setup.template.overwrite: true
  enabled: true
  close_eof: true
  paths: 
    - /usr/share/filebeat/dockerlogs/*.csv

  processors:
  - decode_csv_fields:
     fields:
       message: "message"
     separator: ","
     ignore_missing: false
     overwrite_keys: true
     trim_leading_space: true
     fail_on_error: true
  - drop_fields:
      fields: ["log", "host", "ecs", "input", "agent"]
  - extract_array:
      field: message
      mappings:
          sr: 0
          Identifiant PSI: 1
          libellé PSI: 2
          Identifiant PdR: 3
          T3 Date Prévisionnelle: 4
          DS Reporting PdR: 5
          Status PSI: 6
          Type PdR: 7
  - drop_fields:
        fields: ["message","sr"]

               
  #index: rapport_g035_prov_1
   
filebeat.registry.path: /usr/share/filebeat/data/registry/filebeat/filebeat

output:
  elasticsearch:
    enabled: true
    hosts: ["IPAdress:8081"]
    indices:
      - index: "rapport_g035_prov"
      #- index: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"
      #- index: "filebeat-7.7.0"
#setup.dashboards.kibana_index: file-*
seccomp.enabled: false
logging.metrics.enabled: false

```

but when i chek the index in kibana i found that the index can't read columnes names

 ![map](https://us1.discourse-cdn.com/elastic/original/3X/b/f/bfb1589a313cffc23d7fdbd8de98216559a243a5.png)

i tried to process the csv if filebeat.yml in another way

```auto
filebeat.inputs:
- type: log
  setup.template.overwrite: true
  enabled: true
  close_eof: true
  paths: 
    - /usr/share/filebeat/dockerlogs/*.csv

  processors:
  - decode_csv_fields:
      fields:
        message: decoded.csv
      separator: ","
      ignore_missing: false
      overwrite_keys: true
      trim_leading_space: false
      fail_on_error: true

               
  #index: rapport_g035_prov_1
   
filebeat.registry.path: /usr/share/filebeat/data/registry/filebeat/filebeat

output:
  elasticsearch:
    enabled: true
    hosts: ["IPAdress:8081"]
    indices:
      - index: "rapport_g035_prov"
      #- index: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"
      #- index: "filebeat-7.7.0"
#setup.dashboards.kibana_index: file-*
seccomp.enabled: false
logging.metrics.enabled: false

```

but i got the same error it cant map the index correctli i know ther is a problem in the proccessing of csv in the filebeat.yml but i dont know what is it

---

<div class="post-metadata">

**Author:** ![jsanz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsanz/32/53734_2.png) [@jsanz](https://discuss.elastic.co/u/jsanz)\
**Post date:** [June 27, 2022, 3:09pm UTC](https://discuss.elastic.co/t/upload-csv-fle-in-elasticsearch-using-filebeat/306018/2 "2022-06-27T15:09:53Z")

</div>

have you tried to process the CSV with an [ingest pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html)? That's the preferred method to shape your ingested data since you can centralize the business rules to produce elasticsearch documents inside the very same database and the filebeat component just sends the raw _lines_ to ES.

Take a look to this excellent video explaining how to create the pipeline from the ES stack management interface

[![](https://us1.discourse-cdn.com/elastic/original/3X/1/c/1ce60991e1b94685ee546e270499a59cc6acf213.jpeg "Ingest Pipelines - Daily Elastic Byte S02E11") ](https://www.youtube.com/watch?v=-wOvyubrHIM)

hope it helps!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 25, 2022, 3:10pm UTC](https://discuss.elastic.co/t/upload-csv-fle-in-elasticsearch-using-filebeat/306018/3 "2022-07-25T15:10:54Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.