# Upload CSV File to Kibana Dashboard

**URL:** <https://discuss.elastic.co/t/upload-csv-file-to-kibana-dashboard/342821>\
**Category:** Kibana\
**Created:** [September 12, 2023, 11:01am UTC](https://discuss.elastic.co/t/upload-csv-file-to-kibana-dashboard/342821 "2023-09-12T11:01:19Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [September 12, 2023, 11:01am UTC](https://discuss.elastic.co/t/upload-csv-file-to-kibana-dashboard/342821/1 "2023-09-12T11:01:19Z")

</div>

Hi Team,

I need help on below two points while uploading csv file through kibana dashboard.

1. How to upload a csv file size of more than 100MB through the kibana dashboard.
2. How to upload multiple csv files to same indices.

Thanks,  
Debasis

---

<div class="post-metadata">

**Author:** ![ppisljar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppisljar/32/11588_2.png) [@ppisljar](https://discuss.elastic.co/u/ppisljar)\
**Post date:** [September 12, 2023, 12:17pm UTC](https://discuss.elastic.co/t/upload-csv-file-to-kibana-dashboard/342821/2 "2023-09-12T12:17:07Z")

</div>

thats not possible thru kibana UI, but you can use various methods to get the data indexed to elasticsearch.

- beats: you can use filebeat to monitor your folder for files and automatically index them, here is a sample filebeats config

```auto
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /path/to/your/csv/files/*.csv

output.elasticsearch:
  hosts: ["http://your-elasticsearch-host:9200"]
  index: "your_index_name"

```

you can use es bulk api to index the documents, and maybe a simple python script to do the processing for you

```auto
from elasticsearch import Elasticsearch
from elasticsearch.helpers import bulk

es = Elasticsearch(['http://localhost:9200']) # Replace with your Elasticsearch URL

# Read your CSV file and create a list of dictionaries for each row
# For example, you can use Python's CSV module for this.
data = [{"field1": value1, "field2": value2, ...}, {...}, ...]

# Index the data into Elasticsearch
bulk(es, data, index='your_index_name', doc_type='your_doc_type')

```

---

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [September 14, 2023, 7:49am UTC](https://discuss.elastic.co/t/upload-csv-file-to-kibana-dashboard/342821/3 "2023-09-14T07:49:17Z")

</div>

Thanks @ppisljar for your response to my first question. Could you please help me with the second requirement? For example, I am uploading a text.csv to the index "sample" and next want one more csv file to the same index "sample". Is there any way to achieve this?

Thanks,  
Debasis

---

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [September 14, 2023, 10:57am UTC](https://discuss.elastic.co/t/upload-csv-file-to-kibana-dashboard/342821/4 "2023-09-14T10:57:11Z")

</div>

Hi @ppisljar,

Do you have any reference blog where we can get how filebeats can be configured such way to upload csv files to the particular index.

Thanks,  
Debasis

---

<div class="post-metadata">

**Author:** ![ppisljar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppisljar/32/11588_2.png) [@ppisljar](https://discuss.elastic.co/u/ppisljar)\
**Post date:** [September 14, 2023, 12:58pm UTC](https://discuss.elastic.co/t/upload-csv-file-to-kibana-dashboard/342821/5 "2023-09-14T12:58:58Z")

</div>

here is something i found on the web: [Load CSV data to ElasticSearch using FileBeat](https://www.mayurshingare.dev/blog/load-csv-data-to-elasticsearch/)

---

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [September 18, 2023, 11:06am UTC](https://discuss.elastic.co/t/upload-csv-file-to-kibana-dashboard/342821/6 "2023-09-18T11:06:14Z")

</div>

Thanks @ppisljar. I had created the ingest pipelines but where to check if it is successful or failed because I could not see the data in the corresponding index. Is there any way to check this?

Thanks,  
Debasis

---

<div class="post-metadata">

**Author:** ![ppisljar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppisljar/32/11588_2.png) [@ppisljar](https://discuss.elastic.co/u/ppisljar)\
**Post date:** [September 18, 2023, 11:40am UTC](https://discuss.elastic.co/t/upload-csv-file-to-kibana-dashboard/342821/7 "2023-09-18T11:40:21Z")

</div>

logs should be in **/var/log/filebeat/filebeat**

---

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [September 18, 2023, 12:03pm UTC](https://discuss.elastic.co/t/upload-csv-file-to-kibana-dashboard/342821/8 "2023-09-18T12:03:46Z")

</div>

Below is my unit file /usr/lib/systemd/system/filebeat.service and when I check under /var/log there is no filebeat folder as I mentioned above is there any way to check if ingest pipeline is working or in failed state.

```auto
Preformatted text> UMask=0027
> Environment="GODEBUG='madvdontneed=1'"
> Environment="BEAT_LOG_OPTS="
> Environment="BEAT_CONFIG_OPTS=-c /etc/filebeat/filebeat.yml"
> Environment="BEAT_PATH_OPTS=--path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/filebeat --path.logs /var/log/filebeat"
> ExecStart=/usr/share/filebeat/bin/filebeat --environment systemd $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OPTS
> Restart=always

```

Thanks,  
Debasis

---

<div class="post-metadata">

**Author:** ![ppisljar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppisljar/32/11588_2.png) [@ppisljar](https://discuss.elastic.co/u/ppisljar)\
**Post date:** [September 19, 2023, 11:27am UTC](https://discuss.elastic.co/t/upload-csv-file-to-kibana-dashboard/342821/9 "2023-09-19T11:27:31Z")

</div>

can you confirm filebeat is running ?

try to follow this tutorial to get it running: [Filebeat quick start: installation and configuration | Filebeat Reference [8.10] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation-configuration.html)

---

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [September 20, 2023, 11:56am UTC](https://discuss.elastic.co/t/upload-csv-file-to-kibana-dashboard/342821/10 "2023-09-20T11:56:38Z")

</div>

Hi  
Yes, my filebeat is up and running.

`````auto
[root@cb-1 ~]# systemctl status filebeat
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
   Loaded: loaded (/usr/lib/systemd/system/filebeat.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2023-09-18 16:26:43 IST; 2 days ago
     Docs: https://www.elastic.co/beats/filebeat
 Main PID: 14193 (filebeat)
````Preformatted text`

Thanks,
Debasis
`````

---

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [September 20, 2023, 12:42pm UTC](https://discuss.elastic.co/t/upload-csv-file-to-kibana-dashboard/342821/11 "2023-09-20T12:42:01Z")

</div>

Hi @ppisljar ,

I followed the same doc to install filebeat and it is up and running as per the below command. Is there any way to validate the ingest pipeline if it is working properly or not?

> sytemctl status filebeat

Thanks,  
Debasis

---

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [September 22, 2023, 11:06am UTC](https://discuss.elastic.co/t/upload-csv-file-to-kibana-dashboard/342821/12 "2023-09-22T11:06:18Z")

</div>

HI @ppisljar ,

Could you please help here?

Thanks,  
Debasis

---

<div class="post-metadata">

**Author:** ![kcreddy](https://avatars.discourse-cdn.com/v4/letter/k/6f9a4e/32.png) [@kcreddy](https://discuss.elastic.co/u/kcreddy)\
**Post date:** [September 25, 2023, 7:13am UTC](https://discuss.elastic.co/t/upload-csv-file-to-kibana-dashboard/342821/13 "2023-09-25T07:13:04Z")

</div>

> [@Debasis\_Mallick](#):
>
> is there any way to check if ingest pipeline is working or in failed state.

You could use `GET /_nodes/stats?metric=ingest&filter_path=nodes.*.ingest.pipelines` to get [statistics about your ingest pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html#cluster-nodes-stats-ingest-ex:~:text=To%20further%20refine%20the%20response%2C%20change%20the%20filter_path%20value.%20For%20example%2C%20the%20following%20request%20only%20returns%20ingest%20pipeline%20statistics) to see `failed` count. You can run it from Dev Tools in Kibana.

> Is there any way to validate the ingest pipeline if it is working properly or not?

You can use the [simulate pipeline API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html#:~:text=You%20can%20also%20test%20pipelines%20using%20the%20simulate%20pipeline%20API.%20You%20can%20specify%20a%20configured%20pipeline%20in%20the%20request%20path.%20For%20example%2C%20the%20following%20request%20tests%20my%2Dpipeline) to test ingest pipeline.

---

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [September 26, 2023, 12:15pm UTC](https://discuss.elastic.co/t/upload-csv-file-to-kibana-dashboard/342821/14 "2023-09-26T12:15:01Z")

</div>

Hi @kcreddy Thanks for your response. Now I can see the ingest pipeline details from Dev tools as below. Which means data loaded to my indices but when I search the data under Discover tool nothing showing for sales indices so am I missing anything here.

```auto
> "parse_sales_data": {
> "count": 44462,
> "time_in_millis": 269,
> "current": 0,
> "failed": 1,
> "processors": [
> {
> "csv": {
> "type": "csv",
> "stats": {
> "count": 44462,
> "time_in_millis": 180,
> "current": 0,
> "failed": 0
> }
> }
> },

```

Thanks,  
Debasis

---

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [September 26, 2023, 12:56pm UTC](https://discuss.elastic.co/t/upload-csv-file-to-kibana-dashboard/342821/15 "2023-09-26T12:56:38Z")

</div>

Hi @kcreddy ,

In addition to the above issue, just want to inform you that I followed the below link to create a pipeline of sales (testing the theoretical part since I am new to the elasticsearch world) before doing the actual data load which is in csv format.

> **[Mayur Shingare](https://www.mayurshingare.dev/blog/load-csv-data-to-elasticsearch/)**
>
> Mayur Shingare's stories, thoughts and ideas.

Thanks,  
Debasis

---

<div class="post-metadata">

**Author:** ![kcreddy](https://avatars.discourse-cdn.com/v4/letter/k/6f9a4e/32.png) [@kcreddy](https://discuss.elastic.co/u/kcreddy)\
**Post date:** [September 26, 2023, 2:40pm UTC](https://discuss.elastic.co/t/upload-csv-file-to-kibana-dashboard/342821/16 "2023-09-26T14:40:56Z")

</div>

Can you provide both the ingest pipeline and also filebeat configuration with couple csv rows?

Can you query your `sales` index from Dev Tools and check if you can find documents from there?

```auto
GET sales/_search
{
  "query":{
    "match_all" : {}
  }
}

```

If so, maybe the [Data View](https://www.elastic.co/guide/en/kibana/current/data-views.html#settings-create-pattern) created might be wrong which is not pointing to the index where data is ingested. In this case documents might have been ingested into `sales` index, but your Data View doesn't contain `sales` index.

Also, you seem to have 1 failure in the pipeline. You could have an `on_failure` clause inside your pipeline to add `error.message` field to understand why the failure occurred. More info [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html#handling-pipeline-failures).

```auto
"on_failure": [
    {
      "set": {
        "description": "Record error information",
        "field": "error.message",
        "value": "Processor {{ _ingest.on_failure_processor_type }} with tag {{ _ingest.on_failure_processor_tag }} in pipeline {{ _ingest.on_failure_pipeline }} failed with message {{ _ingest.on_failure_message }}"
      }
    }
  ]

```

---

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [September 26, 2023, 4:11pm UTC](https://discuss.elastic.co/t/upload-csv-file-to-kibana-dashboard/342821/17 "2023-09-26T16:11:22Z")

</div>

Hi @kcreddy ,  
Please find the ingest pipeline details as belwo.

> PUT \_ingest/pipeline/parse\_sales\_data  
> {  
> "processors": [  
> {  
> "csv": {  
> "description": "Parse sales data from scanner",  
> "field": "message",  
> "target\_fields": ["sr","date","customer\_id","transaction\_id","sku\_category","sku","quantity","sales\_amount"],  
> "separator": ",",  
> "ignore\_missing":true,  
> "trim":true  
> },  
> "remove": {  
> "field": ["sr"]  
> }  
> }  
> ]  
> }

Below are some records from scanner-data.csv file

Below are some records from scanner-data.csv file

> ,Date,Customer\_ID,Transaction\_ID,SKU\_Category,SKU,Quantity,Sales\_Amount  
> 1,02/01/2016,2547,1,X52,0EM7L,1,3.13  
> 2,02/01/2016,822,2,2ML,68BRQ,1,5.46  
> 3,02/01/2016,3686,3,0H2,CZUZX,1,6.35  
> 4,02/01/2016,3719,4,0H2,549KK,1,5.59

I made the below changes in filebeat.yml

> # ============================== Filebeat inputs ===============================
> 
> - type: log  
> enabled: true  
> paths:
> - /cbdata/elasticsearch/scanner-data.csv # path to your CSV file  
> exclude\_lines: [^""] # header line  
> index: sales  
> pipeline: parse\_sales\_data
> 
> # ---------------------------- Elasticsearch Output ----------------------------
> 
> output.elasticsearch:
> 
> # Array of hosts to connect to.
> 
> hosts: ["[https://xx.xx.xx.xx:9200](https://xx.xx.xx.xx:9200)","[https://xx.xx.xx.xx:9200](https://xx.xx.xx.xx:9200)","[https://xx.xx.xx.xx:9200](https://xx.xx.xx.xx:9200)"]
> 
> # Protocol - either `http` (default) or `https`.
> 
> protocol: "https"
> 
> # Authentication credentials - either API key or username/password.
> 
> #api\_key: "id:api\_key"  
> username: "elastic"  
> password: "elastic"  
> ssl:  
> enabled: true  
> certificate\_authorities: ["/etc/filebeat/certs/cert.pem"]

---

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [September 26, 2023, 4:12pm UTC](https://discuss.elastic.co/t/upload-csv-file-to-kibana-dashboard/342821/18 "2023-09-26T16:12:52Z")

</div>

Hi @kcreddy ,

As I mentioned earlier I had followed the steps mentioned in below link.

> **[Mayur Shingare](https://www.mayurshingare.dev/blog/load-csv-data-to-elasticsearch/)**
>
> Mayur Shingare's stories, thoughts and ideas.

Thanks,  
Debasis

---

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [September 28, 2023, 5:29am UTC](https://discuss.elastic.co/t/upload-csv-file-to-kibana-dashboard/342821/19 "2023-09-28T05:29:40Z")

</div>

Hi @kcreddy ,

Did you find the time to look into the above issue.

Thanks,  
Debasis

---

<div class="post-metadata">

**Author:** ![kcreddy](https://avatars.discourse-cdn.com/v4/letter/k/6f9a4e/32.png) [@kcreddy](https://discuss.elastic.co/u/kcreddy)\
**Post date:** [September 28, 2023, 7:16am UTC](https://discuss.elastic.co/t/upload-csv-file-to-kibana-dashboard/342821/20 "2023-09-28T07:16:14Z")

</div>

Hey, I was going through the tutorial and was able to ingest without any problem. The data you presented above is different from the ones in the tutorial. For example, the date format is different. There might be WARN or ERROR messages in your filebeat logs indicating failure to index the document due to parsing in wrong format.

If there are other errors, I would check inside the filebeat logs.

[Next page](https://discuss.elastic.co/t/upload-csv-file-to-kibana-dashboard/342821.md?page=2)