# Csv files from FIlebeat to Elasticsearch

**URL:** https://discuss.elastic.co/t/csv-files-from-filebeat-to-elasticsearch/229492
**Category:** Elasticsearch
**Created:** [April 23, 2020, 2:43pm UTC](https://discuss.elastic.co/t/csv-files-from-filebeat-to-elasticsearch/229492 "2020-04-23T14:43:34Z")
**Posts on this page:** 8
**Page:** 1

<div class="post-metadata">

### Author: ![wild\_turkey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wild_turkey/32/66978_2.png) [@wild\_turkey](https://discuss.elastic.co/u/wild_turkey)
#### Post date: [April 23, 2020, 2:43pm UTC](https://discuss.elastic.co/t/csv-files-from-filebeat-to-elasticsearch/229492/1 "2020-04-23T14:43:34Z")

</div>

Hi All,

I'm a beginner to the Elasticstack world and learning to architect the collection of logs for visualization.

One file I'm working on is a CSV file. I want to feed it into Elasticsearch to visualize in Kibana. I assume I use filebeat for this.

Question #1 - What is logstash and would I need it in this scnario?  
Question #2 - Is there a template of a config file that will get me started to collect files from a directory and feed to elasticsearch?

Thank you in advance!

---

<div class="post-metadata">

### Author: ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [April 23, 2020, 3:23pm UTC](https://discuss.elastic.co/t/csv-files-from-filebeat-to-elasticsearch/229492/2 "2020-04-23T15:23:05Z")

</div>

Welcome!

1. Logstash is an ETL (Extract Transform and Load) tool. Used for complex use cases. You don't need it here. You can directly stream the CSV to Elasticsearch with filebeat and use [https://www.elastic.co/guide/en/elasticsearch/reference/7.6/csv-processor.html](https://www.elastic.co/guide/en/elasticsearch/reference/7.6/csv-processor.html) to transform the data.

2. Here is one:

> <https://github.com/dadoonet/bano-elastic/blob/master/filebeat-config/filebeat-cloud.yml>

And the associated ingest pipeline:

> <https://github.com/dadoonet/bano-elastic/blob/master/cloud/ingest-bano.json>

---

<div class="post-metadata">

### Author: ![droberts195](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/droberts195/32/17692_2.png) [@droberts195](https://discuss.elastic.co/u/droberts195)
#### Post date: [April 23, 2020, 3:32pm UTC](https://discuss.elastic.co/t/csv-files-from-filebeat-to-elasticsearch/229492/3 "2020-04-23T15:32:26Z")

</div>

In the not-too-distant future there will be an easier way to get the ingest pipeline config and Filebeat config than typing them out by hand.

Starting in version 7.7 (not released yet but not too far off) after [importing a CSV file using the File Data Visualizer](https://www.elastic.co/blog/importing-csv-and-log-data-into-elasticsearch-with-file-data-visualizer) in Kibana there will be an option to display a sample Filebeat config that would be appropriate for CSV files with the same structure that you uploaded as a sample - it's the change that was made in [https://github.com/elastic/kibana/pull/58152](https://github.com/elastic/kibana/pull/58152) and there's a screenshot in that PR. The File Data Visualizer will leave behind the ingest pipeline that was appropriate for the CSV columns that it saw, so you'll have nearly everything you need. (Just a few details like hostnames and passwords need to be dealt with manually.)

As I said, you cannot do this today but 7.7 is the next release, so it won't be _that_ long before it's possible.

---

<div class="post-metadata">

### Author: ![wild\_turkey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wild_turkey/32/66978_2.png) [@wild\_turkey](https://discuss.elastic.co/u/wild_turkey)
#### Post date: [April 23, 2020, 3:36pm UTC](https://discuss.elastic.co/t/csv-files-from-filebeat-to-elasticsearch/229492/4 "2020-04-23T15:36:32Z")

</div>

Thank you @dadoonet, this is helpful. Excuse my ignorance, but for my education, can you confirm the flow?

- CSV file is on my PC in a c:/CSV folder
- The file is imported through Filebeat, and sent to Elasticsearch.

The ingestion part I have a few grey areas. Is there anything I have to install to use the csv processor? Does this take place before of after the filebeat import?

---

<div class="post-metadata">

### Author: ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [April 23, 2020, 4:07pm UTC](https://discuss.elastic.co/t/csv-files-from-filebeat-to-elasticsearch/229492/5 "2020-04-23T16:07:32Z")

</div>

I confirm the flow.

> [@wild\_turkey](#):
>
> Is there anything I have to install to use the csv processor?

No

> [@wild\_turkey](#):
>
> Does this take place before of after the filebeat import?

You have to define the ingest pipeline before starting filebeat.

---

<div class="post-metadata">

### Author: ![wild\_turkey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wild_turkey/32/66978_2.png) [@wild\_turkey](https://discuss.elastic.co/u/wild_turkey)
#### Post date: [April 23, 2020, 5:11pm UTC](https://discuss.elastic.co/t/csv-files-from-filebeat-to-elasticsearch/229492/6 "2020-04-23T17:11:33Z")

</div>

One more question to help connect the dots (I haven't ingested any files yet, this will be my first time using a Windows installation of Elasticstack)

Would the pipeline be in a different configuration file and run prior?

I've been reading and watching youtube videos, there seems to be many ways to do one half of the configuration but not a step by step of my end to end use case.

I really appreciate the input.

---

<div class="post-metadata">

### Author: ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [April 24, 2020, 2:42am UTC](https://discuss.elastic.co/t/csv-files-from-filebeat-to-elasticsearch/229492/7 "2020-04-24T02:42:57Z")

</div>

You create a pipeline using the REST API.  
So no config file for this. See

> <https://github.com/dadoonet/bano-elastic/blob/c735a3dcad65c7b332df346b365faae69ad81a2e/setup-cloud.sh#L18>

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 22, 2020, 2:43am UTC](https://discuss.elastic.co/t/csv-files-from-filebeat-to-elasticsearch/229492/8 "2020-05-22T02:43:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
