# How to input CSV files into Logstash from a front end UI?

**URL:** <https://discuss.elastic.co/t/how-to-input-csv-files-into-logstash-from-a-front-end-ui/145246>\
**Category:** Logstash\
**Created:** [August 20, 2018, 9:37pm UTC](https://discuss.elastic.co/t/how-to-input-csv-files-into-logstash-from-a-front-end-ui/145246 "2018-08-20T21:37:45Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![APJ](https://avatars.discourse-cdn.com/v4/letter/a/b5ac83/32.png) [@APJ](https://discuss.elastic.co/u/APJ)\
**Post date:** [August 20, 2018, 9:37pm UTC](https://discuss.elastic.co/t/how-to-input-csv-files-into-logstash-from-a-front-end-ui/145246/1 "2018-08-20T21:37:46Z")

</div>

Hi all,

I am developing a JavaScript application (Backend in **Node.js** and Frontend in **Vue.js** ), wherein I want to send CSV data into logstash from the front end or UI, i.e, the user can choose any CSV file from local system and upload it into logstash, which will then be sent to elasticsearch for analysis.

I am not sure or aware of a way to do this. Is there a mechanism to achieve this? If so, can someone please help me out? I would really appreciate your help. Thanks in advance!

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 21, 2018, 7:11am UTC](https://discuss.elastic.co/t/how-to-input-csv-files-into-logstash-from-a-front-end-ui/145246/2 "2018-08-21T07:11:27Z")

</div>

Three options come to mind:

- The backend can store the CSV in a directory that a file input in the Logstash configuration is monitoring with a wildcard.
- The backend can post the CSV to Logstash if you set up an http input.
- The backend can push the CSV to a message broker (Redis, RabbitMQ, Kafka, ...) that Logstash pulls from.

---

<div class="post-metadata">

**Author:** ![APJ](https://avatars.discourse-cdn.com/v4/letter/a/b5ac83/32.png) [@APJ](https://discuss.elastic.co/u/APJ)\
**Post date:** [August 21, 2018, 11:59pm UTC](https://discuss.elastic.co/t/how-to-input-csv-files-into-logstash-from-a-front-end-ui/145246/3 "2018-08-21T23:59:10Z")

</div>

Thanks a lot for the information. Posting the CSV to Logstash is the most suitable option for me. Below is my logstash configuration.

```
input {
  file {
    path => "E:/Downloads/health-data/*.csv"
    start_position => "beginning"
  }
}
filter {
  csv {
     separator => ","
     columns => ["Date","Blood Glucose","Blood Pressure","Weight","Height","BMI"]
  }
}
output {
   elasticsearch {
     hosts => "http://localhost:9200"
     index => "health-metrics"
  }
stdout {}
}

```

My Node.js server is running on port 3000 (localhost). I am assuming I don't need the local path as I will be posting from the server. So, I am unable to figure out how the logstash file should be configured. Could you please help me how I can configure the above config file to make a post request to logstash from my Node.js server?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 22, 2018, 4:35am UTC](https://discuss.elastic.co/t/how-to-input-csv-files-into-logstash-from-a-front-end-ui/145246/4 "2018-08-22T04:35:19Z")

</div>

Just add an http input. I think the port is the only option you need to configure.

I strongly suggest you replace the elasticsearch output with a `stdout { codec => rubydebug }`. Get the input and filters working as you want and only then worry about getting it into ES.

---

<div class="post-metadata">

**Author:** ![APJ](https://avatars.discourse-cdn.com/v4/letter/a/b5ac83/32.png) [@APJ](https://discuss.elastic.co/u/APJ)\
**Post date:** [August 24, 2018, 11:45pm UTC](https://discuss.elastic.co/t/how-to-input-csv-files-into-logstash-from-a-front-end-ui/145246/5 "2018-08-24T23:45:24Z")

</div>

Thanks,

I am trying to send the CSV files from a file system. The logstash configuration monitors for the any files in the directory. Here is the logstash configuration,

```
input {
  file {
    path => "E:\Downloads\Server\public\uploads\*.csv"
    start_position => "beginning"
  }
}

filter {
  csv {
    separator => ","
    columns => ["Date","Blood Glucose","Blood Pressure","Weight","Height","BMI"]
  }
}

output {
  elasticsearch {
    hosts => "http://localhost:9200"
    index => "health-metrics"
  }
  stdout {
    codec => rubydebug
  }
}

```

I checked if the configuration is fine by running the following command,

> .\logstash -f csv\_data.conf --config.test\_and\_exit

The configuration is OK.

But when I try to pipe the data into Elasticsearch, I don't see any index created on the ES server.

Here is the screenshot of my CSV file,

**files-1535150705952**

 ![Screenshot%20(6)](https://us1.discourse-cdn.com/elastic/original/3X/1/6/1686c8a35ddcb7ce937fca05a396dfd2010dbf2f.png)

What is wrong with the configuration? Could you please help me out?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 25, 2018, 1:52pm UTC](https://discuss.elastic.co/t/how-to-input-csv-files-into-logstash-from-a-front-end-ui/145246/6 "2018-08-25T13:52:11Z")

</div>

As I said, comment out the elasticsearch output and make sure you're getting Logstash to process input files.

I think you need to change the filename pattern to use forward slashes instead of backslashes, i.e. E:/Downloads/Server/public/uploads/\*.csv instead of E:\Downloads\Server\public\uploads\*.csv.

---

<div class="post-metadata">

**Author:** ![APJ](https://avatars.discourse-cdn.com/v4/letter/a/b5ac83/32.png) [@APJ](https://discuss.elastic.co/u/APJ)\
**Post date:** [August 27, 2018, 7:48pm UTC](https://discuss.elastic.co/t/how-to-input-csv-files-into-logstash-from-a-front-end-ui/145246/7 "2018-08-27T19:48:05Z")

</div>

Thanks a lot! It is working now. How do I parse the "Date" field so that it serves as a @timestamp for displaying visualizations on Kibana based on the date on X-axis?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 27, 2018, 7:49pm UTC](https://discuss.elastic.co/t/how-to-input-csv-files-into-logstash-from-a-front-end-ui/145246/8 "2018-08-27T19:49:52Z")

</div>

Use a date filter.

---

<div class="post-metadata">

**Author:** ![APJ](https://avatars.discourse-cdn.com/v4/letter/a/b5ac83/32.png) [@APJ](https://discuss.elastic.co/u/APJ)\
**Post date:** [August 27, 2018, 7:56pm UTC](https://discuss.elastic.co/t/how-to-input-csv-files-into-logstash-from-a-front-end-ui/145246/9 "2018-08-27T19:56:16Z")

</div>

I did use the date filter.

```
filter {
  csv {
    separator => ","
    columns => ["Date","Blood Glucose","Blood Pressure","Weight","Height","BMI"]
  }
  date{
    match => ["Date", "MM/dd/yy"],
    target => "Date"
  }
}

```

But, I get an error like this

```
PS E:\elk\logstash\bin> .\logstash -f csv_data.conf --config.test_and_exit
Sending Logstash's logs to E:/elk/logstash/logs which is now configured via log4j2.properties
[2018-08-27T12:34:58,327][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-08-27T12:34:58,769][FATAL][logstash.runner] The given configuration is invalid. Reason: Expected one of #, } at line 14, column 34 (byte 294) after filter {
  csv {
    separator => ","
    columns => ["Date","Blood Glucose","Blood Pressure","Weight","Height","BMI"]
  }
  date{
    match => ["Date", "MM/dd/yy"]
[2018-08-27T12:34:58,779][ERROR][org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

```

What wrong am I doing?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 27, 2018, 7:57pm UTC](https://discuss.elastic.co/t/how-to-input-csv-files-into-logstash-from-a-front-end-ui/145246/10 "2018-08-27T19:57:19Z")

</div>

Remove the comma at the end of the `match` line.

---

<div class="post-metadata">

**Author:** ![APJ](https://avatars.discourse-cdn.com/v4/letter/a/b5ac83/32.png) [@APJ](https://discuss.elastic.co/u/APJ)\
**Post date:** [August 27, 2018, 8:04pm UTC](https://discuss.elastic.co/t/how-to-input-csv-files-into-logstash-from-a-front-end-ui/145246/11 "2018-08-27T20:04:10Z")

</div>

Thanks! Also, how do I parse the fields (Blood Glucose, Height, Weight, etc) so that I can use them as fields on the y-axis of kibana visualizations?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 27, 2018, 8:06pm UTC](https://discuss.elastic.co/t/how-to-input-csv-files-into-logstash-from-a-front-end-ui/145246/12 "2018-08-27T20:06:42Z")

</div>

Convert the fields to numbers, I guess? The csv filter's convert option can assist with that. Keep in mind that the existing mapping of a field in an ES index can't be changed, so after adjusting the configuration you need to reindex your data (e.g. by deleting the existing index).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 24, 2018, 9:57pm UTC](https://discuss.elastic.co/t/how-to-input-csv-files-into-logstash-from-a-front-end-ui/145246/14 "2018-09-24T21:57:46Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.