# Shipping CSV file using Filebeat to Logstash and then from logstash to Elasticsearch

**URL:** https://discuss.elastic.co/t/shipping-csv-file-using-filebeat-to-logstash-and-then-from-logstash-to-elasticsearch/48921
**Category:** Logstash
**Created:** [May 2, 2016, 6:07am UTC](https://discuss.elastic.co/t/shipping-csv-file-using-filebeat-to-logstash-and-then-from-logstash-to-elasticsearch/48921 "2016-05-02T06:07:45Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![gauravkb](https://avatars.discourse-cdn.com/v4/letter/g/a4c791/32.png) [@gauravkb](https://discuss.elastic.co/u/gauravkb)
#### Post date: [May 2, 2016, 6:07am UTC](https://discuss.elastic.co/t/shipping-csv-file-using-filebeat-to-logstash-and-then-from-logstash-to-elasticsearch/48921/1 "2016-05-02T06:07:45Z")

</div>

hi ,

Im trying to ship csv file using filebeat to logstash and then to elasticsearch but nothing is feeding into logstash or elasticsearch

This is how my filebeat.yml looks like..

filebeat:

# List of prospectors to fetch data.

prospectors:  
paths:  
- /opt/logstash/logs/date\_wise2.csv  
#- c:\programdata\elasticsearch\logs\*  
input\_type: csv  
document\_type: date\_wise  
registry\_file: /var/lib/filebeat/registry  
output:  
logstash:  
# The Logstash hosts  
hosts: ["localhost:5044"]

```
# Number of workers per Logstash host.
worker: 1

# Set gzip compression level.
compression_level: 3

```

and this is how my elkconf.conf looks like..

input {  
beats {  
host =\> "localhost"  
port =\> "5044"  
}  
}  
filter  
{  
if [type] == "date\_wise"  
{  
csv  
{  
columns =\> ["Date","Critical","Major","Warning","Minor","Normal","Grand Total"]  
separator =\> ","  
skip\_empty\_columns =\> "true"  
}  
mutate  
{  
convert =\> ["Critical" , "integer"]  
convert =\> ["Major", "integer"]  
convert =\> ["Warning", "integer"]  
convert =\> ["Minor", "integer"]  
convert =\> ["Normal", "integer"]  
convert =\> ["Grand Total", "integer"]  
}

if ([Date] =~ "Date" )  
{  
drop{}  
}  
if ( [Date] =~ "CONCATENATE Message" )  
{  
drop{}  
}  
if ([Date] =~ "Grand Total" )  
{  
drop{}  
}  
date {  
match =\> ["Date","MM/dd/YYYY"]  
}  
}  
}  
output  
{  
if [type] == "date\_wise"  
{  
elasticsearch  
{  
hosts =\> "localhost:9200"  
index =\> "date-wise%{+YYYY.MM.dd}"  
}  
}  
stdout { codec =\> rubydebug }  
}

and this is how my logstash logs looks like

# cat /var/log/logstash/logstash.log

{:timestamp=\>"2016-05-02T10:25:51.641000+0530", :message=\>"SIGTERM received. Shutting down the agent.", :level=\>:warn}  
{:timestamp=\>"2016-05-02T10:25:51.643000+0530", :message=\>"stopping pipeline", :id=\>"main"}  
{:timestamp=\>"2016-05-02T10:25:52.534000+0530", :message=\>"Pipeline main has been shutdown"}  
{:timestamp=\>"2016-05-02T10:26:10.156000+0530", :message=\>"Pipeline main started"}

im not sure where the problem is , but im assuming since i have not specified  
start\_position =\> "beginning" , there would be any issues..

thanks

---

<div class="post-metadata">

### Author: ![rajkumar3v](https://avatars.discourse-cdn.com/v4/letter/r/73ab20/32.png) [@rajkumar3v](https://discuss.elastic.co/u/rajkumar3v)
#### Post date: [July 26, 2016, 5:21am UTC](https://discuss.elastic.co/t/shipping-csv-file-using-filebeat-to-logstash-and-then-from-logstash-to-elasticsearch/48921/2 "2016-07-26T05:21:40Z")

</div>

Hi Gaurav,

have you resolved this issue, kindly share it because am also having the same issue.

the pipeline have started fine and the data is not pushed to elastic.

Thanks,  
Raj

---

<div class="post-metadata">

### Author: ![Tom-Kun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tom-kun/32/10114_2.png) [@Tom-Kun](https://discuss.elastic.co/u/Tom-Kun)
#### Post date: [November 4, 2016, 2:01pm UTC](https://discuss.elastic.co/t/shipping-csv-file-using-filebeat-to-logstash-and-then-from-logstash-to-elasticsearch/48921/3 "2016-11-04T14:01:25Z")

</div>

Hi Gaurav,

First your filebeat.yml file is not correct...  
filebeat:  
#List of prospectors to fetch data  
prospectors:  
paths:  
- /opt/logstash/logs/date\_wise2.csv  
#log: Reads every line of the log file (default)  
#stdin : Reads the standard in.  
**#csv input\_type doesn't exist**  
input\_type: **log**  
document\_type: date\_wise  
registry\_file: /var/lib/filebeat/registry  
output:  
logstash:  
hosts: ["localhost:5044"]

The rest of your configuration seems to be correct.  
If you want to do some test over and over again, you have to delete your "registry" file into your /var/lib/filebeat/registry (by default).

Best regards,  
Tom

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 4:31am UTC](https://discuss.elastic.co/t/shipping-csv-file-using-filebeat-to-logstash-and-then-from-logstash-to-elasticsearch/48921/4 "2017-07-06T04:31:05Z")

</div>