# Logstash pipeline for csv file using filebeat **URL:** \ **Category:** Logstash\ **Created:** [November 5, 2019, 11:14pm UTC](https://discuss.elastic.co/t/logstash-pipeline-for-csv-file-using-filebeat/206689 "2019-11-05T23:14:30Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![newbieELK](https://avatars.discourse-cdn.com/v4/letter/n/839c29/32.png) [@newbieELK](https://discuss.elastic.co/u/newbieELK)\ **Post date:** [November 5, 2019, 11:14pm UTC](https://discuss.elastic.co/t/logstash-pipeline-for-csv-file-using-filebeat/206689/1 "2019-11-05T23:14:30Z")
Hi, Spent too much time on this, would really appreciate any help. My csv file loading in one record. My data is csv: Number,Category,Assignment group,Technology,Assigned to,Opened,O\_Month,Opened Day,Opened Date,Opened Time,Duration,Durn in hrs,Floor,TimeWindow,Closed,C\_Month,PriorZZZy,Short description,Configuration ZZZem,Attribute,Env INC0403021,Application,ZZZ MS-SQL Admin,MS\_SQL,Adam SmZZZh,2019-06-30 23:05:55,Jun,Sunday,06/30/19,11:05 PM,28531,7.9,10:00 PM,10 PM - 12 AM,1/0/1900,Jan,3 - Low,Control-M DS: DMS\_XXXsqldba0040\_YYYY\_DL\_XXX\_P - RC 1 - Ended not OK 6/30/2019 11:05:08 PM,Database - SQL Server,Backup,Prod INC0403009,Application,ZZZ MS-SQL Admin,MS\_SQL,Adam SmZZZh,2019-06-30 19:33:47,Jun,Sunday,06/30/19,7:33 PM,41202,11.4,6:00 PM,6 PM - 8 PM,1/0/1900,Jan,3 - Low,Control-M DS: DS\_XXXsqldba0015\_TRANSLOG\_BCK\_PROD - RC 1 - Ended not OK 6/30/2019 7:13:53 PM,Database - SQL Server,Backup,Prod This is the logstash pipeline: input { beats { port =\> 5045 ssl =\> true ssl\_certificate\_authorities =\> ["xxxxxx.pem"] ssl\_certificate =\> "/etc/logstash/SSL/logstash\_xxxxx.pem" ssl\_key =\> "/etc/logstash/SSL/logstash\_xxxxx.key" } } filter { if "inc" in [tags] { csv { columns =\> ["Number","Category","Assignment group","Technology","Assigned to","Opened","O\_Month","Opened Day","Opened Date","Opened Time","Duration","Durn in hrs","Floor","TimeWindow","Closed","C\_Month","Priority","Short Description","Confirguration","Attribution","Env"] skip\_header =\> true } #Date filter is used to convert date to @Timestamp so that chart in Kibana will show as per date date { match =\> ["Opened", "yyyy-MM-dd HH:mm:ss"] timezone =\> "America/New\_York" target =\> "Opened" ``` } ``` } } output { if "inc" in [tags] { elasticsearch { hosts =\> "localhost:9200" manage\_template =\> false index =\> "inc-%{+YYYY.MM.dd}" } } } ---
**Author:** ![kkulkarni](https://avatars.discourse-cdn.com/v4/letter/k/4af34b/32.png) [@kkulkarni](https://discuss.elastic.co/u/kkulkarni)\ **Post date:** [November 6, 2019, 5:26am UTC](https://discuss.elastic.co/t/logstash-pipeline-for-csv-file-using-filebeat/206689/2 "2019-11-06T05:26:08Z")
I think this is because logstash looks for newline at end of the line and only treats as new record. Please do an enter at the end of the row 2 and try. input { file { ``` # Set this relative to your log folder... path => "C:/Users/xxxxxxx/Desktop/test.csv" start_position => "beginning" } ``` } filter { csv { autodetect\_column\_names =\> true skip\_header =\> true separator =\> "," ``` } date { match => ["Opened", "yyyy-MM-dd HH:mm:ss"] timezone => "America/New_York" target => "Opened" } }# End of filter ``` output { stdout { codec =\> rubydebug } } # Following is the output # { "Category" =\> "Application", "C\_Month" =\> "Jan", "PriorZZZy" =\> "3 - Low", "Technology" =\> "MS\_SQL", "Assignment group" =\> "ZZZ MS-SQL Admin", "Duration" =\> "28531", "Attribute" =\> "Backup", "Short description" =\> "Control-M DS: DMS\_XXXsqldba0040\_YYYY\_DL\_XXX\_P - RC 1 - Ended not OK 6/30/2019 11:05:08 PM", "Env" =\> "Prod", "path" =\> "C:/Users/kkulkarni/Desktop/test.csv", "Durn in hrs" =\> "7.9", "O\_Month" =\> "Jun", "Closed" =\> "1/0/1900", "TimeWindow" =\> "10 PM - 12 AM", "Assigned to" =\> "Adam SmZZZh", "Opened" =\> 2019-07-01T03:05:55.000Z, "host" =\> "HYDLRAP000119", "Number" =\> "INC0403021", "Floor" =\> "10:00 PM", "Opened Day" =\> "Sunday", "@timestamp" =\> 2019-11-06T05:24:12.679Z, "Opened Date" =\> "06/30/19", "Opened Time" =\> "11:05 PM", "message" =\> "INC0403021,Application,ZZZ MS-SQL Admin,MS\_SQL,Adam SmZZZh,2019-06-30 23:05:55,Jun,Sunday,06/30/19,11:05 PM,28531,7.9,10:00 PM,10 PM - 12 AM,1/0/1900,Jan,3 - Low,Control-M DS: DMS\_XXXsqldba0040\_YYYY\_DL\_XXX\_P - RC 1 - Ended not OK 6/30/2019 11:05:08 PM,Database - SQL Server,Backup,Prod\r", "@version" =\> "1", "Configuration ZZZem" =\> "Database - SQL Server" } { "Category" =\> "Application", "C\_Month" =\> "Jan", "PriorZZZy" =\> "3 - Low", "Technology" =\> "MS\_SQL", "Assignment group" =\> "ZZZ MS-SQL Admin", "Duration" =\> "41202", "Attribute" =\> "Backup", "Short description" =\> "Control-M DS: DS\_XXXsqldba0015\_TRANSLOG\_BCK\_PROD - RC 1 - Ended not OK 6/30/2019 7:13:53 PM", "Env" =\> "Prod", "path" =\> "C:/Users/kkulkarni/Desktop/test.csv", "Durn in hrs" =\> "11.4", "O\_Month" =\> "Jun", "Closed" =\> "1/0/1900", "TimeWindow" =\> "6 PM - 8 PM", "Assigned to" =\> "Adam SmZZZh", "Opened" =\> 2019-06-30T23:33:47.000Z, "host" =\> "HYDLRAP000119", "Number" =\> "INC0403009", "Floor" =\> "6:00 PM", "Opened Day" =\> "Sunday", "@timestamp" =\> 2019-11-06T05:24:26.816Z, "Opened Date" =\> "06/30/19", "Opened Time" =\> "7:33 PM", "message" =\> "INC0403009,Application,ZZZ MS-SQL Admin,MS\_SQL,Adam SmZZZh,2019-06-30 19:33:47,Jun,Sunday,06/30/19,7:33 PM,41202,11.4,6:00 PM,6 PM - 8 PM,1/0/1900,Jan,3 - Low,Control-M DS: DS\_XXXsqldba0015\_TRANSLOG\_BCK\_PROD - RC 1 - Ended not OK 6/30/2019 7:13:53 PM,Database - SQL Server,Backup,Prod\r", "@version" =\> "1", "Configuration ZZZem" =\> "Database - SQL Server" } ---
**Author:** ![newbieELK](https://avatars.discourse-cdn.com/v4/letter/n/839c29/32.png) [@newbieELK](https://discuss.elastic.co/u/newbieELK)\ **Post date:** [November 6, 2019, 6:52pm UTC](https://discuss.elastic.co/t/logstash-pipeline-for-csv-file-using-filebeat/206689/3 "2019-11-06T18:52:10Z")
Hi, Thank you for your response. The issue is still there. I should have mentioned that the setup we have is that we have installed filebeat on a windows server which is pushes the file to elasticsearch. from filebeat server log: "@timestamp": "2019-11-06T15:24:05.078Z", "@metadata": { "beat": "", "type": "\_doc", "version": "" }, "message": "Number,Category,Assignment group,Technology,Assigned to,Opened,O\_Month,Opened Day,Opened Date,Opened Time,Duration,Durn in hrs,Floor,TimeWindow,Closed,C\_Month,PriorZZZy,Short description,Configuration ZZZem,Attribute,Env INC0403021,Application,ZZZ MS-SQL Admin,MS\_SQL,Adam SmZZZh,2019-06-30 23:05:55,Jun,Sunday,06/30/19,11:05 PM,28531,7.9,10:00 PM,10 PM - 12 AM,1/0/1900,Jan,3 - Low,Control-M DS: DMS\_XXXsqldba0040\_YYYY\_DL\_XXX\_P - RC 1 - Ended not OK 6/30/2019 11:05:08 PM,Database - SQL Server,Backup,Prod INC0403009,Application,ZZZ MS-SQL Admin,MS\_SQL,Adam SmZZZh,2019-06-30 19:33:47,Jun,Sunday,06/30/19,7:33 PM,41202,11.4,6:00 PM,6 PM - 8 PM,1/0/1900,Jan,3 - Low,Control-M DS: DS\_XXXsqldba0015\_TRANSLOG\_BCK\_PROD - RC 1 - Ended not OK 6/30/2019 7:13:53 PM,Database - SQL Server,Backup,Prod" "tags": [ "incident-mgmt" ], "input": { "type": "log" }, "ecs": { "version": "1.0.0" }, "host": { "os": { "version": "10.0", "family": "windows", "name": "Windows Server 2016 Standard", "kernel": "10.0.14393.3269 (rs1\_release.190929-1234)", "build": "14393.3274", "platform": "windows" }, "id": "c9ee2f53-8d44-4d1a-a813-7e4bed086071", "hostname": "windows2016svr01", "architecture": "x86\_64", "name": "windows2016svr01" }, "agent": { "ephemeral\_id": "f8fdb5b5-027c-4c42-a65a-24e6c7de59a6", "hostname": "windows2016svr01", "id": "3fa94c24-af18-4672-9f07-1cd34d4a32e9", "version": "7.0.0", "type": "filebeat" }, "log": { "offset": 0, "file": { "path": "D:\ELK\_Logs-data\inc6m-formatted6.csv" }, "flags": [ "truncated", "multiline" ] } } ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 4, 2019, 6:52pm UTC](https://discuss.elastic.co/t/logstash-pipeline-for-csv-file-using-filebeat/206689/4 "2019-12-04T18:52:17Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.