Can't parse a CSV file

Elastic Stack Logstash
1

Hi,

I have a simple CSV which I want to parse using Logstash and create some simple graphs in Kibana. But for some reason (unknown to me), the CSV file doesn't get parsed and I cannot see the index in Elasticsearch. Here's my Logstash Conf file and some sample data in that CSV file:


input {  
      file {
          path => "C:\Users\ahmadmar\Desktop\edw_hana_comparison.csv"
          start_position => "beginning"
      }
}
filter {  
    csv {
        columns => ["subject_area","table_count_edw","size_in_gb_edw","db_used_space_bytes","db_defined_space_bytes"]
        separator => ","
      }
 }
output {  
   elasticsearch { 
        action => "index"
        host => "localhost"
        index => "edwhanacomparison"
        workers => 1
    }
  #stdout {
  #       codec => rubydebug
  #}
}



For this conf file, for example, I might have the following sample data in the CSV (comma separated):


subject_area,table_count_edw,size_in_gb_edw,db_used_space_bytes,db_defined_space_bytes
sales,2715,14504,15573200000000.0000000000,26410700000000.0000000000
finance,2580,8067,8661520000000.0000000000,18500300000000.0000000000
supply,2132,3387,3636590000000.0000000000,9214700000000.0000000000
marketing,1416,484,520204000000.0000000000,1761170000000.0000000000
inventory,1367,8039,8631410000000.0000000000,16044900000000.0000000000
item master data,1331,1232,1322660000000.0000000000,5283610000000.0000000000
sourcing,1323,2727,2927900000000.0000000000,7339070000000.0000000000
service management,1237,11962,12843900000000.0000000000,23557500000000.0000000000
customer,770,433,465250000000.0000000000,1137570000000.0000000000
common reference,551,809,868605000000.0000000000,1485580000000.0000000000
logistics,541,3451,3705090000000.0000000000,10196200000000.0000000000
supplier,506,338,363356000000.0000000000,932852000000.0000000000
issue,447,213,228628000000.0000000000,1115990000000.0000000000
location,411,20,21725418496.0000000000,137825000000.0000000000
sales opportunity,358,23,24829823488.0000000000,302353000000.0000000000
unknown,325,482,517389000000.0000000000,1642500000000.0000000000
demand,295,1273,1367190000000.0000000000,2626770000000.0000000000
price,195,303,325639000000.0000000000,546812000000.0000000000
worker,92,107,114895000000.0000000000,221994000000.0000000000
design,64,7,7236625920.0000000000,86974504344.0000000000
calendar,62,1,565151744.0000000000,9244766076.0000000000
commodity,44,1,1416584192.0000000000,15058641780.0000000000
wfds,6,33,35053620224.0000000000,133284000000.0000000000
midas,4,0,2237440.0000000000,4222191246.0000000000



What is wrong with my CONF file? I would appreciate any input!

thanks,
Ahmad
2

Your configuration looks fine, but you've probably processed the file in question at some point (e.g. while you were testing) so the start_position parameter doesn't take effect. See the fine-print in the documentation. Delete the file's sincedb entry or create a new file.

3

Thanks Magnus. I've figured it out.