Unable to log CSV

[AdamD25 (Adam)]

Hi,

I'm new to logstash so forgive any obvious mistakes! I've recently installed the ELK stack and have been trying to setup a pipeline to elasticsearch but have hit a wall. My config file (see below) seems to run without a hitch but for some reason my CSV isn't being logged - I've checked the structure of my CSV and edited it recently so there doesn't seem to be a problem there.

input {
file {
path => "C:\ELK_Stack\data\test_data.csv"
start_position => "beginning"
sincedb_path => "NUL"
}
}
filter {
csv {
separator => ","
columns => [ "propcode", "prop_name", "unqid", "PropType", "Portfolio", "asst_mgr", "OccupStatus", "Occstatus", "TenantName", "ERVunderwr", "LsStart", "LeaseToDate", "NextRRDate", "rentstdate", "Effbrdate", "brknotice", "Tcertexp", "tcertrem", "ContractedRent", "EstimatedRent", "unit_area" ]
}
}
output {
stdout { codec => rubydebug }
}

I've tried using the --debug mode but haven't spotted anything useful, I've put the output on pastebin:

https://pastebin.com/bpgfLzBV

Any help would be hugely appreciate!

Kind regards,

Adam

[guyboertje]

This behaviour is due to the strict way that the "tailing" concept has been interpreted, like the unix tail -f command.
If a file is present when Logstash starts, then reading starts from the end, i.e. only process new content that is seen in the file after LS starts. The start_position is meant for when a file is "discovered" after LS starts where one can override this tail from the end.

The version of the file input that is shipped with LS 6.4.0 has a "read" mode - specifically designed for when a file should be read from start to finish (subject to sincedb position tracking, which you opted out of by setting the sincedb_path to NUL).

Along with read mode is the ability to do some action when the EOF is reached. The default action is to delete the file, but you can change this to "log" as well as adding a path to a file that will hold the path to each file as it is completed. If using the delete option, make sure that you have a copy of the original.

[balumurari1]

Refer the below link. Hope this helps you,
https://discuss.elastic.co/t/load-csv-file-in-logstash/151319/2

Regards,
Balu