LogStash multiple files in single location

[ShashiK (Shashi)]

Logstash works for me when i have a direct file reference -
When I use the single file reference in the logstash config-
path => "C:\LogRepository\transaction\Web\HttpUser_1312.csv"
It works for me.
However when i put in a generic reference -
path => "C:\LogRepository\transaction\Web*.csv"
It does not index any files.
Problem for me is that this location contains individual user transaction data and I cannot say how many files or specifically the name of the files in this location. - this is a direct data dump.
So I am wondering if logstash file filter can support a tree structure, i.e. directory containing directories containing file(s).

[CDR (Colton)]

If I am understanding your problem correctly you have multiple files under the directory C:\LogRepository\transaction\Web. To read in the multiple files with the ending .csv change your path to this:

path => "C:\LogRepository\transaction\Web\*.csv"

This is saying that it will read in any files under the directory Web that end in .csv. What you originally had Logstash interpreted as "read anything under the directory transaction that follows Web*.csv"

If you have multiple directories under Web you can choose this for path:

path => "C:\LogRepository\transaction\Web\**\*.csv"

This means that Logstash will look under any directory under Web for files ending in .csv

https://www.elastic.co/guide/en/logstash/2.2/plugins-inputs-file.html#plugins-inputs-file-path

[ShashiK (Shashi)]

Thanks Colton.
It was a typo in my post.
below is the exact entry i have in my conf file.

input {
file {
path => "C:\Shashi\Tools\LogRepository\transaction\Web\http\*.csv"
type => "transactions"
start_position => "beginning"
}
}

filter {
csv {
separator => ","
columns => ["Username", "InterfaceType", "UserType", "MessageType",
"MessageTemplate", "StartTimeinMillis", "EndTimeinMillis",
"StartTime", "EndTime"]
}
}

output {
elasticsearch {
hosts => ["localhost:9200"]
index => "test_data"
document_type => "transactions"
}
stdout {}
}

Still the same issue......

[CDR (Colton)]

So it is not reading in any of the files that are located in that directory with .csv endings? Or when you say that it is not indexing do you mean that the files are being read in but not indexed in elasticsearch?

[ShashiK (Shashi)]

i dont think it is reading because i do not see any entries in elasticsearch.
Is there any log file i can enable to figure out what is going wrong....
again i do not think there is any issue with any of the input files.
Because when i remove *.csv and put Useri.csv as a specific file name then the records get indexed.