Monitoring csv files in a directory using logstash.conf

Elastic Stack Logstash
1

Dear All,
I am new to ELK stack and have recently installed logstash, Elasticsearch and Kibana (all version 5.4 on CentOS 6.5).

Objective is to monitor a directory for new csv files and parse them using logstash and index them into Elasticsearch for report generation.

I can parse the files through the command line (/usr/share/logstash/bin/logstash -f logstash.conf) and it gets successfully parsed and indexed in ES, however when i put the conf file under /etc/logstash/conf.d folder, it does not process the csv files.

my logstash.conf is as below

input {
file {
path => "/root/ELK/wallet-data/*.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
separator => ","
columns => ["date","description","moneyIn","moneyOut","tx"]
}
}
output {
elasticsearch {
hosts => "http://localhost:9200"
index => "wallet-index"
}
}

I have enabled debug for the logfile and i can see it trying to process the conf file and get the column-names etc but nothing gets indexed in ES.

Not sure what am i doing wrong over here. Any help would be really appreciated.

Thanks
sirsyedian

2

does it work when you put your conf-file into the conf.d-directory and call it via cmd line?

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf

have you also tried to run logstash via service?
(i'm running it as a service deamon and config-files in /conf.d/ will be recognized)

3

I can parse the files through the command line (/usr/share/logstash/bin/logstash -f logstash.conf) and it gets successfully parsed and indexed in ES, however when i put the conf file under /etc/logstash/conf.d folder, it does not process the csv files.

In the second case, how are you starting Logstash?

4

Yes, it works fine when i call the conf file (from within the conf.d direcory) via cmd line.

Also i do have logstash running as a service and i can see in teh log file that it keeps on looking for changes in the conf files within conf.d directory.

5

I used "initctl start logstash" to start teh service and can see it in running state.

initctl status logstash

logstash start/running, process 28522

6

I used "initctl start logstash" to start teh service and can see it in running state.

When you do that Logstash will normally run as the logstash user and that user doesn't have permission to access /root.

7

Thanks Magnus,
A few questions:

1/ I do have /root/ELK and all of its subdirectories owned by logstash user, however i still get the same issue.

2/ How do i force the logstash service to run as root user?

3/ The logging is set to debug but i cant see any indications of permission issue. Is it how it normally works? (I can;t seem to be able to paste the long debug logs in the post (exceed character limit), Not sure if i can attach hte log file to the post somehow?).

Thanks

8

I do have /root/ELK and all of its subdirectories owned by logstash user, however i still get the same issue.

Look for log entries with "discover" in them. Then you should find an entry that indicates whether Logstash finds any files to read and we'll take it from there.

How do i force the logstash service to run as root user?

Why would you want to do that?

3/ The logging is set to debug but i cant see any indications of permission issue. Is it how it normally works? (I can;t seem to be able to paste the long debug logs in the post (exceed character limit), Not sure if i can attach hte log file to the post somehow?).

You can always use a gist or pastebin.

9

Thanks Magnus,
I have run the following test.

1/ started logstash service with a single csv file in /root/ELK/wallet-date directory.

2/ The logs link is copied below for the start of logstash service and its processing of logstash.conf file, but nothing gets indexed in ES.

3/ I added another csv file to /root/ELK/wallet-date directory to check if it gets discovered (I noticed "discover_interval = 15" log at the logstash startup) but nothing got discovered as per the logstash log entries despite the wait.

Logs:
https://pastebin.com/AMhE4FNJ

10

Okay, so the logging has changed. Here's what I was looking for

[2017-05-10T09:08:59,675][DEBUG][logstash.inputs.file     ] _globbed_files: /root/ELK/wallet-data/*.csv: glob is: []

So according to Logstash the filename pattern expands to zero files. This is usually caused by lack of permissions.

11

Thanks. It indeed was something to do with permissions.

Strange though, the files had the following permissions but were failing.
[root@SPARE wallet-data]# ls -ltr /root/ELK/wallet-data/*.csv
-rwxr-xr-x. 1 logstash logstash 15639 May 10 08:58 /root/ELK/wallet-data/test.csv
-rwxr-xr-x. 1 logstash logstash 15639 May 10 09:07 /root/ELK/wallet-data/test2.csv

I moved them to /home/logstash/InComing directory and updated the conf file to point to them and everything started working.
[root@SPARE logstash]# ls -ltr /home/logstash/Incoming/
-rwxr-xr-x. 1 logstash logstash 15639 May 10 10:14 test2.csv
-rwxr-xr-x. 1 root root 15639 May 10 10:17 test.csv

Both existing and new incoming files are being processed fine now. (even the ones with root ownership)
So is it likely the permissions from the top directory (/root) which was impacting the file retrieval.

Thanks alot for your help. Really appreciate it.

12

Strange though, the files had the following permissions but were failing.
[root@SPARE wallet-data]# ls -ltr /root/ELK/wallet-data/*.csv
-rwxr-xr-x. 1 logstash logstash 15639 May 10 08:58 /root/ELK/wallet-data/test.csv
-rwxr-xr-x. 1 logstash logstash 15639 May 10 09:07 /root/ELK/wallet-data/test2.csv

Sure, but what were the permissions of /root, /root/ELK, and /root/ELK/wallet-data? The Logstash user would need to have at least execute permission on all directories (and read permission to the leaf directory).

13

That explains it.
I was missing execute permissions for the logstash user on the /root directory itself.
It was fixed and logstash was able to process files from /root/ELK/wallet-data directory.

Thanks a lot.

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.