Logstash Shipper to Redis

Elastic Stack Logstash
1

Hi,

Im using LS Shipper to send to Redis queue.

The conf input is reading from different log files but I only see logs from type syslog being sent. Application and Web logs are not getting sent.

I've run LS shipper in debug mode and see it has found all the logs with no errors on startup, config looks good.

Anything else to check?

Also is there a way to encrypt the traffic from the LS shipper and Redis ?

Thanks
Aidan

2

The conf input is reading from different log files but I only see logs from type syslog being sent. Application and Web logs are not getting sent.

I've run LS shipper in debug mode and see it has found all the logs with no errors on startup, config looks good.

And new data is being added to the application and web logs?

Also is there a way to encrypt the traffic from the LS shipper and Redis ?

I don't believe Redis supports encryption. You could tunnel the traffic over e.g. SSH, or have the Logstash shippers send via the lumberjack protocol to another Logstash instance on the Redis box, which then passes events to Redis on localhost.

3

I added:
sincedb_path => "/dev/null"
start_position => "beginning"

and it reads the files now.

Thank you.

4

What is the preferred way to set logstash permission on the files as getting an error that was working yesterday?

failed to open /var/log/messages: Permission denied - /var/log/messages {:level=>:warn, :file=>"filewatch/tail.rb", :line=>"131", :method=>"_open_file"}

5

What does ls -l /var/log/messages say?

6

-rw-------+ 1 root root 650898854 Sep 25 04:31 /var/log/messages

I set setfacl -m u:logstash:r /var/log/messages yesterday.

7

Perhaps the file has rotated since then? If so you should be able to add setfacl as a post rotation command. I'd also look into the possibility of configuring what permissions these files should have. Which Linux is this? On Debian and Ubuntu the log files are mode 0640 with "adm" as the group, so adding Logstash to that group solves the problem.

8

Its CentOS.
Does adding an entry in /etc/sysconfig/logstash LS_GROUP=adm work ?

9

Only if the members of the adm group have permission to read the log file. I don't know of a way to configure the permissions of these log files so your best bet might be a post rotation script that adjusts the ACL or the file mode. Syslog might not open all files immediately, and in that case you may have to make sure that the file is created in the post rotation script so that you can change its permissions.

10

Hi Magnus,

Thanks. We will make the necessary changes outside of LS..

I have a question regarding monitoring LS. Are there any planned API’s to allow us to do this?

Can you confirm that Marvel does not need a licence anymore ?

Many thanks
Aidan

11

I have a question regarding monitoring LS. Are there any planned API’s to allow us to do this?

Yes, see the Logstash roadmap.

Can you confirm that Marvel does not need a licence anymore ?

No, I haven't heard anything about that.

12

Regarding Marvel, this is conflicting with what we were told at a training day in Dublin recently. The trainers said that Marvel or Marvel-lite would be available to the community ?

13

Hi Magnus,

Is there any benchmarks on disk i/o that Elasticsearch uses as compared to Graphite ?

Many thanks
Aidan

14

Is there any benchmarks on disk i/o that Elasticsearch uses as compared to Graphite ?

None that I'm aware of, but my hunch is that Graphite is friendlier I/O-wise.

15

Hi Magnus,

I have collectd sending metrics to Logstash and want to create graphs form memory usage, etc.

Any docs for this /

Many thanks
Aidan

16

That's a very open question. Have you looked at the Kibana introduction videos and documentation? (I'm assuming you're sending the data to Elasticsearch and want to use Kibana to plot it.)