Logs shipping to redis queue

Elastic Stack Logstash
1

I'm shipping some logs from server 1 to server 2 using logstash-shipper and receiving the message on redisqueue on server 2

the config file i'm using is this:

input {
file {
path => "/path/to/my/logfile.log"
type => "apache"
}
}

output {
redis { host => "myserverIP" data_type => "list" key => "logstash" }
}

The log file contains 29 logs...however logstash shipper stops after sending 3 logs..why is that?
I do not see any error messages either. Is it some sort of a delay issue somewhere?

2

Did you test with a stdout output?

Maybe you must "reset" the input file syncdb (by changing its path).

1 Like
3

Ok, after fiddling around a little...it was shipping logss.
however, it was extremely slow..what could be there reason behind it?

Another thing, After i recieved the logs on server 2 through redisqueue..i created a logstash config file...indexed at elasticsearch and displayed on kibana..

However, after some time..the logs were gone from kibana magically...(there were 5 logs..and which each refresh..some would disappear)

I don't understand what's happening here.

4

How do you measure "slowness"?

About Kibana, check time window (UI, on top right).

1 Like
5

Also check ignore_older parameter of input file.

1 Like
6

omg it works! Thanks so much!!!!!!!!!!!!!

7

I have 28 logs in total...for some reason it is only shipping 15 of 28 logs...

This is how my config file looks... logstash-shipper.conf

input {
file {
path => [ "/path/to/logs/sample.log" ]
type => "syslog"
start_position => "beginning"
sincedb_path => "/dev/null"
ignore_older => 0
}
}

output {
stdout { codec => rubydebug }
redis { host => "10...*" data_type => "list" key => "rediskey" }
}