Reading Old Log files


#1

Hi,

I have ELK Server setup and the logstash keeps reading the old files along with the new ones.
How can I prevent that?


(Mark Walkom) #2

Can we see your config?


#3

Yes sure:

Input Config :

input {
lumberjack {
port => 5043
type => "logs"
codec => "json"
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}

Filter:
filter {
grok {
match => ["message","(?[0-9.]{1,6}) %{WORD:method} %{TIMESTAMP_ISO8601:RequestTime} %{URIPATHPARAM:RequestUrl} %{TIMESTAMP_ISO8601:RequestTime} %{NUMBER:StatusCode} %{DATA:level}"]
tag_on_failure => []
}
}

output {
elasticsearch { hosts => ["localhost:9200"]
}
stdout { codec => rubydebug }

}


(Mark Walkom) #4

So how are your logs getting sent to LS? Cause there isn't anything there that would indicate why this may happen.


#5

oh sorry! The logs get into logstash using logstash forwarder.


(Mark Walkom) #6

Ok, but you missed my point :slightly_smiling:

How do logs get to Logstash?


#7

Ohh sorry! The logs get into logstash using LSF (logstash forwarder).


#8

Thats the logstash-forwarder conf at the client:
{
"network": {
"servers": [ "log-server:5043" ],

"ssl certificate": "/etc/logstash/logstash-certs/logstash-forwarder.crt",
"ssl ca": "/etc/logstash/logstash-certs/logstash-forwarder.crt",

"timeout": 15
},
"files": [
{
"paths": [ "/opt/logs/*.log"],
"fields": { "type": "log","environment": "production" }
},
{
"paths": [ "/var/log/syslog"],
"fields": { "type": "syslog" }
}
]


(Mark Walkom) #9

Ok, LSF is deprecated, you should use filebeat.

That aside, I don't see a problem with your config, do things get moved aside on the FS?


#10

Yes. I have migrated few servers from LSF to Filebeat. But I have one doubt, since I have around 100+ servers all sending logs to one single server. What is the best way to scale out?
And will this issue go away completely with Filebeat?
Andyes we are using logrotate to move the old logs.


(Mark Walkom) #11

We recommend the use of a broker for that - https://www.elastic.co/guide/en/logstash/2.1/deploying-and-scaling.html#deploying-message-queueing


#12

Alright @warkolm thanks a lot.


(system) #13