Hi everyone, I'm a newer of using logstash, here is my case:
- In right case:
log nginx (hostname: staging) --> filebeat ---> logstash ---> output to file: /opt/logs/staging/2016-09-27.log ( right folder "staging" as I want). - But using redis as broker:
log nginx (hostname: staging) --> filebeat --->redis---> logstash ---> output to file: /opt/logs/%{host}/2016-09-27.log ( it create folder %{host} instead of "staging").
So what's problem?
Here's my logstash config
/etc/logstash/conf.d/02-beats-input.conf:
input { redis { host => "10.84.87.148" port => "6379" type => "redis-input" data_type => "list" key => "filebeat" } }
/etc/logstash/conf.d/14-file-filter.conf:
filter { if [type] == "nginx-access" { grok { match => { "message" => "%{NGINXACCESS}" } } } }
/etc/logstash/conf.d/30-elasticsearch-output.conf:
output { elasticsearch { hosts => ["localhost:9200"] } stdout { codec => rubydebug } file { codec => line { format => "%{message}" } path => "/opt/logs/%{host}/%{+YYYY-MM-dd}.log" } }
And the output of logstash.stdout:
{ "@timestamp" => "2016-10-12T03:15:27.323Z", "beat" => { "hostname" => "staging", "name" => "staging" }, "count" => 1, "fields" => nil, "input_type" => "log", "message" => "10.84.10.186 - - [12/Oct/2016:10:15:26 +0700] \"GET /bigbuckbunny_1100.mp4/seg-9-v1-a1.ts HTTP/1.1\" 200 788472 \"http://osmfhls.kutu.ru/\" \"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36\"", "offset" => 65958, "source" => "/opt/nginx/logs/hls_access.log", "type" => "nginx-access", "@version" => "1", "clientip" => [ [0] "10.84.10.186", [1] "10.84.10.186", [2] "10.84.10.186" ], "ident" => [ [0] "-", [1] "-", [2] "-" ], "auth" => [ [0] "-", [1] "-", [2] "-" ], "timestamp" => [ [0] "12/Oct/2016:10:15:26 +0700", [1] "12/Oct/2016:10:15:26 +0700", [2] "12/Oct/2016:10:15:26 +0700" ], "verb" => [ [0] "GET", [1] "GET", [2] "GET" ], "request" => [ [0] "/bigbuckbunny_1100.mp4/seg-9-v1-a1.ts", [1] "/bigbuckbunny_1100.mp4/seg-9-v1-a1.ts", [2] "/bigbuckbunny_1100.mp4/seg-9-v1-a1.ts" ], "httpversion" => [ [0] "1.1", [1] "1.1", [2] "1.1" ], "response" => [ [0] "200",