Logstash aggregate filter doesn't aggregate all messages


(alex) #1

i have apache wsgi log:

[Sun Mar 05 14:26:12.858516 2017] [wsgi:error] [pid 11254:tid 140473096181504]
[Sun Mar 05 14:26:12.859199 2017] [wsgi:error] [pid 11254:tid 140473096181504] /usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py:852: I
nsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.i
o/en/latest/advanced-usage.html#ssl-warnings
[Sun Mar 05 14:26:12.859210 2017] [wsgi:error] [pid 11254:tid 140473096181504]   InsecureRequestWarning)
[Sun Mar 05 14:26:12.859215 2017] [wsgi:error] [pid 11254:tid 140473096181504]
[Sun Mar 05 14:26:12.859640 2017] [wsgi:error] [pid 11254:tid 140473087788800] /usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py:852: I
nsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.i
o/en/latest/advanced-usage.html#ssl-warnings
[Sun Mar 05 14:26:12.859650 2017] [wsgi:error] [pid 11254:tid 140473087788800]   InsecureRequestWarning)

and i want to combine( aggregate) all log lines with same id for example [pid 11254:tid 140473096181504] to one event.
my config:

input {
beats {
port => 5044
tags => [ "beats" ]
}
filter {
if [type] == "django" {
grok {
match => [
"message", "%{SYSLOG5424SD:timestamp}%{SPACE}%{SYSLOG5424SD:deamon}%{SPACE}%{SYSLOG5424SD:id}%{GREEDYDATA:text}" ]
}
aggregate {
task_id => "%{id}"
code => "
map['id'] = event.get('id')
map['log'] ||= []
map['log'] << {'text' => event.get('text')}
map['type'] = event.get('type')
map['host'] = event.get('host')
event.cancel()
"
push_previous_map_as_event => true
timeout => 160
}
output {

if "beats" in [tags] {
elasticsearch {
hosts => ["10.0.0.1:9200"]
manage_template => false
sniffing => false
index => "%{[@metadata][beat]}-%{[type]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
user => "elastic"
password => "changeme"
}
}

as the result some logs combine some not
Can anybody help me? thanks


(system) #2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.