Openstack log filter

Elastic Stack Logstash
1

Hello guys,

I'm having a problem where I have been stuck for the past 3 days if you would be able to help me I would appreciate very much :smiley:

I'm trying to filter logs coming from openstack more precisely from the nova service, but I want to ignore logs that came with the word DEBUG or INFO, so, up untill now this is my configuration:
/etc/logstash/conf.d/01-logstash-beats-input.conf:
input {
beats {
port => 5044
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash.crt"
ssl_key => "/etc/pki/tls/private/logstash.key"
}
}

/etc/logstash/conf.d/15-openstack-filter.conf
filter {
grok {
match => [ "message", "%{DATE:date} %{TIME:time} %{POSINT:openstack_pid} %{LOGLEVEL:loglevel}" ]
patterns_dir => ["/etc/logstash/conf.d/patterns"]
}
}

/etc/logstash/conf.d/30-elasticsearch-output.conf
output {
elasticsearch {
hosts => "localhost:9200"
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
}
}

The grok file:
/etc/logstash/conf.d/patterns/grok
OPENSTACK_PID ( %{POSINT:pid:int})?

LOGLEVEL ([A-a]lert|ALERT|[A-a]udit|AUDIT|[T|t]race|TRACE|[D|d]ebug|DEBUG|[N|n]otice|NOTICE|[W|w]arn?(?:ing)?|WARN?(?:ING)?|[E|e]rr?(?:or)?|ERR?(?:OR)?|[C|c]rit?(?:ical)?|CRIT?(?:ICAL)?|[F|f]atal|FATAL|[S|s]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)

Example of logs:
2019-05-23 11:23:08.623 19643 INFO nova.api.openstack.placement.requestlog [req-70f52a92-06a1-44b5-8a3a-268d0d8fac3b 8f5dcc068c684184b31f45ea62e09d74 cd495fc09f024f38aba7565fe1041b50 - default default] 192.0.2.1 "GET /placement/resource_providers/a9714c16-c34b-46a8-95dd-8dd28ec1ba33/inventories" status: 200 len: 514 microversion: 1.0
2019-05-23 11:23:35.008 5478 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 856, in query
2019-05-23 11:23:35.008 5478 ERROR oslo_db.sqlalchemy.engines self._affected_rows = self._read_query_result(unbuffered=unbuffered)

Up untill now Kibana is showing me everything, ERROR INFO, everything when I'm trying to ignore at least the logs with INFO LOGLEVEL

Can anyone please help me?

Thank you
Best regards

2

If you want to discard the events with some log levels you could use

if [loglevel] in [ "INFO", "DEBUG" ] { drop {} }
3

Hello again,
I'm sorry to insist on this and let me just add that it worked.
But now I have another problem, I have another machine sending log messages to logstash with INFO that I want to register, so I was thinking if there is a way to filter this messages on the client machine using Filebeat (I think it makes more sense to filter in client and send less "garbage"
I tried this on "filebeat.yml":

Exclude lines. A list of regular expressions to match. It drops the lines that are

matching any regular expression from the list.

#exclude_lines: ['^DBG']
processors:
- drop_event:
when:
regexp:
message: "INFO | DEBUG"

It doesn't work
The idea is to drop INFO or DEBUG messages

Thank you again for the help @Badger

4

I resolved this with

Exclude lines. A list of regular expressions to match. It drops the lines that are

matching any regular expression from the list.

#exclude_lines: ['^DBG']
processors:
- drop_event:
when:
regexp:
message: "INFO|DEBUG"

Thank you very much for the help

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.