System module working but not logging all lines in /var/log/messages


(Chris Searle) #1

Running 6.4 on debian stretch.

OK - so I think I have the system module configured (default paths etc). New to trying the modules system - it's been filebeats and logstash up to now.

In kibana I can see the dashboards and they do show live data - and I can see info from both /var/log/syslog and /var/log/messages data under discover.

But - not all lines from /var/log/messages.

Anything that looks like

timestamp host program[pid]: message

or

timestamp host program: message

work fine.

But anything from the kernel - that just doesn't seem to make it to ES/kibana

The kernel lines look like

timestamp host kernel: [9999999.999999] message

If I look at the module code it seems to show the following patterns:

"patterns": [
    "%{SYSLOGTIMESTAMP:system.syslog.timestamp} %{SYSLOGHOST:system.syslog.hostname} %{DATA:system.syslog.program}(?:\\[%{POSINT:system.syslog.pid}\\])?: %{GREEDYMULTILINE:system.syslog.message}",
    "%{SYSLOGTIMESTAMP:system.syslog.timestamp} %{GREEDYMULTILINE:system.syslog.message}"
],

Even if the first line doesn't match (since there is no : after the ] in the kernel lines (though surely it should interpret that as an entry without PID) - I can't see why the second option isn't working (even if it wouldn't give me hostname and program and PID - just getting the line in the message would help me correlate it with other things going on.

But I cannot find these lines in the discover tab. And I can't find anything in the filebeat log telling me that anything is going wrong - just normal startup stuff and metrics info.

I get the feeling that I'm missing something obvious but it's getting late and I've been staring at this for so long I'm probably overlooking it.

Anyone can point me in the right direction?


(Steffen Siering) #2

Have you checked Elasticsearch logs for errors?

Can you try running filebeat with -d '*' to enable debug logs?


(Chris Searle) #3

Thanks for the suggestions. Will check when I get to work tomorrow.


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.