Kibana unable to parse syslog logs

[ishan.abhinit (ishan)]

I have a text file which contains data in the below format (syslog).

Oct  9 2019 23:39:37 myrtle sshd[41925]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=221.49.202.135  user=root
Oct  9 2019 23:39:37 myrtle sshd[41925]: Failed password for root from 221.49.202.135 port 1930 ssh2
Oct  9 2019 23:39:38 myrtle sshd[41927]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=221.49.202.135  user=root
Oct  9 2019 23:39:38 myrtle sshd[41927]: Failed password for root from 221.49.202.135 port 55212 ssh2
Oct 10 2019 04:28:55 myrtle sshd[41931]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=108.150.44.62  user=root
Oct 10 2019 04:28:55 myrtle sshd[41931]: Failed password for root from 108.150.44.62 port 17735 ssh2
Oct 10 2019 04:28:56 myrtle sshd[41936]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=108.150.44.62  user=root
Oct 10 2019 04:28:56 myrtle sshd[41936]: Failed password for root from 108.150.44.62 port 54304 ssh2
Oct 10 2019 04:28:57 myrtle sshd[41939]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=108.150.44.62  user=root
Oct 10 2019 04:28:57 myrtle sshd[41939]: Failed password for root from 108.150.44.62 port 33925 ssh2
Oct 10 2019 04:28:58 myrtle sshd[41941]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=108.150.44.62  user=root

I am trying to get this parsed on Kibana by uploading a file through Kibana interface. I keep getting the error that "File structure cannot be determined". I tried these override settings:

Number of line to sample: 1000
Data Format: delimited
Delimiter: space
Quote character: not applicable
Timestamp format: MMM dd yyyy HH:mm:ss

NOTE: This worked in my older Kibana setup which was similar to this current one

Please assist.

[stephenb (Stephen Brown)]

What version are you on?

I just tried on 8.7.0 and it loaded without setting anything... and it picked up the data correctly and parsed out the IP and put the rest in a message field

[ishan.abhinit (ishan)]

Exactly, it worked on my previous installation of Kibana. I installed Kibana from here from RPM repository.

The Elasticsearch version is 8.7.0. Did I do something wrong this time?

[stephenb (Stephen Brown)]

I don't know. I literally used your log lines and they parsed and ended up in Kibana using the system module.

Those are your log lines. I just put them in the file. Pointed the system module at it.

Made sure I ran setup .... Did you run set up?

filebeat setup -e

Then ran filebeat and those are the results I posted.

I didn't make any other changes

[ishan.abhinit (ishan)]

I am trying to parse this using Kibana user interface though.

I even tried to parse it using the Filebeat system module. Weird thing is - I have another log file (apache log file which I am parsing using apache module) and I am able to see all the apache log files in Kibana. Its just this syslog file is not showing up in the Kibana. Both the files haves same date ranges.

I had first setup apache files, then ran filebeat setup -e
Then I updated system modules, and again ran filebeat setup -e.

[stephenb (Stephen Brown)]

@ishan.abhinit

m trying to parse this using Kibana user interface though.

Ohh sorry I am answering toooo many...

I loaded that through the Upload File in Kibana 8.7.0 All Defaults

As you asked ... I did not set any delimeters etc..etc..

Apache won't work those are not apache logs

There are different parsers...

What I showed above I did with the File Uploader

You are doing something basic wrong..

Did you clean out the index / delete the index and try to use Upload All Defaults New Index?

[ishan.abhinit (ishan)]

I too did the same. Loaded them through the Upload File in Kibana with defaults. When default settings did not parse it, then I tried to set delimiters.

Apache logs are working perfectly in Kibana. Kibana is no able to parse system logs.

[stephenb (Stephen Brown)]

Hmmm

Step by Step

File Uploader

Contents of file

$ cat syslog.log
Oct  9 2019 23:39:37 myrtle sshd[41925]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=221.49.202.135  user=root
Oct  9 2019 23:39:37 myrtle sshd[41925]: Failed password for root from 221.49.202.135 port 1930 ssh2
Oct  9 2019 23:39:38 myrtle sshd[41927]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=221.49.202.135  user=root
Oct  9 2019 23:39:38 myrtle sshd[41927]: Failed password for root from 221.49.202.135 port 55212 ssh2
Oct 10 2019 04:28:55 myrtle sshd[41931]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=108.150.44.62  user=root
Oct 10 2019 04:28:55 myrtle sshd[41931]: Failed password for root from 108.150.44.62 port 17735 ssh2
Oct 10 2019 04:28:56 myrtle sshd[41936]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=108.150.44.62  user=root
Oct 10 2019 04:28:56 myrtle sshd[41936]: Failed password for root from 108.150.44.62 port 54304 ssh2
Oct 10 2019 04:28:57 myrtle sshd[41939]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=108.150.44.62  user=root
Oct 10 2019 04:28:57 myrtle sshd[41939]: Failed password for root from 108.150.44.62 port 33925 ssh2
Oct 10 2019 04:28:58 myrtle sshd[41941]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=108.150.44.62  user=root

Load the File: No Special Settings

Nothing Touched

Click On Discover

If I use the filebeat system module with syslog input it does not parse your file... it does parse my syslog on my Mac.

[ishan.abhinit (ishan)]

Thank you. This worked. I was earlier loading it from the interface on the 'welcome home' page and it was not working. Do you know what could be the reason?

[stephenb (Stephen Brown)]

Assuming you mean this, it works exactly the same for me.