Hi. I am new to ELK stack, and I cannot find this problem online, so I am posting here. I have a self hosted ELK stack the newest version with the agents version 9.0.3. I also have 1 agent deployed as a fleet server via fleet server policy on the same machine running the ELK stack. All machines are Ubuntu 24.04 machines and 3 of them have agent's that are enrolled via a different Linux policy. Effectively, all machines have at least these integrations via the Elastic Agent:
System Integration
Auditd Logs Integration
Sysmon for Linux Integration
I can confirm Sysmon for linux is putting it's events in /var/log/syslog on the local machine. I can also confirm the data from /var/log/syslog is entering the ELK stack, because I can see those types of event's with that sysmon structure in the "message" field in Kibana Discover from my hosts if I filter for sysmon.
The problem is it coming into the SIEM as data_stream.dataset: "system.syslog" where I think it expects it to come in as data_stream.dataset: "sysmon_linux.log" which is what the Sysmon for Linux Logs Overview dashboard is looking at, and I think that distinction makes sense to have since the event have such a different structure.
Did I do something wrong with the integration or is this intended behaviour? If it's not normal how can I fix it? I did not make any changes to local agent files, only used fleet server to enroll via policy. All machines are synced via time server, have correct localization and timezone, have auditd and sysmon installed. Auditd with recommended rules and config, sysmon with a barebones xml config for linux/ubuntu machines, which is I think 7 event categories in total. All Agents also showing with healthy status and inputs.
According to the integration it should work I think so I am not sure what I did wrong. Any help would be appreciated.
System Integration. This integration reads from /var/log/syslog so I am not surprised that all the events in /var/log/syslog in system.syslog why some are not showing up in sysmon_linux.log I am not not sure.
What I would recommend... If you want sysmon separate I suspect you should send sysmon to a separate file such as /var/log/sysmon* .... as the integration expects... then the integration will only be consuming sysmon.
Hi @stephenb Thanks for the warm welcome and for looking at my issue.
I think you are correct in suspecting that its something related to sysmon not putting its events in the correct log file. When I had initially installed sysmon, I think I did so without specifying a config file, and that seems to affect how/where it gets logged at least in ubuntu. Now that I have applied a sysmon config to all these machines, the events are still not going to /var/log/sysmon*, but that seems to have to do with how systemd/rsyslog/journald components handle this sysmon logging on Ubuntu. It seems that in my current configuration (I just made some adjustments) it's now actually going into journald as opposed to /var/log/syslog like it did before, but as you say, I think I should be able to configure that in either journald config or in the systemd service config for sysmon, that it will output to a new file named /var/log/sysmon.log.
It is interesting however this behaviour, because I have not configured anything like that in my config file:
I only just now because of your help figured out where I am able to change these settings for these integrations. I was constantly clicking on the integration itself within my policy Kibana which only has a settings menu, but it's actually the integration policy that I should edit. Because of that I also never was able to find where it actually looks, or change/verify any other system integration, that clears things up a lot thanks.
This is my Sysmon for Linux Integration version:
In my 'System' integration, I left all default logging enabled and aditionally I enabled under metrics > CPU Core.
I think I am on the right track now, I will report back once I output everything to a new file ../sysmon*. I would include more pictures, but because my account is new I can only post one.
Problem 1 is that the integration looks for /var/log/sysmon* and these files don't exist yet with my current sysmon install. I believe this is simply how ubuntu handles these logs by default, based on the fact that splunk integration for Sysmon for linux also monitors journald. Some events did enter the syslog file earlier, and I am still trying to figure out when they dont and when they do, ubuntu has a drop in file for journald that does specify it should forward to syslog. I have the feeling I now have way less events for sysmon, because I am actually using a config now. But it should be relatively straight forward to get it to a new file called /var/log/sysmon*.
Problem 2 however is that Sysmon on Linux is not reading and logging any of my monitored events at all that are set in my config. When I restart sysmon service, can see those restart events as sysmon events via journalctl just fine, but I cannot see any other events that i am trying to log. When I create a process, file, or network connection as a test on a different terminal tailing journalctl, I am not getting any events back for that. Which means that probably there is more wrong with my sysmon installation/configuration. I read that auditd needs to be set up which is working and those logs I am getting properly, so I am not sure what is wrong with my setup.
That means that since the issue is mostly likely not related to the integration anymore, I will close this topic now. I am quite positive when I get correct logging to that file, the integration will work.
My other post got deleted not sure why, but I am now closing this thread based on that info, thanks for the help.
Hi @stephenb yes I understand your thought process, my older comment got deleted (by elastic forum staff for spam, which I did not do) so it was not visible anymore and some context was lost.
The problem I had was until your first comment I was never able to find where the integration sysmon for linux was looking, because I was clicking on the integration itself in my agents policy which has only the uninstall option, instead I had to click on the integration policy, There in that menu I was indeed able to find and configure the file, until your comment I had not found or noticed this difference yet.
The rest of my problem is indeed related to my sysmon config, simply related to how ubuntu handles it's logging or that logs are not coming in correctly, which I would not ask you to fix or troubleshoot. This is why I tried to mark your answer as correct, because it seems i have some more troubleshooting to do on my end, and we can mark this thread as solved.
Actually, your post was flagged after review, but that flag was ignored. Somehow, in your response, you deleted it. (That happens; I have the audit trail.) No worries, I restored it, so we have the whole topic.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
