I am running the elk stack on 1 ubuntu device along with filebeat, and metricbeat and this device is showing up fine with syslog data and as a host. Im running winlogbeat on another device, and shipping the logs through logstash . The events from the windows machine are showing up under the events tab, but its not showing up as a host, and it claims to only get event from filebeat, even though it clearly is getting events from the windows machine, which only runs winlogbeat.
As you can see there are events from 2 different hosts, but only teleit-vm is showing up properly
**EDIT
I have tried running this .\winlogbeat.exe setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["172.16.10.20:9200"]' but i get the error no connection could be made because the host actively refused it
I have checked the firewall on the ubuntu machine and it is disabled. i can ping it just fine I get the occasional crash on the Elasticsearch but the errors are just gigantic and quite frankly impossible to sort through.
I belive this is similar to an issue I had from using Logstash as an intermediary between Winlogbeat and Elastic. Removing Logstash for windows logs solved all my issues.
Logstash seems to be messing up ECS data, or maybe metadata, idk. This weirdly causes SIEM to attempt very heavy text searching which seems to be what you pastebin logs are showing. It shouldn't happen.
If you can get Winlogbeat to send directly to Elasticsearch I think you'll be in the money. Do you have xpack Security enabled? The -E commands will need to reflect any certificate usage. Editing the .yml files will be easier tbh, commend out the logstash parts and fill in the elastic parts.
Thanks. Im currently trying to go around logstash. I think im honestly dealing with a couple of different issues all together. The winlogbeat cant even connect to the ES atm. I had to bind it to the correct IP (obvicoulsy), but now ES keeps breaking because of filesize limits, which it wont read from /etc/security/limits.conf for some reason.
**EDIT
I can see that it for some reason only runs with ipv6, which is probably why winlogbead cant connect to it. Not sure how i force it to start on ipv4.
This is running with default values. If i change the network.host to 0.0.0.0 or the actual IP adress, it just wont start for some reason =/
This is what im getting currently when i give it ipadresses. I just stops. My limits.conf file looks like this, so im not sure why it is still complaining about it :s
Or add it if it's not there. By default it's commented out though.
If that doesn't work, the centos containers I have don't have those elasticsearch lines in limits.conf, but doing a filter_path=**.max_file_descripts shows 1048576 so there must be some other way to setting it.
edit: Using network.hosts: 0.0.0.0 is most likely correct and I reckon the 'limits.conf' error is just the next error allong and it is getting better.
Try running the ulimit command without sudo and possibly as the elasticsearch user. I belive it's a session based setting so running as sudo will break it.
Its definitly trying to bind to local adapters, pv6 might be a backup connection as it cant find ipv4. Have you checked the VM's networking settings? What is the VM hosted on? I.e. Hyper-V, ESXI, etc.
It might be the 'limits' thing which is stopping the ipv4 listener, or that the ipv4 setup on the vm is weird for whatever reason. Kind of guessing here though.
According to this guy, adding session required pam_limits.so to the /etc/pam.d/common-session* file(s) will sort ulimits out. Classic "40 ways to do something" Linux.
edit: actually, do you know user what is elasticsearch running as? I haven't actually done a raw elasticsearch install and assumed it just ran as "elasticsearch" user. Maybe it's running as the 'siem' user and so the limits and binding privilages dont apply?
ulimit without sudo gives permission denied. I have added the session required pam_limits.so to the /etc/pam.d/common-session* file(s), and it still didnt do anything =/
Its running on ESXI.
I dont actually know which user is running the elasticsearch, so i tried adding all the ones i could think of, including elasticsearch, siem, teleit-vm, and the * in case that works. Still ulimit gives 4096, even after restarting.
Dang I think at this point it might be worth asking on a linux forum/exchange for help with the ulimits, or change your title here see if any linux guru appears. I can't see why it wouldn't work especially using * - nofile 65536 and such... unless we're both misunderstanding how it works (definitly a possibility lol, no offence)
I know this is such a cop-out answer; Have you thought about using docker on the VM? The ELK dockers are pre-built with a perfect working copy of E L or K hosted on Linux, which you then run as a container. All your .yml configs will work with the dockered instances too.
In fact, apparantly, vSphere supports dockered containers without a host, you'll need the paid version of ESXi though.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
