Host not showing up despite events being present

[norup]

I am running the elk stack on 1 ubuntu device along with filebeat, and metricbeat and this device is showing up fine with syslog data and as a host. Im running winlogbeat on another device, and shipping the logs through logstash . The events from the windows machine are showing up under the events tab, but its not showing up as a host, and it claims to only get event from filebeat, even though it clearly is getting events from the windows machine, which only runs winlogbeat.

As you can see there are events from 2 different hosts, but only teleit-vm is showing up properly

Any ideas what is the problem here?

**EDIT
I have tried running this .\winlogbeat.exe setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["172.16.10.20:9200"]' but i get the error no connection could be made because the host actively refused it
I have checked the firewall on the ubuntu machine and it is disabled. i can ping it just fine I get the occasional crash on the Elasticsearch but the errors are just gigantic and quite frankly impossible to sort through.

heres a pastebin of a small snippet of the error... https://pastebin.com/qstux58p

Read that this could be a resource problem, but i just gave the VM 64gb memory and it still breaks as soon as i go to kibana on firefox

[c1jt]

I belive this is similar to an issue I had from using Logstash as an intermediary between Winlogbeat and Elastic. Removing Logstash for windows logs solved all my issues.

Logstash seems to be messing up ECS data, or maybe metadata, idk. This weirdly causes SIEM to attempt very heavy text searching which seems to be what you pastebin logs are showing. It shouldn't happen.

If you can get Winlogbeat to send directly to Elasticsearch I think you'll be in the money. Do you have xpack Security enabled? The -E commands will need to reflect any certificate usage. Editing the .yml files will be easier tbh, commend out the logstash parts and fill in the elastic parts.

Good luck

[norup]

Thanks. Im currently trying to go around logstash. I think im honestly dealing with a couple of different issues all together. The winlogbeat cant even connect to the ES atm. I had to bind it to the correct IP (obvicoulsy), but now ES keeps breaking because of filesize limits, which it wont read from /etc/security/limits.conf for some reason.
**EDIT
I can see that it for some reason only runs with ipv6, which is probably why winlogbead cant connect to it. Not sure how i force it to start on ipv4.
This is running with default values. If i change the network.host to 0.0.0.0 or the actual IP adress, it just wont start for some reason =/

This is what im getting currently when i give it ipadresses. I just stops. My limits.conf file looks like this, so im not sure why it is still complaining about it :s

[c1jt]

Oh, ubuntu likes to ignore limits.conf...

Edit the /etc/pam.d/su file and uncomment this:

session    required   pam_limits.so

Or add it if it's not there. By default it's commented out though.

If that doesn't work, the centos containers I have don't have those elasticsearch lines in limits.conf, but doing a filter_path=**.max_file_descripts shows 1048576 so there must be some other way to setting it.

edit: Using network.hosts: 0.0.0.0 is most likely correct and I reckon the 'limits.conf' error is just the next error allong and it is getting better.

[norup]

unfortunately it already looked like this

Is this bootstrap check why it wont start?

**EDIT
I added

>     transport.host: localhost
>     transport.tcp.port: 9300

to elasticsearch.yml, and now it starts, even though its still complaining about the descriptor size, but it still only listens on ipv6 :S

It looks like its binding the addresses right in the elasticsearch startup log though =/

removed the transport part again and trying to fix the file descriptor issue.

Added things in limit.conf, in /etc/pam.d/su and common something etc. and set fs.file-size= 80000 in sysctl aswell.
Still doesnt work =/

filter_path=**.max_file_descripts didnt return anything

Managed to get this before it crashes

ulimit -Hn says the same thing. Cant change it with sudo ulimit -n 65536 says command not found...

[c1jt]

Try running the ulimit command without sudo and possibly as the elasticsearch user. I belive it's a session based setting so running as sudo will break it.

Its definitly trying to bind to local adapters, pv6 might be a backup connection as it cant find ipv4. Have you checked the VM's networking settings? What is the VM hosted on? I.e. Hyper-V, ESXI, etc.
It might be the 'limits' thing which is stopping the ipv4 listener, or that the ipv4 setup on the vm is weird for whatever reason. Kind of guessing here though.

According to this guy, adding session required pam_limits.so to the /etc/pam.d/common-session* file(s) will sort ulimits out. Classic "40 ways to do something" Linux.

edit: actually, do you know user what is elasticsearch running as? I haven't actually done a raw elasticsearch install and assumed it just ran as "elasticsearch" user. Maybe it's running as the 'siem' user and so the limits and binding privilages dont apply?

[norup]

ulimit without sudo gives permission denied. I have added the session required pam_limits.so to the /etc/pam.d/common-session* file(s), and it still didnt do anything =/
Its running on ESXI.

I dont actually know which user is running the elasticsearch, so i tried adding all the ones i could think of, including elasticsearch, siem, teleit-vm, and the * in case that works. Still ulimit gives 4096, even after restarting.

[c1jt]

Dang I think at this point it might be worth asking on a linux forum/exchange for help with the ulimits, or change your title here see if any linux guru appears. I can't see why it wouldn't work especially using * - nofile 65536 and such... unless we're both misunderstanding how it works (definitly a possibility lol, no offence)

I know this is such a cop-out answer; Have you thought about using docker on the VM? The ELK dockers are pre-built with a perfect working copy of E L or K hosted on Linux, which you then run as a container. All your .yml configs will work with the dockered instances too.
In fact, apparantly, vSphere supports dockered containers without a host, you'll need the paid version of ESXi though.

Best of luck,

[norup]

FINALLY got it working...

In Summary

If you want to increase the limit shown by ulimit -n , you should:

• Modify /etc/systemd/user.conf and /etc/systemd/system.conf with the following line (this takes care of graphical login):

DefaultLimitNOFILE=65535

• Modify /etc/security/limits.conf with the following lines (this takes care of non-GUI login):

mkasberg hard nofile 65535
mkasberg soft nofile 65535

• Reboot your computer for changes to take effect.