Sysmon events not getting to SOC kibana or hunt - connection issues

Hi,

i am using elasticsearch 8.6.2,
Winlogbeat 8.6.2
and sysmon 74

I am trying the ELK system. The data gets into sysmon ok. There are probably many reasons. I was pointing to output.logstash because as I understand it, it gets data from more places than elasticsearch. I saw in a couple of places that logstash data will also wind up in elasticsearch for the pipeline to SO?
I saw that you cannot have two yml outputs at the same time. I assume that as long as the elasticsearch service is running using output.logstash works.
I have sysmon, winlogbeat and elasticsearch services running.
I tried both the logstash and elasticsearch outputs, and i have the same problems:
connection issues :
“ {"file.name":"beater/winlogbeat.go","file.line":149},"message":"Winlogbeat is unable to load the ingest pipelines because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines, you can ignore this warning.","service.name":"winlogbeat","ecs.version":"1.6.0"} ”

{"file.name":"transport/logging.go","file.line":38},"message":"Error dialing dial tcp [::1]:9200: connectex: No connection could be made because the target machine actively refused it.","service.name":"winlogbeat","network":"tcp","address":"localhost:9200","ecs.version":"1.6.0"}

{"file.name":"pipeline/client_worker.go","file.line":150},"message":"Failed to connect to backoff(elasticsearch(http://localhost:9200)): Get "[http://localhost:9200](http://localhost:9200%5C/)": dial tcp [::1]:9200: connectex: No connection could be made because the target machine actively refused it.","service.name":"winlogbeat","ecs.version":"1.6.0"}

{"file.name":"pipeline/client_worker.go","file.line":150},"message":"Failed to connect to backoff(elasticsearch(http://localhost:9200)): Get "[http://localhost:9200](http://localhost:9200%5C/)": dial tcp [::1]:9200: connectex: No connection could be made because the target machine actively refused it.","service.name":"winlogbeat","ecs.version":"1.6.0"}

It was suggested That I try to use telnet to check the connection. I am thinking i should use the host IP’s for ports 5044 and 9200.

#---------------------------- Elasticsearch Output ----------------------------

#output.elasticsearch:
#Boolean flag to enable or disable the output module.
#enabled: true
#Array of hosts to connect to.
#Scheme and port can be left out and will be set to the default (http and 9200)
#In case you specify and additional path, the scheme is required: http://localhost:9200/path # IPv6 addresses should always be defined as: https://[2001:db8::1]:9200
#hosts: ["localhost:9200"]

#------------------------------ Logstash Output -------------------------------
output.logstash:
#Boolean flag to enable or disable the output module.
enabled: true
#The Logstash hosts
hosts: ["192.168.0.0/16:5044"]

thanks for any help and advice

So @iqworks

There is definitely some issues going on...

Apologies, I do not understand what you means ... gets data from more places ...

What is your overall architecture? Which One are you trying?

a) Sysmon-> winlogbeat-> elasticsearch
b) Sysmon-> logstash -> elasticsearch
c) Sysmon-> winlogbeat-> logstash -> elasticsearch

So start from the beginning and tell us what you want to accomplish
If you just want sysmon to get started either a) or b) is simplest c) is harder for sure... valid but harder

Next, If the above are the logs from winlogbeat and the output section of your winlogbeat.yml there is definitely a problem because you are configured to output to logstash and you are getting elasticsearch errors.

My first guess is that you have another .yml file in the same directory as the winlogbeat.yml perhaps you made a copy the .ymls get concatenated I have seen that before.

Second Logstash does not do anything automatic.... do you have it set up and configured?
What does your pipelines look like?

This does not look correct
hosts: ["192.168.0.0/16:5044"]

That means you are telling winlog beat to send to every host
192.168.0.0 - 192.168.255.255. --- ~65K Hosts, pretty sure you do not want to do that.
Don't use a CIDR...

Very helpful Stephen
i will use this scenario:
a) Sysmon-> winlogbeat-> elasticsearch

Here are changes I made:
changed paths

i am just showing this so you can see what i have:
image

whoa, 65K Hosts, nope dont want that.
image

here is what I put in my VM so-allow. I picked an arbitrary IP?
image

Get rid of other yml files

Have to learn about pipeline's:

Going to try this:

PS > .\winlogbeat.exe setup --pipelines

going to have to study pipelines, i will start here:

thanks again for taking the time Stephen!!

As @stephenb wrote, it will be helpful to know what you are trying to do with WinLogBeat (his list with a, b, and c).

Remember, WinLogBeat is just a beater, a log message shipper that accepts log messages from somewhere and sends them somewhere else (and will queue them up if that somewhere else is busy or not responding). I do not know WinLogBeat but I guess that it is listening in on what goes into the Windows Events database, and then sends them to either a Logstash server/service, or directly to a so-called Ingest Pipeline in/on Elasticsearch.

Depending on where you want to send stuff, you need to configure an output-plugin to do the sending. This can be an output.logstash plugin or an output.elasticsearch plugin. It might even be both since I think WinLogBeat needs (right after installation) to talk directly to Elasticsearch and deploy some templates, dashboards, lifecycle polices, Kibana data views, etc) for you to effectively use and view the data.

Just run

PS > .\winlogbeat.exe setup

Without the pipelines.. otherwise it won't load the dashboards and everything I don't know why our docs say that always just run the full setup, not individual pipelines, dashboards, etc.

Trust me on this one ! :slight_smile:

1 Like

HA )-: , I get it, will do. Valuable suggestion. Does everything else look right to you ?

Is the port "hosts: ["192.168.1.216:9200"] " here madnatory, or should I use an available port that my machine shows as open?
thanks as always.

Stephen, in fact i am not sure of the IP address and port I should be using here"
hosts: ["192.168.1.216:9200"]" ?? Maybe this is mty problem in the connection?

Hi Steph, This is what i got when i ran this command:
D:\VM WorkStation\winlogbeat>.\winlogbeat.exe setup
Exiting: couldn't connect to any of the configured Elasticsearch hosts. Errors: [error connecting to Elasticsearch at http://192.168.1.216:9200: Get "http://192.168.1.216:9200": dial tcp 192.168.1.216:9200: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.].

I feel like i am missing just a few small pieces.

thanks as always Stephen

Stephen and Jan, after thinking about it, I might as well show you the IP addresses I am using and where. Here is my VM network settings.


Here is my ipconfig results:
"D:\VM WorkStation\winlogbeat>ipconfig

Windows IP Configuration

Unknown adapter Secure VPN:

** Connection-specific DNS Suffix . :**
** Link-local IPv6 Address . . . . . : fe80::dce:a00:59f:ba1d%14**
** IPv4 Address. . . . . . . . . . . : 100.120.204.67**
** Subnet Mask . . . . . . . . . . . : 255.255.255.0**
** Default Gateway . . . . . . . . . :**

Wireless LAN adapter Local Area Connection 10:*

** Media State . . . . . . . . . . . : Media disconnected**
** Connection-specific DNS Suffix . :**

Wireless LAN adapter Local Area Connection 11:*

** Media State . . . . . . . . . . . : Media disconnected**
** Connection-specific DNS Suffix . :**

Ethernet adapter VMware Network Adapter VMnet1:

** Connection-specific DNS Suffix . :**
** Link-local IPv6 Address . . . . . : fe80::3255:1e49:c85d:d418%2**
** IPv4 Address. . . . . . . . . . . : 192.168.10.1**
** Subnet Mask . . . . . . . . . . . : 255.255.255.0**
** Default Gateway . . . . . . . . . :**

Ethernet adapter VMware Network Adapter VMnet4:

** Connection-specific DNS Suffix . :**
** Link-local IPv6 Address . . . . . : fe80::4844:5fed:e7a1:7ef1%7**
** IPv4 Address. . . . . . . . . . . : 192.168.40.1**
** Subnet Mask . . . . . . . . . . . : 255.255.255.0**
** Default Gateway . . . . . . . . . :**

Ethernet adapter VMware Network Adapter VMnet8:

** Connection-specific DNS Suffix . :**
** Link-local IPv6 Address . . . . . : fe80::af4e:7386:5851:e3bc%13**
** IPv4 Address. . . . . . . . . . . : 192.168.80.1**
** Subnet Mask . . . . . . . . . . . : 255.255.255.0**
** Default Gateway . . . . . . . . . :**

Wireless LAN adapter Wi-Fi:

** Connection-specific DNS Suffix . : lan**
** IPv6 Address. . . . . . . . . . . : 2603:8001:b102:3958::1e63**
** IPv6 Address. . . . . . . . . . . : 2603:8001:b102:3958:d7d7:4f6e:4cce:c3a2**
** Temporary IPv6 Address. . . . . . : 2603:8001:b102:3958:1033:4664:1aaf:acb3**
** Link-local IPv6 Address . . . . . : fe80::4409:2b95:a88f:5ae4%12**
** IPv4 Address. . . . . . . . . . . : 192.168.1.214**
** Subnet Mask . . . . . . . . . . . : 255.255.255.0**
** Default Gateway . . . . . . . . . : fe80::fa5b:3bff:fee0:383d%12**
** 192.168.1.1**

Ethernet adapter Bluetooth Network Connection:

** Media State . . . . . . . . . . . : Media disconnected**
** Connection-specific DNS Suffix . :**

Tunnel adapter Teredo Tunneling Pseudo-Interface:

** Connection-specific DNS Suffix . :**

  • Lin-local IPv6 Address . . . . . : e80::1829:b000:b215:d35a%17**
    ** Default Gateway . . . . . . . . . :**

D:\VM WorkStation\winlogbeat>"

I have 192.168.1.214 as a reserved IP with spectrum. When Installing security, it asked me for a static IP for my new VM, I abritraly selected 192.162.1.216 as my static IP.

Thanks both of your suggestions and advice.

Stephen and Jan, to be honest from all the documention I have read over the last few months, there was not one that told me about the back alleys I would have to take to have sysmon to show up in my SOC.
I am sure that everyone trying to use SOC will have thier own set of circumstances, so I am not blaming the documentation thats out there.

thanks again to both of you.

Hi @iqworks I have be AFK.

It is not back alleys :slight_smile:

Apologies that it has been frustrating but you did a lot of stuff before you asked questions ... Which I LOVE but sometimes it takes a little to back up if you had an error early on which may have.

PLus you are mixing VMs / Virtual Networking etc, you need to finger out what IP Elasticsearch is running on and do they have connectivity....

What host are Kibana and Elasticsearch and Winlogbeat running on.

Are they all on separate VMs? Same VMs?

How Did you install elasticsearch?

Please share your elasticsearch.yml

You / We need to figure out what IP / Host it is running on... whether it is properly configured and reachable from other host... otherwise...it is all guess work.

I have no idea what you mean by installing security.

Let's slow down and back up a bit and answer the questions I asked above.

HA, yea, "fools rush in" comes to mind.
So, I begin by answering your questions :slight_smile:

Yes, i have elasticsearch running as a service (so I think I installed it?)

it is on the same laptop.

I downloaded elasticsearch-8.6.2-windows-x86_64 and unzipped it to a folder.

I sent you the sysmon, elasticsearch and winlogbeat by including them in my email reply, let me know if you did not receive them?

thanks again for your time and advice Stephen!

I don't think email attachments go through... I don't see them... also I suspect most of your issues are related the the VMs and Associated networking.

Most of us actually use the Elastic Discuss site ... here is your thread

https://discuss.elastic.co/t/sysmon-events-not-getting-to-soc-kibana-or-hunt-connection-issues/327966

You need to understand what the host IP is for Elasticsearch and Kibana, Whether they are reachable and accepting connections.

It looks like elasticsearch is running on 129.168.1.120 according to the image at the top of the thread.

image

If so on the VM you install winlogbeat on you should try to curl that endpoint that is what you should configure the winlogbeat elasticsearch output to point to.

But your winlogbeat error seems to be pointing to some other IP...

Elasticsearch running on 129.168.1.120:9200
Winlogbeat pointing to Elasticsearch 192.168.1.216:9200

They need to line up.

Thanks Stephen, I will look at these suggestions you have given me.

Sorry you could not see my yml's. It seems like all three are two big to just dump in a reply. are there maybe certain sections of the 3 ymls you would like to see?

I just got through learning a little more about elasticsearch :
(for others)

I am going to review your last suggestions, especially about the IP address line up. I am working on another project for the next two weeks, but wil get back to this when i get back.

again, thanks Stephen for your time and suggestions

Also, I am installing it on my laptop, not a VM, is that a problem?
Here are my VMware virtual networks


Will be in touch when i get back to this project.

I gotta say I am completely confused by these two statement

Running on a laptop should be no problem... curious what OS ...

But then if you are running on a laptop what are at the VMware Virtual networks?

I am totally confused usually a laptop has 1 or perhaps 2 IPs wifi and ethernet so I have no clue what you are doing ...

But in the end if you want a beat to communicate with elasticsearch you need to put the correct elasticsearch resolvable hostname or IP in the output section in the beat....

Also there should be not problem pasting the yml in these topics people do it all the time...

Good luck let us know when you try again...

Stephen, sorry for the confusion. Of course i am installing sysmon, winlogbeat and elasticsearch on my laptop on windows 10 and 11.
I have heard it could be risky to use the public Ipv4 IP spectrum provides. So, there are certain tools you want only on a VM. I reserved this IP with spectrum and gave .214 and created an IP of .216 to use as a static IP.

I sent a copy of my vmware workstation 16 virtual networks, is this what you mean?
I am sure you can see i am a novice for all of days. I will let you know when i get back on this project.

thanks as always for your valuable time, i will try to write my questions with more clarity.

Still confused ... sorry guess I am not speaking the same language

If this is all on your laptop not sure why you would use any public IP address...

I do all sorts of stuff on my home network using the local 192.x.x.x IPs nothing public.

I run Elastic and Kibana in Docker on my Mac (you could do the same on Windows) .. then I run beats, logstash etc where I need... never leave my home network...

I am not a sysmon guru maybe it all has to do with that...

Hi jan, thanks for getting back, sorry it took so long to respond. I am trying to do Stephens a, sysmon > winlogbeat > elasticsearch.
looked at output plugins (have not seen this requirement in all of what I have read) : Output plugins | Logstash Reference [8.11] | Elastic
but there are so many.
Maybe you have a link that explains why I need a plugin. I have not seen that as a reqiuirement to have this scenario to work?

thanks for your advice or suggestions

For beats the outputs are not "plugins" (that is more of a logstash termnlogy)

I think @jba was just referring to the output options