Custom UDP Logs Integration Not Ingesting Data

Elastic Stack Elastic Agent
1

I currently have the 'Custom UDP Logs' integration setup on an Elastic Agent. It's configured to listen on all interfaces for port 9514:

image
image779×280 19.2 KB

All other options are left as default, with the Syslog Options and custom pipeline being switched on/off with no difference made.

Wireshark was installed on the device that has the agent installed and I can see that it's receiving the logs we need on this port, however, no data is pulling into Elastic.

Since we can see the host with the agent installed is receiving the data, this seems to rule out an issue with the network configuration.Meaning that there could seemingly be an issue with is the Elastic Agent/Integration setup. However, I can't figure out what.

I'm sure I'm missing something simple but any help would be appreciated.

2

Hi @callum,

Thanks for reaching out to the team. Is there potentially a firewall running on the host that is blocking traffic to the agent process?

For example, on some Windows hosts, Windows Defender Firewall can block access on a per applications basis.

Can you describe the system you have installed the agent on in more detail?

3

Also, can you see if that UDP Port is open and listening?

4

Hi,

Appreciate the response.

The port is open and the Windows Firewall fully disabled. I can see the traffic being received through Wireshark so this shows that there isn't anything in the network blocking it.

I think I must've configured something wrong with the integration but unsure what it could be.

Edit:
Apologies I forgot to include more info about the device. It's a Windows Server that's being used as a network probe.

Many thanks,
Callum

5

Hi,

Thanks for the reply.

I can confirm that the port is open:

image
image850×156 10.1 KB

This replicates the setting in the UDP integration. I've confirmed that this changes when I change the integration settings, for example, changing from 0.0.0.0 to the IP of the device sending the logs.

I can also see that the traffic I want is being received through Wireshark. I can view each of the logs when inspecting through this. I don't have access to the GUI at the moment so unable to provide a screenshot of Wireshark but it doesn't appear to be a network issue.

Many thanks,
Callum

6

So when you go to
Kibana - Discover
And pick logs-*
And set the time picker to last 30 days. Do you see anything?

Kibana - Dev Tools

GET _cat/indices/*logs*/?v

what is the results

Oh and welcome to the community. There's something basic going on here as well.

You should also be able to look at the elastic agent filebeat logs and see if there's any errors.

You can do that and discover too

7

Thanks for the reply.

I can't see anything in Discover. I've set the integration to use the following dataset and tags:

image
image890×654 42.1 KB

I'm unable to upload more than one image but when searching for these there are no results. I tried searching for:
tags : "kerio" or event.dataset : "kerio.firewall"

I would post the full results of GET _cat/indices/*logs*/?v but the index that this should be writing to doesn't exist. I believe that it should be writing to logs-kerio.firewall-*namespace* but there's nothing that' similar to this name.

I've copied in some of the error logs from Filebeat over the time that this integration has been configured:

There was a lot of the following error around 6-7 hours after the integration was configured:
[elastic_agent.filebeat][error] failed to publish events: temporary bulk send failure

Then there was an assortment of errors similar to these for a few days afterward:
[elastic_agent.filebeat][error] Error dialing lookup [redacted].eu-west-1.aws.found.io: getaddrinfow: This is usually a temporary error during hostname resolution and means that the local server did not receive a response from an authoritative server.

[elastic_agent.filebeat][error] Failed to connect to backoff(elasticsearch(https://[redacted].eu-west-1.aws.found.io:443)): Get "https://[redacted].eu-west-1.aws.found.io:443": lookup [redacted].eu-west-1.aws.found.io: getaddrinfow: This is usually a temporary error during hostname resolution and means that the local server did not receive a response from an authoritative server.

[elastic_agent.filebeat][error] Error dialing lookup [redacted].eu-west-1.aws.found.io: no such host

[elastic_agent.filebeat][error] failed to perform any bulk index operations: Post "https://[redacted].eu-west-1.aws.found.io:443/_bulk?filter_path=errors%2Citems.%2A.error%2Citems.%2A.status": lookup [redacted].eu-west-1.aws.found.io: no such host

[elastic_agent.filebeat][error] failed to publish events: Post "https://[redacted].eu-west-1.aws.found.io:443/_bulk?filter_path=errors%2Citems.%2A.error%2Citems.%2A.status": lookup [redacted].eu-west-1.aws.found.io: no such host

The device has been pulling in logs from other integrations without any issues.

Is there any documentation on how the Custom UDP/TCP Logs integrations should be setup? I have a feeling I must have misconfigured something but unsure where else to look.

Many thanks,
Callum

8

So, first, you said you left everything else default, but you changed the data stream and added an ingest pipeline. :slight_smile: Those are significant changes, that come with assumption that the data arriving will be in the correct format. I assume you know what you are doing with these.

It may not be the issue, but I would have left the defaults until you got the data flowing...

You can just download the tar.gz of filebeat onto the host and try the udp input ... Agent is basically the same under the covers... run it in the foreground... and you should see the console logs... you can run it with -d * option it will show lots of output...

All those connectivity issues... hmmm not so good....

When you say other integrations are working... is that with the same agent on the same host?