All other options are left as default, with the Syslog Options and custom pipeline being switched on/off with no difference made.
Wireshark was installed on the device that has the agent installed and I can see that it's receiving the logs we need on this port, however, no data is pulling into Elastic.
Since we can see the host with the agent installed is receiving the data, this seems to rule out an issue with the network configuration.Meaning that there could seemingly be an issue with is the Elastic Agent/Integration setup. However, I can't figure out what.
I'm sure I'm missing something simple but any help would be appreciated.
The port is open and the Windows Firewall fully disabled. I can see the traffic being received through Wireshark so this shows that there isn't anything in the network blocking it.
I think I must've configured something wrong with the integration but unsure what it could be.
Edit:
Apologies I forgot to include more info about the device. It's a Windows Server that's being used as a network probe.
This replicates the setting in the UDP integration. I've confirmed that this changes when I change the integration settings, for example, changing from 0.0.0.0 to the IP of the device sending the logs.
I can also see that the traffic I want is being received through Wireshark. I can view each of the logs when inspecting through this. I don't have access to the GUI at the moment so unable to provide a screenshot of Wireshark but it doesn't appear to be a network issue.
I'm unable to upload more than one image but when searching for these there are no results. I tried searching for: tags : "kerio" or event.dataset : "kerio.firewall"
I would post the full results of GET _cat/indices/*logs*/?v but the index that this should be writing to doesn't exist. I believe that it should be writing to logs-kerio.firewall-*namespace* but there's nothing that' similar to this name.
I've copied in some of the error logs from Filebeat over the time that this integration has been configured:
There was a lot of the following error around 6-7 hours after the integration was configured: [elastic_agent.filebeat][error] failed to publish events: temporary bulk send failure
Then there was an assortment of errors similar to these for a few days afterward: [elastic_agent.filebeat][error] Error dialing lookup [redacted].eu-west-1.aws.found.io: getaddrinfow: This is usually a temporary error during hostname resolution and means that the local server did not receive a response from an authoritative server.
[elastic_agent.filebeat][error] Failed to connect to backoff(elasticsearch(https://[redacted].eu-west-1.aws.found.io:443)): Get "https://[redacted].eu-west-1.aws.found.io:443": lookup [redacted].eu-west-1.aws.found.io: getaddrinfow: This is usually a temporary error during hostname resolution and means that the local server did not receive a response from an authoritative server.
[elastic_agent.filebeat][error] Error dialing lookup [redacted].eu-west-1.aws.found.io: no such host
[elastic_agent.filebeat][error] failed to perform any bulk index operations: Post "https://[redacted].eu-west-1.aws.found.io:443/_bulk?filter_path=errors%2Citems.%2A.error%2Citems.%2A.status": lookup [redacted].eu-west-1.aws.found.io: no such host
[elastic_agent.filebeat][error] failed to publish events: Post "https://[redacted].eu-west-1.aws.found.io:443/_bulk?filter_path=errors%2Citems.%2A.error%2Citems.%2A.status": lookup [redacted].eu-west-1.aws.found.io: no such host
The device has been pulling in logs from other integrations without any issues.
Is there any documentation on how the Custom UDP/TCP Logs integrations should be setup? I have a feeling I must have misconfigured something but unsure where else to look.
So, first, you said you left everything else default, but you changed the data stream and added an ingest pipeline. Those are significant changes, that come with assumption that the data arriving will be in the correct format. I assume you know what you are doing with these.
It may not be the issue, but I would have left the defaults until you got the data flowing...
You can just download the tar.gz of filebeat onto the host and try the udp input ... Agent is basically the same under the covers... run it in the foreground... and you should see the console logs... you can run it with -d * option it will show lots of output...
All those connectivity issues... hmmm not so good....
When you say other integrations are working... is that with the same agent on the same host?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
Those are significant changes, that come with assumption that the data arriving will be in the correct format. I assume you know what you are doing with these.