I am currently facing a problem.
I'm planning to set up an ELK service on a redhat ocp platform and install metricbeat on the VM to send the data to a logstash pod in ocp, I'm currently doing the following.
- a pod -> elasticsearch:8.3.11
- a pod -> kibana:8.3.11
- a pod -> logstash:8.3.11
logsash has a service configured (as attached) and this sevice has an ingress configured to open the service externally (as attached).
I am trying to send data from VM to logstash pod in ocp via metricbeat via ingress FQDN, but it is failing, here is what I am trying to find out. - I tried to start a metricbeat:8.3.11 pod in the same ocp, under the same project, and set the data target to logstash's service, which worked, logstash did receive the data and imported it into easticsearch, but when I changed the target from service to ingress FQDN, the data was sent to the logstash pod. service to ingress FQDN, it fails, the failure log is attached (metricbeat-in-ocp-use-ingress-fail.log). 2.
- when I use metricbeat to export data to ingress from VM in VMware platform, I get the same error as attached.
- I think my ingress is functioning fine because I tried to send a request to ingress using curl and it worked, the command is as follows: echo '{"@timestamp": "2023-12-27T07:20:20.355Z", "message": "Test log message" }' | curl -X POST -H "Content-Type: application/json" -d @- ifa-metric.apps.ocp-u.nextbank.com.tw:80 , after running I was able to query this data via kibana.
What I want to know is, it seems that after the above test, my elasticserch&logstash&metribeat don't seem to have too much problem with the settings, I think the problem may be in the ingress, all the data from the metricbeat will fail after going through the ingress, is there any way to help? Is there anything that can be done to help troubleshoot this?