Data from Splunk not reaching Ealsticsearch server through Apache Loadbalancer

[sumith_naduvath]

Hi All,

I am trying to setup an Elasticsearch and Kibana on my server and the input data for Elasticsearch is a splunk log forwarder. The splunk is being managed by a third party company and they have configured the log forwarding to my Elasticsearch server on port 9200.

However I am not seeing any data on Elasticsearch. Here is how we have configured the Elastic Search.

On top, we have a h/w load balancer on which we have configured the public IP (internet facing). From there the request will fwd to webserver(Apache) and below to that we have the application server(elastic). Both Load balancer and Apache are being managed by a different team.

Is there any extra configuration required to get the log onto application server? I am able to access Elasticsearch in browser with publicIP:9200 but there is no data being received by elasticsearch.

Any insights would be greatly appreciated.

Regards,
Sumith

[Christian_Dahlqvist]

I would be surprised if the Splunk log forwarder supported the Elasticsearch bulk format, so suspect you will need to send this data through Logstash, potentially using a TCP input plugin.

[sumith_naduvath]

Hi Christian,

Thanks for your update. I tried adding logtash as mentioned. However now the elastic search is failing with below error.

ADMINSHELL :/etc/elasticsearch # /etc/init.d/elasticsearch status
elasticsearch dead but subsys locked
ADMINSHELL :/etc/elasticsearch #

To add logstash to the system I am doing the below configuration. Logstash will listen on port 9200 and outputs the result to 9300 which in turn will be used by elasticsearch.

Logstash conf :

ADMINSHELL :/usr/share/logstash # cat simple.conf
input {
tcp {
port => 9200
}
}

output {
elasticsearch {
hosts => ["localhost:9300"]
}
}
ADMINSHELL :/usr/share/logstash #

Elasticsearch conf:

network.bind_host: 0.0.0.0
transport.host: localhost
transport.tcp.port: 9300
#Set a custom port for HTTP:
http.port: 9300

I am not able to make elasticsearch running with port 9300. Also with the above logstash configuration I am not sure what service will be running in port 9200 to scan the incoming logs.

Regards,
Sumith

[Christian_Dahlqvist]

Logstash communicates via HTTP, so you need to use port 9200.

[sumith_naduvath]

Hi Christian,

Logstash is configured on port 9200. Since the port is already occupied I have to make elastic search to listen on port 9300. Any issue with this?

[Christian_Dahlqvist]

Do not use port 9200 for the TCP input as this will be misleading. If you do, use e.g. 9201 for the Elasticsearch HTTP port, as it must be different from the transport port.