Ship Logs from application server to ELK server with beats

[vijazy (vijayakumar)]

Hi ELK team,

I am building log analyzer for production environment. My scenario is, i need to setupelasticsearch,Logstash and kibana on one centos7 server which is going to use ELK server another one is apache server(remote server).

I have configured following like

(i).ELK server - elasticsearch,kibana ,logstash with nginx proxy
(ii).Application server(apache server)- installed beats on apache server

(i).ELK server configuration

Elasticsearch Configuration

vi /etc/elasticsearch/elasticsearch.yml

network.host: localhost

http.port: 9200

Kibana Dashboard Configuration

vi /etc/kibana/kibana.yml

server.port: 5601

server.host: "localhost"

elasticsearch.hosts: ["http://localhost:9200"]

Nginx Configuration:

server {

listen 80;

server_name 172.xx.xx.xx;

location / {

proxy_pass http://localhost:5601;

proxy_http_version 1.1;

proxy_set_header Upgrade $http_upgrade;

proxy_set_header Connection 'upgrade';

proxy_set_header Host $host;

proxy_cache_bypass $http_upgrade;

}

}

Logstash Configuration

cat /etc/logstash/conf.d/02-beats-input.conf

input {

beats {

port => 5044

}

}

I have configured filtersection too..

cat /etc/logstash/conf.d/30-elasticsearch-output.conf

output {

elasticsearch {

hosts => ["localhost:9200"]

sniffing => true

manage_template => false

index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"

}

}

ii) Application server Beat configuration

I have installed beats in application server and ships logs with logstash.

Filebeat configuration:

cat /etc/filebeat/filebeat.yml

output.logstash:

#the logstash hosts

hosts: ["172.xx.xx.xx:5044"]

here, i am not sure how to load filebeat index template, index and Dashboard in kibana. When i try to run
"filebeat setup" command from apache server, i got following error.

[root@webserver ~]# filebeat setup
Exiting: Index management requested but the Elasticsearch output is not configured/enabled.

I know if i configure filebeat with elasticsearch output, i can load indexes directly to kibana. But i don't want apache server directly contact with elasticsearch.

Can someone suggest how can i load "index, index template and dashboards" directly from apache server to kibana while configured beats output to logstash.

I have searched lot of sites, but haven't get any answers. Is there any other alternative method(like import method) to load apache server filebeat index in kibana and elk server.

I am eager to wait someone give solution for this.

Thanks
vijazy