Installing Beats (Filebeat, Auditbeat, etc) on remote host

[mokotoy (Mark)]

I have an ELK stack install on a RockNSM 2.5 machine with Elastic version 7.6

I have the server running and I want to install beats on other hosts in my network. I'm running into issues running the beats setup as it keeps rejecting the connection.

My elastic server is 192.168.1.27

I have the following in my auditbeat.yml file on my host computer (Ubuntu server)

output.elasticsearch:
 hosts: ["192.168.1.27:9200"]

setup.kibana:
  host: "192.168.1.27:5601"

To allow connection other than the default localhost, I changed my network.hosts in /etc/elasticsearch/elasticsearch.yml to

 network.host: 0.0.0.0

Additionally, I had to add the following lines:

  discovery.seed_hosts: ["host1"]
  cluster.initial_master_nodes: ["node-1"]

note: “host1” = node.name (same as node.name in the yml file)

and comment out the following:

#discovery.type: single-node

That worked. I tested it with netcat and succeeded.

I ran into another problem with Kibana when I ran the sudo auditbeat setup

I get: "connection refused"

    Overwriting ILM policy is disabled. Set `setup.ilm.overwrite:true` for enabling.

    Index setup finished.
    Loading dashboards (Kibana must be running and reachable)
    Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to 
    http://192.168.1.27:5601/api/status fails: fail to execute the HTTP GET request: Get 
    http://192.168.1.27:5601/api/status: dial tcp 192.168.1.27:5601: connect: connection refused. 
    Response: .

I looked around online and found that I have to define server.host in /etc/kibana/kibana.yml to the Kibana’s IP address otherwise its default is localhost.

I edited the kibana.yml and added server.host: 192.168.1.27 to look like below:

server.port: 5601
server.name: "RockNSM"
server.host: 192.168.1.27
server.defaultRoute: "/app/kibana#/dashboard/6151e9d0-bf83-11e9-85bb-3b744f61312d"
elasticsearch.hosts: "http://127.0.0.1:9200"

I ran sudo auditbeat setup on the host machine and voila! it worked!

Overwriting ILM policy is disabled. Set `setup.ilm.overwrite:true` for enabling.
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards

or at least it told me it worked… HOWEVER…

Now I can’t connect to Kibana. Kibana is running, even when i restart it, everything is running. I go to the <Kibana's IP>/app/kibana#/ and I get a 503 error .

I do a netstat -ano and I get a TIME_WAIT on the state of my connection

I went back to the kibana.yml file and tried to change this

elasticsearch.hosts: "http://127.0.0.1:9200"

to this:

elasticsearch.hosts: "http://192.168.1.27:9200"

and that didn’t work…

In the end, I changed the kibana.yml file back to normal and commented out the server.host

server.port: 5601
server.name: "RockNSM"
#server.host: 192.168.1.27
server.defaultRoute: "/app/kibana#/dashboard/6151e9d0-bf83-11e9-85bb-3b744f61312d"
elasticsearch.hosts: "http://127.0.0.1:9200"

I got my Kibana dashboard back but the my auditbeat host is gone. back to square 1!

I’m once again out of things to try. looking for guidance. Just trying to deploy beats to hosts. I run into similar problems with other beats deployment. Thanks!

-moki