ElasticSearch on Azure - How to Logstash and install plugins?

Elastic Stack Elasticsearch
1

Hi,

We are trying to develop a PoC for Elastic on Azure Cloud

We installed the Elastic template from Azure Marketplace, selected 3 nodes and now we have following configuration:
1 Kibana VM
3 Data VM
3 Master VM

What we can do so far:

  • Connect and log in to Kibana (create simple index pattern from console)
  • Connect to VMs via bash ssh

What is still open and we desperately look for clear documentation:

  1. On which VM runs logstash?

  2. How to start logstash?

  3. Where to store the config files (for logstash but also if using jdbc plugin)?

  4. Installation of Kibana plugins: when trying to install for example https://github.com/Nico-DF/kibana-plugin-traffic-sg I always get 'operation not permitted' even if I am logged in as the owner of the directory
    Is it impossible to install extra plugins on VMs?

  5. Installation of ES plugings like like https://github.com/Azure/azure-diagnostics-tools/tree/master/Logstash/logstash-input-azurewadtable or https://github.com/Azure/azure-diagnostics-tools/tree/master/Logstash/logstash-input-azureblob
    On which VM it should be installed? Master-0 I assume?
    But then under which directory?

Currently the documentation I can find for setting ELK on Azure is mostly


and

But all these are not giving us answers.

Can anyone direct us to a more comprehensive (for dummies) guidelines/tutorial Or at least reply to our questions?

thx

2

I work on the Elastic Azure ARM Marketplace template so let me see if I can answer your questions:

The template currently deploys only Elasticsearch and Kibana, so if you would like to also use Logstash and Beats, you will need to install those separately. We have discussed including Logstash in the template in future and I have opened an issue on github to track this.

See above. If you're deploying Logstash to Ubuntu 16.04 on Azure, you'll probably use systemd to run Logstash.

On the VM(s) running Logstash.

Yes. ssh into the Kibana VM. Typically, you install plugins through the kibana-plugin script for a 5.x version:

sudo /usr/share/kibana/bin/kibana-plugin install <package name or URL>

and restart Kibana service afterwards

sudo service kibana restart

but that particular plugin's instructions use git to clone the repository to the plugins directory. It looks like there is an open PR for that plugin to fix an issue with Kibana 5.5.x and 5.6.x though, so you may need to wait for that to be merged before you can use it. Since we don't maintain that plugin, I would recommend keeping an eye on that github repository.

Those are logstash plugins, so you would install them onto the VM(s) that are running Logstash.

Check out the blog posts that we have about the Azure Marketplace and ARM template offering:

3

Thx a lot @forloop

After some more research and attempts I did managed yesterday to install Logstash and many plugins for both Kibana and Logstash
I'm still stuck with the jdbc plugin but I'll investigate this still a bit more

Now there is still something I still do not understand 100%
What is the best approach for Logstash?

Currently I only installed Logstash on my Kibana VM (I actually installed everything on Kibana VM only so far). Was this a mistake or correct move?
Would it be better to have Logstash installed on all my 3 Data VMs or on all my 3 Master VMs instead?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.