Deployment architecture of ELK stack in on premise Windows Server

I have found myself as an administrator in an environment where ELK stack is used for log aggregation. We need to be pulling in Syslog data from our Network devices, virtualization cluster, and logs from our Windows / Linux machines using Elastic Agents.

A previous administrator had setup ELK stack on a Windows 2019 server. Here is some of the only documentation left behind pertaining to upgrading the ELK components, in an air gapped scenerio.

Elastic Upgrade

  1. Download Elasticsearch, Kibana and Logstash from their website

  2. Place downloads in S:\Elastic\

  3. Log into Server-ELS01

  4. Unzip each ZIP file to C:\ElastickStack (They will have folder names like (-)

  5. Stop all elastic Services

  6. Rename Elastic Folders to .old

  7. Elasticsearch → Elasticsearch.old

  8. Kibana → Kibana.old

  9. Logstash → Logstash.old

  10. Rename new folders to original folder name

  11. Elasticsearch-x.x.x → Elasticsearch

  12. Kibana-x.x.x → Kibana

  13. Logstash-x.x.x → Logstash

  14. Reinstall the elastic service with the commands: (If I don't do this I get Java errors on startup)

  15. cd \ElastickStack\elasticsearch\bin

  16. elasticsearch-service.bat remove

  17. elasticsearch-service.bat install

  18. Set the service to automatic startup

  19. Start the Elasticsearch service

  20. Watch logs for errors in c:\Programdata\Elastic\Elasticsearch\logs

  21. Once running, start the Kibana service

  22. This will take a few minutes

  23. Watch logs for errors in c:\Programdata\Elastic\kibana\logs

  24. You should see and entry in the log "[INFO ][status] Kibana is now available"

  25. You can validate by logging into the web interface
    Once running, start the Logstash service
    Watch logs for errors in c:\Programdata\Elastic\logstash\logs

Java isn't even installed on the server. I am confused because there are elastic binaries that live under %programdata%, but the upgrade instructions show to update the elastic binaries in C:\ElasticStack where these files do exist. I need some general guidance on how you would do this basic single node deployment on windows 2019 so that I can try to piece this together. It seems like things are configured to send Elasticsearch data to the D drive and then backup snapshots onto our network storage, but also seems like logstash is not properly moving data from the queue.

Has anyone had experience with this basic architecture deployment who can explain generally how things should work? Resources you would point me to?

I think elastic ships the proper Java bits.

I thought that may be the case, thanks for clarifying!

Hi @grootlin Welcome to the community.

I just saw this ... be very careful with instructions like that ... did you already try to execute them... if not... I would not... I would read our official documentation on Upgrading the Stack.

Also, it is VERY important to understand What Version you are coming from and Going to. Not all upgrades are equal...

And yes JVM comes packaged with Elasticsearch and Logstash

1 Like

Thanks for your reply. My deployment is on windows server. I haven't found documentation that is geared towards windows. And yes I attempted the upgrade which I ran into some shard limit issues, and ended up restoring to a clean snapshot. Thanks for your reply

1 Like

Detailed installation instructions

Windows

Sounds like you may have succeeded but you should start to form your new upgrade path... Had you run those command across a major version it would not have worked...

So perhaps You got lucky this time and it was just a minor upgrade and that process would work.

is that "ran into some shard limit issues but were ultimately successful once I restored the snapshot, so problem solved" or "ran into some shard limit issues and ended up rolling back with help of the snapshot" ?

I'm gonna take a wild guess you hit the 1000 shard (default) limit.