Storm and ElasticStack with external ZooKeeper Server

AaronNGray · February 15, 2021, 9:33pm

I am trying to create a docker-compose script that marries ElasticStack with Storm for the purposes of StormCrawler development.

I have a storm script that includes a ZooKeeper server.

gist.github.com

https://gist.github.com/AaronNGray/4cbfa25db0633fc516a68f8c163eb60b

README.md

## Docker yaml script for Zookeeper, Storm v1.2.3 nimbus, supervisor, and UI.
```
docker-compose -f storm-1.2.3.yaml pull
docker-compose -f storm-1.2.3.yaml start
```

```
docker-compose -f storm-1.2.3.yaml stop
```

storm-1.2.3.yaml

version: '3.2'

services:
  zookeeper:
    image: zookeeper
    container_name: zookeeper
    restart: always

  nimbus:
    image: storm:1.2.3

This file has been truncated. show original

I tried combining this with the following ElasticSearch - Kibana script :-

I seem to be getting issues when using the separate ZooKeeper Server.

Is there any documentation or examples on how to make the ElasticSearch source work as a ZooKepper client rather than as a server ? Or is it just a case of mapping ports correctly ?

gist.github.com

https://gist.github.com/AaronNGray/0aacbee2d1cb02c1f684ae990d9df4d6

storm.yaml

version: '3.2'

services:
  zookeeper:
    image: zookeeper
    container_name: zookeeper
    restart: always

  nimbus:
    image: storm:1.2.3

This file has been truncated. show original

ElasticSearch does not seem to come up on port 9200 and Kibana fails to serve as a result with :-

"Kibana server is not ready yet"

Many thanks in advance,

Aaron

AaronNGray · February 16, 2021, 6:32pm

Okay I seem to have solved things now, but working from docker-elk :-

https://github.com/AaronNGray/docker-storm-elasticsearch-kibana

Got the docker-compose working just need to get the CI scripts working properly, and then add storm tests.

system · March 16, 2021, 6:32pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.