Cluster configuarton for ELK stack with Kafka

Elastic Stack Logstash
1

Hi All,

I am done with standalone testing of ELK with Kafka now i want to implement Full Clustered ELK + Kafka .

Please share docs and imp points for the same .

2

You'll get more useful answers if you ask a more concrete question.

3

Below is my zookeeper config file ,

Zoo.cfg

############################################

The number of milliseconds of each tick

tickTime=2000

The number of ticks that the initial

synchronization phase can take

initLimit=10

The number of ticks that can pass between

sending a request and getting an acknowledgement

syncLimit=5

the directory where the snapshot is stored.

do not use /tmp for storage, /tmp here is just

example sakes.

dataDir=/zookeeper/data

dataLogDir ======= >>>>> where you would like ZooKeeper to log

dataLogDir=/zookeeper/logs

the port at which the clients will connect

clientPort=2181

the maximum number of client connections.

increase this if you need to handle more clients

maxClientCnxns=60

Be sure to read the maintenance section of the

administrator guide before turning on autopurge.

http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance

The number of snapshots to retain in dataDir

autopurge.snapRetainCount=5

Purge task interval in hours

Set to "0" to disable auto purge feature

autopurge.purgeInterval=0

3DS Zookeeper cluster vkafka0[1-3]dsy

server.1=PQR:2888:3888
server.2=XYZ:2888:3888
server.3=ABC:2888:3888

######################################################

I am trying to start my zookeeper cluster but getting below exception , on all 3 servers . what is wrong with config files .

2017-12-22 08:36:56,993 [myid:] - INFO [main:QuorumPeerConfig@136] - Reading configuration from: /usr/local/zookeeper/bin/../conf/zoo.cfg
2017-12-22 08:36:57,003 [myid:] - INFO [main:QuorumPeer$QuorumServer@184] - Resolved hostname: ABC to address: /ABC
2017-12-22 08:36:57,003 [myid:] - INFO [main:QuorumPeer$QuorumServer@184] - Resolved hostname: XYZ to address: /XYZ
2017-12-22 08:36:57,004 [myid:] - INFO [main:QuorumPeer$QuorumServer@184] - Resolved hostname: PQR to address: /PQR
2017-12-22 08:36:57,004 [myid:] - INFO [main:QuorumPeerConfig@398] - Defaulting to majority quorums
2017-12-22 08:36:57,005 [myid:] - ERROR [main:QuorumPeerMain@88] - Invalid config, exiting abnormally
org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error processing /usr/local/zookeeper/bin/../conf/zoo.cfg
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:156)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:104)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
Caused by: java.lang.IllegalArgumentException: /zookeeper/data/myid file is missing
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:408)
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:152)
... 2 more
Invalid config, exiting abnormally

please help on the same

4

This forum is for Logstash questions. Zookeeper questions belong elsewhere.

/zookeeper/data/myid file is missing

Isn't this error message pretty clear?

5

yes i have created file myid .

But i am getting some other exception now .

Can you help me where i can ask for kafka related queries .

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.