I am new to Azure and Elasticsearch. While I am trying to send event hub messages to elastic search, nothing is showing up in kibana and my console shows me exception which I do no how to solve it. I followed this doc [event hub plugin]
Here is my configuration
input {
azure_event_hubs {
event_hub_connections => ["Endpoint=sb://myevent.servicebus.windows.net/;SharedAccessKeyName=mykeyname;SharedAccessKey=key;EntityPath=mylog"]
threads => 3
decorate_events => true
consumer_group => "logstash"
storage_connection => "DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=mykey;EndpointSuffix=core.windows.net"
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
action => "index"
hosts => ["http://localhost:9200" ]
index => "azure_event_hub"
}
}
I am getting the following Exception. Any help is appreciated
[2019-03-01T21:48:34,016][INFO ][com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler] linkName[mgmt:receiver], remoteSource[Source{address='$management', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter=null, defaultOutcome=null, outcomes=null, capabilities=null}]
[2019-03-01T21:48:34,054][INFO ][com.microsoft.azure.eventprocessorhost.PartitionManager] host logstash-c64b7350-ea75-4d91-b346-004ebef75ef9: Eventhub mylog count of partitions: 2
[2019-03-01T21:48:34,061][INFO ][com.microsoft.azure.eventprocessorhost.PartitionManager] host logstash-c64b7350-ea75-4d91-b346-004ebef75ef9: Found partition with id: 0
[2019-03-01T21:48:34,061][INFO ][com.microsoft.azure.eventprocessorhost.PartitionManager] host logstash-c64b7350-ea75-4d91-b346-004ebef75ef9: Found partition with id: 1
[2019-03-01T21:48:34,061][INFO ][com.microsoft.azure.eventhubs.impl.ClientEntity] close: clientId[a18757]
[2019-03-01T21:48:34,064][INFO ][com.microsoft.azure.eventhubs.impl.ClientEntity] close: clientId[MessagingFactoryd4e746]
[2019-03-01T21:48:34,271][INFO ][com.microsoft.azure.eventhubs.impl.ConnectionHandler] onConnectionLocalClose: hostname[myevent.servicebus.windows.net:5671], errorCondition[null, null]
[2019-03-01T21:48:34,273][INFO ][com.microsoft.azure.eventhubs.impl.BaseLinkHandler] linkName[mgmt:sender]
[2019-03-01T21:48:34,284][INFO ][com.microsoft.azure.eventhubs.impl.BaseLinkHandler] linkName[mgmt:receiver]
[2019-03-01T21:48:34,285][INFO ][com.microsoft.azure.eventhubs.impl.SessionHandler] entityName[mgmt-session], condition[Error{condition=null, description='null', info=null}]
[2019-03-01T21:48:34,287][INFO ][com.microsoft.azure.eventhubs.impl.BaseLinkHandler] linkName[mgmt:sender], ErrorCondition[null, null]
[2019-03-01T21:48:34,288][INFO ][com.microsoft.azure.eventhubs.impl.BaseLinkHandler] linkName[mgmt:receiver], ErrorCondition[null, null]
[2019-03-01T21:48:34,289][INFO ][com.microsoft.azure.eventhubs.impl.MessagingFactory] messagingFactory[MessagingFactoryd4e746], hostName[myevent.servicebus.windows.net], info[mgmtChannel closed]
[2019-03-01T21:48:34,291][INFO ][com.microsoft.azure.eventhubs.impl.ConnectionHandler] onConnectionRemoteClose: hostname[myevent.servicebus.windows.net:5671], errorCondition[null, null]
[2019-03-01T21:48:34,291][INFO ][com.microsoft.azure.eventhubs.impl.ConnectionHandler] onTransportClosed: hostname[myevent.servicebus.windows.net:5671], error[n/a]
[2019-03-01T21:48:34,292][INFO ][com.microsoft.azure.eventhubs.impl.ConnectionHandler] onConnectionUnbound: hostname[myevent.servicebus.windows.net:5671], state[CLOSED], remoteState[CLOSED]
[2019-03-01T21:48:34,293][INFO ][com.microsoft.azure.eventhubs.impl.SessionHandler] entityName[mgmt-session]
[2019-03-01T21:48:34,293][INFO ][com.microsoft.azure.eventhubs.impl.ReactorDispatcher] ScheduleHandler.run() failed with an error
java.nio.channels.ClosedChannelException: null
at sun.nio.ch.SourceChannelImpl.ensureOpen(SourceChannelImpl.java:154) ~[?:1.8.0_191]
at sun.nio.ch.SourceChannelImpl.read(SourceChannelImpl.java:158) ~[?:1.8.0_191]
at com.microsoft.azure.eventhubs.impl.ReactorDispatcher$ScheduleHandler.run(ReactorDispatcher.java:118) [azure-eventhubs-1.3.0.jar:?]
at com.microsoft.azure.eventhubs.impl.ReactorDispatcher$CloseHandler.run(ReactorDispatcher.java:151) [azure-eventhubs-1.3.0.jar:?]
at org.apache.qpid.proton.reactor.impl.SelectableImpl.free(SelectableImpl.java:153) [proton-j-0.31.0.jar:?]
at org.apache.qpid.proton.reactor.impl.ReactorImpl.free(ReactorImpl.java:137) [proton-j-0.31.0.jar:?]
at com.microsoft.azure.eventhubs.impl.MessagingFactory$RunReactor.run(MessagingFactory.java:546) [azure-eventhubs-1.3.0.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_191]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_191]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_191]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
[2019-03-01T21:48:34,710][INFO ][com.microsoft.azure.eventprocessorhost.AzureStorageCheckpointLeaseManager] host logstash-c64b7350-ea75-4d91-b346-004ebef75ef9: Created lease store OK or it already existed
[2019-03-01T21:48:34,928][INFO ][com.microsoft.azure.eventprocessorhost.PartitionManager] host logstash-c64b7350-ea75-4d91-b346-004ebef75ef9: Starting lease scan
[2019-03-01T21:48:34,934][INFO ][logstash.inputs.azureeventhubs] Event Hub registration complete. {:event_hub_name=>"mylog"}
[2019-03-01T21:48:34,945][INFO ][logstash.inputs.azureeventhubs] Event Hub is processing events... {:event_hub_name=>"mylog"}
[2019-03-01T21:48:35,295][INFO ][com.microsoft.azure.eventprocessorhost.PumpManager] host logstash-c64b7350-ea75-4d91-b346-004ebef75ef9: 0: creating new pump
[2019-03-01T21:48:35,305][INFO ][com.microsoft.azure.eventprocessorhost.PartitionPump] host logstash-c64b7350-ea75-4d91-b346-004ebef75ef9: 0: Creating and opening event processor instance