Azure Event Hubs plugin problem

[cRaZyT (Thorsten)]

Hi there,

I've installed logstash 7.13.1 and the logstash-input-azure_event_hubs 1.3.0 plugin.

I got the following error during the initialization:

[ERROR][logstash.inputs.azureeventhubs][main][07395995e41d84f42b907d44ee2f6bbd6eedfb89f0d117d0a2f18870b0b1b694] Event Hub failed during initialization. {:event_hub_name=>"insights-operational-logs", :exception=>#<NoMethodError: undefined method value' for #<String:0x7f715d23>>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-azure_event_hubs-1.3.0/lib/logstash/inputs/azure_event_hubs.rb:390:in block in run'"]}

[ERROR][logstash.inputs.azureeventhubs][main][07395995e41d84f42b907d44ee2f6bbd6eedfb89f0d117d0a2f18870b0b1b694] Event Hub failed during initialization. {:event_hub_name=>"event_hub2", :exception=>#<NoMethodError: undefined method value' for #<String:0x32a4ddfb>>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-azure_event_hubs-1.3.0/lib/logstash/inputs/azure_event_hubs.rb:390:in block in run'"]}

The input looks like this:

input {
   azure_event_hubs {
      config_mode => "advanced"
      threads => 8
      decorate_events => true
      checkpoint_interval => 5
      storage_connection => "DefaultEndpointsProtocol=https;AccountName=aqsdevsa;AccountKey=XXXXX==;EndpointSuffix=core.windows.net"
      event_hubs => [
        { "insights-operational-logs" => {
          event_hub_connections => ["Endpoint=sb://XXXXX.servicebus.windows.net/;SharedAccessKeyName=XXXXX;SharedAccessKey=XXXXX=;EntityPath=insights-operational-logs"]
          consumer_group => "$Default"
        }},
        { "event_hub2" => {
          event_hub_connections => ["Endpoint=sb://XXXXX.servicebus.windows.net/;SharedAccessKeyName=PreviewDataPolicy;SharedAccessKey=XXXXX=;EntityPath=event_hub2"]
          consumer_group => "$Default"
        }}
      ]
   }
}

filter {
   split {
     field => ["records"]
   }
}

output {
  elasticsearch {
    hosts => ["https://XXXXX.azure.elastic-cloud.com:9200"]
    data_stream => false
    index => "eventhub-%{+YYYY.MM.dd}"
    user => "XXXXX"
    password => "XXXXX"
  }
}

I also have tried this filter without success:

filter {
   json {
      source => "message"
   }
   split {
     field => ["records"]
   }
}

The same error message appears even without a filter.

Has some an idea what I made wrong in the pipeline.yml?

Thanky in adavand

Regards
Thorsten

[leandrojmp (Leandro Pereira)]

Your configuration is wrong.

Since you are using the config_mode as advanced you should use event_hub_connection not event_hub_connections.

This settings is also a string, not an array.

It should be:

      event_hubs => [
        { "insights-operational-logs" => {
          event_hub_connection => "Endpoint=sb://XXXXX.servicebus.windows.net/;SharedAccessKeyName=XXXXX;SharedAccessKey=XXXXX=;EntityPath=insights-operational-logs"
          consumer_group => "$Default"
        }},
        { "event_hub2" => {
          event_hub_connection => "Endpoint=sb://XXXXX.servicebus.windows.net/;SharedAccessKeyName=PreviewDataPolicy;SharedAccessKey=XXXXX=;EntityPath=event_hub2"
          consumer_group => "$Default"
        }}
      ]

You can check this in the documentation.

[cRaZyT (Thorsten)]

Thank you leandrojmp, I was probably blind because I totally overlooked that! You really saved my day.