Translate filter for Azure subscription ID

Sorry I can't figure out how to format this properly.

logstash 1:7.3.0-1

I am pulling in data from an Azure event hub using the logstash azure_event_hubs plugin. Without any modification, the data displayed in Kibana shows the subscription ID. I am trying to modify this to display the subscription name. The source JSON diplayed in Azure activty alerts shows the field "subscriptionId": "55555555-5555-5555-5555-555555555555". At some point, the data field is changed to:
"azure": {
"subscription": "55555555-5555-5555-5555-555555555555"

This is then displayed in Kibana as "azure.subcription". I'm using the logstash translate plugin to replace the subscription ID with the name.

Here is the contents of the pipeline yml:
...
input{
azure_event_hubs {
event_hub_connections => ["Endpoint=***"]
threads => 2
decorate_events => true
consumer_group => "logstash"
storage_connection => "DefaultEndpointsProtocol=https;AccountName=***endpointSuffix=core.windows.net"
}
}

filter {
json {
source => "message"
}

if "_jsonparsefailure" not in [tags] {
split {
field => "records"
target => "data"
}

if "_split_type_failure" not in [tags] {
  ruby{
    code => "
      event.get('data').each { |k,v|
        event.set(k, v)
      }
      event.remove('data')
    "
    remove_field => ["message", "records"]
  }

  if "_rubyexception" not in [tags] {
    azure_event{}

    if "_azure_event_failure" not in [tags] {
      if [callerIpAddress]{
        geoip {
          source => "callerIpAddress"
        }
      }
      date {
          match => [ "time", "ISO8601" ]
      }
      mutate {
          copy => { "@metadata" => "[azure][metadata]" }
      }
    }
  }
}

}
translate {
field => "azure.subscription"
destination => "azure.subscription"
override => false
dictionary => {
"55555555-5555-5555-5555-555555555555" => "subscription1"
"55555555-5555-5555-5555-666666666666" => "subscription2"
}
}
}
output {
elasticsearch {
hosts => ["http://10.112.136.4:9200"]
index => "azure-%{+YYYY.MM.dd}"
}
}
...
I have also tried
field => "subscriptionID"
destination => "subscription"

The yml file is copied from /usr/share/logstash/x-pack/modules/azure/configuration/logstash/azure.conf.erb.

The filter does not appear to work and I recieve this message in the logs:
[2019-08-21T17:54:08,495][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>416, "name"=>"[azeventhub]<azure_event_hu
bs", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-input-azure_event_hubs-1.1.2/lib/logstash/inputs/azure_event_hubs.rb:466:in block in join'"}], ["LogStash::Filters::Tran slate", {"destination"=>"azure.subscription", "override"=>"false", "dictionary"=>{"55555555-5555-5555-5555-555555555555"=>"subscription1", "55555555-5555-5555-5555-666666666666"=>"subscription2"}, "id"=>"054cc2deefe4e58bbe8f6fe221e183fb2479ecf3f70a21b980a7d00f93e0414f", "field"=>"azure.subscription"}]=>[{"thread_id"=>414, "name"=>"[azeventhub]>worker0", " current_call"=>"[...]/logstash-core/lib/logstash/java_pipeline.rb:239:inblock in start_workers'"}, {"thread_id"=>415, "name"=>"[azeventhub]>worker1", "current_call"=>"[...]/logstash-core/
lib/logstash/java_pipeline.rb:239:in `block in start_workers'"}]}}

Googling the error has lead to no useful information. I'm not sure where the conversion from subscriptionID to azure.subscription is happening or which input (field) value I need to use for the filter.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.