# Sending azure\_event\_hub messages to elasticsearch **URL:** \ **Category:** Logstash\ **Created:** [February 7, 2019, 10:41pm UTC](https://discuss.elastic.co/t/sending-azure-event-hub-messages-to-elasticsearch/167550 "2019-02-07T22:41:25Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![fringeadmin](https://avatars.discourse-cdn.com/v4/letter/f/bb73d2/32.png) [@fringeadmin](https://discuss.elastic.co/u/fringeadmin)\ **Post date:** [February 7, 2019, 10:41pm UTC](https://discuss.elastic.co/t/sending-azure-event-hub-messages-to-elasticsearch/167550/1 "2019-02-07T22:41:25Z")
I recently installed Elastic Stack for the first time using the steps at [Digital Ocean](https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elastic-stack-on-centos-7). I then followed the steps at the [Azure Event Hubs plugin page](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-azure_event_hubs.html) to link logstash to an Azure event hub. This seemed to go okay, but nothing is showing up in kibana or even elasticsearch as far as I can tell. My config looks like this: ``` input { azure_event_hubs { config_mode => "advanced" event_hubs => [ { "myentitypath" => { event_hub_connection => "Endpoint=sb://myendpoint.servicebus.windows.net/;SharedAccessKeyName=logstash;SharedAccessKey=alongrandomkey;EntityPath=myentitypath" }} ] threads => 8 decorate_events => true consumer_group => "$Default" storage_connection => "DefaultEndpointsProtocol=https;AccountName=mystorageacct;AccountKey=alongrandomkey;EndpointSuffix=core.windows.net" } } ``` I feel like I probably need to specify some output details? The only output config that I have is what I set up for filebeats: ``` output { elasticsearch { hosts => ["localhost:9200"] manage_template => false index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" } } ``` Is there an output format that I need to specify for Azure Event Hub? Do I need to somehow create the indexes in ES? Any help is much appreciated. As you can tell I don't really understand how all this works. 🙂 ---
**Author:** ![NerdSec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nerdsec/32/22056_2.png) [@NerdSec](https://discuss.elastic.co/u/NerdSec)\ **Post date:** [February 8, 2019, 4:25am UTC](https://discuss.elastic.co/t/sending-azure-event-hub-messages-to-elasticsearch/167550/2 "2019-02-08T04:25:20Z")
Hi Nick, Could you verify if you are receiving the information on STDOUT? Try using the following config: ```auto input { .... Your Input Settings ..... } output { stdout { } } ``` This will validate if the input configuration is working, then we can look for error on the output. A typical elasticsearch output looks as follows: ```auto output { elasticsearch { action => "index" hosts => ["http://x.x.x.x:9200"] index => "indice" } } ``` I believe you are missing the action part in the elasticsearch config. ---
**Author:** ![fringeadmin](https://avatars.discourse-cdn.com/v4/letter/f/bb73d2/32.png) [@fringeadmin](https://discuss.elastic.co/u/fringeadmin)\ **Post date:** [February 8, 2019, 2:13pm UTC](https://discuss.elastic.co/t/sending-azure-event-hub-messages-to-elasticsearch/167550/3 "2019-02-08T14:13:48Z")
Thanks for the reply and very helpful stdout config! Using that I am able to see that the azure event hub data is definitely making it to logstash: ``` "@timestamp" => 2019-02-08T14:04:32.149Z, "message" => "{\"records\": [{ \"LogicalServerName\": \"myservername\", \"SubscriptionId\": \"mysubuuid\", \"ResourceGroup\": \"myRG\", \"time\": \"2019-02-08T14:00:21.7540000Z\"... ``` So next I tried adding your output block verbatim and nothing happens. I assume that's because "indice" needs to be an actual indice and not just the word, but how do I know which indice? And how do I create the indices that azure\_event\_hub expects? Filebeats has a handy tool that does it, but I don't see anything similar for this plugin. ---
**Author:** ![fringeadmin](https://avatars.discourse-cdn.com/v4/letter/f/bb73d2/32.png) [@fringeadmin](https://discuss.elastic.co/u/fringeadmin)\ **Post date:** [February 11, 2019, 7:39pm UTC](https://discuss.elastic.co/t/sending-azure-event-hub-messages-to-elasticsearch/167550/4 "2019-02-11T19:39:11Z")
Does anyone have any solutions to this? It seems odd that the documentation tells me how to ingest the data, but not how to display it. Are there any other docs someone can point me to that might help? ---
**Author:** ![fringeadmin](https://avatars.discourse-cdn.com/v4/letter/f/bb73d2/32.png) [@fringeadmin](https://discuss.elastic.co/u/fringeadmin)\ **Post date:** [February 18, 2019, 2:47pm UTC](https://discuss.elastic.co/t/sending-azure-event-hub-messages-to-elasticsearch/167550/5 "2019-02-18T14:47:22Z")
For the benefit of any future viewers, yes, just naming an index will cause logstash to create and use that index. Here's the file I ended up with: ``` input { azure_event_hubs { config_mode => "advanced" event_hubs => [ { "eventhubentityname" => { event_hub_connection => "Endpoint=sb://myeventhub.servicebus.windows.net/;SharedAccessKeyName=logstash;SharedAccessKey=myaccesskey;EntityPath=eventhubentityname" }} ] threads => 8 decorate_events => true consumer_group => "logstash" storage_connection => "DefaultEndpointsProtocol=https;AccountName=mystorageacct;AccountKey=mystorageacctkey;EndpointSuffix=core.windows.net" type => "azure_event_hub" } } # since I have multiple inputs going to ES, I use the type parameter to make sure only the event hub messages go to this index output { if [type] == "azure_event_hub" { #stdout { } elasticsearch { action => "index" hosts => ["http://localhost:9200"] index => "azure_event_hub" } } } ``` Unfortunately this dumps all the useful info to a single 'message' field, so I still need to figure out how to make that doing something practical. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 18, 2019, 4:01pm UTC](https://discuss.elastic.co/t/sending-azure-event-hub-messages-to-elasticsearch/167550/6 "2019-02-18T16:01:27Z")
Start with ``` filter { json { source => "message" } } ``` If that does not produce what you want then show us the output from ``` output { stdout { codec => rubydebug } } ``` and explain what you do not like about it. ---
**Author:** ![fringeadmin](https://avatars.discourse-cdn.com/v4/letter/f/bb73d2/32.png) [@fringeadmin](https://discuss.elastic.co/u/fringeadmin)\ **Post date:** [February 18, 2019, 4:33pm UTC](https://discuss.elastic.co/t/sending-azure-event-hub-messages-to-elasticsearch/167550/7 "2019-02-18T16:33:42Z")
Thanks for the info badger, the json is nested and is a set of events under message. Using your filter above I get a list of records with several very long json strings that could be called event[0], event[1], etc. It looks like basically like this: ``` {"records": [ { "LogicalServerName": "myservername", "SubscriptionId": "mysubid-uuid", "ResourceGroup": "Prod", "time": "2019-02-15T17:44:23.3870000Z", "resourceId": "/SUBSCRIPTIONS/mysubid-uuid/RESOURCEGROUPS/PROD/PROVIDERS/MICROSOFT.SQL/SERVERS/myservername/DATABASES/MASTER", "category": "SQLSecurityAuditEvents", "operationName": "AuditEvent"}, { "LogicalServerName": "myservername", "SubscriptionId": "mysubid-uuid", "ResourceGroup": "Prod", "time": "2019-02-15T17:44:23.3870000Z", "resourceId": "/SUBSCRIPTIONS/mysubid-uuid/RESOURCEGROUPS/PROD/PROVIDERS/MICROSOFT.SQL/SERVERS/myservername/DATABASES/MASTER", "category": "SQLSecurityAuditEvents", "operationName": "AuditEvent"} ]} ``` That led me to the split function in logstash and I ended up with this, which does basically what I want: ``` filter { if [type] == "azure_event_hub" { json { source => "message" } split { field => ["records"] } } } ``` Thanks everyone who helped! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 18, 2019, 4:33pm UTC](https://discuss.elastic.co/t/sending-azure-event-hub-messages-to-elasticsearch/167550/8 "2019-03-18T16:33:46Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.