Azure Module - Multiple event hubs?

Elastic Stack Beats
1

Can't seem to find a solution for this. Here's the situation:

I am setting up a PoC environment, which has two Azure event hubs configured to input Azure AD data (which themselves are coming from separate environments).

I have a filebeat instance I'm trying to pull the data in from, and pass off to a Logstash instance to eventually send to ES Cloud.

Filebeat works fine if I set up a single azure module definition, which points to a specific event hub. It doesn't appear I can duplicate the block to point to a second event hub.

What would be the recommended setup to achieve this goal? I'm kind of new to filebeat modules, so I'm not entirely sure if in this instance I even NEED the module, or if plain inputs are fine?

2

Hey @chris.murray, a couple of questions to test my understanding of your scenario.

Are you trying to pull the same type of data (auditlogs or signinlogs) from two different event hubs / active directories?

Can you share the configuration files from your PoC (even a simplified version would do)?

Modules usually have one input per data type, but I want to learn more about your use case before adding more.

3

Yeah, so for the moment assume they are both activitylogs. (The point about the different log types is good, but the setup isn't that far yet...and likely we'd have different ones for the different log types).

I've kind of figured out that the Azure module, if you put two entries in like so:

filebeat.modules:

# Config for Event Hub 1
- module: azure
  activitylogs:
    enabled: true
    var:
      eventhub: "eventhubname"
      consumer_group: "$Default"
      connection_string: "Endpoint=sb://(rest of connection string)"
      storage_account: "(storageaccountname)"
      storage_account_key: "(Storageaccountkey)"

# Config for Event Hub 2
- module: azure
  activitylogs:
    enabled: true
    var:
      eventhub: "eventhub2name"
      consumer_group: "$Default"
      connection_string: "Endpoint=sb://(rest of connection string)"
      storage_account: "(storageaccountname)"
      storage_account_key: "(Storageaccountkey)"

The second one will be "active" and the first one ignored. If I flip them around it works.

You might wonder why we're doing this. It's a bit of a multi-tenant environment, and the separate event hubs is part of how this particular client separates data...not something I can directly change :smiley:

What we've done as a bandaid for the moment is have two filebeat agents running on the same box we're pulling these logs from (using different filebeat.yml files / path.data paths) and that does work, but feels a bit clunky. I don't know if this is the recommended way or not.

4

Just for a test have you tried to put the 2nd activitylogs: in just under the same - module: azure definition?

5

I have not....will test.

6

That won't work. What version of beats are u using? This was a bug that was fixed. There was a bug with how the filebeat.modules section was parsed. If u copy the config that u have to the modules.d/azure.yml, it will work as expected

2 Likes
7

Should be in filebeat 8.1 or 8.2

8

I think this is a legit use case.

This was a bug that was fixed.

@legoguy1000 is Inconsistencies on filebeat modules configuration load · Issue #11943 · elastic/beats · GitHub the bug you were mentioning?

9

That's an older one, here's the PR that fixed it along with a more recent issue mentioning it. [Filebeat] Fix multiple modules in filebeat.yml by legoguy1000 · Pull Request #29952 · elastic/beats · GitHub

1 Like
10

Oh, great!

According to the release notes, the fix has been released in 8.0.1.

11

You mean, as opposed to it being in filebeat.yml?

12

correct

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.