Logstash receive from multiple source & data analysis

Elastic Stack Logstash
1

Hi,

I'm new to logstash, i've set up the "sexilog" appliance ( it's a elk appliance mostly developped for vmware logs)

I'm trying to catch a extreme networks wifi controller logs.

Actually, all the log are received trough the same default input file from sexilog.

  1. I would like to add a tag to the logs from the wifi controller. to filter only these logs. What's the best way to do it ?
    Actually this is teh default input file

input {

    udp {
            port => 514
            type => esxi
    }

}

Can i copy that file and add the source ip ?

  1. This is the kind of logs i receive:

<182>events: EventType[State Change] MAC[EC:35:86:XX:XX:XX] IP[172.XX.X.XX] SSID[SSIDNAME] BSSID[D8:84:XX:XX:XX:XX]

I suppose i have to use grok to format it correctly, what would be the righ way to do it ?

Thanks in advance for yor help

2

If that is your only input on 514 then you just choose the type field. If you have multiple inputs each input can have a type. What do you mean by source ip?

Just write a grok to match what you are going to get.

That is the correct way

3
  1. Not sure exactly what you mean, but the udp input will create a field with the IP address and (I think) source port of the connecting client. If the IP address is fixed and you want to add a human-readable field or tag you can use a conditional filter similar to this:
if [clientip] == "1.2.3.4" {
  mutate {
    add_tag => ["some-appliance"]
  }
}
  1. Okay, so you basically have a list of key/value pairs (on the form KEY[VALUE]). In most key/value cases one would use the kv filter, but that won't work in this case so we'll have to use grok. The following can be used to parse a single such key/value pair:
EventType\[(?<EventType>[^\]]+)\]
4

HI,

Thanks you for your answers.

Could you confirm that the "if" has to be set in the input file ?

Will try the grok

5

Which configuration file you place the if conditional in doesn't matter. All configuration files are essentially concatenated.

6

hi,

i created a filter file in the logstash conf.d foledr with that :

filter {
if [clientip] == "10.9.8.xx" {
mutate {
add_tag => ["ewc"]
type => ["ewc"]
}
}
}

But it keep the esx type .
Do i need to place that in another place ?

7

Do you have a clientip field in your messages? I only gave you an example of what a conditional could look like. You'll have to adapt it to the set of fields you have at your disposal.

Secondly, it's not clear what you mean by this:

mutate {
  add_tag => ["ewc"]
  type => ["ewc"]
}

Was the intention to set type equal to "ewc"? Or apply the add_tag option unless if type already was equal to "ewc"? Either way it won't work but if I know what you tried to do I can explain what you should do instead.

8

I d'ont have a client ip in the log but i have a host.

I change it to this :

if [host] == "10.9.8.xx" {
mutate {
add_tag => "ewc"
type => "ewc"
}}

I want to change the type, because the default type is esxi.

I just wan t to be able to differenciate the logs from ewc and the logs from esxi, to eassylly filter on the dashboard.
It's more clean if the ewc logs have the ewc type right ?

9

I want to change the type, because the default type is esxi.

type => "ewc" for setting the type will only work for inputs. To change the type in a filter you need this:

mutate {
  replace => {
    "type" => "ewc"
  }
}

It's more clean if the ewc logs have the ewc type right ?

Yes, I agree.

10

yeah that works :slight_smile:

will now try to use grok.

This is what i have in the message:

182>events: EventType[State Change] MAC[88:30:8A:7D:XX:XX IP[172.15.X.XX] SSID[SSID] BSSID[D8:84:66:3C:XX:XX]

I would like to have a column eventType, on mac, one ip etc..

11

Have you looked at the suggestion for how to parse those fields using grok that I gave you last week?

12

I'm just start to have a look on that, but i'm not really sure to hunderstand what to do with it :slight_smile:
i just test it in the grok debugger, and looks to privde the right info.

What will be the correct syntax in the filter ?
something like this ?

grok {
match => [
"message", "EventType[(?[^]]+)]"
"message", "MAC[(?[^]]+)]
]}

13

I suggest you cram everything into a single expression.

grok {
  match => ["message", "... events: EventType\[(?<EventType>[^]]+)\] MAC\[(?<MAC>[^]]+)\] ..."]
}
14

Yeah that's works :slight_smile:

15

I have different event, do i have to add a second expression ?

events: EventType[State Change] MAC[4C:7C:5F:C7:4B:89] IP[10.9.X.XXX] SSID[SSID] BSSID[D8:84:66:3C:XX:XX]

grok {
match => ["message", "... events: EventType[(?[^]]+)] MAC[(?[^]]+)] ..."]
match => ["message", "... events: EventType[(?[^]]+)] MAC[(?[^]]+)] IP[(?[^]]+)]..."]
}

16

I think what you've proposed works but the documentation suggests this instead:

grok {
  match => {
    "message" => [
      "pattern1",
      "pattern2",
      ...,
      "patternN",
    ]
  }
}
17

ok will try this.

i still have a question , in the log i have the Detail part

EventType[Roam] MAC[4C:8D:79:4B:XX:XX] AP[APNAME2-1] FromAP[APNAME2-1] BSSID[D8:84:66:3C:XX:XX] Details: Inside AC from AP/Radio[2] to AP/Radio[1] VNS[SSID]

How can i split that one : Details: Inside AC from AP/Radio[2] to AP/Radio[1]

... events: EventType[(?[^]]+)] MAC[(?[^]]+)] ... Details[(?

[^]]+)]"

Thx

19

HI,

i think i've set up all rule now, elso for the details, don't know if it's the right way to do it.

BSSID[(?[^]]+)] Details: Inside AC from AP/Radio[(?[^]]+)] to AP/Radio[(?[^]]+)] VNS[(?[^]]+)]

The last log i have is this one, i have no idea how to manage it :

events: Access Point AP Serial Number 15281595085E0000: 12/07/15 13:10:24: DCS Measured Noise -87dBm Dropped below Threshold of -80dBm on chann. 5200Mhz 3

By the way Magnus Bäck, thank you very much for your help.