How to alert if two fields match

Elastic Stack Elasticsearch
1

I am very new to messing with Elastic pipelines and I need help. I want to alert based on whether two fields in a log match. I am not sure the correct way to do this.

The logs are from a Cisco DUO integration. I want alerts if the auth device and the access device countries are different from one another.

Based on some posts, I tried to create a new field in the pipline and use a SET to true based on a condition if cisco_duo.auth.access_device.location.country == cisco_duo.auth.auth_device.location.country. I was not able to get this to work so I am not sure if it is just my syntax or if I am on the wrong track altogether.

If I can get that to work, I could create an alert based on whether my new field is true or false.

I am open to taking a completely different route as well!

2

There's a million ways to achieve what you want, but to create a "matched" field at ingest time something like:

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "script": {
          "lang": "painless",
          "source": """
            ctx['fields_match'] = ctx['field1'] == ctx['field2'];
          """
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "field1": "banana",
        "field2": "banana"
      }
    }
  ]
}
3

That is great! Much simpler than what I have been trying.

I am trying to add this to an existing ingest pipeline and it is not working. It is setting TRUE even when they do NOT match. My guess I have some syntax wrong?

Processor type: script
Source: ctx['cisco_duo.auth.device_country_match'] = ctx['cisco_duo.auth.access_device.location.country'] == ctx['cisco_duo.auth.auth_device.location.country']

4

Can you share the fuller log, with _simulate and a couple of sample documents. Maybe you need braces.

You probably should add a check that both fields actually exist too, setting false if either doesn’t.

5

Result shows the field as true so it seems to not be checking.

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "script": {
          "lang": "painless",
          "source": """
            ctx['cisco_duo.auth.device_country_match'] = ctx['cisco_duo.auth.access_device.location.country'] == ctx['cisco_duo.auth.auth_device.location.country'];
          """
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "source": {
      "address": "127.0.0.1",
      "ip": "127.0.0.1",
      "user": {
        "name": "user",
        "id": "xxxxxxxxxxxxxxx",
        "email": "user@user.com"
      }
    },
    "cisco_duo": {
      "auth": {
        "result": "success",
        "reason": "user_approved",
        "access_device": {
          "ip": "127.0.0.1",
          "location": {
            "country": "United States",
            "city": "Gotham",
            "state": "Illinois"
          }
        },
        "event_type": "authentication",
        "factor": "duo_push",
        "auth_device": {
          "ip": "127.0.0.1",
          "name": "mydevice",
          "location": {
            "country": "Nigeria",
            "city": "Metropolis",
            "state": "DC"
          }
        },
        "email": "user@user.com",
        "trusted_endpoint_status": "unknown"
      }
    },
    "input": {
      "type": "httpjson"
    },
    "related": {
      "user": [
        "user"
      ]
    },
    "data_stream": {
      "namespace": "default",
      "type": "logs",
      "dataset": "cisco_duo.auth"
    },
    "event": {
      "agent_id_status": "verified",
      "reason": "user_approved",
      "kind": "event",
      "module": "cisco_duo",
      "category": [
        "authentication"
      ],
      "type": [
        "info"
      ],
      "dataset": "cisco_duo.auth",
      "outcome": "success"
    },
    "user": {
      "name": "user",
      "id": "xxxxxxxxxxxxxxx",
      "email": "user@user.com"
    }
      }
    }
  ]
}
6

This seems to work for me, I reduced example to closer to minimum.

I don't think your example passed the whole document to _simulate. Also to reference nested fields is a bit different.

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "script": {
          "lang": "painless",
          "source": """
            if (ctx['cisco_duo']['auth']['access_device']['location']['country'] == null ) {
              ctx['countries_match'] = false
            } else if (ctx['cisco_duo']['auth']['auth_device']['location']['country'] == null ) { 
              ctx['countries_match'] = false
            } else { ctx['countries_match'] = ctx['cisco_duo']['auth']['access_device']['location']['country'] == ctx['cisco_duo']['auth']['auth_device']['location']['country']
            }
          """
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "address": "127.0.0.1",
        "ip": "127.0.0.1",
        "cisco_duo": {
          "auth": {
            "access_device": {
              "ip": "127.0.0.1",
              "location": {
                "country": "US",
                "city": "Gotham",
                "state": "Illinois"
              }
            },
            "auth_device": {
              "ip": "127.0.0.1",
              "name": "mydevice",
              "location": {
                "country": "US",
                "city": "Metropolis",
                "state": "DC"
              }
            }
          }
        }
      }
    }
  ]
}

also this seems achieve same thing but doesn't create the countries_match field if either of the 2 country values is null, or if they differ.

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "set": {
          "description": "If access_device and auth_device countries are the same then set countries_match to true",
          "if": "ctx['cisco_duo']['auth']['auth_device']['location']['country'] != null && ctx['cisco_duo']['auth']['access_device']['location']['country'] != null && ( ctx['cisco_duo']['auth']['access_device']['location']['country'] == ctx['cisco_duo']['auth']['auth_device']['location']['country'] ) ",
          "field": "countries_match",
          "value": true
        }
      }
    ]
  },
  "docs": [
        {
      "_source": {
        "address": "127.0.0.1",
        "ip": "127.0.0.1",
        "cisco_duo": {
          "auth": {
            "access_device": {
              "ip": "127.0.0.1",
              "location": {
                "country": "US",
                "city": "Gotham",
                "state": "Illinois"
              }
            },
            "auth_device": {
              "ip": "127.0.0.1",
              "name": "mydevice",
              "location": {
                "country": "US",
                "city": "Metropolis",
                "state": "DC"
              }
            }
          }
        }
      }
    }
  ]
}
7

This is perfect! Thank you for your help. This works perfectly and helps me understand the syntax much better.

8

Glad it helped.

Note, just for the record, personally I would not do it this way. Look into Watcher alerts, and "compare" support:

The details are left as an exercise for the reader :slight_smile:

1 Like
9

I am using SecurityOnion so I do not think the watcher would fit my use? It uses sigma to create an elastic eql query which does not support field comparisons.

Happy to continue to hear suggestions!

10

No more suggestion, because I know nothing of SecurityOnion / sigma. If your org mandates to use tools X and Y, very common, then you are stuck to find slightly hacky workarounds to solve relatively simple problems. Been there, got several scars from the experience, know it can be quite annoying. On flip side, if the org allows 1001 different tools, every group chooses its favorite, and you end up with stuff that is un-manageable and overly-complex.