Watcher post data

Elastic Stack Elasticsearch
1

Hi All,

I am executing a watcher which post data to an API. In it self this works fine, however the data format is something I am struggling with. My watcher looks like this.

{
  "trigger": {
    "schedule": {
      "cron": "0 */15 * * * ?"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "clog-*"
        ],
        "types": [],
        "body": {
          "_source": [
            "login",
            "ip",
            "block_data",
            "block_reason",
            "from",
            "dw002"
          ],
          "query": {
            "bool": {
              "must": [
                {
                  "query_string": {
                    "query": "source_affiliate:nlmail AND dw002:true",
                    "analyze_wildcard": true,
                    "default_field": "*"
                  }
                },
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-15m",
                      "lte": "now"
                    }
                  }
                }
              ],
              "filter": [],
              "should": [],
              "must_not": []
            }
          }
        }
      }
    }
  },
  "condition": {
    "script": {
      "source": "boolean trigger=false;ArrayList offenders = new ArrayList();for(int i=0;i<ctx.payload.hits.hits.size();i++){offenders.add(i, ctx.payload.hits.hits[i]._source);}ctx.vars.offenders=offenders;trigger=true;return trigger;",
      "lang": "painless",
      "params": {
        "dw002": 1
      }
    }
  },
  "actions": {
    "xredir_post": {
      "webhook": {
        "scheme": "http",
        "host": "10.80.3.109",
        "port": 80,
        "method": "post",
        "path": "/v1/block_dw002",
        "params": {},
        "headers": {
          "Content-Type": "application/json"
        },
        "auth": {
          "basic": {
            "username": "admin",
            "password": "::es_redacted::"
          }
        },
        "body": "{{ctx.vars.offenders}}"
      }
    }
  }
}

The data it posts looks like this.

"{0={dw002=true, ip=41.65.69.99, block_reason=botnetloginfrom, from=dddd@blabla.nl, block_data=41.65.69.99 list=smtp-known-botnetip.black.dnsbl.blabla.local, login=dddd@blabla.nl}, 1={ip=11.114.156.159, dw002=true, block_reason=botnetloginfrom, from=kees.ddd@blabla.nl, block_data=41.114.156.159 list=smtp-known-botnetip.black.dnsbl.blabla.local, login=kees.ddd@blabla.nl}, 2={ip=191.2.132.77, dw002=true, block_reason=botnetloginfrom, from=dddd@xxxxxx.nl, block_data=190.2.132.77 list=sip24.xxxxxx.com, login=dddd@xxxxxx.nl}, 3={ip=45.76.125.140, dw002=true, block_reason=botnetloginfrom, from=j.dddd@ddddd.nl, block_data=45.76.125.140 list=smtp-known-botnetip.black.dnsbl.blabla.local, login=j.selenthijssen@dddd.nl}, 4={dw002=true, ip=5.147.49.92, block_reason=botnetloginfrom, from=fam.ffff@blabla.nl, block_data=5.147.49.92 list=sip24.xxxxxx.com, login=fam.sssn@blabla.nl}, 5={ip=17.139.24.142, dw002=true, block_reason=botnetloginfrom, from=keesennicole@xxxxxx.nl, block_data=17.139.24.142 list=sip.xxxxxx.com, login=keeddddnicole@xxxxxx.nl}, 6={ip=45.16.125.140, dw002=true, block_reason=botnetloginfrom, from=w.ddddd@blabla.nl, block_data=45.76.125.140 list=smtp-known-botnetip.black.dnsbl.blabla.local, login=w.s11en@blabla.nl}, 7={dw002=true, ip=109.201.133.236, block_reason=botnetloginfrom, from=chwk@xxxxxx.nl, block_data=109.201.133.236 list=sip24.xxxxxx.com, login=chwk@h111e.nl}}"

I basically need to send a array with the _source to the API but I seem to do something wrong..

I hope somebody can give me some advice.

Paul.

2

Please remove references to invaluement's host names from your comments (or anywhere). (1) those are old host names that are on their way out - only a handful of our customers still use those (2) normally queries to these are blocked unless they come from specific IPs that belong to paying customers (3) however, they were recently opened up for a short time in jan/2019 - but that won't last long (4) SOON - AFTER WE MIGRATE OUR LAST CUSTOMERS AWAY FROM USING THESE OLD HOST NAMES - WE WILL START "BLACKLISTING THE WORLD" ON THOSE HOST NAMES.

Again, we haven't official used these in years, only a few customers still use those, and attempted use of these by non-customers is unauthorized and abusive. And will be very dumb very soon since ALL queries to those will start answering 127.0.0.2 in the very near future. So by publishing that here - you're leading others off a cliff. So please delete those host names from your code. (unless you just LIKE leading others off a cliff - if so, then keep it as is - and hopefully they'll read my warning so that they'll understand what to expect.)
Thanks for your help with this!
Rob McEwen, CEO of invaluement.com

4

Hi Rob,

  1. I should have anonymized that data better. My apologies for that, I will do that.
  2. Your making the assumption that I am not a paying customer and there for I am a abuser, this is not the case.
  3. How does you reply help my problem? it does not..
  4. You don't have to SHOUT.

Paul.

ps. You could have contacted me trough a private message, would have been a lot nicer..

5

Paul, Thanks for the fast response. Sorry if I came across as a jerk - I just wanted anyone reading this to understand and take seriously the potential damage that would happen if they started using those invaluement host names - especially if we end up "listing the world" after we soon migrate away from those particular host names. Also, I STILL have no idea who you are. I have no idea what company you represent. You could be some "Paul" I've talked to before a dozen times! (but maybe I just never had your last name?) or you could be an employee of a company we do business with - but I never worked with you directly? I can't find a reference to ANY person I've exchanged emails with who goes by the name "Paul Janzen" - or any "janzen" in my customer database or in any emails I've sent or received. Forgive me if I'm overlooking something? But from my perspective, I didn't know you were a customer since I couldn't find anything recognizable from your post. While I don't doubt what you're saying - I can't read minds and I still don't know who you are or what company you represent. ("Paul" alone isn't specific enough, and I can't find any references to "Janzen"). I wasn't trying to be mean - but sometimes when our host names are published online - many organizations can pick up on that and it can slowly turn into the equivalent of a denial of service attack. Since I didn't know who you were, I needed to word that strongly and clearly to make sure the message was getting across. I hope you understand. And thanks for going back and editing your previous comment.
PS - Please send me an email so that i can connect-the-dots as to who you are and/or who you represent - the suspense is killing me! Thanks!
--Rob McEwen

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.