I lose entries when using the Watcher Index Action

alerting

#1

Hi,

i want to build a warning system if there are too many failed logins. I want to index the entries, which appear in auth.log into a new index. I build an index with

input {
  file {
          path => "/var/log/auth.log"
          type => "auth"
  }
}

At the end i get all my entries into my index "syslog-auth-log-*".
Now i want to use watcher Index Action, who shall write all index entries into a new index called "anomalies", when i have more than 2 failed logins in the past minute.

My problem is when I have more than 10 failed logins, I no longer get all written in my new index "anomalies". When I produce 10 failed logins with an SSH login script, all 10 entries are displayed in the index "anomalie", when I produce 16 failed logins it just shows 10 entries. Why i lose 6 entries here?
In the execution output it says that it is "total": 16 but it only processes 10 more.

  "result": {
    "execution_time": "2018-10-05T09:22:08.026Z",
    "execution_duration": 4,
    "input": {
      "type": "search",
      "status": "success",
      "payload": {
        "_shards": {
          "total": 10,
          "failed": 0,
          "successful": 10,
          "skipped": 0
        },
        "hits": {
          "hits": [     _hits entries start here_
            {
              "_index": "logstash-auth-log-2018.10.",
              "_type": "doc",
              "_source": {
              "_index": "logstash-auth-log-2018.10.",
              "_type": "doc",
              "_source": {
                "ssh_client_ip": "10.100.100.111",
                "ssh_failreason": "unknown_user",
                "@timestamp": "2018-10-05T09:20:35.967Z",
                "message": "Failed password for invalid user 9dBrgF from 10.100.100.111 port 48482 ssh2"
              }
          "_id": "84GIQ2YBncU22ZIzwkoe",
          "_score": 0.03707742
          ],
          **"total": 16,**     _In the hits entries it just shows me 10 entries, not 16! For a better overview i deleted 9 of them. Where are the other 6?_
          "max_score": 0.03707742
        },
        "took": 1,
        "timed_out": false
      },
      "search": {
        "request": {
          "search_type": "query_then_fetch",
          "indices": [
            "logstash-auth-log-*"
          ],
          "types": [],
          "body": {
            "query": {
              "bool": {
                "must": {
                  "match": {
                    "ssh_failreason": "unknown_user"
                  }
                },
                "filter": {
                  "range": {
                    "@timestamp": {
                      "from": "now-2m",
                      "to": "now"
                    }
                  }
                }
              }
            },
            "_source": [
              "@timestamp",
              "ssh_failreason",
              "ssh_client_ip",
              "message"
            ]
          }
        }
      }
    },
    "condition": {
      "type": "compare",
      "status": "success",
      "met": true,
      "compare": {
        "resolved_values": {
          "ctx.payload.hits.total": 16        _I already lost 6 entries_
        }
      }
    },
    "transform": {
      "type": "script",
      "status": "success",
      "payload": {
        "_value": [
          {
            "ssh_client_ip": "10.100.100.111",
            "@timestamp": "2018-10-05T09:20:35.967Z",
            "ssh_failreason": "unknown_user",
            "message": "Failed password for invalid user 9dBrgF from 10.100.100.111 port 48482 ssh2"
          }
        ]
      }
    },
    "actions": [
      {
        "id": "index_payload",
        "type": "index",
        "status": "success",
        "transform": {
          "type": "script",
          "status": "success",
          "payload": {
            "_doc": [
              {
                "ssh_client_ip": "10.100.100.111",
                "@timestamp": "2018-10-05T09:20:35.967Z",
                "ssh_failreason": "unknown_user",
                "message": "Failed password for invalid user 9dBrgF from 10.100.100.111 port 48482 ssh2"
              }
            ]
          }
        },
        "index": {
          "response": [
            {
              "created": true,
              "result": "created",
              "id": "84GIQ2YBncU22ZIzwkoe",
              "version": 1,
              "type": "my-type",
              "index": "anomalies"
            }
          ]
        }
      }
    ]
  }
}

(Andrés Pérez) #2

Hi,
I think that I know the cause of your problem:

The search request has a "size" parameter that sets the number of hits to return Search request parameters .
If omitted, the default value is 10.

You can modify this parameter (see the example in x-pack input search to get only the last result)

Unfortunately, I don't know any elegant, scalable solution other than setting a suitable magic number .

After reading this question: Watcher index action - how to get all the hits data to index it seems that:

  • The limit is 10000
  • It is not a good idea to index a big amount of search results.

(Alexander Reelsen) #3

Can you maybe explain the use-case? Why is a reindexing necessary when you can access the same data just with a proper range query?

Watcher is not supposed to be a big reindexing tool. You could run a reindex request with a query if you wanted to, however, as stated above, maybe there is another solution based on your use-case.

--Alex


#4

use-case: Send Email Action and display on dashboard (vertical bar) when there are more than 2 failed logins in the past minute.

I think, reindexing isn't necessary here, but for me it was the easiest way to get my result. Sometimes i think too complicated and do weird stuff. Now i try to get it work without reindexing, because i think it's more elegant.


(Alexander Reelsen) #5

Indeed, you are not required to have an alert for a dashbaord visualization and should just go with a filter for that visualization. No need to reindex.


#6

Hello again,
i'm using a similar filterconfig like this. I'm overstrained where i have to put the filter to get the dashboard visualization. Which filter you mean explicit? Do you mean a logsstash filter like i posted above or the Kibana filter? I don't know exactly where to start...


(Alexander Reelsen) #7

I was referring to a query that filters, not about logstash configuration, sorry for the confusion!


#8

I get dashboard visualization with the query filter now. I found nothing about conditions in Query DSL. I just want to get results, if i have more than 2 entries in the past 5 minutes.

Like this in watcher:

"condition": {
"compare": {
"ctx.payload.hits.total": {
"gt": 2
}

Can you help me what i need to use?
Thx a lot right now


(Alexander Reelsen) #9

visualizations and watcher are two completely different things. In case of the visualization you would just see an empty chart.

Can you explain a bit more detailed what exactly you are after - I am not really?

--Alex


#10

I want to build a dashboard where anomalies can be detected. For example, a chart should show that an unusual number of failed login attempts have occurred.

Therefore an index was created, where failed login attempts are displayed. Therefore I created a logstashconfig that indexes auth.log entries. Is it possible to index only failed logins if more than 2 failed logins have happened within the last 5 minutes and all that without watcher? For example, if only one failed login appears, do not index.

Originally I used watcher, because I don't see any other way to index more than 2 failed login attempts in a time span than reindexing.

Maybe you understand my point better now.

best regards


(system) #11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.