How to create a runtime field to create an alert?

[sirineb]

Hi,

I've created a field with a script to set the boolean "office_hours" to true of false depending on the log hours. My painless script is :

def office;
if (doc['@timestamp'].value.hour < 5 || doc['@timestamp'].value.hour > 17) {
    office = false;
} else { office = true;}
emit(office);

However, when I try to create a rule in Security > Rules to get an alert for authentications during this time (so when office_hours=false), I can't add office_hours in the query, whereas in Discover I can see it and filter data with it.

I also tried to map the field directly (with Dev Tools). I successfully added the field and it is viewable in Discover and usable in the Alert query but I don't know how to add the script to this request, given that I have an if conditional in my script. Here's my request :

PUT _all/
{
  "mappings": {
    "runtime": {
       "properties": {
          "office_hours": {
             "type": "boolean",
             "doc_values": true,
             "index": true,
             "script": {
                 "source": "myscript"
             }
          }
       }
    }
  }
}

I tried with "source": """myscript""" as it was advised in a topic but it didn't work.

So the question is :

• do I have to add the field using Kibana and it is a bug that the field cannot be used as a filter when creating a Rule ?

or

• do I have to add the field using the Dev Tools and the request ?

and then how to do it differently than I did ?

Thanks !

ES and Kibana 8.3 are running on Ubuntu 20.04.

[Pedro_Jaramillo]

Hi @sirineb, we added support for Data Views (and runtime fields) to 8.4, so you will be able to use the Security Solution rule creation UI to take advantage of runtime fields after adding them in Kibana. More info on that can be found in our docs.

In 8.3, you should be able to add the runtime field mapping directly onto the index via Dev Tools and then use that field during rule creation by following the guidance here. What is the error you're seeing in Dev Tools when trying to index your runtime field?

[sirineb]

Hi @Pedro_Jaramillo, thanks for your answer ! Thanks for the links.

Here's what I get when I try to index the field office :

{
  "error": {
    "root_cause": [
      {
        "type": "x_content_parse_exception",
        "reason": """[8:12] Unexpected character ('}' (code 125)): was expecting double-quote to start field name
 at [Source: (byte[])"{ "mappings": {
    "runtime": {
      "properties": {
        "office": {
          "type": "boolean",
          "script": {
            "source": "\n              def office;\n              if (doc['@timestamp'].value.hour < 5 || doc['@timestamp'].value.hour > 17) { \n                office = false; \n              } else { office = true;}\n              emit(office)",
          }
          "doc_values": true,
          "index": true
        }
      }
    }
  }
}
"; line: 8, column: 12]"""
      }
    ],
    "type": "x_content_parse_exception",
    "reason": """[8:12] Unexpected character ('}' (code 125)): was expecting double-quote to start field name
 at [Source: (byte[])"{ "mappings": {
    "runtime": {
      "properties": {
        "office": {
          "type": "boolean",
          "script": {
            "source": "\n              def office;\n              if (doc['@timestamp'].value.hour < 5 || doc['@timestamp'].value.hour > 17) { \n                office = false; \n              } else { office = true;}\n              emit(office)",
          }
          "doc_values": true,
          "index": true
        }
      }
    }
  }
}
"; line: 8, column: 12]""",
    "caused_by": {
      "type": "json_parse_exception",
      "reason": """Unexpected character ('}' (code 125)): was expecting double-quote to start field name
 at [Source: (byte[])"{ "mappings": {
    "runtime": {
      "properties": {
        "office": {
          "type": "boolean",
          "script": {
            "source": "\n              def office;\n              if (doc['@timestamp'].value.hour < 5 || doc['@timestamp'].value.hour > 17) { \n                office = false; \n              } else { office = true;}\n              emit(office)",
          }
          "doc_values": true,
          "index": true
        }
      }
    }
  }
}
"; line: 8, column: 12]"""
    }
  },
  "status": 400
}

And here's the full request in case it is a syntax issue.

PUT _all/
{ "mappings": {
    "runtime": {
      "properties": {
        "office": {
          "type": "boolean",
          "script": {
            "source": """
              def office;
              if (doc['@timestamp'].value.hour < 5 || doc['@timestamp'].value.hour > 17) { 
                office = false; 
              } else { office = true;}
              emit(office)""",
          }
          "doc_values": true,
          "index": true
        }
      }
    }
  }
}

[sirineb]

Hi @Pedro_Jaramillo,

I finally managed to add the field. I think the problem was with the \n. I transformed it to :

PUT _all/_mapping
{
      "runtime": {
        "office_hours": {
          "type": "boolean",
          "script": {
            "source": "def office;\r\n\r\nif (doc['@timestamp'].value.hour < 5 || doc['@timestamp'].value.hour > 17) {\r\n    office = false;\r\n} else { office = true;}\r\n\r\nemit(office);"
          }
        }
      }
}

I can visualize the field in Discover and it is consistent with the time, the field is well updated, everything's fine on that side !

After that, I wanted to create the rule but it doesn't return any value even though it is the same query as in Discover.

I don't know what I'm doing wrong, can you help me ?

Thanks

[Pedro_Jaramillo]

Hi @sirineb, we want make sure the runtime field is properly being processed and that event timestamps are not a factor in our diagnosis of the issue. To that effect, would you mind re-running the Rule Preview, but this time selecting Last day in the dropdown? Do you see any results then?

[sirineb]

Hi @Pedro_Jaramillo ,

no there aren't any results either.

I also tried to Preview without my runtime field and no results, so the problem might be the Alert itself and not my field.

But just a question to be sure that I am not going on the wrong path : to make an alert during the night, what would you suggest ? I thought about adding a runtime field because it seemed to be the easiest but maybe there're other and better ways to do, what would you do in my case ?

I looked into the Event Correlation rule type but I don't know if I can use this to do what I want above.

Thanks !

[Pedro_Jaramillo]

Hi @sirineb, the approach you've taken to create a runtime field to denote working hours seems sound to me. In the future, we're looking into facilitating the workflow of creating rules to detect events outside of working hours. But for now, the runtime field approach should work. Could you make sure that you have events in your source data index that match the query in the "Last day"? The rule preview should help with that. If it doesn't show any results, it most likely means there are no source events matching the query in that timeframe.

[sirineb]

Hi @Pedro_Jaramillo,

Thanks !
Yes I do have events in the source data index used to run the preview so that's weird.

I tried something else with EQL query and I can now preview and get the data in the alert.

Thanks