Scripted Field Help

Elastic Stack Kibana
1

Hello,

I am trying to use a scripted field to compare the value of one field against another. A sample document extract is below:

Field1 C2_Host: ["check.example.workers.dev"]
Field2 HostHeader: "Host: check.example.workers.dev\r\n"

The scripted field I am trying to create, DomainFronting, is as follows:

//Returns True if the HostHeader is different to C2_Host
if (doc['HostHeader.keyword'].value == "") {
    return "False";
}

else if (doc['HostHeader.keyword'].value == "Not Found") {
    return "False";
}

else {
    String CleanHostHeader = doc['HostHeader.keyword'].value.splitOnToken('Host: ')[1];
    String CleanHostHeader2 = CleanHostHeader.splitOnToken('\\')[0];

    if (doc['C2_Host.keyword'].value == CleanHostHeader2) {
        return "False";
    }
    else {
        return "True";
    }
}

Which should, in theory:

  • Return False is HostHeader.keyword is empty;
  • Return False if HostHeader.keyword has a value of "Not Found";
  • Return False if HostHeader.keyword is the same as C2_Host;
  • Return True if HostHeader.keyword is different to C2_Host;

Unfortunately, for the above example, the outcome is True, despite both values being the same. When I change the return value to instead return CleanHostHeader2, I can see that the string is correct, and an exact match to C2_Host. Can someone steer me in the right direction, as to where I am going wrong please? I feel like I am very nearly there.

Many thanks!

2

Yes @snkhan Question before digging in do you really mean this fields is an Array?
the [] signifies and array ... if so that may be why your script is not working...

2nd you should really be using a runtime field not a scripted field that is the new approach.

3

Indeed it is, but I would have thought doc['C2_Host.keyword'].value would just pick out its value? I am very new to ELK, and running 7.17, so thought Scripted Fields were the way to go. I see the note about it in the interface, but the link simply takes me to the help page, and it seems that RunTime fields cannot be configured in the GUI :frowning:

4

Yes the can very easily and you can see the results ... give me a couple minutes and I will show you

What version are you running...

I can also see that you did not create a mapping... you should that is why you are haveing to use the

QUESTION : if C2_Host is an Array what do you expect to happy if there are 2 or more values? Should it only compare the first? What is the logic...

5

Thanks Stephen. I am running 7.17. Appreciate the comment if there are multiple C2_Host values, ideally I would have it as if CleanHostHeader2 in C2_Host (pseudocode), but for the time being happy for it to just pick the first value, as that is what I am seeing in 99% of cases.

What I also do not understand is, if I return the value of CleanHostHeader2 instead, the string is exactly the same as C2_Host if there is 1 domain, so in that case I would absolutely expect the condition to return False, and yet it still returns True.

6 
    // You had a trailing space
    String CleanHostHeader2 = CleanHostHeader.splitOnToken('\\')[0].trim();  <!--- trim

You have a bug... a trailing space... that will get you started...

Here is my current code for a runtime field

if (doc['HostHeader'].value == "") {
    return emit("false");
}

else if (doc['HostHeader'].value == "Not Found") {
    return emit("false");
}

else {
    String CleanHostHeader = doc['HostHeader'].value.splitOnToken('Host: ')[1];
    
    // You had a trailing space
    String CleanHostHeader2 = CleanHostHeader.splitOnToken('\\')[0].trim();


    if (CleanHostHeader2.equals(doc['C2_Host'].value))  {
     return emit("true");
    }
    else {
     return emit("false");
    }
}

This code will looks like it will look at the first in the array... if you really wanted to you would have to loop through...

1 Like
7

OMG, thank you! That fixed it :slight_smile:

8

Incidentally, where do you create a RunTime field? Via mappings?

9

Ok so now here is the interesting Part

You can add this through the index Pattern

Kibana - Stack Management - Index Pattern - Add Runtime Field

(Mine is 8.x index patterns are now called Data Views (you should get to 8.x lots of good stuff there)

So if you add a runtime field in an index pattern it will work in visualizations etc... BUT it will not work in DSL queries etc...

If you want it to work everywhere you add it to the mapping

I like creating them in the UI then I move them to the mapping ... just a suggestion

You can click through the docs in your index to see how the code is working

Screen Shot 2022-10-09 at 11.30.12 AM
Screen Shot 2022-10-09 at 11.30.12 AM1920×1133 112 KB

Speaking of the mapping you should be declaring yours...

Example plus all your other fields

PUT discuss-test-runtime/
{
  "mappings": {
    "properties": {
      "C2_Host": {"type": "keyword"},
       "HostHeader": {"type": "keyword"}
    }
  }
}

So you should read up on Mapping which is a big topic in general then look at adding your runtime field

All that said adding the field to your index pattern is a great way to start...

1 Like
10

Basically I use emit like printf good old string debugging I always put leading / trailing characters to see what is there...

1 Like
11

I am new to ELK and very very new to Painless, so I was doing the equivalent by returning the value of the variable, haha!

12

Yeah it's pretty much like Java I find in the UI just put the type and then the dot and then it'll show you all the functions. But the editor is not super helpful.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.