Created field not avail in ML fields

[mathurin68]

With alot of help from the discussion board I'm getting closer to this...

I created a field ...

filter {
if [event_id] == 4688 and [event_data][CommandLine] !~ /"(.*?)"/ {
mutate {
add_field => {"event_data.Suspicious" => ""}
copy => {"[event_data][CommandLine]" => "event_data.Suspicious" }
}
}
}
... this seemed to automatically make it a keyword.

But the field isn't available in Machine learning

I event tried adding it directly to the JSON for the ML job and that didn't seem to work.

What am I doing wrong now?

[dmitri (Dimitris Athanasiou)]

Hi Matt,

Could you use the get-field-mappings API to see the mappings of the field you are creating?

[mathurin68]

Thanks Dimitris, just when I thought I was sort of getting it...

This seems to work ...

filter {
if [event_id] == 4688 and [event_data][CommandLine] !~ /"(.*?)"/ {
mutate {
copy => {"[event_data][CommandLine]" => "[event_data][Suspicious]" }
}
}
}

and gives me -

and I guess looks ok in the json of the event in Discover -
"event_data": {
"CommandLine": "cmd.exe /c del C:\temp\backdoor.bat",
"Suspicious": "cmd.exe /c del C:\temp\backdoor.bat",
},

....but nothing in by_field_name for Machine Learning

[dmitri (Dimitris Athanasiou)]

Could you compare the mappings of event_data.Suspicious to something that appears on that list, say event_data.SubStatus?

[richcollier]

My guess is that the field name is actually just called Suspicious and not event_data.Suspicious.

Matt can you confirm?

[mathurin68]

Not sure,

and ...

There is one in there... but I think it might be from a previous bungled attempts I made to add/copy the field, I seen to have both in the dev console but NOT available fields in ML -

"winlogbeat-2017.12.01": {
"mappings": {
"doc": {
"Suspicious": {
"full_name": "Suspicious",
"mapping": {
"Suspicious": {
"type": "keyword",
"ignore_above": 1024
}
}
}
}
}
},

"winlogbeat-2017.12.04": {
"mappings": {
"doc": {
"event_data.Suspicious": {
"full_name": "event_data.Suspicious",
"mapping": {
"Suspicious": {
"type": "keyword",
"ignore_above": 1024
}
}
}
}
}
}

[dmitri (Dimitris Athanasiou)]

Hi Matt,

I have now confirmed copy_to fields are ignored from those drop-downs in the UI. This will be fixed in future release.

As a workaround, you can manually enter the field name in the JSON tab.