I got a query which creates a new field sophos.utm.to.domain. That query extracts the full mail domain. The mail address which is stored in sophos.utm.to.
My Question: How do I create a new field out of this query?
See the new runtime fields feature of mappings
I tried that and get the following error reason: "There are no external requests known to support wildcards that don't support replacing their indices
I'd need to see the JSON of what you tried to respond further
Sure. I tried following query:
PUT filebeat*
{
"mappings": {
"runtime": {
"sophos.utm.to.domain": {
"type": "keyword",
"script": {
"source": "emit(def m = /@((?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\\[(?:(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9]))\\.){3}(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9])|[a-z0-9-]*[a-z0-9]:(?:[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21-\\x5a\\x53-\\x7f]|\\\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f])+)\\]))/.matcher(doc['sophos.utm.to'].value); return m.find() ? m.group(1): '';)"
}
}
}
}
}
And i got this response:
{
"error" : {
"root_cause" : [
{
"type" : "security_exception",
"reason" : "action [indices:admin/create] is unauthorized for user [test]"
}
],
"type" : "security_exception",
"reason" : "action [indices:admin/create] is unauthorized for user [test]",
"caused_by" : {
"type" : "illegal_state_exception",
"reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
}
},
"status" : 403
}
The "test" user is assigned to the superuser role
Something is unhappy with your levels of authorisation.
What happens when you try doing it to a single index rather than using *?
when only addressing the filebeat-test-7.8.0-2021.02.01-000100 index i get following error:
{
"error" : {
"root_cause" : [
{
"type" : "resource_already_exists_exception",
"reason" : "index [filebeat-test-7.8.0-2021.02.01-000100/Cz1ks68kQ_-bMOhb03PjcA] already exists",
"index_uuid" : "Cz1ks68kQ_-bMOhb03PjcA",
"index" : "filebeat-test-7.8.0-2021.02.01-000100"
}
],
"type" : "resource_already_exists_exception",
"reason" : "index [filebeat-test-7.8.0-2021.02.01-000100/Cz1ks68kQ_-bMOhb03PjcA] already exists",
"index_uuid" : "Cz1ks68kQ_-bMOhb03PjcA",
"index" : "filebeat-test-7.8.0-2021.02.01-000100"
},
"status" : 400
}That's the API trying to create an index (which already exists).
To add a partial update to an existing mapping you need to user the _mapping API e.g. adding a new field to an existing index using:
PUT myExistingIndex/_mapping
{
"properties": {
"myNewField": {
"type": "text"
}
}
}
ah, thanks. But how do i put my script inside of this new field?
As per the docs
Note however the examples assume you're creating a new index with a mapping so remove the mappings wrapper for the fields from the JSON example and put _mapping in the URL instead.
It's my bad.. I just found out, that according to this issue on github, the Runtime fields were added in Version 7.11. Currently, our elk stack is running on 7.8. I will upgrade and test your suggestions. Thank you so far! 
1 Like
So i just upgraded my stack to 7.11 but i still get an error:
I try following:
PUT /filebeat-*/_mapping
{
"runtime": {
"sophos.utm.to.domain": {
"type": "keyword",
"script": {
"source": "emit(def m = /@((?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\\[(?:(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9]))\\.){3}(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9])|[a-z0-9-]*[a-z0-9]:(?:[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21-\\x5a\\x53-\\x7f]|\\\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f])+)\\]))/.matcher(doc['sophos.utm.to'].value); return m.find() ? m.group(1): '';)"
}
}
}
}
and i get this error:
{
"error" : {
"root_cause" : [
{
"type" : "script_exception",
"reason" : "compile error",
"script_stack" : [
"emit(def m = /@((?:(?:[a-z0-9](?:[a- ...",
" ^---- HERE"
],
"script" : "emit(def m = /@((?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\\[(?:(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9]))\\.){3}(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9])|[a-z0-9-]*[a-z0-9]:(?:[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21-\\x5a\\x53-\\x7f]|\\\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f])+)\\]))/.matcher(doc['sophos.utm.to'].value); return m.find() ? m.group(1): '';)",
"lang" : "painless",
"position" : {
"offset" : 11,
"start" : 0,
"end" : 36
}
}
],
"type" : "script_exception",
"reason" : "compile error",
"script_stack" : [
"emit(def m = /@((?:(?:[a-z0-9](?:[a- ...",
" ^---- HERE"
],
"script" : "emit(def m = /@((?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\\[(?:(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9]))\\.){3}(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9])|[a-z0-9-]*[a-z0-9]:(?:[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21-\\x5a\\x53-\\x7f]|\\\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f])+)\\]))/.matcher(doc['sophos.utm.to'].value); return m.find() ? m.group(1): '';)",
"lang" : "painless",
"position" : {
"offset" : 11,
"start" : 0,
"end" : 36
},
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "invalid sequence of tokens near ['='].",
"caused_by" : {
"type" : "no_viable_alt_exception",
"reason" : "no_viable_alt_exception: null"
}
}
},
"status" : 400
}OK so I think I got it working, using "scripted fields".
I put following script as painless:
if (doc['sophos.utm.to'].size() == 0) return '';
Matcher m = /@([\.\w\-_]+\.[a-zA-Z]+)/.matcher(doc['sophos.utm.to'].value);
return m.find() ? m.group(1): '';Hello,
the same script with a little adjustment should fit in the script for a runtime keyword field like you tried to do above. Mostly the return statements need to be replaced with emit function calls.
if (doc['sophos.utm.to'].size() == 0) emit('');
Matcher m = /@([\.\w\-_]+\.[a-zA-Z]+)/.matcher(doc['sophos.utm.to'].value);
if (m.find()) emit(m.group(1)) else emit('');
Would love to hear if this works for you, especially given that you upgraded to 7.11 specifically to try runtime fields out.
Cheers
Luca
1 Like
Even better, you could only emit a value when there is one:
if (doc['sophos.utm.to'].size() > 0) {
Matcher m = /@([\.\w\-_]+\.[a-zA-Z]+)/.matcher(doc['sophos.utm.to'].value);
if (m.find()) emit(m.group(1))
}
1 Like
I did that. But now, how do i update my existing indices with the "new" index template?
Index templates don't take effect on existing indices. In that case you need to do a put mapping call against the existing indices. Is your intention to add the same runtime field to existing indices too?
Cheers
Luca
yes i'd like to add this runtime field to existing indices.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.