# Create new field from query

**URL:** <https://discuss.elastic.co/t/create-new-field-from-query/263892>\
**Category:** Elasticsearch\
**Created:** [February 10, 2021, 3:42pm UTC](https://discuss.elastic.co/t/create-new-field-from-query/263892 "2021-02-10T15:42:10Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![paasi6666](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paasi6666/32/77617_2.png) [@paasi6666](https://discuss.elastic.co/u/paasi6666)\
**Post date:** [February 10, 2021, 3:42pm UTC](https://discuss.elastic.co/t/create-new-field-from-query/263892/1 "2021-02-10T15:42:10Z")

</div>

I got a query which creates a new field `sophos.utm.to.domain`. That query extracts the full mail domain. The mail address which is stored in `sophos.utm.to`.

My Question: How do I create a new field out of this query?

 ![2021-02-10_163740](https://us1.discourse-cdn.com/elastic/original/3X/a/7/a73cbc159c99ac27b90a28ff1e1a3325ab8e3492.png)

---

<div class="post-metadata">

**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\
**Post date:** [February 10, 2021, 3:50pm UTC](https://discuss.elastic.co/t/create-new-field-from-query/263892/2 "2021-02-10T15:50:29Z")

</div>

See the new [runtime fields](https://www.elastic.co/guide/en/elasticsearch/reference/7.11/runtime.html) feature of mappings

---

<div class="post-metadata">

**Author:** ![paasi6666](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paasi6666/32/77617_2.png) [@paasi6666](https://discuss.elastic.co/u/paasi6666)\
**Post date:** [February 10, 2021, 4:15pm UTC](https://discuss.elastic.co/t/create-new-field-from-query/263892/3 "2021-02-10T16:15:03Z")

</div>

I tried that and get the following error reason: `"There are no external requests known to support wildcards that don't support replacing their indices`

---

<div class="post-metadata">

**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\
**Post date:** [February 10, 2021, 4:45pm UTC](https://discuss.elastic.co/t/create-new-field-from-query/263892/4 "2021-02-10T16:45:48Z")

</div>

I'd need to see the JSON of what you tried to respond further

---

<div class="post-metadata">

**Author:** ![paasi6666](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paasi6666/32/77617_2.png) [@paasi6666](https://discuss.elastic.co/u/paasi6666)\
**Post date:** [February 10, 2021, 5:03pm UTC](https://discuss.elastic.co/t/create-new-field-from-query/263892/5 "2021-02-10T17:03:00Z")

</div>

Sure. I tried following query:

```
PUT filebeat*
{
  "mappings": {
    "runtime": {
      "sophos.utm.to.domain": {
        "type": "keyword",
        "script": {
          "source": "emit(def m = /@((?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\\[(?:(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9]))\\.){3}(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9])|[a-z0-9-]*[a-z0-9]:(?:[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21-\\x5a\\x53-\\x7f]|\\\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f])+)\\]))/.matcher(doc['sophos.utm.to'].value); return m.find() ? m.group(1): '';)"
        }
      }
    }
  }
} 

```

And i got this response:

```
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "action [indices:admin/create] is unauthorized for user [test]"
      }
    ],
    "type" : "security_exception",
    "reason" : "action [indices:admin/create] is unauthorized for user [test]",
    "caused_by" : {
      "type" : "illegal_state_exception",
      "reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
    }
  },
  "status" : 403
}

```

The "test" user is assigned to the superuser role

---

<div class="post-metadata">

**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\
**Post date:** [February 10, 2021, 5:06pm UTC](https://discuss.elastic.co/t/create-new-field-from-query/263892/6 "2021-02-10T17:06:36Z")

</div>

> [@paasi6666](#):
>
> The "test" user is assigned to the superuser role

Something is unhappy with your levels of authorisation.  
What happens when you try doing it to a single index rather than using `*`?

---

<div class="post-metadata">

**Author:** ![paasi6666](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paasi6666/32/77617_2.png) [@paasi6666](https://discuss.elastic.co/u/paasi6666)\
**Post date:** [February 10, 2021, 5:10pm UTC](https://discuss.elastic.co/t/create-new-field-from-query/263892/7 "2021-02-10T17:10:05Z")

</div>

when only addressing the filebeat-test-7.8.0-2021.02.01-000100 index i get following error:

```
{
  "error" : {
    "root_cause" : [
      {
        "type" : "resource_already_exists_exception",
        "reason" : "index [filebeat-test-7.8.0-2021.02.01-000100/Cz1ks68kQ_-bMOhb03PjcA] already exists",
        "index_uuid" : "Cz1ks68kQ_-bMOhb03PjcA",
        "index" : "filebeat-test-7.8.0-2021.02.01-000100"
      }
    ],
    "type" : "resource_already_exists_exception",
    "reason" : "index [filebeat-test-7.8.0-2021.02.01-000100/Cz1ks68kQ_-bMOhb03PjcA] already exists",
    "index_uuid" : "Cz1ks68kQ_-bMOhb03PjcA",
    "index" : "filebeat-test-7.8.0-2021.02.01-000100"
  },
  "status" : 400
}
```

---

<div class="post-metadata">

**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\
**Post date:** [February 10, 2021, 5:15pm UTC](https://discuss.elastic.co/t/create-new-field-from-query/263892/8 "2021-02-10T17:15:08Z")

</div>

That's the API trying to create an index (which already exists).

To add a partial update to an existing mapping you need to user the `_mapping` API e.g. adding a new field to an existing index using:

```auto
PUT myExistingIndex/_mapping
{
  "properties": {
    "myNewField": {
      "type": "text"
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![paasi6666](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paasi6666/32/77617_2.png) [@paasi6666](https://discuss.elastic.co/u/paasi6666)\
**Post date:** [February 10, 2021, 5:23pm UTC](https://discuss.elastic.co/t/create-new-field-from-query/263892/9 "2021-02-10T17:23:32Z")

</div>

ah, thanks. But how do i put my script inside of this new field?

---

<div class="post-metadata">

**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\
**Post date:** [February 10, 2021, 5:27pm UTC](https://discuss.elastic.co/t/create-new-field-from-query/263892/10 "2021-02-10T17:27:03Z")

</div>

As per [the docs](https://www.elastic.co/guide/en/elasticsearch/reference/7.11/runtime-mapping-fields.html)  
Note however the examples assume you're creating a new index with a mapping so remove the `mappings` wrapper for the fields from the JSON example and put `_mapping` in the URL instead.

---

<div class="post-metadata">

**Author:** ![paasi6666](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paasi6666/32/77617_2.png) [@paasi6666](https://discuss.elastic.co/u/paasi6666)\
**Post date:** [February 10, 2021, 5:41pm UTC](https://discuss.elastic.co/t/create-new-field-from-query/263892/11 "2021-02-10T17:41:07Z")

</div>

> [@Mark\_Harwood](#):
>
> index with a map

It's my bad.. I just found out, that according to this [issue](https://github.com/elastic/elasticsearch/issues/59332) on github, the Runtime fields were added in Version 7.11. Currently, our elk stack is running on 7.8. I will upgrade and test your suggestions. Thank you so far! 🙂

---

<div class="post-metadata">

**Author:** ![paasi6666](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paasi6666/32/77617_2.png) [@paasi6666](https://discuss.elastic.co/u/paasi6666)\
**Post date:** [February 11, 2021, 7:10am UTC](https://discuss.elastic.co/t/create-new-field-from-query/263892/12 "2021-02-11T07:10:07Z")

</div>

So i just upgraded my stack to 7.11 but i still get an error:  
I try following:

```
PUT /filebeat-*/_mapping
{
  "runtime": {
    "sophos.utm.to.domain": {
      "type": "keyword",
      "script": {
        "source": "emit(def m = /@((?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\\[(?:(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9]))\\.){3}(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9])|[a-z0-9-]*[a-z0-9]:(?:[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21-\\x5a\\x53-\\x7f]|\\\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f])+)\\]))/.matcher(doc['sophos.utm.to'].value); return m.find() ? m.group(1): '';)"
      }
    }
  }
}

```

and i get this error:

```
{
  "error" : {
    "root_cause" : [
      {
        "type" : "script_exception",
        "reason" : "compile error",
        "script_stack" : [
          "emit(def m = /@((?:(?:[a-z0-9](?:[a- ...",
          " ^---- HERE"
        ],
        "script" : "emit(def m = /@((?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\\[(?:(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9]))\\.){3}(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9])|[a-z0-9-]*[a-z0-9]:(?:[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21-\\x5a\\x53-\\x7f]|\\\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f])+)\\]))/.matcher(doc['sophos.utm.to'].value); return m.find() ? m.group(1): '';)",
        "lang" : "painless",
        "position" : {
          "offset" : 11,
          "start" : 0,
          "end" : 36
        }
      }
    ],
    "type" : "script_exception",
    "reason" : "compile error",
    "script_stack" : [
      "emit(def m = /@((?:(?:[a-z0-9](?:[a- ...",
      " ^---- HERE"
    ],
    "script" : "emit(def m = /@((?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\\[(?:(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9]))\\.){3}(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9])|[a-z0-9-]*[a-z0-9]:(?:[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21-\\x5a\\x53-\\x7f]|\\\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f])+)\\]))/.matcher(doc['sophos.utm.to'].value); return m.find() ? m.group(1): '';)",
    "lang" : "painless",
    "position" : {
      "offset" : 11,
      "start" : 0,
      "end" : 36
    },
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "invalid sequence of tokens near ['='].",
      "caused_by" : {
        "type" : "no_viable_alt_exception",
        "reason" : "no_viable_alt_exception: null"
      }
    }
  },
  "status" : 400
}
```

---

<div class="post-metadata">

**Author:** ![paasi6666](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paasi6666/32/77617_2.png) [@paasi6666](https://discuss.elastic.co/u/paasi6666)\
**Post date:** [February 11, 2021, 7:47am UTC](https://discuss.elastic.co/t/create-new-field-from-query/263892/13 "2021-02-11T07:47:31Z")

</div>

OK so I think I got it working, using "scripted fields".  
I put following script as painless:

```
if (doc['sophos.utm.to'].size() == 0) return '';
Matcher m = /@([\.\w\-_]+\.[a-zA-Z]+)/.matcher(doc['sophos.utm.to'].value);
return m.find() ? m.group(1): '';
```

---

<div class="post-metadata">

**Author:** ![javanna](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/javanna/32/4698_2.png) [@javanna](https://discuss.elastic.co/u/javanna)\
**Post date:** [February 11, 2021, 10:57am UTC](https://discuss.elastic.co/t/create-new-field-from-query/263892/14 "2021-02-11T10:57:37Z")

</div>

Hello,  
the same script with a little adjustment should fit in the script for a runtime keyword field like you tried to do above. Mostly the return statements need to be replaced with emit function calls.

```auto
if (doc['sophos.utm.to'].size() == 0) emit('');
Matcher m = /@([\.\w\-_]+\.[a-zA-Z]+)/.matcher(doc['sophos.utm.to'].value);
if (m.find()) emit(m.group(1)) else emit('');

```

Would love to hear if this works for you, especially given that you upgraded to 7.11 specifically to try runtime fields out.

Cheers  
Luca

---

<div class="post-metadata">

**Author:** ![javanna](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/javanna/32/4698_2.png) [@javanna](https://discuss.elastic.co/u/javanna)\
**Post date:** [February 11, 2021, 10:59am UTC](https://discuss.elastic.co/t/create-new-field-from-query/263892/15 "2021-02-11T10:59:19Z")

</div>

Even better, you could only emit a value when there is one:

```auto
if (doc['sophos.utm.to'].size() > 0) {
    Matcher m = /@([\.\w\-_]+\.[a-zA-Z]+)/.matcher(doc['sophos.utm.to'].value);
    if (m.find()) emit(m.group(1))
}

```

---

<div class="post-metadata">

**Author:** ![paasi6666](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paasi6666/32/77617_2.png) [@paasi6666](https://discuss.elastic.co/u/paasi6666)\
**Post date:** [February 16, 2021, 7:03am UTC](https://discuss.elastic.co/t/create-new-field-from-query/263892/16 "2021-02-16T07:03:01Z")

</div>

I did that. But now, how do i update my existing indices with the "new" index template?

---

<div class="post-metadata">

**Author:** ![javanna](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/javanna/32/4698_2.png) [@javanna](https://discuss.elastic.co/u/javanna)\
**Post date:** [February 16, 2021, 9:32am UTC](https://discuss.elastic.co/t/create-new-field-from-query/263892/17 "2021-02-16T09:32:13Z")

</div>

Index templates don't take effect on existing indices. In that case you need to do a put mapping call against the existing indices. Is your intention to add the same runtime field to existing indices too?

Cheers  
Luca

---

<div class="post-metadata">

**Author:** ![paasi6666](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paasi6666/32/77617_2.png) [@paasi6666](https://discuss.elastic.co/u/paasi6666)\
**Post date:** [February 16, 2021, 11:57am UTC](https://discuss.elastic.co/t/create-new-field-from-query/263892/18 "2021-02-16T11:57:03Z")

</div>

yes i'd like to add this runtime field to existing indices.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 16, 2021, 11:57am UTC](https://discuss.elastic.co/t/create-new-field-from-query/263892/19 "2021-03-16T11:57:46Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.