# Extract value from message field ( runtime fields)
**URL:** https://discuss.elastic.co/t/extract-value-from-message-field-runtime-fields/311679
**Category:** Elasticsearch
**Created:** [August 8, 2022, 10:57pm UTC](https://discuss.elastic.co/t/extract-value-from-message-field-runtime-fields/311679 "2022-08-08T22:57:14Z")
**Posts on this page:** 18
**Page:** 1
### Author:  [@jplopezy](https://discuss.elastic.co/u/jplopezy)
#### Post date: [August 8, 2022, 10:57pm UTC](https://discuss.elastic.co/t/extract-value-from-message-field-runtime-fields/311679/1 "2022-08-08T22:57:15Z")
Hello everyone,
Currently, I have an OpenVPN log that is sent from the server to elastic... from what I see the fields are pretty generic and the most important one is "message", it has all the information and I can't parse it.
The most straightforward alternative is to go to "Index Patterns" and create a new field and then create a "runtime fields" script... but apparently, I can't find a convention to extract specific values ββfrom the message field because it throws me several errors...
Does anyone have an example of something like this done and working?
I mean create a field called "test" for example and create a "runtime fields" script that extracts a value from the message field and displays it
thanks
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [August 8, 2022, 11:27pm UTC](https://discuss.elastic.co/t/extract-value-from-message-field-runtime-fields/311679/2 "2022-08-08T23:27:30Z")
Hi @jplopezy Welcome to the community!
The easiest way to help would be to provide a couple sample of the log lines and what you want the result to look like (in detail)
Runtime is pretty good but perhaps you should consider and ingest pipeline and parse the message field.
Provide the before (message) and after (parsed) and perhaps someone can help you.
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [August 9, 2022, 12:21am UTC](https://discuss.elastic.co/t/extract-value-from-message-field-runtime-fields/311679/4 "2022-08-09T00:21:03Z")
Ok please provide a couple FULL samples in text (anonymized)... screen shot does no good...
Also I will say it again an ingest pipeline would probably put you in a better position long term (faster queries etc)
But sure you can start with a runtme but several full line in text not screen shot...
Is the message field the full above or just partial... be clear help us help you π
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [August 9, 2022, 1:01am UTC](https://discuss.elastic.co/t/extract-value-from-message-field-runtime-fields/311679/6 "2022-08-09T01:01:44Z")
And you only want to extract the username from the these 2 logs... after the `/CN=`
And the rest of the logs... no user field?
```auto
2019-09-23T17:49:32+0000 [stdout#info] [OVPN 1] OUT: 'Mon Sep 23 17:49:32 2019 Hide-my-ip:7305 VERIFY OK: depth=1, /CN=OpenVPN CA'
2019-09-23T17:49:32+0000 [stdout#info] [OVPN 1] OUT: 'Mon Sep 23 17:49:32 2019 Hide-my-ip:7305 VERIFY OK: depth=0, /CN=lan-with-inet_AUTOLOGIN'
```
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [August 9, 2022, 5:13am UTC](https://discuss.elastic.co/t/extract-value-from-message-field-runtime-fields/311679/8 "2022-08-09T05:13:45Z")
Thanks... I appreciate you patience and detail, and I am happy to provide an example / solutions, I have just learned that we will both get to and answer quicker if I have a good understanding exactly what you are looking for... your initial post was not enough detail.
To add a runtime field that you want to search on you have to add it to the mappings see [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/runtime-examples.html) and some examples... and remember it is added as a `field` not to the `_source` so you will be able to search on it, use it in visualizations and access but it will not be in the `_source` json
My index is named
`discuss-openvpn`
I indexed your data into the `message` field and then I added this
This is THE most Brute Force way to do this ... there are probably more elegant ways.. but as you said you just wanted and example.
This basically will parse use the grok and for the lines that do not match the `username` field is not created for those that do.. `username` will be emitted
```auto
PUT discuss-openvpn/_mappings
{
"runtime": {
"username": {
"type": "keyword",
"script": """
String username=grok('%{GREEDYDATA:leading_data}/CN=%{DATA:username}\'').extract(doc["message"].value)?.username;
if (username != null) emit(username);
"""
}
}
}
```
Then when I search for this and note it is in the `fields` not `_source`
```auto
GET discuss-openvpn/_search
{
"fields": [
"*"
],
"query": {
"exists": {
"field": "username"
}
}
}
#results
{
"took": 4,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 2,
"relation": "eq"
},
"max_score": 1,
"hits": [
{
"_index": "discuss-openvpn",
"_id": "uYPrgIIBkiWX4SWKF193",
"_score": 1,
"_source": {
"@timestamp": "2022-08-09T00:57:30.000Z",
"message": "2019-09-23T17:49:32+0000 [stdout#info] [OVPN 1] OUT: 'Mon Sep 23 17:49:32 2019 Hide-my-ip:7305 VERIFY OK: depth=1, /CN=OpenVPN CA'"
},
"fields": {
"@timestamp": [
"2022-08-09T00:57:30.000Z"
],
"message": [
"2019-09-23T17:49:32+0000 [stdout#info] [OVPN 1] OUT: 'Mon Sep 23 17:49:32 2019 Hide-my-ip:7305 VERIFY OK: depth=1, /CN=OpenVPN CA'"
],
"username": [
"OpenVPN CA"
]
}
},
{
"_index": "discuss-openvpn",
"_id": "u4PrgIIBkiWX4SWKoF-7",
"_score": 1,
"_source": {
"@timestamp": "2022-08-09T00:57:30.000Z",
"message": "2019-09-23T17:49:32+0000 [stdout#info] [OVPN 1] OUT: 'Mon Sep 23 17:49:32 2019 Hide-my-ip:7305 VERIFY OK: depth=0, /CN=lan-with-inet_AUTOLOGIN'"
},
"fields": {
"@timestamp": [
"2022-08-09T00:57:30.000Z"
],
"message": [
"2019-09-23T17:49:32+0000 [stdout#info] [OVPN 1] OUT: 'Mon Sep 23 17:49:32 2019 Hide-my-ip:7305 VERIFY OK: depth=0, /CN=lan-with-inet_AUTOLOGIN'"
],
"username": [
"lan-with-inet_AUTOLOGIN"
]
}
}
]
}
}
```
And it also shows up in Discover
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [August 9, 2022, 8:04pm UTC](https://discuss.elastic.co/t/extract-value-from-message-field-runtime-fields/311679/10 "2022-08-09T20:04:13Z")
The `discuss-openvpn` index is just my example...
You need to replace that with the actual name of your indices...
What / how are you ingesting the logs?
What version are you on?
What is the result of running
`GET _cat/indices/?v`
---
### Author:  [@jplopezy](https://discuss.elastic.co/u/jplopezy)
#### Post date: [August 9, 2022, 8:49pm UTC](https://discuss.elastic.co/t/extract-value-from-message-field-runtime-fields/311679/11 "2022-08-09T20:49:36Z")
@stephenb
On the openvpn server I have the "elastic 7.15.1" agent installed that takes the logs from "/var/log/openvpnas.log"...
On the kibana side, I have an index called "logs-\*" that shows me everything... but I wouldn't like to touch that for its integrity but I don't know how to create a new index that points to those agent logs
thanks
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [August 9, 2022, 9:28pm UTC](https://discuss.elastic.co/t/extract-value-from-message-field-runtime-fields/311679/12 "2022-08-09T21:28:08Z")
> [@jplopezy](#):
>
> "logs-\*"
That is not an index... that is an index pattern,,, 2 different but related items
an index pattern is an "alias" that points to a collection of actual indices.
I am not asking you to create an index.
You asked how to create a runtime field, I am showing you... you **add** a runtime field TO an existing index.
Go to Kibana -\> Dev Tools and run
`GET _cat/indices/?v`
That is a read only command... which will list the actual indices... not the index pattern.
I can also show you how to add the runtime field to the index pattern, which then the runtime field will only show in visualizations and discover that use the index pattern not searches etc as it is not actually added to the indices.... perhaps that is what you want ... instead...
There is a reason I ask a lot of question when people are just getting started π
So which do you want to do
a) Add a runtime field to and index ... which provides more flexibilty but involves updating / adding the runtime field to each index
b) Add the runtime field to the index-pattern `logs-*` so that you can see it in Discover and Visualizations and KQL but will not be available in DSL queries (detailed / low level queries)
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [August 9, 2022, 10:15pm UTC](https://discuss.elastic.co/t/extract-value-from-message-field-runtime-fields/311679/14 "2022-08-09T22:15:16Z")
> [@jplopezy](#):
>
> Option "a" I want to create an "index API"
Well I think you are trying to create a new index, not an Index API (index API is an existing set of REST interfaces as defined by Elasticsearch see [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices.html). ..
And index is the collection or logs / documents stored in elasticsearch ...that is managed via the Index APIs
So back to choice a) Does that that means you want to load the logs data into a new / different index so you can test with it? if so you have 2 choices
a1) Reload the data into a new / another index from scratch using the same method which the data was originally loaded
a2) Pick one of the actual indices and Reindex (i.e. make a copy of it) then work with that.
Which do you want? (so this is getting to be more than just show me a runtime field)
OR you can add a runtime field to an existing index which is non destructive but it does mean adding to its mapping / schema ..
So what is your next choice?
Plus you did not show me the output of which we will need if you choose a2) and really it will help overall.
Go to Kibana -\> Dev Tools and run
`GET _cat/indices/?v`
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [August 9, 2022, 11:48pm UTC](https://discuss.elastic.co/t/extract-value-from-message-field-runtime-fields/311679/16 "2022-08-09T23:48:51Z")
> [@jplopezy](#):
>
> What am I doing wrong?
You are mixing / not quite getting the concepts.
You are adding a runtime field to an empty index, then you are adding that empty index to an index pattern, and expecting the runtime field that is ONLY on the empty index to work on all the other indices you did not add the runtime field too ... it will not work.
So now I will be prescriptive... as you are much closer to option b so lets do that to start.
> [@stephenb](#):
>
> b) Add the runtime field to the index-pattern `logs-*` so that you can see it in Discover and Visualizations and KQL but will not be available in DSL queries (detailed / low level queries)
First delete the empty index you created...
`DELETE openvpn-dashboard`
Then delete your _open_ Index Pattern
Then recreate and index pattern lets use `logs-openvpn-*` and Add Field
Then setup the runtime field
the code
```auto
String username=grok('%{GREEDYDATA:leading_data}/CN=%{DATA:username}\'').extract(doc["message"].value)?.username;
if (username != null) emit(username);
```
Then Discover with your new index pattern,,,
You should see the fields
If you are going to be working with elasticsearch and kibana perhaps it would be of benefit to take some of the free training vids / webinars and or the getting started docs etc.
Perhaps Learn about Index, Index Template, Mappings, Data Streams, Data Types, Index Pattern/ Data Views etc
---
### Author:  [@jplopezy](https://discuss.elastic.co/u/jplopezy)
#### Post date: [August 10, 2022, 12:46am UTC](https://discuss.elastic.co/t/extract-value-from-message-field-runtime-fields/311679/17 "2022-08-10T00:46:56Z")
@stephenb
I remember that I tried that but there it gave me an error because of the type of format...
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [August 10, 2022, 1:08am UTC](https://discuss.elastic.co/t/extract-value-from-message-field-runtime-fields/311679/18 "2022-08-10T01:08:01Z")
Where was that error on Doscover?
That is because your logs `message` field is a special type `match_only_text` that does not support grok / runtime fields... so all this is for not.. nothing I can do about that... or at least nothing obvious.
Can you show the lists of fields on your index patterns?
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [August 10, 2022, 1:27am UTC](https://discuss.elastic.co/t/extract-value-from-message-field-runtime-fields/311679/20 "2022-08-10T01:27:36Z")
> [@jplopezy](#):
>
> what do you think about this option?
Not Possible, You have to do that **before** you index the data ... not after ... you can not change the mapping of a field i.e. the datatype **after** it has been indexed...
If you already have data that is indexed ... you can not change the type... period.
You can add runtime fields but they need do not operate on all types of data.
Per [Here](https://www.elastic.co/guide/en/elasticsearch/reference/current/runtime-mapping-fields.html)
> The `runtime` section can be any of these data types:
>
> - `boolean`
> - `composite`
> - `date`
> - `double`
> - `geo_point`
> - `ip`
> - `keyword`
> - `long`
> - [`lookup`](https://www.elastic.co/guide/en/elasticsearch/reference/current/runtime-retrieving-fields.html#lookup-runtime-fields)
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [August 10, 2022, 1:28am UTC](https://discuss.elastic.co/t/extract-value-from-message-field-runtime-fields/311679/21 "2022-08-10T01:28:13Z")
Do you have an `event.original` field in the index pattern?
Apologies I should have noticed this early on.. could have saved a lot of time... but at least you got some free training .. and we got to work together π
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [August 10, 2022, 1:57am UTC](https://discuss.elastic.co/t/extract-value-from-message-field-runtime-fields/311679/23 "2022-08-10T01:57:34Z")
@jplopezy Look
**OHHHHH i may have found a** way using the `_source`
> However, there are cases where retrieving fields from `_source` is necessary. For example, `text` fields do not have `doc_values` available by default, so you have to retrieve values from `_source` . In other instances, you might choose to disable `doc_values` on a specific field.
**it will not be efficient and I would not recommend at scale...**
Try this as the code in the runtime field
```auto
String username=grok('%{GREEDYDATA:leading_data}/CN=%{DATA:username}\'').extract(params._source.message)?.username;
if (username != null) emit(username);
```
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [August 10, 2022, 5:44am UTC](https://discuss.elastic.co/t/extract-value-from-message-field-runtime-fields/311679/24 "2022-08-10T05:44:31Z")
> [@jplopezy](#):
>
> what I don't understand is the difference between the message I have and yours.
My message field is a `keyword` type (think unparsed, ) versus yours which is a `text` type which means the message is tokenized for free text search.
These 2 types are among the 2 most fundamental concepts in Elasticsearch.
See [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html)
---
### Author:  [@jplopezy](https://discuss.elastic.co/u/jplopezy)
#### Post date: [August 10, 2022, 2:23pm UTC](https://discuss.elastic.co/t/extract-value-from-message-field-runtime-fields/311679/25 "2022-08-10T14:23:44Z")
@stephenb
I have nothing but words of thanks! At the last configuration, it worked correctly! Thank you for your help and all the time you spent helping me.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 7, 2022, 2:24pm UTC](https://discuss.elastic.co/t/extract-value-from-message-field-runtime-fields/311679/26 "2022-09-07T14:24:31Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.