# All fields from logstash coming in as text to ES, Pls let me know how to set as keyword?
**URL:** https://discuss.elastic.co/t/all-fields-from-logstash-coming-in-as-text-to-es-pls-let-me-know-how-to-set-as-keyword/364076
**Category:** Elasticsearch
**Created:** [July 30, 2024, 4:28pm UTC](https://discuss.elastic.co/t/all-fields-from-logstash-coming-in-as-text-to-es-pls-let-me-know-how-to-set-as-keyword/364076 "2024-07-30T16:28:50Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@Bhanu\_Praveen](https://discuss.elastic.co/u/Bhanu_Praveen)
#### Post date: [July 30, 2024, 4:28pm UTC](https://discuss.elastic.co/t/all-fields-from-logstash-coming-in-as-text-to-es-pls-let-me-know-how-to-set-as-keyword/364076/1 "2024-07-30T16:28:50Z")
All fields from logstash coming in as text to ES, Pls let me know how to set as keyword?
Below is my json log string:
```auto
{
"traceId": "o0uyveRU/8BkjL+lbpDlnQ==",
"spanId": "73rmqIRmo34=",
"operationName": "ProcessWorkItem",
"startTime": "2024-07-22T06:07:12.650881409Z",
"duration": "0.256620153s",
"tags": [
{
"key": "otel.library.name",
"vStr": "com.mdi.core.workmgmt.TypedConsumer"
},
{
"key": "ta.work.type",
"vStr": "INBOUND_ENTITY_INTEGRATION"
},
{
"key": "ta.companyCode",
"vStr": "AMBER_ROAD"
},
{
"key": "ta.work.status",
"vStr": "COMPLETED"
},
{
"key": "ta.work.loadDuration",
"vType": "INT64"
},
{
"key": "service.version",
"vStr": "24.2"
},
{
"key": "work.processorclass",
"vStr": "com.mdi.core.entity.integration.EntityStdIntegrationWorkProcessor"
},
{
"key": "ta.work.receivedTime",
"vType": "INT64",
"vInt64": "1721628429423"
},
{
"key": "ta.work.delegateDuration",
"vType": "INT64",
"vInt64": "257"
},
{
"key": "ta.work.busKeySearch",
"vStr": "AMBER_ROAD;SO_TEST_93;SALES_ORDER"
},
{
"key": "ta.orgCode",
"vStr": "AMBER_ROAD"
},
{
"key": "ta.work.id",
"vStr": "33267221"
},
{
"key": "length",
"vType": "INT64",
"vInt64": "3680"
},
{
"key": "ta.work.reportStatusDuration",
"vType": "INT64",
"vInt64": "4"
},
{
"key": "ta.work.errorCount",
"vType": "INT64"
},
{
"key": "ta.work.delegateConsumerTime",
"vType": "INT64",
"vInt64": "253"
},
{
"key": "span.kind",
"vStr": "internal"
},
{
"key": "otel.status_code",
"vStr": "OK"
},
{
"key": "internal.span.format",
"vStr": "otlp"
}
],
"process": {
"serviceName": "TRADE_SINGLETON",
"tags": [
{
"key": "deployment.environment",
"vStr": "PSR"
},
{
"key": "e2.environment.name",
"vStr": "OTEL_PST"
},
{
"key": "e2.instance.name",
"vStr": "TRADE_SINGLETON_1"
},
{
"key": "e2.product.family",
"vStr": "GTM"
},
{
"key": "e2.product.tenant.id",
"vStr": "WMMERCURYTA"
},
{
"key": "service.instance.id",
"vStr": "URN:PSR:GTM:TA:OTEL_PST:TRADE_SINGLETON:TRADE_SINGLETON_1"
},
{
"key": "service.namespace",
"vStr": "TA"
},
{
"key": "telemetry.sdk.language",
"vStr": "java"
},
{
"key": "telemetry.sdk.name",
"vStr": "opentelemetry"
},
{
"key": "telemetry.sdk.version",
"vStr": "1.34.0"
}
]
},
"tag": "jaeger_spans"
}
```
I am using filters as below:
```auto
filter
{
json
{
source => "message"
}
json
{
source => "message"
target => "tmessage"
}
#ruby {
# code => "event.set('startTimeMillis', ((event.get('startTime').to_f*1000).to_i))"
# }
mutate
{
add_field => {"process.serviceName" => "%{[tmessage][process][serviceName]}"}
}
ruby {
code => '
event.get("[tmessage][tags]").each { |a|
name = a["key"]
name=name.gsub(/\./,"@")
if a["vStr"]
value = a["vStr"]
elsif a["vInt64"]
value = a["vInt64"]
else
value = ""
end
event.set("tag." + name, value)
}
'
}
ruby {
code => '
event.get("[tmessage][process][tags]").each { |a|
name = a["key"]
name=name.gsub(/\./,"@")
if a["vStr"]
value = a["vStr"]
elsif a["vInt64"]
value = a["vInt64"]
else
value = ""
end
event.set("process.tag." + name, value)
}
'
}
mutate {
convert => { "duration" => "float" }
convert => { "tag.length" => "float" }
}
mutate {
remove_field => ["message", "tmessage","tags"]
}
}
```
Pls let me know where would be the issue or how to get fields of string type as keyword in Elastic?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 30, 2024, 4:41pm UTC](https://discuss.elastic.co/t/all-fields-from-logstash-coming-in-as-text-to-es-pls-let-me-know-how-to-set-as-keyword/364076/2 "2024-07-30T16:41:03Z")
In order to control how fields are mapped you need to create [an index template](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html).
Logstash configuration can change how the data being sent to Elasticsearch is structured, but does not control how the fields are mapped once Elasticsearch receives the data.
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [July 30, 2024, 4:45pm UTC](https://discuss.elastic.co/t/all-fields-from-logstash-coming-in-as-text-to-es-pls-let-me-know-how-to-set-as-keyword/364076/3 "2024-07-30T16:45:29Z")
From #Elastic Search to #Elasticsearch
---
### Author:  [@Bhanu\_Praveen](https://discuss.elastic.co/u/Bhanu_Praveen)
#### Post date: [July 31, 2024, 1:28am UTC](https://discuss.elastic.co/t/all-fields-from-logstash-coming-in-as-text-to-es-pls-let-me-know-how-to-set-as-keyword/364076/4 "2024-07-31T01:28:26Z")
ya Christian, i got a default indices created with below mappings:
```auto
{
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"duration": {
"type": "float"
},
"event": {
"properties": {
"original": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"sequence": {
"type": "long"
}
}
},
"host": {
"properties": {
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"operationName": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"process": {
"properties": {
"serviceName": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"tag": {
"properties": {
"deployment@environment": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"e2@environment@name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"e2@instance@name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"e2@product@family": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"e2@product@tenant@id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"service@instance@id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"service@namespace": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"telemetry@sdk@language": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"telemetry@sdk@name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"telemetry@sdk@version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
},
"spanId": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"startTime": {
"type": "date"
},
"tag": {
"properties": {
"internal@span@format": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"length": {
"type": "float"
},
"otel@library@name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"otel@status_code": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"service@version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"span@kind": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"ta@companyCode": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"ta@orgCode": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"ta@work@busKeySearch": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"ta@work@delegateConsumerTime": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"ta@work@delegateDuration": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"ta@work@errorCount": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"ta@work@id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"ta@work@loadDuration": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"ta@work@receivedTime": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"ta@work@reportStatusDuration": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"ta@work@status": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"ta@work@type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"work@processorclass": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"traceId": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
```
Then i copied this mapping, changed type:text to keyword and created a index template as below:
```auto
{
"dynamic_templates": [],
"properties": {
"duration": {
"type": "long",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"spanId": {
"type": "keyword",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"traceId": {
"type": "keyword",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"process": {
"type": "object",
"properties": {
"tag": {
"type": "object",
"properties": {
"telemetry@sdk@name": {
"type": "keyword",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"service@instance@id": {
"type": "keyword",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"service@namespace": {
"type": "keyword",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"e2@instance@name": {
"type": "keyword",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"e2@product@family": {
"type": "keyword",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"deployment@environment": {
"type": "keyword",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"e2@environment@name": {
"type": "keyword",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"e2@product@tenant@id": {
"type": "keyword",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"telemetry@sdk@language": {
"type": "keyword",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"telemetry@sdk@version": {
"type": "keyword",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
}
}
},
"serviceName": {
"type": "keyword",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
}
}
},
"@timestamp": {
"type": "date"
},
"@version": {
"type": "text",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"host": {
"type": "object",
"properties": {
"name": {
"type": "keyword",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
}
}
},
"operationName": {
"type": "keyword",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"startTime": {
"type": "long"
},
"tag": {
"type": "object",
"properties": {
"otel@library@name": {
"type": "keyword",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"span@kind": {
"type": "keyword",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"ta@work@status": {
"type": "keyword",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"length": {
"type": "long",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"ta@work@delegateConsumerTime": {
"type": "keyword",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"internal@span@format": {
"type": "keyword",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"service@version": {
"type": "keyword",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"ta@companyCode": {
"type": "keyword",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"ta@work@delegateDuration": {
"type": "long",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"work@processorclass": {
"type": "keyword",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"ta@work@reportStatusDuration": {
"type": "long",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"otel@status_code": {
"type": "keyword",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"ta@work@loadDuration": {
"type": "long",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"ta@work@errorCount": {
"type": "long",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"ta@orgCode": {
"type": "keyword",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"ta@work@busKeySearch": {
"type": "keyword",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"ta@work@type": {
"type": "keyword",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"ta@work@receivedTime": {
"type": "long",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"ta@work@id": {
"type": "keyword",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
}
}
},
"event": {
"type": "object",
"properties": {
"sequence": {
"type": "long"
},
"original": {
"type": "text",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
}
}
}
}
}
```
But still i am unable to get the data type correctly. Pls let me know what is the issue?
---
### Author:  [@Bhanu\_Praveen](https://discuss.elastic.co/u/Bhanu_Praveen)
#### Post date: [August 1, 2024, 1:56am UTC](https://discuss.elastic.co/t/all-fields-from-logstash-coming-in-as-text-to-es-pls-let-me-know-how-to-set-as-keyword/364076/5 "2024-08-01T01:56:21Z")
Got it thanks
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 29, 2024, 1:56am UTC](https://discuss.elastic.co/t/all-fields-from-logstash-coming-in-as-text-to-es-pls-let-me-know-how-to-set-as-keyword/364076/6 "2024-08-29T01:56:47Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.