All fields from logstash coming in as text to ES, Pls let me know how to set as keyword?

Bhanu_Praveen · July 30, 2024, 4:28pm

Below is my json log string:

{
	"traceId": "o0uyveRU/8BkjL+lbpDlnQ==",
	"spanId": "73rmqIRmo34=",
	"operationName": "ProcessWorkItem",
	"startTime": "2024-07-22T06:07:12.650881409Z",
	"duration": "0.256620153s",
	"tags": [
		{
			"key": "otel.library.name",
			"vStr": "com.mdi.core.workmgmt.TypedConsumer"
		},
		{
			"key": "ta.work.type",
			"vStr": "INBOUND_ENTITY_INTEGRATION"
		},
		{
			"key": "ta.companyCode",
			"vStr": "AMBER_ROAD"
		},
		{
			"key": "ta.work.status",
			"vStr": "COMPLETED"
		},
		{
			"key": "ta.work.loadDuration",
			"vType": "INT64"
		},
		{
			"key": "service.version",
			"vStr": "24.2"
		},
		{
			"key": "work.processorclass",
			"vStr": "com.mdi.core.entity.integration.EntityStdIntegrationWorkProcessor"
		},
		{
			"key": "ta.work.receivedTime",
			"vType": "INT64",
			"vInt64": "1721628429423"
		},
		{
			"key": "ta.work.delegateDuration",
			"vType": "INT64",
			"vInt64": "257"
		},
		{
			"key": "ta.work.busKeySearch",
			"vStr": "AMBER_ROAD;SO_TEST_93;SALES_ORDER"
		},
		{
			"key": "ta.orgCode",
			"vStr": "AMBER_ROAD"
		},
		{
			"key": "ta.work.id",
			"vStr": "33267221"
		},
		{
			"key": "length",
			"vType": "INT64",
			"vInt64": "3680"
		},
		{
			"key": "ta.work.reportStatusDuration",
			"vType": "INT64",
			"vInt64": "4"
		},
		{
			"key": "ta.work.errorCount",
			"vType": "INT64"
		},
		{
			"key": "ta.work.delegateConsumerTime",
			"vType": "INT64",
			"vInt64": "253"
		},
		{
			"key": "span.kind",
			"vStr": "internal"
		},
		{
			"key": "otel.status_code",
			"vStr": "OK"
		},
		{
			"key": "internal.span.format",
			"vStr": "otlp"
		}
	],
	"process": {
		"serviceName": "TRADE_SINGLETON",
		"tags": [
			{
				"key": "deployment.environment",
				"vStr": "PSR"
			},
			{
				"key": "e2.environment.name",
				"vStr": "OTEL_PST"
			},
			{
				"key": "e2.instance.name",
				"vStr": "TRADE_SINGLETON_1"
			},
			{
				"key": "e2.product.family",
				"vStr": "GTM"
			},
			{
				"key": "e2.product.tenant.id",
				"vStr": "WMMERCURYTA"
			},
			{
				"key": "service.instance.id",
				"vStr": "URN:PSR:GTM:TA:OTEL_PST:TRADE_SINGLETON:TRADE_SINGLETON_1"
			},
			{
				"key": "service.namespace",
				"vStr": "TA"
			},
			{
				"key": "telemetry.sdk.language",
				"vStr": "java"
			},
			{
				"key": "telemetry.sdk.name",
				"vStr": "opentelemetry"
			},
			{
				"key": "telemetry.sdk.version",
				"vStr": "1.34.0"
			}
		]
	},
	"tag": "jaeger_spans"
}

I am using filters as below:

filter 
{ 
	json 
	{
		source => "message"
	}
		
	json 
	{
		source => "message"
		target => "tmessage"
	}
	
	#ruby { 
	#	code => "event.set('startTimeMillis', ((event.get('startTime').to_f*1000).to_i))" 
	#	}

	mutate
	{
		add_field => {"process.serviceName" => "%{[tmessage][process][serviceName]}"}
	}
	
	ruby {
		code => '
			event.get("[tmessage][tags]").each { |a|
				name = a["key"]
				name=name.gsub(/\./,"@")
				if a["vStr"]
					value = a["vStr"]
				elsif a["vInt64"] 
					value = a["vInt64"]
				else
					value = ""
				end
				event.set("tag." + name, value)
				}
			'
		}
				
	ruby {
		code => '
			event.get("[tmessage][process][tags]").each { |a|
				name = a["key"]
				name=name.gsub(/\./,"@")
				if a["vStr"]
					value = a["vStr"]
				elsif a["vInt64"]
					value = a["vInt64"]
				else
					value = ""
				end
				event.set("process.tag." + name, value)
			}
		'
		}
		
		  mutate {
			convert => { "duration"  => "float" }
			convert => { "tag.length"  => "float" }
			}
		
	mutate {
		remove_field => [ "message", "tmessage","tags"]
	}

}

Pls let me know where would be the issue or how to get fields of string type as keyword in Elastic?

Christian_Dahlqvist · July 30, 2024, 4:41pm

In order to control how fields are mapped you need to create an index template.

Logstash configuration can change how the data being sent to Elasticsearch is structured, but does not control how the fields are mapped once Elasticsearch receives the data.

dadoonet · July 30, 2024, 4:45pm

From Elastic Search to Elasticsearch

Bhanu_Praveen · July 31, 2024, 1:28am

ya Christian, i got a default indices created with below mappings:

{
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "@version": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "duration": {
        "type": "float"
      },
      "event": {
        "properties": {
          "original": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "sequence": {
            "type": "long"
          }
        }
      },
      "host": {
        "properties": {
          "name": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      },
      "operationName": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "process": {
        "properties": {
          "serviceName": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "tag": {
            "properties": {
              "deployment@environment": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "e2@environment@name": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "e2@instance@name": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "e2@product@family": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "e2@product@tenant@id": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "service@instance@id": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "service@namespace": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "telemetry@sdk@language": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "telemetry@sdk@name": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "telemetry@sdk@version": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          }
        }
      },
      "spanId": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "startTime": {
        "type": "date"
      },
      "tag": {
        "properties": {
          "internal@span@format": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "length": {
            "type": "float"
          },
          "otel@library@name": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "otel@status_code": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "service@version": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "span@kind": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "ta@companyCode": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "ta@orgCode": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "ta@work@busKeySearch": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "ta@work@delegateConsumerTime": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "ta@work@delegateDuration": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "ta@work@errorCount": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "ta@work@id": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "ta@work@loadDuration": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "ta@work@receivedTime": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "ta@work@reportStatusDuration": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "ta@work@status": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "ta@work@type": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "work@processorclass": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      },
      "traceId": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      }
    }
  }
}

Then i copied this mapping, changed type:text to keyword and created a index template as below:


{
  "dynamic_templates": [],
  "properties": {
    "duration": {
      "type": "long",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "spanId": {
      "type": "keyword",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "traceId": {
      "type": "keyword",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "process": {
      "type": "object",
      "properties": {
        "tag": {
          "type": "object",
          "properties": {
            "telemetry@sdk@name": {
              "type": "keyword",
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              }
            },
            "service@instance@id": {
              "type": "keyword",
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              }
            },
            "service@namespace": {
              "type": "keyword",
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              }
            },
            "e2@instance@name": {
              "type": "keyword",
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              }
            },
            "e2@product@family": {
              "type": "keyword",
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              }
            },
            "deployment@environment": {
              "type": "keyword",
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              }
            },
            "e2@environment@name": {
              "type": "keyword",
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              }
            },
            "e2@product@tenant@id": {
              "type": "keyword",
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              }
            },
            "telemetry@sdk@language": {
              "type": "keyword",
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              }
            },
            "telemetry@sdk@version": {
              "type": "keyword",
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              }
            }
          }
        },
        "serviceName": {
          "type": "keyword",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        }
      }
    },
    "@timestamp": {
      "type": "date"
    },
    "@version": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "host": {
      "type": "object",
      "properties": {
        "name": {
          "type": "keyword",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        }
      }
    },
    "operationName": {
      "type": "keyword",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "startTime": {
      "type": "long"
    },
    "tag": {
      "type": "object",
      "properties": {
        "otel@library@name": {
          "type": "keyword",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "span@kind": {
          "type": "keyword",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "ta@work@status": {
          "type": "keyword",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "length": {
          "type": "long",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "ta@work@delegateConsumerTime": {
          "type": "keyword",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "internal@span@format": {
          "type": "keyword",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "service@version": {
          "type": "keyword",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "ta@companyCode": {
          "type": "keyword",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "ta@work@delegateDuration": {
          "type": "long",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "work@processorclass": {
          "type": "keyword",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "ta@work@reportStatusDuration": {
          "type": "long",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "otel@status_code": {
          "type": "keyword",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "ta@work@loadDuration": {
          "type": "long",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "ta@work@errorCount": {
          "type": "long",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "ta@orgCode": {
          "type": "keyword",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "ta@work@busKeySearch": {
          "type": "keyword",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "ta@work@type": {
          "type": "keyword",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "ta@work@receivedTime": {
          "type": "long",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "ta@work@id": {
          "type": "keyword",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        }
      }
    },
    "event": {
      "type": "object",
      "properties": {
        "sequence": {
          "type": "long"
        },
        "original": {
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        }
      }
    }
  }
}

But still i am unable to get the data type correctly. Pls let me know what is the issue?

Bhanu_Praveen · August 1, 2024, 1:56am

Got it thanks

system · August 29, 2024, 1:56am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Logstash--Using ruby to convert json to fields. Data going to elastic as text. How to set to long? Logstash	1	16	July 30, 2024
I am curious when sending data from logstash to elasticsearch Elasticsearch	8	373	August 4, 2020
Field NUMBER save as String Logstash	1	335	January 1, 2019
Possible to grok an Elasticsearch document fields? Logstash	4	428	January 12, 2017
Split row data to fields Logstash	7	2112	July 6, 2017

All fields from logstash coming in as text to ES, Pls let me know how to set as keyword?

Related Topics