Hello,
I am trying to setup custom logging for Mulesoft APIs and trying to send data to an ELK server (instead of Kibana, it uses OpenSearch from AWS). I am also using a custom JSON Logger that is required for separate logging. The issues I encounter is that I have to map the field "message" to be able to search queries for data specific in that field
Ex:
message: {
"correlationId" : "5aa2f4c0-7d09-11ed-aba3-4a89e7edc87b",
"message" : "Start",
"tracePoint" : "START",
"priority" : "INFO",
"elapsed" : 0,
"locationInfo" : {
"lineInFile" : "36",
"component" : "json-logger:logger",
"fileName" : "local-to-salesforce.xml",
"rootContainer" : "post-accounts"
},
"timestamp" : "2022-12-16T06:17:44.816Z",
"content" : {
"payload" : [ { <some payload>
},
"applicationName" : "local-to-salesforce",
"applicationVersion" : "1.0.0",
"environment" : "dev",
"threadName" : "[MuleRuntime].uber.24: [logs-to-opensearch].post-accounts.BLOCKING @59c71deb"
}
I know that that Elasticsearch does auto mapping to the fields that it receives in the logs, the problem is that the "message" field (first one) is mapped to text and keyword and I cannot do a sort after tracePoint values (for example). I tried updating the mapping of that field, but maybe I am doing something wrong and this might not be the best approach or might be entirely wrong one.
I have the following update I want to do:
PUT mule-logs/_mapping
{
"mappings": {
"properties": {
"message": {
"type": "text",
"properties": {
"correlationId": {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"message": {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"tracePoint": {
"type": "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"priority": {
"type": "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"elapsed": {
"type": "long"
},
"locationInfo": {
"type": "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"timestamp": {
"type": "date"
},
"content": {
"type": "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"applicationName": {
"type": "string"
},
"applicationVersion": {
"type": "text"
},
"environment": {
"type": "string"
},
"threadName": {
"type": "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
}
}
And I get the following error:
{
"error" : {
"root_cause" : [
{
"type" : "mapper_parsing_exception",
"reason" : "Root mapping definition has unsupported parameters: [mappings : {properties={message={type=text, properties={elapsed={type=long}, applicationVersion={type=text}, locationInfo={type=text, fields={keyword={ignore_above=256, type=keyword}}}, environment={type=string}, correlationId={type=text, fields={keyword={ignore_above=256, type=keyword}}}, tracePoint={type=text, fields={keyword={ignore_above=256, type=keyword}}}, message={type=text, fields={keyword={ignore_above=256, type=keyword}}}, priority={type=text, fields={keyword={ignore_above=256, type=keyword}}}, content={type=text, fields={keyword={ignore_above=256, type=keyword}}}, applicationName={type=string}, threadName={type=text, fields={keyword={ignore_above=256, type=keyword}}}, timestamp={type=date}}}}}]"
}
],
"type" : "mapper_parsing_exception",
"reason" : "Root mapping definition has unsupported parameters: [mappings : {properties={message={type=text, properties={elapsed={type=long}, applicationVersion={type=text}, locationInfo={type=text, fields={keyword={ignore_above=256, type=keyword}}}, environment={type=string}, correlationId={type=text, fields={keyword={ignore_above=256, type=keyword}}}, tracePoint={type=text, fields={keyword={ignore_above=256, type=keyword}}}, message={type=text, fields={keyword={ignore_above=256, type=keyword}}}, priority={type=text, fields={keyword={ignore_above=256, type=keyword}}}, content={type=text, fields={keyword={ignore_above=256, type=keyword}}}, applicationName={type=string}, threadName={type=text, fields={keyword={ignore_above=256, type=keyword}}}, timestamp={type=date}}}}}]"
},
"status" : 400
}
Which I understand it's because the syntax is incorrect. Any idea how should I approach this? Thank you!
)