Mutate and gsub usage

Elastic Stack Logstash
1

Hi,

I was trying to fetch below fileds from the log

2024-01-10 04:21:52.018 -06:00 [INF] [2696100223720240110042151-10] {"Message":"Device_Response_2696100223720240110042151-10","ApiEndPoint":"ws://10.136.41.18:50000/","ResponseCode":"","Store":"2696","Terminal":"100","TransactionId":"2237","CorrelationId":"2696100223720240110042151-10","RequestPayload":"{"request":{"resource":{"type":"info"},"flow_id":"2696100223720240110042151-10","endpoint":"/upp/v1/device"}","ResponsePayload":"{"response" : {"endpoint" : "/upp/v1/device","flow_id" : "2696100223720240110042151-10","resource" : {"status" : "completed","unit_data" : {"application" : "Unified Payment Platform","battery" : {"battery_charging_state" : "NotAvailable","battery_level" : "N/A"},"contactless_emv" : {"config" : "/HOST/EMVCLESS.XML","contactless_interface_support" : "Yes","discover_kernel_version" : "020000","expresspay_v2_kernel_version" : "NONE","expresspay_v3_kernel_version" : "070100","interac_kernel_version" : "020500","jspeedy_kernel_version" : "040300","paypass_v3_app_version" : "030700","paypass_v3_kernel_version" : "090200","upi_kernel_version" : "040700","visa_paywave_kernel_version" : "070803"},"emv" : {"config" : "/HOST/EMVCONTACT.XML","engine_version" : "063400","kernel_version" : "090600"},"general" : {"device_model" : "L7000","ecr_no" : "3456","manufacture" : "INGNAR","manufacture_serial_no" : "24694446","unit_serial_no" : "3011295624694446"},"memory" : {"flash_size" : "491040","ram_size" : "506116"},"transaction" : {"cashback_limit" : "999999"},"version" : {"application_version" : "7.83.27-0022","digitizer_version" : "0000","eftl_version" : "0100","eftp_version" : "0100","os_version" : "0506","pci_version" : "0.0.0.0.00.00-0000","security_module_version" : "0508"}}}","TimeTaken":"00:00:00.3931037","TimeStamp":null,"Exception":null,"LogLevel":"Information","LogLevelEnum":2}

Fields to fetch:

  1. Terminal
  2. Apiendpoint
  3. store
  4. application version
  5. transaction id

Wrote a grok pattern like below

%{TIMESTAMP_ISO8601:app_timestamp}%{SPACE}%{GREEDYDATA}\]%{SPACE}%{GREEDYDATA:response}RequestPayload%{GREEDYDATA}application_version%{GREEDYDATA:application_version}digitizer_version

Grok Simulate:

{
  "response": "{\"Message\":\"Device_Response_2696100223720240110042151-10\",\"ApiEndPoint\":\"ws://10.136.41.18:50000/\",\"ResponseCode\":\"\",\"Store\":\"2696\",\"Terminal\":\"100\",\"TransactionId\":\"2237\",\"CorrelationId\":\"2696100223720240110042151-10\",\"",
  "app_timestamp": "2024-01-10 04:21:52.018",
  "application_version": "\" : \"7.83.27-0022\",\""

How can I get the

  1. Terminal
  2. Apiendpoint
  3. store
  4. transaction id
    using mutate and gsub?
2

edit 0

3

If your [message] contain valid JSON then you could use something like

    mutate { gsub => ["message", "^[^{]+{", "{" ] }
    json { source => "message" target => "[@metadata][json]" }
    mutate {
        rename => {
            "[@metadata][json][TransactionId]" => "TransactionId"
            "[@metadata][json][Terminal]" => "Terminal"
        }
    }

to pull out the fields you want. However, it is either missing escapes on the excess double quotes or it needs some double quotes removed and } added.

Since you want so little of the JSON data another approach would be

    grok {
        break_on_match => false
        match => {
            "message" => [
                '"TransactionId"\s*:\s*"%{NUMBER:TransactionId}"',
                '"ApiEndPoint"\s*:\s*"%{URI:ApiEndPoint}"',
                '"Terminal"\s*:\s*"%{NUMBER:Terminal}"'
            ]
        }
    }
1 Like
4

Hi,

I have tried above code and its not fetching the fields. Can you guide to fix the below code which I have written below to fetch data.

filter 
{

    if "Terminal" in [message] and "ApiEndPoint" in [message] and "application_version" in [message] and  "store" in [message] and "TransactionId" in [message]
    {

       grok 
       {
            match => { 
            "message" =>"%{TIMESTAMP_ISO8601:app_timestamp}%{SPACE}%{GREEDYDATA}\]%{SPACE}%{GREEDYDATA:response}RequestPayload%{GREEDYDATA}application_version%{GREEDYDATA:application_version}digitizer_version"
          }
		  
		 }
		  
		     mutate {
       gsub => ["response", "^[^{]+{", "{" ] 
    }
		  mutate {
            remove_field => [ "Message" ]
          }
		  
		  if ApiEndPoint in [response]
	{
	
	grok
	{
	match => {
	"response"
	=>"%{URI:ApiEndPoint}"
	}
 }
 
 mutate {
            remove_field => [ "ResponseCode" ]
          }
kv {
    value_split => ":"
    field_split => ","
}

}
    else
    {
      drop{}
    }
    
 }
![response|690x59](upload://oloVv0I8j4ndCrqyEBipEUD3Bzj.png)
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.