Convert json nested fields into integer

Elastic Stack Logstash
1

Hi. I am trying to convert a field read by a json filter in logstash. I am used to convert fields from other sources, but with this nested field from a json log file, I cannot convert a quoted number string to a integer.

 grok {
	 match => { "message" => "^(?m)%{TIMESTAMP_ISO8601:time} \[%{WORD:LogLevel}\s\] %{GREEDYDATA:Method}\]-%{GREEDYDATA:DATA}$"}
	 timeout_millis => 10000
     id => "alllog1"
	 }
    json{
  source => "DATA"
  }
	 mutate {convert => { "events.data.custom_attributes.txn_total_time_to_complete_in_sec" => "integer" }}
	mutate {convert => { "[events][data][custom_attribute][txn_total_time_to_complete_in_sec]" => "integer" }}
	mutate {convert => { "[txn_total_time_to_complete_in_sec]" => "integer" }}
	mutate {convert => { "txn_total_time_to_complete_in_sec" => "integer" }}
	mutate {convert => { "[events.data.custom_attributes][txn_total_time_to_complete_in_sec]" => "integer" }}

Sample logs:

2022-07-08 01:56:16,940 [INFO ] -pos_click_[pickup-cash]_[txn-confirm]-{"environment":"dev","user_identities":{"customer_id":"asd"},"user_attributes":{"agt_network_id":"df","agt_transaction_ids":["9b639c7a-79bc-4d1b-af9c-dd9802d1a5d5","73fcde89-bd7c-499b-abcd-ae42d73e2c07","2c4cf377-367c-4e08-ae15-5e0a88aa6c3d","09048a19-92c5-4b55-b69e-996615e5145e","5d89ac18-c528-4205-bb20-ce1bcf3eec40","3fdef907-42da-419f-a60e-fc8deff5de0e"],"agt_partner_id":"","agt_pos_application":"WP2","agt_session_start_time":"2022-07-08T09:51:18Z","agt_brand":"sd","agt_terminal_id":"sad","agt_operator_last_logout":"07/07/2022 18:53:07 PKT","agt_user":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36","agt_account_number":"APK111360","agt_session_id":"sd","agt_marketing_languages":null,"agt_operator_id":"ase","agt_time_zone":"Asia/Karachi","agt_app_tag_version":"1.0.129-L27-R1-M2","agt_browser_version":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36","agt_session_end_time":"","agt_fsid":"as","agt_browser_language":"er","agt_os_version":"Windows NT 10.","agt_country":"wer","agt_ip_address":"","agt_domain_url":"https://test.com","agt_language":"EN","agt_operator_last_login":"07/07/2022 18:34:41 PKT"},"events":[{"event_type":"custom_event","data":{"custom_event_type":"other","event_name":"pos_click_[pickup-cash]_[txn-confirm]","custom_attributes":{"event_completion_time_in_sec":"1.44","transaction_status":"completed","browser_start_time":"2022:07:08:05:56:13:5613","nodejs_version":"{'elavon':'1.0.0','pcSignature':'6.1.0','rsw':'6.0.0','pinpad':'5.0.0','qch':'5.0.0','termAuth':'10.0.0','verifone':'8.0.0','printerSwitching':'5.0.0','node':'14.1.0','wer':'9.0.0','wer':'3.0.0','rwer':'2.0.0'}","receive_currency":"PKR","txn_start_time_in_gmt":"2022:07:08:05:54:02:542","is_rsw_txn":"N","is_safenet_txn":"N","is_mixcase_hash_password":"N","is_printerSwitching_enabled":"N","primary_id_type":"Computerized National Identity Card (CNIC)","agent_verified_compliance_data":"Y","receiver_state":"er","transaction_term_id":"ert","txn_end_time_in_gmt":"2022:07:08:05:56:13:5613","receiver_name_type":"D","send_amount":"345.00","enabled_analytics_config":"werer, MPARTICLE","sender_country":"rt","page_name":"pickup-cash","transaction_name":"pickup-cash","is_wupc":"N","transfer_tax_total":"0.00","is_docretrieval_txn":"N","receiver_country":"PK","transaction_id":"a3995295-9f7e-4ab5-8866-35d488f594ae","message_charges":"0.00","is_nodejs_enabled":"Y","new_customer":"Y","exchange_rate":"205.3810","is_dsp_txn":"N","sender_name_type":"D","transaction_type":"PAID","is_doccapture_txn":"N","send_currency":"USD","browser_end_time":"2022:07:08:05:56:15:5615","transfer_fee":"7.00","txn_total_time_to_complete_in_sec":"131","total_amount":"307.00","is_bam_agent":"N","captured_compliance_ucb_fids":"{'01':'TEMPLATE_ID','02':'ID_TYPE','24':'THIRDPARTY_OPTION','03':'ID_NUMBER','M2':'MOBILE_PHONE_NUMBER_CODE','06':'ID_ISSUER','O6':'TEMPLATE_VERSION','29':'CURRENT_ADDRESS_LINE1','07':'DATE_OF_BIRTH','08':'OCCUPATION','09':'ID_ISSUE_DATE','C1':'MOBILE_PHONE_NUMBER','C4':'COUNTRY_CODE_PHONE_NUMBER','89':'RELATIONSHIP_TO_RECEIVER_OR_SENDER','78':'ID_HAS_EXPIRATION_DATE','30':'CURRENT_ADDRESS_CITY','53':'PROVINCE','31':'CURRENT_ADDRESS_POSTAL_CODE','10':'ID_EXPIRATION_DATE','56':'PHONE_NUMBER','67878':'RECEIVER_HAS_PHONE_NUMBER','99':'ACK_FLAG','33':'COUNTRY_OF_BIRTH','34':'NATIONALITY','82':'I_ACT_ON_MY_BEHALF','86':'CURRENT_ADDRESS_PARISH_STATE_PROVINCE_NAME','20':'CURRENT_ADDRESS_COUNTRY','21':'PURPOSE_OF_TRANSACTION'}","transaction_account_number":"tyuy","receive_amount":"5454.30","template_id":"UNI_01_P|PK_1.1","receive_city":"tyuty"}}}]}
2

Try with:

  mutate { convert => { 
     "[events][0][data][custom_attributes][txn_total_time_to_complete_in_sec]" => "integer" 
     "[events][0][data][custom_attributes][transfer_fee]" => "integer" 
     }
  }

Result:
"transaction_type" => "PAID",
"transfer_fee" => 7,
"txn_total_time_to_complete_in_sec" => 131,
"send_currency" => "USD",
"is_bam_agent" => "N",
"nodejs_version"

3

still it is same as string.

image

4

Your mapping might be old - textual, do remapping, must be number. If problems are still there, use debugger, will show you details:

output {
    stdout {  codec => rubydebug{}  }
}
5

did this one still same issue.

image
image771×133 5.35 KB

6

Quoted are strings, without quote is number or time. So your conversion is OK. Check mapping.

If is the string value will be:
"txn_total_time_to_complete_in_sec": "131"

7

sorry my bad it is working fine

1 Like
8

This one working fine. thank you so much

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.