How to remove a filed

Elastic Stack Logstash
1

Hi Team,

Please can you suggest how to remove field "payload" from below pattern:

I am trying using remove_field => [ "payload" ]. Please suggest.

{match => { "message" => "[%{GREEDYDATA:BEGIN}]|%{GREEDYDATA:service_id}|%{GREEDYDATA:log_level}|%{GREEDYDATA:log_type}|%{GREEDYDATA:log_code}|%{GREEDYDATA:log_message}|%{GREEDYDATA:log_guid}|%{GREEDYDATA:org_code}|%{GREEDYDATA:process_name}|%{GREEDYDATA:si_id}|%{GREEDYDATA:operation_name}|%{GREEDYDATA:service_requestor}|%{GREEDYDATA:service_provider}|%{GREEDYDATA:machine_name}|%{GREEDYDATA:engine_name}|%{GREEDYDATA:payload}|%{GREEDYDATA:pre_step}|%{GREEDYDATA:post_step}|%{GREEDYDATA:log_timestamp}|%{GREEDYDATA:si_id}|%{GREEDYDATA:EOD}
remove_field => [ "payload" ]

2

If you don't want to catch the payload field just replace %{GREEDYDATA:payload} with %{GREEDYDATA}.

But that grok expression is awful. That amount of GREEDYDATA patterns is extremely inefficient and could give the wrong results. Just use a csv filter.

3

Thankyou magnus
I tried using mutate { remove_field => ["payload"]}
And it worked

4

Hi Team,

For below input:
BEGIN|ABC1|3|fdfd||SRServiceRequest|sfdf-20171023-17.02.00.979478|||fdfda|dfd|dfdf||sdfd|fdf|dfgfdkj|fdfdf|AUDIT2|2017-10-23T17:02:01.113+08:00|fdfdf|EOD

i am using grok pattern:
[%{GREEDYDATA:BEGIN}]|%{GREEDYDATA:service_id}|%{GREEDYDATA:log_level}|%{GREEDYDATA:log_type}|%{GREEDYDATA:log_code}|%{GREEDYDATA:log_message}|%{GREEDYDATA:log_guid}|%{GREEDYDATA:org_code}|%{GREEDYDATA:process_name}|%{GREEDYDATA:si_id}|%{GREEDYDATA:operation_name}|%{GREEDYDATA:service_requestor}|%{GREEDYDATA:service_provider}|%{GREEDYDATA:machine_name}|%{GREEDYDATA:engine_name}|%{GREEDYDATA:payload}|%{GREEDYDATA:pre_step}|%{GREEDYDATA:post_step}|%{GREEDYDATA:log_timestamp}|%{GREEDYDATA:si_id}|%{GREEDYDATA:EOD}

In below output i see service_id is populating from BEGIN instead of ABC1 alone.

Output below:
{
"BEGIN": [
null
],
"service_id": [
"BEGIN|ABC1"
],
"log_level": [
"3"
],
.........more.

Please can you suggest how to populate BEING with BEGIN constant instead of null.

5

As I said: That grok expression is awful. That amount of GREEDYDATA patterns is extremely inefficient and could give the wrong results. Just use a csv filter.

Side note: Always post configuration and log entries as preformatted text to avoid getting the text damaged by the discussion forum software. In this case it has probably damaged your grok expression so I can't tell what you're using.

6

Hi Magnus,

Sorry. Kindly could you suggest pattern so being element is populated instead of null and filed service_id does not take BEGIN.

INPUT:
BEGIN|REWARDPOINTSACCOUNT100|3|AUDIT||SRServiceRequest|IS@SG-156-20171023-17.02.00.979478|||d6e66635-8bc1-486c-9fa5-4e3c04f1dcba|retrievePartyRewardSummary|ICAL||s01sbw5app2a|RewardPointsAccount_SG_v1_0_513_2-Process_Archive|<?xml version="1.0" encoding="UTF-8"?> <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Header><ns4:MsgDetl xmlns:ns4="http://schemas.care.com/soi/common/4_0" xmlns:ns3="http://schemas.care.com/soi/retrievePartyRewardSummary/4_0" xmlns:ns2="http://schemas.care.com/soi/common/4_1"><ns4:MsgVersion>4.0</ns4:MsgVersion><ns4:MsgUID>IS@SG-156-20171023-17.02.00.979478</ns4:MsgUID><ns4:SvcVersion>4.0.0</ns4:SvcVersion></ns4:MsgDetl><ns4:Trace xmlns:ns4="http://schemas.care.com/soi/common/4_0" xmlns:ns3="http://schemas.care.com/soi/retrievePartyRewardSummary/4_0" xmlns:ns2="http://schemas.care.com/soi/common/4_1"><ns4:RqDateTime>2017-10-23T17:02:00.980+08:00</ns4:RqDateTime><ns4:RqClient><ns4:RqClientId>ICAL</ns4:RqClientId><ns4:RqClientOrg>0001</ns4:RqClientOrg><ns4:RqClientCtry>SG</ns4:RqClientCtry></ns4:RqClient><ns4:Operator><ns4:OpInternalId>DRADICA2</ns4:OpInternalId><ns4:OpLoginId>CRMUATA</ns4:OpLoginId><ns4:OpRole>TELLER</ns4:OpRole></ns4:Operator></ns4:Trace><ns4:ExtendedHeader xmlns:ns4="http://schemas.care.com/soi/common/4_0" xmlns:ns3="http://schemas.care.com/soi/retrievePartyRewardSummary/4_0" xmlns:ns2="http://schemas.care.com/soi/common/4_1"/></soap:Header><soap:Body><ns2:retrievePartyRewardSummary xmlns:ns2="http://schemas.care.com/soi/retrievePartyRewardSummary/4_0" xmlns:ns3="http://schemas.care.com/soi/common/4_0" xmlns:ns4="http://schemas.care.com/soi/common/4_1"><ns4:CommonRq><ns4:OrgCode>0001</ns4:OrgCode><ns4:ChannelId>96</ns4:ChannelId></ns4:CommonRq><ns4:TrailDetl><ns4:OpId>CRMUATA</ns4:OpId><ns4:OpDateTime>2017-10-23T17:02:00.980+08:00</ns4:OpDateTime><ns4:OpMainUnit>0117</ns4:OpMainUnit></ns4:TrailDetl><ns4:CISInternalId><ns4:CISCIN>S1139026I</ns4:CISCIN><ns4:CISCINsfx>00</ns4:CISCINsfx></ns4:CISInternalId></ns2:retrievePartyRewardSummary></soap:Body></soap:Envelope>|d6e66635-8bc1-486c-9fa5-4e3c04f1dcba|AUDIT2|2017-10-23T17:02:01.113+08:00|d6e66635-8bc1-486c-9fa5-4e3c04f1dcba|EOD

PATTERN:
[%{GREEDYDATA:BEGIN}|%{GREEDYDATA:service_id}|%{GREEDYDATA:log_level}|%{GREEDYDATA:log_type}|%{GREEDYDATA:log_code}|%{GREEDYDATA:log_message}|%{GREEDYDATA:log_guid}|%{GREEDYDATA:org_code}|%{GREEDYDATA:process_name}|%{GREEDYDATA:si_id}|%{GREEDYDATA:operation_name}|%{GREEDYDATA:service_requestor}|%{GREEDYDATA:service_provider}|%{GREEDYDATA:machine_name}|%{GREEDYDATA:engine_name}|%{GREEDYDATA:pay_load}|%{GREEDYDATA:pre_step}|%{GREEDYDATA:post_step}|%{GREEDYDATA:log_timestamp}|%{GREEDYDATA:si_id}|%{GREEDYDATA:EOD}

OUTPUT:

{
"BEGIN": [
null
],
"service_id": [
"BEGIN|REWARDPOINTSACCOUNT100"
],
"log_level": [
"3"
],
"log_type": [
"AUDIT"
],
"log_code": [
""
],
"log_message": [
"SRServiceRequest"
],
"log_guid": [
"IS@SG-156-20171023-17.02.00.979478"
],
"org_code": [
""
],
"process_name": [
""
],
"si_id": [
"d6e66635-8bc1-486c-9fa5-4e3c04f1dcba",
"d6e66635-8bc1-486c-9fa5-4e3c04f1dcba"
],
"operation_name": [
"retrievePartyRewardSummary"
],
"service_requestor": [
"ICAL"
],
"service_provider": [
""
],
"machine_name": [
"s01sbw5app2a"
],
"engine_name": [
"RewardPointsAccount_SG_v1_0_513_2-Process_Archive"
],
"pay_load": [
"<?xml version="1.0" encoding="UTF-8"?> <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Header><ns4:MsgDetl xmlns:ns4="http://schemas.care.com/soi/common/4_0" xmlns:ns3="http://schemas.care.com/soi/retrievePartyRewardSummary/4_0" xmlns:ns2="http://schemas.care.com/soi/common/4_1"><ns4:MsgVersion>4.0</ns4:MsgVersion><ns4:MsgUID>IS@SG-156-20171023-17.02.00.979478</ns4:MsgUID><ns4:SvcVersion>4.0.0</ns4:SvcVersion></ns4:MsgDetl><ns4:Trace xmlns:ns4="http://schemas.care.com/soi/common/4_0" xmlns:ns3="http://schemas.care.com/soi/retrievePartyRewardSummary/4_0" xmlns:ns2="http://schemas.care.com/soi/common/4_1"><ns4:RqDateTime>2017-10-23T17:02:00.980+08:00</ns4:RqDateTime><ns4:RqClient><ns4:RqClientId>ICAL</ns4:RqClientId><ns4:RqClientOrg>0001</ns4:RqClientOrg><ns4:RqClientCtry>SG</ns4:RqClientCtry></ns4:RqClient><ns4:Operator><ns4:OpInternalId>DRADICA2</ns4:OpInternalId><ns4:OpLoginId>CRMUATA</ns4:OpLoginId><ns4:OpRole>TELLER</ns4:OpRole></ns4:Operator></ns4:Trace><ns4:ExtendedHeader xmlns:ns4="http://schemas.care.com/soi/common/4_0" xmlns:ns3="http://schemas.care.com/soi/retrievePartyRewardSummary/4_0" xmlns:ns2="http://schemas.care.com/soi/common/4_1"/></soap:Header><soap:Body><ns2:retrievePartyRewardSummary xmlns:ns2="http://schemas.care.com/soi/retrievePartyRewardSummary/4_0" xmlns:ns3="http://schemas.care.com/soi/common/4_0" xmlns:ns4="http://schemas.care.com/soi/common/4_1"><ns4:CommonRq><ns4:OrgCode>0001</ns4:OrgCode><ns4:ChannelId>96</ns4:ChannelId></ns4:CommonRq><ns4:TrailDetl><ns4:OpId>CRMUATA</ns4:OpId><ns4:OpDateTime>2017-10-23T17:02:00.980+08:00</ns4:OpDateTime><ns4:OpMainUnit>0117</ns4:OpMainUnit></ns4:TrailDetl><ns4:CISInternalId><ns4:CISCIN>S1139026I</ns4:CISCIN><ns4:CISCINsfx>00</ns4:CISCINsfx></ns4:CISInternalId></ns2:retrievePartyRewardSummary></soap:Body></soap:Envelope>"
],
"pre_step": [
"d6e66635-8bc1-486c-9fa5-4e3c04f1dcba"
],
"post_step": [
"AUDIT2"
],
"log_timestamp": [
"2017-10-23T17:02:01.113+08:00"
],
"EOD": [
"EOD"
]
}

7

You can try with below grok pattern

image
image943×112 7.3 KB

8

You can try with below grok pattern

This might appear to work but it's still insanely inefficient and potentially fragile.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.