How to remove dynamical field?

Elastic Stack Logstash
1

i have got the log looks like this below.

{ "message" => "#, @varbind_list=[#, #>, #, #>], @specific_trap=1, @source_ip="10.10.10.13", @agent_addr=#, @generic_trap=6>",
"host" => "10.10.10.13",
"@version" => "1",
"@timestamp" => "2016-04-08T06:22:17.711Z",
"type" => "snmptrap",
"BGP4-MIB::bgpPeerLastError_10_10_10_54" => "\x00\x00",
"BGP4-MIB::bgpPeerState_10_10_10_54" => "3",
"RFC1155-SMI::enterprises_9_9_187_1_2_1_1_7_10_10_10_54" => "",
"RFC1155-SMI::enterprises_9_9_187_1_2_1_1_8_10_10_10_54" => "1",
"source_ip" => "10.10.10.13"
}
{
"message" => "#, @varbind_list=[#>], @specific_trap=2, @source_ip="10.10.10.12", @agent_addr=#, @generic_trap=6>",
"host" => "10.10.10.12",
"@version" => "1",
"@timestamp" => "2016-04-08T07:09:13.040Z",
"type" => "snmptrap",
"MSDP-MIB::msdpPeerState_10_10_10_54" => "1",
"source_ip" => "10.10.10.12"
}
}
i only want to get "message" 、"host" 、 "@version"、 "@timestamp" 、"type" and "source_ip" ,
how to remove the other field ?

Can somebody give me a hint how I can fix the problem?

2

Have a look at the prune filter.

4

thank you for your help !
but if i want to add new field for the six field name and add a new field for the six field value (below type field) ,
for example from my log ,
log _ 1 ,
six_field:BGP4-MIB::bgpPeerLastError_10_10_10_54
six_field_value : \x00\x00

log_2,
six_field:MSDP-MIB::msdpPeerState_10_10_10_54
six_field_value : 1

5

Sorry, I don't understand the question. Perhaps you can give an example of an input event and the desired output event.

6

Sorry, I didn't say it clearly..
i can use prune to get "message" 、"host" 、 "@version"、 "@timestamp" 、"type" field , but i want to want to get the other dynamical field,
for example ,
{ "message" => "#, @varbind_list=[#, #>, #, #>], @specific_trap=1, @source_ip="10.10.10.13", @agent_addr=#, @generic_trap=6>",
"host" => "10.10.10.13",
"@version" => "1",
"@timestamp" => "2016-04-08T06:22:17.711Z",
"type" => "snmptrap",
"BGP4-MIB::bgpPeerLastError_10_10_10_54" => "\x00\x00",
"BGP4-MIB::bgpPeerState_10_10_10_54" => "3",
"RFC1155-SMI::enterprises_9_9_187_1_2_1_1_7_10_10_10_54" => "",
"RFC1155-SMI::enterprises_9_9_187_1_2_1_1_8_10_10_10_54" => "1",
"source_ip" => "10.10.10.13"
}

i want to get the output looks like below ,
message :[...]
timestamp:[....]
host:[....]
version:[...]
type:[.....]
BGP4-MIB::bgpPeerLastError :\x00\x00

but the line "BGP4-MIB::bgpPeerLastError :\x00\x00 " is dynamical ,so can you please tell me how to get the BGP4-MIB::bgpPeerLastError :\x00\x00 ?

please help. Thanks.

7

Oh, I see. I think you need to use a ruby filter for this. This might do it but probably isn't very efficient:

ruby {
  code => "
    prefixes = ['BGP4-MIB::bgpPeerLastError', 'BGP4-MIB::bgpPeerState']
    event.to_hash.each_pair { |k, v|
      prefixes.each { |p|
        if k.start_with? p
          event[p] = v
          event.remove(k)
        end
      }
    }
  "
}
8

i am sorry i think i didn't say it clearly.
the line "BGP4-MIB::bgpPeerLastError :\x00\x00 " is not always have the same prefix ,
for example ,
when i get the log as follows,
{ "message" => "#, @varbind_list=[#, #>, #, #>], @specific_trap=1, @source_ip="10.10.10.13", @agent_addr=#, @generic_trap=6>",
"host" => "10.10.10.13",
"@version" => "1",
"@timestamp" => "2016-04-08T06:22:17.711Z",
"type" => "snmptrap",
"RFC1155-SMI::enterprises_9_9_187_1_2_5_1_17_32_1_14_16_255_255_17_0_0_0_0_0_0_0_0_2" => "\x00\x00",
"RFC1155-SMI::enterprises_9_9_187_1_2_5_1_3_32_1_14_16_255_255_17_0_0_0_0_0_0_0_0_2" => "3",
"RFC1155-SMI::enterprises_9_9_187_1_2_5_1_28_32_1_14_16_255_255_17_0_0_0_0_0_0_0_0_2" => "",
"RFC1155-SMI::enterprises_9_9_187_1_2_5_1_29_32_1_14_16_255_255_17_0_0_0_0_0_0_0_0_2" => "1",
"source_ip" => "10.10.10.13"
}

then i want to get the output looks like below ,
message :[...]
timestamp:[....]
host:[....]
version:[...]
type:[.....]
RFC1155-SMI::enterprises :\x00\x00

and for example 2 ,
when i get the log as follows,
{ "message" => "#, @varbind_list=[#, #>, #, #>], @specific_trap=1, @source_ip="10.10.10.13", @agent_addr=#, @generic_trap=6>",
"host" => "10.10.10.13",
"@version" => "1",
"@timestamp" => "2016-04-08T06:22:17.711Z",
"type" => "snmptrap",
"MSDP-MIB::msdpPeerState_10_10_11_90" => "1",
"source_ip" => "10.10.10.13"
}

then i want to get the output looks like below ,
message :[...]
timestamp:[....]
host:[....]
version:[...]
type:[.....]
MSDP-MIB::msdpPeerState_10_10_11_90 :1

i think maybe it's can use prefix to filter ,
because I neither know the field names, nor the number of fields.
Do you know other method?
I truly appreciate ... your help in resolving the problem.

9

As long as the set of interesting prefixes is known the example I gave should work for your second example. For the first example, how is Logstash supposed to choose between the four fields with the common prefix?

10

for the first example,
should get as follows,
RFC1155-SMI::enterprises" => "\x00\x00"

but when i use the ruby code ,i got the error
Ruby exception occurred: can't add a new key into hash during iteration {:level=>:error}

11

for the first example,
should get as follows,
RFC1155-SMI::enterprises" => "\x00\x00"

I understand that, but how should Logstash reach that conclusion? Since there are multiple fields with that prefix how is Logstash supposed to know which one to pick?

Ruby exception occurred: can't add a new key into hash during iteration {:level=>:error}

Change event.to_hash.each_pair to event.to_hash.clone.each_pair.

12

Count from the message beginning of the number of sixth .
but i don't know how to make logstash to do ?

for example,
{ "message" => "..",
"host" => "10.10.10.13",
"@version" => "1",
"@timestamp" => "2016-04-08T06:22:17.711Z",
"type" => "snmptrap",
"MSDP-MIB::msdpPeerState_10_10_11_90" => "1",
"field_7"=>"field_7 content",
"source_ip" => "10.10.10.13"
}

i want to pick the sixth field is "MSDP-MIB::msdpPeerState_10_10_11_90" => "1",
i want the output looks like as below,

"message" => "..",
"host" => "10.10.10.13",
"@version" => "1",
"@timestamp" => "2016-04-08T06:22:17.711Z",
"type" => "snmptrap",
"MSDP-MIB::msdpPeerState_10_10_11_90" => "1",
"source_ip" => "10.10.10.13"

13

Hello,
Anybody has any good suggestion?