How to remove dynamical field?


(saye) #1

i have got the log looks like this below.

{ "message" => "#, @varbind_list=[#, #>, #, #>], @specific_trap=1, @source_ip="10.10.10.13", @agent_addr=#, @generic_trap=6>",
"host" => "10.10.10.13",
"@version" => "1",
"@timestamp" => "2016-04-08T06:22:17.711Z",
"type" => "snmptrap",
"BGP4-MIB::bgpPeerLastError_10_10_10_54" => "\x00\x00",
"BGP4-MIB::bgpPeerState_10_10_10_54" => "3",
"RFC1155-SMI::enterprises_9_9_187_1_2_1_1_7_10_10_10_54" => "",
"RFC1155-SMI::enterprises_9_9_187_1_2_1_1_8_10_10_10_54" => "1",
"source_ip" => "10.10.10.13"
}
{
"message" => "#, @varbind_list=[#>], @specific_trap=2, @source_ip="10.10.10.12", @agent_addr=#, @generic_trap=6>",
"host" => "10.10.10.12",
"@version" => "1",
"@timestamp" => "2016-04-08T07:09:13.040Z",
"type" => "snmptrap",
"MSDP-MIB::msdpPeerState_10_10_10_54" => "1",
"source_ip" => "10.10.10.12"
}
}
i only want to get "message" 、"host" 、 "@version"、 "@timestamp" 、"type" and "source_ip" ,
how to remove the other field ?

Can somebody give me a hint how I can fix the problem?


(Magnus Bäck) #2

Have a look at the prune filter.


(saye) #4

thank you for your help !
but if i want to add new field for the six field name and add a new field for the six field value (below type field) ,
for example from my log ,
log _ 1 ,
six_field:BGP4-MIB::bgpPeerLastError_10_10_10_54
six_field_value : \x00\x00

log_2,
six_field:MSDP-MIB::msdpPeerState_10_10_10_54
six_field_value : 1


(Magnus Bäck) #5

Sorry, I don't understand the question. Perhaps you can give an example of an input event and the desired output event.


(saye) #6

Sorry, I didn't say it clearly..
i can use prune to get "message" 、"host" 、 "@version"、 "@timestamp" 、"type" field , but i want to want to get the other dynamical field,
for example ,
{ "message" => "#, @varbind_list=[#, #>, #, #>], @specific_trap=1, @source_ip="10.10.10.13", @agent_addr=#, @generic_trap=6>",
"host" => "10.10.10.13",
"@version" => "1",
"@timestamp" => "2016-04-08T06:22:17.711Z",
"type" => "snmptrap",
"BGP4-MIB::bgpPeerLastError_10_10_10_54" => "\x00\x00",
"BGP4-MIB::bgpPeerState_10_10_10_54" => "3",
"RFC1155-SMI::enterprises_9_9_187_1_2_1_1_7_10_10_10_54" => "",
"RFC1155-SMI::enterprises_9_9_187_1_2_1_1_8_10_10_10_54" => "1",
"source_ip" => "10.10.10.13"
}

i want to get the output looks like below ,
message :[...]
timestamp:[....]
host:[....]
version:[...]
type:[.....]
BGP4-MIB::bgpPeerLastError :\x00\x00

but the line "BGP4-MIB::bgpPeerLastError :\x00\x00 " is dynamical ,so can you please tell me how to get the BGP4-MIB::bgpPeerLastError :\x00\x00 ?

please help. Thanks.


(Magnus Bäck) #7

Oh, I see. I think you need to use a ruby filter for this. This might do it but probably isn't very efficient:

ruby {
  code => "
    prefixes = ['BGP4-MIB::bgpPeerLastError', 'BGP4-MIB::bgpPeerState']
    event.to_hash.each_pair { |k, v|
      prefixes.each { |p|
        if k.start_with? p
          event[p] = v
          event.remove(k)
        end
      }
    }
  "
}

(saye) #8

i am sorry i think i didn't say it clearly.
the line "BGP4-MIB::bgpPeerLastError :\x00\x00 " is not always have the same prefix ,
for example ,
when i get the log as follows,
{ "message" => "#, @varbind_list=[#, #>, #, #>], @specific_trap=1, @source_ip="10.10.10.13", @agent_addr=#, @generic_trap=6>",
"host" => "10.10.10.13",
"@version" => "1",
"@timestamp" => "2016-04-08T06:22:17.711Z",
"type" => "snmptrap",
"RFC1155-SMI::enterprises_9_9_187_1_2_5_1_17_32_1_14_16_255_255_17_0_0_0_0_0_0_0_0_2" => "\x00\x00",
"RFC1155-SMI::enterprises_9_9_187_1_2_5_1_3_32_1_14_16_255_255_17_0_0_0_0_0_0_0_0_2" => "3",
"RFC1155-SMI::enterprises_9_9_187_1_2_5_1_28_32_1_14_16_255_255_17_0_0_0_0_0_0_0_0_2" => "",
"RFC1155-SMI::enterprises_9_9_187_1_2_5_1_29_32_1_14_16_255_255_17_0_0_0_0_0_0_0_0_2" => "1",
"source_ip" => "10.10.10.13"
}

then i want to get the output looks like below ,
message :[...]
timestamp:[....]
host:[....]
version:[...]
type:[.....]
RFC1155-SMI::enterprises :\x00\x00

and for example 2 ,
when i get the log as follows,
{ "message" => "#, @varbind_list=[#, #>, #, #>], @specific_trap=1, @source_ip="10.10.10.13", @agent_addr=#, @generic_trap=6>",
"host" => "10.10.10.13",
"@version" => "1",
"@timestamp" => "2016-04-08T06:22:17.711Z",
"type" => "snmptrap",
"MSDP-MIB::msdpPeerState_10_10_11_90" => "1",
"source_ip" => "10.10.10.13"
}

then i want to get the output looks like below ,
message :[...]
timestamp:[....]
host:[....]
version:[...]
type:[.....]
MSDP-MIB::msdpPeerState_10_10_11_90 :1

i think maybe it's can use prefix to filter ,
because I neither know the field names, nor the number of fields.
Do you know other method?
I truly appreciate ... your help in resolving the problem.


(Magnus Bäck) #9

As long as the set of interesting prefixes is known the example I gave should work for your second example. For the first example, how is Logstash supposed to choose between the four fields with the common prefix?


(saye) #10

for the first example,
should get as follows,
RFC1155-SMI::enterprises" => "\x00\x00"

but when i use the ruby code ,i got the error
Ruby exception occurred: can't add a new key into hash during iteration {:level=>:error}


(Magnus Bäck) #11

for the first example,
should get as follows,
RFC1155-SMI::enterprises" => "\x00\x00"

I understand that, but how should Logstash reach that conclusion? Since there are multiple fields with that prefix how is Logstash supposed to know which one to pick?

Ruby exception occurred: can't add a new key into hash during iteration {:level=>:error}

Change event.to_hash.each_pair to event.to_hash.clone.each_pair.


(saye) #12

Count from the message beginning of the number of sixth .
but i don't know how to make logstash to do ?

for example,
{ "message" => "..",
"host" => "10.10.10.13",
"@version" => "1",
"@timestamp" => "2016-04-08T06:22:17.711Z",
"type" => "snmptrap",
"MSDP-MIB::msdpPeerState_10_10_11_90" => "1",
"field_7"=>"field_7 content",
"source_ip" => "10.10.10.13"
}

i want to pick the sixth field is "MSDP-MIB::msdpPeerState_10_10_11_90" => "1",
i want the output looks like as below,

"message" => "..",
"host" => "10.10.10.13",
"@version" => "1",
"@timestamp" => "2016-04-08T06:22:17.711Z",
"type" => "snmptrap",
"MSDP-MIB::msdpPeerState_10_10_11_90" => "1",
"source_ip" => "10.10.10.13"


Logstash filter - get field?
(saye) #13

Hello,
Anybody has any good suggestion?


(system) #14