Translate keys/fields

Elastic Stack Logstash
1

So, after that dissect doubt, the new one is: how can I translate keys or fieldnames?

The files I'm reading are IAS/NPS, so they come with "key,value" and key being a number. But to better understand, navigate, use at will, it's better to have their names/descriptions instead of numbers. I.e: 4 = "NAS-IP-Address". (edit: I'm using https://iso.csusb.edu/tools/nps-log-interpreter as base, since it parses all the samples I needed, so far. But I now it to be lacking a couple fields, like codes 4147 and 4148)

So, after that first doubt about dissect I finally got to a place where I have all the fields I want, with their correct values parsed. But field numbers, not field names. And I can't find a way to do this.

If it helps in any way at all, here's where I've got so far:

input {
	stdin { }
}

filter {

	mutate {
		gsub => ["message", "\r", ""]
	}

	dissect {
		mapping => {
			"message" => "%{NASIPAddress},%{UserName},%{log_timestamp},%{+log_timestamp},%{ServiceType},%{ServerName},%{values}"
		}
	}

	mutate {
		gsub => ["values", "([^,]+),([^,]+),?", "\1=\2,"]
		split => ["values", ","]
	}

	kv {
		source => "values"
	}

	# translate {
	# 	field => "[message]"
	# 	destination => "[new_message]"
	# 	dictionary => {
	# 		"7" => "FramedProtocol"
	# 		"8" => "FramedIPAddress"
	# 		"9" => "FramedIPNetmask"
	# 		"10" => "FramedRouting"
	# 		"12" => "FramedMTU"
	# 		"13" => "FramedCompression"
	# 		"14" => "LoginIPHost"
	# 		"15" => "LoginService"
	# 		"16" => "LoginTCPPort"
	# 		"18" => "ReplyMessage"
	# 		"19" => "CallbackNumber"
	# 		"20" => "CallbackID"
	# 		"22" => "FramedRoute"
	# 		"23" => "FramedIPXNetwork"
	# 		"25" => "Class"
	# 		"26" => "VendorSpecific"
	# 		"27" => "SessionTimeout"
	# 		"28" => "IdleTimeout"
	# 		"29" => "TerminationAction"
	# 		"30" => "CalledStationID"
	# 		"31" => "CallingStationID"
	# 		"32" => "NASIdentifier"
	# 		"34" => "LoginLATService"
	# 		"35" => "LoginLATNode"
	# 		"36" => "LoginLATGroup"
	# 		"37" => "FramedAppleTalkLink"
	# 		"38" => "FramedAppleTalkNetwork"
	# 		"39" => "FramedAppleTalkZone"
	# 		"40" => "AcctStatusType"
	# 		"41" => "AcctDelayTime"
	# 		"42" => "AcctInputOctets"
	# 		"43" => "AcctOutputOctets"
	# 		"44" => "AcctSessionID"
	# 		"45" => "AcctAuthentic"
	# 		"46" => "AcctSessionTime"
	# 		"47" => "AcctInputPackets"
	# 		"48" => "AcctOutputPackets"
	# 		"49" => "AcctTerminateCause"
	# 		"50" => "AcctMultiSSNID"
	# 		"51" => "AcctLinkCount"
	# 		"55" => "EventTimestamp"
	# 		"61" => "NASPortType"
	# 		"62" => "PortLimit"
	# 		"63" => "LoginLATPort"
	# 		"64" => "TunnelType"
	# 		"65" => "TunnelMediumType"
	# 		"66" => "TunnelClientEndpt"
	# 		"67" => "TunnelServerEndpt"
	# 		"68" => "AcctTunnelConnection"
	# 		"75" => "PasswordRetry"
	# 		"76" => "Prompt"
	# 		"77" => "ConnectInfo"
	# 		"78" => "ConfigurationToken"
	# 		"81" => "TunnelPvtGroupID"
	# 		"82" => "TunnelAssignmentID"
	# 		"83" => "TunnelPreference"
	# 		"85" => "AcctInterimInterval"
	# 		"4108" => "ClientIPAddress"
	# 		"4116" => "NASManufacturer"
	# 		"4121" => "MSCHAPError"
	# 		"4127" => "AuthenticationType"
	# 		"4128" => "ClientFriendlyName"
	# 		"4129" => "SAMAccountName"
	# 		"4130" => "FullyQualifiedUserName"
	# 		"4132" => "EAPFriendlyName"
	# 		"4136" => "PacketType"
	# 		"4142" => "ReasonCode"
	# 		"4149" => "NPPolicyName"
	# 		"4154" => "ProxyPolicyName"
	# 		"4155" => "ProviderType"
	# 		"4156" => "ProviderName"
	# 		"4157" => "RemoteServerAddress"
	# 	}
	# }

}

output {

	stdout { codec => rubydebug }
	stdout { codec => json }
}

Removing values, I can share the rubydebug this far:

{
               "51" =>
             "4159" =>
                "6" =>
             "4120" =>
    "log_timestamp" =>
             "4142" =>
             "4136" =>
                "4" =>
               "66" =>
               "40" =>
               "55" =>
               "65" =>
             "host" =>
         "@version" =>
      "ServiceType" =>
               "12" =>
               "50" =>
             "4148" =>
                "7" =>
       "@timestamp" => 2019-09-26T02:44:33.716Z,
             "4160" =>
               "25" =>
               "27" =>
          "message" =>
                "8" =>
       "ServerName" =>
             "4154" =>
       "4294967206" =>
               "31" =>
               "28" =>
               "61" =>
                "5" =>
             "4147" =>
               "45" =>
     "NASIPAddress" =>
             "4108" =>
               "44" =>
               "64" =>
           "values" => [
        [ 0] "4",
        [ 1] "6",
        [ 2] "7",
        [ 3] "5",
        [ 4] "61",
        [ 5] "64",
        [ 6] "65",
        [ 7] "31",
        [ 8] "66",
        [ 9] "25",
        [10] "44",
        [11] "8",
        [12] "12",
        [13] "27",
        [14] "28",
        [15] "50",
        [16] "51",
        [17] "55",
        [18] "45",
        [19] "40",
        [20] "4108",
        [21] "4147",
        [22] "4148",
        [23] "4160",
        [24] "4159",
        [25] "4120",
        [26] "4294967206",
        [27] "4154",
        [28] "4136",
        [29] "4142"
    ],
         "UserName" =>
}
2

After some thinking, I went another way with this. Instead of trying to translate the keys after making them keys, I thought about replacing them before.

So I mutate/gsub to remove that \r, then I dissect the first 6 static fields and leave all the rest as %{values}. And here we start the fun: more gsubs! The first changing X,Y, into X=Y, and a load of gsubs changing \b<code>\b= into <keyname>=!! I got to even add an IAS dictionary afterwards <3

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.