Grok for IP then geoip and ip2proxy plugin?

[mortimerj (kevin)]

Hello!

I'm trying to grok just the IP address from the field ClientIP, which can present as [1.2.3.4:4444] or 1.2.3.4:4444, then create a new field (clientipfixed), and apply the geoip and ip2proxy plugins to enrich it. I don't see clientipfixed getting created or enriched in my index after creation? I've included the config below thanks!

input {
file {
path => "/home/ubuntu/Desktop/test/*.csv"
sincedb_path => "/dev/null"
start_position => "beginning"
}
}
filter {
csv {
separator => ","
columns => ["CreationDate","UserIds","Operations","CreationTime","Id","Operation","OrganizationId","RecordType","ResultStatus","UserKey","UserType","Version","Workload","ClientIP","UserId","ClientIPAddress","ClientInfoString","ClientProcessName","ClientVersion","ExternalAccess","InternalLogonType","LogonType","LogonUserSid","MailboxGuid","MailboxOwnerSid","MailboxOwnerUPN","OrganizationName","OriginatingServer","SessionId","AffectedItems","CrossMailboxOperation","Folder","ObjectId","AzureActiveDirectoryEventType","ExtendedProperties","ModifiedProperties","Actor","ActorContextId","ActorIpAddress","InterSystemsId","IntraSystemId","SupportTicketId","Target","TargetContextId","ApplicationId","LogonError","AppId","Item","ClientAppId","DestFolder","CorrelationId","EventSource","ItemType","ListId","ListItemUniqueId","Site","UserAgent","WebId","SourceFileExtension","SiteUrl","SourceFileName","SourceRelativeUrl","CustomUniqueId","OperationProperties","DoNotDistributeEvent"]
}
grok{
match => { "ClientIP" => "%{IP:clientipfixed}" }
}
geoip{
source => "clientipfixed"
}
ip2proxy{
source => "clientipfixed"
database => "/home/ubuntu/Desktop/test/ip2proxydb"
}
}
output {
elasticsearch {
hosts => "http://localhost:9200"
index => "mappingtest"
}
stdout {}
}

[Badger]

What does [ClientIP] look like on stdout?

[mortimerj (kevin)]

I'm seeing many errors in /var/log/logstash/logstash-plain.log that include the following, but I've sed'd all " out of the csv, and opened it to control-f check and make sure there aren't any "s.

:exception=>#<CSV::MalformedCSVError: Unclosed quoted field on line 1.>}

The resulting index and pattern have the following:

1. No field created for clientipfixed in index pattern
2. IPs like 1.1.1.1:1234 are indexed
3. IPs like [1.1.1.1]:1234 are not indexed
4. IPs that don't have port are indexed and enriched

[mortimerj (kevin)]

I changed

match => { "ClientIP" => "%{IP:clientipfixed}" }

to

match => { "message" => "%{IP:clientipfixed}" }

and it worked...

Sorry for dumb question!