Grok for IP then geoip and ip2proxy plugin?


I'm trying to grok just the IP address from the field ClientIP, which can present as [] or, then create a new field (clientipfixed), and apply the geoip and ip2proxy plugins to enrich it. I don't see clientipfixed getting created or enriched in my index after creation? I've included the config below thanks!

input {
file {
path => "/home/ubuntu/Desktop/test/*.csv"
sincedb_path => "/dev/null"
start_position => "beginning"
filter {
csv {
separator => ","
columns => ["CreationDate","UserIds","Operations","CreationTime","Id","Operation","OrganizationId","RecordType","ResultStatus","UserKey","UserType","Version","Workload","ClientIP","UserId","ClientIPAddress","ClientInfoString","ClientProcessName","ClientVersion","ExternalAccess","InternalLogonType","LogonType","LogonUserSid","MailboxGuid","MailboxOwnerSid","MailboxOwnerUPN","OrganizationName","OriginatingServer","SessionId","AffectedItems","CrossMailboxOperation","Folder","ObjectId","AzureActiveDirectoryEventType","ExtendedProperties","ModifiedProperties","Actor","ActorContextId","ActorIpAddress","InterSystemsId","IntraSystemId","SupportTicketId","Target","TargetContextId","ApplicationId","LogonError","AppId","Item","ClientAppId","DestFolder","CorrelationId","EventSource","ItemType","ListId","ListItemUniqueId","Site","UserAgent","WebId","SourceFileExtension","SiteUrl","SourceFileName","SourceRelativeUrl","CustomUniqueId","OperationProperties","DoNotDistributeEvent"]
match => { "ClientIP" => "%{IP:clientipfixed}" }
source => "clientipfixed"
source => "clientipfixed"
database => "/home/ubuntu/Desktop/test/ip2proxydb"
output {
elasticsearch {
hosts => "http://localhost:9200"
index => "mappingtest"
stdout {}

What does [ClientIP] look like on stdout?

I'm seeing many errors in /var/log/logstash/logstash-plain.log that include the following, but I've sed'd all " out of the csv, and opened it to control-f check and make sure there aren't any "s.

:exception=>#<CSV::MalformedCSVError: Unclosed quoted field on line 1.>}

The resulting index and pattern have the following:

  1. No field created for clientipfixed in index pattern
  2. IPs like are indexed
  3. IPs like []:1234 are not indexed
  4. IPs that don't have port are indexed and enriched

I changed

match => { "ClientIP" => "%{IP:clientipfixed}" }


match => { "message" => "%{IP:clientipfixed}" }

and it worked... :man_facepalming:

Sorry for dumb question!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.