Muliline logstash

Elastic Stack Logstash
1

general question here as I'm not sure I understand the multiline filter quite yet. sorry if this has been answered many times before. I tried searching a bit but didn't find a clear example. what I'm looking to do is combine 4 line entries into what will look like one line so I can grok "message" appropriately... it's from a Cisco ACS server and what it looks like is happening is when it generates a log it really generates 5 lines for every log as follows:

Feb 11 18:02:46 xxxxx CSCOacs_Passed_Authentications 0027684799 5 0 2016-02-11 18:02:46.659 +00:00 0152726068 5201 NOTICE Passed-Authentication: Authentication succeeded, ACSVersion=acs-5.4.0.46(unresolved)-B.221, ConfigVersionId=172, Device IP Address=x.x.x.x.(xxx-xxxxx), DestinationIPAddress=x.x.x.x(xxxxxx), DestinationPort=49, UserName=xxx, Protocol=Tacacs, RequestLatency=34, Type=Authentication, Action=Login, Privilege-Level=1, Authen-Type=ASCII, Service=Login, User=xxxx, Port=64941, Remote-Address=x.x.x.x(xxxx),
Feb 11 18:02:46 lxacs1 CSCOacs_Passed_Authentications 0027684799 5 1 UserName=xxxx, AcsSessionID=xxxxx/228692893/8870500, AuthenticationIdentityStore=AD1, AuthenticationMethod=PAP_ASCII, SelectedAccessService=Default Device Admin, SelectedShellProfile=PermitEnable, IdentityGroup=IdentityGroup:All Groups:ACSViewOnly, Step=13013 , Step=15008 , Step=15004 , Step=15012 , Step=15041 , Step=15004 , Step=15013 , Step=13045 , Step=13015 , Step=13014 , Step=15037 , Step=15041 , Step=15004 , Step=15013 ,
Feb 11 18:02:46 xxxxx CSCOacs_Passed_Authentications 0027684799 5 2 Step=24430 , Step=24416 , Step=24420 , Step=24402 , Step=22037 , Step=15044 , Step=15004 , Step=15035 , Step=15042 , Step=15036 , Step=15004 , Step=13015 , SelectedAuthenticationIdentityStores=AD1, SelectedAuthenticationIdentityStores=AD1, NetworkDeviceName=xxx-xxxxx, NetworkDeviceGroups=Location:All Locations:xxxxxx, NetworkDeviceGroups=Device Type:All Device Types:Cisco:Cisco ASAs, ServiceSelectionMatchedRule=Rule-2,
Feb 11 18:02:46 xxxx CSCOacs_Passed_Authentications 0027684799 5 3 IdentityPolicyMatchedRule=ActiveDirectory, ADDomain=xxxxx.xxx, GroupMappingPolicyMatchedRule=ViewOnly, AuthorizationPolicyMatchedRule=ViewOnly, Action=Login, Privilege-Level=1, Authen-Type=ASCII, Service=Login, Port=64941, Remote-Address=x.x.x.x(xxxx), ExternalGroups=xxx.xxx/Groups/Security Groups/ISX Viewers, ExternalGroups=xxxx.xxxx/Groups/Security Groups/xxxxxxxx, ExternalGroups=xxx.xxx/Users/xxxxxxxxx,
Feb 11 18:02:46 xxxx CSCOacs_Passed_Authentications 0027684799 5 4 ExternalGroups=xxxxx.xxxx/Users/Domain Users, ExternalGroups=xxxx.xxxx/Builtin/Users, ExternalGroups=xxxx.xxxx/Builtin/Event Log Readers, IdentityAccessRestricted=false, displayName=xxxxx, sAMAccountName=xxxxxx, Response={Type=Authentication; Authen-Reply-Status=Pass; }

sorry I know it's a mess... but here's what I was going to key on:

CSCOacs_Passed_Authentications 0027684799 5 0
CSCOacs_Passed_Authentications 0027684799 5 1
CSCOacs_Passed_Authentications 0027684799 5 2
CSCOacs_Passed_Authentications 0027684799 5 3
CSCOacs_Passed_Authentications 0027684799 5 4

in this the number 0027684799 will increment by 1 for each new log entry and if we reboot the box it will start back over at 0000000001... when it hits all 9999.. it rolls over too.

in the first line there's an additional line:

CSCOacs_Passed_Authentications 0027684799 5 0 2016-02-11 18:02:46.659 +00:00 0152726068 5201

5201 is the actual "log" type that I would key off of for grok parsing. as that number is unique and will tell me what type of log it really is..

my hope is that with the "multiline" filter I can basically add the 5 lines together and make "message" look like one huge single line entry. I can then parse Message based on the unique ID "5201" in this case the way I need to.

any suggestions would be greatly appreciated.

thanks,

Lee

2

ok so I'm starting to get something that looks good but not sure it's the best way to go about it... for my specific example above i've got a sameple test file with the 5 log entries in them and am testing using the below logstash config...:

 input {
   file {
     path => ["/home/elk/logstash-test-dir/acsdebug.log"]
     start_position => "beginning"
     type => "acslog"
   }
 }

 filter {
      multiline {
           'pattern' => 'CSCOacs_Passed_Authentications\s\d{10}\s([1-5])\s([1-5])'
            'what' => 'previous'
      }
 }
 output {stdout{ codec => rubydebug }}

in my actual setup my input is a shared log file that has many devices all sending data to the same file.

What happens if I get a syslog line of an unrelated message? would it be better if I break this particular log data from ACS into a separate file so it's a separate input to logstash? currently these logs are going into a single flat syslog file along with other router/firewall logs. i'm afraid I'll get one of these ACS logs in in the middle of them will be some router or firewall log... thoughts?

3

ok. I think this is solved. when I tried to roll the multiline via a filter section to logstash I would get a "warning" message that stated:

Defaulting filter worker threads to 1 because there are some filters that might not work with multiple worker threads {:count was=>6, :filters=>["multiline"], :level=>:warn}

so what I did was use multiline as input from a separate file versus as a filter looking through the global syslog. I believe this will resolve any potential issues where I could get some other unrelated logs inserted between a multiline event. I also updated my regex on the multiline to make it fit all ACS log cases as there are many different types of logs that are in this multiline format. new config looks like:

input {
  file {
    path => ["/var/log/network.log"]
    start_position => "beginning"
    type => "syslog"
    tags => [ "netsyslog" ]
  }
  file {
    path => "/var/log/acslog.log"
    start_position => "end"
    type => "syslog"
    tags => [ "acssyslog" ]
    codec => multiline {
       'pattern' => 'CSCO.*\s\d{10}\s\d\s([1-9])'
       'what' => 'previous'
     }
  }
}

and that all appears to be working good.

4

I'm experiencing the same "Defaulting filter worker threads to 1 ..." error on a multi-CPU based virtual system. Just to confirm, you no longer get the "Defaulting filter worker threads to 1 ..." message in your Logstash logfiles? I'm confused b/c it was explained in a different blog inquiry that the multiline functionality is not thread safe and that is why Logstash defaults the filter to 1 CPU on a multi-CPU'ed system. Please advise. Regards, Gary.