Multiline | Not even working


(WhipSnake) #1

I have having difficulty getting the below multiline code to work. I feel I am close just no sure what I am doing wrong. I want it to take the lines that don't have time stamps and append it to the time stamp that is above it. I had read in another post that spaces and such in the .yml are important and I made sure mine were ok. Filebeat does run and it sends the stuff to logstash but for some reason when I look in Kibana the logs are being split.

Config File:

filebeat.prospectors:

- input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - E:\AOLogs\grid.log

multiline:
  pattern: '\d{1,2}\s+(Jan(uary)?|Feb(ruary)?|Mar(ch)?|Apr(il)?|May|Jun(e)?|Jul(y)?|Aug(ust)?|Sep(tember)?|Oct(ober)?|Nov(ember)?|Dec(ember)?)\s\d{4}\s\d{2}:\d{2}:\d{2},\d{3}'
  negate: true
  match: after

#================================ General =====================================

name: DevAO

#================================ Outputs =====================================

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.

output.logstash:
  # The Logstash hosts
 hosts: ["Server.Domain.Hidden:5044"]

#================================ Logging =====================================

logging.level: debug

logging.selectors: ["*"]

Sample Log File:

06 Oct 2017 02:45:17,261 [Thread=AMP - Activity Processor - Parallel Adapter Invocation Worker - 6272000] DEBUG AdapterManager  [PeerName=CDP] [JobID=b518a7772085ac34:-7822094:15e28c16e22:-7ff61-1507275916300] [AdapterName=IPNOPSHA_DB]  unsetMDC(AdapterRequest)
06 Oct 2017 02:45:17,261 [Thread=AMP - Activity Processor - Parallel Adapter Invocation Worker - 6272000] DEBUG AdapterManager  [PeerName=CDP]    Thread [42150793] - Received response from local adapter "AdapterConfiguration1488257317210--650700716" : <adapter-response><execution-duration>2</execution-duration><status>success</status><messageBase64></messageBase64><peer-name>CDP</peer-name><data><sqlAdapterResponse><metadata><queryExecuted>UPDATE AO.Shadow_ipnops_blackout
SET b_move_to_queue = 1,
new_assignment_group = 'ipnops-automation',
b_update_ticket = 1,
touch_count = 1,
b_override = 1,
ticket_update = '' + Convert(nvarchar,GETDATE(),120) + ' (Shadow-Master_ipnops_blackout): Moving to ipnops-automation ' + CHAR(13) + CHAR(10) + '---' + CHAR(13) + CHAR(10) + ticket_update
WHERE  ( (  ( alert1 = 'NT_CPU')  OR  (  (alert1 like 'NT_LOGICAL_DISK%') )  OR  ( alert2 like 'PA:%')  OR  ( (alert1 in (select t.a1 from (SELECT alert1 as a1, alert2 as a2, alert3 as a3 FROM ao.Patrol_CLI_Master_Alert1) t where (a2 is null or a2 = alert2) and (a3 is null or a3 = alert3))))  OR  ( (alert1 = 'ping' or alert3 = 'Availability') and CI_Type not in ('router', 'Router', 'circuit', 'Circuit') and alert1 != 'Interface')  OR  ( alert1 = 'NT_SERVICES' or alert1 = 'NT_Service')   ))   AND (Affected_CI NOT IN (SELECT Affected_CI FROM AO.BlackOut_Task WHERE GETDATE() between Start_Time_Moline and End_Time_Moline)) AND NOT (( 1 = 0 )) </queryExecuted><execution-milliseconds>2</execution-milliseconds><status>success</status></metadata></sqlAdapterResponse></data></adapter-response>
06 Oct 2017 02:45:17,261 [Thread=AMP - Activity Processor - Parallel Adapter Invocation Worker - 6272000] DEBUG AdapterManager  [PeerName=CDP]    Request succeeded, adding entry for peer named  [CDP] in adapter-job cache for adapter name [IPNOPSHA_DB] and job ID [[JobID representation=b518a7772085ac34:-7822094:15e28c16e22:-7ff61-1507275916300, timestamp=Fri Oct 06 02:45:16 CDT 2017]].
06 Oct 2017 02:45:17,261 [Thread=AMP - Activity Processor - Parallel Adapter Invocation Worker - 6272000] DEBUG AdapterManager  [PeerName=CDP]    returning response <adapter-response><execution-duration>2</execution-duration><status>success</status><messageBase64></messageBase64><peer-name>CDP</peer-name><data><sqlAdapterResponse><metadata><queryExecuted>UPDATE AO.Shadow_ipnops_blackout
SET b_move_to_queue = 1,
new_assignment_group = 'ipnops-automation',
b_update_ticket = 1,
touch_count = 1,
b_override = 1,
ticket_update = '' + Convert(nvarchar,GETDATE(),120) + ' (Shadow-Master_ipnops_blackout): Moving to ipnops-automation ' + CHAR(13) + CHAR(10) + '---' + CHAR(13) + CHAR(10) + ticket_update
WHERE  ( (  ( alert1 = 'NT_CPU')  OR  (  (alert1 like 'NT_LOGICAL_DISK%') )  OR  ( alert2 like 'PA:%')  OR  ( (alert1 in (select t.a1 from (SELECT alert1 as a1, alert2 as a2, alert3 as a3 FROM ao.Patrol_CLI_Master_Alert1) t where (a2 is null or a2 = alert2) and (a3 is null or a3 = alert3))))  OR  ( (alert1 = 'ping' or alert3 = 'Availability') and CI_Type not in ('router', 'Router', 'circuit', 'Circuit') and alert1 != 'Interface')  OR  ( alert1 = 'NT_SERVICES' or alert1 = 'NT_Service')   ))   AND (Affected_CI NOT IN (SELECT Affected_CI FROM AO.BlackOut_Task WHERE GETDATE() between Start_Time_Moline and End_Time_Moline)) AND NOT (( 1 = 0 )) </queryExecuted><execution-milliseconds>2</execution-milliseconds><status>success</status></metadata></sqlAdapterResponse></data></adapter-response>
06 Oct 2017 02:45:17,411 [Thread=AMP - Activity Processor - Parallel Adapter Invocation Worker - 6272001] DEBUG AdapterManager  [PeerName=CDP]    mapped IPNOPSHA_DB to AdapterConfiguration1488257317210--650700716
06 Oct 2017 02:45:17,411 [Thread=AMP - Activity Processor - Parallel Adapter Invocation Worker - 6272001] DEBUG AdapterManager  [PeerName=CDP]    Received request to perform action on [IPNOPSHA_DB].

(Steffen Siering) #2

Please properly format config files and logs using the </> button.

The multiline setting should be part of the prospector setting, not global.


(WhipSnake) #3

That worked they should plaster this doc more places for idiot first timers

https://www.elastic.co/guide/en/beats/libbeat/current/config-file-format.html


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.