I have having difficulty getting the below multiline code to work. I feel I am close just no sure what I am doing wrong. I want it to take the lines that don't have time stamps and append it to the time stamp that is above it. I had read in another post that spaces and such in the .yml are important and I made sure mine were ok. Filebeat does run and it sends the stuff to logstash but for some reason when I look in Kibana the logs are being split.
Config File:
filebeat.prospectors:
- input_type: log
# Paths that should be crawled and fetched. Glob based paths.
paths:
- E:\AOLogs\grid.log
multiline:
pattern: '\d{1,2}\s+(Jan(uary)?|Feb(ruary)?|Mar(ch)?|Apr(il)?|May|Jun(e)?|Jul(y)?|Aug(ust)?|Sep(tember)?|Oct(ober)?|Nov(ember)?|Dec(ember)?)\s\d{4}\s\d{2}:\d{2}:\d{2},\d{3}'
negate: true
match: after
#================================ General =====================================
name: DevAO
#================================ Outputs =====================================
# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.
output.logstash:
# The Logstash hosts
hosts: ["Server.Domain.Hidden:5044"]
#================================ Logging =====================================
logging.level: debug
logging.selectors: ["*"]
Sample Log File:
06 Oct 2017 02:45:17,261 [Thread=AMP - Activity Processor - Parallel Adapter Invocation Worker - 6272000] DEBUG AdapterManager [PeerName=CDP] [JobID=b518a7772085ac34:-7822094:15e28c16e22:-7ff61-1507275916300] [AdapterName=IPNOPSHA_DB] unsetMDC(AdapterRequest)
06 Oct 2017 02:45:17,261 [Thread=AMP - Activity Processor - Parallel Adapter Invocation Worker - 6272000] DEBUG AdapterManager [PeerName=CDP] Thread [42150793] - Received response from local adapter "AdapterConfiguration1488257317210--650700716" : <adapter-response><execution-duration>2</execution-duration><status>success</status><messageBase64></messageBase64><peer-name>CDP</peer-name><data><sqlAdapterResponse><metadata><queryExecuted>UPDATE AO.Shadow_ipnops_blackout
SET b_move_to_queue = 1,
new_assignment_group = 'ipnops-automation',
b_update_ticket = 1,
touch_count = 1,
b_override = 1,
ticket_update = '' + Convert(nvarchar,GETDATE(),120) + ' (Shadow-Master_ipnops_blackout): Moving to ipnops-automation ' + CHAR(13) + CHAR(10) + '---' + CHAR(13) + CHAR(10) + ticket_update
WHERE ( ( ( alert1 = 'NT_CPU') OR ( (alert1 like 'NT_LOGICAL_DISK%') ) OR ( alert2 like 'PA:%') OR ( (alert1 in (select t.a1 from (SELECT alert1 as a1, alert2 as a2, alert3 as a3 FROM ao.Patrol_CLI_Master_Alert1) t where (a2 is null or a2 = alert2) and (a3 is null or a3 = alert3)))) OR ( (alert1 = 'ping' or alert3 = 'Availability') and CI_Type not in ('router', 'Router', 'circuit', 'Circuit') and alert1 != 'Interface') OR ( alert1 = 'NT_SERVICES' or alert1 = 'NT_Service') )) AND (Affected_CI NOT IN (SELECT Affected_CI FROM AO.BlackOut_Task WHERE GETDATE() between Start_Time_Moline and End_Time_Moline)) AND NOT (( 1 = 0 )) </queryExecuted><execution-milliseconds>2</execution-milliseconds><status>success</status></metadata></sqlAdapterResponse></data></adapter-response>
06 Oct 2017 02:45:17,261 [Thread=AMP - Activity Processor - Parallel Adapter Invocation Worker - 6272000] DEBUG AdapterManager [PeerName=CDP] Request succeeded, adding entry for peer named [CDP] in adapter-job cache for adapter name [IPNOPSHA_DB] and job ID [[JobID representation=b518a7772085ac34:-7822094:15e28c16e22:-7ff61-1507275916300, timestamp=Fri Oct 06 02:45:16 CDT 2017]].
06 Oct 2017 02:45:17,261 [Thread=AMP - Activity Processor - Parallel Adapter Invocation Worker - 6272000] DEBUG AdapterManager [PeerName=CDP] returning response <adapter-response><execution-duration>2</execution-duration><status>success</status><messageBase64></messageBase64><peer-name>CDP</peer-name><data><sqlAdapterResponse><metadata><queryExecuted>UPDATE AO.Shadow_ipnops_blackout
SET b_move_to_queue = 1,
new_assignment_group = 'ipnops-automation',
b_update_ticket = 1,
touch_count = 1,
b_override = 1,
ticket_update = '' + Convert(nvarchar,GETDATE(),120) + ' (Shadow-Master_ipnops_blackout): Moving to ipnops-automation ' + CHAR(13) + CHAR(10) + '---' + CHAR(13) + CHAR(10) + ticket_update
WHERE ( ( ( alert1 = 'NT_CPU') OR ( (alert1 like 'NT_LOGICAL_DISK%') ) OR ( alert2 like 'PA:%') OR ( (alert1 in (select t.a1 from (SELECT alert1 as a1, alert2 as a2, alert3 as a3 FROM ao.Patrol_CLI_Master_Alert1) t where (a2 is null or a2 = alert2) and (a3 is null or a3 = alert3)))) OR ( (alert1 = 'ping' or alert3 = 'Availability') and CI_Type not in ('router', 'Router', 'circuit', 'Circuit') and alert1 != 'Interface') OR ( alert1 = 'NT_SERVICES' or alert1 = 'NT_Service') )) AND (Affected_CI NOT IN (SELECT Affected_CI FROM AO.BlackOut_Task WHERE GETDATE() between Start_Time_Moline and End_Time_Moline)) AND NOT (( 1 = 0 )) </queryExecuted><execution-milliseconds>2</execution-milliseconds><status>success</status></metadata></sqlAdapterResponse></data></adapter-response>
06 Oct 2017 02:45:17,411 [Thread=AMP - Activity Processor - Parallel Adapter Invocation Worker - 6272001] DEBUG AdapterManager [PeerName=CDP] mapped IPNOPSHA_DB to AdapterConfiguration1488257317210--650700716
06 Oct 2017 02:45:17,411 [Thread=AMP - Activity Processor - Parallel Adapter Invocation Worker - 6272001] DEBUG AdapterManager [PeerName=CDP] Received request to perform action on [IPNOPSHA_DB].