Multiline codec with Timestamp in between log message

Saravana37 · January 11, 2020, 5:06am

Hi ,

My Sample looks as follows.

    ObjMgrBusCompLog	Create	4	000010c25c361790:0	2020-01-11 05:58:14	Begin: construct BusComp "Employee" at 18d25dc8

    ObjMgrBusCompLog	Create	4	000010c25c361790:0	2020-01-11 05:58:14	End: construct BusComp "Employee" at 18d25dc8

    ObjMgrSqlLog	Detail	4	000010c25c361790:0	2020-01-11 05:58:14	SELECT statement with ID: 193DF750
    SELECT
          T6.CONFLICT_ID,
          T6.LAST_UPD,
          T6.CREATED,
          T6.LAST_UPD_BY,
          T6.CREATED_BY,
          T6.MODIFICATION_NUM,
          T6.ROW_ID,
          T6.ROW_ID,
          T14.AGENT_FLG,
          T12.CTI_ACD_USERID,
          T12.X_MARKET,
          T12.X_PREF_BRKDN_CNTRY,
          T12.X_PREF_CASE_TYPE,
          T12.X_PREF_COUNTRY,
          T12.X_PREF_CUST_LANGUAGE,
          T2.NAME,
          T12.X_PREF_PDQ_GROUP,
          T12.X_PREF_SERVICE_TYPE,
          T13.CURR_PRD_OBJ_NAME,
          T14.EMP_FLG,
          T8.EMPLOYEE_TYPE_CD,
          T10.SCHED_ENGINE_CD,
          T14.FAX_PH_NUM,
          T14.FST_NAME,
          T14.LAST_NAME,
          T12.LOGIN,
          T14.MEMBER_FLG,
          T14.MID_NAME,
          T14.NEW_USER_RESP_NAME,
          T6.NAME,
          T14.BU_ID,
          T3.BU_ID,
          T14.PR_SYNC_USER_ID,
          T14.PR_PER_ADDR_ID,
          T14.PR_HELD_POSTN_ID,
          T14.PR_RESP_ID,
          T14.PR_POSTN_ID,
          T14.PROVIDER_FLG,
          T9.OWN_INST_ID,
          T9.INTEGRATION_ID,
          T8.SALARY_GRADE_ID,
          T8.SALARY_PLAN_ID,
          T10.TIME_ZONE_ID,
          T8.CUR_WRK_SHFT_ID,
          T14.PR_DEPT_OU_ID,
          T8.SYNC_SUCCESS_FLG,
          T14.TIMEZONE_ID,
          T7.NAME,
          T12.X_IWS_USERNAME,
          T12.X_IWS_FLAG,
          T5.NAME,
          T11.LOGIN,
          T1.NAME,
          T12.ROW_ID,
          T12.PAR_ROW_ID,
          T12.MODIFICATION_NUM,
          T12.CREATED_BY,
          T12.LAST_UPD_BY,
          T12.CREATED,
          T12.LAST_UPD,
          T12.CONFLICT_ID,
          T12.PAR_ROW_ID,
          T14.ROW_ID,
          T14.PAR_ROW_ID,
          T14.MODIFICATION_NUM,
          T14.CREATED_BY,
          T14.LAST_UPD_BY,
          T14.CREATED,
          T14.LAST_UPD,
          T14.CONFLICT_ID,
          T14.PAR_ROW_ID,
          T8.ROW_ID,
          T8.PAR_ROW_ID,
          T8.MODIFICATION_NUM,
          T8.CREATED_BY,
          T8.LAST_UPD_BY,
          T8.CREATED,
          T8.LAST_UPD,
          T8.CONFLICT_ID,
          T8.PAR_ROW_ID,
          T9.ROW_ID,
          T9.PAR_ROW_ID,
          T9.MODIFICATION_NUM,
          T9.CREATED_BY,
          T9.LAST_UPD_BY,
          T9.CREATED,
          T9.LAST_UPD,
          T9.CONFLICT_ID,
          T9.PAR_ROW_ID,
          T4.ROW_ID,
          T1.ROW_ID
       FROM 
           SIEBEL.S_RESP T1,
           SIEBEL.S_LANG T2,
           SIEBEL.S_POSTN T3,
           SIEBEL.S_PARTY T4,
           SIEBEL.S_TIMEZONE_LANG T5,
           SIEBEL.S_PARTY T6,
           SIEBEL.S_TIMEZONE T7,
           SIEBEL.S_EMP_PER T8,
           SIEBEL.S_CONTACT_SS T9,
           SIEBEL.S_SRV_REGN T10,
           SIEBEL.S_USER T11,
           SIEBEL.S_USER T12,
           SIEBEL.S_BU T13,
           SIEBEL.S_CONTACT T14
       WHERE 
          T14.PR_HELD_POSTN_ID = T3.PAR_ROW_ID (+) AND
          T3.BU_ID = T13.ROW_ID (+) AND
          T12.X_PREF_CUST_LANGUAGE = T2.LANG_CD (+) AND
          T8.SRV_REGN_ID = T10.ROW_ID (+) AND
          T14.TIMEZONE_ID = T7.ROW_ID (+) AND
          T7.ROW_ID = T5.PAR_ROW_ID (+) AND T5.LANG_ID (+) = :1 AND
          T6.ROW_ID = T12.PAR_ROW_ID AND
          T6.ROW_ID = T14.PAR_ROW_ID AND
          T6.ROW_ID = T8.PAR_ROW_ID AND
          T6.ROW_ID = T9.PAR_ROW_ID (+) AND
          T14.PR_SYNC_USER_ID = T4.ROW_ID (+) AND
          T14.PR_SYNC_USER_ID = T11.PAR_ROW_ID (+) AND
          T14.PR_RESP_ID = T1.ROW_ID (+) AND
          (T14.EMP_FLG = 'Y') AND
          (T6.ROW_ID = :2)
       ORDER BY
          T14.EMP_FLG, T14.LAST_NAME, T14.FST_NAME

    ObjMgrSqlLog	Detail	4	000010c25c361790:0	2020-01-11 05:58:14	Bind variable 1: ENU

    ObjMgrSqlLog	Detail	4	000010c25c361790:0	2020-01-11 05:58:14	Bind variable 2: 1-1IXAY8D

    ObjMgrSqlLog	Detail	4	000010c25c361790:0	2020-01-11 05:58:14	
    ***** SQL Statement Prepare Time for SQL Cursor with ID 193DF750: 0.000 seconds *****
  `

I want to tell filebeat that where ever it finds a Timestamp in the message (in middle of message) , should consider the whole line as one single event . For example below multiline should consider as single event.

 ObjMgrSqlLog	Detail	4	000010c25c361790:0	2020-01-11 05:58:14	SELECT statement with ID: 193DF750`
SELECT
  T6.CONFLICT_ID,
  T6.LAST_UPD,
  T6.CREATED,
  T6.LAST_UPD_BY,
  T6.CREATED_BY,
  T6.MODIFICATION_NUM,
  T6.ROW_ID,
  T6.ROW_ID,
  T14.AGENT_FLG,
  T12.CTI_ACD_USERID,
  T12.X_MARKET,
  T12.X_PREF_BRKDN_CNTRY,
  T12.X_PREF_CASE_TYPE,
  T12.X_PREF_COUNTRY,
  T12.X_PREF_CUST_LANGUAGE,
  T2.NAME,
  T12.X_PREF_PDQ_GROUP,
  T12.X_PREF_SERVICE_TYPE,
  T13.CURR_PRD_OBJ_NAME,
  T14.EMP_FLG,
  T8.EMPLOYEE_TYPE_CD,
  T10.SCHED_ENGINE_CD,
  T14.FAX_PH_NUM,
  T14.FST_NAME,
  T14.LAST_NAME,
  T12.LOGIN,
  T14.MEMBER_FLG,
  T14.MID_NAME,
  T14.NEW_USER_RESP_NAME,
  T6.NAME,
  T14.BU_ID,
  T3.BU_ID,
  T14.PR_SYNC_USER_ID,
  T14.PR_PER_ADDR_ID,
  T14.PR_HELD_POSTN_ID,
  T14.PR_RESP_ID,
  T14.PR_POSTN_ID,
  T14.PROVIDER_FLG,
  T9.OWN_INST_ID,
  T9.INTEGRATION_ID,
  T8.SALARY_GRADE_ID,
  T8.SALARY_PLAN_ID,
  T10.TIME_ZONE_ID,
  T8.CUR_WRK_SHFT_ID,
  T14.PR_DEPT_OU_ID,
  T8.SYNC_SUCCESS_FLG,
  T14.TIMEZONE_ID,
  T7.NAME,
  T12.X_IWS_USERNAME,
  T12.X_IWS_FLAG,
  T5.NAME,
  T11.LOGIN,
  T1.NAME,
  T12.ROW_ID,
  T12.PAR_ROW_ID,
  T12.MODIFICATION_NUM,
  T12.CREATED_BY,
  T12.LAST_UPD_BY,
  T12.CREATED,
  T12.LAST_UPD,
  T12.CONFLICT_ID,
  T12.PAR_ROW_ID,
  T14.ROW_ID,
  T14.PAR_ROW_ID,
  T14.MODIFICATION_NUM,
  T14.CREATED_BY,
  T14.LAST_UPD_BY,
  T14.CREATED,
  T14.LAST_UPD,
  T14.CONFLICT_ID,
  T14.PAR_ROW_ID,
  T8.ROW_ID,
  T8.PAR_ROW_ID,
  T8.MODIFICATION_NUM,
  T8.CREATED_BY,
  T8.LAST_UPD_BY,
  T8.CREATED,
  T8.LAST_UPD,
  T8.CONFLICT_ID,
  T8.PAR_ROW_ID,
  T9.ROW_ID,
  T9.PAR_ROW_ID,
  T9.MODIFICATION_NUM,
  T9.CREATED_BY,
  T9.LAST_UPD_BY,
  T9.CREATED,
  T9.LAST_UPD,
  T9.CONFLICT_ID,
  T9.PAR_ROW_ID,
  T4.ROW_ID,
  T1.ROW_ID
FROM 
   SIEBEL.S_RESP T1,
   SIEBEL.S_LANG T2,
   SIEBEL.S_POSTN T3,
   SIEBEL.S_PARTY T4,
   SIEBEL.S_TIMEZONE_LANG T5,
   SIEBEL.S_PARTY T6,
   SIEBEL.S_TIMEZONE T7,
   SIEBEL.S_EMP_PER T8,
   SIEBEL.S_CONTACT_SS T9,
   SIEBEL.S_SRV_REGN T10,
   SIEBEL.S_USER T11,
   SIEBEL.S_USER T12,
   SIEBEL.S_BU T13,
   SIEBEL.S_CONTACT T14
WHERE 
  T14.PR_HELD_POSTN_ID = T3.PAR_ROW_ID (+) AND
  T3.BU_ID = T13.ROW_ID (+) AND
  T12.X_PREF_CUST_LANGUAGE = T2.LANG_CD (+) AND
  T8.SRV_REGN_ID = T10.ROW_ID (+) AND
  T14.TIMEZONE_ID = T7.ROW_ID (+) AND
  T7.ROW_ID = T5.PAR_ROW_ID (+) AND T5.LANG_ID (+) = :1 AND
  T6.ROW_ID = T12.PAR_ROW_ID AND
  T6.ROW_ID = T14.PAR_ROW_ID AND
  T6.ROW_ID = T8.PAR_ROW_ID AND
  T6.ROW_ID = T9.PAR_ROW_ID (+) AND
  T14.PR_SYNC_USER_ID = T4.ROW_ID (+) AND
  T14.PR_SYNC_USER_ID = T11.PAR_ROW_ID (+) AND
  T14.PR_RESP_ID = T1.ROW_ID (+) AND
  (T14.EMP_FLG = 'Y') AND
  (T6.ROW_ID = :2)
ORDER BY
  T14.EMP_FLG, T14.LAST_NAME, T14.FST_NAME

I can see examples for Timestamp in Other forums as start date of Time stamp ^ Timestamp , But in My log time stamp is in middle , How can i specify this ? Any suggestions please ?

shaunak · January 13, 2020, 6:03pm

How about something like this:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /path/to/your/logs/*.log
  multiline.pattern: '\d{4}-\d{2}-\d{2}'
  multiline.negate: true
  multiline.match: after

The key there, of course, is the regular expression pattern you choose to specify in the multiline.pattern setting. I've started you off with a pretty basic one above. It should work for the sample logs you've provided. But you might need a more sophisticated pattern depending on any edge cases, e.g. if a timestamp could appear as part of a query that's being logged.

You can read more about multiline matching in Filebeat here: https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html.

Hope that helps,

Shaunak

Saravana37 · January 14, 2020, 4:59am

Hello @shaunak , Thanks for your reply.

I tried above lines but still getting only SELECT in the message, Not the complete query . Can you please help ?

Saravana37 · January 14, 2020, 6:34am

Hello @shaunak , Apologies and it is working . But it is working if i have single input file ,

If i have multiple input files , multiline pattern is getting applied for the last input section . How can i specify this multiline gobally ? Please help.

Saravana37 · January 14, 2020, 7:16am

Hello @shaunak ,

It works, thank you very much. I have given multiple paths like array now and the multiline pattern is working for all.

system · February 11, 2020, 7:16am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.