Multiline pattern for covering all inconsistencies

zaeemmasood · October 31, 2023, 10:07pm

Hello All,

The application log generates messages which include various lines (not the same number every time). It also contains messages in XML form and various other kind.

Is there a way I could define a multiline.pattern which covers "all" possible options and gives a blanket coverage to the entire message?

I just need to push the entire message in one field.

Thanks

leandrojmp · October 31, 2023, 10:53pm

You need to provide some sample of your messages and what can constitute one message with multiple lines.

zaeemmasood · November 1, 2023, 3:36pm

Thanks. Here are a few examples. Lot of variety is present

2023-11-01 07:19:13,385~|~INFO~|~hostname~|~com.vice.ServiceClient~|~Service Response : ^M
{
  "redacted" : "Y",
  "redacted" : "Y",
  "redacted" : "Y",
  "redacted" : "Y",
  "redacted" : "Y",
  "redacted" : "",
  "redacted" : false,
  "redacted" : false,
  "redacted" : "(GMT+01:00) Brussels, Copenhagen, Madrid, Paris",
  "redacted" : false,
  "redacted" : false,
  "redacted" : "",
  "redacted" : false,
  }~|~

2023-11-01 07:19:58,320~|~INFO~|~hostname~|~com.Controller~|~Key, Value :  cookie = _cls_v=fd4e54b5-ccab-40e3-a60c-a43de201bde2; bmuid=1698822663948-2C82CBD6-78CC-49B2-A8D8-EF7D198076DE; langid=en-US; BROWSER_TOKEN=5261046812441792; choice=B; ak_bmsc=CF22EB4EB4D4121B60C5A9460B7150A2~000000000000000000000000000000~YAAQlHjOF4FVZmOLAQAA3EMxihUfOL08EE6+jwEZiBy/e7mF2bmaLcYWpsAftkzvEVnccCvtE5yq3GcLUN1pc5OWm+AuLJ4V3fub1uEqIHQ1raczDLKgMLpKptDxiO//oAzeFX0sCMynmWWWulP8xSefd5WNZb65y7PvV5tWBiN74P+3EKmrv8ZwqP55lcCmZ9dJXdkXLqLoY3d0fvHbJGa2yRIZQgU3MUXJAKTtc8IUCznMAICMtvtkiu/ZHVcj3hrUwp0SdG980v1t9fbzDbFhnrAPeOg9nOkn4tua6vXDYFg+ND0p9ir0uRmoEYXm+gkYAWYlF0Hg3r2qIEHZdamx3Adp/T419gGPlD1y1fZ0BlqFXo9/RyDLchwCXmhY1xnXfPqYKvxd7nj5gL0=; cdSNum=1698832092589-sjc0000102-021e42a6-2e97-4c8e-af65-e50c20cdf97a; sz=F4F9B118EB0BA262CFF57B40E9880623~YAAQlKg4F3+Dbm6LAQAACY6KihWeb+v4uQzNVwzD8p87SNZrLDN5iVlPryqb0ubKowFwfosHw5KlQ2PJJyWzK9lYLF8TlWmdktqFshYyBIO5R47iDSg9RUpk70CP5pMCdZBKikTtbfiIa/j6uyWUDmkyfoPbPbG4bk/t9oeiWJDVG5IGEit47wlMqeiOT/s7BfuAf9Gxn77eLQD4BjDlODmtjUIhUIWSv5GURa0kDmdYsU2kedLn0Zt+wy6Zbq2oynQXqKiQ9gbThKIRna/GHk4n4/ujRblSo0MAzm/1K6OJz7qgTbVDLrbzF3NrWBXJEKxAeBK5keajfKpg1k9F~3291206~4339012; _s=84e9763c-17ce-4221-962c-4daa5d3f3cf7:1; id=IN; P=6f67828f994e36eeaba721b7f75662db272125e5682e2ebaa7763c9c8f0d4ba3; loken=4d5d2be9-a4d7-4547-9628-c0831046d46d; UP=true; globalPref_date_format=MONTH_DAY_YEAR; format=USUK; globalPref_default_locale=en_US; DATA_LOCALE=GLOBAL; Association_Mode=None; D=82019491; Dqa=82019491; L1BvcnRhbA2=f_v0MQiDrbGI84wI8EBv0NMcooWqLSiCaw3gmSoKcD5c4fAFouvR3jElBTXjksZTWFstbpqxSh6NFVk0dsWkkllWj2BsQ5-3cqgOlhkYosU1; be=portal; PU_SP=18d91c4b46cfb4274d146316715cdba7966c5560ad5c03967189eb339a5f845b16830bfc71b98a32b5492366d0628bec60e7ff33feb725752dcf865aa1fb90139109c63efd4695625f4a9818df46d462f120124cb00151ede8991a5958722a5a9859227feb34c8ae7ae02b836c833e2a6f2f7f2b5ac17e14fbefddc92f514d7bcfae0572b6411995ca083c8c116062e9dff34ad46b0c77dde878e49392e083c9699e9de4a512c3f5c7e0ac36ec95491273b87adcaf115e201420a24e7afc2c0f3fe6c513dd453caffd71bf5ffbfcd99413222baa9aeab4c81e83cfd637f2cb5ac9bb4ed3e2491488dee82950714159c6d5660280a6f3d0b19e2c632a75a9e2f2; ON=LOGGEDOFF; .ASP=8D3DFE647EA2F4B7C2B36EA92BDBF14BAE8D13514D556ADBB4C3AB42CF57E9C49E7D5A923F4BB04FF90F08B5AEAE872523F7E291FFE340957C6E0CE1CBD6FD1EA7E328980AC5D27C1273231B7C0A43F083A7F659E444B763D63C2F299890C13FB25FDD7576C9B9A2EE45145E51B8B45D95FCF319D730FC9703B3F8E57E2CD99107AF8ADEFF107851B20B1E1C350C047F941C3EA2CD719CB1CB23D49BA7DA17230D44B636E90ECC95544BD27F7D68C9D387158BBF890CFB857EA53A91AE81A2F9E638352F10DB7383D86CAA863700F4008D0F74F2DB78C15473F84E2AC4FC70F6; session=-1; X-REZX=TBGtv3QMIjgOQfu7KkYp_vgdglx-1wHWGyQODJKV4dDWzRNcH_ZiISLTN7k8kGfybzT70kpJGe6Qe4J4Vg_xIIWjaquglzLNyWQnk3r736s1:9r1ZvvVW3sl9_ik0vqeWjvViVuwO2nejjtZ45SrMvBN8DWk8HNjpwl3dBwMW7rtWiRgNGP64-xCKD5r4HsA6bZ-UV13lhzKIQKnHpISxLNE1; ZIDI=06e65c05-36a0-4458-aff8-01341804ece1; BZCID=24897784-f387-40e0-b97f-d00613ac9905; Type=form; Oer=%C2U%ACK%u0109%A2%B4G%D2%9A%A8t%ACv%C4%8D; SSite=None; lnkt=e034cd94-a795-44f9-942a-1254477c809b; Aud=8467; Diuz=bWFnbTg0Njc; UM_Bsd=R:35|g:34b19421-e450-4491-90f1-3fd751b0ab48|n:zoomer1_a6584e1e-55e3-4797-bf22-e75ff520b393; T1=R:35|i:60813|e:33; bsv=DCD9A25B4B642F202B5D963F0F096B4D~YAAQlHjOF+CUbGOLAQAAZY2cihVDm8WoOb1wMB/jOlD/O4lTYlRCK7BGdShSLrlDdZOeq/yw+McNg6PCHpw9TclAPWQoNsSk/qfQdkJiPwZxrXbG75aEHPV1NeCZucd2qvAerQ8sK1aA7Ox+fK1A280xkmjSpjtB3sDd8yqqzqwNrn5EaPvg+z94F6Cc/nYTc9ljUBn9PMFF59wEB2B8Gj6/0KNsfxFETgSZc4ubhQgi3Rvn2gvbS15ozfxFPGpEgIRmUvyE~1; za_abck=EF8ECF49AAEC5D9F3D94980A37D7BA6F~0~YAAQlHjOF/iUbGOLAQAAp4+cigrZjjmPdFKCLiSMlYHV2BJgnX3homhMBQ1Ypjxni7CIn3tWG99cXkEP0/ddxY5/MuI5hj3/fcM+CyLdR2UIHo/uYcB7CKfrHTf8eYVGPrkdU1dUa2GFLiL64N5+fN2UcvTJ6FG+jlpVOjOebJ6tZ8ll8wsCiDEa2hudxhHb8y+SUe7wPebu7yQG3ICtdtQnaWCAuToshaItWz+9+aMHCt9+o26dwWHarxZohBKIb27pMREkXTip8eJzLTwgynUr30MisLYjJBowiAuXtpm+z5nkVGgQyZ0xB5TRFPwdWca1IJYMSkEGSQNL7I5ZqV1aszQIlcqIw3vRK7cdMDK0wRwnsZhbRsibGiTFe8Nn4sQ=~-1~-1~-1 ~|~

2023-11-01 07:21:37,846~|~INFO~|~hostname~|~com.Client~|~Service Response : ^M"{\"Id\":\"ssd3d\",\"Parity\":22973235,\"utyId\":null,\"prefDr\":null,\"es\":[\"032\",\"750\",\"850\",\"267\",\"950\",\"268\",\"940\",\"269\",\"600\",\"930\",\"920\",\"953\",\"436\",\"889\",\"803\"],\"xdds\":[\"AR\",\"ZM\",\"TH\",\"JP\",\"PH\",\"IT\",\"PK\",\"GB\",\"MY\",\"PL\",\"US\"],\"sdsdes\":[\"USER_ENT\",\"AVLQ\",\"sdss_2000\",\"TR\",\"GTFIIP\",\"SERC\",\"AARP\",\"dfffY\",\"FLOW\",\"lemMNT\",\"GFIPI\",\"PMTE2C\",\"ACDDARP\",\"MB\",\"IB\",\"PAYMUUT\",\"PRF\",\"RNR\",\"OODD\",\"CLIDEF\",\"SR\",\"OONTS\",\"TION\",\"NT_AS\",\"EFI\",\"INQ\",\"NQ\",\"DPI\",\"TASS\",\"ASSADM\",\"RUN\",\"DSWP\",\"PP\",\"CINQ\",\"ASGNR\",\"SINQ\",\"ROD\",\"TEDD\",\"SMPPT\",\"LCINQ\",\"INT\",\"RDP\",\"CH\",\"LOATE\",\"PPCONC\",\"TBBDELIB\",\"MMT\",\"ININQ\",\"IVSUPP\",\"WN\",\"NQ\",\"SRM\",\"CFS\",\"CTMGKT\",\"FIINQ\",\"MGR\",\"UST\",\"EAS\",\"DEBPT\",\"MGR\",\"UP\",\"AP\",\"MARP\"ULARP\",\"I\",\"ONCD\",\"TEF\",\"TDOCINQ\",\"IIHED\",\"TPI\",\"T\",\"IIBRIES\",\"MENTS\",\"CMCOLPI\",\"RDE\",\"ECE\",\"ITARP\",\"PGHORPAY\",\"OOKG\",\"SHPI\",\"TSSO\"],\"ponner\":null,\"OOgions\":[\"GG\"]}"~|~

2023-11-01 07:46:04,924~|~INFO~|~hostname~|~com.Client~|~Returning Response for user qabcdf :: {
  "status" : "NONE",
  "level" : "JJOKKJ"
}~|~

leandrojmp · November 1, 2023, 4:24pm

For all those messages you just need a pattern that match the date, because if the line does not start with a date, then it is part of the above line.

Check these examples from the documentation

In your case you would need something like this:

parsers:
- multiline:
    type: pattern
    pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
    negate: true
    match: after

zaeemmasood · November 2, 2023, 1:57pm

Thanks.

Going by your advice, my filebeat.yml now looks like below:

- type: log
  paths:
    - /opt/auth/Auth.log

  fields:
     type: abgho-auth
  fields_under_root: true

  filebeat.registry.path: /opt/auth/data/registry
  setup.dashboards.enabled: true

  multiline.type: pattern
  multiline.pattern: '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after

Now the messages seem to pile up under one field. See below:

Also can you please educate me on what does the regex '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}' signify?

Also the date/ timestamp in my case is 2023-11-02 02:21:58,820

I guess I need to find an equivalent pattern for that.

leandrojmp · November 2, 2023, 2:01pm

This regex is wrong, it is an example regex from the documentation that will match the example logs, which are different from yours.

You should use the one I shared in the previous post:

This is the correct pattern:

pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'

[0-9] means that it will match any number and the {4} means that it will match a sequence of 4 numbers, the - will match a literal -.

So the pattern will match anything between 0000-00-00 and 9999-99-99, which is enough to match all dates in the format yyyy-MM-dd.

zaeemmasood · November 2, 2023, 2:46pm

Thanks @leandrojmp

It helped me a lot

system · November 30, 2023, 4:46pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.