Multiline pattern for covering all inconsistencies

Elastic Stack Beats
1

Hello All,

The application log generates messages which include various lines (not the same number every time). It also contains messages in XML form and various other kind.

Is there a way I could define a multiline.pattern which covers "all" possible options and gives a blanket coverage to the entire message?

I just need to push the entire message in one field.

Thanks

2

You need to provide some sample of your messages and what can constitute one message with multiple lines.

3

Thanks. Here are a few examples. Lot of variety is present :slight_smile:

2023-11-01 07:19:13,385~|~INFO~|~hostname~|~com.vice.ServiceClient~|~Service Response : ^M
{
  "redacted" : "Y",
  "redacted" : "Y",
  "redacted" : "Y",
  "redacted" : "Y",
  "redacted" : "Y",
  "redacted" : "",
  "redacted" : false,
  "redacted" : false,
  "redacted" : "(GMT+01:00) Brussels, Copenhagen, Madrid, Paris",
  "redacted" : false,
  "redacted" : false,
  "redacted" : "",
  "redacted" : false,
  }~|~

2023-11-01 07:19:58,320~|~INFO~|~hostname~|~com.Controller~|~Key, Value :  cookie = _cls_v=fd4e54b5-ccab-40e3-a60c-a43de201bde2; bmuid=1698822663948-2C82CBD6-78CC-49B2-A8D8-EF7D198076DE; langid=en-US; BROWSER_TOKEN=5261046812441792; choice=B; ak_bmsc=CF22EB4EB4D4121B60C5A9460B7150A2~000000000000000000000000000000~YAAQlHjOF4FVZmOLAQAA3EMxihUfOL08EE6+jwEZiBy/e7mF2bmaLcYWpsAftkzvEVnccCvtE5yq3GcLUN1pc5OWm+AuLJ4V3fub1uEqIHQ1raczDLKgMLpKptDxiO//oAzeFX0sCMynmWWWulP8xSefd5WNZb65y7PvV5tWBiN74P+3EKmrv8ZwqP55lcCmZ9dJXdkXLqLoY3d0fvHbJGa2yRIZQgU3MUXJAKTtc8IUCznMAICMtvtkiu/ZHVcj3hrUwp0SdG980v1t9fbzDbFhnrAPeOg9nOkn4tua6vXDYFg+ND0p9ir0uRmoEYXm+gkYAWYlF0Hg3r2qIEHZdamx3Adp/T419gGPlD1y1fZ0BlqFXo9/RyDLchwCXmhY1xnXfPqYKvxd7nj5gL0=; cdSNum=1698832092589-sjc0000102-021e42a6-2e97-4c8e-af65-e50c20cdf97a; sz=F4F9B118EB0BA262CFF57B40E9880623~YAAQlKg4F3+Dbm6LAQAACY6KihWeb+v4uQzNVwzD8p87SNZrLDN5iVlPryqb0ubKowFwfosHw5KlQ2PJJyWzK9lYLF8TlWmdktqFshYyBIO5R47iDSg9RUpk70CP5pMCdZBKikTtbfiIa/j6uyWUDmkyfoPbPbG4bk/t9oeiWJDVG5IGEit47wlMqeiOT/s7BfuAf9Gxn77eLQD4BjDlODmtjUIhUIWSv5GURa0kDmdYsU2kedLn0Zt+wy6Zbq2oynQXqKiQ9gbThKIRna/GHk4n4/ujRblSo0MAzm/1K6OJz7qgTbVDLrbzF3NrWBXJEKxAeBK5keajfKpg1k9F~3291206~4339012; _s=84e9763c-17ce-4221-962c-4daa5d3f3cf7:1; id=IN; P=6f67828f994e36eeaba721b7f75662db272125e5682e2ebaa7763c9c8f0d4ba3; loken=4d5d2be9-a4d7-4547-9628-c0831046d46d; UP=true; globalPref_date_format=MONTH_DAY_YEAR; format=USUK; globalPref_default_locale=en_US; DATA_LOCALE=GLOBAL; Association_Mode=None; D=82019491; Dqa=82019491; L1BvcnRhbA2=f_v0MQiDrbGI84wI8EBv0NMcooWqLSiCaw3gmSoKcD5c4fAFouvR3jElBTXjksZTWFstbpqxSh6NFVk0dsWkkllWj2BsQ5-3cqgOlhkYosU1; be=portal; PU_SP=18d91c4b46cfb4274d146316715cdba7966c5560ad5c03967189eb339a5f845b16830bfc71b98a32b5492366d0628bec60e7ff33feb725752dcf865aa1fb90139109c63efd4695625f4a9818df46d462f120124cb00151ede8991a5958722a5a9859227feb34c8ae7ae02b836c833e2a6f2f7f2b5ac17e14fbefddc92f514d7bcfae0572b6411995ca083c8c116062e9dff34ad46b0c77dde878e49392e083c9699e9de4a512c3f5c7e0ac36ec95491273b87adcaf115e201420a24e7afc2c0f3fe6c513dd453caffd71bf5ffbfcd99413222baa9aeab4c81e83cfd637f2cb5ac9bb4ed3e2491488dee82950714159c6d5660280a6f3d0b19e2c632a75a9e2f2; ON=LOGGEDOFF; .ASP=8D3DFE647EA2F4B7C2B36EA92BDBF14BAE8D13514D556ADBB4C3AB42CF57E9C49E7D5A923F4BB04FF90F08B5AEAE872523F7E291FFE340957C6E0CE1CBD6FD1EA7E328980AC5D27C1273231B7C0A43F083A7F659E444B763D63C2F299890C13FB25FDD7576C9B9A2EE45145E51B8B45D95FCF319D730FC9703B3F8E57E2CD99107AF8ADEFF107851B20B1E1C350C047F941C3EA2CD719CB1CB23D49BA7DA17230D44B636E90ECC95544BD27F7D68C9D387158BBF890CFB857EA53A91AE81A2F9E638352F10DB7383D86CAA863700F4008D0F74F2DB78C15473F84E2AC4FC70F6; session=-1; X-REZX=TBGtv3QMIjgOQfu7KkYp_vgdglx-1wHWGyQODJKV4dDWzRNcH_ZiISLTN7k8kGfybzT70kpJGe6Qe4J4Vg_xIIWjaquglzLNyWQnk3r736s1:9r1ZvvVW3sl9_ik0vqeWjvViVuwO2nejjtZ45SrMvBN8DWk8HNjpwl3dBwMW7rtWiRgNGP64-xCKD5r4HsA6bZ-UV13lhzKIQKnHpISxLNE1; ZIDI=06e65c05-36a0-4458-aff8-01341804ece1; BZCID=24897784-f387-40e0-b97f-d00613ac9905; Type=form; Oer=%C2U%ACK%u0109%A2%B4G%D2%9A%A8t%ACv%C4%8D; SSite=None; lnkt=e034cd94-a795-44f9-942a-1254477c809b; Aud=8467; Diuz=bWFnbTg0Njc; UM_Bsd=R:35|g:34b19421-e450-4491-90f1-3fd751b0ab48|n:zoomer1_a6584e1e-55e3-4797-bf22-e75ff520b393; T1=R:35|i:60813|e:33; bsv=DCD9A25B4B642F202B5D963F0F096B4D~YAAQlHjOF+CUbGOLAQAAZY2cihVDm8WoOb1wMB/jOlD/O4lTYlRCK7BGdShSLrlDdZOeq/yw+McNg6PCHpw9TclAPWQoNsSk/qfQdkJiPwZxrXbG75aEHPV1NeCZucd2qvAerQ8sK1aA7Ox+fK1A280xkmjSpjtB3sDd8yqqzqwNrn5EaPvg+z94F6Cc/nYTc9ljUBn9PMFF59wEB2B8Gj6/0KNsfxFETgSZc4ubhQgi3Rvn2gvbS15ozfxFPGpEgIRmUvyE~1; za_abck=EF8ECF49AAEC5D9F3D94980A37D7BA6F~0~YAAQlHjOF/iUbGOLAQAAp4+cigrZjjmPdFKCLiSMlYHV2BJgnX3homhMBQ1Ypjxni7CIn3tWG99cXkEP0/ddxY5/MuI5hj3/fcM+CyLdR2UIHo/uYcB7CKfrHTf8eYVGPrkdU1dUa2GFLiL64N5+fN2UcvTJ6FG+jlpVOjOebJ6tZ8ll8wsCiDEa2hudxhHb8y+SUe7wPebu7yQG3ICtdtQnaWCAuToshaItWz+9+aMHCt9+o26dwWHarxZohBKIb27pMREkXTip8eJzLTwgynUr30MisLYjJBowiAuXtpm+z5nkVGgQyZ0xB5TRFPwdWca1IJYMSkEGSQNL7I5ZqV1aszQIlcqIw3vRK7cdMDK0wRwnsZhbRsibGiTFe8Nn4sQ=~-1~-1~-1 ~|~

2023-11-01 07:21:37,846~|~INFO~|~hostname~|~com.Client~|~Service Response : ^M"{\"Id\":\"ssd3d\",\"Parity\":22973235,\"utyId\":null,\"prefDr\":null,\"es\":[\"032\",\"750\",\"850\",\"267\",\"950\",\"268\",\"940\",\"269\",\"600\",\"930\",\"920\",\"953\",\"436\",\"889\",\"803\"],\"xdds\":[\"AR\",\"ZM\",\"TH\",\"JP\",\"PH\",\"IT\",\"PK\",\"GB\",\"MY\",\"PL\",\"US\"],\"sdsdes\":[\"USER_ENT\",\"AVLQ\",\"sdss_2000\",\"TR\",\"GTFIIP\",\"SERC\",\"AARP\",\"dfffY\",\"FLOW\",\"lemMNT\",\"GFIPI\",\"PMTE2C\",\"ACDDARP\",\"MB\",\"IB\",\"PAYMUUT\",\"PRF\",\"RNR\",\"OODD\",\"CLIDEF\",\"SR\",\"OONTS\",\"TION\",\"NT_AS\",\"EFI\",\"INQ\",\"NQ\",\"DPI\",\"TASS\",\"ASSADM\",\"RUN\",\"DSWP\",\"PP\",\"CINQ\",\"ASGNR\",\"SINQ\",\"ROD\",\"TEDD\",\"SMPPT\",\"LCINQ\",\"INT\",\"RDP\",\"CH\",\"LOATE\",\"PPCONC\",\"TBBDELIB\",\"MMT\",\"ININQ\",\"IVSUPP\",\"WN\",\"NQ\",\"SRM\",\"CFS\",\"CTMGKT\",\"FIINQ\",\"MGR\",\"UST\",\"EAS\",\"DEBPT\",\"MGR\",\"UP\",\"AP\",\"MARP\"ULARP\",\"I\",\"ONCD\",\"TEF\",\"TDOCINQ\",\"IIHED\",\"TPI\",\"T\",\"IIBRIES\",\"MENTS\",\"CMCOLPI\",\"RDE\",\"ECE\",\"ITARP\",\"PGHORPAY\",\"OOKG\",\"SHPI\",\"TSSO\"],\"ponner\":null,\"OOgions\":[\"GG\"]}"~|~

2023-11-01 07:46:04,924~|~INFO~|~hostname~|~com.Client~|~Returning Response for user qabcdf :: {
  "status" : "NONE",
  "level" : "JJOKKJ"
}~|~
4

For all those messages you just need a pattern that match the date, because if the line does not start with a date, then it is part of the above line.

Check these examples from the documentation

In your case you would need something like this:

parsers:
- multiline:
    type: pattern
    pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
    negate: true
    match: after
5

Thanks.

Going by your advice, my filebeat.yml now looks like below:

- type: log
  paths:
    - /opt/auth/Auth.log

  fields:
     type: abgho-auth
  fields_under_root: true

  filebeat.registry.path: /opt/auth/data/registry
  setup.dashboards.enabled: true

  multiline.type: pattern
  multiline.pattern: '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after

Now the messages seem to pile up under one field. See below:

Capture
Capture1002×452 17 KB

Also can you please educate me on what does the regex '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}' signify?

Also the date/ timestamp in my case is 2023-11-02 02:21:58,820

I guess I need to find an equivalent pattern for that.

6

This regex is wrong, it is an example regex from the documentation that will match the example logs, which are different from yours.

You should use the one I shared in the previous post:

This is the correct pattern:

pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'

[0-9] means that it will match any number and the {4} means that it will match a sequence of 4 numbers, the - will match a literal -.

So the pattern will match anything between 0000-00-00 and 9999-99-99, which is enough to match all dates in the format yyyy-MM-dd.

7

Thanks @leandrojmp

It helped me a lot

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.