# Multiline pattern for covering all inconsistencies
**URL:** https://discuss.elastic.co/t/multiline-pattern-for-covering-all-inconsistencies/346162
**Category:** Beats
**Tags:** filebeat
**Created:** [October 31, 2023, 10:07pm UTC](https://discuss.elastic.co/t/multiline-pattern-for-covering-all-inconsistencies/346162 "2023-10-31T22:07:22Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)
#### Post date: [October 31, 2023, 10:07pm UTC](https://discuss.elastic.co/t/multiline-pattern-for-covering-all-inconsistencies/346162/1 "2023-10-31T22:07:22Z")
Hello All,
The application log generates messages which include various lines (not the same number every time). It also contains messages in XML form and various other kind.
Is there a way I could define a multiline.pattern which covers "all" possible options and gives a blanket coverage to the entire message?
I just need to push the entire message in one field.
Thanks
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [October 31, 2023, 10:53pm UTC](https://discuss.elastic.co/t/multiline-pattern-for-covering-all-inconsistencies/346162/2 "2023-10-31T22:53:05Z")
> [@zaeemmasood](#):
>
> Is there a way I could define a multiline.pattern which covers "all" possible options and gives a blanket coverage to the entire message?
You need to provide some sample of your messages and what can constitute one message with multiple lines.
---
### Author:  [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)
#### Post date: [November 1, 2023, 3:36pm UTC](https://discuss.elastic.co/t/multiline-pattern-for-covering-all-inconsistencies/346162/3 "2023-11-01T15:36:24Z")
Thanks. Here are a few examples. Lot of variety is present 🙂
```auto
2023-11-01 07:19:13,385~|~INFO~|~hostname~|~com.vice.ServiceClient~|~Service Response : ^M
{
"redacted" : "Y",
"redacted" : "Y",
"redacted" : "Y",
"redacted" : "Y",
"redacted" : "Y",
"redacted" : "",
"redacted" : false,
"redacted" : false,
"redacted" : "(GMT+01:00) Brussels, Copenhagen, Madrid, Paris",
"redacted" : false,
"redacted" : false,
"redacted" : "",
"redacted" : false,
}~|~
```
```auto
2023-11-01 07:19:58,320~|~INFO~|~hostname~|~com.Controller~|~Key, Value : cookie = _cls_v=fd4e54b5-ccab-40e3-a60c-a43de201bde2; bmuid=1698822663948-2C82CBD6-78CC-49B2-A8D8-EF7D198076DE; langid=en-US; BROWSER_TOKEN=5261046812441792; choice=B; ak_bmsc=CF22EB4EB4D4121B60C5A9460B7150A2~000000000000000000000000000000~YAAQlHjOF4FVZmOLAQAA3EMxihUfOL08EE6+jwEZiBy/e7mF2bmaLcYWpsAftkzvEVnccCvtE5yq3GcLUN1pc5OWm+AuLJ4V3fub1uEqIHQ1raczDLKgMLpKptDxiO//oAzeFX0sCMynmWWWulP8xSefd5WNZb65y7PvV5tWBiN74P+3EKmrv8ZwqP55lcCmZ9dJXdkXLqLoY3d0fvHbJGa2yRIZQgU3MUXJAKTtc8IUCznMAICMtvtkiu/ZHVcj3hrUwp0SdG980v1t9fbzDbFhnrAPeOg9nOkn4tua6vXDYFg+ND0p9ir0uRmoEYXm+gkYAWYlF0Hg3r2qIEHZdamx3Adp/T419gGPlD1y1fZ0BlqFXo9/RyDLchwCXmhY1xnXfPqYKvxd7nj5gL0=; cdSNum=1698832092589-sjc0000102-021e42a6-2e97-4c8e-af65-e50c20cdf97a; sz=F4F9B118EB0BA262CFF57B40E9880623~YAAQlKg4F3+Dbm6LAQAACY6KihWeb+v4uQzNVwzD8p87SNZrLDN5iVlPryqb0ubKowFwfosHw5KlQ2PJJyWzK9lYLF8TlWmdktqFshYyBIO5R47iDSg9RUpk70CP5pMCdZBKikTtbfiIa/j6uyWUDmkyfoPbPbG4bk/t9oeiWJDVG5IGEit47wlMqeiOT/s7BfuAf9Gxn77eLQD4BjDlODmtjUIhUIWSv5GURa0kDmdYsU2kedLn0Zt+wy6Zbq2oynQXqKiQ9gbThKIRna/GHk4n4/ujRblSo0MAzm/1K6OJz7qgTbVDLrbzF3NrWBXJEKxAeBK5keajfKpg1k9F~3291206~4339012; _s=84e9763c-17ce-4221-962c-4daa5d3f3cf7:1; id=IN; P=6f67828f994e36eeaba721b7f75662db272125e5682e2ebaa7763c9c8f0d4ba3; loken=4d5d2be9-a4d7-4547-9628-c0831046d46d; UP=true; globalPref_date_format=MONTH_DAY_YEAR; format=USUK; globalPref_default_locale=en_US; DATA_LOCALE=GLOBAL; Association_Mode=None; D=82019491; Dqa=82019491; L1BvcnRhbA2=f_v0MQiDrbGI84wI8EBv0NMcooWqLSiCaw3gmSoKcD5c4fAFouvR3jElBTXjksZTWFstbpqxSh6NFVk0dsWkkllWj2BsQ5-3cqgOlhkYosU1; be=portal; PU_SP=18d91c4b46cfb4274d146316715cdba7966c5560ad5c03967189eb339a5f845b16830bfc71b98a32b5492366d0628bec60e7ff33feb725752dcf865aa1fb90139109c63efd4695625f4a9818df46d462f120124cb00151ede8991a5958722a5a9859227feb34c8ae7ae02b836c833e2a6f2f7f2b5ac17e14fbefddc92f514d7bcfae0572b6411995ca083c8c116062e9dff34ad46b0c77dde878e49392e083c9699e9de4a512c3f5c7e0ac36ec95491273b87adcaf115e201420a24e7afc2c0f3fe6c513dd453caffd71bf5ffbfcd99413222baa9aeab4c81e83cfd637f2cb5ac9bb4ed3e2491488dee82950714159c6d5660280a6f3d0b19e2c632a75a9e2f2; ON=LOGGEDOFF; .ASP=8D3DFE647EA2F4B7C2B36EA92BDBF14BAE8D13514D556ADBB4C3AB42CF57E9C49E7D5A923F4BB04FF90F08B5AEAE872523F7E291FFE340957C6E0CE1CBD6FD1EA7E328980AC5D27C1273231B7C0A43F083A7F659E444B763D63C2F299890C13FB25FDD7576C9B9A2EE45145E51B8B45D95FCF319D730FC9703B3F8E57E2CD99107AF8ADEFF107851B20B1E1C350C047F941C3EA2CD719CB1CB23D49BA7DA17230D44B636E90ECC95544BD27F7D68C9D387158BBF890CFB857EA53A91AE81A2F9E638352F10DB7383D86CAA863700F4008D0F74F2DB78C15473F84E2AC4FC70F6; session=-1; X-REZX=TBGtv3QMIjgOQfu7KkYp_vgdglx-1wHWGyQODJKV4dDWzRNcH_ZiISLTN7k8kGfybzT70kpJGe6Qe4J4Vg_xIIWjaquglzLNyWQnk3r736s1:9r1ZvvVW3sl9_ik0vqeWjvViVuwO2nejjtZ45SrMvBN8DWk8HNjpwl3dBwMW7rtWiRgNGP64-xCKD5r4HsA6bZ-UV13lhzKIQKnHpISxLNE1; ZIDI=06e65c05-36a0-4458-aff8-01341804ece1; BZCID=24897784-f387-40e0-b97f-d00613ac9905; Type=form; Oer=%C2U%ACK%u0109%A2%B4G%D2%9A%A8t%ACv%C4%8D; SSite=None; lnkt=e034cd94-a795-44f9-942a-1254477c809b; Aud=8467; Diuz=bWFnbTg0Njc; UM_Bsd=R:35|g:34b19421-e450-4491-90f1-3fd751b0ab48|n:zoomer1_a6584e1e-55e3-4797-bf22-e75ff520b393; T1=R:35|i:60813|e:33; bsv=DCD9A25B4B642F202B5D963F0F096B4D~YAAQlHjOF+CUbGOLAQAAZY2cihVDm8WoOb1wMB/jOlD/O4lTYlRCK7BGdShSLrlDdZOeq/yw+McNg6PCHpw9TclAPWQoNsSk/qfQdkJiPwZxrXbG75aEHPV1NeCZucd2qvAerQ8sK1aA7Ox+fK1A280xkmjSpjtB3sDd8yqqzqwNrn5EaPvg+z94F6Cc/nYTc9ljUBn9PMFF59wEB2B8Gj6/0KNsfxFETgSZc4ubhQgi3Rvn2gvbS15ozfxFPGpEgIRmUvyE~1; za_abck=EF8ECF49AAEC5D9F3D94980A37D7BA6F~0~YAAQlHjOF/iUbGOLAQAAp4+cigrZjjmPdFKCLiSMlYHV2BJgnX3homhMBQ1Ypjxni7CIn3tWG99cXkEP0/ddxY5/MuI5hj3/fcM+CyLdR2UIHo/uYcB7CKfrHTf8eYVGPrkdU1dUa2GFLiL64N5+fN2UcvTJ6FG+jlpVOjOebJ6tZ8ll8wsCiDEa2hudxhHb8y+SUe7wPebu7yQG3ICtdtQnaWCAuToshaItWz+9+aMHCt9+o26dwWHarxZohBKIb27pMREkXTip8eJzLTwgynUr30MisLYjJBowiAuXtpm+z5nkVGgQyZ0xB5TRFPwdWca1IJYMSkEGSQNL7I5ZqV1aszQIlcqIw3vRK7cdMDK0wRwnsZhbRsibGiTFe8Nn4sQ=~-1~-1~-1 ~|~
```
```auto
2023-11-01 07:21:37,846~|~INFO~|~hostname~|~com.Client~|~Service Response : ^M"{\"Id\":\"ssd3d\",\"Parity\":22973235,\"utyId\":null,\"prefDr\":null,\"es\":[\"032\",\"750\",\"850\",\"267\",\"950\",\"268\",\"940\",\"269\",\"600\",\"930\",\"920\",\"953\",\"436\",\"889\",\"803\"],\"xdds\":[\"AR\",\"ZM\",\"TH\",\"JP\",\"PH\",\"IT\",\"PK\",\"GB\",\"MY\",\"PL\",\"US\"],\"sdsdes\":[\"USER_ENT\",\"AVLQ\",\"sdss_2000\",\"TR\",\"GTFIIP\",\"SERC\",\"AARP\",\"dfffY\",\"FLOW\",\"lemMNT\",\"GFIPI\",\"PMTE2C\",\"ACDDARP\",\"MB\",\"IB\",\"PAYMUUT\",\"PRF\",\"RNR\",\"OODD\",\"CLIDEF\",\"SR\",\"OONTS\",\"TION\",\"NT_AS\",\"EFI\",\"INQ\",\"NQ\",\"DPI\",\"TASS\",\"ASSADM\",\"RUN\",\"DSWP\",\"PP\",\"CINQ\",\"ASGNR\",\"SINQ\",\"ROD\",\"TEDD\",\"SMPPT\",\"LCINQ\",\"INT\",\"RDP\",\"CH\",\"LOATE\",\"PPCONC\",\"TBBDELIB\",\"MMT\",\"ININQ\",\"IVSUPP\",\"WN\",\"NQ\",\"SRM\",\"CFS\",\"CTMGKT\",\"FIINQ\",\"MGR\",\"UST\",\"EAS\",\"DEBPT\",\"MGR\",\"UP\",\"AP\",\"MARP\"ULARP\",\"I\",\"ONCD\",\"TEF\",\"TDOCINQ\",\"IIHED\",\"TPI\",\"T\",\"IIBRIES\",\"MENTS\",\"CMCOLPI\",\"RDE\",\"ECE\",\"ITARP\",\"PGHORPAY\",\"OOKG\",\"SHPI\",\"TSSO\"],\"ponner\":null,\"OOgions\":[\"GG\"]}"~|~
```
```auto
2023-11-01 07:46:04,924~|~INFO~|~hostname~|~com.Client~|~Returning Response for user qabcdf :: {
"status" : "NONE",
"level" : "JJOKKJ"
}~|~
```
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [November 1, 2023, 4:24pm UTC](https://discuss.elastic.co/t/multiline-pattern-for-covering-all-inconsistencies/346162/4 "2023-11-01T16:24:29Z")
For all those messages you just need a pattern that match the date, because if the line does not start with a date, then it is part of the above line.
Check these [examples from the documentation](https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html#_timestamps)
In your case you would need something like this:
```auto
parsers:
- multiline:
type: pattern
pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
negate: true
match: after
```
---
### Author:  [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)
#### Post date: [November 2, 2023, 1:57pm UTC](https://discuss.elastic.co/t/multiline-pattern-for-covering-all-inconsistencies/346162/5 "2023-11-02T13:57:24Z")
Thanks.
Going by your advice, my filebeat.yml now looks like below:
```auto
- type: log
paths:
- /opt/auth/Auth.log
fields:
type: abgho-auth
fields_under_root: true
filebeat.registry.path: /opt/auth/data/registry
setup.dashboards.enabled: true
multiline.type: pattern
multiline.pattern: '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after
```
Now the messages seem to pile up under one field. See below:
Also can you please educate me on what does the regex ` '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'` signify?
Also the date/ timestamp in my case is `2023-11-02 02:21:58,820`
I guess I need to find an equivalent pattern for that.
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [November 2, 2023, 2:01pm UTC](https://discuss.elastic.co/t/multiline-pattern-for-covering-all-inconsistencies/346162/6 "2023-11-02T14:01:19Z")
> [@zaeemmasood](#):
>
> Also can you please educate me on what does the regex ` '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'` signify?
This regex is wrong, it is an example regex from the documentation that will match the example logs, which are different from yours.
You should use the one I shared in the previous post:
> [@leandrojmp](#):
>
> ```auto
> parsers:
> - multiline:
> type: pattern
> pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
> negate: true
> match: after
>
> ```
This is the correct pattern:
> `pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'`
`[0-9]` means that it will match any number and the `{4}` means that it will match a sequence of 4 numbers, the `-` will match a literal `-`.
So the pattern will match anything between `0000-00-00` and `9999-99-99`, which is enough to match all dates in the format `yyyy-MM-dd`.
---
### Author:  [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)
#### Post date: [November 2, 2023, 2:46pm UTC](https://discuss.elastic.co/t/multiline-pattern-for-covering-all-inconsistencies/346162/7 "2023-11-02T14:46:00Z")
Thanks @leandrojmp
It helped me a lot
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 30, 2023, 4:46pm UTC](https://discuss.elastic.co/t/multiline-pattern-for-covering-all-inconsistencies/346162/8 "2023-11-30T16:46:56Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.