# Multiline, Multi Field input question - newbie here **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [November 10, 2018, 9:28pm UTC](https://discuss.elastic.co/t/multiline-multi-field-input-question-newbie-here/156166 "2018-11-10T21:28:41Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![newmember](https://avatars.discourse-cdn.com/v4/letter/n/ce7236/32.png) [@newmember](https://discuss.elastic.co/u/newmember)\ **Post date:** [November 10, 2018, 9:28pm UTC](https://discuss.elastic.co/t/multiline-multi-field-input-question-newbie-here/156166/1 "2018-11-10T21:28:41Z")
I am putting together bits and pieces from examples to create my first custom filebeats input. I have 10s of thousands of these files, that I would like to read into ES. ``` cdv_nrings=8 cdv_phone=16188835888 cdv_informat=NONE cdv_tries=1 cdv_callTime=0 cdv_newApp=arcVXML2 cdv_retryInterval=0 cdv_initialScript=http://10.30.30.17:8080/pre/vui/aOut/1176825 cdv_applicationData=15740 # 2015/09/18 17:03:16 ##Fri Sep 18 17:03:59 2015 #OutboundRetCode:603 VXML Event: error.com.arc.tel_initiatecall.tel_failure ``` I was thinking that I want to define some group names to match my unique field names: My regex doesn't work in the online testers, I cant see to describe the new line properly, maybe that's not my problem? ` multiline.pattern: '=(?Pre\w+$)\n=(?Pre\w+$)\n=(?Pre\w+$)\n=(?Pre\w+$)\n=(?Pre\w+$)\n=(?Pre\w+$)\n=(?Pre\w+$)\n=(?Pre\w+$)\n=(?Pre\w+$)\n#(?Pre\w+$)\n##(?Pre\w+$)\n#(?Pre\w+$)'` Then in my filebeat.yml file I would match the group name to the field name: ``` - type: log enabled: true close_eof: true paths: - C:\OCS\work\0.CDF* fields: log_type: work_active cdv_nrings: cdv_nrings cdv_phone: cdv_phone cdv_informat: cdv_informat cdv_tries: cdv_tries cdv_callTime: cdv_callTime cdv_newApp: cdv_newApp cdv_retryInterval: cdv_retryInterval cdv_initialScript: cdv_initialScript cdv_applicationData: cdv_applicationData date: date daydate: daydate OutboundRetCode: OutboundRetCode multiline.pattern: '=(?Pre\w+$)\n=(?Pre\w+$)\n=(?Pre\w+$)\n=(?Pre\w+$)\n=(?Pre\w+$)\n=(?Pre\w+$)\n=(?Pre\w+$)\n=(?Pre\w+$)\n=(?Pre\w+$)\n#(?Pre\w+$)\n##(?Pre\w+$)\n#(?Pre\w+$)' multiline.negate: false multiline.match: before ``` I tested my config and that passed: ``` C:\filebeat-6.4.3-windows-x86_64>filebeat test config filebeat.yml Config OK ``` Then I would set something up for Kibana Template but I have not got to this part at this time. I am on the correct track here? How does my regex look? Thanks Update: I tried this regex and I got closer but not perfect > ^(.+)=(.+)(\r\n\s+(.+))_|^#\s(.+)(\r\n\s+(.+))_|^##(.+)(\r\n\s+(.+))\*|^#(.+):(.+)(\r\n\s +(.+)) ---
**Author:** ![ShaneP](https://avatars.discourse-cdn.com/v4/letter/s/e9bcb4/32.png) [@ShaneP](https://discuss.elastic.co/u/ShaneP)\ **Post date:** [November 11, 2018, 5:55am UTC](https://discuss.elastic.co/t/multiline-multi-field-input-question-newbie-here/156166/2 "2018-11-11T05:55:13Z")
Wow, really [kissanime](https://kissanime.software/) great work mate.Keep it up you are almost [letgo](https://letgo.onl/) on the level of perfection. Regards, Shane. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [November 12, 2018, 8:59pm UTC](https://discuss.elastic.co/t/multiline-multi-field-input-question-newbie-here/156166/3 "2018-11-12T20:59:39Z")
Have you got multiple consecutive events? Also use `----` separator (or some other marker) to show where exactly you want to split the multiline. Having some more samples helps in seeing and understanding a pattern. ---
**Author:** ![newmember](https://avatars.discourse-cdn.com/v4/letter/n/ce7236/32.png) [@newmember](https://discuss.elastic.co/u/newmember)\ **Post date:** [November 13, 2018, 12:42am UTC](https://discuss.elastic.co/t/multiline-multi-field-input-question-newbie-here/156166/4 "2018-11-13T00:42:07Z")
Thanks The group of 12 lines at the top of the case equals one file. We generate 100,000+ files a day all with the same layout. Thanks I hope this helps ---
**Author:** ![newmember](https://avatars.discourse-cdn.com/v4/letter/n/ce7236/32.png) [@newmember](https://discuss.elastic.co/u/newmember)\ **Post date:** [November 13, 2018, 12:47am UTC](https://discuss.elastic.co/t/multiline-multi-field-input-question-newbie-here/156166/5 "2018-11-13T00:47:09Z")
I was reading up on how grok works, not that I have a grok log statement but I get the idea of moving the regex statement from identifying groups in regex and just creating a regex per feild and adding those statements to yml file. I'll try my thought tonight an update the case. ---
**Author:** ![newmember](https://avatars.discourse-cdn.com/v4/letter/n/ce7236/32.png) [@newmember](https://discuss.elastic.co/u/newmember)\ **Post date:** [November 13, 2018, 8:09am UTC](https://discuss.elastic.co/t/multiline-multi-field-input-question-newbie-here/156166/6 "2018-11-13T08:09:25Z")
I updated the yml to use kv and I deleted the registry then ran filebeat,exe again. It started but nothing loaded. Thoughts please? ``` - type: log enabled: true close_eof: true paths: - C:\OCS\work\0.CDF* filter { kv { source => "message" field_split => "=" include_keys => ["cdv_nrings", "cdv_phone", "cdv_informat", "cdv_tries", "cdv_callTime", "cdv_newApp", "cdv_retryInterval", "cdv_initialScript", "cdv_applicationData", "date", "daydate", "OutboundRetCode"] trim => "<>[]," trimkey => "<>[]," } } ``` ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [November 13, 2018, 12:31pm UTC](https://discuss.elastic.co/t/multiline-multi-field-input-question-newbie-here/156166/7 "2018-11-13T12:31:09Z")
So one file == 1 event? In this case you don't need a complicated regex. Just try to capture everything, no matter the contents. Grok or kv filter is not part of filebeat, but logstash or Ingest node. The `filter` config as used is a Logstash configuration. If you want to use it like this, publish the event to Logstash. If you want to turn your work into a filebeat module, better start with [Ingest Node](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html) (config `pipeline` in elasticsearch output). ---
**Author:** ![newmember](https://avatars.discourse-cdn.com/v4/letter/n/ce7236/32.png) [@newmember](https://discuss.elastic.co/u/newmember)\ **Post date:** [November 19, 2018, 7:41pm UTC](https://discuss.elastic.co/t/multiline-multi-field-input-question-newbie-here/156166/8 "2018-11-19T19:41:01Z")
Thanks Is it best practice to have the message "cdv\_informat=NONE" or should I create a field called "cdv\_informat" with a sample value of "NONE" in this example? Thanks ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [November 20, 2018, 10:29am UTC](https://discuss.elastic.co/t/multiline-multi-field-input-question-newbie-here/156166/9 "2018-11-20T10:29:39Z")
Better create a field. So you can query/search/visualise specific fields and kibana. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 18, 2018, 10:29am UTC](https://discuss.elastic.co/t/multiline-multi-field-input-question-newbie-here/156166/10 "2018-12-18T10:29:42Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.