# Handling multiline with filebeat **URL:** https://discuss.elastic.co/t/handling-multiline-with-filebeat/301000 **Category:** Beats **Tags:** filebeat **Created:** [March 29, 2022, 6:44pm UTC](https://discuss.elastic.co/t/handling-multiline-with-filebeat/301000 "2022-03-29T18:44:26Z") **Posts on this page:** 9 **Page:** 1
### Author: ![stecino](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@stecino](https://discuss.elastic.co/u/stecino) #### Post date: [March 29, 2022, 6:44pm UTC](https://discuss.elastic.co/t/handling-multiline-with-filebeat/301000/1 "2022-03-29T18:44:26Z")
Hello, I have the following snippet, and I am trying to capture all of it as a multiline \<\< JESI\>\> [ERROR] [TIME:29 Mar 2022 04:34:53][Tid:OUTBOUND\_RECOVERY\_01339]Exception @[NodeId=6;Element=Channel@amdocs/eai/adaptor/outbound/QueryAccountByCustID.xml]Error=Exception @[NodeId=6;Element=Channel@amdocs/eai/adaptor/outbound/QueryAccountByCustID.xml]Error=amdocs.ecommerce.esi.exceptions.CommunicationException: [LoginFault [ApiFault exceptionCode='INVALID\_LOGIN' exceptionMessage='Invalid username, password, security token; or user locked out.' ] ] amdocs.ecommerce.esi.exceptions.CommunicationException: [LoginFault [ApiFault exceptionCode='INVALID\_LOGIN' exceptionMessage='Invalid username, password, security token; or user locked out.' ] ] ``` at amdocs.ecommerce.esi.proxy.SFDCProxy.connect(SFDCProxy.java:206) at amdocs.ecommerce.esi.handlers.channels.SFDCProxyChannel.publish(SFDCProxyChannel.java:78) at amdocs.ecommerce.esi.utils.flow.invoke.Channel.doExecute(Channel.java:305) at amdocs.ecommerce.esi.utils.flow.BaseElement.execute(BaseElement.java:216) at amdocs.ecommerce.esi.utils.flow.ProcessFlow.execute(ProcessFlow.java:1004) at amdocs.ecommerce.esi.utils.flow.ProcessFlowEngine.execute(ProcessFlowEngine.java:638) at amdocs.ecommerce.esi.utils.flow.FLOW.doExecute(FLOW.java:339) at amdocs.ecommerce.esi.utils.flow.BaseElement.execute(BaseElement.java:216) at amdocs.ecommerce.esi.utils.flow.TRY.doExecute(TRY.java:81) at amdocs.ecommerce.esi.utils.flow.BaseElement.execute(BaseElement.java:216) at amdocs.ecommerce.esi.utils.flow.TRYCATCH.doExecute(TRYCATCH.java:218) at amdocs.ecommerce.esi.utils.flow.BaseElement.execute(BaseElement.java:216) at amdocs.ecommerce.esi.utils.flow.ELSE.doExecute(ELSE.java:76) at amdocs.ecommerce.esi.utils.flow.BaseElement.execute(BaseElement.java:216) at amdocs.ecommerce.esi.utils.flow.IF.doExecute(IF.java:147) at amdocs.ecommerce.esi.utils.flow.BaseElement.execute(BaseElement.java:216) at amdocs.ecommerce.esi.utils.flow.ProcessFlow.execute(ProcessFlow.java:1004) at amdocs.ecommerce.esi.utils.flow.ProcessFlowEngine.execute(ProcessFlowEngine.java:677) at amdocs.ecommerce.esi.eai.api.flows.sub.GenericProcessFlowHandler.executeFlow(GenericProcessFlowHandler.java:51) ``` ---
### Author: ![sholzhauer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sholzhauer/32/110282_2.png) [@sholzhauer](https://discuss.elastic.co/u/sholzhauer) #### Post date: [March 30, 2022, 7:04am UTC](https://discuss.elastic.co/t/handling-multiline-with-filebeat/301000/2 "2022-03-30T07:04:47Z")
You can use a multiline configuration: ```auto - type: log paths: - "/var/log/application.log" ignore_older: 1h fields_under_root: true multiline: pattern: '^<< JESI>>' negate: true match: after timeout: 5s ``` ---
### Author: ![stecino](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@stecino](https://discuss.elastic.co/u/stecino) #### Post date: [April 7, 2022, 2:54am UTC](https://discuss.elastic.co/t/handling-multiline-with-filebeat/301000/3 "2022-04-07T02:54:20Z")
Thanks for the reply. I went ahead an implemented it, but it seems now that events are appended to each other as one document in Elasticsearch. Could I be doing something wrong? I am sending the events from filebeat to kafka, then I read kafka topic and write it to ES ```auto << JESI>> [INFO] [TIME:6 Apr 2022 17:55:19][Tid:ADAPTOR_POST_SUB_01_05323][ID=20220406175519_239735_dfwlnpkgjesi-01_40820_bfebb84e-29b8-4666-afc6-001acc567c7c;Flow=amdocs/eai/adaptor/outbound/OUTBOUND_UPSERT_CUST.xml]callPrimaryMCByRepIdService from Input : 1 << JESI>> [INFO] [TIME:6 Apr 2022 17:55:19][Tid:ADAPTOR_POST_SUB_01_05323][ID=20220406175519_239735_dfwlnpkgjesi-01_40820_bfebb84e-29b8-4666-afc6-001acc567c7c;Flow=amdocs/eai/adaptor/outbound/OUTBOUND_UPSERT_CUST.xml]primaryMcID from Input : 195280 << JESI>> [INFO] [TIME:6 Apr 2022 17:55:19][Tid:ADAPTOR_POST_SUB_01_05323][ID=20220406175519_239736_dfwlnpkgjesi-01_40820_4a1ee972-dbee-4a4e-bb6f-7289e7879431;Flow=amdocs/eai/adaptor/outbound/QueryUserByRepID.xml]Query = SELECT Id FROM User where Dex_kGen_Employee_ID__c = '195280' LIMIT 1 << JESI>> [INFO] [TIME:6 Apr 2022 17:55:19][Tid:ADAPTOR_POST_SUB_01_05323][ID=20220406175519_239736_dfwlnpkgjesi-01_40820_4a1ee972-dbee-4a4e-bb6f-7289e7879431;Flow=amdocs/eai/adaptor/outbound/QueryUserByRepID.xml]Query = [SELECT Id FROM User where Dex_kGen_Employee_ID__c = '195280' LIMIT 1] java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor172.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at amdocs.ecommerce.esi.proxy.SFDCProxy.connect(SFDCProxy.java:198) at amdocs.ecommerce.esi.handlers.channels.SFDCProxyChannel.publish(SFDCProxyChannel.java:78) at amdocs.ecommerce.esi.utils.flow.invoke.Channel.doExecute(Channel.java:305) at amdocs.ecommerce.esi.utils.flow.BaseElement.execute(BaseElement.java:216) at amdocs.ecommerce.esi.utils.flow.ProcessFlow.execute(ProcessFlow.java:1004) at amdocs.ecommerce.esi.utils.flow.ProcessFlowEngine.execute(ProcessFlowEngine.java:638) at amdocs.ecommerce.esi.utils.flow.FLOW.doExecute(FLOW.java:339) at amdocs.ecommerce.esi.utils.flow.BaseElement.execute(BaseElement.java:216) at amdocs.ecommerce.esi.utils.flow.THEN.doExecute(THEN.java:79) at amdocs.ecommerce.esi.utils.flow.BaseElement.execute(BaseElement.java:216) at amdocs.ecommerce.esi.utils.flow.IF.doExecute(IF.java:141) at amdocs.ecommerce.esi.utils.flow.BaseElement.execute(BaseElement.java:216) ``` ---
### Author: ![stecino](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@stecino](https://discuss.elastic.co/u/stecino) #### Post date: [April 7, 2022, 3:00am UTC](https://discuss.elastic.co/t/handling-multiline-with-filebeat/301000/4 "2022-04-07T03:00:06Z")
Also, is there a way for me to drop events via filebeat, if an event with specific string match is found? ---
### Author: ![sholzhauer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sholzhauer/32/110282_2.png) [@sholzhauer](https://discuss.elastic.co/u/sholzhauer) #### Post date: [April 7, 2022, 2:34pm UTC](https://discuss.elastic.co/t/handling-multiline-with-filebeat/301000/5 "2022-04-07T14:34:40Z")
How do you mean, appended in kibana? Can you provide an example (json export) ---
### Author: ![stecino](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@stecino](https://discuss.elastic.co/u/stecino) #### Post date: [April 7, 2022, 5:28pm UTC](https://discuss.elastic.co/t/handling-multiline-with-filebeat/301000/6 "2022-04-07T17:28:39Z")
```auto { "_index": "kgjesi-2022.04.07", "_type": "kg-jesi-logs-st1", "_id": "AYAFBDR9l1hx9GHPv1PX", "_version": 1, "_score": 2, "_source": { "@timestamp": "2022-04-07T17:13:08.782Z", "beat": { "hostname": "dfwlnpkgjesi-01", "name": "dfwlnpkgjesi-01" }, "input_type": "log", "message": "<< JESI>> [ERROR] [TIME:7 Apr 2022 12:13:07][Tid:ADAPTOR_POST_SUB_01_06319]Exception @[NodeId=6;Element=Channel@amdocs/eai/adaptor/outbound/QueryAccountByCustID.xml]Error=Exception @[NodeId=6;Element=Channel@amdocs/eai/adaptor/outbound/QueryAccountByCustID.xml]Error=amdocs.ecommerce.esi.exceptions.CommunicationException: [LoginFault [ApiFault exceptionCode='INVALID_LOGIN'\n exceptionMessage='Invalid username, password, security token; or user locked out.'\n]\n]\n\n<< JESI>> [ERROR] [TIME:7 Apr 2022 12:13:07][Tid:ADAPTOR_POST_SUB_01_06319]\nFlow:[Id=20220407121257_103850_dfwlnpkgjesi-01_40820_bd76cc0f-9412-40bb-894e-f28f5fa31478;name=amdocs/eai/adaptor/outbound/QueryAccountByCustID.xml;nodeId=6;name=]\nMessage:Execute - Process (20220407121257_103850_dfwlnpkgjesi-01_40820_bd76cc0f-9412-40bb-894e-f28f5fa31478-amdocs/eai/adaptor/outbound/QueryAccountByCustID.xml) Terminated with Error!!!amdocs.ecommerce.esi.exceptions.CommunicationException: [LoginFault [ApiFault exceptionCode='INVALID_LOGIN'\n exceptionMessage='Invalid username, password, security token; or user locked out.'\n]\n]\n\n<< JESI>> [ERROR] [TIME:7 Apr 2022 12:13:07][Tid:ADAPTOR_POST_SUB_01_06319]Exception @[NodeId=6.2.1.1.8.1.1.1.2;Element=FLOW@amdocs/eai/adaptor/outbound/OUTBOUND_UPSERT_CUST.xml]Error=Exception @[NodeId=6.2.1.1.8.1.1.1.2;Element=FLOW@amdocs/eai/adaptor/outbound/OUTBOUND_UPSERT_CUST.xml]Error=amdocs.ecommerce.esi.exceptions.CommunicationException: [LoginFault [ApiFault exceptionCode='INVALID_LOGIN'\n exceptionMessage='Invalid username, password, security token; or user locked out.'\n]\n]\n\n<< JESI>> [ERROR] [TIME:7 Apr 2022 12:13:07][Tid:ADAPTOR_POST_SUB_01_06319]Exception @[NodeId=6.2.1.1.8.1.1.1;Element=TRY@amdocs/eai/adaptor/outbound/OUTBOUND_UPSERT_CUST.xml]Error=Exception @[NodeId=6.2.1.1.8.1.1.1;Element=TRY@amdocs/eai/adaptor/outbound/OUTBOUND_UPSERT_CUST.xml]Error=amdocs.ecommerce.esi.exceptions.CommunicationException: [LoginFault [ApiFault exceptionCode='INVALID_LOGIN'\n exceptionMessage='Invalid username, password, security token; or user locked out.'\n]\n]\n\n<< JESI>> [ERROR] [TIME:7 Apr 2022 12:13:07][Tid:ADAPTOR_POST_SUB_01_06319]\nFlow:[Id=20220407121257_103849_dfwlnpkgjesi-01_40820_5baefb47-c2c4-487d-a753-c9af60335f6a;name=amdocs/eai/adaptor/outbound/OUTBOUND_UPSERT_CUST.xml;nodeId=6.2.1.1.8.1.1.2.1;name=]\nMessage:Execute - [handleRetrieveAccountContactException] Java code Failed!\n<< JESI>> [ERROR] [TIME:7 Apr 2022 12:13:07][Tid:ADAPTOR_POST_SUB_01_06319]Exception @[NodeId=6.2.1.1.8.1.1.2.1;Element=CODE@amdocs/eai/adaptor/outbound/OUTBOUND_UPSERT_CUST.xml]Error=Exception @[NodeId=6.2.1.1.8.1.1.2.1;Element=CODE@amdocs/eai/adaptor/outbound/OUTBOUND_UPSERT_CUST.xml]Error=Code=JESIERR-4;Description=[LoginFault [ApiFault exceptionCode='INVALID_LOGIN'\n exceptionMessage='Invalid username, password, security token; or user locked out.'\n]\n]\n\n<< JESI>> [ERROR] [TIME:7 Apr 2022 12:13:07][Tid:ADAPTOR_POST_SUB_01_06319]Exception @[NodeId=6.2.1.1.8.1.1.2;Element=CATCH@amdocs/eai/adaptor/outbound/OUTBOUND_UPSERT_CUST.xml]Error=Exception @[NodeId=6.2.1.1.8.1.1.2;Element=CATCH@amdocs/eai/adaptor/outbound/OUTBOUND_UPSERT_CUST.xml]Error=Code=JESIERR-4;Description=[LoginFault [ApiFault exceptionCode='INVALID_LOGIN'\n exceptionMessage='Invalid username, password, security token; or user locked out.'\n]\n]\n\n<< JESI>> [ERROR] [TIME:7 Apr 2022 12:13:07][Tid:ADAPTOR_POST_SUB_01_06319]Exception @[NodeId=6.2.1.1.8.1.1;Element=TRYCATCH@amdocs/eai/adaptor/outbound/OUTBOUND_UPSERT_CUST.xml]Error=Exception @[NodeId=6.2.1.1.8.1.1;Element=TRYCATCH@amdocs/eai/adaptor/outbound/OUTBOUND_UPSERT_CUST.xml]Error=Code=JESIERR-4;Description=[LoginFault [ApiFault exceptionCode='INVALID_LOGIN'\n exceptionMessage='Invalid username, password, security token; or user locked out.'\n]\n]\n\n<< JESI>> [ERROR] [TIME:7 Apr 2022 12:13:07][Tid:ADAPTOR_POST_SUB_01_06319]Exception @[NodeId=6.2.1.1.8.1;Element=IF-THEN@amdocs/eai/adaptor/outbound/OUTBOUND_UPSERT_CUST.xml]Error=Exception @[NodeId=6.2.1.1.8.1;Element=IF-THEN@amdocs/eai/adaptor/outbound/OUTBOUND_UPSERT_CUST.xml]Error=Code=JESIERR-4;Description=[LoginFault [ApiFault exceptionCode='INVALID_LOGIN'\n exceptionMessage='Invalid username, password, security token; or user locked out.'\n]\n]\n\n<< JESI>> [ERROR] [TIME:7 Apr 2022 12:13:07][Tid:ADAPTOR_POST_SUB_01_06319]Exception @[NodeId=6.2.1.1.8;Element=IF@amdocs/eai/adaptor/outbound/OUTBOUND_UPSERT_CUST.xml]Error=Exception @[NodeId=6.2.1.1.8;Element=IF@amdocs/eai/adaptor/outbound/OUTBOUND_UPSERT_CUST.xml]Error=Code=JESIERR-4;Description=[LoginFault [ApiFault exceptionCode='INVALID_LOGIN'\n exceptionMessage='Invalid username, password, security token; or user locked out.'\n]\n]\n\n<< JESI>> [ERROR] [TIME:7 Apr 2022 12:13:07][Tid:ADAPTOR_POST_SUB_01_06319]Exception @[NodeId=6.2.1.1;Element=TRY@amdocs/eai/adaptor/outbound/OUTBOUND_UPSERT_CUST.xml]Error=Exception @[NodeId=6.2.1.1;Element=TRY@amdocs/eai/adaptor/outbound/OUTBOUND_UPSERT_CUST.xml]Error=Code=JESIERR-4;Description=[LoginFault [ApiFault exceptionCode='INVALID_LOGIN'\n exceptionMessage='Invalid username, password, security token; or user locked out.'\n]\n]\n\n<< JESI>> [ERROR] [TIME:7 Apr 2022 12:13:07][Tid:ADAPTOR_POST_SUB_01_06319]\nFlow:[Id=20220407121257_103849_dfwlnpkgjesi-01_40820_5baefb47-c2c4-487d-a753-c9af60335f6a;name=amdocs/eai/adaptor/outbound/OUTBOUND_UPSERT_CUST.xml;nodeId=6.2.1.2.1;name=]\nMessage:Execute - [handleExecuteException] Java code Failed!\n<< JESI>> [ERROR] [TIME:7 Apr 2022 12:13:07][Tid:ADAPTOR_POST_SUB_01_06319]Exception @[NodeId=6.2.1.2.1;Element=CODE@amdocs/eai/adaptor/outbound/OUTBOUND_UPSERT_CUST.xml]Error=Exception @[NodeId=6.2.1.2.1;Element=CODE@amdocs/eai/adaptor/outbound/OUTBOUND_UPSERT_CUST.xml]Error=Code=JESIERR-1;Description=amdocs.ecommerce.common.exceptions.SystemException:\n<< JESI>> [ERROR] [TIME:7 Apr 2022 12:13:07][Tid:ADAPTOR_POST_SUB_01_06319]Exception @[NodeId=6.2.1.2;Element=CATCH@amdocs/eai/adaptor/outbound/OUTBOUND_UPSERT_CUST.xml]Error=Exception @[NodeId=6.2.1.2;Element=CATCH@amdocs/eai/adaptor/outbound/OUTBOUND_UPSERT_CUST.xml]Error=Code=JESIERR-1;Description=amdocs.ecommerce.common.exceptions.SystemException:\n<< JESI>> [ERROR] [TIME:7 Apr 2022 12:13:07][Tid:ADAPTOR_POST_SUB_01_06319]Exception @[NodeId=6.2.1;Element=TRYCATCH@amdocs/eai/adaptor/outbound/OUTBOUND_UPSERT_CUST.xml]Error=Exception @[NodeId=6.2.1;Element=TRYCATCH@amdocs/eai/adaptor/outbound/OUTBOUND_UPSERT_CUST.xml]Error=Code=JESIERR-1;Description=amdocs.ecommerce.common.exceptions.SystemException:\n<< JESI>> [ERROR] [TIME:7 Apr 2022 12:13:07][Tid:ADAPTOR_POST_SUB_01_06319]Exception @[NodeId=6.2;Element=IF-ELSE@amdocs/eai/adaptor/outbound/OUTBOUND_UPSERT_CUST.xml]Error=Exception @[NodeId=6.2;Element=IF-ELSE@amdocs/eai/adaptor/outbound/OUTBOUND_UPSERT_CUST.xml]Error=Code=JESIERR-1;Description=amdocs.ecommerce.common.exceptions.SystemException:\n<< JESI>> [ERROR] [TIME:7 Apr 2022 12:13:07][Tid:ADAPTOR_POST_SUB_01_06319]Exception @[NodeId=6;Element=IF@amdocs/eai/adaptor/outbound/OUTBOUND_UPSERT_CUST.xml]Error=Exception @[NodeId=6;Element=IF@amdocs/eai/adaptor/outbound/OUTBOUND_UPSERT_CUST.xml]Error=Code=JESIERR-1;Description=amdocs.ecommerce.common.exceptions.SystemException:\n<< JESI>> [ERROR] [TIME:7 Apr 2022 12:13:07][Tid:ADAPTOR_POST_SUB_01_06319]\nFlow:[Id=20220407121257_103849_dfwlnpkgjesi-01_40820_5baefb47-c2c4-487d-a753-c9af60335f6a;name=amdocs/eai/adaptor/outbound/OUTBOUND_UPSERT_CUST.xml;nodeId=6.2.1.2.1;name=]\nMessage:Execute - Process (20220407121257_103849_dfwlnpkgjesi-01_40820_5baefb47-c2c4-487d-a753-c9af60335f6a-amdocs/eai/adaptor/outbound/OUTBOUND_UPSERT_CUST.xml) Terminated with Error!!!Code=JESIERR-1;Description=amdocs.ecommerce.common.exceptions.SystemException:\n<< JESI>> [ERROR] [TIME:7 Apr 2022 12:13:07][Tid:ADAPTOR_POST_SUB_01_06319][EG:Name=ADAPTOR_POST_SUB_01_06;Type=JDBC(1649351587317)][InterfaceException] Code[JESIERR-1] Description[amdocs.ecommerce.common.exceptions.SystemException:] StackTrace[amdocs.ecommerce.esi.exceptions.SubscribeException: \n\tat amdocs.ecommerce.esi.handlers.SubscribeMessageManagerImpl.postSubscribe(SubscribeMessageManagerImpl.java:704)\n\tat amdocs.ecommerce.esi.api.EJBInterface.EJBInterfaceAPIBean.postSubscribe(EJBInterfaceAPIBean.java:265)\n\tat amdocs.ecommerce.esi.api.EJBInterface.EJBInterfaceAPIOfflineImpl.postSubscribe(EJBInterfaceAPIOfflineImpl.java:970)\n\tat amdocs.ecommerce.esi.handlers.events.SubscribeTimerEventHandler.callFlow(SubscribeTimerEventHandler.java:227)\n\tat amdocs.ecommerce.esi.handlers.events.SubscribeTimerEventHandler.doBeanExecute(SubscribeTimerEventHandler.java:282)\n\tat amdocs.ecommerce.esi.api.EventGeneratorManager.EventGeneratorManagerBean.executeOnEventRecord(EventGeneratorManagerBean.java:163)\n\tat amdocs.ecommerce.esi.api.EventGeneratorManager.EventGeneratorManagerOfflineImpl.executeOnEventRecord(EventGeneratorManagerOfflineImpl.java:2239)\n\tat amdocs.ecommerce.esi.handlers.events.JDBCPublishTimerHandler.executeOne(JDBCPublishTimerHandler.java:1312)\n\tat amdocs.ecommerce.esi.handlers.events.JDBCPublishTimerHandler.execute(JDBCPublishTimerHandler.java:1225)\n\tat amdocs.ecommerce.esi.handlers.events.BaseGenericEvent.processEvent(BaseGenericEvent.java:651)\n\tat amdocs.ecommerce.esi.api.EventGeneratorManager.EventGeneratorManagerBean.processEvent(EventGeneratorManagerBean.java:227)\n\tat amdocs.ecommerce.esi.api.EventGeneratorManager.EventGeneratorManagerOfflineImpl.processEvent(EventGeneratorManagerOfflineImpl.java:112)\n\tat amdocs.ecommerce.esi.handlers.events.BaseGenericEvent.callEGBean(BaseGenericEvent.java:821)\n\tat amdocs.ecommerce.esi.handlers.events.BaseTimerEvent.onTimer(BaseTimerEvent.java:502)\n\tat amdocs.ecommerce.esi.handlers.events.BaseTimerEvent.run(BaseTimerEvent.java:342)\n\tat amdocs.ecommerce.esi.handlers.events.GenericJavaTimerTask.run(Ge\n<< JESI>> [INFO] [TIME:7 Apr 2022 12:13:08][Tid:BANK_ADP_POST_SUB_01_00388][ID=20220407121308_103867_dfwlnpkgjesi-01_40820_d0ee945e-71c1-4305-ad17-4ca6a63d8b19;Flow=amdocs/eai/adaptor/outbound/authrequest/NEW_ORDER_AUTH.xml]... Checking CHASE_SERVER = 1.", "offset": 15902514, "source": "/kgnusr/rhd/prdaweb/jesiPRD_PUB/jesidomain/logs/JESI_amdocsOnline.log", "tags": [ "jesiPRD_PUB" ], "type": "kg-jesi-logs-st1", "@version": "1", "kafka": { "msg_size": 10806, "topic": "kg-jesi-logs-st1", "consumer_group": "logstash", "partition": 7, "key": null }, "platform": "kgjesi" }, "fields": { "@timestamp": [ 1649351588782 ] } } ``` You can see the new lines in the message field ---
### Author: ![sholzhauer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sholzhauer/32/110282_2.png) [@sholzhauer](https://discuss.elastic.co/u/sholzhauer) #### Post date: [April 12, 2022, 1:02pm UTC](https://discuss.elastic.co/t/handling-multiline-with-filebeat/301000/7 "2022-04-12T13:02:10Z")
Could you perhaps share your Filebeat and logstash configurations ---
### Author: ![stecino](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@stecino](https://discuss.elastic.co/u/stecino) #### Post date: [April 12, 2022, 5:05pm UTC](https://discuss.elastic.co/t/handling-multiline-with-filebeat/301000/8 "2022-04-12T17:05:12Z")
Here is the filebeat config ```auto filebeat: prospectors: - paths: - /kgnusr/rhd/prdcweb/prdAcrm1_01/WLServers/config/logs/online.log document_type: kg-acrm-logs-st1 tags: ["prdAcrm1_01"] multiline.pattern: '^<>' multiline.negate: true multiline.match: after exclude_files: ['\.gz$'] - paths: - /kgnusr/rhd/prdcweb/prdAcrm2_01/WLServers/config/logs/online.log document_type: kg-acrm-logs-st1 tags: ["prdAcrm2_01"] multiline.pattern: '^<>' multiline.negate: true multiline.match: after exclude_files: ['\.gz$'] - paths: - /kgnusr/rhd/prdcweb/prdAcrm3_01/WLServers/config/logs/online.log document_type: kg-acrm-logs-st1 tags: ["prdAcrm3_01"] multiline.pattern: '^<>' multiline.negate: true multiline.match: after exclude_files: ['\.gz$'] - paths: - /kgnusr/rhd/prdcweb/prdAcrm4_01/WLServers/config/logs/online.log document_type: kg-acrm-logs-st1 tags: ["prdAcrm4_01"] multiline.pattern: '^<>' multiline.negate: true multiline.match: after exclude_files: ['\.gz$'] close_older: 480m logging: level: debug # enable file rotation with default configuration to_files: true # do not log to syslog to_syslog: false files: path: /filebeat-5.0.0-alpha3-linux-x64/var/log name: filebeat.log keepfiles: 7 output: kafka: hosts: ["blah:9092"] topics: - topic: '%{[type]}' use_type: true worker: 3 compression: none ``` And the logstash ```auto input { kafka { zk_connect => "blah:2181" white_list => "kg-acrm-logs-st1" decorate_events => true codec => json } } filter { mutate { add_field => { "platform" => "kgacrm" } } if "prdAcrm" in [tags] { if [type] == "kg-acrm-logs-st1" { mutate { add_field => { "kg_host" => "%{[beat][hostname]}" } remove_field => ["[beat]"] } } date { match => ["timestamp", "MMM dd HH:mm:ss","MMM d HH:mm:ss","MMM dd yyyy HH:mm:ss","MMM d yyyy HH:mm:ss"] target => "@timestamp" timezone => "America/Chicago" } } } output { elasticsearch { hosts => ["blah:9200", "blah2:9200"] index => "%{platform}-%{+YYYY.MM.dd}" } stdout { codec => rubydebug } } ``` ---
### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system) #### Post date: [May 10, 2022, 7:05pm UTC](https://discuss.elastic.co/t/handling-multiline-with-filebeat/301000/9 "2022-05-10T19:05:13Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.