Ignore Error Messages

Elastic Stack Logstash
1

Team,

While analyzing system log files in Linux, I came up with the below entry.
How do we ignore such huge blocks in Logstash ??

I believe if we do not ignore this, it will be considered as a log and we will get incorrect data on the Kibana Dashboard.

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0
ND-INFO  : Staring BCIAgent for Netdiagnostics ...
BCIMode Info : NDControlConnModel.isndcHostAtStartup false NDControlConnModel.isinstanceAtStartUp= false  NDControlConnModel.isndlPortAtStartUp = true
time is included.

Starting Interoperability Server:  9.3

2018-08-20 23:57:03,757:INFO   ::main: Initializing data cache...                                   [system]: YCPSystem                
2018-08-20 23:58:46,391:WARN   ::main: PROMOTION_RULES_CACHE registered                             [system]: YPMPromotionRulesCacheManager
2018-08-20 23:58:46,491:WARN   ::main: PRICELIST_HEADER_CACHE registered                            [system]: YPMPricingHeaderCacheManager
2018-08-20 23:58:46,494:INFO   ::main: PricingRule and PriceListHeader Cache Loading has been skipped [system]: YPMExtensionPointInitializer
2018-08-20 23:59:31,650:ERROR  ::main: Could not start adapter successfully for serverName: DataPumpAndImpSyncServer [system]: IntegrationAdapter       
2018-08-20 23:59:31,652:ERROR  ::main: [1534827571651]Cannot Resolve Variable.                      [system]: IntegrationAdapter       
2018-08-20 23:59:31,655:ERRORDTL::main: [1534827571651]<?xml version="1.0" encoding="UTF-8"?>
<Errors>
    <Error ErrorCode="YCP75_00014"
        ErrorDescription="Cannot Resolve Variable." ErrorRelatedMoreInfo="">
        <Attribute Name="ErrorCode" Value="YCP75_00014"/>
        <Attribute Name="ErrorDescription" Value="Cannot Resolve Variable."/>
        <Attribute Name="Variable" Value="KOHLS_POC_STORE_ORDER_SYNC_Q"/>
        <Stack>com.yantra.yfc.util.YFCException
	at com.yantra.yfc.util.YFCVariableResolver$DefaultPropertyLookup.resolveVariable(YFCVariableResolver.java:39)
	at com.yantra.yfc.util.YFCVariableResolver.getResolvedString(YFCVariableResolver.java:114)
	at com.yantra.yfc.util.YFCVariableResolver.getResolvedString(YFCVariableResolver.java:96)
	at com.yantra.yfc.util.YFCVariableResolver.getResolvedString(YFCVariableResolver.java:93)
	at com.yantra.yfc.util.YFCVariableResolver.getResolvedString(YFCVariableResolver.java:90)
	at com.yantra.interop.services.jms.JMSTransportProperties.setMainQueueName(JMSTransportProperties.java:98)
	at com.yantra.interop.services.jms.JMSTransportProperties.init(JMSTransportProperties.java:468)
	at com.yantra.interop.services.TransportPropsFactory.getTransportProps(TransportPropsFactory.java:56)
	at com.yantra.interop.services.TransportPropsFactory.getTransportProps(TransportPropsFactory.java:47)
	at com.yantra.integration.configurator.ConfigurationHelper.getTransportProps(ConfigurationHelper.java:222)
	at com.yantra.integration.configurator.ConfigurationHelper.readConfiguration(ConfigurationHelper.java:166)
	at com.yantra.integration.adapter.IntegrationAdapter.initConfig(IntegrationAdapter.java:141)
	at com.yantra.integration.adapter.IntegrationAdapter.start(IntegrationAdapter.java:381)
	at com.yantra.integration.adapter.IntegrationAdapter.main(IntegrationAdapter.java:515)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.yantra.integration.adapter.IntegrationAdapterLauncher.launch(IntegrationAdapterLauncher.java:134)
	at com.yantra.integration.adapter.IntegrationAdapterLauncher.main(IntegrationAdapterLauncher.java:61)
</Stack>
2

Join the lines of that message into a single event (use the multiline codec if you're using the file input). If you really want to drop such events you can use the drop filter wrapped in a conditional (see example in the drop plugin documentation).

3

Thanks for your response.

I am currently using filebeat as my input and in that case I cannot use"multiline codec"..

I have gone through the drop filter and just want to understand if the option can only be used for a single line or can we use it for a block ??

How can I use it for the above block of Junk Data that was posted earlier ??

Thank You

4

I am currently using filebeat as my input and in that case I cannot use"multiline codec"..

No, you should use Filebeat's multiline feature.

I have gone through the drop filter and just want to understand if the option can only be used for a single line or can we use it for a block ??

The drop filter acts on events. Events may contain one or more lines.