# Require suggestion for the processing multi-line logs: filebeat or logstash **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [October 18, 2018, 9:44am UTC](https://discuss.elastic.co/t/require-suggestion-for-the-processing-multi-line-logs-filebeat-or-logstash/152976 "2018-10-18T09:44:47Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![ajb](https://avatars.discourse-cdn.com/v4/letter/a/ecae2f/32.png) [@ajb](https://discuss.elastic.co/u/ajb)\ **Post date:** [October 18, 2018, 9:44am UTC](https://discuss.elastic.co/t/require-suggestion-for-the-processing-multi-line-logs-filebeat-or-logstash/152976/1 "2018-10-18T09:44:47Z")
Hi Elastic team, Im trying to process logs from liferay (a CMS application) **liferay logs daily** single file contains above 1000 lines ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/7/07fc20fb550cc42041e015fb4e4b5c9ad234b434.png) **Sample log lines inside log file** ``` 13:56:59,178 INFO [ContainerBackgroundProcessor[StandardEngine[Catalina]]][HotDeployEvent:145] Plugin tf-seller-portlet requires marketplace-portlet 13:56:59,180 INFO [ContainerBackgroundProcessor[StandardEngine[Catalina]]][PortletHotDeployListener:525] Unregistering portlets for tf-seller-portlet 13:56:59,314 INFO [ContainerBackgroundProcessor[StandardEngine[Catalina]]][PortletHotDeployListener:560] 1 portlet for tf-seller-portlet was unregistered 13:56:59,317 INFO [ContainerBackgroundProcessor[StandardEngine[Catalina]]][PluginPackageUtil:1016] Reading plugin package for tf-seller-portlet 13:05:34,755 WARN [ajp-bio-8009-exec-1][SecurityPortletContainerWrapper:623] Reject process action for https://www.abc.com/login1111 on 49 13:05:34,767 WARN [ajp-bio-8009-exec-2][SecurityPortletContainerWrapper:623] Reject process action for https://www.abc.com/login1111 on 49 13:35:36,194 ERROR [liferay/search_writer/SYSTEM_ENGINE-1][IndexAccessorImpl:336] Check Lucene directory failed for 0 org.apache.lucene.store.LockReleaseFailedException: Cannot forcefully unlock a NativeFSLock which is held by another indexer component: /opt/foss/data/lucene/0/write.lock at org.apache.lucene.store.NativeFSLock.release(NativeFSLockFactory.java:294) at org.apache.lucene.index.IndexWriter.unlock(IndexWriter.java:4654) at com.liferay.portal.search.lucene.IndexAccessorImpl._checkLuceneDir(IndexAccessorImpl.java:132) at com.liferay.portal.search.lucene.IndexAccessorImpl.(IndexAccessorImpl.java:87) at com.liferay.portal.search.lucene.LuceneHelperImpl.getIndexAccessor(LuceneHelperImpl.java:391) at com.liferay.portal.search.lucene.LuceneHelperImpl.updateDocument(LuceneHelperImpl.java:791) at com.liferay.portal.search.lucene.LuceneHelperUtil.updateDocument(LuceneHelperUtil.java:428) at com.liferay.portal.search.lucene.LuceneIndexWriter.updateDocument(LuceneIndexWriter.java:118) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:23) at java.lang.reflect.Method.invoke(Method.java:606) at com.liferay.portal.kernel.messaging.proxy.ProxyRequest.execute(ProxyRequest.java:85) at com.liferay.portal.kernel.messaging.proxy.ProxyMessageListener.receive(ProxyMessageListener.java:51) at com.liferay.portal.kernel.messaging.InvokerMessageListener.receive(InvokerMessageListener.java:72) at com.liferay.portal.kernel.messaging.ParallelDestination$1.run(ParallelDestination.java:69) at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask._runTask(ThreadPoolExecutor.java:682) at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask.run(ThreadPoolExecutor.java:593) at java.lang.Thread.run(Thread.java:745) ``` **Trying to achieve** 1. Filter the single log file into ERROR , INFO and others accordingly and process them with grok 2. Trying to use separate grok for only ERROR lines as these have stack trace 3. Output all to elastic search and also output to separate files as INFO, ERROR accordingly **Below is pipeline config** my config is not proper **Q1)** is it better to use filebeat for dividing the single log into INFO, ERROR etc.. and then process to logstash or directly input to logstash ``` input { file { path => "/home/foss/liferay/*.log" start_position => beginning codec => multiline { pattern => "^%{TIME}" negate => true what => "previous" auto_flush_interval => 5 } } } filter { grok { match => { "message" => "%{TIME:logTime}\s*%{LOGLEVEL:logLevel}\s*\[%{GREEDYDATA:logThread}\]\s*\[%{GREEDYDATA:logClassName}:%{NUMBER:logLineNumber}\]\s*%{GREEDYDATA:logMessage}" } } if "ERROR" in [logLevel] { mutate { replace => { type => "ERROR" } } grok { match => {"mesage" => "%{TIME:logTime}\s*%{LOGLEVEL:logLevel}\s*\[%{GREEDYDATA:logThread}\]\s*\[%{GREEDYDATA:logClassName}:%{NUMBER:logLineNumber}\]\s*%{GREEDYDATA:logMessage}" } } } ruby { code => " event.set('datetime', [event.get('path').split('/')[-1].split('.')[-2], event.get('logTime').split(',')[0]].join(' ')) " } date { match => ["datetime", "yyyy-MM-dd HH:mm:ss"] } } output { elasticsearch { hosts => ["localhost:9200"] # load balances between the hosts document_type => "default" # Defaults to "type" field index => "liferay" http_compression => true } if [type] == "ERROR" { stdout { file => /home/foss/proccessed/"%{type}-%{datetime}" } } ``` ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [October 18, 2018, 10:08pm UTC](https://discuss.elastic.co/t/require-suggestion-for-the-processing-multi-line-logs-filebeat-or-logstash/152976/2 "2018-10-18T22:08:13Z")
What do you expect by using filebeat? You want to remove logstash and do everything in filebeat? You have 2 outputs in your logstash config. This is not supported in Filebeat. ---
**Author:** ![ajb](https://avatars.discourse-cdn.com/v4/letter/a/ecae2f/32.png) [@ajb](https://discuss.elastic.co/u/ajb)\ **Post date:** [October 19, 2018, 3:42am UTC](https://discuss.elastic.co/t/require-suggestion-for-the-processing-multi-line-logs-filebeat-or-logstash/152976/3 "2018-10-19T03:42:11Z")
Hi steffen, Thank you for your response, So for the input part Im just trying to know how to proceed with my use-case should i go with filebeat first (whether it can separate lines according to INFO, ERROR & WARN along with forming fields ) then send it to logstash or should i use directly logstash ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [October 19, 2018, 12:35pm UTC](https://discuss.elastic.co/t/require-suggestion-for-the-processing-multi-line-logs-filebeat-or-logstash/152976/4 "2018-10-19T12:35:56Z")
Both are viable options. Filebeat splits logs by lines and has a multiline input. Somewhat similar to the file input. Why exactly you want to use Filebeat? If both Filebeat and Logstash run on the same host, then there is no need for Filebeat. If you want to offload processing/parsing to another host, use Filebeat + Logstash. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 16, 2018, 12:36pm UTC](https://discuss.elastic.co/t/require-suggestion-for-the-processing-multi-line-logs-filebeat-or-logstash/152976/5 "2018-11-16T12:36:00Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.