# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=769 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 770 --- ## [Logstash not able to handle large volume of logs](https://discuss.elastic.co/t/logstash-not-able-to-handle-large-volume-of-logs/130228)
**Author:** [@ashif14](https://discuss.elastic.co/u/ashif14)\ **Replies:** 3\ **Last updated:** [May 2, 2018, 10:37am UTC](https://discuss.elastic.co/t/logstash-not-able-to-handle-large-volume-of-logs/130228 "2018-05-02T10:37:51Z")
We have two 32GB logstash servers, and we are receiving almost 3-4millions events per hours in logstash. Also we have queue depth of 4GB each. Still it's not able to process these logs, and getting JVM out of heap excep… --- ## [Multiline support in logstash](https://discuss.elastic.co/t/multiline-support-in-logstash/129995)
**Author:** [@sonusr](https://discuss.elastic.co/u/sonusr)\ **Replies:** 2\ **Last updated:** [May 2, 2018, 10:30am UTC](https://discuss.elastic.co/t/multiline-support-in-logstash/129995 "2018-05-02T10:30:08Z")
HI, Is it possible to send log4j multi-line logs as single event from logstash without filebeat? I tried the following multiline codec in logstash conf file and no luck input { file { path =\> "/var/log/mylog.log" c… --- ## [ELK integration with ITSM](https://discuss.elastic.co/t/elk-integration-with-itsm/128876)
**Author:** [@Gayu\_Ravi](https://discuss.elastic.co/u/Gayu_Ravi)\ **Replies:** 5\ **Last updated:** [May 2, 2018, 9:39am UTC](https://discuss.elastic.co/t/elk-integration-with-itsm/128876 "2018-05-02T09:39:18Z")
Hi I want to integrate my ITSM tool like JIRA,BMC remedy,ServiceNow with ELK to load my ticket info. In Splunk few connectors are there for this integration. Does any similar connectors available in ELK? Anyother idea t… --- ## [SOAP web services Elasticsearch](https://discuss.elastic.co/t/soap-web-services-elasticsearch/129985)
**Author:** [@Patricie](https://discuss.elastic.co/u/Patricie)\ **Replies:** 2\ **Last updated:** [May 2, 2018, 9:30am UTC](https://discuss.elastic.co/t/soap-web-services-elasticsearch/129985 "2018-05-02T09:30:03Z")
I'm new to Elasticsearch and I want to know if it is a suitable solution for our team. We need to log our SOAP requests and responses (they're in XML format). We work with .NET on the server side (we're just maintaining… --- ## [@timestamp problem](https://discuss.elastic.co/t/timestamp-problem/130213)
**Author:** [@chenzilong444](https://discuss.elastic.co/u/chenzilong444)\ **Replies:** 0\ **Last updated:** [May 2, 2018, 9:26am UTC](https://discuss.elastic.co/t/timestamp-problem/130213 "2018-05-02T09:26:01Z")
i know @timestamp is Upload time but @timestamp is utc and my timezone is +08:00 why it know my timezone --- ## [Persistent queue breaks UTF-8 international language](https://discuss.elastic.co/t/persistent-queue-breaks-utf-8-international-language/130175)
**Author:** [@ilpc](https://discuss.elastic.co/u/ilpc)\ **Replies:** 2\ **Last updated:** [May 2, 2018, 8:46am UTC](https://discuss.elastic.co/t/persistent-queue-breaks-utf-8-international-language/130175 "2018-05-02T08:46:48Z")
my.conf: input { file { path =\> \[ "/var/log/logstash/input.log" \] } } output { file { path =\> "/var/log/logstash/output.log" } } input.log: 1가2나3다4 It works fine on memory queue. logstash.yml: path.data: /var/… --- ## [How to create two kv filter](https://discuss.elastic.co/t/how-to-create-two-kv-filter/129580)
**Author:** [@lcguy](https://discuss.elastic.co/u/lcguy)\ **Replies:** 21\ **Last updated:** [May 2, 2018, 8:24am UTC](https://discuss.elastic.co/t/how-to-create-two-kv-filter/129580 "2018-05-02T08:24:14Z")
Hi, I have two syslogs input feeds from two sources. One is coming with comma and another one is with space. So, if I configure only with udp 5514, how do I split these two inputs using two filters as below. filter { … --- ## [SNMPTrap input garbage](https://discuss.elastic.co/t/snmptrap-input-garbage/129318)
**Author:** [@jasonyates](https://discuss.elastic.co/u/jasonyates)\ **Replies:** 1\ **Last updated:** [May 2, 2018, 7:57am UTC](https://discuss.elastic.co/t/snmptrap-input-garbage/129318 "2018-05-02T07:57:32Z")
I appear to have issues with receiving MAC addresses via SNMP Trap from a Cisco WLC, they end up as garbage in the resulting JSON. "cldcClientMacAddress":"˜\\u0000Æ\>Š," I have verified several things based on similar po… --- ## [Backspace(\\b) doesn't work with the gsub filter](https://discuss.elastic.co/t/backspace-b-doesnt-work-with-the-gsub-filter/130191)
**Author:** [@jvayao](https://discuss.elastic.co/u/jvayao)\ **Replies:** 2\ **Last updated:** [May 2, 2018, 7:07am UTC](https://discuss.elastic.co/t/backspace-b-doesnt-work-with-the-gsub-filter/130191 "2018-05-02T07:07:40Z")
Version: 6.x Operating System: Any Backspace(\\b) doesn't work with the gsub filter: Config: mutate { gsub =\> \[ "Data","\\u0000","", "Data","\\u0001","", "Data","\\u0002","", "Data","\\u0003","", "Data","\\u0004… --- ## [Nested grok on custom syslogs of vernish servers](https://discuss.elastic.co/t/nested-grok-on-custom-syslogs-of-vernish-servers/130137)
**Author:** [@Ferdous\_Shibly](https://discuss.elastic.co/u/Ferdous_Shibly)\ **Replies:** 3\ **Last updated:** [May 2, 2018, 6:25am UTC](https://discuss.elastic.co/t/nested-grok-on-custom-syslogs-of-vernish-servers/130137 "2018-05-02T06:25:44Z")
Hi, I am trying to process custom access logs of vernish servers. The log format is: %h %t %r %\>s %b %{resp.http.X-Cache}V %{req.http.user-agent}V %{req.http.referer}V %{geoip.city}V Here is a sample logs: Mar 25 … --- ## [Logstash input-\> file-\> path...Why the wildcard \* doesn't work?](https://discuss.elastic.co/t/logstash-input-file-path-why-the-wildcard-doesnt-work/130177)
**Author:** [@cuiyingjia](https://discuss.elastic.co/u/cuiyingjia)\ **Replies:** 1\ **Last updated:** [May 2, 2018, 6:06am UTC](https://discuss.elastic.co/t/logstash-input-file-path-why-the-wildcard-doesnt-work/130177 "2018-05-02T06:06:17Z")
I tried both on Windows7 and Windows10 , ELK are all latest version. If the file path is a specific file as below , it works: input{ file { path =\> "C:\\PATH\\Folder1\\Branch1\\target.log" start\_position =\> "begi… --- ## [Logstash Reference - Parsing Logs with Logstash](https://discuss.elastic.co/t/logstash-reference-parsing-logs-with-logstash/129917)
**Author:** [@yckoh](https://discuss.elastic.co/u/yckoh)\ **Replies:** 6\ **Last updated:** [May 2, 2018, 1:39am UTC](https://discuss.elastic.co/t/logstash-reference-parsing-logs-with-logstash/129917 "2018-05-02T01:39:01Z")
Hi, I've tried to send log lines from filebeat to Logstash as per the instructions in the reference. However, even though it shows that the pipeline has started successfully it seems to not give me any output on my comma… --- ## [Logstash with filter wont start: Failed to execute action](https://discuss.elastic.co/t/logstash-with-filter-wont-start-failed-to-execute-action/130105)
**Author:** [@sindred](https://discuss.elastic.co/u/sindred)\ **Replies:** 2\ **Last updated:** [May 1, 2018, 2:26pm UTC](https://discuss.elastic.co/t/logstash-with-filter-wont-start-failed-to-execute-action/130105 "2018-05-01T14:26:49Z")
My filter: input { beats { port =\> "5044" } } filter { multiline { pattern =\> "^\\s" what =\> "previous" } dissect { mapping =\> { "message" =\> "%{timestamp} | %{level} | %{subsystem} | … --- ## [Logstash stopped running](https://discuss.elastic.co/t/logstash-stopped-running/130120)
**Author:** [@Ant](https://discuss.elastic.co/u/Ant)\ **Replies:** 2\ **Last updated:** [May 1, 2018, 1:45pm UTC](https://discuss.elastic.co/t/logstash-stopped-running/130120 "2018-05-01T13:45:15Z")
I have a cloud elastic instance and a local VM running logstash that is reading data from RabbitMQ which is also local. This has been running fine for 4 days then today I nocitce that the queue on RabbitMQ is growing as… --- ## [How to use multiline pattern matching grok filter?](https://discuss.elastic.co/t/how-to-use-multiline-pattern-matching-grok-filter/129632)
**Author:** [@RameshNagargoje](https://discuss.elastic.co/u/RameshNagargoje)\ **Replies:** 5\ **Last updated:** [May 1, 2018, 9:45am UTC](https://discuss.elastic.co/t/how-to-use-multiline-pattern-matching-grok-filter/129632 "2018-05-01T09:45:53Z")
My use case is to find the line which comes before some pattern line and if that line is found in then dump in ES. Here is a small example. In this, if I am finding "dhd\_bus\_rxctl: rxcnt\_timeout" line before the lin… --- ## [How to skip grok filter match if one of the grok is matched](https://discuss.elastic.co/t/how-to-skip-grok-filter-match-if-one-of-the-grok-is-matched/130094)
**Author:** [@RameshNagargoje](https://discuss.elastic.co/u/RameshNagargoje)\ **Replies:** 3\ **Last updated:** [May 1, 2018, 8:51am UTC](https://discuss.elastic.co/t/how-to-skip-grok-filter-match-if-one-of-the-grok-is-matched/130094 "2018-05-01T08:51:06Z")
Hi, I am having 1000 of grok in one filter. I want to skip to check if one of the grok is matched. Here is my config if\[type\] == 'kmsg' { grok { match =\> { logs =\> ".\*Unable to handle kernel paging request at vir… --- ## [Logstash(5.4.3) fails to transfer logs even when logstash starts as a service](https://discuss.elastic.co/t/logstash-5-4-3-fails-to-transfer-logs-even-when-logstash-starts-as-a-service/130093)
**Author:** [@nikhil.k](https://discuss.elastic.co/u/nikhil.k)\ **Replies:** 2\ **Last updated:** [May 1, 2018, 7:41am UTC](https://discuss.elastic.co/t/logstash-5-4-3-fails-to-transfer-logs-even-when-logstash-starts-as-a-service/130093 "2018-05-01T07:41:40Z")
Hi Everyone, In my scenario, I installed logstash using yum. The logstash version is 5.4.3. I am sending /var/log/messages to a file(output.txt) which is present on same system. Operating System - Centos Below is the … --- ## [Which algorithms are used in grok filter to find exact match?](https://discuss.elastic.co/t/which-algorithms-are-used-in-grok-filter-to-find-exact-match/130090)
**Author:** [@RameshNagargoje](https://discuss.elastic.co/u/RameshNagargoje)\ **Replies:** 9\ **Last updated:** [May 1, 2018, 7:27am UTC](https://discuss.elastic.co/t/which-algorithms-are-used-in-grok-filter-to-find-exact-match/130090 "2018-05-01T07:27:29Z")
Hi all, I am using grok filter to find an exact match in logs. My logs size would be 10 mb. And streaming of logs is continuing going on. I am not getting that every time match is searched in 10 mb of log size Will it be… --- ## [Need help in grok pattern](https://discuss.elastic.co/t/need-help-in-grok-pattern/129942)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 4\ **Last updated:** [May 1, 2018, 2:44am UTC](https://discuss.elastic.co/t/need-help-in-grok-pattern/129942 "2018-05-01T02:44:23Z")
How do I drop certain words in between my log lines? I dont want to write a parser for them or not useful. E.g. Below log 29-Apr-2018 12:01:19.760 rpz: info: client 192.168.5.103#60868 (abc.com): rpz QNAME Local-Data … --- ## [Expose logstash API to host](https://discuss.elastic.co/t/expose-logstash-api-to-host/130067)
**Author:** [@rajisankar](https://discuss.elastic.co/u/rajisankar)\ **Replies:** 2\ **Last updated:** [April 30, 2018, 10:08pm UTC](https://discuss.elastic.co/t/expose-logstash-api-to-host/130067 "2018-04-30T22:08:07Z")
I am running Logstash 5.5 in a docker container. Since Logstash API binds to port 9600, i expose the port in the container hoping to access API endpoints from the host. Also, i am running metricbeat on the host to collec… --- ## [How to get only matched lines from "Message"?](https://discuss.elastic.co/t/how-to-get-only-matched-lines-from-message/129989)
**Author:** [@RameshNagargoje](https://discuss.elastic.co/u/RameshNagargoje)\ **Replies:** 1\ **Last updated:** [April 30, 2018, 10:04pm UTC](https://discuss.elastic.co/t/how-to-get-only-matched-lines-from-message/129989 "2018-04-30T22:04:08Z")
Hi, I am using grok filter to find match, lines in "message",But i not getting how to get that lines from "message". Here is my filter grok { match =\> { logs =\> "dhd\_check\_hang: Event HANG send up due to"} } And he… --- ## [How to take the first two lines after splitting on new lines](https://discuss.elastic.co/t/how-to-take-the-first-two-lines-after-splitting-on-new-lines/130061)
**Author:** [@Harsh\_Verma](https://discuss.elastic.co/u/Harsh_Verma)\ **Replies:** 1\ **Last updated:** [April 30, 2018, 9:59pm UTC](https://discuss.elastic.co/t/how-to-take-the-first-two-lines-after-splitting-on-new-lines/130061 "2018-04-30T21:59:57Z")
I want to store first two lines in one of my logs in a different variable using split() function of Mutate filter and then concatenating using a delimiter like comma, is that possible using split() and add\_field() comman… --- ## [Logstash maximum input](https://discuss.elastic.co/t/logstash-maximum-input/129994)
**Author:** [@Ahmed\_Gamal1](https://discuss.elastic.co/u/Ahmed_Gamal1)\ **Replies:** 1\ **Last updated:** [April 30, 2018, 9:48pm UTC](https://discuss.elastic.co/t/logstash-maximum-input/129994 "2018-04-30T21:48:49Z")
we are having logstash instance that curretnly have 6 jdbc input and this may exceed in the future (may be new 2 inputs per month)...Does logstash have maximum input? or if we exceeded number of inputs we will have slow … --- ## [Filter winlogbeat by eventID](https://discuss.elastic.co/t/filter-winlogbeat-by-eventid/128770)
**Author:** [@PandKing](https://discuss.elastic.co/u/PandKing)\ **Replies:** 7\ **Last updated:** [April 30, 2018, 8:34pm UTC](https://discuss.elastic.co/t/filter-winlogbeat-by-eventid/128770 "2018-04-30T20:34:17Z")
Hi, I am sending wineventlogs to logstash at about 60k/m and we need to send about 100 matching eventID's to an output. It would be ugly to add 100 or statements to this filter and just wondering what the best way migh… --- ## [Setting and testing logstash and elastic search intsallation](https://discuss.elastic.co/t/setting-and-testing-logstash-and-elastic-search-intsallation/129971)
**Author:** [@rbradley](https://discuss.elastic.co/u/rbradley)\ **Replies:** 11\ **Last updated:** [April 30, 2018, 8:31pm UTC](https://discuss.elastic.co/t/setting-and-testing-logstash-and-elastic-search-intsallation/129971 "2018-04-30T20:31:33Z")
Hi, I am trying to load a csv file into logtsash and output to elastic search. i have read some of the topics and my error is similar to them. \[Logstatsh%20error\] I dont think index has been created. what should i d… --- ## [How to process event which has \*.1.\* pattern](https://discuss.elastic.co/t/how-to-process-event-which-has-1-pattern/130052)
**Author:** [@HRG](https://discuss.elastic.co/u/HRG)\ **Replies:** 2\ **Last updated:** [April 30, 2018, 6:43pm UTC](https://discuss.elastic.co/t/how-to-process-event-which-has-1-pattern/130052 "2018-04-30T18:43:14Z")
Experts: I would like to drop all the events and process only those events which match .1. criteria in the message. I have written as below but not working. Could you please help how to write using regex. if \[message\]… --- ## [Can I digitally sign the incoming logs](https://discuss.elastic.co/t/can-i-digitally-sign-the-incoming-logs/130030)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 1\ **Last updated:** [April 30, 2018, 6:31pm UTC](https://discuss.elastic.co/t/can-i-digitally-sign-the-incoming-logs/130030 "2018-04-30T18:31:58Z")
Hi, Since I am forwarding my incoming logs to storage wondering if I can somehow digitally sign those for future forensic and maintain integrity? Is there any such mechanism available with logstash? --- ## [Removing specific fields](https://discuss.elastic.co/t/removing-specific-fields/130013)
**Author:** [@Nikolay\_Petrov](https://discuss.elastic.co/u/Nikolay_Petrov)\ **Replies:** 1\ **Last updated:** [April 30, 2018, 6:27pm UTC](https://discuss.elastic.co/t/removing-specific-fields/130013 "2018-04-30T18:27:35Z")
Hi, I'm trying to remove fields like this "\*\_bytes" using logstash filter, without any success so far: { ... "mountpoints": { "/soft": { "size\_bytes": "21003628544", "available\_bytes": "20957556736", "used\_bytes… --- ## [Error while starting filebeat](https://discuss.elastic.co/t/error-while-starting-filebeat/129975)
**Author:** [@Tamal\_Kundu](https://discuss.elastic.co/u/Tamal_Kundu)\ **Replies:** 3\ **Last updated:** [April 30, 2018, 6:20pm UTC](https://discuss.elastic.co/t/error-while-starting-filebeat/129975 "2018-04-30T18:20:51Z")
\[fuseadmin@a0110pcsgesb01 filebeat-5.5.0-linux-x86\_64\]$ ./filebeat -e -c filebeat.yml 2018/04/29 19:35:47.194084 beat.go:285: INFO Home path: \[/app/filebeat-5.5.0-linux-x86\_64\] Config path: \[/app/filebeat-5.5.0-linux-x8… --- ## [Logstash translate plugin dynamic dictionary path](https://discuss.elastic.co/t/logstash-translate-plugin-dynamic-dictionary-path/129889)
**Author:** [@krishna\_chaitanya](https://discuss.elastic.co/u/krishna_chaitanya)\ **Replies:** 3\ **Last updated:** [April 30, 2018, 4:44pm UTC](https://discuss.elastic.co/t/logstash-translate-plugin-dynamic-dictionary-path/129889 "2018-04-30T16:44:27Z")
Hi, I am creating a csv file which I want to use as a dictionary inside translate plugin. Since the mapping might change the next day, I named my csv file this way: my-mapping-YYYY-MM-dd.csv where YYYY, MM, dd are ye… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=768) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=770)