# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=869 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 870 --- ## [Duplicate Entries for multiline log processed by Logstash](https://discuss.elastic.co/t/duplicate-entries-for-multiline-log-processed-by-logstash/108925)
**Author:** [@pborges](https://discuss.elastic.co/u/pborges)\ **Replies:** 0\ **Last updated:** [November 23, 2017, 4:05pm UTC](https://discuss.elastic.co/t/duplicate-entries-for-multiline-log-processed-by-logstash/108925 "2017-11-23T16:05:56Z")
Hello, I'm new to ELK usage and I would greatly appreciate feedback on the following problem that has been driving me crazy for the last couple of days. I have a Filebeat that is shipping logs to Logstash, including mu… --- ## [JMX plugin - call an operation?](https://discuss.elastic.co/t/jmx-plugin-call-an-operation/108917)
**Author:** [@TimWard](https://discuss.elastic.co/u/TimWard)\ **Replies:** 0\ **Last updated:** [November 23, 2017, 3:37pm UTC](https://discuss.elastic.co/t/jmx-plugin-call-an-operation/108917 "2017-11-23T15:37:18Z")
I'm wanting to find out whether there are any deadlocked threads in a Java application, which looks like a JMX call to java.lang.Threading.findDeadlockedThreads() - no parameters to pass, a null response means OK, a non-… --- ## [How to turn fields in a text file into terms in kibana](https://discuss.elastic.co/t/how-to-turn-fields-in-a-text-file-into-terms-in-kibana/107551)
**Author:** [@yishain11](https://discuss.elastic.co/u/yishain11)\ **Replies:** 1\ **Last updated:** [November 23, 2017, 3:16pm UTC](https://discuss.elastic.co/t/how-to-turn-fields-in-a-text-file-into-terms-in-kibana/107551 "2017-11-23T15:16:32Z")
I have a text file, formmated like JSON: "changedFieldsList": \[ { "fieldId": "ORDER\_ID", "fieldType": "NUMBER", "fieldValue": "55841", "fieldChanged": "Y" }, { "fieldId": "ORDER\_DATE", "fieldType": "DATE", … --- ## [Create a global patterns\_dir](https://discuss.elastic.co/t/create-a-global-patterns-dir/108898)
**Author:** [@Vafa\_Ronaghi](https://discuss.elastic.co/u/Vafa_Ronaghi)\ **Replies:** 1\ **Last updated:** [November 23, 2017, 2:25pm UTC](https://discuss.elastic.co/t/create-a-global-patterns-dir/108898 "2017-11-23T14:25:05Z")
Hello, Is it possible to define a global patterns\_dir for all cases in filter or it must be defined in each grok section. Se below: if ("xxx" in \[path\]) { grok { \*\*patterns\_dir =\> \[ "/usr/share/logs… --- ## [Logstash 5.6 how to add\_field from redis?](https://discuss.elastic.co/t/logstash-5-6-how-to-add-field-from-redis/108855)
**Author:** [@feranwq](https://discuss.elastic.co/u/feranwq)\ **Replies:** 1\ **Last updated:** [November 23, 2017, 2:20pm UTC](https://discuss.elastic.co/t/logstash-5-6-how-to-add-field-from-redis/108855 "2017-11-23T14:20:49Z")
input log like this """ {"@timestamp":"2017-11-23T16:10:20+08:00","host":"192.168.3.251","userip":"192.168.3.254","size":1973,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http\_host":"192.168.3.251","url"… --- ## [Inspect Logstash resource consumption](https://discuss.elastic.co/t/inspect-logstash-resource-consumption/108902)
**Author:** [@ntviet18](https://discuss.elastic.co/u/ntviet18)\ **Replies:** 0\ **Last updated:** [November 23, 2017, 2:09pm UTC](https://discuss.elastic.co/t/inspect-logstash-resource-consumption/108902 "2017-11-23T14:09:53Z")
My Logstash cluster with 4 nodes was running perfectly fine and using 1 Gib of memory each. After I configured persistent queue, suddenly each node use up to 8-10 Gib each. I am suspecting some kind of buffering happen, … --- ## [Http input return code](https://discuss.elastic.co/t/http-input-return-code/108722)
**Author:** [@cvi](https://discuss.elastic.co/u/cvi)\ **Replies:** 3\ **Last updated:** [November 23, 2017, 1:27pm UTC](https://discuss.elastic.co/t/http-input-return-code/108722 "2017-11-23T13:27:07Z")
Hi, I am testing the http input plugin with logstash 5.6 and I didn't find a way to answer something else than "ok". For instance, could something in the logstash pipeline filter control what to send back (e.g. an error… --- ## [Logstach 2.3.4 is not working](https://discuss.elastic.co/t/logstach-2-3-4-is-not-working/108832)
**Author:** [@anilagrawal6991](https://discuss.elastic.co/u/anilagrawal6991)\ **Replies:** 4\ **Last updated:** [November 23, 2017, 9:50am UTC](https://discuss.elastic.co/t/logstach-2-3-4-is-not-working/108832 "2017-11-23T09:50:27Z")
I am using Logstach 2.3.4 and elastic search 2.3.5 on windows 10. but when I tried to insert data into the elastic search using Logstach its not working its only show "io/console not supported; tty will not be manipula… --- ## [Logstash startup completed nothing proceeds after this command in logstash](https://discuss.elastic.co/t/logstash-startup-completed-nothing-proceeds-after-this-command-in-logstash/108715)
**Author:** [@ayushbij27](https://discuss.elastic.co/u/ayushbij27)\ **Replies:** 5\ **Last updated:** [November 23, 2017, 9:50am UTC](https://discuss.elastic.co/t/logstash-startup-completed-nothing-proceeds-after-this-command-in-logstash/108715 "2017-11-23T09:50:24Z")
I am trying to index the data from logs file to elastisearch but I was unable to do so because after the Logstash Startup completed message appears nothing actually happens. So I changed conf file to input { file { pa… --- ## [Elasticsearch input plugin stops](https://discuss.elastic.co/t/elasticsearch-input-plugin-stops/108844)
**Author:** [@ismael\_boumedien](https://discuss.elastic.co/u/ismael_boumedien)\ **Replies:** 1\ **Last updated:** [November 23, 2017, 9:36am UTC](https://discuss.elastic.co/t/elasticsearch-input-plugin-stops/108844 "2017-11-23T09:36:49Z")
Hello i'm using the logstash elasticsearch plugin input, but i find out that logstash stops. I wish to keep logstash tour to retrieve new events coming from elasticsearch. My configuration: input { elasticsearch { ho… --- ## [TCP syslog output not working properly in logstash](https://discuss.elastic.co/t/tcp-syslog-output-not-working-properly-in-logstash/108830)
**Author:** [@vishal0708gautam](https://discuss.elastic.co/u/vishal0708gautam)\ **Replies:** 0\ **Last updated:** [November 23, 2017, 7:53am UTC](https://discuss.elastic.co/t/tcp-syslog-output-not-working-properly-in-logstash/108830 "2017-11-23T07:53:27Z")
We are using logstash for UDP & TCP syslog events receiving and forwarding. The input for TCP & UDP in logstash is working properly and receiving all the syslog events but TCP output syslog forwarding its not working pro… --- ## [Providing config files in docker command](https://discuss.elastic.co/t/providing-config-files-in-docker-command/108708)
**Author:** [@Sanju\_Mon](https://discuss.elastic.co/u/Sanju_Mon)\ **Replies:** 5\ **Last updated:** [November 23, 2017, 6:49am UTC](https://discuss.elastic.co/t/providing-config-files-in-docker-command/108708 "2017-11-23T06:49:10Z")
Hi, I am trying to run logstash on docker by using the custom configuration for logstash. Logstash for Docker was pulled using the following commands: docker pull docker.elastic.co/logstash/logstash:6.0.0 docker pull … --- ## [Failing to parse csv](https://discuss.elastic.co/t/failing-to-parse-csv/108613)
**Author:** [@chikondot](https://discuss.elastic.co/u/chikondot)\ **Replies:** 5\ **Last updated:** [November 23, 2017, 6:31am UTC](https://discuss.elastic.co/t/failing-to-parse-csv/108613 "2017-11-23T06:31:23Z")
Good day, I am trying to parse multiple csv files in a directory for trend analysis and report generation, Index is being created but not all information is being parsed. The error below is being displayed \[ERROR\] 201… --- ## [How to store and resend syslog events when output server goes offline](https://discuss.elastic.co/t/how-to-store-and-resend-syslog-events-when-output-server-goes-offline/108822)
**Author:** [@vishal0708gautam](https://discuss.elastic.co/u/vishal0708gautam)\ **Replies:** 0\ **Last updated:** [November 23, 2017, 6:25am UTC](https://discuss.elastic.co/t/how-to-store-and-resend-syslog-events-when-output-server-goes-offline/108822 "2017-11-23T06:25:57Z")
Hello Experts, We are using logstash as syslog receiver and forwarder both is done using TCP & UDP. Logstash outputs syslog event to a server whenever it receives syslog in input. But in some cases server may not be ava… --- ## [Unable to import CSV to elasticsearch via logstash](https://discuss.elastic.co/t/unable-to-import-csv-to-elasticsearch-via-logstash/108726)
**Author:** [@JoelJoy](https://discuss.elastic.co/u/JoelJoy)\ **Replies:** 2\ **Last updated:** [November 23, 2017, 5:22am UTC](https://discuss.elastic.co/t/unable-to-import-csv-to-elasticsearch-via-logstash/108726 "2017-11-23T05:22:37Z")
Hi the Created the Config as below. usecase.config: input { file { path =\> "C:\\ElasticStack\\UseCase\_Karthi.csv" start\_position =\> "beginning" } } filter { csv { separator =\> "," columns =\> \["Use case number"… --- ## [Configure logstash with two same type sources](https://discuss.elastic.co/t/configure-logstash-with-two-same-type-sources/108814)
**Author:** [@hugo53](https://discuss.elastic.co/u/hugo53)\ **Replies:** 1\ **Last updated:** [November 23, 2017, 4:56am UTC](https://discuss.elastic.co/t/configure-logstash-with-two-same-type-sources/108814 "2017-11-23T04:56:51Z")
Hi there, I'm trying to configure logstash for my application, in the case: Two sources with same type 'raw' (it's a type of buynan-logstash, a nodejs module) input { tcp { type =\> "raw" … --- ## [Difference between input type "syslog" and "tcp"?](https://discuss.elastic.co/t/difference-between-input-type-syslog-and-tcp/108574)
**Author:** [@jcarnat](https://discuss.elastic.co/u/jcarnat)\ **Replies:** 2\ **Last updated:** [November 22, 2017, 10:22pm UTC](https://discuss.elastic.co/t/difference-between-input-type-syslog-and-tcp/108574 "2017-11-22T22:22:29Z")
Hi, What is the difference between using the syslog and the tcp inputs when dealing with logs? After reading the 5.6/plugins-inputs-syslog and 5.6/plugins-inputs-tcp pages, I can't guess the pros / cons of each plugins… --- ## [Generating multiple events from a single message](https://discuss.elastic.co/t/generating-multiple-events-from-a-single-message/108772)
**Author:** [@pri](https://discuss.elastic.co/u/pri)\ **Replies:** 2\ **Last updated:** [November 22, 2017, 9:05pm UTC](https://discuss.elastic.co/t/generating-multiple-events-from-a-single-message/108772 "2017-11-22T21:05:00Z")
Hi, I have a csv file that looks like this: The first line is the header column Item, Type, Category, Q1, Q2, Q3, Q4 Tools, License, Cost Saving, 15000, 0, 6000, 899 I am trying to consume this in Logstash and … --- ## [\[ATTENTION\] Logstash 6.0.0 - Known Issue Indexing to Elasticsearch 6.0](https://discuss.elastic.co/t/attention-logstash-6-0-0-known-issue-indexing-to-elasticsearch-6-0/108789)
**Author:** [@alvin](https://discuss.elastic.co/u/alvin)\ **Replies:** 0\ **Last updated:** [November 22, 2017, 8:16pm UTC](https://discuss.elastic.co/t/attention-logstash-6-0-0-known-issue-indexing-to-elasticsearch-6-0/108789 "2017-11-22T20:16:15Z")
This issue affects Logstash 6.0.0 users indexing to Elasticsearch 6.0 when multiple “type” field values are present in Logstash events. It affects both new users on Logstash 6.0.0, and existing users upgrading to Logstas… --- ## [Logstash aggregate filter - sendmail logs](https://discuss.elastic.co/t/logstash-aggregate-filter-sendmail-logs/105917)
**Author:** [@kuznick](https://discuss.elastic.co/u/kuznick)\ **Replies:** 1\ **Last updated:** [November 22, 2017, 7:47pm UTC](https://discuss.elastic.co/t/logstash-aggregate-filter-sendmail-logs/105917 "2017-11-22T19:47:29Z")
I would like to forward sendmail logs based on reception domain. Only possible solution I can see is to aggregate sendmail logs based on session id and forward it. sample log is 2017-10-06T00:00:25+00:00 smtpser01 send… --- ## [Logstash Replacing A Value](https://discuss.elastic.co/t/logstash-replacing-a-value/108781)
**Author:** [@motts](https://discuss.elastic.co/u/motts)\ **Replies:** 1\ **Last updated:** [November 22, 2017, 7:13pm UTC](https://discuss.elastic.co/t/logstash-replacing-a-value/108781 "2017-11-22T19:13:52Z")
I am working on filtering ftp logs that are in xml format. Everything works fine, but this one column that's labeled "host". If that column has data in it, that data will be put into elasticsearch. However, if that colu… --- ## [Logstash 6.0 to elasticsearch 5.6.4 get a 503](https://discuss.elastic.co/t/logstash-6-0-to-elasticsearch-5-6-4-get-a-503/108748)
**Author:** [@rj-reilly](https://discuss.elastic.co/u/rj-reilly)\ **Replies:** 6\ **Last updated:** [November 22, 2017, 6:08pm UTC](https://discuss.elastic.co/t/logstash-6-0-to-elasticsearch-5-6-4-get-a-503/108748 "2017-11-22T18:08:00Z")
Hi, I upgraded logstash to 6.0 to take advantage of multiple pipelines and logstash receives a 503 from the already working elasticsearch. below is my config and conf logstash.yml path.data: /var/lib/logstash path.lo… --- ## [File input pipeline not working in Logstash 5.6.3](https://discuss.elastic.co/t/file-input-pipeline-not-working-in-logstash-5-6-3/108469)
**Author:** [@sgladkovich](https://discuss.elastic.co/u/sgladkovich)\ **Replies:** 10\ **Last updated:** [November 22, 2017, 6:03pm UTC](https://discuss.elastic.co/t/file-input-pipeline-not-working-in-logstash-5-6-3/108469 "2017-11-22T18:03:33Z")
Hello, I have a single-node ElasticStack 5.6.3 on Ubuntu 14.04 in AWS, and I'm trying to get AWS RDS logs sent to it. I have a script (on the same node) that downloads the log files with no issues. Here's my Logstash … --- ## [Extract rows from log using http plugin](https://discuss.elastic.co/t/extract-rows-from-log-using-http-plugin/106344)
**Author:** [@mfellah](https://discuss.elastic.co/u/mfellah)\ **Replies:** 5\ **Last updated:** [November 22, 2017, 4:39pm UTC](https://discuss.elastic.co/t/extract-rows-from-log-using-http-plugin/106344 "2017-11-22T16:39:12Z")
Hi, I'm trying to use http plugin to receive log with this format : 2017-10-31-07:29:05 - DEBUG Waiting for callback, for command id : 56232 2017-10-31-07:29:05 - DEBUG Waiting for callback... 2017-10-31-07:29:06… --- ## [Loogstash is not parsing the message correctly to kibana](https://discuss.elastic.co/t/loogstash-is-not-parsing-the-message-correctly-to-kibana/108344)
**Author:** [@Jonesthomas](https://discuss.elastic.co/u/Jonesthomas)\ **Replies:** 6\ **Last updated:** [November 22, 2017, 3:58pm UTC](https://discuss.elastic.co/t/loogstash-is-not-parsing-the-message-correctly-to-kibana/108344 "2017-11-22T15:58:33Z")
hi,my sample log {"LogLevel":"ERROR","LogMsg":"{\\"itemId\\":0,\\"module\\":\\"/curie/encounter\\",\\"action\\":\\"/addToken\\",\\"errorMessage\\":\\"java.lang.RuntimeException: org.apache.ibatis.exceptions.PersistenceException: \\\\n#… --- ## [Logstash JDBC Input infinite loop](https://discuss.elastic.co/t/logstash-jdbc-input-infinite-loop/108739)
**Author:** [@mc1392](https://discuss.elastic.co/u/mc1392)\ **Replies:** 5\ **Last updated:** [November 22, 2017, 3:45pm UTC](https://discuss.elastic.co/t/logstash-jdbc-input-infinite-loop/108739 "2017-11-22T15:45:37Z")
I'm using Logstash 5.6.3 and I think I am stuck in an infinite loop. I'm using the upsert functionality, with my Id field as the document\_id. Seems the same record is getting updated multiple times. The process never … --- ## [Enrich logs with client certificate subject](https://discuss.elastic.co/t/enrich-logs-with-client-certificate-subject/108698)
**Author:** [@cruscio](https://discuss.elastic.co/u/cruscio)\ **Replies:** 1\ **Last updated:** [November 22, 2017, 3:41pm UTC](https://discuss.elastic.co/t/enrich-logs-with-client-certificate-subject/108698 "2017-11-22T15:41:43Z")
When Logstash is secured with SSL mutual authentication, is it possible to have Logstash add the client certificate's subject as a field in an event before forwarding to Elasticsearch? --- ## [Logstash JDBC Input Complete Action](https://discuss.elastic.co/t/logstash-jdbc-input-complete-action/108620)
**Author:** [@Jeff\_Bursik](https://discuss.elastic.co/u/Jeff_Bursik)\ **Replies:** 2\ **Last updated:** [November 22, 2017, 3:29pm UTC](https://discuss.elastic.co/t/logstash-jdbc-input-complete-action/108620 "2017-11-22T15:29:51Z")
This has been asked in other threads however a resolution or a solution doesn't appear to have ever been posted so I'd like to ask agin: Is it possible to have a post action once Logstash has completed the stream/output… --- ## [Http\_poller header set in post](https://discuss.elastic.co/t/http-poller-header-set-in-post/108742)
**Author:** [@michimau](https://discuss.elastic.co/u/michimau)\ **Replies:** 0\ **Last updated:** [November 22, 2017, 3:29pm UTC](https://discuss.elastic.co/t/http-poller-header-set-in-post/108742 "2017-11-22T15:29:00Z")
I am trying to fetch data from a sparql endpoint. i can make a successfull http call with the following parameters: endpoint: cr.eionet.europa.eu/sparql header set like: "query"= "see the query in logstash.conf...eas… --- ## [Date Custom Field Timezone](https://discuss.elastic.co/t/date-custom-field-timezone/108673)
**Author:** [@MaikLinnemann](https://discuss.elastic.co/u/MaikLinnemann)\ **Replies:** 6\ **Last updated:** [November 22, 2017, 3:10pm UTC](https://discuss.elastic.co/t/date-custom-field-timezone/108673 "2017-11-22T15:10:53Z")
Dear Forum, i have a config which is as follows: filter { if \[type\] == "nginx-default" { grok { patterns\_dir =\> "/etc/logstash/patterns" match =\> \[ "message", "%{COMBINEDAPACHELOG}" \] } date {… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=868) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=870)