# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=659 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 660 --- ## [Split ES event](https://discuss.elastic.co/t/split-es-event/154152)
**Author:** [@humalog](https://discuss.elastic.co/u/humalog)\ **Replies:** 12\ **Last updated:** [October 26, 2018, 2:48pm UTC](https://discuss.elastic.co/t/split-es-event/154152 "2018-10-26T14:48:56Z")
Hello, at the moment I have one event. Console brings the following output: { port =\> \[ \[0\] "22", \[1\] "443", \[2\] "623", \[3\] "0", \[4\] "22", \[5\] "80" \], "path" =\>"/home/data/… --- ## [Logstash - dump s3 into aws elasticsearch (vpc) - s3 is hanging forever](https://discuss.elastic.co/t/logstash-dump-s3-into-aws-elasticsearch-vpc-s3-is-hanging-forever/154072)
**Author:** [@murali12180](https://discuss.elastic.co/u/murali12180)\ **Replies:** 1\ **Last updated:** [October 26, 2018, 12:07pm UTC](https://discuss.elastic.co/t/logstash-dump-s3-into-aws-elasticsearch-vpc-s3-is-hanging-forever/154072 "2018-10-26T12:07:12Z")
I can transfer data from sql server to aws ES using logstash conf, however, i'm having issues with dumping s3 bucket into aws es. error: \[INFO \] 2018-10-25 23:29:58.266 \[\[main\]\ **Author:** [@gunygoogoo](https://discuss.elastic.co/u/gunygoogoo)\ **Replies:** 5\ **Last updated:** [October 26, 2018, 12:03pm UTC](https://discuss.elastic.co/t/advice-on-logstash-beats-and-basic-log-aggregation/154031 "2018-10-26T12:03:33Z") Hi, I've been reading a number of articles as I try to put a design together for log aggregation. My situation is that I have a relatively small (no huge traffic loads) distributed application. The overall app will be … --- ## [Extract Fields from JSON Array](https://discuss.elastic.co/t/extract-fields-from-json-array/150718)
**Author:** [@cappy](https://discuss.elastic.co/u/cappy)\ **Replies:** 10\ **Last updated:** [October 26, 2018, 11:31am UTC](https://discuss.elastic.co/t/extract-fields-from-json-array/150718 "2018-10-26T11:31:33Z")
Hello, I would like to extract the fields from the following JSON: { "startTime": "2018-10-01T20:33:56", "results": \[{ "flags": \[ "ScanPe::no\_resources", "ScanYara::compiling\_error" \], "flavors": { "mi… --- ## [Dynamic index](https://discuss.elastic.co/t/dynamic-index/154146)
**Author:** [@wuwo1952368901](https://discuss.elastic.co/u/wuwo1952368901)\ **Replies:** 3\ **Last updated:** [October 26, 2018, 11:12am UTC](https://discuss.elastic.co/t/dynamic-index/154146 "2018-10-26T11:12:42Z")
I want to dynamic index from meassage nodename,but it's only show "%{\[node\_name\]}" ,it's not my node\_name. How can i config this? filter { grok { match =\> { "message"=\>"#%{DATA:request\_time}#%{DATA:node\_name}#%{DATA:… --- ## [Logstash 6.4 having errors during start](https://discuss.elastic.co/t/logstash-6-4-having-errors-during-start/153415)
**Author:** [@motts](https://discuss.elastic.co/u/motts)\ **Replies:** 1\ **Last updated:** [October 26, 2018, 11:07am UTC](https://discuss.elastic.co/t/logstash-6-4-having-errors-during-start/153415 "2018-10-26T11:07:38Z")
Hello, I installed logstash 6.4 and I am seeing this error: It seems your ruby installation is missing psych (for YAML output). To eliminate this warning, please install libyaml and reinstall your ruby. I installed li… --- ## [Time format in importing csv](https://discuss.elastic.co/t/time-format-in-importing-csv/154174)
**Author:** [@Nay\_Myo\_Zaw](https://discuss.elastic.co/u/Nay_Myo_Zaw)\ **Replies:** 1\ **Last updated:** [October 26, 2018, 10:46am UTC](https://discuss.elastic.co/t/time-format-in-importing-csv/154174 "2018-10-26T10:46:28Z")
When I import datetime data as index it's changed to as below: 2018-10-20 11:55:46 -\> January 20th 2018, 11:55:46.000 here is my date filter from conf date { locale =\> "en" match =\> \["Created","YYYY-MM-DD hh:mm:ss"… --- ## [Logstash pipelines throwing NoMethodError exceptions with undefined method ever since I added an S3 Output plugin (6.4)](https://discuss.elastic.co/t/logstash-pipelines-throwing-nomethoderror-exceptions-with-undefined-method-ever-since-i-added-an-s3-output-plugin-6-4/152529)
**Author:** [@shog1466112](https://discuss.elastic.co/u/shog1466112)\ **Replies:** 1\ **Last updated:** [October 26, 2018, 9:48am UTC](https://discuss.elastic.co/t/logstash-pipelines-throwing-nomethoderror-exceptions-with-undefined-method-ever-since-i-added-an-s3-output-plugin-6-4/152529 "2018-10-26T09:48:35Z")
Hi there, I recently added an S3 output plugin to my logstash pipelines (two of them). Ever since I've been getting these exceptions which I haven't been able to get to the bottom of. The containers running logstash de… --- ## [Oracle Logs with JDBC Input](https://discuss.elastic.co/t/oracle-logs-with-jdbc-input/154154)
**Author:** [@Miguel\_Leite](https://discuss.elastic.co/u/Miguel_Leite)\ **Replies:** 0\ **Last updated:** [October 26, 2018, 9:01am UTC](https://discuss.elastic.co/t/oracle-logs-with-jdbc-input/154154 "2018-10-26T09:01:04Z")
Hey there, I was wondering if it would be possible to use the Logstash JDBC Plug-in to get log data from Oracle, like the listener.log and the alert.log... I'm having lots of issues trying to parse the listener.log... T… --- ## [Http\_poller input plugin issue](https://discuss.elastic.co/t/http-poller-input-plugin-issue/153576)
**Author:** [@shirm](https://discuss.elastic.co/u/shirm)\ **Replies:** 4\ **Last updated:** [October 26, 2018, 8:12am UTC](https://discuss.elastic.co/t/http-poller-input-plugin-issue/153576 "2018-10-26T08:12:32Z")
I am trying to create a dashboard for our devops tools, e.g GIT commits/pull requests TeamCity Build Status Sonar metrics Etc I am trying to fetch these data via Logstash I am using the input plugin, http\_poller Be… --- ## [Logstash 6.2.4 stuck in infinite retry loop](https://discuss.elastic.co/t/logstash-6-2-4-stuck-in-infinite-retry-loop/153527)
**Author:** [@pankhuri](https://discuss.elastic.co/u/pankhuri)\ **Replies:** 0\ **Last updated:** [October 23, 2018, 6:48am UTC](https://discuss.elastic.co/t/logstash-6-2-4-stuck-in-infinite-retry-loop/153527 "2018-10-23T06:48:24Z")
Hi, I am using logstash 6.2.4 with following yml settings - pipeline.batch.size: 600 pipeline.workers: 1 dead\_letter\_queue.enable: true The conf file used to run logstash application is - input { file { p… --- ## [Logstash-jms-input (JMS consumer died)](https://discuss.elastic.co/t/logstash-jms-input-jms-consumer-died/154131)
**Author:** [@Muhammad\_Rizky](https://discuss.elastic.co/u/Muhammad_Rizky)\ **Replies:** 0\ **Last updated:** [October 26, 2018, 7:36am UTC](https://discuss.elastic.co/t/logstash-jms-input-jms-consumer-died/154131 "2018-10-26T07:36:50Z")
Hei buddies, i've an issue when i run logstash config with logstash-input-jms. This is my logstash config: input { jms { destination =\> "log.to.logstash.pipeline" username =\> "" password =\> "!" yaml\_file =\> "/hom… --- ## [Bulk error in logstash](https://discuss.elastic.co/t/bulk-error-in-logstash/154097)
**Author:** [@Abhijith\_Vijayan](https://discuss.elastic.co/u/Abhijith_Vijayan)\ **Replies:** 4\ **Last updated:** [October 26, 2018, 6:15am UTC](https://discuss.elastic.co/t/bulk-error-in-logstash/154097 "2018-10-26T06:15:15Z")
I am getting the below error from logstash frequently. Is this because of elasticsearch? \[2018-10-26T08:31:12,884\]\[ERROR\]\[logstash.outputs.elasticsearch\] Attempted to send a bulk request to elasticsearch, but no there … --- ## [Synchronize data to es and customize es index structure](https://discuss.elastic.co/t/synchronize-data-to-es-and-customize-es-index-structure/154098)
**Author:** [@fdb58704a2c5cc6a3286](https://discuss.elastic.co/u/fdb58704a2c5cc6a3286)\ **Replies:** 1\ **Last updated:** [October 26, 2018, 5:54am UTC](https://discuss.elastic.co/t/synchronize-data-to-es-and-customize-es-index-structure/154098 "2018-10-26T05:54:43Z")
i have two table: customer,customer\_order customer have customer\_name,sex,customerid columns. and i want synchronize the two table to elasticsearch customerOrder index the customerOrder structure like { orderid:x, o… --- ## [How to avoid storing original xml when using the XML filter](https://discuss.elastic.co/t/how-to-avoid-storing-original-xml-when-using-the-xml-filter/154091)
**Author:** [@anp](https://discuss.elastic.co/u/anp)\ **Replies:** 1\ **Last updated:** [October 26, 2018, 5:46am UTC](https://discuss.elastic.co/t/how-to-avoid-storing-original-xml-when-using-the-xml-filter/154091 "2018-10-26T05:46:25Z")
I have defined this XML filter: xml { source =\> "xml\_descriptors" target =\> "json\_descriptors" force\_array =\> false } With the above my ES index contains both the original xml\_descriptors (as XML) and the transform… --- ## [Failed to start logstash](https://discuss.elastic.co/t/failed-to-start-logstash/153984)
**Author:** [@arvind.jadaun](https://discuss.elastic.co/u/arvind.jadaun)\ **Replies:** 6\ **Last updated:** [October 26, 2018, 4:40am UTC](https://discuss.elastic.co/t/failed-to-start-logstash/153984 "2018-10-26T04:40:45Z")
Hi Guys, This is first time i am starting logstash on my windows machine and below is the error msg i have. Is there anyone out there who has similar challenges like me. Seeking for your inputs. Thanks! \> \[2018-10-25T1… --- ## [XML tag filtering](https://discuss.elastic.co/t/xml-tag-filtering/154029)
**Author:** [@humalog](https://discuss.elastic.co/u/humalog)\ **Replies:** 2\ **Last updated:** [October 26, 2018, 4:37am UTC](https://discuss.elastic.co/t/xml-tag-filtering/154029 "2018-10-26T04:37:04Z")
Hello, i have an xml file. I use input codec multiline. My filter is an xml xpath filter and output is stdout. I have a question about xml tag filtering. The XML structure looks like this: \ \ \ **Author:** [@thangld](https://discuss.elastic.co/u/thangld)\ **Replies:** 1\ **Last updated:** [October 26, 2018, 3:21am UTC](https://discuss.elastic.co/t/how-to-decode-email-subject-using-logstash-filter/153827 "2018-10-26T03:21:54Z") Hello elastic community! I'm new to logstash. I'm using logstash filter to parse CISCO Ironport logs. But some of it has encode email Subject. How can I decode this one with logstash? Is there any filters to do it? Ex… --- ## [Explosion of fields from nginx log parsing](https://discuss.elastic.co/t/explosion-of-fields-from-nginx-log-parsing/154078)
**Author:** [@jgrevich](https://discuss.elastic.co/u/jgrevich)\ **Replies:** 0\ **Last updated:** [October 26, 2018, 1:49am UTC](https://discuss.elastic.co/t/explosion-of-fields-from-nginx-log-parsing/154078 "2018-10-26T01:49:07Z")
We exceeded the default number of total fields for one index and it appears to be related to how we're parsing log events from nginx (and perhaps others). Does anyone know what might cause these types of fields to be cr… --- ## [Logstash Persistent Queue inhibits the ability to see near real time status](https://discuss.elastic.co/t/logstash-persistent-queue-inhibits-the-ability-to-see-near-real-time-status/154076)
**Author:** [@swtrux](https://discuss.elastic.co/u/swtrux)\ **Replies:** 0\ **Last updated:** [October 26, 2018, 1:29am UTC](https://discuss.elastic.co/t/logstash-persistent-queue-inhibits-the-ability-to-see-near-real-time-status/154076 "2018-10-26T01:29:43Z")
I want to open an enhancement request for this but wanted to get feedback here first... When using persistent queueing in Logstash access to near real time data is not available until queued data is worked off. The pre… --- ## [Websocket input client threw expection- No Method error](https://discuss.elastic.co/t/websocket-input-client-threw-expection-no-method-error/153926)
**Author:** [@Nidhi\_Mishra](https://discuss.elastic.co/u/Nidhi_Mishra)\ **Replies:** 3\ **Last updated:** [October 26, 2018, 12:58am UTC](https://discuss.elastic.co/t/websocket-input-client-threw-expection-no-method-error/153926 "2018-10-26T00:58:24Z")
The Config file of Logstash for the Websocket Client- input { websocket { codec =\> "plain" url =\> "http://localhost:8080/WebsocketHome/" } } output { elasticsearch… --- ## [Logstash ignores my if condition](https://discuss.elastic.co/t/logstash-ignores-my-if-condition/154066)
**Author:** [@rtfm.pls](https://discuss.elastic.co/u/rtfm.pls)\ **Replies:** 2\ **Last updated:** [October 26, 2018, 12:07am UTC](https://discuss.elastic.co/t/logstash-ignores-my-if-condition/154066 "2018-10-26T00:07:23Z")
Some of input log messages are in JSON format. I would like to use "time" value as "@timestamp". This is an example of log message in JSON format : { "level":30, "time":1531171074631, "msg":"hello world", "pid"… --- ## [HTTP NLog to Logstash Not Parsing Right](https://discuss.elastic.co/t/http-nlog-to-logstash-not-parsing-right/154070)
**Author:** [@hmendoza](https://discuss.elastic.co/u/hmendoza)\ **Replies:** 0\ **Last updated:** [October 25, 2018, 10:25pm UTC](https://discuss.elastic.co/t/http-nlog-to-logstash-not-parsing-right/154070 "2018-10-25T22:25:58Z")
I have a similar issue than Empty result when using TCP input to collect NLog JSON events and no one posted a response to that (automatically got closed after 28 days). I captured the output that NLog is sending to Elas… --- ## [Change character encoding in Logstash](https://discuss.elastic.co/t/change-character-encoding-in-logstash/153806)
**Author:** [@varun1992](https://discuss.elastic.co/u/varun1992)\ **Replies:** 2\ **Last updated:** [October 25, 2018, 7:09pm UTC](https://discuss.elastic.co/t/change-character-encoding-in-logstash/153806 "2018-10-25T19:09:28Z")
How to specify the character encoding as UTF 16 or ASCII in logstash? My input section as below. I am reading a CSV file using CSV plugin in filter section input { file { path =\> "C:/ITIL\_Incidents.csv" … --- ## [Elastic Json parse into logstash](https://discuss.elastic.co/t/elastic-json-parse-into-logstash/151840)
**Author:** [@moughrom](https://discuss.elastic.co/u/moughrom)\ **Replies:** 29\ **Last updated:** [October 25, 2018, 5:36pm UTC](https://discuss.elastic.co/t/elastic-json-parse-into-logstash/151840 "2018-10-25T17:36:26Z")
Hello, I am retrieving indices from an elasticsearch containing syslogs messages through a curl commande then they are read by a logstash. and want to extract 4 fields from this messages (datetime\_received,"ip\_host\_pkt… --- ## [Logstash Output plugin Elastic/Kafka Polling Behaviour](https://discuss.elastic.co/t/logstash-output-plugin-elastic-kafka-polling-behaviour/154043)
**Author:** [@londonx](https://discuss.elastic.co/u/londonx)\ **Replies:** 0\ **Last updated:** [October 25, 2018, 4:40pm UTC](https://discuss.elastic.co/t/logstash-output-plugin-elastic-kafka-polling-behaviour/154043 "2018-10-25T16:40:00Z")
Is there anyway to disable the periodic polling of the hosts listed in the output? I am using the Kafka output plugin and I assume the behaviour here is similar with the Elastic plugin. I tried to be clever and create … --- ## [Do we need persistent queues if using Kafka input?](https://discuss.elastic.co/t/do-we-need-persistent-queues-if-using-kafka-input/154038)
**Author:** [@javadevmtl](https://discuss.elastic.co/u/javadevmtl)\ **Replies:** 0\ **Last updated:** [October 25, 2018, 3:55pm UTC](https://discuss.elastic.co/t/do-we-need-persistent-queues-if-using-kafka-input/154038 "2018-10-25T15:55:13Z")
If Logstash fails while inserting to Elastic but it has already committed the Kafka offset, we can lose data right? --- ## [Mutate Filter - Multiple Datatype Conversion](https://discuss.elastic.co/t/mutate-filter-multiple-datatype-conversion/153940)
**Author:** [@vishnuduttpv](https://discuss.elastic.co/u/vishnuduttpv)\ **Replies:** 1\ **Last updated:** [October 25, 2018, 3:51pm UTC](https://discuss.elastic.co/t/mutate-filter-multiple-datatype-conversion/153940 "2018-10-25T15:51:13Z")
Hi, I am using the following logstash input configuration. Previously I was using only the dissect filter. I added the mutate filter to change the datatypes for ease of use in KIBANA. filter { dissect { mapping =\>… --- ## [Logstash - Calculate a midpoint from a Geo polygon](https://discuss.elastic.co/t/logstash-calculate-a-midpoint-from-a-geo-polygon/154030)
**Author:** [@adde](https://discuss.elastic.co/u/adde)\ **Replies:** 1\ **Last updated:** [October 25, 2018, 3:08pm UTC](https://discuss.elastic.co/t/logstash-calculate-a-midpoint-from-a-geo-polygon/154030 "2018-10-25T15:08:17Z")
Hello, I want to calculate a midpoint from a geo-polygon. I know I should use the ruby filter but I don't know how. My formula looks like this: pmidlat = plat2 - ((plat2 - plat3) / 2) pmidlong = plong4 - ((plong4 - p… --- ## [Use Logstash Dissect for Java Stack Trace](https://discuss.elastic.co/t/use-logstash-dissect-for-java-stack-trace/153892)
**Author:** [@topcan5](https://discuss.elastic.co/u/topcan5)\ **Replies:** 2\ **Last updated:** [October 25, 2018, 2:37pm UTC](https://discuss.elastic.co/t/use-logstash-dissect-for-java-stack-trace/153892 "2018-10-25T14:37:17Z")
I am using logstash dissect plugin to parse my log. How can I use it to handle stacktrace? I am looking for something like %{stacktrace}. Can this be done by using dissect? Or I have to use Grok? 18-10-24 14:46:45 m… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=658) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=660)