# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=843 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 844 --- ## [Logstash how to transform custom json data to multiple data](https://discuss.elastic.co/t/logstash-how-to-transform-custom-json-data-to-multiple-data/114642)
**Author:** [@beiming](https://discuss.elastic.co/u/beiming)\ **Replies:** 2\ **Last updated:** [January 11, 2018, 6:58am UTC](https://discuss.elastic.co/t/logstash-how-to-transform-custom-json-data-to-multiple-data/114642 "2018-01-11T06:58:20Z")
Hi, I use Logstash 6.1.1, and use http plugin put json data to logstash. some data like: "@timestamp": "2018-01-09T02:50:59.776Z", "headers": { ...... }, "key1": "value1" "key2": "value2" "logs": \["2018-01-09 0… --- ## [Cast mongodb \_id object to string](https://discuss.elastic.co/t/cast-mongodb-id-object-to-string/114844)
**Author:** [@mahajan344](https://discuss.elastic.co/u/mahajan344)\ **Replies:** 1\ **Last updated:** [January 11, 2018, 5:56am UTC](https://discuss.elastic.co/t/cast-mongodb-id-object-to-string/114844 "2018-01-11T05:56:51Z")
Hi, I am working on creating a pipeline to get data from MongoDB to Elasticsearch using Logstash. I am using dbschema mongodb jdbc drivers. I am able to connect to database using driver but I am facing issue with \_id .… --- ## [\[Throttle filter\]How can I string this field?](https://discuss.elastic.co/t/throttle-filter-how-can-i-string-this-field/114781)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 1\ **Last updated:** [January 11, 2018, 1:21am UTC](https://discuss.elastic.co/t/throttle-filter-how-can-i-string-this-field/114781 "2018-01-11T01:21:38Z")
Hi, I'm struggling with the throttle filter. What I want seems pretty straight forward but I can't get it to work. They "KEY" value needs to contain the fields you want to throttle. I figured out "field" throttles the … --- ## [Unable to remove %{host} field when using kafka output plugin with codec plain](https://discuss.elastic.co/t/unable-to-remove-host-field-when-using-kafka-output-plugin-with-codec-plain/114684)
**Author:** [@berlin](https://discuss.elastic.co/u/berlin)\ **Replies:** 8\ **Last updated:** [January 10, 2018, 10:30pm UTC](https://discuss.elastic.co/t/unable-to-remove-host-field-when-using-kafka-output-plugin-with-codec-plain/114684 "2018-01-10T22:30:02Z")
Hi, Please review the below configuration: input { sqs { access\_key\_id =\> "" secret\_access\_key =\> "" polling\_frequency =\> "1" queue =\> "xxx" region =\> "eu-west-1" threads =\> "1" codec =\> p… --- ## [Unenclosed quoted field on line 1 - frustrating error message](https://discuss.elastic.co/t/unenclosed-quoted-field-on-line-1-frustrating-error-message/114959)
**Author:** [@dorj1234](https://discuss.elastic.co/u/dorj1234)\ **Replies:** 2\ **Last updated:** [January 10, 2018, 9:04pm UTC](https://discuss.elastic.co/t/unenclosed-quoted-field-on-line-1-frustrating-error-message/114959 "2018-01-10T21:04:30Z")
Hello, I wanted to share some frustration, and maybe even hope for a solution - the logs are extremely unhelpful when it comes to my specific logstash experience. Here is an example where logs tell me there is an un-en… --- ## [Logstash config file processing order](https://discuss.elastic.co/t/logstash-config-file-processing-order/114927)
**Author:** [@Ranieri\_Mazili\_de\_Ol](https://discuss.elastic.co/u/Ranieri_Mazili_de_Ol)\ **Replies:** 1\ **Last updated:** [January 10, 2018, 8:45pm UTC](https://discuss.elastic.co/t/logstash-config-file-processing-order/114927 "2018-01-10T20:45:17Z")
Considering that the basic structure of a logstash.conf is: input { ... } filter { ... } output { ... } Looking above we assume that logstash will process the incoming events on that order (input, filter, out… --- ## [Logstash: Output: Success or Failure Condition handling and Email trigger](https://discuss.elastic.co/t/logstash-output-success-or-failure-condition-handling-and-email-trigger/114473)
**Author:** [@Kiran4Elastic](https://discuss.elastic.co/u/Kiran4Elastic)\ **Replies:** 0\ **Last updated:** [January 8, 2018, 9:05am UTC](https://discuss.elastic.co/t/logstash-output-success-or-failure-condition-handling-and-email-trigger/114473 "2018-01-08T09:05:04Z")
0 down vote favorite Task: Pulling data from SQLServer and Pushing records to Elasticsearch. Achieving this through triggering the logstash cmd after certain upstream conditional triggers finished. Planning to do thi… --- ## [Logstash failing after installing x-pack, elastic stack 6](https://discuss.elastic.co/t/logstash-failing-after-installing-x-pack-elastic-stack-6/112826)
**Author:** [@yishain11](https://discuss.elastic.co/u/yishain11)\ **Replies:** 2\ **Last updated:** [January 10, 2018, 7:31pm UTC](https://discuss.elastic.co/t/logstash-failing-after-installing-x-pack-elastic-stack-6/112826 "2018-01-10T19:31:41Z")
Hi everyone! I try to add to my Elastic stack 6 x-pack, on my CenOS7 VM. Everything worked fine before x-pack was installed. I followed closely this manual: Installing X-Pack | X-Pack for the Elastic Stack \[6.2\] | Ela… --- ## [Is it possible to send logs from Logstash 2.x to Logstash 6.x?](https://discuss.elastic.co/t/is-it-possible-to-send-logs-from-logstash-2-x-to-logstash-6-x/114907)
**Author:** [@amruth](https://discuss.elastic.co/u/amruth)\ **Replies:** 5\ **Last updated:** [January 10, 2018, 5:38pm UTC](https://discuss.elastic.co/t/is-it-possible-to-send-logs-from-logstash-2-x-to-logstash-6-x/114907 "2018-01-10T17:38:45Z")
Hi, I am trying to setup to send logs from Logstash 2.x to Logstash 6.x. Can someone please say if it's possible? Thanks --- ## [Deleted Dictionary data not removed from memory until restart](https://discuss.elastic.co/t/deleted-dictionary-data-not-removed-from-memory-until-restart/114756)
**Author:** [@cdouglas](https://discuss.elastic.co/u/cdouglas)\ **Replies:** 1\ **Last updated:** [January 10, 2018, 4:07pm UTC](https://discuss.elastic.co/t/deleted-dictionary-data-not-removed-from-memory-until-restart/114756 "2018-01-10T16:07:11Z")
Hello, I am using Logstash 5 with dictionary lookups and a refresh interval of 60 seconds. I am having an issue where if I remove a line from a yml file, the change is not picked up until i restart Logstash. Adding or … --- ## [Maximum throughput Logstash can handle](https://discuss.elastic.co/t/maximum-throughput-logstash-can-handle/114861)
**Author:** [@ilpc](https://discuss.elastic.co/u/ilpc)\ **Replies:** 1\ **Last updated:** [January 10, 2018, 3:53pm UTC](https://discuss.elastic.co/t/maximum-throughput-logstash-can-handle/114861 "2018-01-10T15:53:20Z")
I have a very huge and old log gathering system which is based on daily cronjobs and HDFS. This system is not good for real-time analysis so I decided to introduce Logstash. My system produces 850GB of log each day. I… --- ## [Logstash Netflow module](https://discuss.elastic.co/t/logstash-netflow-module/114118)
**Author:** [@DarkVodka34](https://discuss.elastic.co/u/DarkVodka34)\ **Replies:** 6\ **Last updated:** [January 10, 2018, 3:28pm UTC](https://discuss.elastic.co/t/logstash-netflow-module/114118 "2018-01-10T15:28:35Z")
Howdy im very new to linux and have been following a tut on getting this all working. Im having big trouble getting the module to install I get this error root@OMV-MediaServer:/usr/share/logstash# bin/logstash --module… --- ## [How to remove keyword type in logstash config?](https://discuss.elastic.co/t/how-to-remove-keyword-type-in-logstash-config/114903)
**Author:** [@John\_Doe](https://discuss.elastic.co/u/John_Doe)\ **Replies:** 0\ **Last updated:** [January 10, 2018, 3:12pm UTC](https://discuss.elastic.co/t/how-to-remove-keyword-type-in-logstash-config/114903 "2018-01-10T15:12:47Z")
Is there any way to drop columns for keyword type? I can't find any post for it and please help me. Actually I don't need keyword types for text. Thanks. --- ## [Docker syslog with logstash tcp input plugin](https://discuss.elastic.co/t/docker-syslog-with-logstash-tcp-input-plugin/114899)
**Author:** [@rokopoko](https://discuss.elastic.co/u/rokopoko)\ **Replies:** 0\ **Last updated:** [January 10, 2018, 2:56pm UTC](https://discuss.elastic.co/t/docker-syslog-with-logstash-tcp-input-plugin/114899 "2018-01-10T14:56:52Z")
Hi, is it safe to use the docker syslog logging Driver with the tcp Input plugin and the persistent Queue enabled in logstash? Do logging Events in case of back pressure (Input Queue full) correctly prevent the syslog t… --- ## [Replace timestamp from logfile - dateparsefailure](https://discuss.elastic.co/t/replace-timestamp-from-logfile-dateparsefailure/114800)
**Author:** [@A.Klos](https://discuss.elastic.co/u/A.Klos)\ **Replies:** 6\ **Last updated:** [January 10, 2018, 2:47pm UTC](https://discuss.elastic.co/t/replace-timestamp-from-logfile-dateparsefailure/114800 "2018-01-10T14:47:12Z")
Hi, my logs starting with: 10.01.2018 | 07:43:13,961 | INFO | 09.01.2018 | 23:59:02,119 | DEBUG | I tried following: %{DATE\_EU:Datum} | (%{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second}%{ISO8601\_TIMEZONE:timezone}) B… --- ## [Logstash Kafka Output How to specify the Kafka Partition](https://discuss.elastic.co/t/logstash-kafka-output-how-to-specify-the-kafka-partition/114759)
**Author:** [@ajstark123](https://discuss.elastic.co/u/ajstark123)\ **Replies:** 1\ **Last updated:** [January 10, 2018, 2:44pm UTC](https://discuss.elastic.co/t/logstash-kafka-output-how-to-specify-the-kafka-partition/114759 "2018-01-10T14:44:51Z")
We have a filebeat sending logs to logstash 1. Logstash 1 writes the logs to Kafka. Logstash 2 read the logs from kafka and writes the logs to elastic. This pipeline is work we have logs going into elastic. The issu… --- ## [No results found - Kibana](https://discuss.elastic.co/t/no-results-found-kibana/113813)
**Author:** [@wmcgilvery](https://discuss.elastic.co/u/wmcgilvery)\ **Replies:** 4\ **Last updated:** [January 10, 2018, 1:42pm UTC](https://discuss.elastic.co/t/no-results-found-kibana/113813 "2018-01-10T13:42:46Z")
I don't always check this server but have found out today that I am getting no results showing in Kibana. There are no errors. Elasticsearch log \[2018-01-02 11:45:12,114\]\[INFO \]\[env \] \[Justin Ham… --- ## [Logstash error log](https://discuss.elastic.co/t/logstash-error-log/114869)
**Author:** [@alpha\_gamma](https://discuss.elastic.co/u/alpha_gamma)\ **Replies:** 1\ **Last updated:** [January 10, 2018, 12:37pm UTC](https://discuss.elastic.co/t/logstash-error-log/114869 "2018-01-10T12:37:25Z")
logstash-plain-2018-01-09.loglogstash-plain-2018-01-09.log\] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of #, … --- ## [How to perform join with two or more csv files into one index of elastic search using logstash](https://discuss.elastic.co/t/how-to-perform-join-with-two-or-more-csv-files-into-one-index-of-elastic-search-using-logstash/114865)
**Author:** [@Jagadeesh](https://discuss.elastic.co/u/Jagadeesh)\ **Replies:** 1\ **Last updated:** [January 10, 2018, 12:20pm UTC](https://discuss.elastic.co/t/how-to-perform-join-with-two-or-more-csv-files-into-one-index-of-elastic-search-using-logstash/114865 "2018-01-10T12:20:38Z")
I have created two CSV files A.csv and B.csv. A.csv has the fields id,name,city etc. (10 records) B.csv has fields id,age,blood\_group etc (10 records) In both the above files id field is common. so can I create an ind… --- ## [Receive this error](https://discuss.elastic.co/t/receive-this-error/114817)
**Author:** [@sbampa](https://discuss.elastic.co/u/sbampa)\ **Replies:** 8\ **Last updated:** [January 10, 2018, 11:48am UTC](https://discuss.elastic.co/t/receive-this-error/114817 "2018-01-10T11:48:53Z")
Hi all, i'm receiving the following errors every 2-5 mins: \[2018-01-10T09:55:35,117\]\[WARN \]\[logstash.outputs.elasticsearch\] Marking url as dead. Last error: \[LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnre… --- ## [How to parse csv using logstash](https://discuss.elastic.co/t/how-to-parse-csv-using-logstash/114823)
**Author:** [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)\ **Replies:** 7\ **Last updated:** [January 10, 2018, 11:33am UTC](https://discuss.elastic.co/t/how-to-parse-csv-using-logstash/114823 "2018-01-10T11:33:01Z")
I have CSV files where columns are in following sequence--\> File1: A,B,C,D File2: A,C,D,B I have to parse only A and B columns. how to map this and push to ES. --- ## [Wrong indexing if log is set to DEBUG](https://discuss.elastic.co/t/wrong-indexing-if-log-is-set-to-debug/114831)
**Author:** [@sbampa](https://discuss.elastic.co/u/sbampa)\ **Replies:** 1\ **Last updated:** [January 10, 2018, 11:11am UTC](https://discuss.elastic.co/t/wrong-indexing-if-log-is-set-to-debug/114831 "2018-01-10T11:11:12Z")
Hi all, i have a jboss that produces logs as follow: \[2018-01-10 00:01:03,032 DEBUG\] \[csmf\_37803-8AD3F44ACA4061B5F48ADF6851B4937E\] \[com.notio.report.web.ReportServlet\] GetRemoteHost: 172.16.32.6 \[2018-01-10 00:01:04,46… --- ## [Logstash command not found error](https://discuss.elastic.co/t/logstash-command-not-found-error/114797)
**Author:** [@alpha\_gamma](https://discuss.elastic.co/u/alpha_gamma)\ **Replies:** 9\ **Last updated:** [January 10, 2018, 11:08am UTC](https://discuss.elastic.co/t/logstash-command-not-found-error/114797 "2018-01-10T11:08:06Z")
I have ubuntu 16.04 system, I ahve logstash installed in /etc/logstash the command sudo service logstash start works well but the command /bin/logstash -f logstash.conf does work I have one more location /usr/share/… --- ## [Replace empty field with a value](https://discuss.elastic.co/t/replace-empty-field-with-a-value/114723)
**Author:** [@cilzzz](https://discuss.elastic.co/u/cilzzz)\ **Replies:** 2\ **Last updated:** [January 10, 2018, 8:37am UTC](https://discuss.elastic.co/t/replace-empty-field-with-a-value/114723 "2018-01-10T08:37:54Z")
Hi, I have a log file that contains a field called Status. i have two values: one null and the other OK. by null i mean there are no value. in mu logstash config file i used : mutate { gsub =\> \[ "Status … --- ## [Log processing at scale](https://discuss.elastic.co/t/log-processing-at-scale/114773)
**Author:** [@shwesinhan](https://discuss.elastic.co/u/shwesinhan)\ **Replies:** 0\ **Last updated:** [January 10, 2018, 2:18am UTC](https://discuss.elastic.co/t/log-processing-at-scale/114773 "2018-01-10T02:18:14Z")
Hi all, is anyway to pull down the aws cloudwatch loggroup's log from two logstash hosts at scale? does logstash input support auto scalability / load balancing for coudwatch log input? i'm using logstash v.6.1.1. t… --- ## [Handle EventTime msgpack extension in logstash-codec-fluent](https://discuss.elastic.co/t/handle-eventtime-msgpack-extension-in-logstash-codec-fluent/114788)
**Author:** [@cosmo0920](https://discuss.elastic.co/u/cosmo0920)\ **Replies:** 0\ **Last updated:** [January 10, 2018, 5:41am UTC](https://discuss.elastic.co/t/handle-eventtime-msgpack-extension-in-logstash-codec-fluent/114788 "2018-01-10T05:41:07Z")
Hi all, I'm tried to add new Fluentd v1.0/v0.14's EventTime msgpack protocol extension into logstash-codec-fluent. It is already pull requested in https://github.com/logstash-plugins/logstash-codec-fluent/pull/18. Cur… --- ## [SNMP Trap Monitoring with Logstash](https://discuss.elastic.co/t/snmp-trap-monitoring-with-logstash/114742)
**Author:** [@swasheck](https://discuss.elastic.co/u/swasheck)\ **Replies:** 1\ **Last updated:** [January 9, 2018, 9:00pm UTC](https://discuss.elastic.co/t/snmp-trap-monitoring-with-logstash/114742 "2018-01-09T21:00:01Z")
Currently running ELK 6.1.1 (Docker sebp/elk) and trying to get SNMP trap monitoring functional. My current config is as follows: input { snmptrap { type =\> "snmptrap" host =\> "0.0.0.0" p… --- ## [IRC Input Plugin - Identify with nickserv?](https://discuss.elastic.co/t/irc-input-plugin-identify-with-nickserv/114350)
**Author:** [@egdetti](https://discuss.elastic.co/u/egdetti)\ **Replies:** 1\ **Last updated:** [January 9, 2018, 7:18pm UTC](https://discuss.elastic.co/t/irc-input-plugin-identify-with-nickserv/114350 "2018-01-09T19:18:27Z")
Is there a way to get the IRC Input plugin to identify with the irc server's nickserv upon connection? Many IRC channels limit access to identified users only. Thanks for the help. --- ## [Json\_encode returns null](https://discuss.elastic.co/t/json-encode-returns-null/114728)
**Author:** [@Sebastian\_Fratini](https://discuss.elastic.co/u/Sebastian_Fratini)\ **Replies:** 5\ **Last updated:** [January 9, 2018, 7:07pm UTC](https://discuss.elastic.co/t/json-encode-returns-null/114728 "2018-01-09T19:07:56Z")
I am parsing data from the pm2 manager for node and it works fine, however some of my data ends in a text field instead of an structure. This is my logstash configuration: if \[fields\]\[type\] == "pm2" { json { … --- ## [Allow duplicate entires into logstash](https://discuss.elastic.co/t/allow-duplicate-entires-into-logstash/114702)
**Author:** [@Milton\_Sam\_Moses](https://discuss.elastic.co/u/Milton_Sam_Moses)\ **Replies:** 2\ **Last updated:** [January 9, 2018, 6:33pm UTC](https://discuss.elastic.co/t/allow-duplicate-entires-into-logstash/114702 "2018-01-09T18:33:49Z")
Hello, Experts I have 2000 entries in a csv file, after ingesting it to elasticsearch I see only 505 hits. My requirement is to have all the 2000 entries even though it has duplicates. Is there a way ? -Sam [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=842) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=844)