# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=920 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 921 --- ## [Preprocess Tweets before indexation](https://discuss.elastic.co/t/preprocess-tweets-before-indexation/98081)
**Author:** [@Melvyn\_Peignon](https://discuss.elastic.co/u/Melvyn_Peignon)\ **Replies:** 1\ **Last updated:** [August 24, 2017, 8:14pm UTC](https://discuss.elastic.co/t/preprocess-tweets-before-indexation/98081 "2017-08-24T20:14:17Z")
I want to index tweets using logstash and elasticsearch. I am using the twitter plugin. Now the tweet that are sent contain too many useless information for my application. I would like to select only some fields and ren… --- ## [Winlogbeat collecting custom event views possible?](https://discuss.elastic.co/t/winlogbeat-collecting-custom-event-views-possible/98010)
**Author:** [@g4nj4wizard](https://discuss.elastic.co/u/g4nj4wizard)\ **Replies:** 1\ **Last updated:** [August 24, 2017, 8:02pm UTC](https://discuss.elastic.co/t/winlogbeat-collecting-custom-event-views-possible/98010 "2017-08-24T20:02:38Z")
Hi, i wonder if it is possible to configure winlogbeat to collect a custom created event view? Or do i have to generate a forwarder? I cant find my custom views in powershell with get-winevent. --- ## [Adding filebeat index patterns to use filebeat index or just use custom index without adding filebeat index patterns](https://discuss.elastic.co/t/adding-filebeat-index-patterns-to-use-filebeat-index-or-just-use-custom-index-without-adding-filebeat-index-patterns/97980)
**Author:** [@BentCoder](https://discuss.elastic.co/u/BentCoder)\ **Replies:** 1\ **Last updated:** [August 24, 2017, 8:01pm UTC](https://discuss.elastic.co/t/adding-filebeat-index-patterns-to-use-filebeat-index-or-just-use-custom-index-without-adding-filebeat-index-patterns/97980 "2017-08-24T20:01:50Z")
Hi, If I use filebeat-%{+yyyy.MM.dd} as index name, every day I get new index created in ES. I then can use filebeat-\* in Kibana to see logs. For this, I have to add index patterns and load template. If I use custo… --- ## [Aggregate more than one metric with Metrics filter plugin](https://discuss.elastic.co/t/aggregate-more-than-one-metric-with-metrics-filter-plugin/98283)
**Author:** [@keithn](https://discuss.elastic.co/u/keithn)\ **Replies:** 0\ **Last updated:** [August 24, 2017, 7:22pm UTC](https://discuss.elastic.co/t/aggregate-more-than-one-metric-with-metrics-filter-plugin/98283 "2017-08-24T19:22:15Z")
I’m using Metricbeat to ship system metrics (CPU, Memory) to Logstash. Is it possible to configure the Logstash Metrics filter plugin to aggregate more than one metric at a time? I want to get the min, max and mean for… --- ## [Cannot run logstash as a service, with active pipeline](https://discuss.elastic.co/t/cannot-run-logstash-as-a-service-with-active-pipeline/98244)
**Author:** [@pshah7](https://discuss.elastic.co/u/pshah7)\ **Replies:** 9\ **Last updated:** [August 24, 2017, 4:38pm UTC](https://discuss.elastic.co/t/cannot-run-logstash-as-a-service-with-active-pipeline/98244 "2017-08-24T16:38:32Z")
Hello, I have installed ELK stack v5.5.2 and is working completely fine. I have started logstash as a service but it fails to initialize the pipeline. So every time I have to start logstash pipeline with /usr/share/lo… --- ## [No output from TCP input](https://discuss.elastic.co/t/no-output-from-tcp-input/97753)
**Author:** [@capaneus](https://discuss.elastic.co/u/capaneus)\ **Replies:** 3\ **Last updated:** [August 24, 2017, 3:53pm UTC](https://discuss.elastic.co/t/no-output-from-tcp-input/97753 "2017-08-24T15:53:34Z")
Hi All, I’m trying logstash with python TCP input, I followed some sample code here: Adapted a bit the code, but I’m sending the exact same message through tcp. From python output: (MainThread) 2017-08-21 12:07:35… --- ## [How to determine if field contains a value?](https://discuss.elastic.co/t/how-to-determine-if-field-contains-a-value/97842)
**Author:** [@inhinyera16](https://discuss.elastic.co/u/inhinyera16)\ **Replies:** 3\ **Last updated:** [August 24, 2017, 2:38pm UTC](https://discuss.elastic.co/t/how-to-determine-if-field-contains-a-value/97842 "2017-08-24T14:38:55Z")
Hello, Here’s my sample data in a field called “additional text” “additionalText” : \[ "A “commonMIBAlarm” event has occurred, from xxxxxx device, named xxxxxx Severity=major ComponentID=Navigation System&Name=Navigati… --- ## [Xml parser with xpath giving wrong result in xml filter plugin](https://discuss.elastic.co/t/xml-parser-with-xpath-giving-wrong-result-in-xml-filter-plugin/97666)
**Author:** [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Replies:** 5\ **Last updated:** [August 24, 2017, 2:33pm UTC](https://discuss.elastic.co/t/xml-parser-with-xpath-giving-wrong-result-in-xml-filter-plugin/97666 "2017-08-24T14:33:07Z")
Hi All i am having a xml filter, which looks like this xml { source =\> "message" target =\> "message\_parsed" add\_tag =\> \["xml\_parsed"\] remove\_namespaces =\> true store\_xml =\> … --- ## [Logstash trying to parse string into date match only in production](https://discuss.elastic.co/t/logstash-trying-to-parse-string-into-date-match-only-in-production/98246)
**Author:** [@Yasho](https://discuss.elastic.co/u/Yasho)\ **Replies:** 0\ **Last updated:** [August 24, 2017, 2:08pm UTC](https://discuss.elastic.co/t/logstash-trying-to-parse-string-into-date-match-only-in-production/98246 "2017-08-24T14:08:48Z")
This a totally strange behaviour i am seeing in logstash, I am processing a csv file, where the first column is a date filed, in general when we run the logstash it runs smooth for about 2 - 4 hrs, after which we start … --- ## [SSL with Logstash 5.4 run on IBM JRE 8](https://discuss.elastic.co/t/ssl-with-logstash-5-4-run-on-ibm-jre-8/98201)
**Author:** [@kodgule](https://discuss.elastic.co/u/kodgule)\ **Replies:** 1\ **Last updated:** [August 24, 2017, 2:05pm UTC](https://discuss.elastic.co/t/ssl-with-logstash-5-4-run-on-ibm-jre-8/98201 "2017-08-24T14:05:40Z")
How can I enable Logstash to accept SSL traffic from Filebeat? In my case, I am running Logstash with IBM JRE 8. I know it works fine over Oracle JDK/ Open JDK. But I have a requirement to use IBM JDK, hence the ask. H… --- ## [Logstash is creating a csv file with 0 bytes](https://discuss.elastic.co/t/logstash-is-creating-a-csv-file-with-0-bytes/98097)
**Author:** [@sudi\_2611](https://discuss.elastic.co/u/sudi_2611)\ **Replies:** 2\ **Last updated:** [August 24, 2017, 12:31pm UTC](https://discuss.elastic.co/t/logstash-is-creating-a-csv-file-with-0-bytes/98097 "2017-08-24T12:31:54Z")
i am new to logstash, from few days i have noticed that logstash has started creating 0 byte file -rw-r–r-- 1 logstash logstash 1859370519 Aug 21 19:57 fsimage -r–r--r-- 1 logstash logstash 0 Aug 23 09:49 fsimage.csv.1… --- ## [Logstash kafka input plugin:can not read data from kafka](https://discuss.elastic.co/t/logstash-kafka-input-plugin-can-not-read-data-from-kafka/98164)
**Author:** [@xiaoxinba](https://discuss.elastic.co/u/xiaoxinba)\ **Replies:** 2\ **Last updated:** [August 24, 2017, 12:28pm UTC](https://discuss.elastic.co/t/logstash-kafka-input-plugin-can-not-read-data-from-kafka/98164 "2017-08-24T12:28:06Z")
Now,my logstash can not consume data from kafka; logstash configuration: input { kafka { bootstrap\_servers =\> group\_id =\> } } filter{ XXXX } output{ elasticsearch{XXX} } when i see logstash comsuer offset from kafka, … --- ## [Add a tag if a field exists](https://discuss.elastic.co/t/add-a-tag-if-a-field-exists/97634)
**Author:** [@Bitdog](https://discuss.elastic.co/u/Bitdog)\ **Replies:** 3\ **Last updated:** [August 24, 2017, 11:46am UTC](https://discuss.elastic.co/t/add-a-tag-if-a-field-exists/97634 "2017-08-24T11:46:26Z")
I’ve been working with Logstash for about 6 weeks. Trying to tag a message based on a field. When the grok match fails I get a \_grokparsefailure tag. Here is the add\_tag section and the entire filter below. When I ne… --- ## [Nested Objects in a json Response displayed in Kibnana](https://discuss.elastic.co/t/nested-objects-in-a-json-response-displayed-in-kibnana/97565)
**Author:** [@giridharan78](https://discuss.elastic.co/u/giridharan78)\ **Replies:** 4\ **Last updated:** [August 24, 2017, 11:11am UTC](https://discuss.elastic.co/t/nested-objects-in-a-json-response-displayed-in-kibnana/97565 "2017-08-24T11:11:00Z")
Continuing the discussion from I would like to access nested data from an array: The data inside the array are not getting indexed and hence not displayed on the kibana dashboard. What is the best way to process the dat… --- ## [Logstash server parameter dependency](https://discuss.elastic.co/t/logstash-server-parameter-dependency/98163)
**Author:** [@Shrawan\_Bhagwat](https://discuss.elastic.co/u/Shrawan_Bhagwat)\ **Replies:** 0\ **Last updated:** [August 24, 2017, 6:51am UTC](https://discuss.elastic.co/t/logstash-server-parameter-dependency/98163 "2017-08-24T06:51:17Z")
Hi all, i am using logstash from more than 2 years. i have recently observed that same configuration of logstash is not working on some of my servers. troubleshooting: I have started logstash in debug mode, and found … --- ## [Beats: regarding backpressure](https://discuss.elastic.co/t/beats-regarding-backpressure/95869)
**Author:** [@dawiro](https://discuss.elastic.co/u/dawiro)\ **Replies:** 2\ **Last updated:** [August 24, 2017, 6:48am UTC](https://discuss.elastic.co/t/beats-regarding-backpressure/95869 "2017-08-24T06:48:14Z")
Hi, What i’ve read on the subject of backpressure says that when logstash becomes overloaded it will apply backpressure to throttle tranmission rates from clients… Can i ask what logstash uses as the determinant of bei… --- ## [Logstash 2.2.0 on windows 10](https://discuss.elastic.co/t/logstash-2-2-0-on-windows-10/97759)
**Author:** [@sunilmchaudhari](https://discuss.elastic.co/u/sunilmchaudhari)\ **Replies:** 2\ **Last updated:** [August 24, 2017, 5:06am UTC](https://discuss.elastic.co/t/logstash-2-2-0-on-windows-10/97759 "2017-08-24T05:06:13Z")
Hi, I am starting logstash 2.2.0 with file input on windows 10 OS. But it says, Unable to find JRuby. If you are a user, this is a bug. If you are a developer, please run ‘rake bootstrap’. Running ‘rake’ requires the… --- ## [Unable to register last access file](https://discuss.elastic.co/t/unable-to-register-last-access-file/97873)
**Author:** [@jogoinar10](https://discuss.elastic.co/u/jogoinar10)\ **Replies:** 3\ **Last updated:** [August 24, 2017, 1:32am UTC](https://discuss.elastic.co/t/unable-to-register-last-access-file/97873 "2017-08-24T01:32:44Z")
I observed that when I use elasticsearch as Input in logstash, it does not register the last instance it sends. Ex. my server got disconnected, so it will start to overwrite again the data. Another thing is that if my … --- ## [Logstash 429 errors when pushing data to Elastic Search](https://discuss.elastic.co/t/logstash-429-errors-when-pushing-data-to-elastic-search/98140)
**Author:** [@sgunnala](https://discuss.elastic.co/u/sgunnala)\ **Replies:** 1\ **Last updated:** [August 24, 2017, 12:18am UTC](https://discuss.elastic.co/t/logstash-429-errors-when-pushing-data-to-elastic-search/98140 "2017-08-24T00:18:11Z")
Filebeats -\> Kafka 0.9 -\>Logstash 5.0.2 -\> ElasticSearch 5.2 We push more then 30k/sec events from Logstash to ES using multiple indexers consuming from Kafka. some peak numbers we see upto 70 mil per 30 mins. ES is 18 … --- ## [Clean install and running as a service: no files in /etc/logstash/conf.d/\*](https://discuss.elastic.co/t/clean-install-and-running-as-a-service-no-files-in-etc-logstash-conf-d/97920)
**Author:** [@Zilean](https://discuss.elastic.co/u/Zilean)\ **Replies:** 4\ **Last updated:** [August 23, 2017, 9:36pm UTC](https://discuss.elastic.co/t/clean-install-and-running-as-a-service-no-files-in-etc-logstash-conf-d/97920 "2017-08-23T21:36:42Z")
Hi, I am setting up logstash 5.5.1 in a RHEL 7 and my java version is 1.8.0\_144 (Oracle). I have been following the tutorials and it seems that the pipeline configuration files are placed in the /etc/logstash/conf.d di… --- ## [\[\[main\]\ **Author:** [@leonard\_halfstater](https://discuss.elastic.co/u/leonard_halfstater)\ **Replies:** 1\ **Last updated:** [August 23, 2017, 9:23pm UTC](https://discuss.elastic.co/t/main-beats-error-logstash-pipeline-a-plugin-had-an-unrecoverable-error-will-restart-this-plugin/97936 "2017-08-23T21:23:54Z") I am getting this error when i am trying to give input as beats to logstash and elasticsearch as an output --- ## [Logstash conf reload error](https://discuss.elastic.co/t/logstash-conf-reload-error/98085)
**Author:** [@user1](https://discuss.elastic.co/u/user1)\ **Replies:** 1\ **Last updated:** [August 23, 2017, 9:05pm UTC](https://discuss.elastic.co/t/logstash-conf-reload-error/98085 "2017-08-23T21:05:59Z")
Hi , PLZ help me… logstash not able to listen filebeat output \[2017-08-23T19:06:43,864\]\[ERROR\]\[logstash.agent \] Pipeline aborted due to error {:exception=\>#\ **Author:** [@bm1391](https://discuss.elastic.co/u/bm1391)\ **Replies:** 1\ **Last updated:** [August 23, 2017, 8:51pm UTC](https://discuss.elastic.co/t/a-question-setting-up-logstash-with-raspberry-pi/97830 "2017-08-23T20:51:42Z") Hi all, I am setting up a raspberry pi 3 to send syslogs (rsyslog) to a server running logstash. I have installed filebeat on the raspberry pi in order to ship the logs to the logstash server. For some reason, the logst… --- ## [Picking substring from field in logstash](https://discuss.elastic.co/t/picking-substring-from-field-in-logstash/97797)
**Author:** [@Vivek\_Sharma](https://discuss.elastic.co/u/Vivek_Sharma)\ **Replies:** 1\ **Last updated:** [August 23, 2017, 8:47pm UTC](https://discuss.elastic.co/t/picking-substring-from-field-in-logstash/97797 "2017-08-23T20:47:19Z")
Hi I have a log field in the pattern: PRODUCT\_GLOBAL\_20170706024756466\_0000000004069390 My requirement is to extract 0000000004069390 from the field and store it in a new field named FieldX. Can anyone please help me … --- ## [Geo location](https://discuss.elastic.co/t/geo-location/97758)
**Author:** [@shaharz](https://discuss.elastic.co/u/shaharz)\ **Replies:** 1\ **Last updated:** [August 23, 2017, 8:43pm UTC](https://discuss.elastic.co/t/geo-location/97758 "2017-08-23T20:43:24Z")
I would like to understand how does Elastic translate the Geo Location of users by their IP? How accurate is it indeed? And a general question : is it possible that an internet provider will assign the same IP (random … --- ## [Logstash cassandra pluggin](https://discuss.elastic.co/t/logstash-cassandra-pluggin/98125)
**Author:** [@bbhandari0121](https://discuss.elastic.co/u/bbhandari0121)\ **Replies:** 0\ **Last updated:** [August 23, 2017, 8:07pm UTC](https://discuss.elastic.co/t/logstash-cassandra-pluggin/98125 "2017-08-23T20:07:11Z")
Hi Team, I am trying to dump the data from Logstash to Cassandara, i didnt find any plugin under the logstash section. Can any one please provide me the solution to achieve this ? --- ## [Field value String manipulation in logstash filter](https://discuss.elastic.co/t/field-value-string-manipulation-in-logstash-filter/97726)
**Author:** [@Jonesthomas](https://discuss.elastic.co/u/Jonesthomas)\ **Replies:** 2\ **Last updated:** [August 23, 2017, 6:49pm UTC](https://discuss.elastic.co/t/field-value-string-manipulation-in-logstash-filter/97726 "2017-08-23T18:49:55Z")
I need to manipulate the string value of my json log, This is my sample log file {"LogLevel":"ERROR","LogMsg":"{\\"itemId\\":0,\\"module\\":\\"/curie/calendar/appointment\\",\\"action\\":\\"/getSlotByDate/{doctorId}/{date}\\",\\"e… --- ## [Carriage return and float to integer](https://discuss.elastic.co/t/carriage-return-and-float-to-integer/98106)
**Author:** [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\ **Replies:** 0\ **Last updated:** [August 23, 2017, 4:16pm UTC](https://discuss.elastic.co/t/carriage-return-and-float-to-integer/98106 "2017-08-23T16:16:19Z")
Hello, I try to upload new type of data. In flat files on format csv, each ten lines i have a return carriage. 101;2907035;/suivi\_message\_alerte;3;150328;150390;101;853531;/acc1200;0;0;;1593384;00;150330;0;2 101;29037… --- ## [JDBC plugin ignores and duplicates entities](https://discuss.elastic.co/t/jdbc-plugin-ignores-and-duplicates-entities/98092)
**Author:** [@MayaK](https://discuss.elastic.co/u/MayaK)\ **Replies:** 0\ **Last updated:** [August 23, 2017, 2:03pm UTC](https://discuss.elastic.co/t/jdbc-plugin-ignores-and-duplicates-entities/98092 "2017-08-23T14:03:53Z")
Hi everyone, I tried to move my data from a single postgreSQL table to Elasticsearch. I have like 6,6m rows and I get the same amount of documents created within my index. The problem now is that logstash creates like… --- ## [Support for REDIS zsets v2](https://discuss.elastic.co/t/support-for-redis-zsets-v2/98090)
**Author:** [@amaury-d](https://discuss.elastic.co/u/amaury-d)\ **Replies:** 0\ **Last updated:** [August 23, 2017, 2:00pm UTC](https://discuss.elastic.co/t/support-for-redis-zsets-v2/98090 "2017-08-23T14:00:53Z")
Hello, Wanted to ping a second time on the pull request to support zset in logstash: I’ve added rspec and more documentation and got no news since 13th June. As time has passed there is now a small conflict on t… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=919) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=921)