# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=547 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 548 --- ## [Consume a rest Api like openweather websites by logstash to store in elasticsearch](https://discuss.elastic.co/t/consume-a-rest-api-like-openweather-websites-by-logstash-to-store-in-elasticsearch/180850)
**Author:** [@decodeit](https://discuss.elastic.co/u/decodeit)\ **Replies:** 5\ **Last updated:** [May 16, 2019, 12:34pm UTC](https://discuss.elastic.co/t/consume-a-rest-api-like-openweather-websites-by-logstash-to-store-in-elasticsearch/180850 "2019-05-16T12:34:23Z")
I am new to elasticsearch. What I am looking for is to be able to call any external webservices like openweather which requires any api key or either id and password, and be able to consume the response by logstash in js… --- ## [Installing pluging using bin/logstash-plugin install](https://discuss.elastic.co/t/installing-pluging-using-bin-logstash-plugin-install/181377)
**Author:** [@kumar4](https://discuss.elastic.co/u/kumar4)\ **Replies:** 1\ **Last updated:** [May 16, 2019, 12:23pm UTC](https://discuss.elastic.co/t/installing-pluging-using-bin-logstash-plugin-install/181377 "2019-05-16T12:23:47Z")
hi dears , i am trying to install a plugin i use the commande bin/logstash-plugin install : but no such file or directory; i want to know the exact directory of logstash plugin to execute this command --- ## [Logstash 6.7.1 - Events disappear after tcp input plugin in pipeline](https://discuss.elastic.co/t/logstash-6-7-1-events-disappear-after-tcp-input-plugin-in-pipeline/181346)
**Author:** [@ariemenschneider](https://discuss.elastic.co/u/ariemenschneider)\ **Replies:** 0\ **Last updated:** [May 16, 2019, 9:24am UTC](https://discuss.elastic.co/t/logstash-6-7-1-events-disappear-after-tcp-input-plugin-in-pipeline/181346 "2019-05-16T09:24:33Z")
Hi, we use LS 6.7.1 on RHEL 7.6 (installed with rpm). There we have about 20 pipelines running on 2 LS instances on different servers with ~3-4k syslog events / second. We use the central pipeline management feature of … --- ## [Help with formatting a logstash output exec using curl](https://discuss.elastic.co/t/help-with-formatting-a-logstash-output-exec-using-curl/181116)
**Author:** [@johnwood](https://discuss.elastic.co/u/johnwood)\ **Replies:** 6\ **Last updated:** [May 16, 2019, 9:15am UTC](https://discuss.elastic.co/t/help-with-formatting-a-logstash-output-exec-using-curl/181116 "2019-05-16T09:15:01Z")
Hi - I am really struggling with what I assume to be the correct escaping of the JSON payload of this curl command in a logstash output exec. Any tips or guidance or examples would be most welcome! Thanks John I cure… --- ## [Logstash not picking some conf files](https://discuss.elastic.co/t/logstash-not-picking-some-conf-files/181320)
**Author:** [@Nithani25](https://discuss.elastic.co/u/Nithani25)\ **Replies:** 0\ **Last updated:** [May 16, 2019, 6:52am UTC](https://discuss.elastic.co/t/logstash-not-picking-some-conf-files/181320 "2019-05-16T06:52:27Z")
Hi Team, I working on a setup where i have 5 conf files available on single node cluster(No cluster), the setup was working perfectly fine but off late i could observe that two of my conf files are not getting pickled u… --- ## [Logstash the pipeline stopped processing new events](https://discuss.elastic.co/t/logstash-the-pipeline-stopped-processing-new-events/180930)
**Author:** [@Haitianisgood](https://discuss.elastic.co/u/Haitianisgood)\ **Replies:** 4\ **Last updated:** [May 16, 2019, 1:45am UTC](https://discuss.elastic.co/t/logstash-the-pipeline-stopped-processing-new-events/180930 "2019-05-16T01:45:24Z")
ENV： logstash 6.2.4 kafka\_2.12-2.0.0 My logstash has running several months，but it did't provide of service when I check it today! The process of logstash was still available. The log report : \[ERROR\]\[logstash.pipeli… --- ## [Logstash 7.0.1 Centos 7 persistent queue vm held lock error](https://discuss.elastic.co/t/logstash-7-0-1-centos-7-persistent-queue-vm-held-lock-error/181083)
**Author:** [@JeremyC](https://discuss.elastic.co/u/JeremyC)\ **Replies:** 1\ **Last updated:** [May 15, 2019, 10:38pm UTC](https://discuss.elastic.co/t/logstash-7-0-1-centos-7-persistent-queue-vm-held-lock-error/181083 "2019-05-15T22:38:04Z")
Hello, I am trying to add Persistent queues to logstash, however i am getting the below error. And i not sure how it resolve it. I am not sure if issue with config or logstash itself. What i have done: I have stopped … --- ## [How to strip white space from all fields in logstash](https://discuss.elastic.co/t/how-to-strip-white-space-from-all-fields-in-logstash/181255)
**Author:** [@rahulkothanath](https://discuss.elastic.co/u/rahulkothanath)\ **Replies:** 5\ **Last updated:** [May 15, 2019, 8:51pm UTC](https://discuss.elastic.co/t/how-to-strip-white-space-from-all-fields-in-logstash/181255 "2019-05-15T20:51:26Z")
I am creating fields using grok match with position and resulting json fields contain white spaces. I need to trim the white space from all fields . input { stdin { } } filter { grok { match =\> { "message… --- ## [Grok parse failure and date format error](https://discuss.elastic.co/t/grok-parse-failure-and-date-format-error/181240)
**Author:** [@khergner](https://discuss.elastic.co/u/khergner)\ **Replies:** 3\ **Last updated:** [May 15, 2019, 6:21pm UTC](https://discuss.elastic.co/t/grok-parse-failure-and-date-format-error/181240 "2019-05-15T18:21:32Z")
Hi, I want to parse to below line but grok pattern get error. How can ı parse below line? 12/May/19 20:03:14 118.102.70.71 --- ## [Convert date](https://discuss.elastic.co/t/convert-date/181244)
**Author:** [@schneider](https://discuss.elastic.co/u/schneider)\ **Replies:** 3\ **Last updated:** [May 15, 2019, 6:08pm UTC](https://discuss.elastic.co/t/convert-date/181244 "2019-05-15T18:08:55Z")
I want to convert data 1,data 3, data 4 and data 5 to date but the log is ex: NormalTransac#0112019 Apr 29 13:30:49 -03:00#0112019/04/29 13:30:49 -03:00#0112019 Apr 29 13:30:49 -03:00#0111689530397.4535.1556555444285@t… --- ## [Netflow Module Not Creating ES Index](https://discuss.elastic.co/t/netflow-module-not-creating-es-index/180222)
**Author:** [@etsai7](https://discuss.elastic.co/u/etsai7)\ **Replies:** 9\ **Last updated:** [May 15, 2019, 5:00pm UTC](https://discuss.elastic.co/t/netflow-module-not-creating-es-index/180222 "2019-05-15T17:00:12Z")
Hello, When I launch logstash 6.5.4 with the netflow module, using the line: bin/logstash --modules netflow --setup -M netflow.var.input.udp.port=2055 it does not seem to create an ES index for Netflow as the docs s… --- ## [Logstash- getting specific lines](https://discuss.elastic.co/t/logstash-getting-specific-lines/181086)
**Author:** [@Godson\_Raju](https://discuss.elastic.co/u/Godson_Raju)\ **Replies:** 1\ **Last updated:** [May 15, 2019, 2:32pm UTC](https://discuss.elastic.co/t/logstash-getting-specific-lines/181086 "2019-05-15T14:32:22Z")
I was wondering if I can get the 10 lines following the Keyword "Exception" in the message field of a Tomcat log file. I am able to get the lines with a specific keyword but failed to get the lines following the line wit… --- ## [Bind Logstash monitoring API to different port](https://discuss.elastic.co/t/bind-logstash-monitoring-api-to-different-port/180909)
**Author:** [@jfreeman](https://discuss.elastic.co/u/jfreeman)\ **Replies:** 3\ **Last updated:** [May 15, 2019, 1:25pm UTC](https://discuss.elastic.co/t/bind-logstash-monitoring-api-to-different-port/180909 "2019-05-15T13:25:16Z")
According to the document https://www.elastic.co/guide/en/logstash/current/monitoring.html to bind monitoring API to a different port we have to launch Logstash with the --http.port doing so with port 443 I get the belo… --- ## [Removing fields with certain value (or if mapping is not found) from the output](https://discuss.elastic.co/t/removing-fields-with-certain-value-or-if-mapping-is-not-found-from-the-output/180941)
**Author:** [@saif3r](https://discuss.elastic.co/u/saif3r)\ **Replies:** 7\ **Last updated:** [May 15, 2019, 1:21pm UTC](https://discuss.elastic.co/t/removing-fields-with-certain-value-or-if-mapping-is-not-found-from-the-output/180941 "2019-05-15T13:21:25Z")
Hello Guys, I have a Logstash pipeline, responsible for pulling xml files. Each xml is split as suggested here by @Badger : https://discuss.elastic.co/t/turning-one-xml-into-multiple-events-where-not-all-values-are-alwa… --- ## [@timestamp to unix epoch conversion in Logstash 7.0.1](https://discuss.elastic.co/t/timestamp-to-unix-epoch-conversion-in-logstash-7-0-1/181142)
**Author:** [@pastechecker](https://discuss.elastic.co/u/pastechecker)\ **Replies:** 5\ **Last updated:** [May 15, 2019, 1:05pm UTC](https://discuss.elastic.co/t/timestamp-to-unix-epoch-conversion-in-logstash-7-0-1/181142 "2019-05-15T13:05:51Z")
Hello! I am attempting to convert @timestamp to unix epoch. When running in a online parser: https://repl.it/repls/FlatPristineEmulation it does what it supposed to do. My test configuration: input {stdin { }} fi… --- ## [How to parse multiple json files in hierarchical directory structure](https://discuss.elastic.co/t/how-to-parse-multiple-json-files-in-hierarchical-directory-structure/180147)
**Author:** [@stour](https://discuss.elastic.co/u/stour)\ **Replies:** 5\ **Last updated:** [May 15, 2019, 1:00pm UTC](https://discuss.elastic.co/t/how-to-parse-multiple-json-files-in-hierarchical-directory-structure/180147 "2019-05-15T13:00:51Z")
Hi, I am new in ELK Stack. I want to parse multiple json files from hierarchical directory structure. Following is the directory structure. data |-- 2017 | |-- 01 | | |-- 01 | | | |-- a.json | | | \`-… --- ## [Match multiple values into one field](https://discuss.elastic.co/t/match-multiple-values-into-one-field/181115)
**Author:** [@victor.nilsson](https://discuss.elastic.co/u/victor.nilsson)\ **Replies:** 3\ **Last updated:** [May 15, 2019, 12:45pm UTC](https://discuss.elastic.co/t/match-multiple-values-into-one-field/181115 "2019-05-15T12:45:21Z")
Hi, We want to match multiple values into one field from a single Document/Message. For example, the following message: "Testlog, Field1=value1,asdasda,asdasd,asdasd,Field2=value2" With the following grok patterns: "… --- ## [Use geoip plugin to load VLAN IDs](https://discuss.elastic.co/t/use-geoip-plugin-to-load-vlan-ids/181034)
**Author:** [@yasvanth](https://discuss.elastic.co/u/yasvanth)\ **Replies:** 4\ **Last updated:** [May 15, 2019, 12:44pm UTC](https://discuss.elastic.co/t/use-geoip-plugin-to-load-vlan-ids/181034 "2019-05-15T12:44:50Z")
Hi Team, I would like to know whether the my idea/approach is possible. Using geoip plugin I need to query mmdb which contains list of network mapped to vlan id as given below: netowork | vlan\_id| vlan\_description … --- ## [Merge two mongodb collections](https://discuss.elastic.co/t/merge-two-mongodb-collections/181177)
**Author:** [@Aymen\_Ben\_Moussa](https://discuss.elastic.co/u/Aymen_Ben_Moussa)\ **Replies:** 0\ **Last updated:** [May 15, 2019, 12:20pm UTC](https://discuss.elastic.co/t/merge-two-mongodb-collections/181177 "2019-05-15T12:20:37Z")
I have two mongodb collections Users and Transactions, since elasticsearch have different approaches when it comes to joins i got stuck on how to make these two collections into one index with nested fields, embeeded, p… --- ## [File input duplicates](https://discuss.elastic.co/t/file-input-duplicates/181169)
**Author:** [@Sali7007](https://discuss.elastic.co/u/Sali7007)\ **Replies:** 1\ **Last updated:** [May 15, 2019, 12:13pm UTC](https://discuss.elastic.co/t/file-input-duplicates/181169 "2019-05-15T12:13:21Z")
Hi I have a file which is rewritten daily with new data. Everything was working fine till I missed to update the file with new data. So, Logstash processed the old data itself again generating duplicate data. Is ther… --- ## [Capture events from only first half of a file in composite format](https://discuss.elastic.co/t/capture-events-from-only-first-half-of-a-file-in-composite-format/181164)
**Author:** [@Abhilash\_B](https://discuss.elastic.co/u/Abhilash_B)\ **Replies:** 0\ **Last updated:** [May 15, 2019, 11:29am UTC](https://discuss.elastic.co/t/capture-events-from-only-first-half-of-a-file-in-composite-format/181164 "2019-05-15T11:29:58Z")
I have files which looks like this. My logstash pipeline looks like this (Have omitted the output part intentionally). I require only the top part of the file which essentially consists of key value pairs as in single r… --- ## [Compatibility of Logstash 2.x with Elastic 7.X](https://discuss.elastic.co/t/compatibility-of-logstash-2-x-with-elastic-7-x/181160)
**Author:** [@maxdj007](https://discuss.elastic.co/u/maxdj007)\ **Replies:** 0\ **Last updated:** [May 15, 2019, 11:11am UTC](https://discuss.elastic.co/t/compatibility-of-logstash-2-x-with-elastic-7-x/181160 "2019-05-15T11:11:02Z")
I need a logstash version that runs with java 7 and hence I need to use the version 2.X. I need to verify if the version 2.x of Logstash is compatible with Elasticsearch version 7.X --- ## [Source file size and number of lines](https://discuss.elastic.co/t/source-file-size-and-number-of-lines/180986)
**Author:** [@saif3r](https://discuss.elastic.co/u/saif3r)\ **Replies:** 2\ **Last updated:** [May 15, 2019, 10:48am UTC](https://discuss.elastic.co/t/source-file-size-and-number-of-lines/180986 "2019-05-15T10:48:18Z")
Hello, I'm processing raw .xml file using file plugin in Logstash. I'm gathering some metadata about the file like path and host but I would like to get input file size and number of lines as well. Is that possible? If … --- ## [Logstash offline plugin installation-OpenTimeout](https://discuss.elastic.co/t/logstash-offline-plugin-installation-opentimeout/181022)
**Author:** [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Replies:** 1\ **Last updated:** [May 15, 2019, 10:23am UTC](https://discuss.elastic.co/t/logstash-offline-plugin-installation-opentimeout/181022 "2019-05-15T10:23:44Z")
hi all I installed logstash-input-jmx on logstash-6.6.0 (installed on windows) successfully. now i want to install it on a server with no internet access. based on the (https://www.elastic.co/guide/en/logstash/current/o… --- ## [Matching a nested field in a JSON formatted log](https://discuss.elastic.co/t/matching-a-nested-field-in-a-json-formatted-log/181132)
**Author:** [@martin\_g](https://discuss.elastic.co/u/martin_g)\ **Replies:** 2\ **Last updated:** [May 15, 2019, 10:19am UTC](https://discuss.elastic.co/t/matching-a-nested-field-in-a-json-formatted-log/181132 "2019-05-15T10:19:30Z")
I'm experimenting with logging with the Elastic stack. I have forced something to write messages to a logfile, which will be read by FileBeat and send to Logstash. I've managed to get my first grok filter working. Howe… --- ## [\[SOLVED\] Logstash memcached does not work as expected](https://discuss.elastic.co/t/solved-logstash-memcached-does-not-work-as-expected/181029)
**Author:** [@pastechecker](https://discuss.elastic.co/u/pastechecker)\ **Replies:** 5\ **Last updated:** [May 15, 2019, 10:06am UTC](https://discuss.elastic.co/t/solved-logstash-memcached-does-not-work-as-expected/181029 "2019-05-15T10:06:09Z")
Hello. I am attempting to set a value in a memcached using logstash 7.0.1 input { stdin { } } filter { grok { match =\> \["message", "%{GREEDYDATA:uri\_fqdn}$" \] } fingerprint { method =\> "MD5" enable\_metric =\> tr… --- ## [Logstash plugin installation issue- unable to download data](https://discuss.elastic.co/t/logstash-plugin-installation-issue-unable-to-download-data/181105)
**Author:** [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Replies:** 2\ **Last updated:** [May 15, 2019, 9:28am UTC](https://discuss.elastic.co/t/logstash-plugin-installation-issue-unable-to-download-data/181105 "2019-05-15T09:28:22Z")
hi all I am using logstash version 6.6.0 on windows, and now I am interested of logstash plugin installation. for this purpose, i ran following command in CMD which gives me Error. C:\\logstash-6.6.0\\bin\>logstash-plugin… --- ## [How to run logstash in background in unix?](https://discuss.elastic.co/t/how-to-run-logstash-in-background-in-unix/180779)
**Author:** [@mandar.raj](https://discuss.elastic.co/u/mandar.raj)\ **Replies:** 7\ **Last updated:** [May 15, 2019, 8:24am UTC](https://discuss.elastic.co/t/how-to-run-logstash-in-background-in-unix/180779 "2019-05-15T08:24:46Z")
How to run logstash in background in unix ? --- ## [\[Solved\]How to convert a json format string to json object for only one field of a full log text in ruby filter?](https://discuss.elastic.co/t/solved-how-to-convert-a-json-format-string-to-json-object-for-only-one-field-of-a-full-log-text-in-ruby-filter/180312)
**Author:** [@Adminstrator](https://discuss.elastic.co/u/Adminstrator)\ **Replies:** 6\ **Last updated:** [May 15, 2019, 7:41am UTC](https://discuss.elastic.co/t/solved-how-to-convert-a-json-format-string-to-json-object-for-only-one-field-of-a-full-log-text-in-ruby-filter/180312 "2019-05-15T07:41:38Z")
I use filebeat to collcet log.Then push log to kafka ,then use logstash to parse every log record.then push to another kafka again.then use another logstash push the log to elasticsearch. And the log is splited by '\\t'.… --- ## [Logstash, An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception](https://discuss.elastic.co/t/logstash-an-exceptioncaught-event-was-fired-and-it-reached-at-the-tail-of-the-pipeline-it-usually-means-the-last-handler-in-the-pipeline-did-not-handle-the-exception/180515)
**Author:** [@faba47305d2d194daaea](https://discuss.elastic.co/u/faba47305d2d194daaea)\ **Replies:** 4\ **Last updated:** [May 15, 2019, 4:10am UTC](https://discuss.elastic.co/t/logstash-an-exceptioncaught-event-was-fired-and-it-reached-at-the-tail-of-the-pipeline-it-usually-means-the-last-handler-in-the-pipeline-did-not-handle-the-exception/180515 "2019-05-15T04:10:15Z")
Hello, I am trying set up ELK stack on my server. I installed ELK on my server. now I am trying to connect Filebeat with Logstash. However I am having trouble with Logstash. On my logstash log, \[WARN \] 2019-05-10 16… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=546) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=548)