# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=468 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 469 --- ## [Logstash exec input on cluster](https://discuss.elastic.co/t/logstash-exec-input-on-cluster/201935)
**Author:** [@Vijayakumar\_Kannan](https://discuss.elastic.co/u/Vijayakumar_Kannan)\ **Replies:** 0\ **Last updated:** [October 2, 2019, 10:02am UTC](https://discuss.elastic.co/t/logstash-exec-input-on-cluster/201935 "2019-10-02T10:02:18Z")
Hi, I want to run command "ssh ip -c ls" every 1 hour in logstash exec input plugin. If i have 3 logstash nodes in cluster every logstash nodes will execute this ? if yes how do i set only one logstash node execute the … --- ## [How to add characters to field containing the value to be compared for a match in translate pluging](https://discuss.elastic.co/t/how-to-add-characters-to-field-containing-the-value-to-be-compared-for-a-match-in-translate-pluging/201927)
**Author:** [@namali](https://discuss.elastic.co/u/namali)\ **Replies:** 0\ **Last updated:** [October 2, 2019, 9:43am UTC](https://discuss.elastic.co/t/how-to-add-characters-to-field-containing-the-value-to-be-compared-for-a-match-in-translate-pluging/201927 "2019-10-02T09:43:09Z")
common filed of csv1 and csv2 is citycode, but citycode in csv2 file has additional character 'CV' in this scenario how to read csv2 from translate pluging. and also read csv3 to get out put index. input csv files csv1 … --- ## [New opensource tool for preventing misconfiguration in Logstash](https://discuss.elastic.co/t/new-opensource-tool-for-preventing-misconfiguration-in-logstash/201902)
**Author:** [@Mickeyempow](https://discuss.elastic.co/u/Mickeyempow)\ **Replies:** 0\ **Last updated:** [October 2, 2019, 7:11am UTC](https://discuss.elastic.co/t/new-opensource-tool-for-preventing-misconfiguration-in-logstash/201902 "2019-10-02T07:11:32Z")
Hi All - we at empow developed a new opensource tool to prevent misconfiguration in Logstash - here's an article on how to use is: https://blog.empow.co/preventing-logstash-misconfiguration And here's the link to the Gi… --- ## [Logstash configuration issue(message=\>"Could not execute action)](https://discuss.elastic.co/t/logstash-configuration-issue-message-could-not-execute-action/201900)
**Author:** [@vijayakrishna.rg](https://discuss.elastic.co/u/vijayakrishna.rg)\ **Replies:** 0\ **Last updated:** [October 2, 2019, 7:04am UTC](https://discuss.elastic.co/t/logstash-configuration-issue-message-could-not-execute-action/201900 "2019-10-02T07:04:50Z")
Logstash: 7.3.0 Elastic: 7.3.0 filebeat: 7.3.0 i am configuring logstash on one of the Redhat machine to forward nginx logs, getting below issue, can someone please suggest me, Thanks. nginx.conf input { beats { … --- ## [Aggregate filter not catching up: input start\_position = "beginning"](https://discuss.elastic.co/t/aggregate-filter-not-catching-up-input-start-position-beginning/201428)
**Author:** [@Paul\_Ainslie](https://discuss.elastic.co/u/Paul_Ainslie)\ **Replies:** 5\ **Last updated:** [October 2, 2019, 3:14am UTC](https://discuss.elastic.co/t/aggregate-filter-not-catching-up-input-start-position-beginning/201428 "2019-10-02T03:14:38Z")
One of the things I love about logstash is the "sincedb" database which tracks current position and — if interrupted — remembers the location of where it left off. So, if Logstash is down for, say 4 hours, once restarte… --- ## [ELK with Docker - Setting up new Index Pattern](https://discuss.elastic.co/t/elk-with-docker-setting-up-new-index-pattern/201883)
**Author:** [@sud0](https://discuss.elastic.co/u/sud0)\ **Replies:** 0\ **Last updated:** [October 2, 2019, 1:26am UTC](https://discuss.elastic.co/t/elk-with-docker-setting-up-new-index-pattern/201883 "2019-10-02T01:26:37Z")
Hi guys. I'm trying to setup an ELK using Filebeat and Docker. I've created a new conf file under /etc/logstash/conf.d with the filters that I had from previous tests with ELK years ago. # UAT Web industry filter if … --- ## [HALP! Changing two backslashes to one!](https://discuss.elastic.co/t/halp-changing-two-backslashes-to-one/201707)
**Author:** [@joaociocca](https://discuss.elastic.co/u/joaociocca)\ **Replies:** 2\ **Last updated:** [October 1, 2019, 9:29pm UTC](https://discuss.elastic.co/t/halp-changing-two-backslashes-to-one/201707 "2019-10-01T21:29:03Z")
So, I thought this would be trivial, right? gsub =\> \["message","\\\\\\\\","\\\\"\] but... nop, it doesn't work. Logstash no like it. There's this field with domain and username that comes out as domain\\\\username from the log … --- ## [How to reload logstash conf files - Docker container](https://discuss.elastic.co/t/how-to-reload-logstash-conf-files-docker-container/201817)
**Author:** [@kmiklas](https://discuss.elastic.co/u/kmiklas)\ **Replies:** 3\ **Last updated:** [October 1, 2019, 7:00pm UTC](https://discuss.elastic.co/t/how-to-reload-logstash-conf-files-docker-container/201817 "2019-10-01T19:00:03Z")
What is the easiest way to tell Logstash to reload its configuration (.conf) files? I am now running the Elastic Stack, and Logstash, in a Docker container (sebp-elk, to be specific). Now, I am restarting the entire co… --- ## [Logstash JDBC index field with multiple values](https://discuss.elastic.co/t/logstash-jdbc-index-field-with-multiple-values/199498)
**Author:** [@Kevin\_Juliano](https://discuss.elastic.co/u/Kevin_Juliano)\ **Replies:** 5\ **Last updated:** [September 17, 2019, 8:40pm UTC](https://discuss.elastic.co/t/logstash-jdbc-index-field-with-multiple-values/199498 "2019-09-17T20:40:20Z")
Newbie here, I am using Logstash to index data on our PostgreSQL database, the thing is, we have a very complex query for just one record,e.g 1 table has many relations. Is it possible to format the index like this: { … --- ## [JDBC pipelines.yml setup](https://discuss.elastic.co/t/jdbc-pipelines-yml-setup/201850)
**Author:** [@fefo69](https://discuss.elastic.co/u/fefo69)\ **Replies:** 0\ **Last updated:** [October 1, 2019, 6:00pm UTC](https://discuss.elastic.co/t/jdbc-pipelines-yml-setup/201850 "2019-10-01T18:00:16Z")
Hi, I'm having an issue when trying to add two pipelines for jdbc ingest In my current setup I have 5 pipelines defined in pipelines.yml, in one of them I have one jdbc input definition which its working fine, but when… --- ## [Logstash config JSON parse error](https://discuss.elastic.co/t/logstash-config-json-parse-error/201828)
**Author:** [@rohitg](https://discuss.elastic.co/u/rohitg)\ **Replies:** 3\ **Last updated:** [October 1, 2019, 6:07pm UTC](https://discuss.elastic.co/t/logstash-config-json-parse-error/201828 "2019-10-01T18:07:22Z")
Hi, I am trying to parse a json file using logstash, but getting json parse error. \[WARN \]\[logstash.filters.json \] Error parsing json {:source=\>"message", :raw=\>"\\t\\t\\t\\t\\t1568996833,", :exception=\>#\ **Author:** [@humartinez](https://discuss.elastic.co/u/humartinez)\ **Replies:** 3\ **Last updated:** [October 1, 2019, 5:26pm UTC](https://discuss.elastic.co/t/date-in-correct-timezone/201801 "2019-10-01T17:26:45Z") Hi there, Im stuck in the way that Elastic stores datetime fields. Im in GMT-3 and I need to enter the date in a way that kibana could present it right. Currenlty Im using logstash to store some data in ES, and it stor… --- ## [Range Filter](https://discuss.elastic.co/t/range-filter/201843)
**Author:** [@Tony\_Chirillo](https://discuss.elastic.co/u/Tony_Chirillo)\ **Replies:** 1\ **Last updated:** [October 1, 2019, 5:16pm UTC](https://discuss.elastic.co/t/range-filter/201843 "2019-10-01T17:16:03Z")
With the Logstash range filter, is it possible to use a wildcard to target all similarly name fields? For example, instead of listing out all the event\_data fields one by one, I want to target all of them - something lik… --- ## [Performing a function if a value is inside an array](https://discuss.elastic.co/t/performing-a-function-if-a-value-is-inside-an-array/201826)
**Author:** [@Jasonespo](https://discuss.elastic.co/u/Jasonespo)\ **Replies:** 1\ **Last updated:** [October 1, 2019, 5:03pm UTC](https://discuss.elastic.co/t/performing-a-function-if-a-value-is-inside-an-array/201826 "2019-10-01T17:03:56Z")
This is my array: "device\_tag": \[ "DeviceA", "DeviceB" \] Essentially I want to do something if tag is Device A and something else if Device is tag B. if \[device\_tag\] == "DeviceA" { mutate { do something here } } … --- ## [Grok multi-line problem extracting data from between two markers](https://discuss.elastic.co/t/grok-multi-line-problem-extracting-data-from-between-two-markers/201811)
**Author:** [@astickler](https://discuss.elastic.co/u/astickler)\ **Replies:** 0\ **Last updated:** [October 1, 2019, 1:27pm UTC](https://discuss.elastic.co/t/grok-multi-line-problem-extracting-data-from-between-two-markers/201811 "2019-10-01T13:27:48Z")
My source is filebeat with correctly configured multi-line support. Using logstash (v5.6.16), I am trying to extract the data between two marker strings, the second of which is optional. This all works fine unless the so… --- ## [Pulling date from Elastic and convert it to local time zone](https://discuss.elastic.co/t/pulling-date-from-elastic-and-convert-it-to-local-time-zone/201345)
**Author:** [@saif3r](https://discuss.elastic.co/u/saif3r)\ **Replies:** 3\ **Last updated:** [October 1, 2019, 12:39pm UTC](https://discuss.elastic.co/t/pulling-date-from-elastic-and-convert-it-to-local-time-zone/201345 "2019-10-01T12:39:59Z")
Hello, I'm using Beats to to put data onto my Elasticsearch server. All machines incluing the monitored server are based in CET timezone. In Kibana i can see the @timestamp stored in CET, but when i pull it out using Lo… --- ## [Ship FortiGate logs to logstash](https://discuss.elastic.co/t/ship-fortigate-logs-to-logstash/201794)
**Author:** [@Asmaa\_Sarih](https://discuss.elastic.co/u/Asmaa_Sarih)\ **Replies:** 0\ **Last updated:** [October 1, 2019, 12:36pm UTC](https://discuss.elastic.co/t/ship-fortigate-logs-to-logstash/201794 "2019-10-01T12:36:10Z")
Hi, How to configure FortiGate firewall to send logs to Logstash --- ## [While moving data from sql sever to elastic search taking moretime(logstash)](https://discuss.elastic.co/t/while-moving-data-from-sql-sever-to-elastic-search-taking-moretime-logstash/201775)
**Author:** [@rameshkr1994](https://discuss.elastic.co/u/rameshkr1994)\ **Replies:** 0\ **Last updated:** [October 1, 2019, 10:29am UTC](https://discuss.elastic.co/t/while-moving-data-from-sql-sever-to-elastic-search-taking-moretime-logstash/201775 "2019-10-01T10:29:41Z")
Dear All. while moving data from sql server to elastic search using logstash taking more time why. i have 25 crores records in sql server side and i am trying to move but its taking more time and giving some error. i… --- ## [How to configure multiple pipelines with single input plugin?](https://discuss.elastic.co/t/how-to-configure-multiple-pipelines-with-single-input-plugin/201773)
**Author:** [@Ashish\_Khare](https://discuss.elastic.co/u/Ashish_Khare)\ **Replies:** 0\ **Last updated:** [October 1, 2019, 10:28am UTC](https://discuss.elastic.co/t/how-to-configure-multiple-pipelines-with-single-input-plugin/201773 "2019-10-01T10:28:02Z")
Hello, I have configured two separate piplines for two seperate log files. Both log streams are coming to single port 5044. How do I configure single input for both pipelines? --- ## [Ship Kafka logs](https://discuss.elastic.co/t/ship-kafka-logs/200706)
**Author:** [@Eva](https://discuss.elastic.co/u/Eva)\ **Replies:** 20\ **Last updated:** [October 1, 2019, 10:01am UTC](https://discuss.elastic.co/t/ship-kafka-logs/200706 "2019-10-01T10:01:06Z")
Hello, can anyone please give me some info or point me to the doc on how to ship kafka logs to elk. I did check some which did not work out. I also read that it is slightly different than other server logs that we ship T… --- ## [Filter and mapping by multiple fields in translate filter and dictionary](https://discuss.elastic.co/t/filter-and-mapping-by-multiple-fields-in-translate-filter-and-dictionary/201766)
**Author:** [@fabrei](https://discuss.elastic.co/u/fabrei)\ **Replies:** 0\ **Last updated:** [October 1, 2019, 9:37am UTC](https://discuss.elastic.co/t/filter-and-mapping-by-multiple-fields-in-translate-filter-and-dictionary/201766 "2019-10-01T09:37:20Z")
I have a logstash event which e.g. looks like this: { "field1": "value1", "field2": "value2" } Now I want to perform a translate filter which filters and maps by the two fields ("field1" or/and "field2"). So, when … --- ## [Logstash throwing error: Failed to publish events caused by: write tcp IP:PORT](https://discuss.elastic.co/t/logstash-throwing-error-failed-to-publish-events-caused-by-write-tcp-ip-port/201744)
**Author:** [@Ghassan\_Zein](https://discuss.elastic.co/u/Ghassan_Zein)\ **Replies:** 1\ **Last updated:** [October 1, 2019, 8:44am UTC](https://discuss.elastic.co/t/logstash-throwing-error-failed-to-publish-events-caused-by-write-tcp-ip-port/201744 "2019-10-01T08:44:38Z")
I am using filebeat on port 5064 to send logs with Index Pattern "dev-logs-arangodb-\*" from arangodb logs to logstash and then kibana. Filebeat is able to fetch the updates of logs, but not able to send them to logstash… --- ## [Strange error](https://discuss.elastic.co/t/strange-error/201437)
**Author:** [@BeMoore](https://discuss.elastic.co/u/BeMoore)\ **Replies:** 2\ **Last updated:** [October 1, 2019, 8:33am UTC](https://discuss.elastic.co/t/strange-error/201437 "2019-10-01T08:33:34Z")
Sep 27 17:22:55 ldnlinsocelk01 systemd\[1\]: Started logstash. Sep 27 17:22:55 ldnlinsocelk01 logstash\[15386\]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be r… --- ## [Logstash output to create an index based on 2 hour data](https://discuss.elastic.co/t/logstash-output-to-create-an-index-based-on-2-hour-data/201729)
**Author:** [@Bob94](https://discuss.elastic.co/u/Bob94)\ **Replies:** 1\ **Last updated:** [October 1, 2019, 6:46am UTC](https://discuss.elastic.co/t/logstash-output-to-create-an-index-based-on-2-hour-data/201729 "2019-10-01T06:46:54Z")
Hi, Is there in any way i can create an index from logstash output and segregate the index based on 2 hours index ? %{+yyyy.mm.dd.hh} will create the data in every hour...what i need is in every 2 hour.. --- ## [Slow performance of Logstash elasticsearch filter plugin](https://discuss.elastic.co/t/slow-performance-of-logstash-elasticsearch-filter-plugin/201609)
**Author:** [@msk\_76](https://discuss.elastic.co/u/msk_76)\ **Replies:** 4\ **Last updated:** [October 1, 2019, 6:30am UTC](https://discuss.elastic.co/t/slow-performance-of-logstash-elasticsearch-filter-plugin/201609 "2019-10-01T06:30:54Z")
I have around 4.5Million records in my input data of logstash to which I am doing a lookup of an existing index in ES using following ES filter plugin. This is just like adding department information to a user\_name field… --- ## [Logstash startup Error:Could not find or load main class Files\\Microsoft](https://discuss.elastic.co/t/logstash-startup-error-could-not-find-or-load-main-class-files-microsoft/201722)
**Author:** [@Kozanic](https://discuss.elastic.co/u/Kozanic)\ **Replies:** 0\ **Last updated:** [October 1, 2019, 4:55am UTC](https://discuss.elastic.co/t/logstash-startup-error-could-not-find-or-load-main-class-files-microsoft/201722 "2019-10-01T04:55:51Z")
Hi All, New to Elastic but not having a good start. Have managed to get Metricbeat working and reporting into out Cloud instance, now need to get Logstash working so I can run some SQL querys against a DB. Have gone t… --- ## [Issue with HTTP Poller when using headers - invalid url](https://discuss.elastic.co/t/issue-with-http-poller-when-using-headers-invalid-url/201549)
**Author:** [@Sketchy](https://discuss.elastic.co/u/Sketchy)\ **Replies:** 2\ **Last updated:** [October 1, 2019, 3:41am UTC](https://discuss.elastic.co/t/issue-with-http-poller-when-using-headers-invalid-url/201549 "2019-10-01T03:41:50Z")
Having trouble getting http\_poller to work querying an API, as soon as I try adding headers it starts failing with Invalid URL without any other details as to why its failing. There is little documentation on using heade… --- ## [.monitoring-logstash not working after i started using pipeline.yml](https://discuss.elastic.co/t/monitoring-logstash-not-working-after-i-started-using-pipeline-yml/201516)
**Author:** [@mrjayarajj](https://discuss.elastic.co/u/mrjayarajj)\ **Replies:** 2\ **Last updated:** [October 1, 2019, 3:23am UTC](https://discuss.elastic.co/t/monitoring-logstash-not-working-after-i-started-using-pipeline-yml/201516 "2019-10-01T03:23:27Z")
in my local i didnt use pipeline.yml , but i came to know that its recommended to use pipleline.yml l i was monitoring the logstash using kibana as below after i started using the pipleline.yml , this monitoring fun… --- ## [Fresh Install of 7.3.2 Crashes Creating Template](https://discuss.elastic.co/t/fresh-install-of-7-3-2-crashes-creating-template/201553)
**Author:** [@hackdefendr](https://discuss.elastic.co/u/hackdefendr)\ **Replies:** 1\ **Last updated:** [October 1, 2019, 1:26am UTC](https://discuss.elastic.co/t/fresh-install-of-7-3-2-crashes-creating-template/201553 "2019-10-01T01:26:05Z")
I hate starting over from scratch, but upgrading from 6.8.3 to 7.3.2 did not go well. So I removed and purged my whole ELK system. I still have all my JSON logs, so I can import them without any issue. With that said, … --- ## [Unable to create multiple index in elasticsearch from logstash with multiple input](https://discuss.elastic.co/t/unable-to-create-multiple-index-in-elasticsearch-from-logstash-with-multiple-input/201706)
**Author:** [@Asmaa\_Sarih](https://discuss.elastic.co/u/Asmaa_Sarih)\ **Replies:** 0\ **Last updated:** [October 1, 2019, 12:58am UTC](https://discuss.elastic.co/t/unable-to-create-multiple-index-in-elasticsearch-from-logstash-with-multiple-input/201706 "2019-10-01T00:58:13Z")
Hi Team, Can anyone help me in confugiring multiple indexes from multiple input with logstash, I am unable to create multiple index in elastic (with multiple if conditions) . One index is getting created but not both, … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=467) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=469)