# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=329 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 330 --- ## [Logstash - Improving speed of reading data from Redis queue](https://discuss.elastic.co/t/logstash-improving-speed-of-reading-data-from-redis-queue/241260)
**Author:** [@vasek](https://discuss.elastic.co/u/vasek)\ **Replies:** 0\ **Last updated:** [July 15, 2020, 9:56am UTC](https://discuss.elastic.co/t/logstash-improving-speed-of-reading-data-from-redis-queue/241260 "2020-07-15T09:56:09Z")
Hi, can you please help how to increase speed of reading messages from redis queue by Logstash? An application generates 100 mil. documents / day. I have 3 logstash instances. I've tried these input configuration: i… --- ## [Can we run Logstash API endpoint in HTTPS (secured)?](https://discuss.elastic.co/t/can-we-run-logstash-api-endpoint-in-https-secured/241251)
**Author:** [@Agniv\_Gon](https://discuss.elastic.co/u/Agniv_Gon)\ **Replies:** 0\ **Last updated:** [July 15, 2020, 8:36am UTC](https://discuss.elastic.co/t/can-we-run-logstash-api-endpoint-in-https-secured/241251 "2020-07-15T08:36:51Z")
In my logstash I have configured logstash.yml with below: http.host: "0.0.0.0" http.port: 9000 And it is working fine \[Api Webserver\] agent - Successfully started Logstash API endpoint {:port=\>9000} Is there a way we… --- ## [Logstash pipeline execution order](https://discuss.elastic.co/t/logstash-pipeline-execution-order/240539)
**Author:** [@tbege](https://discuss.elastic.co/u/tbege)\ **Replies:** 2\ **Last updated:** [July 15, 2020, 4:17am UTC](https://discuss.elastic.co/t/logstash-pipeline-execution-order/240539 "2020-07-15T04:17:38Z")
Hello, I am working on a sync mechanism between a DB2 database and Elasticsearch, I am transporting the data with the JDBC plugin. However, I am running into the issue of the pipeline order, meaning: I have 3 masterd… --- ## [exception=\>#\ **Author:** [@srbhklkrn](https://discuss.elastic.co/u/srbhklkrn)\ **Replies:** 7\ **Last updated:** [July 15, 2020, 2:37am UTC](https://discuss.elastic.co/t/exception-logstash-unexpected-character-code-58-expected-a-valid-value-number-string-array-object-true-false-or-null/241198 "2020-07-15T02:37:37Z") I need to parse and have all the JSON object in one message but it is splitting into different messages as you can see in stdout below JSON: \[ { "SOURCE": "Source A", "Model": "Mo… --- ## [When \`.logstash\_jdbc\_last\_run\` is created?](https://discuss.elastic.co/t/when-logstash-jdbc-last-run-is-created/240902)
**Author:** [@ohoroyoi](https://discuss.elastic.co/u/ohoroyoi)\ **Replies:** 1\ **Last updated:** [July 15, 2020, 2:18am UTC](https://discuss.elastic.co/t/when-logstash-jdbc-last-run-is-created/240902 "2020-07-15T02:18:48Z")
Even if I create a file arbitrarily, the text in the file is erased and sql\_last\_value is not created. I originally had this file, but I deleted it. So I can't get sql\_last\_value. How should I solve it? --- ## [Logstash Cache](https://discuss.elastic.co/t/logstash-cache/241202)
**Author:** [@csgeek](https://discuss.elastic.co/u/csgeek)\ **Replies:** 4\ **Last updated:** [July 14, 2020, 10:34pm UTC](https://discuss.elastic.co/t/logstash-cache/241202 "2020-07-14T22:34:58Z")
I'm working on an event filter written in ruby that needs to do a lookup. I have the data stored in a JSON file that i'm loading locally and I'd like to know if there is any concept of a shared cache that I can leverage… --- ## [Rename Pipeline.yml](https://discuss.elastic.co/t/rename-pipeline-yml/241192)
**Author:** [@Thiago\_Paiva](https://discuss.elastic.co/u/Thiago_Paiva)\ **Replies:** 2\ **Last updated:** [July 14, 2020, 9:36pm UTC](https://discuss.elastic.co/t/rename-pipeline-yml/241192 "2020-07-14T21:36:07Z")
Hello folks, I currently need to upload my pipeline.yml to git, But my logstash exists in 5 (Five) servers. And Three(3) servers use the same pipeline.yml and Two use other pipeline.yml. So I want to rename one like Pip… --- ## [Cisco syslogs showing up as grokparsefailure when sent to Logstash](https://discuss.elastic.co/t/cisco-syslogs-showing-up-as-grokparsefailure-when-sent-to-logstash/240587)
**Author:** [@jabberwalker](https://discuss.elastic.co/u/jabberwalker)\ **Replies:** 4\ **Last updated:** [July 14, 2020, 8:39pm UTC](https://discuss.elastic.co/t/cisco-syslogs-showing-up-as-grokparsefailure-when-sent-to-logstash/240587 "2020-07-14T20:39:31Z")
I am working through a Udemy course to familiarize myself with Elasticsearch but I've ran into a wall when applying it to our current environment when it comes to Logstash. I am trying to send data from Cisco syslog but… --- ## [Logstash unable to reach Elasticsearch: port issues](https://discuss.elastic.co/t/logstash-unable-to-reach-elasticsearch-port-issues/241187)
**Author:** [@rahulnama](https://discuss.elastic.co/u/rahulnama)\ **Replies:** 5\ **Last updated:** [July 14, 2020, 8:38pm UTC](https://discuss.elastic.co/t/logstash-unable-to-reach-elasticsearch-port-issues/241187 "2020-07-14T20:38:04Z")
Hello Team I deployed Elasticsearch on Kubernetes and it's working fine. Also, I created an ingress so it's redirecting from 80 =\> 9200. But logstash is looking for 9200 so connection is failing. However, if I try h… --- ## [Running logstash using systemd with specific settings](https://discuss.elastic.co/t/running-logstash-using-systemd-with-specific-settings/241184)
**Author:** [@kaplan.dani](https://discuss.elastic.co/u/kaplan.dani)\ **Replies:** 0\ **Last updated:** [July 14, 2020, 7:29pm UTC](https://discuss.elastic.co/t/running-logstash-using-systemd-with-specific-settings/241184 "2020-07-14T19:29:14Z")
I run logstash with this command line and it works correctly: bin/logstash -f my\_pipeline.conf --config.reload.automatic --path.settings /etc/logstash when I run it using systemctl (on debian) it will not work, I guess… --- ## [Your socket connection to the server was not read from or written to within the timeout period](https://discuss.elastic.co/t/your-socket-connection-to-the-server-was-not-read-from-or-written-to-within-the-timeout-period/237638)
**Author:** [@Annette1](https://discuss.elastic.co/u/Annette1)\ **Replies:** 12\ **Last updated:** [July 14, 2020, 6:58pm UTC](https://discuss.elastic.co/t/your-socket-connection-to-the-server-was-not-read-from-or-written-to-within-the-timeout-period/237638 "2020-07-14T18:58:08Z")
when using logstash to s3 output. have tried many strategies of file\_size, file\_time, an rotation strategy to fix with no joy. --- ## [Grok working online but not in Kibana debugger or Logstash](https://discuss.elastic.co/t/grok-working-online-but-not-in-kibana-debugger-or-logstash/241176)
**Author:** [@HassaanSalik](https://discuss.elastic.co/u/HassaanSalik)\ **Replies:** 1\ **Last updated:** [July 14, 2020, 6:07pm UTC](https://discuss.elastic.co/t/grok-working-online-but-not-in-kibana-debugger-or-logstash/241176 "2020-07-14T18:07:48Z")
Hi, I am trying to parse gc-log file through grok as shown in this tutorial. The following grok is working fine in both Grok Constructor and Grok Debugger. %{TIMESTAMP\_ISO8601:timestamp}: %{NUMBER:jvm\_time}: \\\[%{DATA:… --- ## [Can't connect filebeat to logstash on the separate instances](https://discuss.elastic.co/t/cant-connect-filebeat-to-logstash-on-the-separate-instances/241132)
**Author:** [@Denis\_Denis](https://discuss.elastic.co/u/Denis_Denis)\ **Replies:** 2\ **Last updated:** [July 14, 2020, 5:28pm UTC](https://discuss.elastic.co/t/cant-connect-filebeat-to-logstash-on-the-separate-instances/241132 "2020-07-14T17:28:31Z")
Hello, I'm new with Elastic Stack and currently trying to build logging system with infrastructure type filebeat -\> logstash -\> elastic search -\> kibana. I'm using the separate AWS EC2 instance for logstash, elastic sea… --- ## [Logstash Worker Threads Blocked by Pipeline](https://discuss.elastic.co/t/logstash-worker-threads-blocked-by-pipeline/241165)
**Author:** [@mattb1](https://discuss.elastic.co/u/mattb1)\ **Replies:** 0\ **Last updated:** [July 14, 2020, 4:44pm UTC](https://discuss.elastic.co/t/logstash-worker-threads-blocked-by-pipeline/241165 "2020-07-14T16:44:56Z")
Hi, Im using the 7.4.2 Version of the elastic Stack. One of my Logstash Pipelines Keeps blocking the Worker Threads after a While. Essentialy grinding all of the ingestion to a halt. As no other Pipelines can be execu… --- ## [Seperate dynamic comma seperated field into multiple fields](https://discuss.elastic.co/t/seperate-dynamic-comma-seperated-field-into-multiple-fields/241103)
**Author:** [@sz5ps5](https://discuss.elastic.co/u/sz5ps5)\ **Replies:** 1\ **Last updated:** [July 14, 2020, 4:30pm UTC](https://discuss.elastic.co/t/seperate-dynamic-comma-seperated-field-into-multiple-fields/241103 "2020-07-14T16:30:18Z")
Apologies I am new to ELK and Logstash so this may be a simple solution. I have a field that is currently in the elk stack, it is passing through Logstash. The field is dynamic in that is can have any number of values. … --- ## [Adding additional fields for Logstash Kinesis Input](https://discuss.elastic.co/t/adding-additional-fields-for-logstash-kinesis-input/240542)
**Author:** [@beyondcompare](https://discuss.elastic.co/u/beyondcompare)\ **Replies:** 5\ **Last updated:** [July 14, 2020, 4:19pm UTC](https://discuss.elastic.co/t/adding-additional-fields-for-logstash-kinesis-input/240542 "2020-07-14T16:19:04Z")
Hi everyone, I'd like to ask for some help from you please about adding additional fields to Logstash Kinesis Input. I have three logstash servers consuming a Kinesis Stream using Kinesis Input Plugin. Example like th… --- ## [Splitting nested /hierarchal documents in Logstash](https://discuss.elastic.co/t/splitting-nested-hierarchal-documents-in-logstash/241156)
**Author:** [@Ahmed\_Adly](https://discuss.elastic.co/u/Ahmed_Adly)\ **Replies:** 0\ **Last updated:** [July 14, 2020, 3:33pm UTC](https://discuss.elastic.co/t/splitting-nested-hierarchal-documents-in-logstash/241156 "2020-07-14T15:33:58Z")
Hello, I'm using Logstash to read external REST API and feed elastic search. the API data looks like this: { ArrayOfParent: \[ Parent: { ArrayOfChildTypeOne: \[ {ID: 1, Name: child1} \], ArrayOfChil… --- ## [Logstash unable to parse with kv filter](https://discuss.elastic.co/t/logstash-unable-to-parse-with-kv-filter/240820)
**Author:** [@msk\_76](https://discuss.elastic.co/u/msk_76)\ **Replies:** 17\ **Last updated:** [July 14, 2020, 3:13pm UTC](https://discuss.elastic.co/t/logstash-unable-to-parse-with-kv-filter/240820 "2020-07-14T15:13:30Z")
My logstash configuration is working fine for most of the log cases but below exceptions are not working. My log has following format The json part consist of usual format "" "" both have space separator. In normal… --- ## [Make the elasticsearch-input return only one hit](https://discuss.elastic.co/t/make-the-elasticsearch-input-return-only-one-hit/241146)
**Author:** [@Alessio\_Creo](https://discuss.elastic.co/u/Alessio_Creo)\ **Replies:** 0\ **Last updated:** [July 14, 2020, 2:10pm UTC](https://discuss.elastic.co/t/make-the-elasticsearch-input-return-only-one-hit/241146 "2020-07-14T14:10:20Z")
Hi, I'm trying to use Logstash to ingest just one hit from an Elasticsearch index. This is my input configuration input{ elasticsearch{ hosts=\>\["logmgrappa.risorse.int:9200","logmgrappb.risorse.int:9200","logmgr… --- ## [Logstash date filter and Timestamp](https://discuss.elastic.co/t/logstash-date-filter-and-timestamp/241110)
**Author:** [@Nikita\_Shetty](https://discuss.elastic.co/u/Nikita_Shetty)\ **Replies:** 3\ **Last updated:** [July 14, 2020, 12:29pm UTC](https://discuss.elastic.co/t/logstash-date-filter-and-timestamp/241110 "2020-07-14T12:29:06Z")
I have field called created\_date in my table and the data is in the format - 2020-07-14 10:41:24.879884 +05:30. Want to parse this data and map it to the timestamp field in logstash. Used the below log pattern in the con… --- ## [Get json from Http poller , store it in local variable and use it in elastic search output](https://discuss.elastic.co/t/get-json-from-http-poller-store-it-in-local-variable-and-use-it-in-elastic-search-output/239610)
**Author:** [@Abolurah](https://discuss.elastic.co/u/Abolurah)\ **Replies:** 4\ **Last updated:** [July 14, 2020, 9:56am UTC](https://discuss.elastic.co/t/get-json-from-http-poller-store-it-in-local-variable-and-use-it-in-elastic-search-output/239610 "2020-07-14T09:56:18Z")
Hi All I'm trying to configure the Logstash, that it will poll credentials from Http poller plugin store it is some local variable, get data from local elastic search (multiple Inputs), and will send it to remote AWS El… --- ## [Grok parse failure splitting in to 3 blocks](https://discuss.elastic.co/t/grok-parse-failure-splitting-in-to-3-blocks/239983)
**Author:** [@karlm](https://discuss.elastic.co/u/karlm)\ **Replies:** 3\ **Last updated:** [July 14, 2020, 9:21am UTC](https://discuss.elastic.co/t/grok-parse-failure-splitting-in-to-3-blocks/239983 "2020-07-14T09:21:33Z")
I have a log file, with multiple lines like so 03-Jul-2020 10:24:00.773 +01:00 \[ERR\] this is the error message message I have a grok filter trying to extract the log in 3 separate blocks 1. DateTime 2. Priority 3. … --- ## [\_dateparsefailure in tags](https://discuss.elastic.co/t/dateparsefailure-in-tags/240676)
**Author:** [@Govinda8594](https://discuss.elastic.co/u/Govinda8594)\ **Replies:** 2\ **Last updated:** [July 14, 2020, 7:21am UTC](https://discuss.elastic.co/t/dateparsefailure-in-tags/240676 "2020-07-14T07:21:24Z")
Hello Team, i am using filebeat to ship all logs to logstash,this logs are docker container logs here is my filebeat.yml file filebeat.inputs: - type: log paths: - 'var/lib/docker/containers/ec037f29884667f4e27… --- ## [Reading cloud event avro Message](https://discuss.elastic.co/t/reading-cloud-event-avro-message/241070)
**Author:** [@revooru\_nischal](https://discuss.elastic.co/u/revooru_nischal)\ **Replies:** 0\ **Last updated:** [July 14, 2020, 7:15am UTC](https://discuss.elastic.co/t/reading-cloud-event-avro-message/241070 "2020-07-14T07:15:38Z")
Hi , I was trying to find a plugin which is able to read a kafka message which is of format \[cloud event avro\](https://github.com/cloudevents/spec/blob/v1.0/avro-format.md eg : {"data\_base64":"LFNSVl9Qb3N0T25lVHJhbnNh… --- ## [How to check client\_ip ipv4 or ipv6?](https://discuss.elastic.co/t/how-to-check-client-ip-ipv4-or-ipv6/240654)
**Author:** [@Champion\_Xie](https://discuss.elastic.co/u/Champion_Xie)\ **Replies:** 7\ **Last updated:** [July 14, 2020, 3:30am UTC](https://discuss.elastic.co/t/how-to-check-client-ip-ipv4-or-ipv6/240654 "2020-07-14T03:30:59Z")
how to use mutate add\_field check client\_ip version v4 or v6? if \['client\_ip'\].length \>15 { mutate{ add\_field =\> { "ip\_type" =\> "ipv6" } } } else { mutate{ add\_field =\> { "ip\_type" =\> "ipv4" } }… --- ## [Filebeat not receiving logs and error in logstash](https://discuss.elastic.co/t/filebeat-not-receiving-logs-and-error-in-logstash/240653)
**Author:** [@Sai\_Avinash\_Duddupud](https://discuss.elastic.co/u/Sai_Avinash_Duddupud)\ **Replies:** 12\ **Last updated:** [July 12, 2020, 4:07pm UTC](https://discuss.elastic.co/t/filebeat-not-receiving-logs-and-error-in-logstash/240653 "2020-07-12T16:07:52Z")
I am trying to setup filebeat and logstash on my server1 and send data to elasticsearch located on server2 and visualize it using kibana. Following are filebeat logs and when i run filebeat test output it showed the res… --- ## [Logstash csv output plugin problem](https://discuss.elastic.co/t/logstash-csv-output-plugin-problem/240691)
**Author:** [@kase](https://discuss.elastic.co/u/kase)\ **Replies:** 8\ **Last updated:** [July 14, 2020, 12:39am UTC](https://discuss.elastic.co/t/logstash-csv-output-plugin-problem/240691 "2020-07-14T00:39:51Z")
Hi there, I want to output elasticsearch documents to csv file with using logstash, but faced some problems... tail -f /var/log/logstash/logstash-plain.log \[2020-07-10T11:11:02,561\]\[ERROR\]\[logstash.javapipeline \]\[e… --- ## [Need help with resolving - security\_exception - no permissions for \[indices:admin/create\] and User \[name=logstash](https://discuss.elastic.co/t/need-help-with-resolving-security-exception-no-permissions-for-indices-admin-create-and-user-name-logstash/241038)
**Author:** [@roligupt](https://discuss.elastic.co/u/roligupt)\ **Replies:** 0\ **Last updated:** [July 13, 2020, 11:30pm UTC](https://discuss.elastic.co/t/need-help-with-resolving-security-exception-no-permissions-for-indices-admin-create-and-user-name-logstash/241038 "2020-07-13T23:30:53Z")
I just deployed a new ELK instance in K8 platform. i keep getting these errors - \[logstash.outputs.elasticsearch\]\[main\] retrying failed action with response code: 403 ({"type"=\>"security\_exception", "reason"=\>"no permis… --- ## [Cannot import json template for cloudfront logs to cluster](https://discuss.elastic.co/t/cannot-import-json-template-for-cloudfront-logs-to-cluster/240958)
**Author:** [@gadg3ts](https://discuss.elastic.co/u/gadg3ts)\ **Replies:** 6\ **Last updated:** [July 13, 2020, 9:10pm UTC](https://discuss.elastic.co/t/cannot-import-json-template-for-cloudfront-logs-to-cluster/240958 "2020-07-13T21:10:28Z")
Hi, I'm attempting to get my cloudfront logs into a local elasticsearch cluster (7.8.0) and am having a bit of trouble. The template I'm attempting to import is from here https://aws.amazon.com/premiumsupport/knowledg… --- ## [Ruby help with array of json objects](https://discuss.elastic.co/t/ruby-help-with-array-of-json-objects/241022)
**Author:** [@rajsolanki](https://discuss.elastic.co/u/rajsolanki)\ **Replies:** 14\ **Last updated:** [July 13, 2020, 7:42pm UTC](https://discuss.elastic.co/t/ruby-help-with-array-of-json-objects/241022 "2020-07-13T19:42:52Z")
hi sorry if its repeat question. In my document there are few fields which are array of json objects. I want to extract one of json array object's value and assign it to my new field. So for that i m using ruby code t… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=328) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=330)