# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=324 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 325 --- ## [Bulk requests are retried indefinitely](https://discuss.elastic.co/t/bulk-requests-are-retried-indefinitely/242488)
**Author:** [@cjmarkham](https://discuss.elastic.co/u/cjmarkham)\ **Replies:** 2\ **Last updated:** [July 24, 2020, 5:45pm UTC](https://discuss.elastic.co/t/bulk-requests-are-retried-indefinitely/242488 "2020-07-24T17:45:14Z")
I have an issue where a bulk request fails from the ELS output plugin. After reading the docs, I saw that these requests are retried indefinitely. However, this blocks the whole pipeline so no new events are consumed. I… --- ## [Should I put the index template & policy in ElasticSearch Cluster? or In Logstash?](https://discuss.elastic.co/t/should-i-put-the-index-template-policy-in-elasticsearch-cluster-or-in-logstash/242532)
**Author:** [@abb](https://discuss.elastic.co/u/abb)\ **Replies:** 0\ **Last updated:** [July 24, 2020, 3:31pm UTC](https://discuss.elastic.co/t/should-i-put-the-index-template-policy-in-elasticsearch-cluster-or-in-logstash/242532 "2020-07-24T15:31:35Z")
I am having trouble finding the path to put my template and policy in ES or Logstash. I know logstash stores default template path at vendor -\> Path When I referred ES 7.8.0 documentation for policy (Here) & it also sa… --- ## [Logstash output plugin for slack](https://discuss.elastic.co/t/logstash-output-plugin-for-slack/242297)
**Author:** [@marttonso](https://discuss.elastic.co/u/marttonso)\ **Replies:** 1\ **Last updated:** [July 24, 2020, 8:34am UTC](https://discuss.elastic.co/t/logstash-output-plugin-for-slack/242297 "2020-07-24T08:34:35Z")
Hello, I'd like to get a few more urgent log messages into slack channel. They're not critical enough to route to Pagerduty and create alerts & incidents, otherwise I could send them to Slack from there. The output plu… --- ## [Using Logstash http input filter, what is the max file size limit can be sent to it?](https://discuss.elastic.co/t/using-logstash-http-input-filter-what-is-the-max-file-size-limit-can-be-sent-to-it/240839)
**Author:** [@Agniv\_Gon](https://discuss.elastic.co/u/Agniv_Gon)\ **Replies:** 1\ **Last updated:** [July 24, 2020, 7:06am UTC](https://discuss.elastic.co/t/using-logstash-http-input-filter-what-is-the-max-file-size-limit-can-be-sent-to-it/240839 "2020-07-24T07:06:55Z")
I am using Logstash 7.7.1 and using below conf for my logstash. I am trying to send .gz file and in header passing Content-Encoding: gzip which is working fine. But I am clueless about the max size limit which I can send… --- ## [Logstash http input plugin accepting gzip file but how to detect only text format?](https://discuss.elastic.co/t/logstash-http-input-plugin-accepting-gzip-file-but-how-to-detect-only-text-format/240683)
**Author:** [@Agniv\_Gon](https://discuss.elastic.co/u/Agniv_Gon)\ **Replies:** 4\ **Last updated:** [July 24, 2020, 6:51am UTC](https://discuss.elastic.co/t/logstash-http-input-plugin-accepting-gzip-file-but-how-to-detect-only-text-format/240683 "2020-07-24T06:51:22Z")
I am using the below conf for Logstash HTTP input plugin: I am trying to send .gz file and in header passing Content-Encoding: gzip. It is successfully doing its job but I am interested only in text format inside the gzi… --- ## [Trying to calculate differences between 2 timestamps](https://discuss.elastic.co/t/trying-to-calculate-differences-between-2-timestamps/242443)
**Author:** [@windowsrefund\_refund](https://discuss.elastic.co/u/windowsrefund_refund)\ **Replies:** 4\ **Last updated:** [July 24, 2020, 5:19am UTC](https://discuss.elastic.co/t/trying-to-calculate-differences-between-2-timestamps/242443 "2020-07-24T05:19:27Z")
input { generator { message =\> "Jul 24 03:35:03 myhost metricbeat: 2020-07-24T03:35:03.788Z#011INFO#011\[publisher\]#011pipeline/retry.go:225#011 done" count =\> 1 } } filter { grok { match =\> { "messag… --- ## [Send kafka topics to different elastic search hosts through logstash](https://discuss.elastic.co/t/send-kafka-topics-to-different-elastic-search-hosts-through-logstash/242446)
**Author:** [@ankit9787](https://discuss.elastic.co/u/ankit9787)\ **Replies:** 1\ **Last updated:** [July 24, 2020, 5:14am UTC](https://discuss.elastic.co/t/send-kafka-topics-to-different-elastic-search-hosts-through-logstash/242446 "2020-07-24T05:14:00Z")
I have created one logstash conf file and reading kafka topic and trying to send to different elastic search deployed on different servers. Can you please help, what I am doing wrong. Nothings happening. Below my code W… --- ## [Logstash SNS output failing](https://discuss.elastic.co/t/logstash-sns-output-failing/242273)
**Author:** [@Vishnuvardhan\_K](https://discuss.elastic.co/u/Vishnuvardhan_K)\ **Replies:** 7\ **Last updated:** [July 24, 2020, 5:08am UTC](https://discuss.elastic.co/t/logstash-sns-output-failing/242273 "2020-07-24T05:08:58Z")
Hi, I am trying to send the logs to the SNS topic through Filebeat and Logstash. But it is failing with the attached below the message. When I try to send the output to stdout, then it works without any issues. Can so… --- ## [Ship logs from multiple filebeat instances running on different machines to single logstash server](https://discuss.elastic.co/t/ship-logs-from-multiple-filebeat-instances-running-on-different-machines-to-single-logstash-server/242435)
**Author:** [@Gyanprakash\_Mishra](https://discuss.elastic.co/u/Gyanprakash_Mishra)\ **Replies:** 0\ **Last updated:** [July 24, 2020, 3:49am UTC](https://discuss.elastic.co/t/ship-logs-from-multiple-filebeat-instances-running-on-different-machines-to-single-logstash-server/242435 "2020-07-24T03:49:07Z")
I have following scenario : multiple applications running on different server spread across region. How should I configure system so that i have filebeat instance running on each application server and sending data to on… --- ## [Logstash s3 output plugin storing files as .txt instead of .json on s3](https://discuss.elastic.co/t/logstash-s3-output-plugin-storing-files-as-txt-instead-of-json-on-s3/242431)
**Author:** [@jericomaramot](https://discuss.elastic.co/u/jericomaramot)\ **Replies:** 0\ **Last updated:** [July 24, 2020, 3:18am UTC](https://discuss.elastic.co/t/logstash-s3-output-plugin-storing-files-as-txt-instead-of-json-on-s3/242431 "2020-07-24T03:18:33Z")
Hi, I need to store my logstash output to s3 as json file. But upon checking the output files in s3, it is stored as .txt format. Here is my output code: output { s3 { region =\> "${elkRegion}" … --- ## [How to track the stream of documents inputed in Logstash by Elastic search input plugin?](https://discuss.elastic.co/t/how-to-track-the-stream-of-documents-inputed-in-logstash-by-elastic-search-input-plugin/242417)
**Author:** [@lucasbemol](https://discuss.elastic.co/u/lucasbemol)\ **Replies:** 0\ **Last updated:** [July 23, 2020, 11:10pm UTC](https://discuss.elastic.co/t/how-to-track-the-stream-of-documents-inputed-in-logstash-by-elastic-search-input-plugin/242417 "2020-07-23T23:10:53Z")
I'm having some problems with ElasticSearch. Keeping it short, I need to look into my logs and normalize them in various indexes. My logs are stored in both an index of ElasticSearch and in a relational database. These … --- ## [NegativeArraySizeException: -1680966849, HostUnreachableError and OOM](https://discuss.elastic.co/t/negativearraysizeexception-1680966849-hostunreachableerror-and-oom/242366)
**Author:** [@rajdevworks](https://discuss.elastic.co/u/rajdevworks)\ **Replies:** 4\ **Last updated:** [July 23, 2020, 10:04pm UTC](https://discuss.elastic.co/t/negativearraysizeexception-1680966849-hostunreachableerror-and-oom/242366 "2020-07-23T22:04:30Z")
On our ELK stack, while ingesting bulk logs, we are seeing HostUnreachableError often followed by connection restoration message in logstash.err. Following that we see NegativeArraySizeException: -1680966849 multiple tim… --- ## [How I can set up logstash with debian packages](https://discuss.elastic.co/t/how-i-can-set-up-logstash-with-debian-packages/241533)
**Author:** [@Splunk](https://discuss.elastic.co/u/Splunk)\ **Replies:** 3\ **Last updated:** [July 16, 2020, 11:11pm UTC](https://discuss.elastic.co/t/how-i-can-set-up-logstash-with-debian-packages/241533 "2020-07-16T23:11:45Z")
I feel like it is a waste of time to deal with Debian packages, can anyone tell me how I can set up logstash with Debian packages? I followed the instruction on elastic website, how can i access to logstash conf file h… --- ## [Logstash multiple pipelines processing](https://discuss.elastic.co/t/logstash-multiple-pipelines-processing/242392)
**Author:** [@FranB](https://discuss.elastic.co/u/FranB)\ **Replies:** 2\ **Last updated:** [July 23, 2020, 8:27pm UTC](https://discuss.elastic.co/t/logstash-multiple-pipelines-processing/242392 "2020-07-23T20:27:22Z")
Hello, i have this doubt: if i have 4 pipelines in the pipelines.yml file, one instance of logstash will process and output at the same time/in parallel for those 4 different type of pipelines or will they work as the f… --- ## [Logstash error after configuring ssl in the logstash.conf](https://discuss.elastic.co/t/logstash-error-after-configuring-ssl-in-the-logstash-conf/242262)
**Author:** [@Matish\_Bhuyan](https://discuss.elastic.co/u/Matish_Bhuyan)\ **Replies:** 9\ **Last updated:** [July 23, 2020, 7:39pm UTC](https://discuss.elastic.co/t/logstash-error-after-configuring-ssl-in-the-logstash-conf/242262 "2020-07-23T19:39:59Z")
Hi All , I have configured my elk stack to use ssl settings and after the change in Kibana and elasticsearch i have updated the logstash configuration , I am able to connect to my elasticsearch cluster from logstash as… --- ## [How to create key value from Nested fields](https://discuss.elastic.co/t/how-to-create-key-value-from-nested-fields/242084)
**Author:** [@Ramalakshmi](https://discuss.elastic.co/u/Ramalakshmi)\ **Replies:** 12\ **Last updated:** [July 23, 2020, 6:52pm UTC](https://discuss.elastic.co/t/how-to-create-key-value-from-nested-fields/242084 "2020-07-23T18:52:30Z")
Hi, I am getting json data in kibana. The data having nested fields like aws.datasource.resource.instanceDetails.tags { "key": "Environment", "value": "true" }, { "key": "nga", "… --- ## [Logstash not saving additional fields](https://discuss.elastic.co/t/logstash-not-saving-additional-fields/242220)
**Author:** [@tomba2k](https://discuss.elastic.co/u/tomba2k)\ **Replies:** 6\ **Last updated:** [July 23, 2020, 4:26pm UTC](https://discuss.elastic.co/t/logstash-not-saving-additional-fields/242220 "2020-07-23T16:26:14Z")
On working elastic stack I'm trying to send mysql slowlog to OSS elasticsearch cluster. Data is saved to ES, including \[event\]\[dataset\] for example, but everything under \[mysql\]\[slowlog\] or even \[mysql\] arrays is being … --- ## [Logstash relay](https://discuss.elastic.co/t/logstash-relay/242100)
**Author:** [@mariof](https://discuss.elastic.co/u/mariof)\ **Replies:** 6\ **Last updated:** [July 23, 2020, 3:21pm UTC](https://discuss.elastic.co/t/logstash-relay/242100 "2020-07-23T15:21:25Z")
i'm interested in setting up a logstash relay. i need to be able to listen for syslog and forward the messages via at least one other logstash server before hitting a group of logstash servers. do you think this is poss… --- ## [Logstash Error: Unrecognized SSL message, plaintext connection?](https://discuss.elastic.co/t/logstash-error-unrecognized-ssl-message-plaintext-connection/242356)
**Author:** [@Marios\_Karatisoglou](https://discuss.elastic.co/u/Marios_Karatisoglou)\ **Replies:** 0\ **Last updated:** [July 23, 2020, 2:42pm UTC](https://discuss.elastic.co/t/logstash-error-unrecognized-ssl-message-plaintext-connection/242356 "2020-07-23T14:42:01Z")
Hello community, I use Logstash in centos 7 and when I start it, it gives the following error \*\*\[2020-07-23T10:37:01,699\]\[INFO \]\[logstash.runner \] Starting Logstash {"logstash.version"=\>"6.8.10"}\*\* \*\*\[2020… --- ## [Terminate Logstash when using pipeline-to-pipelines communication](https://discuss.elastic.co/t/terminate-logstash-when-using-pipeline-to-pipelines-communication/238110)
**Author:** [@glenacota](https://discuss.elastic.co/u/glenacota)\ **Replies:** 3\ **Last updated:** [July 23, 2020, 10:49am UTC](https://discuss.elastic.co/t/terminate-logstash-when-using-pipeline-to-pipelines-communication/238110 "2020-07-23T10:49:04Z")
Hi, this is a duplicated of a question I posted last year, and yet unanswered/unsolved (original question). I would like to use one of the pipeline-to-pipeline architecture patterns, say, the forked path pattern. The in… --- ## [Using kv to produce fields with every word capitalized](https://discuss.elastic.co/t/using-kv-to-produce-fields-with-every-word-capitalized/242058)
**Author:** [@username11](https://discuss.elastic.co/u/username11)\ **Replies:** 2\ **Last updated:** [July 23, 2020, 10:07am UTC](https://discuss.elastic.co/t/using-kv-to-produce-fields-with-every-word-capitalized/242058 "2020-07-23T10:07:58Z")
I am using kv to parse a certain log that contains comma-separated key-value pairs. However, the keys may contain multiple space-separated words with inconsistent capitalization, and I'm trying to bring the resulting fie… --- ## [Define field as nested type using Logstash](https://discuss.elastic.co/t/define-field-as-nested-type-using-logstash/242289)
**Author:** [@Yoorae\_Kim](https://discuss.elastic.co/u/Yoorae_Kim)\ **Replies:** 9\ **Last updated:** [July 23, 2020, 9:06am UTC](https://discuss.elastic.co/t/define-field-as-nested-type-using-logstash/242289 "2020-07-23T09:06:39Z")
Hello, I am currently shipping my data to Elasticsearch using Logstash. The inner-hit query I am trying to perform is returning an error saying "failed to create query: \[nested\] nested object under path \[transactions\]… --- ## [How to Parse Json http body from a POST request with tcp plugin](https://discuss.elastic.co/t/how-to-parse-json-http-body-from-a-post-request-with-tcp-plugin/240565)
**Author:** [@Bouraoui\_Kacem](https://discuss.elastic.co/u/Bouraoui_Kacem)\ **Replies:** 4\ **Last updated:** [July 23, 2020, 8:19am UTC](https://discuss.elastic.co/t/how-to-parse-json-http-body-from-a-post-request-with-tcp-plugin/240565 "2020-07-23T08:19:30Z")
I send a Post request with a json content from PostMan to logstash pipeline with an input tcp plugin , the model of my data is like : { "results:\[ values :"\\"{..json object.. }\\"" \] } my logstahs input is : … --- ## [How to Read logs from Solaris 11.3 SPARC and push to logstash on rhel](https://discuss.elastic.co/t/how-to-read-logs-from-solaris-11-3-sparc-and-push-to-logstash-on-rhel/241358)
**Author:** [@dhanshake](https://discuss.elastic.co/u/dhanshake)\ **Replies:** 3\ **Last updated:** [July 23, 2020, 7:54am UTC](https://discuss.elastic.co/t/how-to-read-logs-from-solaris-11-3-sparc-and-push-to-logstash-on-rhel/241358 "2020-07-23T07:54:40Z")
Hey, Any alternative to Filebeat which helps in reading application / database logs from Solaris server (Solaris 11.3 SPARC) and push it to my logstash on my rhel ? P.s filebeat is not supported in Solaris SPARC syste… --- ## [S3 input Plugin Logstash](https://discuss.elastic.co/t/s3-input-plugin-logstash/242298)
**Author:** [@Vignesh2912](https://discuss.elastic.co/u/Vignesh2912)\ **Replies:** 0\ **Last updated:** [July 23, 2020, 7:36am UTC](https://discuss.elastic.co/t/s3-input-plugin-logstash/242298 "2020-07-23T07:36:02Z")
Hi Everyone, I am getting the below error in the Logstash. Could you please provide some insight on how to fix this.?? Already tried reinstalling the plugin. \[2020-07-23T06:50:07,402\]\[ERROR\]\[logstash.pipeline \] … --- ## [Processing large amoung of data with Logstash](https://discuss.elastic.co/t/processing-large-amoung-of-data-with-logstash/242020)
**Author:** [@evannobre](https://discuss.elastic.co/u/evannobre)\ **Replies:** 5\ **Last updated:** [July 23, 2020, 7:28am UTC](https://discuss.elastic.co/t/processing-large-amoung-of-data-with-logstash/242020 "2020-07-23T07:28:21Z")
Hi, Sorry for the text, but I tried to detail my problem as much as possible to make it clear. I am part of a medical devices team. We run hundreds of tests over the devices to assure the quality and we save the result… --- ## [Last line of log repeatedly gets indexed/processed to elastic/kibana](https://discuss.elastic.co/t/last-line-of-log-repeatedly-gets-indexed-processed-to-elastic-kibana/242293)
**Author:** [@Aj\_t](https://discuss.elastic.co/u/Aj_t)\ **Replies:** 0\ **Last updated:** [July 23, 2020, 7:15am UTC](https://discuss.elastic.co/t/last-line-of-log-repeatedly-gets-indexed-processed-to-elastic-kibana/242293 "2020-07-23T07:15:20Z")
I have a filebeat \> logstash \> elastic set up and I am working with a multi-line log format. I am not sure why the last line of the log file is repeatedly getting indexed. What I would ideally want is that filebeat wo… --- ## [How to load tables from mysql to elasticsearch](https://discuss.elastic.co/t/how-to-load-tables-from-mysql-to-elasticsearch/242275)
**Author:** [@Krishna\_Sai\_Nag\_G](https://discuss.elastic.co/u/Krishna_Sai_Nag_G)\ **Replies:** 5\ **Last updated:** [July 23, 2020, 6:14am UTC](https://discuss.elastic.co/t/how-to-load-tables-from-mysql-to-elasticsearch/242275 "2020-07-23T06:14:24Z")
Hi, I want load multiple tables from mysql to different indexes in elastic search(i.e; each table in mysql to different index in elastic search), Is it possible to load using single logstash config file. Anyone can hel… --- ## [Integer comparison not working](https://discuss.elastic.co/t/integer-comparison-not-working/242248)
**Author:** [@opoplawski](https://discuss.elastic.co/u/opoplawski)\ **Replies:** 3\ **Last updated:** [July 23, 2020, 3:35am UTC](https://discuss.elastic.co/t/integer-comparison-not-working/242248 "2020-07-23T03:35:39Z")
I'm trying to conditionally set some fields in my pipeline: filter { if \[log\]\[file\]\[path\] == "/var/log/e2guardian/access.log" { grok { match =\> { "message" =\> "(?\\\d\\d\\d\\d\\.\\d\\d\\.\\d\\d \\d\\d:\\d\\d:… --- ## [I'm not able to see logs in \[Filebeat Logstash\] Logstash Logs ECS dashboard](https://discuss.elastic.co/t/im-not-able-to-see-logs-in-filebeat-logstash-logstash-logs-ecs-dashboard/241961)
**Author:** [@Bhavin\_Varsur](https://discuss.elastic.co/u/Bhavin_Varsur)\ **Replies:** 1\ **Last updated:** [July 23, 2020, 12:41am UTC](https://discuss.elastic.co/t/im-not-able-to-see-logs-in-filebeat-logstash-logstash-logs-ecs-dashboard/241961 "2020-07-23T00:41:08Z")
I installed both filebeat and logstash. i aslo create a pipeline in logstash which is input data from beats and output to elasticsearch. when i run pipeline the logs are succesfully index in logstash.index pattern cre… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=323) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=325)