# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=352 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 353 --- ## [Using Cloud.id and Cloud.auth in logstash](https://discuss.elastic.co/t/using-cloud-id-and-cloud-auth-in-logstash/234336)
**Author:** [@Apollyon](https://discuss.elastic.co/u/Apollyon)\ **Replies:** 2\ **Last updated:** [May 26, 2020, 3:07pm UTC](https://discuss.elastic.co/t/using-cloud-id-and-cloud-auth-in-logstash/234336 "2020-05-26T15:07:07Z")
Hello All, I've been playing around with Logstash recently and thought it'll be cool to pick your lots brain about something. When in the logstash.yml, I place the cloud.id and cloud.auth, which is well and ok. But the… --- ## [Logstash pipeline to pipeline - outputs too many documents](https://discuss.elastic.co/t/logstash-pipeline-to-pipeline-outputs-too-many-documents/234365)
**Author:** [@Shai\_Lers](https://discuss.elastic.co/u/Shai_Lers)\ **Replies:** 0\ **Last updated:** [May 26, 2020, 3:07pm UTC](https://discuss.elastic.co/t/logstash-pipeline-to-pipeline-outputs-too-many-documents/234365 "2020-05-26T15:07:03Z")
Hi, i am using filebeat in order to write to a pipeline: input { beats { port =\> "5050" } } output { if \[message\] =~ /value/ { pipeline { send\_to =\> \[reciving\_pipeline\] } } } a… --- ## [\[org.logstash.execution.ShutdownWatcherExt\] - Logstash seems to be stuck!](https://discuss.elastic.co/t/org-logstash-execution-shutdownwatcherext-logstash-seems-to-be-stuck/234335)
**Author:** [@Apollyon](https://discuss.elastic.co/u/Apollyon)\ **Replies:** 0\ **Last updated:** [May 26, 2020, 12:42pm UTC](https://discuss.elastic.co/t/org-logstash-execution-shutdownwatcherext-logstash-seems-to-be-stuck/234335 "2020-05-26T12:42:21Z")
Hello all, Setup is Logstash (7.7.0) agent on AWS linux instance - using s3 input and Elasticsearch output in confs. Elastic Cloud (7.7.0) Logstash seems to be running and all but then when it comes to seeing the dat… --- ## [Logstash http input posted content not available](https://discuss.elastic.co/t/logstash-http-input-posted-content-not-available/234364)
**Author:** [@lkalaivanan](https://discuss.elastic.co/u/lkalaivanan)\ **Replies:** 0\ **Last updated:** [May 26, 2020, 3:02pm UTC](https://discuss.elastic.co/t/logstash-http-input-posted-content-not-available/234364 "2020-05-26T15:02:02Z")
Hi All Infra: cadvisor --\> logstash --\> elasticsearch --\> kibana We are trying to achieve the above infra. Like cadvisor to post container\_stats to the elasticsearch via logstash Log stash config : input { http { … --- ## [Filter Elasticsearch Plugin](https://discuss.elastic.co/t/filter-elasticsearch-plugin/234363)
**Author:** [@aannee](https://discuss.elastic.co/u/aannee)\ **Replies:** 0\ **Last updated:** [May 26, 2020, 3:01pm UTC](https://discuss.elastic.co/t/filter-elasticsearch-plugin/234363 "2020-05-26T15:01:28Z")
Hi, it's my first time using this filter-plugin. I'm just wondering if its possible to join two elasticsearch index tusing this plugin. I'm thinking of using this plugin because in elasticsearch its not possible to just… --- ## [Logstash jdbc input jdbc\_fetch\_size not working making first load extremely slower](https://discuss.elastic.co/t/logstash-jdbc-input-jdbc-fetch-size-not-working-making-first-load-extremely-slower/230289)
**Author:** [@MChat](https://discuss.elastic.co/u/MChat)\ **Replies:** 3\ **Last updated:** [May 26, 2020, 2:33pm UTC](https://discuss.elastic.co/t/logstash-jdbc-input-jdbc-fetch-size-not-working-making-first-load-extremely-slower/230289 "2020-05-26T14:33:24Z")
I am trying to pull all the data from oracle DB table for first load and then it would be delta based on tracking column and schedule. Below is the jdbc conf in logstash: input { jdbc { jdbc\_connection\_string =\> "… --- ## [Maps are not showing data if logstash is used](https://discuss.elastic.co/t/maps-are-not-showing-data-if-logstash-is-used/233679)
**Author:** [@PraveenKT](https://discuss.elastic.co/u/PraveenKT)\ **Replies:** 12\ **Last updated:** [May 26, 2020, 1:48pm UTC](https://discuss.elastic.co/t/maps-are-not-showing-data-if-logstash-is-used/233679 "2020-05-26T13:48:17Z")
When i directly send data to Elasticsearch i am able to see the maps data in SIEM and Netflow geolocations. But if i send data through the Logstash i am missing data in SIEM maps and Netflow geoip locations. To get dat… --- ## [Logstash doesn't send data to Elastic](https://discuss.elastic.co/t/logstash-doesnt-send-data-to-elastic/234346)
**Author:** [@lm699](https://discuss.elastic.co/u/lm699)\ **Replies:** 0\ **Last updated:** [May 26, 2020, 1:31pm UTC](https://discuss.elastic.co/t/logstash-doesnt-send-data-to-elastic/234346 "2020-05-26T13:31:38Z")
Hello everybody, I've recently installed Elastic, Logstash and Kibana to parse apache's logs. But, logstash seems sending data only when I run the following command : sudo /usr/share/logstash/bin/logstash --debug --pat… --- ## [Logstash - Mutate lowercase doesn't work with filed name](https://discuss.elastic.co/t/logstash-mutate-lowercase-doesnt-work-with-filed-name/234271)
**Author:** [@stmx38](https://discuss.elastic.co/u/stmx38)\ **Replies:** 1\ **Last updated:** [May 26, 2020, 12:23pm UTC](https://discuss.elastic.co/t/logstash-mutate-lowercase-doesnt-work-with-filed-name/234271 "2020-05-26T12:23:45Z")
Hello, I'm trying to unify some of our fields to be the same case. Now we have TARGET\_INDEX and target\_index I have tried to apply mutate lowercase filter as described in the documentation: mutate { lowercase =\> \[… --- ## [java.lang.OutOfMemoryError: Java heap space error in logstash conf file](https://discuss.elastic.co/t/java-lang-outofmemoryerror-java-heap-space-error-in-logstash-conf-file/233985)
**Author:** [@avneshkumar98](https://discuss.elastic.co/u/avneshkumar98)\ **Replies:** 2\ **Last updated:** [May 26, 2020, 9:35am UTC](https://discuss.elastic.co/t/java-lang-outofmemoryerror-java-heap-space-error-in-logstash-conf-file/233985 "2020-05-26T09:35:49Z")
hey, I'm facing this issue while importing logs in kibana using conf file. \[2020-05-23T13:55:37,084\]\[ERROR\]\[org.logstash.Logstash \] java.lang.OutOfMemoryError: Java heap space and my conf file : input { file{ pat… --- ## [Logstash - pagerduty plugin \[severity level\]](https://discuss.elastic.co/t/logstash-pagerduty-plugin-severity-level/234284)
**Author:** [@Donatas\_Racys](https://discuss.elastic.co/u/Donatas_Racys)\ **Replies:** 0\ **Last updated:** [May 26, 2020, 9:09am UTC](https://discuss.elastic.co/t/logstash-pagerduty-plugin-severity-level/234284 "2020-05-26T09:09:20Z")
Hi, I have a use case with Logstash - Pagerduty plugin - we want to send different level alerts to Pagerduty, but severity parameter is missing. Is there a way to add severity parameter to Pagerduty plugin ? Severity - … --- ## [How to avoid multiple doc/event uploading to remote Elasticsearch after connection issue](https://discuss.elastic.co/t/how-to-avoid-multiple-doc-event-uploading-to-remote-elasticsearch-after-connection-issue/234171)
**Author:** [@Abolurah](https://discuss.elastic.co/u/Abolurah)\ **Replies:** 2\ **Last updated:** [May 26, 2020, 8:48am UTC](https://discuss.elastic.co/t/how-to-avoid-multiple-doc-event-uploading-to-remote-elasticsearch-after-connection-issue/234171 "2020-05-26T08:48:04Z")
Hi all, I'm working on events uploading from local Elasticsearch to remote Elasticsearch with Logstash. I noticed if remote Elastic is disconnected for some reason, the error looks like this: "Attempted to send a bulk… --- ## [Send the same data to two outputs elasticsearch](https://discuss.elastic.co/t/send-the-same-data-to-two-outputs-elasticsearch/234274)
**Author:** [@am95](https://discuss.elastic.co/u/am95)\ **Replies:** 0\ **Last updated:** [May 26, 2020, 8:17am UTC](https://discuss.elastic.co/t/send-the-same-data-to-two-outputs-elasticsearch/234274 "2020-05-26T08:17:45Z")
Hi, I wanna send the same data from one logstash to two elasticsearch nodes ans i don't knwo how te set the pipeline, any help please ? i've set in the logstash output this : output { elasticsearch { hosts =\> \["node… --- ## [Add host name field](https://discuss.elastic.co/t/add-host-name-field/234044)
**Author:** [@ahiyaz](https://discuss.elastic.co/u/ahiyaz)\ **Replies:** 4\ **Last updated:** [May 26, 2020, 8:12am UTC](https://discuss.elastic.co/t/add-host-name-field/234044 "2020-05-26T08:12:46Z")
hi im shipping sflow data to Logz.io using logstash. due to multi-sites architecture im trying to add hostname (agent hostname) filed to my data that sent from my logstash but with no success. ive tried to add it to d… --- ## [Scaling logstash with s3 output doesnt work](https://discuss.elastic.co/t/scaling-logstash-with-s3-output-doesnt-work/234164)
**Author:** [@mctj](https://discuss.elastic.co/u/mctj)\ **Replies:** 5\ **Last updated:** [May 25, 2020, 6:39pm UTC](https://discuss.elastic.co/t/scaling-logstash-with-s3-output-doesnt-work/234164 "2020-05-25T18:39:03Z")
Hey all, I have multiple machines in my environment (lets say 100). On each of those machines I configured filebeat and metricbeat to send data to Kafka. Each machine has 2 dedicated topics in kafka(one for metricbe… --- ## [Append object to nested field but get overwritten](https://discuss.elastic.co/t/append-object-to-nested-field-but-get-overwritten/234268)
**Author:** [@Arnovdlaan](https://discuss.elastic.co/u/Arnovdlaan)\ **Replies:** 0\ **Last updated:** [May 26, 2020, 8:05am UTC](https://discuss.elastic.co/t/append-object-to-nested-field-but-get-overwritten/234268 "2020-05-26T08:05:47Z")
Hello, We've got a problem with a logstash JDBC configuration. We have a large tabel (~ 15 million records) which we want to insert into our elasticsearch. A source (MySQL) record has a domain column which is used as t… --- ## [How can i create secure websocket WSS client connection in logstash 7.4?](https://discuss.elastic.co/t/how-can-i-create-secure-websocket-wss-client-connection-in-logstash-7-4/234242)
**Author:** [@talbehat](https://discuss.elastic.co/u/talbehat)\ **Replies:** 1\ **Last updated:** [May 26, 2020, 7:47am UTC](https://discuss.elastic.co/t/how-can-i-create-secure-websocket-wss-client-connection-in-logstash-7-4/234242 "2020-05-26T07:47:41Z")
is there any way to use wss client in logstash? --- ## [Mutliple file input not processed (or may be erased in elasticsearch)](https://discuss.elastic.co/t/mutliple-file-input-not-processed-or-may-be-erased-in-elasticsearch/234147)
**Author:** [@pmo](https://discuss.elastic.co/u/pmo)\ **Replies:** 2\ **Last updated:** [May 26, 2020, 7:18am UTC](https://discuss.elastic.co/t/mutliple-file-input-not-processed-or-may-be-erased-in-elasticsearch/234147 "2020-05-26T07:18:17Z")
Hi, I' ve to treat many json files. When I try to import file by file via logstash by putting the name of each file, it works. But when I use "\*", it seams that only 1 file is treated. Or may be there all but only the l… --- ## [Join two index using input elasticsearch/exec plugin in logstash](https://discuss.elastic.co/t/join-two-index-using-input-elasticsearch-exec-plugin-in-logstash/234256)
**Author:** [@aannee](https://discuss.elastic.co/u/aannee)\ **Replies:** 0\ **Last updated:** [May 26, 2020, 7:17am UTC](https://discuss.elastic.co/t/join-two-index-using-input-elasticsearch-exec-plugin-in-logstash/234256 "2020-05-26T07:17:35Z")
Hi, It's my first time facing request like this so I'm thinking if is it possible to join 2 elasticsearch index using logstash? Because I think its not possible to have a join query in elasticsearch. --- ## [SSL Connection Error with SQL Database](https://discuss.elastic.co/t/ssl-connection-error-with-sql-database/234253)
**Author:** [@vikramnr](https://discuss.elastic.co/u/vikramnr)\ **Replies:** 0\ **Last updated:** [May 26, 2020, 6:51am UTC](https://discuss.elastic.co/t/ssl-connection-error-with-sql-database/234253 "2020-05-26T06:51:06Z")
I just want to run logstash with using JDBC plugin but I'm unable to connect to the server. Below is error that I receive. I tried some ways suggested in github but my problem isn't being solved. Please let me know if … --- ## [Conditional(else if ) is not working in logstash(6.8.7) pipeline](https://discuss.elastic.co/t/conditional-else-if-is-not-working-in-logstash-6-8-7-pipeline/229924)
**Author:** [@taybur.rahaman](https://discuss.elastic.co/u/taybur.rahaman)\ **Replies:** 2\ **Last updated:** [April 28, 2020, 3:44am UTC](https://discuss.elastic.co/t/conditional-else-if-is-not-working-in-logstash-6-8-7-pipeline/229924 "2020-04-28T03:44:09Z")
Can you please help me to find out the issue of this pipeline. filter{ if \[log\_group\] in \[ "/aws/lambda/dev", "/aws/lambda/dev-ful" \] { mutate { add\_field =\> { "\[@metadata\]\[target\_index\]" =\> "… --- ## [Using Kafka as Staging for Logstash Pipelines to club diffrent fields from diffrent logstash pipelines and push to Elasticsearch as a single document](https://discuss.elastic.co/t/using-kafka-as-staging-for-logstash-pipelines-to-club-diffrent-fields-from-diffrent-logstash-pipelines-and-push-to-elasticsearch-as-a-single-document/234233)
**Author:** [@suhas\_elk](https://discuss.elastic.co/u/suhas_elk)\ **Replies:** 0\ **Last updated:** [May 26, 2020, 3:42am UTC](https://discuss.elastic.co/t/using-kafka-as-staging-for-logstash-pipelines-to-club-diffrent-fields-from-diffrent-logstash-pipelines-and-push-to-elasticsearch-as-a-single-document/234233 "2020-05-26T03:42:20Z")
Hi Team, We have Logstash Pipeline which receives input from diffrent paths pushed by filebeat , but we have requirement where we need to club diffrent fields which will be processed at diffrent time stamps and push as … --- ## [After Updating Logstash to 7.7 from 7.6 regex isn't working the same](https://discuss.elastic.co/t/after-updating-logstash-to-7-7-from-7-6-regex-isnt-working-the-same/234012)
**Author:** [@chrisdm](https://discuss.elastic.co/u/chrisdm)\ **Replies:** 9\ **Last updated:** [May 26, 2020, 2:47am UTC](https://discuss.elastic.co/t/after-updating-logstash-to-7-7-from-7-6-regex-isnt-working-the-same/234012 "2020-05-26T02:47:46Z")
I recently updated to Logstash 7.7 from 7.6, after this update the following is occurring: A typical log line looks like this: 2019-10-02T04:00:09.254Z - ^\[\[32minfo^\[\[39m: Activity:content-finish 35692 When processing… --- ## [Unable to replace @timestamp from log message data using date filter plugin](https://discuss.elastic.co/t/unable-to-replace-timestamp-from-log-message-data-using-date-filter-plugin/234208)
**Author:** [@John\_Westbrook](https://discuss.elastic.co/u/John_Westbrook)\ **Replies:** 1\ **Last updated:** [May 25, 2020, 9:34pm UTC](https://discuss.elastic.co/t/unable-to-replace-timestamp-from-log-message-data-using-date-filter-plugin/234208 "2020-05-25T21:34:57Z")
Greetings, I am trying to parse log records with the following format and capture the log message date in @timestamp: May 24 00:00:00 localhost haproxy-https\[7130\]: 92.34.251.94 - - \[24/May/2020:07:00:00 +0000\] "GET /… --- ## [Help with grok pattern Logstash](https://discuss.elastic.co/t/help-with-grok-pattern-logstash/234106)
**Author:** [@Amish](https://discuss.elastic.co/u/Amish)\ **Replies:** 2\ **Last updated:** [May 25, 2020, 2:27pm UTC](https://discuss.elastic.co/t/help-with-grok-pattern-logstash/234106 "2020-05-25T14:27:16Z")
I need help with creating a grok pattern to match the following data and create them as new fields. I have attempted to use the Grok debugger online and it matches my custom patterns but when I apply this in Logstash it … --- ## [Unable to run logstash pod on kubernetes](https://discuss.elastic.co/t/unable-to-run-logstash-pod-on-kubernetes/233161)
**Author:** [@Abhinav\_Shrivastava](https://discuss.elastic.co/u/Abhinav_Shrivastava)\ **Replies:** 1\ **Last updated:** [May 25, 2020, 1:57pm UTC](https://discuss.elastic.co/t/unable-to-run-logstash-pod-on-kubernetes/233161 "2020-05-25T13:57:07Z")
Hello, I am trying to deploy elk stack on Kubernetes cluster (running on my laptop) using helm chart. Elastic search and kibana have deployed successfully, however, logstash pod is crashing ans showing "CrashLoopBackOff… --- ## [Adding Datasource from 3rd Party Applications](https://discuss.elastic.co/t/adding-datasource-from-3rd-party-applications/233714)
**Author:** [@Chad](https://discuss.elastic.co/u/Chad)\ **Replies:** 4\ **Last updated:** [May 25, 2020, 1:33pm UTC](https://discuss.elastic.co/t/adding-datasource-from-3rd-party-applications/233714 "2020-05-25T13:33:16Z")
Hi, i'm new in ELK, and i'm looking to get logs from my AD AuditPlus tool ( screenshot attached). How to do I go about it from setting it up , up to creation of dashboard. --- ## [ELK docker setup in alfresco](https://discuss.elastic.co/t/elk-docker-setup-in-alfresco/234156)
**Author:** [@piyush\_badola](https://discuss.elastic.co/u/piyush_badola)\ **Replies:** 0\ **Last updated:** [May 25, 2020, 1:18pm UTC](https://discuss.elastic.co/t/elk-docker-setup-in-alfresco/234156 "2020-05-25T13:18:37Z")
Hi all, i am relatively new to ELK and am trying to setup ELK using docker in alfresco. I am following this as a reference for the setup: my elasticsearch and kibana is up and running but there seems to be no data av… --- ## [How to remove multiple head lines in CSV with logstash configuration file](https://discuss.elastic.co/t/how-to-remove-multiple-head-lines-in-csv-with-logstash-configuration-file/233849)
**Author:** [@Ryan4](https://discuss.elastic.co/u/Ryan4)\ **Replies:** 1\ **Last updated:** [May 25, 2020, 11:32am UTC](https://discuss.elastic.co/t/how-to-remove-multiple-head-lines-in-csv-with-logstash-configuration-file/233849 "2020-05-25T11:32:55Z")
Hello All, I need handle CSV files from Windows File Resource manage. The file has multiple head lines that need to remove. How can I do it with setting logstash configuration file. The following is a sample. "Large F… --- ## [Rename variable field in json file using logstash](https://discuss.elastic.co/t/rename-variable-field-in-json-file-using-logstash/234003)
**Author:** [@Kush\_Wadhwa](https://discuss.elastic.co/u/Kush_Wadhwa)\ **Replies:** 1\ **Last updated:** [May 25, 2020, 11:27am UTC](https://discuss.elastic.co/t/rename-variable-field-in-json-file-using-logstash/234003 "2020-05-25T11:27:48Z")
Hi, I have a jason file which has event0,event1, event2......event50000. I would like to rename all of it to event in Logstash so that the field name is event and corresponding value of event0, event 1 are stored under … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=351) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=353)