# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=135 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 136 --- ## [Logstash shutdown with no error](https://discuss.elastic.co/t/logstash-shutdown-with-no-error/306807)
**Author:** [@Hosein\_Kashefikaram](https://discuss.elastic.co/u/Hosein_Kashefikaram)\ **Replies:** 2\ **Last updated:** [June 9, 2022, 4:43pm UTC](https://discuss.elastic.co/t/logstash-shutdown-with-no-error/306807 "2022-06-09T16:43:26Z")
I got following messages after starting Logstash Jun 09 10:48:10 local systemd\[1\]: Started logstash. Jun 09 10:48:10 local logstash\[3191\]: Using bundled JDK: /usr/share/logstash/jdk Jun 09 10:48:11 local logstash\[3191\]:… --- ## [Need help to configure http output plugin for https](https://discuss.elastic.co/t/need-help-to-configure-http-output-plugin-for-https/306773)
**Author:** [@guillaumeV](https://discuss.elastic.co/u/guillaumeV)\ **Replies:** 1\ **Last updated:** [June 9, 2022, 2:17pm UTC](https://discuss.elastic.co/t/need-help-to-configure-http-output-plugin-for-https/306773 "2022-06-09T14:17:00Z")
Hello i would like to send data via http output plugin (Http output plugin | Logstash Reference \[8.19\] | Elastic) to a https url . I have a very simple example with "https://www.google.com" which is not working is it pos… --- ## [Grok pattern for snort alerts](https://discuss.elastic.co/t/grok-pattern-for-snort-alerts/306625)
**Author:** [@anushka1203](https://discuss.elastic.co/u/anushka1203)\ **Replies:** 4\ **Last updated:** [June 9, 2022, 1:54pm UTC](https://discuss.elastic.co/t/grok-pattern-for-snort-alerts/306625 "2022-06-09T13:54:27Z")
HI everyone, Need help constructing grok pattern for the snort alert log file. I have the so far, but the output is incomplete - %{MONTHNUM:month}\\/%{MONTHDAY:day}-%{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second}\\s+\\\[\\\*\\… --- ## [Connection problem between Logstash and Elasticsearch running on Docker](https://discuss.elastic.co/t/connection-problem-between-logstash-and-elasticsearch-running-on-docker/306757)
**Author:** [@Francesco66](https://discuss.elastic.co/u/Francesco66)\ **Replies:** 4\ **Last updated:** [June 9, 2022, 11:24am UTC](https://discuss.elastic.co/t/connection-problem-between-logstash-and-elasticsearch-running-on-docker/306757 "2022-06-09T11:24:15Z")
Hello team, I have a Logstash that I can successfully run and start by using the following docker command: ------------------------- docker run --rm -d --name logstash-Casex -p 8080:8080 -p 8081:8081 -v /Logstash\_custom\_… --- ## [Can't etablish connection with logstash and my filebeat 7.17.1](https://discuss.elastic.co/t/cant-etablish-connection-with-logstash-and-my-filebeat-7-17-1/306722)
**Author:** [@Dreinale](https://discuss.elastic.co/u/Dreinale)\ **Replies:** 5\ **Last updated:** [June 9, 2022, 11:15am UTC](https://discuss.elastic.co/t/cant-etablish-connection-with-logstash-and-my-filebeat-7-17-1/306722 "2022-06-09T11:15:48Z")
Hi all, I want to do a monitoring of my log but i don't understand why my logstash doesn't"t work with filebeat. For now i have Elasticsearch: Elasticsearch.yml node.name: master-node-1 node.master: true cluster.init… --- ## [Logstash Docker on GCP Cloud Run: Expected the service InnerService \[FAILED\] to be TERMINATED, but the service has FAILED](https://discuss.elastic.co/t/logstash-docker-on-gcp-cloud-run-expected-the-service-innerservice-failed-to-be-terminated-but-the-service-has-failed/306785)
**Author:** [@hegdesandesh25](https://discuss.elastic.co/u/hegdesandesh25)\ **Replies:** 0\ **Last updated:** [June 9, 2022, 10:42am UTC](https://discuss.elastic.co/t/logstash-docker-on-gcp-cloud-run-expected-the-service-innerservice-failed-to-be-terminated-but-the-service-has-failed/306785 "2022-06-09T10:42:36Z")
Continuing the discussion from Logstash Docker on GCP Cloud Run: Failed to find a usable hardware address from the network interfaces: I have a similar use-case, running custom logstash container from gcloud-run with th… --- ## [Docker Logstash Error: Cannot assign requested address](https://discuss.elastic.co/t/docker-logstash-error-cannot-assign-requested-address/306711)
**Author:** [@Francesco66](https://discuss.elastic.co/u/Francesco66)\ **Replies:** 2\ **Last updated:** [June 9, 2022, 5:42am UTC](https://discuss.elastic.co/t/docker-logstash-error-cannot-assign-requested-address/306711 "2022-06-09T05:42:23Z")
Hello , I am running a Logstash as Docker container using the following command: .......... docker run --rm -d --name logstash-Casex -p 8080:8080 -p 8081:8081 -v /Logstash\_custom\_configs/config/pipelines.yml:/usr/share/l… --- ## [GeoIP filter does not work with a subfield](https://discuss.elastic.co/t/geoip-filter-does-not-work-with-a-subfield/306750)
**Author:** [@austin0918](https://discuss.elastic.co/u/austin0918)\ **Replies:** 2\ **Last updated:** [June 9, 2022, 3:20am UTC](https://discuss.elastic.co/t/geoip-filter-does-not-work-with-a-subfield/306750 "2022-06-09T03:20:16Z")
I have a field client.ip that needs to get geo info. I tried below GeoIP filter but it didn't work. Looks like GeoIP does not work with a subfield. Please advise. filter { if \[client\]\[ip\] { geoip { source =\> "\[cli… --- ## [Recommended resource setting for a Logstash Pod](https://discuss.elastic.co/t/recommended-resource-setting-for-a-logstash-pod/306747)
**Author:** [@amruth](https://discuss.elastic.co/u/amruth)\ **Replies:** 0\ **Last updated:** [June 9, 2022, 12:48am UTC](https://discuss.elastic.co/t/recommended-resource-setting-for-a-logstash-pod/306747 "2022-06-09T00:48:26Z")
Is there any recommended resource setting for a Logstash Pod handling small amount(100GB) of data everyday and no filtering in place? Logstash just reads data from SQLServer and pushes it to Elasticsearch without any fil… --- ## [Add\_field in logstash does not work](https://discuss.elastic.co/t/add-field-in-logstash-does-not-work/306616)
**Author:** [@PriyaM21](https://discuss.elastic.co/u/PriyaM21)\ **Replies:** 0\ **Last updated:** [June 8, 2022, 2:04am UTC](https://discuss.elastic.co/t/add-field-in-logstash-does-not-work/306616 "2022-06-08T02:04:50Z")
I am trying to add a new custom field from the message but new field do not show up in Kabana. There is no error in logstash logs. Below is my logstash code if \[fields\]\[app\_name\] == "cobra" { if \[source\] =~ /^".\*(\\|/… --- ## [Logstash connectivity from aws to on-premise](https://discuss.elastic.co/t/logstash-connectivity-from-aws-to-on-premise/306638)
**Author:** [@narasingarao.katta](https://discuss.elastic.co/u/narasingarao.katta)\ **Replies:** 0\ **Last updated:** [June 8, 2022, 7:04am UTC](https://discuss.elastic.co/t/logstash-connectivity-from-aws-to-on-premise/306638 "2022-06-08T07:04:42Z")
Hi i want move my cloud watch logs to elk which is hosted out of aws . can any one give some inputs in this --- ## [Modify the string into another form in Logstash](https://discuss.elastic.co/t/modify-the-string-into-another-form-in-logstash/306732)
**Author:** [@Pranjal\_Sett](https://discuss.elastic.co/u/Pranjal_Sett)\ **Replies:** 1\ **Last updated:** [June 8, 2022, 6:35pm UTC](https://discuss.elastic.co/t/modify-the-string-into-another-form-in-logstash/306732 "2022-06-08T18:35:10Z")
Hello, I am having one csv file, in that csv file there is a attribute named as Time field. And the format of that time field is not as a standard format. Here is the format: \[30/Nov/2017:15:28:27 And I want to change… --- ## [Drop filter to avoid documents to be sent to ES not working](https://discuss.elastic.co/t/drop-filter-to-avoid-documents-to-be-sent-to-es-not-working/306584)
**Author:** [@Carlos\_T](https://discuss.elastic.co/u/Carlos_T)\ **Replies:** 8\ **Last updated:** [June 8, 2022, 6:29pm UTC](https://discuss.elastic.co/t/drop-filter-to-avoid-documents-to-be-sent-to-es-not-working/306584 "2022-06-08T18:29:50Z")
Hi everybody. I've have a ES + Kib + Logstash + Filebeat environment ready for testing purposes, and after installing Filebeat so it sends the data to Logstash, I've realiced that I'm recieving more documents than necce… --- ## [Insert to elasticsearch from logstash IF NOT EXISTS](https://discuss.elastic.co/t/insert-to-elasticsearch-from-logstash-if-not-exists/306368)
**Author:** [@seanziee](https://discuss.elastic.co/u/seanziee)\ **Replies:** 7\ **Last updated:** [June 8, 2022, 6:09pm UTC](https://discuss.elastic.co/t/insert-to-elasticsearch-from-logstash-if-not-exists/306368 "2022-06-08T18:09:58Z")
Hi, I currently have an index that I set unique ids and sometimes there is replica data. In my use case, older data is often more accurate than newer data. So I would like to have logstash only insert if the \_id current… --- ## [Parse ISO 8601 duration format (PT(n)H(n)M(n)S)](https://discuss.elastic.co/t/parse-iso-8601-duration-format-pt-n-h-n-m-n-s/306605)
**Author:** [@Iss](https://discuss.elastic.co/u/Iss)\ **Replies:** 5\ **Last updated:** [June 8, 2022, 5:53pm UTC](https://discuss.elastic.co/t/parse-iso-8601-duration-format-pt-n-h-n-m-n-s/306605 "2022-06-08T17:53:03Z")
Hi! I have a ISO 8601 duration format P(n)Y(n)M(n)DT(n)H(n)M(n)S fields as "duration"=\>"PT0H0M0S" I have to parse it to get time duration in seconds or in format HH:MM:SS. How can I do it with logstash filter? Than… --- ## [Using IBM common data provider for z systems](https://discuss.elastic.co/t/using-ibm-common-data-provider-for-z-systems/306723)
**Author:** [@Akhil2](https://discuss.elastic.co/u/Akhil2)\ **Replies:** 0\ **Last updated:** [June 8, 2022, 4:05pm UTC](https://discuss.elastic.co/t/using-ibm-common-data-provider-for-z-systems/306723 "2022-06-08T16:05:45Z")
Hello all, Hope you are doing well. I have been using ELK 7.16.2. and I want to use IBM Common Data Provider to stream logs to my Logstash instance. But I have a few questions about installation of IBM CDPz, from where… --- ## [Grok pattern matching in debugger but not on logstash](https://discuss.elastic.co/t/grok-pattern-matching-in-debugger-but-not-on-logstash/306624)
**Author:** [@smtp\_server](https://discuss.elastic.co/u/smtp_server)\ **Replies:** 3\ **Last updated:** [June 8, 2022, 3:44pm UTC](https://discuss.elastic.co/t/grok-pattern-matching-in-debugger-but-not-on-logstash/306624 "2022-06-08T15:44:57Z")
Obvious from question, I've been trying to look for fault for so long but still don't seem to be coming around solution. Here's config filter{ grok{ pattern\_definitions =\> { "CUSTOMMONTH" =\> "(Jan|Feb|Mar|Apr|Ma… --- ## [Logstash not working, if installed in a location where the path contains parentheses (on windows)](https://discuss.elastic.co/t/logstash-not-working-if-installed-in-a-location-where-the-path-contains-parentheses-on-windows/306721)
**Author:** [@simon137](https://discuss.elastic.co/u/simon137)\ **Replies:** 0\ **Last updated:** [June 8, 2022, 3:40pm UTC](https://discuss.elastic.co/t/logstash-not-working-if-installed-in-a-location-where-the-path-contains-parentheses-on-windows/306721 "2022-06-08T15:40:29Z")
Hello together, As the title suggests, with a sample path of "C:\\ProgramData\\Test)Test\\logstash", the call to ".\\bin\\logstash.bat" fails with the message: "Test\\logstash\\jdk\\bin\\java.exe" cannot be processed syntactical… --- ## [Logstash not sending all data to elasti search](https://discuss.elastic.co/t/logstash-not-sending-all-data-to-elasti-search/306699)
**Author:** [@Mary2022](https://discuss.elastic.co/u/Mary2022)\ **Replies:** 0\ **Last updated:** [June 8, 2022, 1:39pm UTC](https://discuss.elastic.co/t/logstash-not-sending-all-data-to-elasti-search/306699 "2022-06-08T13:39:31Z")
Hi, My logstash is getting data continuously. I know this because I can see it at the logs (logstash debugging mode) but the if I go to kibana I see that 8 or 7 hrs ago I got 10 min of data and that was it when it was … --- ## [Elasticsearch input plugin used to \_count documents. Is it possible?](https://discuss.elastic.co/t/elasticsearch-input-plugin-used-to-count-documents-is-it-possible/306682)
**Author:** [@rschirin](https://discuss.elastic.co/u/rschirin)\ **Replies:** 0\ **Last updated:** [June 8, 2022, 10:46am UTC](https://discuss.elastic.co/t/elasticsearch-input-plugin-used-to-count-documents-is-it-possible/306682 "2022-06-08T10:46:44Z")
Hey there, I was trying to use the Elasticsearch input plugin just to count documents in a specific timerange or to count the specific \_type documents or so on, but I saw that I cannot use the \_count method and I saw al… --- ## [How to identify whether a elasticsearch document is modified or not?](https://discuss.elastic.co/t/how-to-identify-whether-a-elasticsearch-document-is-modified-or-not/306537)
**Author:** [@Souvik\_Das](https://discuss.elastic.co/u/Souvik_Das)\ **Replies:** 1\ **Last updated:** [June 8, 2022, 8:37am UTC](https://discuss.elastic.co/t/how-to-identify-whether-a-elasticsearch-document-is-modified-or-not/306537 "2022-06-08T08:37:38Z")
Hi All, If a logstash pipeline (conf file) is used to update and insert both i.e., enabled below in the conf file. doc\_as\_upsert =\> true action =\> "update" How to know which documents are updated and which are inserte… --- ## [Configuration of Logstash to remove domain from URL](https://discuss.elastic.co/t/configuration-of-logstash-to-remove-domain-from-url/305667)
**Author:** [@wii](https://discuss.elastic.co/u/wii)\ **Replies:** 2\ **Last updated:** [June 8, 2022, 6:06am UTC](https://discuss.elastic.co/t/configuration-of-logstash-to-remove-domain-from-url/305667 "2022-06-08T06:06:17Z")
Is it possible to configure Logstash to remove domain name when receiving URL input? E.g: Picture above as an example, I would like to eliminate everthing behind 'path' to send to Elasticsearch. If it is possible, ma… --- ## [Updating only a single field of an elasticsearch data through logstash](https://discuss.elastic.co/t/updating-only-a-single-field-of-an-elasticsearch-data-through-logstash/306540)
**Author:** [@dishant.sharma](https://discuss.elastic.co/u/dishant.sharma)\ **Replies:** 0\ **Last updated:** [June 7, 2022, 8:37am UTC](https://discuss.elastic.co/t/updating-only-a-single-field-of-an-elasticsearch-data-through-logstash/306540 "2022-06-07T08:37:48Z")
I have a logstash pipeline put into place through which I am getting logs. I want to handle a use-case through logstash configuration in which I have to append a particular value to a field. Currently, it is happening th… --- ## [How to group logstash output files based on incoming input date?](https://discuss.elastic.co/t/how-to-group-logstash-output-files-based-on-incoming-input-date/306275)
**Author:** [@newelastic](https://discuss.elastic.co/u/newelastic)\ **Replies:** 6\ **Last updated:** [June 7, 2022, 10:56pm UTC](https://discuss.elastic.co/t/how-to-group-logstash-output-files-based-on-incoming-input-date/306275 "2022-06-07T22:56:20Z")
Hello, I've thousands of records in my Elasticsearch which span across different dates, month and year. I would like to output the data by year, month and date wise using output plugin. Here is my pipeline. Can someone… --- ## [Logstash - Error to connecting ES and SQL Server](https://discuss.elastic.co/t/logstash-error-to-connecting-es-and-sql-server/306594)
**Author:** [@Furok](https://discuss.elastic.co/u/Furok)\ **Replies:** 0\ **Last updated:** [June 7, 2022, 4:45pm UTC](https://discuss.elastic.co/t/logstash-error-to-connecting-es-and-sql-server/306594 "2022-06-07T16:45:20Z")
Hi guys, I'm trying to connect SQL server with ES using logstash, nevertheless, I have the following error . \[Ruby-0-Thread-9: :1\] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error… --- ## [Parse dynamic field names](https://discuss.elastic.co/t/parse-dynamic-field-names/306558)
**Author:** [@Soren\_vdc](https://discuss.elastic.co/u/Soren_vdc)\ **Replies:** 1\ **Last updated:** [June 7, 2022, 4:43pm UTC](https://discuss.elastic.co/t/parse-dynamic-field-names/306558 "2022-06-07T16:43:27Z")
Hi, I want to parse some code in message field where the field names contains a number. Example: responseGENERATED\_100=76 SENT\_100=76 responseGENERATED\_180=221 SENT\_180=221 responseGENERATED\_190=0 SENT\_183=0 The fil… --- ## [Logstash - Error parsing a concrete value into object field](https://discuss.elastic.co/t/logstash-error-parsing-a-concrete-value-into-object-field/306559)
**Author:** [@Thuunder7](https://discuss.elastic.co/u/Thuunder7)\ **Replies:** 1\ **Last updated:** [June 7, 2022, 4:38pm UTC](https://discuss.elastic.co/t/logstash-error-parsing-a-concrete-value-into-object-field/306559 "2022-06-07T16:38:35Z")
Hello guys, I have a pipeline on Logstash that is parsing and filtering some data from a kafka topic. Everything is working fine but from time to time i got this error message: "error"=\>{"type"=\>"mapper\_parsing\_excepti… --- ## [Do these "could not index" errors actually end up in Elastic anywhere?](https://discuss.elastic.co/t/do-these-could-not-index-errors-actually-end-up-in-elastic-anywhere/306582)
**Author:** [@ckes](https://discuss.elastic.co/u/ckes)\ **Replies:** 2\ **Last updated:** [June 7, 2022, 3:58pm UTC](https://discuss.elastic.co/t/do-these-could-not-index-errors-actually-end-up-in-elastic-anywhere/306582 "2022-06-07T15:58:40Z")
I've noticed quite a few of these errors with the elastic output plugin in /var/log/logstash/logstash-plain.log: "Could not index event to Elasticsearch. {:status=\>400, :action=\>\["create", ... The solution to fix the… --- ## [Using drop to filter messages](https://discuss.elastic.co/t/using-drop-to-filter-messages/306158)
**Author:** [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\ **Replies:** 10\ **Last updated:** [June 7, 2022, 1:24pm UTC](https://discuss.elastic.co/t/using-drop-to-filter-messages/306158 "2022-06-07T13:24:16Z")
Hi All, I am using grok filter to parse messages coming into Logstash from filebeat. We have ELK 7.6.2 stack. I need to filter out and process "only" the following message(s) in the gc log as follows. Note that pretty… --- ## [How to Improve ES responsiveness when getting warning: retrying failed action with response code: 429](https://discuss.elastic.co/t/how-to-improve-es-responsiveness-when-getting-warning-retrying-failed-action-with-response-code-429/305173)
**Author:** [@truptir](https://discuss.elastic.co/u/truptir)\ **Replies:** 2\ **Last updated:** [June 7, 2022, 1:19pm UTC](https://discuss.elastic.co/t/how-to-improve-es-responsiveness-when-getting-warning-retrying-failed-action-with-response-code-429/305173 "2022-06-07T13:19:21Z")
My flow is Logs files will be processed from FileBeat \> Logstash \> Elasticsearch. I m getting the below 2 errors many times in Logstash logs. Error 1: \[INFO \]\[logstash.outputs.elasticsearch\]\[inventory\] retrying faile… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=134) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=136)