# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=470 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 471 --- ## [Logstash and Filebeat on Kubernetes - Invalid Frame Type, received: 69](https://discuss.elastic.co/t/logstash-and-filebeat-on-kubernetes-invalid-frame-type-received-69/201328)
**Author:** [@iridian](https://discuss.elastic.co/u/iridian)\ **Replies:** 2\ **Last updated:** [September 27, 2019, 6:30pm UTC](https://discuss.elastic.co/t/logstash-and-filebeat-on-kubernetes-invalid-frame-type-received-69/201328 "2019-09-27T18:30:27Z")
This issue matches a few others I've seen. But the difference is I've already ensured my configs do NOT have the same issues that they had: SSL is turned off explicitly on both Logstash AND filebeat/metricbeat I am abs… --- ## [Is there a faster way to test logstash configs with stdin?](https://discuss.elastic.co/t/is-there-a-faster-way-to-test-logstash-configs-with-stdin/201113)
**Author:** [@joaociocca](https://discuss.elastic.co/u/joaociocca)\ **Replies:** 8\ **Last updated:** [September 27, 2019, 4:34pm UTC](https://discuss.elastic.co/t/is-there-a-faster-way-to-test-logstash-configs-with-stdin/201113 "2019-09-27T16:34:16Z")
I found this post from Dec 2018, but the automatic reload doesn't work with stdin (at least up to 6.8, which is my limit at the moment)... Any tips on how to test a developing configuration faster? Or we really have to s… --- ## [Translate keys/fields](https://discuss.elastic.co/t/translate-keys-fields/201127)
**Author:** [@joaociocca](https://discuss.elastic.co/u/joaociocca)\ **Replies:** 1\ **Last updated:** [September 26, 2019, 11:46pm UTC](https://discuss.elastic.co/t/translate-keys-fields/201127 "2019-09-26T23:46:32Z")
So, after that dissect doubt, the new one is: how can I translate keys or fieldnames? The files I'm reading are IAS/NPS, so they come with "key,value" and key being a number. But to better understand, navigate, use at w… --- ## [How can we import only new/updated records if there is no unique/timestamp in sql server (using logstash)](https://discuss.elastic.co/t/how-can-we-import-only-new-updated-records-if-there-is-no-unique-timestamp-in-sql-server-using-logstash/201421)
**Author:** [@rameshkr1994](https://discuss.elastic.co/u/rameshkr1994)\ **Replies:** 1\ **Last updated:** [September 27, 2019, 4:12pm UTC](https://discuss.elastic.co/t/how-can-we-import-only-new-updated-records-if-there-is-no-unique-timestamp-in-sql-server-using-logstash/201421 "2019-09-27T16:12:45Z")
Dear All. i have some data in sql server side but there is no any unique records there but we want to import only newly / updated data from sql server to elastic search as index . please help for this use case. Tha… --- ## [Using gsub to replace an unknown amount of multiple white space to single whitespace](https://discuss.elastic.co/t/using-gsub-to-replace-an-unknown-amount-of-multiple-white-space-to-single-whitespace/201363)
**Author:** [@Jasonespo](https://discuss.elastic.co/u/Jasonespo)\ **Replies:** 2\ **Last updated:** [September 27, 2019, 4:08pm UTC](https://discuss.elastic.co/t/using-gsub-to-replace-an-unknown-amount-of-multiple-white-space-to-single-whitespace/201363 "2019-09-27T16:08:54Z")
Hi guys, Does anyone know how to replace multiple whitespace using gsub? Below doesn't work - but something like this? gsub =\> \[ "syslog\_event", "\\s+", " ", \] Thanks --- ## [Attempted to resurrect connection to dead ES instance, but got an error. java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors](https://discuss.elastic.co/t/attempted-to-resurrect-connection-to-dead-es-instance-but-got-an-error-java-security-cert-certpathvalidatorexception-path-does-not-chain-with-any-of-the-trust-anchors/201423)
**Author:** [@akshatBais](https://discuss.elastic.co/u/akshatBais)\ **Replies:** 0\ **Last updated:** [September 27, 2019, 3:09pm UTC](https://discuss.elastic.co/t/attempted-to-resurrect-connection-to-dead-es-instance-but-got-an-error-java-security-cert-certpathvalidatorexception-path-does-not-chain-with-any-of-the-trust-anchors/201423 "2019-09-27T15:09:57Z")
I put the .cert file in elasticsearch and logstash . My elasticsearch-kibana connection is working fins with ssl config. But wil logstash ssl config it is not somehow connecting or contacting elasticsearch instance. I… --- ## [How to Split the mail body message field into different new fields](https://discuss.elastic.co/t/how-to-split-the-mail-body-message-field-into-different-new-fields/201419)
**Author:** [@Nitin\_Malhar](https://discuss.elastic.co/u/Nitin_Malhar)\ **Replies:** 0\ **Last updated:** [September 27, 2019, 2:42pm UTC](https://discuss.elastic.co/t/how-to-split-the-mail-body-message-field-into-different-new-fields/201419 "2019-09-27T14:42:51Z")
Hi I am new to ELK please may I know how to split mail body into different fields so that i can create a table kind of data out of mail body. --- ## [Update field value if it matches a certain word](https://discuss.elastic.co/t/update-field-value-if-it-matches-a-certain-word/201305)
**Author:** [@naveenrt23](https://discuss.elastic.co/u/naveenrt23)\ **Replies:** 3\ **Last updated:** [September 26, 2019, 8:32pm UTC](https://discuss.elastic.co/t/update-field-value-if-it-matches-a-certain-word/201305 "2019-09-26T20:32:13Z")
Hello, I am trying to replace the value of a field if the field value matches with a certain regex. I am looking at the update function in mutate filter. output: "repo" =\> "test-xyz;buildNumber=2.1", how do i set a … --- ## [Parsing XML data with Xpath](https://discuss.elastic.co/t/parsing-xml-data-with-xpath/201213)
**Author:** [@Hawkeye148](https://discuss.elastic.co/u/Hawkeye148)\ **Replies:** 5\ **Last updated:** [September 27, 2019, 11:29am UTC](https://discuss.elastic.co/t/parsing-xml-data-with-xpath/201213 "2019-09-27T11:29:51Z")
Hello, I am new to Logstash and the whole Elasticsearch stack. I am trying to make a Logstash config file to ingest Cisco Telemetry data. This is the structure of the data I am trying to ingest: \[host 10.10.20.30 sessi… --- ## [\[Solved\] Logstash - Pipeline to Pipeline Config](https://discuss.elastic.co/t/solved-logstash-pipeline-to-pipeline-config/201212)
**Author:** [@jo3rg](https://discuss.elastic.co/u/jo3rg)\ **Replies:** 3\ **Last updated:** [September 27, 2019, 10:34am UTC](https://discuss.elastic.co/t/solved-logstash-pipeline-to-pipeline-config/201212 "2019-09-27T10:34:54Z")
Hi, I'm trying to make our huge logstash configuration easier readable and flexible and therefore thought of splitting it into several pipelines. I came across the pipeline to pipeline communication feature: and al… --- ## [Logstash 7x not reading from log file, same working for 6x versions](https://discuss.elastic.co/t/logstash-7x-not-reading-from-log-file-same-working-for-6x-versions/201252)
**Author:** [@aditya127](https://discuss.elastic.co/u/aditya127)\ **Replies:** 2\ **Last updated:** [September 27, 2019, 9:54am UTC](https://discuss.elastic.co/t/logstash-7x-not-reading-from-log-file-same-working-for-6x-versions/201252 "2019-09-27T09:54:48Z")
Here is my conf file details : input { file { path =\> \["C:\\allLogs.log"\] } } output { elasticsearch { hosts =\> \["localhost:9200"\] index =\> "u\_logs" } stdout { codec =\> rubydebug } } It works for 6.2.4 versio… --- ## [Codec multiline in output section](https://discuss.elastic.co/t/codec-multiline-in-output-section/196985)
**Author:** [@uma\_rengasamy](https://discuss.elastic.co/u/uma_rengasamy)\ **Replies:** 7\ **Last updated:** [September 27, 2019, 8:08am UTC](https://discuss.elastic.co/t/codec-multiline-in-output-section/196985 "2019-09-27T08:08:43Z")
Can we use codec multiline in o/p section with amazon\_es plugin ? --- ## [Get Checkpoint logs that are sending with OPSEC](https://discuss.elastic.co/t/get-checkpoint-logs-that-are-sending-with-opsec/201356)
**Author:** [@gargantua](https://discuss.elastic.co/u/gargantua)\ **Replies:** 0\ **Last updated:** [September 27, 2019, 7:30am UTC](https://discuss.elastic.co/t/get-checkpoint-logs-that-are-sending-with-opsec/201356 "2019-09-27T07:30:02Z")
Hello, I am using Checkpoint as a firewall and Checkpoint sends logs with OPSEC framework that has different layout. I found some really old documentations for integration (using thereby FW1-LogGrabber and OPSEC SDK). S… --- ## [Set a environment variable as a part of the path in Output file](https://discuss.elastic.co/t/set-a-environment-variable-as-a-part-of-the-path-in-output-file/201238)
**Author:** [@nahiko2000](https://discuss.elastic.co/u/nahiko2000)\ **Replies:** 2\ **Last updated:** [September 27, 2019, 7:09am UTC](https://discuss.elastic.co/t/set-a-environment-variable-as-a-part-of-the-path-in-output-file/201238 "2019-09-27T07:09:30Z")
Hi! I have two logstash nodes in a cluster I want both Logstash to output a txt file with a name something like index\_name-2019.09-node1 being "node1" the HOSTNAME of the logstash node. So each logstash node would ou… --- ## [Logstash and filebeat](https://discuss.elastic.co/t/logstash-and-filebeat/201160)
**Author:** [@bcochofel](https://discuss.elastic.co/u/bcochofel)\ **Replies:** 3\ **Last updated:** [September 27, 2019, 6:23am UTC](https://discuss.elastic.co/t/logstash-and-filebeat/201160 "2019-09-27T06:23:55Z")
Hi, I have filebeat configured to send some modules to logstash and on logstash I have a pipeline that handles syslog (file from the documentation) I want to add some more custom logs on filebeat and I'm wondering what… --- ## [Can dissect use a variable number of fields?](https://discuss.elastic.co/t/can-dissect-use-a-variable-number-of-fields/200952)
**Author:** [@joaociocca](https://discuss.elastic.co/u/joaociocca)\ **Replies:** 10\ **Last updated:** [September 26, 2019, 11:47pm UTC](https://discuss.elastic.co/t/can-dissect-use-a-variable-number-of-fields/200952 "2019-09-26T23:47:27Z")
I'm trying to use Logstash to ingest some old, OLD IAS/RRAS logs that seem to be customized. They all begin the same, so I can get dissect to ingest this correctly: filter { dissect { mapping =\> { "message" =\> "%{… --- ## [Struggling with Grok/Dissect](https://discuss.elastic.co/t/struggling-with-grok-dissect/201161)
**Author:** [@jamesp220291](https://discuss.elastic.co/u/jamesp220291)\ **Replies:** 1\ **Last updated:** [September 26, 2019, 10:09pm UTC](https://discuss.elastic.co/t/struggling-with-grok-dissect/201161 "2019-09-26T22:09:13Z")
Hi I am struggling to dissect/use grok on the below log entry - 127.0.0.1 - - \[25/Sep/2019:11:53:28 +0200\] "GET /Request HTTP/1.1" 200 93 "-" "Version/7.1.1 (Build: 194445; Windows 10 (10.0); Zt 1.1.3 windows\_x86\_64) z… --- ## [XML file to elastic search](https://discuss.elastic.co/t/xml-file-to-elastic-search/201321)
**Author:** [@newelk](https://discuss.elastic.co/u/newelk)\ **Replies:** 1\ **Last updated:** [September 26, 2019, 9:59pm UTC](https://discuss.elastic.co/t/xml-file-to-elastic-search/201321 "2019-09-26T21:59:17Z")
Hi Experts, I'm trying a simple usecase using ELK to start with. The usecase is, I've an XML file like below ====================== \ \ \2\ … --- ## [Logstash with Mulesoft Cloud](https://discuss.elastic.co/t/logstash-with-mulesoft-cloud/201298)
**Author:** [@shiv\_khullar](https://discuss.elastic.co/u/shiv_khullar)\ **Replies:** 1\ **Last updated:** [September 26, 2019, 7:29pm UTC](https://discuss.elastic.co/t/logstash-with-mulesoft-cloud/201298 "2019-09-26T19:29:07Z")
Hi, I am new to ELK and would like to get the Mulesoft Cloud Application logs to Logstash. Any idea how can we do it? I am new to Logstash and ELK so any step by step tutorial would be really helpful. --- ## [AWS S3 bucket logs to Elastic search service (AWS) through log stash](https://discuss.elastic.co/t/aws-s3-bucket-logs-to-elastic-search-service-aws-through-log-stash/199737)
**Author:** [@sk545](https://discuss.elastic.co/u/sk545)\ **Replies:** 6\ **Last updated:** [September 26, 2019, 6:35pm UTC](https://discuss.elastic.co/t/aws-s3-bucket-logs-to-elastic-search-service-aws-through-log-stash/199737 "2019-09-26T18:35:49Z")
I set up log stash on EC2 Linux (AWS) and Elastic search service on AWS and i have created one separate file (with input and output) under /etc/logstash/conf.d path but I am not getting logs in Kibana and conn… --- ## [Could not index event to Elasticsearch - Logstash with auth](https://discuss.elastic.co/t/could-not-index-event-to-elasticsearch-logstash-with-auth/201286)
**Author:** [@SirJune](https://discuss.elastic.co/u/SirJune)\ **Replies:** 0\ **Last updated:** [September 26, 2019, 5:39pm UTC](https://discuss.elastic.co/t/could-not-index-event-to-elasticsearch-logstash-with-auth/201286 "2019-09-26T17:39:30Z")
Hi, This is related to my previous posts about LON and LAT. Anyway, i tried adding basic authentication in my stack. Filebeat works with authentication. I followed the docs (7.3) to setup Logstash, one config is not w… --- ## [Connect imap Gmail](https://discuss.elastic.co/t/connect-imap-gmail/201279)
**Author:** [@Klaus\_SIlva](https://discuss.elastic.co/u/Klaus_SIlva)\ **Replies:** 0\ **Last updated:** [September 26, 2019, 5:16pm UTC](https://discuss.elastic.co/t/connect-imap-gmail/201279 "2019-09-26T17:16:44Z")
Hi, I got the problem on logstash when imap.gmai.com. My your configuration in input imap, below with error input { imap { host =\> "imap.gmail.com" user =\> "\*\*\*\*\*@gmail.com" password =\> "\*\*\*… --- ## [Http filter plugin parse error - Unexpected character ('\\\\\\\\' (code 92)): was expecting double-quote to start field name](https://discuss.elastic.co/t/http-filter-plugin-parse-error-unexpected-character-code-92-was-expecting-double-quote-to-start-field-name/201098)
**Author:** [@MMH](https://discuss.elastic.co/u/MMH)\ **Replies:** 1\ **Last updated:** [September 26, 2019, 4:00pm UTC](https://discuss.elastic.co/t/http-filter-plugin-parse-error-unexpected-character-code-92-was-expecting-double-quote-to-start-field-name/201098 "2019-09-26T16:00:18Z")
I am using Logstash 6.5 with Elasticsearch 6.5 What I am trying to achieve is to remove from ELS document fields, that in input has null values. Following advice from other thread, I am planning to use ruby filter to s… --- ## [If statment not working event.dataset](https://discuss.elastic.co/t/if-statment-not-working-event-dataset/201261)
**Author:** [@forestnd](https://discuss.elastic.co/u/forestnd)\ **Replies:** 1\ **Last updated:** [September 26, 2019, 3:02pm UTC](https://discuss.elastic.co/t/if-statment-not-working-event-dataset/201261 "2019-09-26T15:02:02Z")
Hay all I am treying with logstash to add an tag where ever filebeats pass an nignx.access if \[event.dataset\] == "nginx.access" { mutate { add\_tag =\> \[ "nginx" \] } } But for some reason they are not … --- ## [Failed to execute action : Logstash : Pipeline action create error](https://discuss.elastic.co/t/failed-to-execute-action-logstash-pipeline-action-create-error/201182)
**Author:** [@Sushma07](https://discuss.elastic.co/u/Sushma07)\ **Replies:** 1\ **Last updated:** [September 26, 2019, 2:25pm UTC](https://discuss.elastic.co/t/failed-to-execute-action-logstash-pipeline-action-create-error/201182 "2019-09-26T14:25:55Z")
hi, have following setup for Logstash-to-Logstash Communication between 2 servers : Amazon Linux machine Logstash version : 7.x And this is my logstash.conf file: input { stdin { } } output { lumberjack { codec… --- ## [Logstash multiple output parallel](https://discuss.elastic.co/t/logstash-multiple-output-parallel/201151)
**Author:** [@ansarpalakottal](https://discuss.elastic.co/u/ansarpalakottal)\ **Replies:** 1\ **Last updated:** [September 26, 2019, 2:23pm UTC](https://discuss.elastic.co/t/logstash-multiple-output-parallel/201151 "2019-09-26T14:23:35Z")
How can I write parallel output to multiple elasticsearch ? I don't want standard load balancing. --- ## [Can't parse custom grok expression responsible for extracting file extension](https://discuss.elastic.co/t/cant-parse-custom-grok-expression-responsible-for-extracting-file-extension/201247)
**Author:** [@saif3r](https://discuss.elastic.co/u/saif3r)\ **Replies:** 1\ **Last updated:** [September 26, 2019, 2:01pm UTC](https://discuss.elastic.co/t/cant-parse-custom-grok-expression-responsible-for-extracting-file-extension/201247 "2019-09-26T14:01:30Z")
Hello, I'm using custom grok pattern to extract file extension from field called message Here's the configuration snippet: filter { grok { match =\> { "message" =\> "(?\(\\.\[^.\\\\/:\*?"\<\>|\\r\\n\]+$))" } }} … --- ## [Cannot bind to localhost 5000 port](https://discuss.elastic.co/t/cannot-bind-to-localhost-5000-port/201216)
**Author:** [@jacobmathews](https://discuss.elastic.co/u/jacobmathews)\ **Replies:** 0\ **Last updated:** [September 26, 2019, 11:19am UTC](https://discuss.elastic.co/t/cannot-bind-to-localhost-5000-port/201216 "2019-09-26T11:19:49Z")
hi...I am not able to connect to my localhost 5000 port in which I have configured by logstash…I am attaching a screenshot of my "filebeat test output",logstash input conf and filebeat.yml configurations.kindly help me w… --- ## [Logstash pipeline in a transaction](https://discuss.elastic.co/t/logstash-pipeline-in-a-transaction/199661)
**Author:** [@pszemesy](https://discuss.elastic.co/u/pszemesy)\ **Replies:** 2\ **Last updated:** [September 26, 2019, 11:00am UTC](https://discuss.elastic.co/t/logstash-pipeline-in-a-transaction/199661 "2019-09-26T11:00:29Z")
Hi All, I have to create two separate Elasticsearch output indices from the received data with Logstash. Till now it is a simple case, but I have to do the insertion into the indices in a transaction - if the insertion … --- ## [Logstash stop receiving or emitting events to Elasticsearch and sit idle after sometime](https://discuss.elastic.co/t/logstash-stop-receiving-or-emitting-events-to-elasticsearch-and-sit-idle-after-sometime/201203)
**Author:** [@sherifabdlnaby](https://discuss.elastic.co/u/sherifabdlnaby)\ **Replies:** 0\ **Last updated:** [September 26, 2019, 9:58am UTC](https://discuss.elastic.co/t/logstash-stop-receiving-or-emitting-events-to-elasticsearch-and-sit-idle-after-sometime/201203 "2019-09-26T09:58:35Z")
Hello, We're running a single-node ELK cluster (Elasticsearch and Logstash) on the same host, but each in their own docker container, both version 7.3.0. The system has been stable for the past 5 months, ingesting arou… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=469) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=471)