# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=264 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 265 --- ## [Hey i want to add some text into my message can anybody help me how to do it?](https://discuss.elastic.co/t/hey-i-want-to-add-some-text-into-my-message-can-anybody-help-me-how-to-do-it/260321)
**Author:** [@jolaniya](https://discuss.elastic.co/u/jolaniya)\ **Replies:** 4\ **Last updated:** [January 13, 2021, 8:51am UTC](https://discuss.elastic.co/t/hey-i-want-to-add-some-text-into-my-message-can-anybody-help-me-how-to-do-it/260321 "2021-01-13T08:51:23Z")
Hey i want to add some text into my message can anybody help me how to do it? --- ## [Invalid format: \[yyyy-MM-ddTHH:mm:ssZ\]: Unknown pattern letter: T"](https://discuss.elastic.co/t/invalid-format-yyyy-mm-ddthhssz-unknown-pattern-letter-t/260874)
**Author:** [@vaclav1](https://discuss.elastic.co/u/vaclav1)\ **Replies:** 2\ **Last updated:** [January 13, 2021, 7:45am UTC](https://discuss.elastic.co/t/invalid-format-yyyy-mm-ddthhssz-unknown-pattern-letter-t/260874 "2021-01-13T07:45:30Z")
Hello team, strange thing happened. I have following mapping for the field in els with multiple formats: "createdAt" : { "type" : "date", "format" : "yyyy-MM-dd HH:mm:ssZ||yyyy-MM-dd… --- ## [If condition regex error](https://discuss.elastic.co/t/if-condition-regex-error/260939)
**Author:** [@nyquillus](https://discuss.elastic.co/u/nyquillus)\ **Replies:** 1\ **Last updated:** [January 13, 2021, 7:29am UTC](https://discuss.elastic.co/t/if-condition-regex-error/260939 "2021-01-13T07:29:02Z")
Hi, I created a working logstash.conf with if/else conditions. Now I'm trying to implement regex to make the filters less messy. The current filter is like this: filter { if \[tag\] == /^\[a|b|c\]\*$/ { muta… --- ## [Error response code 503](https://discuss.elastic.co/t/error-response-code-503/260940)
**Author:** [@Tabriz](https://discuss.elastic.co/u/Tabriz)\ **Replies:** 0\ **Last updated:** [January 13, 2021, 6:48am UTC](https://discuss.elastic.co/t/error-response-code-503/260940 "2021-01-13T06:48:10Z")
1 on kibana web interface appears that error message \> No result data 2 on redhat 6 logstash service down automatically 3 ithere are n the logstash log file errors a)too many attempts at sending even,dropping b)ret… --- ## [Unable to restrict duplicate documents using http\_poller](https://discuss.elastic.co/t/unable-to-restrict-duplicate-documents-using-http-poller/260511)
**Author:** [@Gauti](https://discuss.elastic.co/u/Gauti)\ **Replies:** 0\ **Last updated:** [January 8, 2021, 8:06am UTC](https://discuss.elastic.co/t/unable-to-restrict-duplicate-documents-using-http-poller/260511 "2021-01-08T08:06:55Z")
Hi All, I have been trying to ingest some, time series data into elasticsearch using logstash http poller plugin. my data is from a monitoring tool which capture the parameters like hostname, memory usage, cpu load, s… --- ## [Document count increase every time with jdbc driver](https://discuss.elastic.co/t/document-count-increase-every-time-with-jdbc-driver/260932)
**Author:** [@beachjf](https://discuss.elastic.co/u/beachjf)\ **Replies:** 0\ **Last updated:** [January 13, 2021, 5:18am UTC](https://discuss.elastic.co/t/document-count-increase-every-time-with-jdbc-driver/260932 "2021-01-13T05:18:41Z")
input { jdbc { jdbc\_driver\_library =\> "" jdbc\_driver\_class =\> "com.microsoft.sqlserver.jdbc.SQLServerDriver" jdbc\_connection\_string =\> "jdbc:sqlserver://aasdf;instanceName=aaSQL;databasename=aaa" jdbc\_u… --- ## [Can the Logstash csv plug-in write to a remote server via sFTP?](https://discuss.elastic.co/t/can-the-logstash-csv-plug-in-write-to-a-remote-server-via-sftp/260815)
**Author:** [@Manuel\_Wong](https://discuss.elastic.co/u/Manuel_Wong)\ **Replies:** 1\ **Last updated:** [January 12, 2021, 11:48pm UTC](https://discuss.elastic.co/t/can-the-logstash-csv-plug-in-write-to-a-remote-server-via-sftp/260815 "2021-01-12T23:48:17Z")
Hello, wish you all an excellent new year! I'd like to ask if the Logstash csv plug-in can write to a remove server via sFTP. I plan to setup the logstash in a container within a K8s cluster but the location where the c… --- ## [Elastic Stack architecture recommandation in production](https://discuss.elastic.co/t/elastic-stack-architecture-recommandation-in-production/260915)
**Author:** [@abdallah](https://discuss.elastic.co/u/abdallah)\ **Replies:** 1\ **Last updated:** [January 12, 2021, 10:53pm UTC](https://discuss.elastic.co/t/elastic-stack-architecture-recommandation-in-production/260915 "2021-01-12T22:53:00Z")
Hi, we have an ELK cluster with 3 ES data, 2 ES master. We have also 2 Logstash and 1 kibana. Is it recommended to have a load balancer between logstash and ES nodes? Thank you --- ## [Logstash stops writing to output file](https://discuss.elastic.co/t/logstash-stops-writing-to-output-file/260860)
**Author:** [@jorism](https://discuss.elastic.co/u/jorism)\ **Replies:** 1\ **Last updated:** [January 12, 2021, 4:17pm UTC](https://discuss.elastic.co/t/logstash-stops-writing-to-output-file/260860 "2021-01-12T16:17:36Z")
We have configured beats to collect access logs on different servers, and write those to logstash. After a while, logstash (latest version stops writing to the output file. The process keeps running but it stops writin… --- ## [Logstash multi pipeline optimal configuration](https://discuss.elastic.co/t/logstash-multi-pipeline-optimal-configuration/260832)
**Author:** [@Dimitrios\_Dellios](https://discuss.elastic.co/u/Dimitrios_Dellios)\ **Replies:** 0\ **Last updated:** [January 12, 2021, 1:27pm UTC](https://discuss.elastic.co/t/logstash-multi-pipeline-optimal-configuration/260832 "2021-01-12T13:27:57Z")
Greetings! I am currently running Logstash 7.9.2, on a production node, with Logstash being the only one of the stack installed there. My node has 8 cores. I am running a multiple pipeline configuration, with some pipel… --- ## [Logstash grok filter not working](https://discuss.elastic.co/t/logstash-grok-filter-not-working/260270)
**Author:** [@vinodhini](https://discuss.elastic.co/u/vinodhini)\ **Replies:** 3\ **Last updated:** [January 12, 2021, 1:12pm UTC](https://discuss.elastic.co/t/logstash-grok-filter-not-working/260270 "2021-01-12T13:12:17Z")
I am trying to get my grok filter working. "\[20/Oct/2020:16:37:57.750 -0500\] BIND RESULT instanceName="abc.com" threadID=9 conn=2312717 op=0 msgID=1 request erIP="123.12.12.1" version="3" dn="uid=xyz,ou=appids,ou=admin… --- ## [\[parsing date error\]: date\_time\_parse\_exception: Failed to parse with all enclosed parsers](https://discuss.elastic.co/t/parsing-date-error-date-time-parse-exception-failed-to-parse-with-all-enclosed-parsers/260804)
**Author:** [@Abdelhalim](https://discuss.elastic.co/u/Abdelhalim)\ **Replies:** 2\ **Last updated:** [January 12, 2021, 11:13am UTC](https://discuss.elastic.co/t/parsing-date-error-date-time-parse-exception-failed-to-parse-with-all-enclosed-parsers/260804 "2021-01-12T11:13:51Z")
Hello, I am using elasticsearch, Kibana and Logstash all version 7.10.1 I would like to ingest data from URLhaus, so I have created a template like that: PUT \_template/urlhaus { "index\_patterns": \["urlhaus-\*"\], "… --- ## [Understanding Unusual Logstash Traffic Flow Rates](https://discuss.elastic.co/t/understanding-unusual-logstash-traffic-flow-rates/260821)
**Author:** [@dawiro](https://discuss.elastic.co/u/dawiro)\ **Replies:** 0\ **Last updated:** [January 12, 2021, 10:59am UTC](https://discuss.elastic.co/t/understanding-unusual-logstash-traffic-flow-rates/260821 "2021-01-12T10:59:28Z")
Hi, I'd like to understand some apparently odd behaviour we're seeing with traffic hitting our logstashes vs what is being ingested into elasticsearch and s3. Note that we're using the output isolator pattern to decoupl… --- ## [How configure logstash in Kubernetes to point to a specific directory that is mapped as emptydir or local](https://discuss.elastic.co/t/how-configure-logstash-in-kubernetes-to-point-to-a-specific-directory-that-is-mapped-as-emptydir-or-local/260814)
**Author:** [@umen](https://discuss.elastic.co/u/umen)\ **Replies:** 0\ **Last updated:** [January 12, 2021, 10:16am UTC](https://discuss.elastic.co/t/how-configure-logstash-in-kubernetes-to-point-to-a-specific-directory-that-is-mapped-as-emptydir-or-local/260814 "2021-01-12T10:16:14Z")
hello all i need to configure logstash to listen to incoming logs in a directory. (which will be manually uploaded logs to ) And to process them and show them in kibana . i already have filebeat monitoring running the… --- ## [Parsing Json from couchDb to ElasticSearch via Logstash](https://discuss.elastic.co/t/parsing-json-from-couchdb-to-elasticsearch-via-logstash/260797)
**Author:** [@skr007](https://discuss.elastic.co/u/skr007)\ **Replies:** 0\ **Last updated:** [January 12, 2021, 8:35am UTC](https://discuss.elastic.co/t/parsing-json-from-couchdb-to-elasticsearch-via-logstash/260797 "2021-01-12T08:35:42Z")
Can anyone please help me with this issue? --- ## [No SSL verification for Logstash filter: Elasticsearch](https://discuss.elastic.co/t/no-ssl-verification-for-logstash-filter-elasticsearch/260489)
**Author:** [@tinhn](https://discuss.elastic.co/u/tinhn)\ **Replies:** 1\ **Last updated:** [January 12, 2021, 7:17am UTC](https://discuss.elastic.co/t/no-ssl-verification-for-logstash-filter-elasticsearch/260489 "2021-01-12T07:17:21Z")
Hello, I am fairly new to setting up security, so I'm sorry if there are holes in my logic. I'm running Elasticsearch (7.9.1 OSS with opendistro) + Kibana on one machine and Logstash (7.8.0) on another machine. Within … --- ## [Google pubsub plugin not working as expected](https://discuss.elastic.co/t/google-pubsub-plugin-not-working-as-expected/260786)
**Author:** [@curiousmind](https://discuss.elastic.co/u/curiousmind)\ **Replies:** 0\ **Last updated:** [January 12, 2021, 5:34am UTC](https://discuss.elastic.co/t/google-pubsub-plugin-not-working-as-expected/260786 "2021-01-12T05:34:18Z")
I am using the google pub-sub input plugin. the following is my input configuration google\_pubsub { project\_id =\> "krail-proj-kgov-rai-doc-92zx4" topic =\> "rail\_logs" subscription =\> "rail… --- ## [Stop Logstash with a Filter or Ruby-Code](https://discuss.elastic.co/t/stop-logstash-with-a-filter-or-ruby-code/259000)
**Author:** [@cwiechmann](https://discuss.elastic.co/u/cwiechmann)\ **Replies:** 3\ **Last updated:** [January 11, 2021, 10:59pm UTC](https://discuss.elastic.co/t/stop-logstash-with-a-filter-or-ruby-code/259000 "2021-01-11T22:59:09Z")
Hi All, I'm using Logstash to prepare/enrich documents before they are send to Elasticsearch. The enrichment is not really optional and will have an impact later, when the data is queried. Therefore, I need to make sure… --- ## [Client write error, trying connect {:e=\>#\ **Author:** [@PraveenKT](https://discuss.elastic.co/u/PraveenKT)\ **Replies:** 3\ **Last updated:** [January 11, 2021, 9:08pm UTC](https://discuss.elastic.co/t/client-write-error-trying-connect-e-ioerror-connection-reset-by-peer/260571 "2021-01-11T21:08:47Z") After restarting logstash, after few hours, beats data is not sending to elasticsearch. Beats ---\> Logstash ----\> logstash ---\> Elasticsearch. I have 2 logstash servers in the site, I am getting below error on one logst… --- ## [Launch Logstash Pipeline through API](https://discuss.elastic.co/t/launch-logstash-pipeline-through-api/260761)
**Author:** [@ctan](https://discuss.elastic.co/u/ctan)\ **Replies:** 0\ **Last updated:** [January 11, 2021, 8:00pm UTC](https://discuss.elastic.co/t/launch-logstash-pipeline-through-api/260761 "2021-01-11T20:00:36Z")
I would like to find a way to launch a Logstash pipeline through the API. The use case is that an application is going to write directly to Elasticsearch, and when it is done we want Logstash to kick-off and do it's thin… --- ## [Logstash connection to Azure mssql](https://discuss.elastic.co/t/logstash-connection-to-azure-mssql/260722)
**Author:** [@vikram\_singh](https://discuss.elastic.co/u/vikram_singh)\ **Replies:** 0\ **Last updated:** [January 11, 2021, 12:56pm UTC](https://discuss.elastic.co/t/logstash-connection-to-azure-mssql/260722 "2021-01-11T12:56:58Z")
Hi, I am trying to connect logstash to azure mssql db. But is is showing Error: unable to load /tmp/mssql-jdbc-8.4.1.jre8.jar from :jdbc\_driver\_library, file not readable (please check user and group permissions for t… --- ## [Logstash, jdbc and .logstash\_jdbc\_last\_run](https://discuss.elastic.co/t/logstash-jdbc-and-logstash-jdbc-last-run/260654)
**Author:** [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Replies:** 3\ **Last updated:** [January 11, 2021, 6:56pm UTC](https://discuss.elastic.co/t/logstash-jdbc-and-logstash-jdbc-last-run/260654 "2021-01-11T18:56:04Z")
In all my cases I do not want to use .logstash\_jdbc\_last\_run how do I avoid using it. I uses following option but it still try to use .logstash\_jdbc\_last\_run and fails sometime when two or more jdbc trying to use that. … --- ## [Convert key value pair into hash](https://discuss.elastic.co/t/convert-key-value-pair-into-hash/260691)
**Author:** [@subash](https://discuss.elastic.co/u/subash)\ **Replies:** 1\ **Last updated:** [January 11, 2021, 3:41pm UTC](https://discuss.elastic.co/t/convert-key-value-pair-into-hash/260691 "2021-01-11T15:41:22Z")
I have a field p1 with the value as partition=root,size=23.28GB,free=15.76GB p1 : partition=root,size=23.28GB,free=15.76GB I want to store this in an array field and the values in a hash, like below. \[partition\]\[root… --- ## [Logstash io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 12766298444, max: 12771524608)](https://discuss.elastic.co/t/logstash-io-netty-util-internal-outofdirectmemoryerror-failed-to-allocate-16777216-byte-s-of-direct-memory-used-12766298444-max-12771524608/260067)
**Author:** [@pavank](https://discuss.elastic.co/u/pavank)\ **Replies:** 6\ **Last updated:** [January 11, 2021, 3:09pm UTC](https://discuss.elastic.co/t/logstash-io-netty-util-internal-outofdirectmemoryerror-failed-to-allocate-16777216-byte-s-of-direct-memory-used-12766298444-max-12771524608/260067 "2021-01-11T15:09:26Z")
Hi All, We are using Logstash 6.3.1 and recently we have been facing issues with getting the log flowing from Filebeat agents. Attaching the configuration and log samples. We have increased the heap memory from 4GB to … --- ## [Dissect filter mapping](https://discuss.elastic.co/t/dissect-filter-mapping/260705)
**Author:** [@ashok9177](https://discuss.elastic.co/u/ashok9177)\ **Replies:** 1\ **Last updated:** [January 11, 2021, 3:08pm UTC](https://discuss.elastic.co/t/dissect-filter-mapping/260705 "2021-01-11T15:08:25Z")
Hi, I want extract only one field after \[HOT\_KV\_LOG\] from the below log message with dissect filter, anyone help me \[HOT\_KV\_LOG\] \[msg\_process\] level\_\_keyword="INFO" type\_\_keyword="MESSAGE\_PROCESSED" e2e\_latency\_\_number… --- ## [Logstash crashing, user error with pipeline and SSL certs configuration](https://discuss.elastic.co/t/logstash-crashing-user-error-with-pipeline-and-ssl-certs-configuration/260612)
**Author:** [@Eden\_Corbin](https://discuss.elastic.co/u/Eden_Corbin)\ **Replies:** 2\ **Last updated:** [January 11, 2021, 12:03pm UTC](https://discuss.elastic.co/t/logstash-crashing-user-error-with-pipeline-and-ssl-certs-configuration/260612 "2021-01-11T12:03:55Z")
I'm struggling a bit getting certs in place for just logstash and filebeat. My goal is to secure log trasnport between VPS instances and my logstash server. Although I'm testing with everything on the same server at the … --- ## [No space left on device with large sql imports despite plenty available](https://discuss.elastic.co/t/no-space-left-on-device-with-large-sql-imports-despite-plenty-available/260706)
**Author:** [@Baygon](https://discuss.elastic.co/u/Baygon)\ **Replies:** 0\ **Last updated:** [January 11, 2021, 10:46am UTC](https://discuss.elastic.co/t/no-space-left-on-device-with-large-sql-imports-despite-plenty-available/260706 "2021-01-11T10:46:43Z")
Hi, I created a brand new instance with a 50Gb SSD disk. 2 vCPU and 12Gb RAM. I'm using logstash to index one large table from a mysql db on another instance. df -h returns 32Gb disk available: root@el1:~# df -h Fil… --- ## [Output only s3 file create events to elastic search and not the contents of the files](https://discuss.elastic.co/t/output-only-s3-file-create-events-to-elastic-search-and-not-the-contents-of-the-files/260697)
**Author:** [@suhas\_1993](https://discuss.elastic.co/u/suhas_1993)\ **Replies:** 1\ **Last updated:** [January 11, 2021, 10:48am UTC](https://discuss.elastic.co/t/output-only-s3-file-create-events-to-elastic-search-and-not-the-contents-of-the-files/260697 "2021-01-11T10:48:02Z")
Hello team, Is there a config that enables Logstash s3 input plugin to output only s3 filenames under the bucket where Logstash config is polling on? The current below config is outputting file contents with message fi… --- ## [Logstash keystore Error](https://discuss.elastic.co/t/logstash-keystore-error/260689)
**Author:** [@baddack](https://discuss.elastic.co/u/baddack)\ **Replies:** 0\ **Last updated:** [January 11, 2021, 9:26am UTC](https://discuss.elastic.co/t/logstash-keystore-error/260689 "2021-01-11T09:26:26Z")
Hello everyone, Can someone help me to solve this error please? adminsys@logstash:/etc/logstash # /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t Thread.exclusive is deprecated, use Thread::Mutex Send… --- ## [SQL data to Elasticsearch using logstash via Mysql connector JDBC plugin](https://discuss.elastic.co/t/sql-data-to-elasticsearch-using-logstash-via-mysql-connector-jdbc-plugin/260686)
**Author:** [@panfan](https://discuss.elastic.co/u/panfan)\ **Replies:** 0\ **Last updated:** [January 11, 2021, 8:46am UTC](https://discuss.elastic.co/t/sql-data-to-elasticsearch-using-logstash-via-mysql-connector-jdbc-plugin/260686 "2021-01-11T08:46:19Z")
Hello, I am having trouble giving input to the elasticearch from data of MySQL via logstash. The following is my simple config file: input { jdbc { clean\_run =\> true jdbc\_driver\_library =\> "mysql-connector-java-5.1… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=263) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=265)