# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=313 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 314 --- ## [Persistent queue not working?](https://discuss.elastic.co/t/persistent-queue-not-working/245852)
**Author:** [@TomHome](https://discuss.elastic.co/u/TomHome)\ **Replies:** 0\ **Last updated:** [August 21, 2020, 4:57am UTC](https://discuss.elastic.co/t/persistent-queue-not-working/245852 "2020-08-21T04:57:40Z")
Hi, I'd like to use Persistent Queues (PQ) as an internal overflow buffer when we get spikes of logs into our redis instance (input redis {} in each pipeline). But the PQ does not grow despite our redis instance being s… --- ## [Logstash sincedb file issue](https://discuss.elastic.co/t/logstash-sincedb-file-issue/245831)
**Author:** [@rahulnama](https://discuss.elastic.co/u/rahulnama)\ **Replies:** 0\ **Last updated:** [August 20, 2020, 11:08pm UTC](https://discuss.elastic.co/t/logstash-sincedb-file-issue/245831 "2020-08-20T23:08:08Z")
Hi Team My elk is on docker and I'm mounting sincedb file to mounted volume. I changed the directory of logstash file input and recreated the docker container. I see log data of only one file in elasticsearch though h… --- ## [Free up disk space](https://discuss.elastic.co/t/free-up-disk-space/245799)
**Author:** [@Bruceclegg](https://discuss.elastic.co/u/Bruceclegg)\ **Replies:** 1\ **Last updated:** [August 20, 2020, 5:03pm UTC](https://discuss.elastic.co/t/free-up-disk-space/245799 "2020-08-20T17:03:42Z")
We have an ELK server that periodically runs out of space. In the spring I worked out a method using curator to delete indices that were more than 30 days old. But I still don't know much about ELK - Because everyone i… --- ## [Logstash filter drop message with $ into string](https://discuss.elastic.co/t/logstash-filter-drop-message-with-into-string/245784)
**Author:** [@davide.stallone](https://discuss.elastic.co/u/davide.stallone)\ **Replies:** 9\ **Last updated:** [August 20, 2020, 3:54pm UTC](https://discuss.elastic.co/t/logstash-filter-drop-message-with-into-string/245784 "2020-08-20T15:54:22Z")
Hi, I'm trying to build a logstash filter to drop message that contains dollar at the end of string. I'm receiving log from a domain controller and I would like to get message that has in the field "user.name" only rea… --- ## [Using environment variables](https://discuss.elastic.co/t/using-environment-variables/245621)
**Author:** [@goncalobsantos](https://discuss.elastic.co/u/goncalobsantos)\ **Replies:** 3\ **Last updated:** [August 20, 2020, 3:13pm UTC](https://discuss.elastic.co/t/using-environment-variables/245621 "2020-08-20T15:13:24Z")
I am trying to execute a conf file and get the following error: Connecion refused - failed to open TCP connection The error does not occur when I export the http and https\_proxy variables before executing the conf file… --- ## [SQL Server data to Elasticsearch using LogStash](https://discuss.elastic.co/t/sql-server-data-to-elasticsearch-using-logstash/245488)
**Author:** [@sukumar.koganti](https://discuss.elastic.co/u/sukumar.koganti)\ **Replies:** 30\ **Last updated:** [August 20, 2020, 2:15pm UTC](https://discuss.elastic.co/t/sql-server-data-to-elasticsearch-using-logstash/245488 "2020-08-20T14:15:18Z")
HI I am tryting to integrate the Sql server data to Elasticsearch with Logstash in my local machine. I am using the Microsoft Sql Server 2019 and jdbc using the below --Microsoft JDBC Driver 4.2 for SQL Server This i… --- ## [Create root field for all fields](https://discuss.elastic.co/t/create-root-field-for-all-fields/245740)
**Author:** [@logger](https://discuss.elastic.co/u/logger)\ **Replies:** 2\ **Last updated:** [August 20, 2020, 1:49pm UTC](https://discuss.elastic.co/t/create-root-field-for-all-fields/245740 "2020-08-20T13:49:01Z")
Hi there, I have a question about ruby or anything else if it is possible. @Badger had a nice ruby script a few years ago. But it doesnt work right now and I am not familiar with ruby. ruby { init =\> ' E… --- ## [Print some line to stdout, Logstash filter pipeline](https://discuss.elastic.co/t/print-some-line-to-stdout-logstash-filter-pipeline/245748)
**Author:** [@vaclav1](https://discuss.elastic.co/u/vaclav1)\ **Replies:** 1\ **Last updated:** [August 20, 2020, 1:33pm UTC](https://discuss.elastic.co/t/print-some-line-to-stdout-logstash-filter-pipeline/245748 "2020-08-20T13:33:20Z")
Hello, is there a way to print a line to stdout in filter pipeline some field, or any text? I have found there is stdout plugin to use, but have not found a single correct example how. Would be good to have some possibi… --- ## [Multiple docker container for multiple pipeline of logstash](https://discuss.elastic.co/t/multiple-docker-container-for-multiple-pipeline-of-logstash/245684)
**Author:** [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Replies:** 4\ **Last updated:** [August 20, 2020, 12:56pm UTC](https://discuss.elastic.co/t/multiple-docker-container-for-multiple-pipeline-of-logstash/245684 "2020-08-20T12:56:32Z")
Currently I run 20 pipeline in a system some just pulls data few times a day some just once a day some every few minute I was thinking to run multiple docker image on a single system, where slower pipeline is running… --- ## [JSON parsing depth in logstash](https://discuss.elastic.co/t/json-parsing-depth-in-logstash/245205)
**Author:** [@rakesh15](https://discuss.elastic.co/u/rakesh15)\ **Replies:** 4\ **Last updated:** [August 20, 2020, 11:02am UTC](https://discuss.elastic.co/t/json-parsing-depth-in-logstash/245205 "2020-08-20T11:02:37Z")
Hi All, Im trying to index the camel logs by using grok pattern and json filter in logstash and then push the same to ES. Below is the sample log snippet that I'm trying to parse. 07:32:49.805 \[Camel (MyCamel) thread … --- ## [Two Logstash instances in the same machine are faster than only one](https://discuss.elastic.co/t/two-logstash-instances-in-the-same-machine-are-faster-than-only-one/245753)
**Author:** [@ricferr1976](https://discuss.elastic.co/u/ricferr1976)\ **Replies:** 0\ **Last updated:** [August 20, 2020, 10:58am UTC](https://discuss.elastic.co/t/two-logstash-instances-in-the-same-machine-are-faster-than-only-one/245753 "2020-08-20T10:58:28Z")
Hello. I have a setup with a log source that is sending about 4k EPS and i have an HAProxy that is receiving those messages and sending them to currently only one Logstash machine. That machine has 8 cores and 8GB of R… --- ## [Logstash to azure blob](https://discuss.elastic.co/t/logstash-to-azure-blob/245725)
**Author:** [@nikul\_jain](https://discuss.elastic.co/u/nikul_jain)\ **Replies:** 0\ **Last updated:** [August 20, 2020, 8:10am UTC](https://discuss.elastic.co/t/logstash-to-azure-blob/245725 "2020-08-20T08:10:20Z")
What is the best way to have output to azure blob --- ## [Using alt key code in logstash](https://discuss.elastic.co/t/using-alt-key-code-in-logstash/245094)
**Author:** [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Replies:** 7\ **Last updated:** [August 20, 2020, 7:52am UTC](https://discuss.elastic.co/t/using-alt-key-code-in-logstash/245094 "2020-08-20T07:52:00Z")
Hi all, i am using logstash to write some data into txt file so that my output configuration of logstash is as following: output { csv { fields =\> \["My\_Data"\] path =\> "D:\\app\_%{year}\_%{mounth}\_%{day}.txt" } stdout { c… --- ## [SNMPTRAP v3](https://discuss.elastic.co/t/snmptrap-v3/245722)
**Author:** [@shaigbdb](https://discuss.elastic.co/u/shaigbdb)\ **Replies:** 0\ **Last updated:** [August 20, 2020, 7:47am UTC](https://discuss.elastic.co/t/snmptrap-v3/245722 "2020-08-20T07:47:54Z")
Hi, I'm looking for a way to get information from snmptrap V3 into Elasticsearch. The Logstash snmptrap plugin does not support V3. I was able to get the traps into LS and ES using the UDP plugin, however I'm getting g… --- ## [Logstash collection for the fields newly added in indexes](https://discuss.elastic.co/t/logstash-collection-for-the-fields-newly-added-in-indexes/245710)
**Author:** [@Dayanidhi\_M](https://discuss.elastic.co/u/Dayanidhi_M)\ **Replies:** 0\ **Last updated:** [August 20, 2020, 6:07am UTC](https://discuss.elastic.co/t/logstash-collection-for-the-fields-newly-added-in-indexes/245710 "2020-08-20T06:07:57Z")
How to load logstash collection to exiting indexes which few new fields added to it....am using JDBC plugin to for logstash collection.......please share your inputs and thoughts which is highly helpful for me --- ## [Getting "Encountered a retryable error. Will Retry with exponential backoff {:code=\>403, :url=\>"http://127.0.0.1:8800/\_bulk"}"](https://discuss.elastic.co/t/getting-encountered-a-retryable-error-will-retry-with-exponential-backoff-code-403-url-http-127-0-0-1-8800-bulk/245703)
**Author:** [@mostafa\_kamal](https://discuss.elastic.co/u/mostafa_kamal)\ **Replies:** 1\ **Last updated:** [August 20, 2020, 5:39am UTC](https://discuss.elastic.co/t/getting-encountered-a-retryable-error-will-retry-with-exponential-backoff-code-403-url-http-127-0-0-1-8800-bulk/245703 "2020-08-20T05:39:23Z")
I am trying to send logs from filebeat to logstash, but getting the following error message \[2020-08-20T06:52:31,108\]\[ERROR\]\[logstash.outputs.elasticsearch\]\[main\]\[797474f8edf561358316c00f5117e9f69c7d9881a02a0e167b0d2d76… --- ## [HTTP POLLER NTLM AUTH](https://discuss.elastic.co/t/http-poller-ntlm-auth/245683)
**Author:** [@gutierrezfj](https://discuss.elastic.co/u/gutierrezfj)\ **Replies:** 1\ **Last updated:** [August 20, 2020, 5:04am UTC](https://discuss.elastic.co/t/http-poller-ntlm-auth/245683 "2020-08-20T05:04:29Z")
What is the parameter for an NTML type authentication using HTTP POLLER?. I need to verify the response of many websites but first I must go through the NTML authentication. I appreciate your help! --- ## [Is there a way to specify multiple cacert files](https://discuss.elastic.co/t/is-there-a-way-to-specify-multiple-cacert-files/245694)
**Author:** [@Qingkun\_Li](https://discuss.elastic.co/u/Qingkun_Li)\ **Replies:** 2\ **Last updated:** [August 20, 2020, 3:50am UTC](https://discuss.elastic.co/t/is-there-a-way-to-specify-multiple-cacert-files/245694 "2020-08-20T03:50:31Z")
Can we specify multiple cacert files? I have multiple Elasticsearch clusters configured with the same domain name (e.g. es.xxx.com), and use DNS to decide which Elasticsearch cluster to use. So my logstash output setting… --- ## [Help to customizing Index and Type](https://discuss.elastic.co/t/help-to-customizing-index-and-type/245251)
**Author:** [@Moksha20](https://discuss.elastic.co/u/Moksha20)\ **Replies:** 6\ **Last updated:** [August 19, 2020, 10:58pm UTC](https://discuss.elastic.co/t/help-to-customizing-index-and-type/245251 "2020-08-19T22:58:28Z")
Hi, I have migrated ELK 6.x to ELK 7.8.In previous version I used to ship log using logship.py where as now I have to use filebeats. I am facing two issues here 1)'index' and 'type' I am getting as meta format that is… --- ## [Loaderror: rubygems/security - config file testing](https://discuss.elastic.co/t/loaderror-rubygems-security-config-file-testing/245677)
**Author:** [@biggtimber](https://discuss.elastic.co/u/biggtimber)\ **Replies:** 0\ **Last updated:** [August 19, 2020, 9:00pm UTC](https://discuss.elastic.co/t/loaderror-rubygems-security-config-file-testing/245677 "2020-08-19T21:00:31Z")
I'm running 7.8 that was installed from the ansible-elasticsearch repo. After trying to put in and or/test a logstash config. The test return with this error. (LoadError) load error: rubygems/security -- java.lang.Excep… --- ## [Failed to execute action](https://discuss.elastic.co/t/failed-to-execute-action/245517)
**Author:** [@Feedy](https://discuss.elastic.co/u/Feedy)\ **Replies:** 5\ **Last updated:** [August 19, 2020, 7:51pm UTC](https://discuss.elastic.co/t/failed-to-execute-action/245517 "2020-08-19T19:51:19Z")
Recently upgraded Logstash to 7.8.1 and unable to start Logstash. The error message I am getting: \[2020-08-18T22:56:00,488\]\[ERROR\]\[logstash.agent \] Failed to execute action {:id=\>:main, :action\_type=\>LogStash:… --- ## [Solved: removing domain name from hostname (host.name) with Logstash mutate split and replace filter](https://discuss.elastic.co/t/solved-removing-domain-name-from-hostname-host-name-with-logstash-mutate-split-and-replace-filter/245661)
**Author:** [@Ryan\_Downey](https://discuss.elastic.co/u/Ryan_Downey)\ **Replies:** 0\ **Last updated:** [August 19, 2020, 6:29pm UTC](https://discuss.elastic.co/t/solved-removing-domain-name-from-hostname-host-name-with-logstash-mutate-split-and-replace-filter/245661 "2020-08-19T18:29:20Z")
I ran into an issue in Kibana where the logs for a server were not showing up after I clicked on a server in the Metrics -\> Inventory -\> Host logs. There were however logs in the Winlogbeat index that were specific to e… --- ## [Logstash node is missing from Monitoring in Kibana](https://discuss.elastic.co/t/logstash-node-is-missing-from-monitoring-in-kibana/244789)
**Author:** [@ELK\_User\_Arun](https://discuss.elastic.co/u/ELK_User_Arun)\ **Replies:** 5\ **Last updated:** [August 19, 2020, 6:13pm UTC](https://discuss.elastic.co/t/logstash-node-is-missing-from-monitoring-in-kibana/244789 "2020-08-19T18:13:26Z")
Hi All, We have two logstash nodes running in our DEV cluster and only one is visible in Kibana under monitoring. I compared the logstash.yml file of both the nodes side-by-side but I could not find any differences bet… --- ## [How to get multiple entries of similar pattern from the same input line using grok pattern?](https://discuss.elastic.co/t/how-to-get-multiple-entries-of-similar-pattern-from-the-same-input-line-using-grok-pattern/245208)
**Author:** [@venkata.kodapaka](https://discuss.elastic.co/u/venkata.kodapaka)\ **Replies:** 8\ **Last updated:** [August 19, 2020, 2:56pm UTC](https://discuss.elastic.co/t/how-to-get-multiple-entries-of-similar-pattern-from-the-same-input-line-using-grok-pattern/245208 "2020-08-19T14:56:17Z")
Hi Team, I'm aware of grok pattern and able to parse and store the data into elastic search using logstash configuration file with filter and grok patterns. For example: If the data input line is: Start-field1|field2… --- ## [Elasticsearch index does not recognize date (Logstash pipeline)](https://discuss.elastic.co/t/elasticsearch-index-does-not-recognize-date-logstash-pipeline/245616)
**Author:** [@Luis\_Dieguez](https://discuss.elastic.co/u/Luis_Dieguez)\ **Replies:** 0\ **Last updated:** [August 19, 2020, 1:58pm UTC](https://discuss.elastic.co/t/elasticsearch-index-does-not-recognize-date-logstash-pipeline/245616 "2020-08-19T13:58:56Z")
I've been doing more testing and I'm almost done learning about ELK. But I have a little problem. By creating a pipeline and outputting the elasticsearch cluster as follows: output { elasticsearch { hosts =\> \[… --- ## [Logstash plugin development; dev container image; review requested](https://discuss.elastic.co/t/logstash-plugin-development-dev-container-image-review-requested/245589)
**Author:** [@cknz](https://discuss.elastic.co/u/cknz)\ **Replies:** 0\ **Last updated:** [August 19, 2020, 11:32am UTC](https://discuss.elastic.co/t/logstash-plugin-development-dev-container-image-review-requested/245589 "2020-08-19T11:32:59Z")
Hi everyone, particularly those with an interest in Logstash plugin development and build tooling. I'm tooling up to do some logstash plugin development, and found myself going a bit crazy getting my development environ… --- ## [ADD FIELD OVERWRITES DATA](https://discuss.elastic.co/t/add-field-overwrites-data/244008)
**Author:** [@Daniel\_Lopez](https://discuss.elastic.co/u/Daniel_Lopez)\ **Replies:** 3\ **Last updated:** [August 19, 2020, 9:57am UTC](https://discuss.elastic.co/t/add-field-overwrites-data/244008 "2020-08-19T09:57:46Z")
Hi guys, I'm trying to add data in a field like an array, but my grok or mutate overwrites it, could someone take a look? This is my code: \`filter { if \[message\] =~ "TargetID:" { drop{} } grok { patterns\_dir =\> \[… --- ## [Tail mode in logstash file input is not reading the recently uploaded files](https://discuss.elastic.co/t/tail-mode-in-logstash-file-input-is-not-reading-the-recently-uploaded-files/245538)
**Author:** [@akshatha](https://discuss.elastic.co/u/akshatha)\ **Replies:** 0\ **Last updated:** [August 19, 2020, 7:40am UTC](https://discuss.elastic.co/t/tail-mode-in-logstash-file-input-is-not-reading-the-recently-uploaded-files/245538 "2020-08-19T07:40:18Z")
Hi , I am providing csv file as input to logstash and defined tail mode, but tail mode is not working. the modified files are not getting appended and logstash is reading all the files, but i want it to read the recent… --- ## [Logstash Add Extra Space on a particular field based on the config](https://discuss.elastic.co/t/logstash-add-extra-space-on-a-particular-field-based-on-the-config/245520)
**Author:** [@mastersmit](https://discuss.elastic.co/u/mastersmit)\ **Replies:** 0\ **Last updated:** [August 19, 2020, 4:29am UTC](https://discuss.elastic.co/t/logstash-add-extra-space-on-a-particular-field-based-on-the-config/245520 "2020-08-19T04:29:01Z")
I have a particular requirement where I have extracted out a field based on the GROK patterns. And based on the length of the attribute, i have to add space at the end of the string. For example: Attribute: PartA: tes… --- ## [How not to output a matching field in grok pattern?](https://discuss.elastic.co/t/how-not-to-output-a-matching-field-in-grok-pattern/245435)
**Author:** [@Tristan\_D](https://discuss.elastic.co/u/Tristan_D)\ **Replies:** 2\ **Last updated:** [August 19, 2020, 1:25am UTC](https://discuss.elastic.co/t/how-not-to-output-a-matching-field-in-grok-pattern/245435 "2020-08-19T01:25:28Z")
I don't want the mathed http\_version field in the output. How should I to do avoid it? I can't remove it with the non-caputering group symbol. This is my nginx log 10.102.136.44 - - \[26/Nov/2019:19:49:36 +0800\] "POST … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=312) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=314)