# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=589 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 590 --- ## [An elasticsearch index as logstash input](https://discuss.elastic.co/t/an-elasticsearch-index-as-logstash-input/170833)
**Author:** [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Replies:** 2\ **Last updated:** [March 6, 2019, 10:17am UTC](https://discuss.elastic.co/t/an-elasticsearch-index-as-logstash-input/170833 "2019-03-06T10:17:40Z")
Hi all, I am using following configuration in logstash: input { jdbc { jdbc\_driver\_library =\> "F:\\driver\\sqljdbc\_6.0\\enu\\jre8\\sqljdbc42.jar" jdbc\_driver\_class =\> "com.microsoft.sqlserver.jdbc.SQLServerDriver" … --- ## [Logstash crashed 2 days ago, now not parsing files](https://discuss.elastic.co/t/logstash-crashed-2-days-ago-now-not-parsing-files/170937)
**Author:** [@jamesp220291](https://discuss.elastic.co/u/jamesp220291)\ **Replies:** 6\ **Last updated:** [March 6, 2019, 10:09am UTC](https://discuss.elastic.co/t/logstash-crashed-2-days-ago-now-not-parsing-files/170937 "2019-03-06T10:09:51Z")
Hi My Elastic search crashed 2 days ago, I have since restarted it and all the services are running. However logstash does not seem to be parsing the build up of files, or any new files. New files/logs are fetched fro… --- ## [Merging two indices in logstash based on the same field](https://discuss.elastic.co/t/merging-two-indices-in-logstash-based-on-the-same-field/171044)
**Author:** [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Replies:** 0\ **Last updated:** [March 6, 2019, 7:52am UTC](https://discuss.elastic.co/t/merging-two-indices-in-logstash-based-on-the-same-field/171044 "2019-03-06T07:52:35Z")
hi all, I am using following config file: input { jdbc { jdbc\_driver\_library =\> "F:\\driver\\sqljdbc\_6.0\\enu\\jre8\\sqljdbc42.jar" jdbc\_driver\_class =\> "com.microsoft.sqlserver.jdbc.SQLServerDriver" jdbc\_connec… --- ## [Cannot remove "Type" field added with Logstash](https://discuss.elastic.co/t/cannot-remove-type-field-added-with-logstash/171053)
**Author:** [@a.sailor](https://discuss.elastic.co/u/a.sailor)\ **Replies:** 0\ **Last updated:** [March 6, 2019, 8:31am UTC](https://discuss.elastic.co/t/cannot-remove-type-field-added-with-logstash/171053 "2019-03-06T08:31:52Z")
Hello ELK gurus, i'm completing my company's cluster configuration but now i have one big issue. I have grok patterns configured like this: grok { #portal.redirector-access match =\> { "message" =\> "%{DATA:faceser… --- ## [Duplicate \_id docs and docs with multiple \_id values - how is this possible?](https://discuss.elastic.co/t/duplicate-id-docs-and-docs-with-multiple-id-values-how-is-this-possible/170848)
**Author:** [@cawoodm](https://discuss.elastic.co/u/cawoodm)\ **Replies:** 2\ **Last updated:** [March 6, 2019, 8:12am UTC](https://discuss.elastic.co/t/duplicate-id-docs-and-docs-with-multiple-id-values-how-is-this-possible/170848 "2019-03-06T08:12:52Z")
We have generated an \_id field which we set in our elasticsearch output so that, in case we ever load the same log twice, we don't get duplicate events. However, we are now seeing lots of duplicate \_id fields (see below… --- ## [Too many events](https://discuss.elastic.co/t/too-many-events/170911)
**Author:** [@Costi](https://discuss.elastic.co/u/Costi)\ **Replies:** 6\ **Last updated:** [March 6, 2019, 7:38am UTC](https://discuss.elastic.co/t/too-many-events/170911 "2019-03-06T07:38:02Z")
Hi! I am using jdbc plugin to make a query in an SQL server db and then i import the result of the query in ES. In SQL, when i run select all i get 200.000 rows and after i export them into ES i get over 1 million even… --- ## [Zip file pass through](https://discuss.elastic.co/t/zip-file-pass-through/171008)
**Author:** [@kharvey](https://discuss.elastic.co/u/kharvey)\ **Replies:** 0\ **Last updated:** [March 6, 2019, 12:37am UTC](https://discuss.elastic.co/t/zip-file-pass-through/171008 "2019-03-06T00:37:15Z")
This may be a strange request, but is there a way that I can use Filebeat / Logstash to do a straight pass through of a zip file? I have a site that has 10's of gigs of log files that I want to process a day, but they a… --- ## [Data is not coming as per custom template in kibana from logstash](https://discuss.elastic.co/t/data-is-not-coming-as-per-custom-template-in-kibana-from-logstash/170995)
**Author:** [@anujjai](https://discuss.elastic.co/u/anujjai)\ **Replies:** 1\ **Last updated:** [March 5, 2019, 10:32pm UTC](https://discuss.elastic.co/t/data-is-not-coming-as-per-custom-template-in-kibana-from-logstash/170995 "2019-03-05T22:32:16Z")
Hi , i am getting issue while displaying the content in kibana as per my template and can see error in log as \_grokparsefailure". we can see out template are getting install successfully in elastic search My configura… --- ## [No Logstash errors, but no data](https://discuss.elastic.co/t/no-logstash-errors-but-no-data/170971)
**Author:** [@johnfowles](https://discuss.elastic.co/u/johnfowles)\ **Replies:** 5\ **Last updated:** [March 5, 2019, 10:25pm UTC](https://discuss.elastic.co/t/no-logstash-errors-but-no-data/170971 "2019-03-05T22:25:46Z")
Working indexing and adding fields to OpenVPN logs in pfsense, the original config (stolen from git) works fine and parses out all the pfsense syslog inputs correctly. When I add in the following I get no errors in the … --- ## [Placing Mutliline file into 1 document](https://discuss.elastic.co/t/placing-mutliline-file-into-1-document/170983)
**Author:** [@Elk\_huh](https://discuss.elastic.co/u/Elk_huh)\ **Replies:** 3\ **Last updated:** [March 5, 2019, 9:32pm UTC](https://discuss.elastic.co/t/placing-mutliline-file-into-1-document/170983 "2019-03-05T21:32:23Z")
I have a file that looks like this and I keep getting the first 7 lines as individual documents and the last one its own document. How do i get it so all of these come back as 1 document I have so far filter{ kv {} } i… --- ## [Pipelines.yml pipeline to pipeline address unavailable error](https://discuss.elastic.co/t/pipelines-yml-pipeline-to-pipeline-address-unavailable-error/170808)
**Author:** [@kharvey](https://discuss.elastic.co/u/kharvey)\ **Replies:** 4\ **Last updated:** [March 5, 2019, 9:10pm UTC](https://discuss.elastic.co/t/pipelines-yml-pipeline-to-pipeline-address-unavailable-error/170808 "2019-03-05T21:10:38Z")
Logstash version 6.6.1 Logstash command: /usr/share/logstash/bin/logstash --config.reload.automatic --path.settings=/etc/logstash I am attempting to handle multiple pipelines using the distributor pattern in my pipeli… --- ## [Loop through and translate nested object](https://discuss.elastic.co/t/loop-through-and-translate-nested-object/170742)
**Author:** [@Ant](https://discuss.elastic.co/u/Ant)\ **Replies:** 6\ **Last updated:** [March 5, 2019, 5:33pm UTC](https://discuss.elastic.co/t/loop-through-and-translate-nested-object/170742 "2019-03-05T17:33:09Z")
Hi, I'm pretty sure I need to use ruby to achive this but I don't really know anything about it. I've done a translation with logstash already where I enrich an ID field that occurs once per document. What I need to do… --- ## [LS not ingesting from Redis after LS 5.6 upgrade](https://discuss.elastic.co/t/ls-not-ingesting-from-redis-after-ls-5-6-upgrade/170960)
**Author:** [@drivera](https://discuss.elastic.co/u/drivera)\ **Replies:** 0\ **Last updated:** [March 5, 2019, 5:25pm UTC](https://discuss.elastic.co/t/ls-not-ingesting-from-redis-after-ls-5-6-upgrade/170960 "2019-03-05T17:25:35Z")
Running Redis 3.2.3 and LS 5.6.15. My setup is LS\>Redis\>LS\>ES. Everything in my stack is now 5.6.15. After the (package manager) upgrade I'm no longer getting events from Redis. So Far, I've re-installed the input and ou… --- ## [How to treat JSON as a string](https://discuss.elastic.co/t/how-to-treat-json-as-a-string/170905)
**Author:** [@BKG](https://discuss.elastic.co/u/BKG)\ **Replies:** 4\ **Last updated:** [March 5, 2019, 2:19pm UTC](https://discuss.elastic.co/t/how-to-treat-json-as-a-string/170905 "2019-03-05T14:19:52Z")
Hi, What would be the best way to handle JSON which appears in amongst my normal logs. We don't require parsing of individual fields, just need it to be displayed as a string as this JSON appears very rarely. As you ca… --- ## [Logstash Filter Verifier does not think that my Windows Logstash batch file is executable](https://discuss.elastic.co/t/logstash-filter-verifier-does-not-think-that-my-windows-logstash-batch-file-is-executable/170906)
**Author:** [@Dom\_Ule](https://discuss.elastic.co/u/Dom_Ule)\ **Replies:** 0\ **Last updated:** [March 5, 2019, 1:10pm UTC](https://discuss.elastic.co/t/logstash-filter-verifier-does-not-think-that-my-windows-logstash-batch-file-is-executable/170906 "2019-03-05T13:10:57Z")
Hi all, @magnusbaeck, I hope you can help me with this mystery. I downloaded LFV 1.5.0 and LS 6.6.0 and ran the following on Windows 7 (64 bit): C:\\apps\\ELK\\logstash-filter-verifier\_1.5.0\_windows\_386\>logstash-filter-ve… --- ## [Error when reading log files using logstash](https://discuss.elastic.co/t/error-when-reading-log-files-using-logstash/170891)
**Author:** [@Dheerajbtu](https://discuss.elastic.co/u/Dheerajbtu)\ **Replies:** 6\ **Last updated:** [March 5, 2019, 1:06pm UTC](https://discuss.elastic.co/t/error-when-reading-log-files-using-logstash/170891 "2019-03-05T13:06:56Z")
I have a JSON file with below information. {"apiId":210966762,"apiVersionId":15552430,"orgId":"eeb3ccb6-a2f4-4c7b-9459-7202183bce03","hostId":"mule.qa.2","receivedTs":"2019-01-28T23:59:54.691-05:00","repliedTs":"2019-01… --- ## [Logstash takes a minute to show the help](https://discuss.elastic.co/t/logstash-takes-a-minute-to-show-the-help/170738)
**Author:** [@arcady](https://discuss.elastic.co/u/arcady)\ **Replies:** 3\ **Last updated:** [March 5, 2019, 12:40pm UTC](https://discuss.elastic.co/t/logstash-takes-a-minute-to-show-the-help/170738 "2019-03-05T12:40:42Z")
Hello, I work with windows 10. When I try to run logstash it takes around 1 minute to launch. And I'm only trying to display the help message (-h). Do any of you knows what I'm doing wrong? I tried with WSL (just un c… --- ## [Logstash and grok](https://discuss.elastic.co/t/logstash-and-grok/170859)
**Author:** [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\ **Replies:** 3\ **Last updated:** [March 5, 2019, 11:55am UTC](https://discuss.elastic.co/t/logstash-and-grok/170859 "2019-03-05T11:55:57Z")
What if i import logs without using any grok? --- ## [Logstash S3 input plugin configuration read files from multiple directories](https://discuss.elastic.co/t/logstash-s3-input-plugin-configuration-read-files-from-multiple-directories/170888)
**Author:** [@prabhakar\_talari](https://discuss.elastic.co/u/prabhakar_talari)\ **Replies:** 0\ **Last updated:** [March 5, 2019, 11:52am UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-configuration-read-files-from-multiple-directories/170888 "2019-03-05T11:52:21Z")
Hi All, We have requirement which is little bit different to write configuration logstash. Below are the files and location details s3bucket/logs/prod/todate/prod.txt s3bucket/logs/dev/todate/dev.txt The input confi… --- ## [I need to parse this log message format](https://discuss.elastic.co/t/i-need-to-parse-this-log-message-format/170765)
**Author:** [@Tanya\_Sharma](https://discuss.elastic.co/u/Tanya_Sharma)\ **Replies:** 10\ **Last updated:** [March 5, 2019, 11:29am UTC](https://discuss.elastic.co/t/i-need-to-parse-this-log-message-format/170765 "2019-03-05T11:29:40Z")
Hello everyone, The message format of the logs is: \< \[2019-03-04T12:29:49,990\]\[INFO \]\[logstash.agent \] Successfully started Logstash API endpoint {:port=\>9600} /\> I have to skip the square brackets and parse… --- ## [Reading existing files with S3 Logstash Input Plugin](https://discuss.elastic.co/t/reading-existing-files-with-s3-logstash-input-plugin/170702)
**Author:** [@febrezo](https://discuss.elastic.co/u/febrezo)\ **Replies:** 4\ **Last updated:** [March 5, 2019, 10:08am UTC](https://discuss.elastic.co/t/reading-existing-files-with-s3-logstash-input-plugin/170702 "2019-03-05T10:08:29Z")
Hi! I'm trying to reach some files we have in a S3 instance with Logstash. My .conf file looks like this: input { s3 { "access\_key\_id" =\> "my\_key" "secret\_access\_key" =\> my\_secret\_key" "bucket" =\> "my\_buc… --- ## [Apache error logs](https://discuss.elastic.co/t/apache-error-logs/170856)
**Author:** [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\ **Replies:** 0\ **Last updated:** [March 5, 2019, 9:49am UTC](https://discuss.elastic.co/t/apache-error-logs/170856 "2019-03-05T09:49:27Z")
I am trying to import apache error logs but unable create the pipeline via filebeat. My grok file looks like: input{ beats { port =\> "5044" } } filter{ grok{ match =\> { "message" =\> "%{TIME:timestamp} %{LOGLEVEL… --- ## [Xxx.com:443 failed to respond", :class=\>"Manticore::ClientProtocolException](https://discuss.elastic.co/t/xxx-com-443-failed-to-respond-class-manticore-clientprotocolexception/170826)
**Author:** [@jeraldsm](https://discuss.elastic.co/u/jeraldsm)\ **Replies:** 1\ **Last updated:** [March 5, 2019, 6:41am UTC](https://discuss.elastic.co/t/xxx-com-443-failed-to-respond-class-manticore-clientprotocolexception/170826 "2019-03-05T06:41:24Z")
We are observing intermittent connectivity issues from logstash to elasticsearch: Following error is seen in the logstash logs intermittently: {:timestamp=\>"2019-03-05T05:27:32.888000+0100", :message=\>"Attempted to se… --- ## [Encountered a retryable error. Will Retry with exponential backoff {:code=\>400, :url=\>"http://elasticsearch.topesc:9200/\_bulk"}](https://discuss.elastic.co/t/encountered-a-retryable-error-will-retry-with-exponential-backoff-code-400-url-http-elasticsearch-topesc-9200-bulk/170822)
**Author:** [@sun\_changlong](https://discuss.elastic.co/u/sun_changlong)\ **Replies:** 0\ **Last updated:** [March 5, 2019, 5:29am UTC](https://discuss.elastic.co/t/encountered-a-retryable-error-will-retry-with-exponential-backoff-code-400-url-http-elasticsearch-topesc-9200-bulk/170822 "2019-03-05T05:29:48Z")
logstash.config input { udp { port =\> 55144 type =\> "log" } } output { stdout { codec =\> rubydebug } elasticsearch { hosts =\> \["\*\*\*\*\*\*"\] index =\> "alarm" document\_type =\> "udp" … --- ## [Sequential processing jdbc input to elastic search](https://discuss.elastic.co/t/sequential-processing-jdbc-input-to-elastic-search/170819)
**Author:** [@PagerBae](https://discuss.elastic.co/u/PagerBae)\ **Replies:** 0\ **Last updated:** [March 5, 2019, 3:41am UTC](https://discuss.elastic.co/t/sequential-processing-jdbc-input-to-elastic-search/170819 "2019-03-05T03:41:56Z")
Hi, I am new to logstash and elasticsearch. I have some separated jdbc query that can't be joined by SQL syntax (different source). I want to process it in logstash in this sequential: Run Query A , check if query A … --- ## [Failed to read pipelines yaml file](https://discuss.elastic.co/t/failed-to-read-pipelines-yaml-file/170794)
**Author:** [@wikedeye](https://discuss.elastic.co/u/wikedeye)\ **Replies:** 4\ **Last updated:** [March 5, 2019, 12:41am UTC](https://discuss.elastic.co/t/failed-to-read-pipelines-yaml-file/170794 "2019-03-05T00:41:52Z")
I am sorry if this is an easy problem to fix. I have been trying to figure out what is wrong for a while and unfortunately I do not have enough experience with linux to understand some of what I see as solutions. I am … --- ## [XML request](https://discuss.elastic.co/t/xml-request/170810)
**Author:** [@biggizod](https://discuss.elastic.co/u/biggizod)\ **Replies:** 0\ **Last updated:** [March 4, 2019, 11:52pm UTC](https://discuss.elastic.co/t/xml-request/170810 "2019-03-04T23:52:59Z")
Hi guys. I am trying to build config file in logtash and need your help, Imnew with logtash, please . So We need to connect to Cisco AXL API and send sql request. I have prepared SQL request but don't know how to run … --- ## [CSV \_dateparsefailure error](https://discuss.elastic.co/t/csv-dateparsefailure-error/170401)
**Author:** [@devmal044](https://discuss.elastic.co/u/devmal044)\ **Replies:** 8\ **Last updated:** [March 4, 2019, 8:59pm UTC](https://discuss.elastic.co/t/csv-dateparsefailure-error/170401 "2019-03-04T20:59:33Z")
Not sure whats going on but I keep getting a dateparsefailure tag on all my data that comes in. Ive tried this on two different ELk instances and I still get the tag attached to my data. Any idea whats going on? I have a… --- ## [Separate indexes for each kubernetes namespace](https://discuss.elastic.co/t/separate-indexes-for-each-kubernetes-namespace/169131)
**Author:** [@gurusrinivasamurthy](https://discuss.elastic.co/u/gurusrinivasamurthy)\ **Replies:** 9\ **Last updated:** [March 4, 2019, 8:21pm UTC](https://discuss.elastic.co/t/separate-indexes-for-each-kubernetes-namespace/169131 "2019-03-04T20:21:35Z")
Hi Team Setup: ELK cluster is setup using docker-compose on one beefy bare metal server. On the kubernetes side I am running filebeat as a DaemonSet to ship container logs to logstash. Our K8S cluster will have aroun… --- ## [Index template comment doesn't work](https://discuss.elastic.co/t/index-template-comment-doesnt-work/170788)
**Author:** [@kesha](https://discuss.elastic.co/u/kesha)\ **Replies:** 0\ **Last updated:** [March 4, 2019, 8:14pm UTC](https://discuss.elastic.co/t/index-template-comment-doesnt-work/170788 "2019-03-04T20:14:10Z")
Logstash: 6.5.4 Index template: { "\_rev": "$Rev: 902 $", "\_comment": "Copyright © 2018, CCG, All rights reserved. ", "order": 0, "index\_patterns": \["ossec-alerts-\*"\], "settings": { "index" : { … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=588) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=590)