# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=442 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 443 --- ## [Test filebeat and logstash independently](https://discuss.elastic.co/t/test-filebeat-and-logstash-independently/208997)
**Author:** [@carlsoane](https://discuss.elastic.co/u/carlsoane)\ **Replies:** 1\ **Last updated:** [November 22, 2019, 9:18pm UTC](https://discuss.elastic.co/t/test-filebeat-and-logstash-independently/208997 "2019-11-22T21:18:52Z")
My production logstash pipeline receives events from the filebeat agent using the "beats" input plugin. For unit testing, I would like to be able to decouple filebeat and logstash. I was thinking I would generate a file… --- ## [Problems installing plugins with Docker and no Internet](https://discuss.elastic.co/t/problems-installing-plugins-with-docker-and-no-internet/208865)
**Author:** [@Jonny3](https://discuss.elastic.co/u/Jonny3)\ **Replies:** 3\ **Last updated:** [November 22, 2019, 8:34pm UTC](https://discuss.elastic.co/t/problems-installing-plugins-with-docker-and-no-internet/208865 "2019-11-22T20:34:44Z")
Hello everyone, I'm working in a ELK system with docker and now I'm installing the Logstash plugins Grok, Mutate and Aggregate but I have the problem that my docker image has not access to Internet so I can't use the sim… --- ## [Timeouts for meraki filters 7.4.2](https://discuss.elastic.co/t/timeouts-for-meraki-filters-7-4-2/209115)
**Author:** [@Erin](https://discuss.elastic.co/u/Erin)\ **Replies:** 2\ **Last updated:** [November 22, 2019, 7:48pm UTC](https://discuss.elastic.co/t/timeouts-for-meraki-filters-7-4-2/209115 "2019-11-22T19:48:04Z")
Hi guys i am running into issues with the latest 7.4 logstash. my meraki filters are timing out that the message value is too large. I have tried to set a greedydata on the last field type and others. i am sending these … --- ## [DNS Filter wont work](https://discuss.elastic.co/t/dns-filter-wont-work/209094)
**Author:** [@elasticcp](https://discuss.elastic.co/u/elasticcp)\ **Replies:** 5\ **Last updated:** [November 22, 2019, 7:35pm UTC](https://discuss.elastic.co/t/dns-filter-wont-work/209094 "2019-11-22T19:35:50Z")
Hey i tried to resolve a hostname in to a DNS. My Input data Looks like this: \> { \> "query\_status": "ok", \> "urls": \[ \> { \> "id": "223622", \> "urlhaus\_reference": "https:\\/\\/urlh… --- ## [Logstash running as service is not sending logs](https://discuss.elastic.co/t/logstash-running-as-service-is-not-sending-logs/208777)
**Author:** [@alesabat](https://discuss.elastic.co/u/alesabat)\ **Replies:** 6\ **Last updated:** [November 22, 2019, 7:02pm UTC](https://discuss.elastic.co/t/logstash-running-as-service-is-not-sending-logs/208777 "2019-11-22T19:02:09Z")
Hi there! I have a rhel ppc64le with tomcat. Wanted to send logs to ELK cluster (filebeat discarted because has no ppc version, so I choose logstash). If I run logstash from CLI, it works fine and I can discover logs i… --- ## [Output logs to multiple index from logstash](https://discuss.elastic.co/t/output-logs-to-multiple-index-from-logstash/208991)
**Author:** [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)\ **Replies:** 6\ **Last updated:** [November 22, 2019, 6:51pm UTC](https://discuss.elastic.co/t/output-logs-to-multiple-index-from-logstash/208991 "2019-11-22T18:51:47Z")
Hi there, I am trying to send multiple log files from filebeat tologstash to elasticsearch. In kibana, I would like each log file under a separate index. This is my logstash.conf file: # Sample Logstash configuration f… --- ## [Best method of delimiter splitting](https://discuss.elastic.co/t/best-method-of-delimiter-splitting/209095)
**Author:** [@gborg](https://discuss.elastic.co/u/gborg)\ **Replies:** 2\ **Last updated:** [November 22, 2019, 6:30pm UTC](https://discuss.elastic.co/t/best-method-of-delimiter-splitting/209095 "2019-11-22T18:30:34Z")
Hello I have had a few fields with deliniter seperated strings that I have expanded into their own fields With grok Is that the best method ? Explanation: You have a field: ":foo:bar:goat" I want the first field, I… --- ## [Extract Month and Year from date field](https://discuss.elastic.co/t/extract-month-and-year-from-date-field/209090)
**Author:** [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Replies:** 7\ **Last updated:** [November 22, 2019, 6:17pm UTC](https://discuss.elastic.co/t/extract-month-and-year-from-date-field/209090 "2019-11-22T18:17:22Z")
I am trying to extract Month from date field. But following is actually extracting month from @timestamp and not from first input { generator { message =\> '{"first\_report": "2019-05-30 14:57:59.11"}' coun… --- ## [Logstash 7.1.1 - java memory settings not applied](https://discuss.elastic.co/t/logstash-7-1-1-java-memory-settings-not-applied/209101)
**Author:** [@ArunANayagam](https://discuss.elastic.co/u/ArunANayagam)\ **Replies:** 0\ **Last updated:** [November 22, 2019, 5:42pm UTC](https://discuss.elastic.co/t/logstash-7-1-1-java-memory-settings-not-applied/209101 "2019-11-22T17:42:45Z")
Hi, I have a docker-compose file like this, version: "3.4" services: logstash: build: context: . volumes: - /opt/elk-logstash/conf/pipeline:/usr/share/logstash/pipeline/ - /opt/elk-logstash/… --- ## [Logstash 7.3.2 deployment not starting after upgrade from 7.1.1](https://discuss.elastic.co/t/logstash-7-3-2-deployment-not-starting-after-upgrade-from-7-1-1/209097)
**Author:** [@zgergely](https://discuss.elastic.co/u/zgergely)\ **Replies:** 0\ **Last updated:** [November 22, 2019, 4:42pm UTC](https://discuss.elastic.co/t/logstash-7-3-2-deployment-not-starting-after-upgrade-from-7-1-1/209097 "2019-11-22T16:42:41Z")
Hi, We have an ELK 7.1.1 deployment running fine on IKS (IBMCLOUD Kubernetes Service). We tried a re-deploy changing the 7.1.1 images for all the components with 7.3.2. All are working fine except logstash is not starti… --- ## [Logstash 6.8.4 and 6.8.5 jdbc input plugin erasing last\_run\_metadata](https://discuss.elastic.co/t/logstash-6-8-4-and-6-8-5-jdbc-input-plugin-erasing-last-run-metadata/209096)
**Author:** [@lpkirby](https://discuss.elastic.co/u/lpkirby)\ **Replies:** 0\ **Last updated:** [November 22, 2019, 4:42pm UTC](https://discuss.elastic.co/t/logstash-6-8-4-and-6-8-5-jdbc-input-plugin-erasing-last-run-metadata/209096 "2019-11-22T16:42:31Z")
Starting in Logstash 6.8.4 and still present in logstash 6.8.5... Whenever I start or restart logstash all of the files defined in all the jdbc input last\_run\_metadata\_path settings are erased. This causes all my queri… --- ## [Grok pattern match but replace a character from result](https://discuss.elastic.co/t/grok-pattern-match-but-replace-a-character-from-result/209084)
**Author:** [@GaryD007](https://discuss.elastic.co/u/GaryD007)\ **Replies:** 2\ **Last updated:** [November 22, 2019, 4:22pm UTC](https://discuss.elastic.co/t/grok-pattern-match-but-replace-a-character-from-result/209084 "2019-11-22T16:22:27Z")
Hi all I am trying to extract the timestamp from a log file but the date and time are separated by a semicolon. Is there a way I can make my Grok pattern work but replace the semicolon with a space ? Here is a sample f… --- ## [Logstash multiple indices](https://discuss.elastic.co/t/logstash-multiple-indices/209071)
**Author:** [@111245](https://discuss.elastic.co/u/111245)\ **Replies:** 0\ **Last updated:** [November 22, 2019, 1:59pm UTC](https://discuss.elastic.co/t/logstash-multiple-indices/209071 "2019-11-22T13:59:16Z")
Hello! I ask for help The question is: I send data using filebeat to logstash, logstash sends to elasticsearch All logs are written in a single logstash- \* indices I would like to write logs from different servers to… --- ## [GROK DEBUGGER](https://discuss.elastic.co/t/grok-debugger/208490)
**Author:** [@Uma1](https://discuss.elastic.co/u/Uma1)\ **Replies:** 2\ **Last updated:** [November 22, 2019, 1:33pm UTC](https://discuss.elastic.co/t/grok-debugger/208490 "2019-11-22T13:33:49Z")
19/Nov/2019:10:30:57 +0530 in grok debugger What will be the syntax for this type of logs --- ## [Backpressure Handling with new Traffic](https://discuss.elastic.co/t/backpressure-handling-with-new-traffic/209061)
**Author:** [@skaldenmet](https://discuss.elastic.co/u/skaldenmet)\ **Replies:** 0\ **Last updated:** [November 22, 2019, 12:50pm UTC](https://discuss.elastic.co/t/backpressure-handling-with-new-traffic/209061 "2019-11-22T12:50:46Z")
Hi, I couldnt find any information on how logstash is handling new traffic when a larger persistent queue is in place. Does logstash prioritize new traffic or does new traffic get into the back of the persistent queue l… --- ## [Getting Grok Parse failure error](https://discuss.elastic.co/t/getting-grok-parse-failure-error/209021)
**Author:** [@akki12ka4](https://discuss.elastic.co/u/akki12ka4)\ **Replies:** 1\ **Last updated:** [November 22, 2019, 11:53am UTC](https://discuss.elastic.co/t/getting-grok-parse-failure-error/209021 "2019-11-22T11:53:58Z")
Hi, I am getting grok parse error for below input: 1999001529.000000\[20291121 01:45:04.000000\] Feature: dummy/2.1/subdummy/myevent Event: One or more users unavailable. Please grant the acess. Id: 1501 Type\_Id: 107696… --- ## [Looks like RabbitMQ output plugin does not support ACK/NACK](https://discuss.elastic.co/t/looks-like-rabbitmq-output-plugin-does-not-support-ack-nack/208484)
**Author:** [@gab.bernasconi](https://discuss.elastic.co/u/gab.bernasconi)\ **Replies:** 3\ **Last updated:** [November 22, 2019, 11:35am UTC](https://discuss.elastic.co/t/looks-like-rabbitmq-output-plugin-does-not-support-ack-nack/208484 "2019-11-22T11:35:55Z")
Our use case is as follows: ELK target version: 6.8.2 Data comes in via Logstash, where it gets parsed and enriched All the data needs to go to Elasticsearch (we use the elasticsearch output plugin) Some of the data ne… --- ## [Replace logstash with kafka](https://discuss.elastic.co/t/replace-logstash-with-kafka/208635)
**Author:** [@varunjithtk](https://discuss.elastic.co/u/varunjithtk)\ **Replies:** 3\ **Last updated:** [November 22, 2019, 11:33am UTC](https://discuss.elastic.co/t/replace-logstash-with-kafka/208635 "2019-11-22T11:33:37Z")
If I all my log generating modules can write into kafka why do i need logstash when kafka can directly write into elastic. Is there any disadvantage if I use this setup? One advantage I see is , in my understanding Logst… --- ## [How to unistall logstash using yum on CentOS](https://discuss.elastic.co/t/how-to-unistall-logstash-using-yum-on-centos/209010)
**Author:** [@rishab](https://discuss.elastic.co/u/rishab)\ **Replies:** 3\ **Last updated:** [November 22, 2019, 11:03am UTC](https://discuss.elastic.co/t/how-to-unistall-logstash-using-yum-on-centos/209010 "2019-11-22T11:03:37Z")
I am using logstash on apache server on my AWS instance. --- ## [Logstash (6.8.4) template handling and include\_type\_name](https://discuss.elastic.co/t/logstash-6-8-4-template-handling-and-include-type-name/207496)
**Author:** [@mw\_jko](https://discuss.elastic.co/u/mw_jko)\ **Replies:** 5\ **Last updated:** [November 22, 2019, 9:37am UTC](https://discuss.elastic.co/t/logstash-6-8-4-template-handling-and-include-type-name/207496 "2019-11-22T09:37:20Z")
Hi there, I'm currently running an ELK stack with version 6.8.4 in a single node setup and preparing it for migration to 7.4.x. My data is stored through filebeat -\> logstash -\> elasticsearch where logstash also does t… --- ## [Logstash 6.4.2 to use index pattern for new indices](https://discuss.elastic.co/t/logstash-6-4-2-to-use-index-pattern-for-new-indices/209033)
**Author:** [@samiujan](https://discuss.elastic.co/u/samiujan)\ **Replies:** 0\ **Last updated:** [November 22, 2019, 9:18am UTC](https://discuss.elastic.co/t/logstash-6-4-2-to-use-index-pattern-for-new-indices/209033 "2019-11-22T09:18:09Z")
Hi I am using Logstash 6.4.2 to read and output data to a new ES 6.4.2 index every day - Each index has the pattern "filebeat-YYYY-MM-DD" I want the new index to use a specific template I have added an index template … --- ## [Is it possible skip columns from csv file while uploading data into elasticsearch?](https://discuss.elastic.co/t/is-it-possible-skip-columns-from-csv-file-while-uploading-data-into-elasticsearch/208901)
**Author:** [@sanket07](https://discuss.elastic.co/u/sanket07)\ **Replies:** 4\ **Last updated:** [November 21, 2019, 5:53pm UTC](https://discuss.elastic.co/t/is-it-possible-skip-columns-from-csv-file-while-uploading-data-into-elasticsearch/208901 "2019-11-21T17:53:49Z")
I have around 50 columns in csv but only 10 are useful for me. Is there any way that when I upload the data, only data for that 10 columns should be inserted in elasticsearch and only those 10 fileds should appear in kib… --- ## [GROK, regexp, string. Help!](https://discuss.elastic.co/t/grok-regexp-string-help/208818)
**Author:** [@user112233](https://discuss.elastic.co/u/user112233)\ **Replies:** 5\ **Last updated:** [November 22, 2019, 6:27am UTC](https://discuss.elastic.co/t/grok-regexp-string-help/208818 "2019-11-22T06:27:43Z")
Trying to parse a log file line using GROK. Line: 14-nov-2019 3:03:37.767 queries: client @000000986596B420 10.10.35.11#49780 (31.22.11.10.in-addr.arpa): query: 31.22.11.10.in-addr.arpa IN PTR + (10.10.36.10) This fil… --- ## [Logstash is not able to send complete data](https://discuss.elastic.co/t/logstash-is-not-able-to-send-complete-data/208986)
**Author:** [@Avinaba\_Ghosh\_Hajra](https://discuss.elastic.co/u/Avinaba_Ghosh_Hajra)\ **Replies:** 1\ **Last updated:** [November 22, 2019, 6:10am UTC](https://discuss.elastic.co/t/logstash-is-not-able-to-send-complete-data/208986 "2019-11-22T06:10:41Z")
Logstash is running with a pipeline from command line with w = 1. After successfully starting, it sends some data successfully to elasticsearch. But after sending dome data it will not able so send the data with the fol… --- ## [Logtstash not recognizing the columns created in Geojson format](https://discuss.elastic.co/t/logtstash-not-recognizing-the-columns-created-in-geojson-format/208832)
**Author:** [@Ramya\_Tanikanti](https://discuss.elastic.co/u/Ramya_Tanikanti)\ **Replies:** 0\ **Last updated:** [November 21, 2019, 8:47am UTC](https://discuss.elastic.co/t/logtstash-not-recognizing-the-columns-created-in-geojson-format/208832 "2019-11-21T08:47:22Z")
Hi, I have fields like A\_Latitude, A\_Longitude, B\_Latitude and B\_Longitude. I would like to make use of this data and create Maps in Kibana. The problem is data is getting into elasticsearch, but the gejson columns creat… --- ## [Failed to read pipelines yaml file even when in correct folder](https://discuss.elastic.co/t/failed-to-read-pipelines-yaml-file-even-when-in-correct-folder/208987)
**Author:** [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)\ **Replies:** 2\ **Last updated:** [November 21, 2019, 11:46pm UTC](https://discuss.elastic.co/t/failed-to-read-pipelines-yaml-file-even-when-in-correct-folder/208987 "2019-11-21T23:46:04Z")
Hi, My pipelines.yml is in the logstash/config folder and I still get this error- Settings ------------------- \[2019-11-21T13:50:15,239\]\[DEBUG\]\[logstash.config.source.multilocal\] Reading pipeline configurations from YA… --- ## [Is there a disadvantage to setting a large client\_inactivity\_timeout?](https://discuss.elastic.co/t/is-there-a-disadvantage-to-setting-a-large-client-inactivity-timeout/208789)
**Author:** [@seanziee](https://discuss.elastic.co/u/seanziee)\ **Replies:** 4\ **Last updated:** [November 21, 2019, 11:35pm UTC](https://discuss.elastic.co/t/is-there-a-disadvantage-to-setting-a-large-client-inactivity-timeout/208789 "2019-11-21T23:35:50Z")
I have a set of logs that sometimes only sends once every 3 days. Is it okay if I set the client\_inactivity\_timeout =\> 300000? What is the advantage to setting this value to a lower vs a higher value? Can I set it to 999… --- ## [Issues Installing Amazon\_ES Plug In in Windows](https://discuss.elastic.co/t/issues-installing-amazon-es-plug-in-in-windows/208794)
**Author:** [@mcburn13](https://discuss.elastic.co/u/mcburn13)\ **Replies:** 1\ **Last updated:** [November 21, 2019, 9:57pm UTC](https://discuss.elastic.co/t/issues-installing-amazon-es-plug-in-in-windows/208794 "2019-11-21T21:57:35Z")
Trying to plug in my IAM creds to get through an ES access policy in AWS, and getting an error that can't find the amazon\_es plugin. So, there seems to be very little specific information on installing this plug in on Wi… --- ## [Where the conditional goes?](https://discuss.elastic.co/t/where-the-conditional-goes/208960)
**Author:** [@Incauto](https://discuss.elastic.co/u/Incauto)\ **Replies:** 9\ **Last updated:** [November 21, 2019, 8:07pm UTC](https://discuss.elastic.co/t/where-the-conditional-goes/208960 "2019-11-21T20:07:42Z")
where in the file a conditional that check if a field contains certain string, and if contains it, drop the log, goes? it goes after the grok? it contains the grok? it goes in the output? The following code doesnt work… --- ## [Working with inconsistent logs](https://discuss.elastic.co/t/working-with-inconsistent-logs/208968)
**Author:** [@Hyperion0](https://discuss.elastic.co/u/Hyperion0)\ **Replies:** 0\ **Last updated:** [November 21, 2019, 7:23pm UTC](https://discuss.elastic.co/t/working-with-inconsistent-logs/208968 "2019-11-21T19:23:52Z")
This night be a silly question but. Is there a reasonable way to handle inconsistencies in text logs? For example I have a text log, the bulk of witch is well formed but at certain points carriage returns are inserted i… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=441) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=443)