# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=387 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 388 --- ## [Dynamic variable for http poller input plugin](https://discuss.elastic.co/t/dynamic-variable-for-http-poller-input-plugin/223912)
**Author:** [@elk2](https://discuss.elastic.co/u/elk2)\ **Replies:** 4\ **Last updated:** [March 20, 2020, 1:08pm UTC](https://discuss.elastic.co/t/dynamic-variable-for-http-poller-input-plugin/223912 "2020-03-20T13:08:15Z")
I want to download and parse a list of csv with yearly weather data with logstash. "YEAR" would be my variable and I need to increase from 1998 to 2019. I tried to use this simple conf file but not even works. input {… --- ## [Problem with logstash-plain.log](https://discuss.elastic.co/t/problem-with-logstash-plain-log/223795)
**Author:** [@Pablo95](https://discuss.elastic.co/u/Pablo95)\ **Replies:** 16\ **Last updated:** [March 20, 2020, 12:51pm UTC](https://discuss.elastic.co/t/problem-with-logstash-plain-log/223795 "2020-03-20T12:51:07Z")
Hi, i'm having problems with my logstash-plain.log file. When I execute the "less" command, this is the code that i receive: \[2020-03-06T11:35:43,304\]\[WARN \]\[logstash.config.source.multilocal\] Ignoring the 'pipelines.ym… --- ## [Compressing RabbitMQ](https://discuss.elastic.co/t/compressing-rabbitmq/224419)
**Author:** [@KevSex](https://discuss.elastic.co/u/KevSex)\ **Replies:** 0\ **Last updated:** [March 20, 2020, 12:48pm UTC](https://discuss.elastic.co/t/compressing-rabbitmq/224419 "2020-03-20T12:48:44Z")
Is there any documentation or examples anyone can provide on how to gzip documents before sending to a rabbitmq-output? (can't see any configuration options for compressing documents on the logstash-rabbitmq-output) In … --- ## [CSV overwrite/ delete index](https://discuss.elastic.co/t/csv-overwrite-delete-index/224196)
**Author:** [@FoRMaTC](https://discuss.elastic.co/u/FoRMaTC)\ **Replies:** 4\ **Last updated:** [March 20, 2020, 8:45am UTC](https://discuss.elastic.co/t/csv-overwrite-delete-index/224196 "2020-03-20T08:45:45Z")
Hi, I'm downloading a CSV from our ticketing system and I want to import that into elastic every 15 minutes. However I want to start with a clean slate over and over again. What i've done now is wrote a small perl scr… --- ## [Logstash.Config syntax/query/error](https://discuss.elastic.co/t/logstash-config-syntax-query-error/224324)
**Author:** [@Joshua\_Wilford](https://discuss.elastic.co/u/Joshua_Wilford)\ **Replies:** 1\ **Last updated:** [March 20, 2020, 12:44am UTC](https://discuss.elastic.co/t/logstash-config-syntax-query-error/224324 "2020-03-20T00:44:39Z")
Hey Community, Been beating my head against the wall all week over this error. I think its syntax but I can't find the error. Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :excep… --- ## [Enriching Docs with Cached Elasticsearch Data](https://discuss.elastic.co/t/enriching-docs-with-cached-elasticsearch-data/224325)
**Author:** [@scottdfedorov](https://discuss.elastic.co/u/scottdfedorov)\ **Replies:** 0\ **Last updated:** [March 19, 2020, 11:06pm UTC](https://discuss.elastic.co/t/enriching-docs-with-cached-elasticsearch-data/224325 "2020-03-19T23:06:35Z")
Howdy, I am curious if anyone has used the jdbc\_static (or streaming) filter plugin to query and cache Elasticsearch data? I am aware of the elasticsearch filter plugin, but that plugin does not cache data. We store so… --- ## [SNMP map Values to Router Interface](https://discuss.elastic.co/t/snmp-map-values-to-router-interface/224312)
**Author:** [@hitman22](https://discuss.elastic.co/u/hitman22)\ **Replies:** 0\ **Last updated:** [March 19, 2020, 8:52pm UTC](https://discuss.elastic.co/t/snmp-map-values-to-router-interface/224312 "2020-03-19T20:52:26Z")
Hello, I am query my Router with SNMP with Logstash and I want to ask if it is possible to map the Fields for example ifHCOutOctets.1 to GigabitEthner1.ifHCOutOctets or something like that and the same also for ifHCInO… --- ## [Logstash Regex Unrecognized character escape](https://discuss.elastic.co/t/logstash-regex-unrecognized-character-escape/224298)
**Author:** [@james.garside](https://discuss.elastic.co/u/james.garside)\ **Replies:** 0\ **Last updated:** [March 19, 2020, 6:22pm UTC](https://discuss.elastic.co/t/logstash-regex-unrecognized-character-escape/224298 "2020-03-19T18:22:45Z")
Hello all, I have this regex which is working for most names but throws an error for some. It seems my use of escape charaters is wrong. \[client\]\[user\]\[name\] =~ /staff\\\\.\*/ or \[client\]\[user\]\[name\] =~ /students\\\\.\*/ … --- ## [Logstash pipeline check for index](https://discuss.elastic.co/t/logstash-pipeline-check-for-index/224179)
**Author:** [@zozo6015](https://discuss.elastic.co/u/zozo6015)\ **Replies:** 4\ **Last updated:** [March 19, 2020, 4:44pm UTC](https://discuss.elastic.co/t/logstash-pipeline-check-for-index/224179 "2020-03-19T16:44:12Z")
Hello, Is there a way to check in logstash pipeline if the exists. I am trying to use elasticsearch filter but I am getting an error on the first document because the index is not yet created. Regards, Peter --- ## [Logstash geoip that works with Kibana SIEM UI](https://discuss.elastic.co/t/logstash-geoip-that-works-with-kibana-siem-ui/224288)
**Author:** [@Vincent\_Maury](https://discuss.elastic.co/u/Vincent_Maury)\ **Replies:** 0\ **Last updated:** [March 19, 2020, 3:46pm UTC](https://discuss.elastic.co/t/logstash-geoip-that-works-with-kibana-siem-ui/224288 "2020-03-19T15:46:18Z")
I'm using Logstash 7.6.0 with Elasticsearch & Kibana, and I figured out we need to change a couple of things to have geoip work with SIEM UI: The Logstash config sample: input { generator { count =\> 1 message =\> "… --- ## [Create map visualizsation of alb logs data](https://discuss.elastic.co/t/create-map-visualizsation-of-alb-logs-data/224283)
**Author:** [@vinayak\_k\_g](https://discuss.elastic.co/u/vinayak_k_g)\ **Replies:** 0\ **Last updated:** [March 19, 2020, 3:15pm UTC](https://discuss.elastic.co/t/create-map-visualizsation-of-alb-logs-data/224283 "2020-03-19T15:15:51Z")
Hi We have setup alb logs to show on our kibana dashboards successfully so far. Now we want to map the geography data. Have followed this article created the geo field but the data shown for geo ip is 0,0. Please le… --- ## [Trying create index in conf.d with propeties](https://discuss.elastic.co/t/trying-create-index-in-conf-d-with-propeties/224092)
**Author:** [@Marcos\_Gomes](https://discuss.elastic.co/u/Marcos_Gomes)\ **Replies:** 2\ **Last updated:** [March 19, 2020, 2:43pm UTC](https://discuss.elastic.co/t/trying-create-index-in-conf-d-with-propeties/224092 "2020-03-19T14:43:54Z")
I have to put the property "index.max\_docvalue\_fields\_search": "1600" I've tried with query =\> "index.max\_docvalue\_fields\_search": "1600" I tried to put index =\> "max\_docvalue\_fields\_search": "1600" And also my last a… --- ## [Logstash 7.5.0 as Windows service using SC commands on CMD](https://discuss.elastic.co/t/logstash-7-5-0-as-windows-service-using-sc-commands-on-cmd/224268)
**Author:** [@sagardhakate](https://discuss.elastic.co/u/sagardhakate)\ **Replies:** 0\ **Last updated:** [March 19, 2020, 1:00pm UTC](https://discuss.elastic.co/t/logstash-7-5-0-as-windows-service-using-sc-commands-on-cmd/224268 "2020-03-19T13:00:58Z")
I am not able to run logstash 7.5.0 as windows service using SC commands on command prompt. C:\>sc \\VIMPABCD-22 create "Elasticsearch Logstash-7.5.0" binpath=C:\\Users\\sagar.dhakate\\Software\\ElasticStack\\logstash-7.5.0\\lo… --- ## [Index name not recognize +YYYY.MM.dd some times](https://discuss.elastic.co/t/index-name-not-recognize-yyyy-mm-dd-some-times/224140)
**Author:** [@Jose\_Antonio\_Arenas](https://discuss.elastic.co/u/Jose_Antonio_Arenas)\ **Replies:** 1\ **Last updated:** [March 19, 2020, 11:08am UTC](https://discuss.elastic.co/t/index-name-not-recognize-yyyy-mm-dd-some-times/224140 "2020-03-19T11:08:04Z")
Hi, When I try to create an index in logstash output using: elasticsearch { hosts =\> "myelasticsearch:9200" index =\> "my\_index-%{+YYYY.MM.dd}" } In the log of elasticsearch the name of the index is "my\_index-" and… --- ## [Issue With Parsing Unique Problem. Please Please help](https://discuss.elastic.co/t/issue-with-parsing-unique-problem-please-please-help/223924)
**Author:** [@pariksh](https://discuss.elastic.co/u/pariksh)\ **Replies:** 15\ **Last updated:** [March 19, 2020, 9:21am UTC](https://discuss.elastic.co/t/issue-with-parsing-unique-problem-please-please-help/223924 "2020-03-19T09:21:34Z")
Hy Everyone myfile.log 17:54:24,429 DEBUG ExUsernamePasswordAuthenticationFilter.successfulAuthentication():319 - Authentication success. Updating SecurityContextHolder to contain: org.springframework.security.authenti… --- ## [Using Logstasg to enrich data](https://discuss.elastic.co/t/using-logstasg-to-enrich-data/224233)
**Author:** [@GVK](https://discuss.elastic.co/u/GVK)\ **Replies:** 0\ **Last updated:** [March 19, 2020, 9:07am UTC](https://discuss.elastic.co/t/using-logstasg-to-enrich-data/224233 "2020-03-19T09:07:54Z")
Hey there! I have some inquiries about the usage of Logstash. I have a few "Crawlers" that request controllers param. I successfully inserted them into elasticsearch but the data had no structure and thus made it hard … --- ## [UsingWinlogbeat and filebeat](https://discuss.elastic.co/t/usingwinlogbeat-and-filebeat/224217)
**Author:** [@Sahana1](https://discuss.elastic.co/u/Sahana1)\ **Replies:** 0\ **Last updated:** [March 19, 2020, 6:01am UTC](https://discuss.elastic.co/t/usingwinlogbeat-and-filebeat/224217 "2020-03-19T06:01:05Z")
Hi Syntax to specify the both beats in logstash with different index in the logstash output section. eg :-Filebeat :- index :filebeat winlogbeat :- Index:wineventlog thanks --- ## [How to remove field in logstash output](https://discuss.elastic.co/t/how-to-remove-field-in-logstash-output/224055)
**Author:** [@neal1991](https://discuss.elastic.co/u/neal1991)\ **Replies:** 3\ **Last updated:** [March 19, 2020, 5:12am UTC](https://discuss.elastic.co/t/how-to-remove-field-in-logstash-output/224055 "2020-03-19T05:12:25Z")
I have set up an ELK stack. For the logstash instance, it has two output including Kafka and elasticsearch. For the output of elasticsearch, I want to keep the field @timestamp. For the output of Kafka, I want to remove… --- ## [Extract SNMP table from "SNMP input plugin" with key-value or Split filter](https://discuss.elastic.co/t/extract-snmp-table-from-snmp-input-plugin-with-key-value-or-split-filter/223991)
**Author:** [@piellick](https://discuss.elastic.co/u/piellick)\ **Replies:** 1\ **Last updated:** [March 19, 2020, 3:26am UTC](https://discuss.elastic.co/t/extract-snmp-table-from-snmp-input-plugin-with-key-value-or-split-filter/223991 "2020-03-19T03:26:29Z")
Hello everyone, SNMP input plugin doesn't include (for the moment) an option to split SNMP table : indexed documents looks like this : { "\_index": "XXXX-2020.03.17", "\_type": "doc", "\_id": "Puby6XABeaR9KJCdhk0… --- ## [Logstash on Docker: how to run a pipeline?](https://discuss.elastic.co/t/logstash-on-docker-how-to-run-a-pipeline/224204)
**Author:** [@fdbatista](https://discuss.elastic.co/u/fdbatista)\ **Replies:** 0\ **Last updated:** [March 19, 2020, 12:42am UTC](https://discuss.elastic.co/t/logstash-on-docker-how-to-run-a-pipeline/224204 "2020-03-19T00:42:58Z")
Hello, community. I have started digging around Logstash and I am trying to use docker image for it. The thing is that every tutorial I have seen just shows how to integrate all components within a docker-compose file,… --- ## [Elasticsearch input missing \[@metadata\]\[\_index\]](https://discuss.elastic.co/t/elasticsearch-input-missing-metadata-index/223594)
**Author:** [@joaociocca](https://discuss.elastic.co/u/joaociocca)\ **Replies:** 13\ **Last updated:** [March 18, 2020, 11:13pm UTC](https://discuss.elastic.co/t/elasticsearch-input-missing-metadata-index/223594 "2020-03-18T23:13:29Z")
I'm doing some work with ES-input, ES-filter for lookup and ES-output into a new index... but the lookup uses \[@metadata\]\[\_index\] information. And I've been noticing a HUGE load of errors on LS log for missing that. Any … --- ## [Logstash Output to Apache Flume](https://discuss.elastic.co/t/logstash-output-to-apache-flume/224194)
**Author:** [@vpayala](https://discuss.elastic.co/u/vpayala)\ **Replies:** 0\ **Last updated:** [March 18, 2020, 10:20pm UTC](https://discuss.elastic.co/t/logstash-output-to-apache-flume/224194 "2020-03-18T22:20:41Z")
Hello, Our current logstash filters and sends information to Elasticsearch, but we have a new requirement to ship raw application logs to our partners' Apache Flume endpoint (avro source). Which logstash output plugin … --- ## [Logstash logging behind in consuming data from kinesis stream](https://discuss.elastic.co/t/logstash-logging-behind-in-consuming-data-from-kinesis-stream/224171)
**Author:** [@asrinivasan](https://discuss.elastic.co/u/asrinivasan)\ **Replies:** 0\ **Last updated:** [March 18, 2020, 6:26pm UTC](https://discuss.elastic.co/t/logstash-logging-behind-in-consuming-data-from-kinesis-stream/224171 "2020-03-18T18:26:24Z")
Hi I have a setup with 4 logstash servers , they are configured with kinesis input plugin, I wanted to check if i can have multiple conf's on each logstash for the same kinesis stream. so lets say 2 per logstash so will… --- ## [Parsing json nested in json from log firehose](https://discuss.elastic.co/t/parsing-json-nested-in-json-from-log-firehose/223593)
**Author:** [@Drob](https://discuss.elastic.co/u/Drob)\ **Replies:** 2\ **Last updated:** [March 18, 2020, 6:22pm UTC](https://discuss.elastic.co/t/parsing-json-nested-in-json-from-log-firehose/223593 "2020-03-18T18:22:04Z")
Example Log: \<6\> 2020-03-13T21:20:46Z aea1b92e-07fe-4e21-7684-7f94 doppler\[17\]: {"cf\_app\_id":"174545fd-aee6-4d7c-819c-6c94cdf3b68c", "cf\_app\_name":"cobra-inbound-read-api", "cf\_org\_id":"eb84d2e9-e049-45c3-9a08-a776662b… --- ## [Logstash match add\_field using dynamic field](https://discuss.elastic.co/t/logstash-match-add-field-using-dynamic-field/224150)
**Author:** [@elady](https://discuss.elastic.co/u/elady)\ **Replies:** 0\ **Last updated:** [March 18, 2020, 4:15pm UTC](https://discuss.elastic.co/t/logstash-match-add-field-using-dynamic-field/224150 "2020-03-18T16:15:09Z")
Hey. I am parsing a big log file with many different log line types. What I am trying to do is to both pars parameters and keep a full line in a field called "text". The problem is Logstash fails to create the pipelin… --- ## [What is the best way to write grok pattern for below Log](https://discuss.elastic.co/t/what-is-the-best-way-to-write-grok-pattern-for-below-log/223646)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 8\ **Last updated:** [March 18, 2020, 4:04pm UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-write-grok-pattern-for-below-log/223646 "2020-03-18T16:04:25Z")
Hi Folks, Below is the line that I am trying to write a GROK pattern however later part starts with JSON and my codec =\> json is not matching due to the first part. Any idea or clue how do I match the rules or do I nee… --- ## [Is it okay to have multiple grok filter files?](https://discuss.elastic.co/t/is-it-okay-to-have-multiple-grok-filter-files/224062)
**Author:** [@calanon](https://discuss.elastic.co/u/calanon)\ **Replies:** 1\ **Last updated:** [March 18, 2020, 3:47pm UTC](https://discuss.elastic.co/t/is-it-okay-to-have-multiple-grok-filter-files/224062 "2020-03-18T15:47:47Z")
I wanted to have a clean structure with my grok filters. For example I have php, nginx and a bunch of other filters for some of my application logs. Would it therefore be okay to have for example: 11-nginx.conf 13-ph… --- ## [Configure logstash input from DB values](https://discuss.elastic.co/t/configure-logstash-input-from-db-values/223873)
**Author:** [@Ryan2](https://discuss.elastic.co/u/Ryan2)\ **Replies:** 2\ **Last updated:** [March 18, 2020, 2:46pm UTC](https://discuss.elastic.co/t/configure-logstash-input-from-db-values/223873 "2020-03-18T14:46:23Z")
Hi Team, as I am new to the ELK technology i need some help on this. I've a requirement in which i need to get log location(More than 1 for sure)s from a DB table and pass those values in logstash input to view the data… --- ## [Connect to postgres dB using logstash](https://discuss.elastic.co/t/connect-to-postgres-db-using-logstash/223487)
**Author:** [@Thamini\_NellaiKumar](https://discuss.elastic.co/u/Thamini_NellaiKumar)\ **Replies:** 2\ **Last updated:** [March 18, 2020, 1:30pm UTC](https://discuss.elastic.co/t/connect-to-postgres-db-using-logstash/223487 "2020-03-18T13:30:35Z")
I have a postgres dB in one server and elk in another server. I want to load data from postgres dB to elastic search using logstash. How can I do this? In which server, should the jdbc driver be installed? --- ## [How do I grok this Cisco VPN log?](https://discuss.elastic.co/t/how-do-i-grok-this-cisco-vpn-log/224120)
**Author:** [@francescouk](https://discuss.elastic.co/u/francescouk)\ **Replies:** 0\ **Last updated:** [March 18, 2020, 1:26pm UTC](https://discuss.elastic.co/t/how-do-i-grok-this-cisco-vpn-log/224120 "2020-03-18T13:26:59Z")
For some reason I´m doing something wrong with this grok but I cannot find the problem: \<164\>Mar 18 2020 08:12:55: %ASA-4-722051: Group \ User \ IP \ IPv4 Address \