# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=503 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 504 --- ## [How to save 'sql\_last\_value' in between reading pages in jdbc-input plugin?](https://discuss.elastic.co/t/how-to-save-sql-last-value-in-between-reading-pages-in-jdbc-input-plugin/192558)
**Author:** [@mahmoud1468](https://discuss.elastic.co/u/mahmoud1468)\ **Replies:** 0\ **Last updated:** [July 28, 2019, 10:19am UTC](https://discuss.elastic.co/t/how-to-save-sql-last-value-in-between-reading-pages-in-jdbc-input-plugin/192558 "2019-07-28T10:19:33Z")
Hi, I have enabled paging ("jdbc\_paging\_enabled" =\> true) in "JDBC-INPUT" plugin in a Logstash pipeline. I would like to save 'sql\_last\_value' at the end of each page instead of the whole query. Is there any way to do… --- ## [Logstash query multiple documents from time series indices and output to a single nested elasticsearch document](https://discuss.elastic.co/t/logstash-query-multiple-documents-from-time-series-indices-and-output-to-a-single-nested-elasticsearch-document/192543)
**Author:** [@jdcohen220](https://discuss.elastic.co/u/jdcohen220)\ **Replies:** 0\ **Last updated:** [July 27, 2019, 11:25pm UTC](https://discuss.elastic.co/t/logstash-query-multiple-documents-from-time-series-indices-and-output-to-a-single-nested-elasticsearch-document/192543 "2019-07-27T23:25:44Z")
Hi, New to logstash and wondering if it is possible for the following use case. Doing some searching it is not entirely clear if it is possible to query elasticsearch in filter stage for multiple documents and nest into… --- ## ["index.lifecycle.rollover\_alias does not point to index](https://discuss.elastic.co/t/index-lifecycle-rollover-alias-does-not-point-to-index/192525)
**Author:** [@mhsh64](https://discuss.elastic.co/u/mhsh64)\ **Replies:** 5\ **Last updated:** [July 27, 2019, 5:22pm UTC](https://discuss.elastic.co/t/index-lifecycle-rollover-alias-does-not-point-to-index/192525 "2019-07-27T17:22:18Z")
Please help: logstash output: I have a index template for logstash, which i defined in logstash output: output { elasticsearch { hosts =\> \["https://elasticsearchurl.domain.com:9200"\] user =\> 'logst… --- ## [Custom patterns can´t get resolved in logstash](https://discuss.elastic.co/t/custom-patterns-can-t-get-resolved-in-logstash/192262)
**Author:** [@CyberSeppi](https://discuss.elastic.co/u/CyberSeppi)\ **Replies:** 2\ **Last updated:** [July 27, 2019, 2:48pm UTC](https://discuss.elastic.co/t/custom-patterns-can-t-get-resolved-in-logstash/192262 "2019-07-27T14:48:55Z")
I´m trying to setup an environment for grok debugging and made this with a docker. Everything works fine, until logstash tries to resolve a custom pattern. Here is my environment I start the docker with docker run -i… --- ## [Can i define S3 output file content?](https://discuss.elastic.co/t/can-i-define-s3-output-file-content/192351)
**Author:** [@brandonbai](https://discuss.elastic.co/u/brandonbai)\ **Replies:** 2\ **Last updated:** [July 27, 2019, 1:38am UTC](https://discuss.elastic.co/t/can-i-define-s3-output-file-content/192351 "2019-07-27T01:38:17Z")
Hi, When I use the following code to remove the message field, filter { .... if "\_grokparsefailure" not in \[tags\] { mutate { remove\_field =\> \[ "message" \] } } .... } the log content output to s3 beco… --- ## [Nested fields as the grok match input](https://discuss.elastic.co/t/nested-fields-as-the-grok-match-input/192502)
**Author:** [@jbrowe](https://discuss.elastic.co/u/jbrowe)\ **Replies:** 3\ **Last updated:** [July 26, 2019, 11:28pm UTC](https://discuss.elastic.co/t/nested-fields-as-the-grok-match-input/192502 "2019-07-26T23:28:21Z")
I am trying to parse ADFS authentication logs from a winlogbeat data stream. My winlogbeat pre-parses these logs by putting everything unrecognized into fields event\_data.param1, event\_data.param2, etc, which are not mea… --- ## [Push\_map\_as\_event\_on\_timeout undocumented side effects (Logstash version 6.8.1)](https://discuss.elastic.co/t/push-map-as-event-on-timeout-undocumented-side-effects-logstash-version-6-8-1/192330)
**Author:** [@ktbd](https://discuss.elastic.co/u/ktbd)\ **Replies:** 6\ **Last updated:** [July 26, 2019, 10:09pm UTC](https://discuss.elastic.co/t/push-map-as-event-on-timeout-undocumented-side-effects-logstash-version-6-8-1/192330 "2019-07-26T22:09:54Z")
The earlier part of my config contains a lot of regex which breaks up a message from /var/log/php7.0-fpm.log into a number of parts and assigns \[@metatadata\]\[php\_variety\] . Additional processing is then performed based o… --- ## [Logstash exec input plugin empty message](https://discuss.elastic.co/t/logstash-exec-input-plugin-empty-message/192316)
**Author:** [@mtudisco](https://discuss.elastic.co/u/mtudisco)\ **Replies:** 3\ **Last updated:** [July 26, 2019, 8:38pm UTC](https://discuss.elastic.co/t/logstash-exec-input-plugin-empty-message/192316 "2019-07-26T20:38:53Z")
Hi, i'm trying to use logstash exec input plugin to run a shell script, however i'm not getting any result. i have a sample shell script /scripts/myscript.sh the contents is just ls if i start logstash with the foll… --- ## [Elasticsearch filter plugin configuration not working](https://discuss.elastic.co/t/elasticsearch-filter-plugin-configuration-not-working/192497)
**Author:** [@mtudisco](https://discuss.elastic.co/u/mtudisco)\ **Replies:** 0\ **Last updated:** [July 26, 2019, 7:59pm UTC](https://discuss.elastic.co/t/elasticsearch-filter-plugin-configuration-not-working/192497 "2019-07-26T19:59:14Z")
I'm new to elasticsearch and i'm trying to get the last known state from a database to compare it with the actual status, however something is not working input { exec { command =\> "sh -c '/usr/local/bin/chkdb.sh'" i… --- ## [How to parse Apache log : Logstash plugin or Filebeat module?](https://discuss.elastic.co/t/how-to-parse-apache-log-logstash-plugin-or-filebeat-module/191212)
**Author:** [@Paul\_ALTEN](https://discuss.elastic.co/u/Paul_ALTEN)\ **Replies:** 2\ **Last updated:** [July 26, 2019, 6:06pm UTC](https://discuss.elastic.co/t/how-to-parse-apache-log-logstash-plugin-or-filebeat-module/191212 "2019-07-26T18:06:09Z")
I'm in troubles using ELK stack for analysing my Apache's logs. Is it better to parse them with Filebeat using the Apache's module directly shipping to Elasticsearch ? Or it's better to use grok plugin from Logstash fo… --- ## [Logstash is not getting started](https://discuss.elastic.co/t/logstash-is-not-getting-started/192484)
**Author:** [@Pradyumna\_Jha](https://discuss.elastic.co/u/Pradyumna_Jha)\ **Replies:** 2\ **Last updated:** [July 26, 2019, 5:35pm UTC](https://discuss.elastic.co/t/logstash-is-not-getting-started/192484 "2019-07-26T17:35:24Z")
When we are starting logstash it is perpetually stuck at \[2019-07-26T22:23:21,607\]\[INFO \]\[logstash.runner \] Starting Logstash {"logstash.version"=\>"7.2.0"} Requesting all to please reply and help ASAP. --- ## [ILM setup: Logstash vs elasticsearch](https://discuss.elastic.co/t/ilm-setup-logstash-vs-elasticsearch/192471)
**Author:** [@fefo69](https://discuss.elastic.co/u/fefo69)\ **Replies:** 3\ **Last updated:** [July 26, 2019, 5:05pm UTC](https://discuss.elastic.co/t/ilm-setup-logstash-vs-elasticsearch/192471 "2019-07-26T17:05:57Z")
Dear all, I'm trying to create indexes from logstash output plugin but policy assignment and rollover is not working. Elasticsearch and logstash version 7.2 This is my test setup: Logstash output plugin: elasticsearc… --- ## [Index my Linkedin or facebook page](https://discuss.elastic.co/t/index-my-linkedin-or-facebook-page/192275)
**Author:** [@manideep](https://discuss.elastic.co/u/manideep)\ **Replies:** 1\ **Last updated:** [July 26, 2019, 4:40pm UTC](https://discuss.elastic.co/t/index-my-linkedin-or-facebook-page/192275 "2019-07-26T16:40:51Z")
Hi Everyone, I am trying to index my Linkedin content in Elasticsearch. I have gone through filebeat, logstash and elasticsearch concepts and wanted to put all together using this simple example. what is the best app… --- ## [Offline plugin install of Java plugin](https://discuss.elastic.co/t/offline-plugin-install-of-java-plugin/192439)
**Author:** [@Cory\_Huebner](https://discuss.elastic.co/u/Cory_Huebner)\ **Replies:** 0\ **Last updated:** [July 26, 2019, 12:42pm UTC](https://discuss.elastic.co/t/offline-plugin-install-of-java-plugin/192439 "2019-07-26T12:42:33Z")
I am attempting to install a custom output Logstash plugin on Logstash 7.2; the plugin is written in Java based on the instructions found on this Logstash documentation page. I am attempting to deploy into a Logstash do… --- ## [Persistent queue not fully emptied after ES output unavailable](https://discuss.elastic.co/t/persistent-queue-not-fully-emptied-after-es-output-unavailable/192445)
**Author:** [@rbayet](https://discuss.elastic.co/u/rbayet)\ **Replies:** 0\ **Last updated:** [July 26, 2019, 1:06pm UTC](https://discuss.elastic.co/t/persistent-queue-not-fully-emptied-after-es-output-unavailable/192445 "2019-07-26T13:06:08Z")
Hi, I have a setup with a JMS input which subs to a durable topic on a message broker. Events contain log data in JSON format, so in my filters, I have a json, split and a bit of ruby code. The single output is ES. A… --- ## [Logstash.outputs.elasticsearch\] UNEXPECTED POOL ERROR {:e=\>#\}](https://discuss.elastic.co/t/logstash-outputs-elasticsearch-unexpected-pool-error-e-logstash-no-available-connections/192364)
**Author:** [@divyadaka](https://discuss.elastic.co/u/divyadaka)\ **Replies:** 7\ **Last updated:** [July 26, 2019, 12:38pm UTC](https://discuss.elastic.co/t/logstash-outputs-elasticsearch-unexpected-pool-error-e-logstash-no-available-connections/192364 "2019-07-26T12:38:08Z")
I am new to ELK stack and also to td-agent. purpose of this installation is to store application logs...i had installed elasticsearch(9200),kibana(5601),logastash (5044)and td-agent(8888).... But elasticsearch ,kibana i… --- ## [Syslog message is incomplete](https://discuss.elastic.co/t/syslog-message-is-incomplete/192384)
**Author:** [@mondei1](https://discuss.elastic.co/u/mondei1)\ **Replies:** 10\ **Last updated:** [July 26, 2019, 12:32pm UTC](https://discuss.elastic.co/t/syslog-message-is-incomplete/192384 "2019-07-26T12:32:38Z")
Hey community, I have to setup a ELK Stack and everything runs perfectly until I had to implement syslogs. My problem here is that the syslog I receive in Logstash differs from the syslog that I receive with rsyslog (lo… --- ## [\[Logstash\]:ConfigurationError ":message=\>"Expected one of #, input, filter, output "](https://discuss.elastic.co/t/logstash-configurationerror-message-expected-one-of-input-filter-output/192045)
**Author:** [@Moustapha\_Berrou](https://discuss.elastic.co/u/Moustapha_Berrou)\ **Replies:** 5\ **Last updated:** [July 26, 2019, 9:55am UTC](https://discuss.elastic.co/t/logstash-configurationerror-message-expected-one-of-input-filter-output/192045 "2019-07-26T09:55:29Z")
Dear all, i am uploading a json file with Logstash 7.2.0 in to Elasticsearch =\> Grafana (All with Docker) input{ file{ codec =\> multiline { pattern =\> '^{' negate =\> true what =\> previous }… --- ## [New conf not working even when I specify the config file path](https://discuss.elastic.co/t/new-conf-not-working-even-when-i-specify-the-config-file-path/192392)
**Author:** [@yan\_darcy](https://discuss.elastic.co/u/yan_darcy)\ **Replies:** 0\ **Last updated:** [July 26, 2019, 8:52am UTC](https://discuss.elastic.co/t/new-conf-not-working-even-when-i-specify-the-config-file-path/192392 "2019-07-26T08:52:07Z")
Hi, I am trying to using Logstash to import data from MySQL to Elasticsearch. I make a test on my config file, but the output is not from my configure file. My test command: current directory: /home/pwangbo/data/elasti… --- ## [How to parse json fields from a log which have multiple event but without delimiter](https://discuss.elastic.co/t/how-to-parse-json-fields-from-a-log-which-have-multiple-event-but-without-delimiter/191858)
**Author:** [@uma\_rengasamy](https://discuss.elastic.co/u/uma_rengasamy)\ **Replies:** 7\ **Last updated:** [July 26, 2019, 7:36am UTC](https://discuss.elastic.co/t/how-to-parse-json-fields-from-a-log-which-have-multiple-event-but-without-delimiter/191858 "2019-07-26T07:36:59Z")
How to parse json fields from a log which have multiple event but without delimiter? The logs are cloudwatch logs streamed to kinesis firehose and stored in s3 which is streamed to logstash --- ## [Logstash java codec running error](https://discuss.elastic.co/t/logstash-java-codec-running-error/190128)
**Author:** [@zhi\_asher](https://discuss.elastic.co/u/zhi_asher)\ **Replies:** 10\ **Last updated:** [July 26, 2019, 3:06am UTC](https://discuss.elastic.co/t/logstash-java-codec-running-error/190128 "2019-07-26T03:06:28Z")
HI，@danhermann，I'm from github issue (https://github.com/elastic/logstash/issues/10680), the promble transf here logstash compile version: 7.1.2, logstash running version: 7.1.1 java\_codec\_example.rb file content # … --- ## [Logstash - ES to BigQuery - doesn't load all rows into BigQuery](https://discuss.elastic.co/t/logstash-es-to-bigquery-doesnt-load-all-rows-into-bigquery/190114)
**Author:** [@srikrishb](https://discuss.elastic.co/u/srikrishb)\ **Replies:** 1\ **Last updated:** [July 26, 2019, 2:19am UTC](https://discuss.elastic.co/t/logstash-es-to-bigquery-doesnt-load-all-rows-into-bigquery/190114 "2019-07-26T02:19:39Z")
Hello, I am using logstash 7.1.1 to fetch data from elasticsearch cluster to BigQuery database. My logstash instance ran successfully but ran into some problems: Logstash config file: input { elasticsearch { hosts =\>… --- ## [Ruby Shenanigans](https://discuss.elastic.co/t/ruby-shenanigans/192341)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 0\ **Last updated:** [July 26, 2019, 1:06am UTC](https://discuss.elastic.co/t/ruby-shenanigans/192341 "2019-07-26T01:06:51Z")
I have this pipeline that's a bit convoluted and wanted to know if this will work. The intent is, if old.new\_duration exists and, for example, has a value of 10, instead of it being replaced, it gets added to the value … --- ## [Logstash with input from file and input from beats (port 5044) create two indices with the same data](https://discuss.elastic.co/t/logstash-with-input-from-file-and-input-from-beats-port-5044-create-two-indices-with-the-same-data/191348)
**Author:** [@izhael](https://discuss.elastic.co/u/izhael)\ **Replies:** 2\ **Last updated:** [July 25, 2019, 9:38pm UTC](https://discuss.elastic.co/t/logstash-with-input-from-file-and-input-from-beats-port-5044-create-two-indices-with-the-same-data/191348 "2019-07-25T21:38:26Z")
Good morning, I have two configuration files in Logstash conf directory: First file: File: firewall.conf \*\* Logs from Firewall \*\* input { file { path =\> "/var/log/remote/firewall/\*-firewall.log" ignore\_older =\> 60 … --- ## [Error after upgrading to Logstash 7.1.1](https://discuss.elastic.co/t/error-after-upgrading-to-logstash-7-1-1/192147)
**Author:** [@Reena\_Deberry](https://discuss.elastic.co/u/Reena_Deberry)\ **Replies:** 5\ **Last updated:** [July 25, 2019, 9:01pm UTC](https://discuss.elastic.co/t/error-after-upgrading-to-logstash-7-1-1/192147 "2019-07-25T21:01:22Z")
\[ERROR\]\[logstash.agent \] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Something is wrong with your configurati… --- ## [Logstash Grok New Line Syntax](https://discuss.elastic.co/t/logstash-grok-new-line-syntax/192148)
**Author:** [@userR](https://discuss.elastic.co/u/userR)\ **Replies:** 11\ **Last updated:** [July 25, 2019, 9:11pm UTC](https://discuss.elastic.co/t/logstash-grok-new-line-syntax/192148 "2019-07-25T21:11:29Z")
Hi, I am attempting to parse log data that are on different lines. When its read through filebeat and sent to logstash, the message contains \\n. Unfortunetly, I am unable to deal with this and am getting an error when I… --- ## [Match specific word in grok filter](https://discuss.elastic.co/t/match-specific-word-in-grok-filter/192299)
**Author:** [@naveenrt23](https://discuss.elastic.co/u/naveenrt23)\ **Replies:** 1\ **Last updated:** [July 25, 2019, 7:43pm UTC](https://discuss.elastic.co/t/match-specific-word-in-grok-filter/192299 "2019-07-25T19:43:37Z")
I have a log message which is composed of multiple values joined by pipes. 20190615|4|method|userend|/test/value/123/1.1|500|2 mY grok filter: mutate { add\_field =\> { "\[@metadata\]\[copyOfMessage\]" =\>… --- ## [Weird field/output issues](https://discuss.elastic.co/t/weird-field-output-issues/192286)
**Author:** [@eclipsed450](https://discuss.elastic.co/u/eclipsed450)\ **Replies:** 2\ **Last updated:** [July 25, 2019, 5:55pm UTC](https://discuss.elastic.co/t/weird-field-output-issues/192286 "2019-07-25T17:55:02Z")
Hi, I'm having some weird issues. Below is my config. I'm finding that if I leave the if \[type\] == "s3\_emr", then the output will not output anything; however if I comment it out, the gelf output will kick in and start … --- ## [Logstash http input Certificate Information in output](https://discuss.elastic.co/t/logstash-http-input-certificate-information-in-output/192288)
**Author:** [@Jonathan\_Samples](https://discuss.elastic.co/u/Jonathan_Samples)\ **Replies:** 0\ **Last updated:** [July 25, 2019, 4:41pm UTC](https://discuss.elastic.co/t/logstash-http-input-certificate-information-in-output/192288 "2019-07-25T16:41:00Z")
I am using 2-way SSL with an http input. I have a use case where I need to pull the CN out of the client certificate and append it to the output. Is there any way to do this? --- ## [SNMP Input Plugin Integer Overflow](https://discuss.elastic.co/t/snmp-input-plugin-integer-overflow/191649)
**Author:** [@pprugger](https://discuss.elastic.co/u/pprugger)\ **Replies:** 11\ **Last updated:** [July 25, 2019, 4:30pm UTC](https://discuss.elastic.co/t/snmp-input-plugin-integer-overflow/191649 "2019-07-25T16:30:28Z")
Hello everyone! We are gathering our firewall data from our cisco ASA 5545 via snmp v3. Now when we query ciscoMemoryPoolFree somewhere an overflow occurs. Return value: CISCO-MEMORY-POOL-MIB::ciscoMemoryPoolFree.1 =… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=502) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=504)