# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=419 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 420 --- ## [Sqs input not working](https://discuss.elastic.co/t/sqs-input-not-working/215215)
**Author:** [@zozo6015](https://discuss.elastic.co/u/zozo6015)\ **Replies:** 0\ **Last updated:** [January 15, 2020, 9:19pm UTC](https://discuss.elastic.co/t/sqs-input-not-working/215215 "2020-01-15T21:19:42Z")
I am trying to import SES logs from aws using sqs input. I have configured the pipeline. Was working for some time but after moving to 7.x it stopped working. My config looks like this: input { sqs { access\_key\_i… --- ## [Conditional checks no longer work](https://discuss.elastic.co/t/conditional-checks-no-longer-work/215201)
**Author:** [@zozo6015](https://discuss.elastic.co/u/zozo6015)\ **Replies:** 2\ **Last updated:** [January 15, 2020, 8:39pm UTC](https://discuss.elastic.co/t/conditional-checks-no-longer-work/215201 "2020-01-15T20:39:46Z")
Hello, I am running logstash 7.5.0 and I am wondering if anything changed on the conditional checks because I have some checks like: if \[agent\]\[type\] == "filebeat" { ... } and it's not matching anymore. --- ## [Drop documents with empty field](https://discuss.elastic.co/t/drop-documents-with-empty-field/215161)
**Author:** [@JeremyP](https://discuss.elastic.co/u/JeremyP)\ **Replies:** 5\ **Last updated:** [January 15, 2020, 6:07pm UTC](https://discuss.elastic.co/t/drop-documents-with-empty-field/215161 "2020-01-15T18:07:06Z")
I'm having difficulty finding the right syntax to drop a document with an empty field. I'm using aggregate maps. Here is my code.... filter { aggregate { task\_id =\> "%{asset\_id}%{vulnerability\_id}" code =\> … --- ## [Logstash filter date "atypical" failure](https://discuss.elastic.co/t/logstash-filter-date-atypical-failure/215039)
**Author:** [@blenolopes](https://discuss.elastic.co/u/blenolopes)\ **Replies:** 4\ **Last updated:** [January 15, 2020, 5:38pm UTC](https://discuss.elastic.co/t/logstash-filter-date-atypical-failure/215039 "2020-01-15T17:38:41Z")
Hello logstashers, I have an "atypical" problem with a particular log format using date filter. Follow the tests. bin/logstash --version logstash 7.5.0 Success ~$ echo "14/01/2020 16:03:00,143" | bin/logstash … --- ## [Logstash parsing remote files from filebeat](https://discuss.elastic.co/t/logstash-parsing-remote-files-from-filebeat/215181)
**Author:** [@victor\_ide](https://discuss.elastic.co/u/victor_ide)\ **Replies:** 0\ **Last updated:** [January 15, 2020, 5:21pm UTC](https://discuss.elastic.co/t/logstash-parsing-remote-files-from-filebeat/215181 "2020-01-15T17:21:48Z")
Hello logstashers! I was wondering if it is possible to read remote files like PDF, docx and import them into elasticsearch using the attachment processor + filebeat, where filebeat would send files to logstash, then lo… --- ## [Removing leading character and trailing 2 characters](https://discuss.elastic.co/t/removing-leading-character-and-trailing-2-characters/214634)
**Author:** [@DWbank](https://discuss.elastic.co/u/DWbank)\ **Replies:** 14\ **Last updated:** [January 15, 2020, 3:58pm UTC](https://discuss.elastic.co/t/removing-leading-character-and-trailing-2-characters/214634 "2020-01-15T15:58:40Z")
I am using kv to parse my fields due to the fields change depends on the logs that are sent from the siem. kv { value\_split =\> “=’” field\_split =\> “’ “… --- ## [Log Management for Multiple Application Instances](https://discuss.elastic.co/t/log-management-for-multiple-application-instances/214940)
**Author:** [@amin224](https://discuss.elastic.co/u/amin224)\ **Replies:** 2\ **Last updated:** [January 15, 2020, 3:44pm UTC](https://discuss.elastic.co/t/log-management-for-multiple-application-instances/214940 "2020-01-15T15:44:45Z")
Hello, sorry if this has been asked before but i am new to ELK stack and did not find an answer for my question. If i have an application that is running multiple instances for example instance1, instance2, instance3 and… --- ## [S3 dynamic date prefix in folder name by using current date as backup\_add\_prefix](https://discuss.elastic.co/t/s3-dynamic-date-prefix-in-folder-name-by-using-current-date-as-backup-add-prefix/215104)
**Author:** [@Xandr90](https://discuss.elastic.co/u/Xandr90)\ **Replies:** 1\ **Last updated:** [January 15, 2020, 3:38pm UTC](https://discuss.elastic.co/t/s3-dynamic-date-prefix-in-folder-name-by-using-current-date-as-backup-add-prefix/215104 "2020-01-15T15:38:53Z")
So what i want to do is use Logstash to rename incoming files in a s3 bucket. I run it via docker and use the following logstash.conf input { s3 { access\_key\_id =\> "some\_key" secret\_access\_key =\> "some\_secret" reg… --- ## [Using date filter within split filter](https://discuss.elastic.co/t/using-date-filter-within-split-filter/214402)
**Author:** [@maksimize](https://discuss.elastic.co/u/maksimize)\ **Replies:** 4\ **Last updated:** [January 15, 2020, 2:47pm UTC](https://discuss.elastic.co/t/using-date-filter-within-split-filter/214402 "2020-01-15T14:47:59Z")
Hi All I'm receiving batch event from azure eventhub and I am using split filter to split each event into it's own record the problem I'm facing is that each split event has it's own datetime and I would like to use th… --- ## [Json::ParserError: Unrecognized token](https://discuss.elastic.co/t/json-unrecognized-token/215006)
**Author:** [@akuninja](https://discuss.elastic.co/u/akuninja)\ **Replies:** 6\ **Last updated:** [January 15, 2020, 2:37pm UTC](https://discuss.elastic.co/t/json-unrecognized-token/215006 "2020-01-15T14:37:18Z")
//////////////////////////////////THIS PART I SOLVED MYSELF /////////////////////////////////////// I am still getting a broken token within my logstash output, and I do not know where this timestamp (possibly) token co… --- ## [Logstash grok filter to generate priority number in logs](https://discuss.elastic.co/t/logstash-grok-filter-to-generate-priority-number-in-logs/215114)
**Author:** [@BIs1](https://discuss.elastic.co/u/BIs1)\ **Replies:** 0\ **Last updated:** [January 15, 2020, 10:44am UTC](https://discuss.elastic.co/t/logstash-grok-filter-to-generate-priority-number-in-logs/215114 "2020-01-15T10:44:35Z")
Hi Everyone, I am very new to Elastic Stack so I have very less knowledge regarding Logstash grok filter. Basically I am trying to get the logs with priority number so that syslog\_pri {} plugin will generate all the nec… --- ## [Drop line rather than the whole message](https://discuss.elastic.co/t/drop-line-rather-than-the-whole-message/215133)
**Author:** [@whoatemyjam](https://discuss.elastic.co/u/whoatemyjam)\ **Replies:** 6\ **Last updated:** [January 15, 2020, 1:55pm UTC](https://discuss.elastic.co/t/drop-line-rather-than-the-whole-message/215133 "2020-01-15T13:55:14Z")
hi I am trying to drop lines which match a regex filter { if \[message\] =~ "\\e\\\[8mha.\*\\e\\\[0m" { drop{} } } this above one removes the whole message , how do i drop only the line that contains the matching regex Than… --- ## [Jenkins logs via logstash](https://discuss.elastic.co/t/jenkins-logs-via-logstash/215122)
**Author:** [@whoatemyjam](https://discuss.elastic.co/u/whoatemyjam)\ **Replies:** 3\ **Last updated:** [January 15, 2020, 1:52pm UTC](https://discuss.elastic.co/t/jenkins-logs-via-logstash/215122 "2020-01-15T13:52:46Z")
Hi we are trying to send Jenkins build logs logs to ES via logstash and filebeat. We have tried Jenkins logstash plugin but it doesnot work with pipelines. Our filebeat config is like filebeat.inputs: - type: log e… --- ## [If statement not working with gsub](https://discuss.elastic.co/t/if-statement-not-working-with-gsub/215030)
**Author:** [@nino](https://discuss.elastic.co/u/nino)\ **Replies:** 4\ **Last updated:** [January 15, 2020, 12:23pm UTC](https://discuss.elastic.co/t/if-statement-not-working-with-gsub/215030 "2020-01-15T12:23:11Z")
Hello all, i want to remove "ZC" in a field and converted in a negative float "account\_value" =\> "767.19ZC", "@timestamp" =\> 2020-01-06T23:00:00.000Z why this statement is not working? filter{ if "ZC" in "fi… --- ## [Issues sending monitoring data from logstash to elasticsearch](https://discuss.elastic.co/t/issues-sending-monitoring-data-from-logstash-to-elasticsearch/215103)
**Author:** [@asp](https://discuss.elastic.co/u/asp)\ **Replies:** 7\ **Last updated:** [January 15, 2020, 11:43am UTC](https://discuss.elastic.co/t/issues-sending-monitoring-data-from-logstash-to-elasticsearch/215103 "2020-01-15T11:43:25Z")
Hi, I have some issues shipping internal monitoring data from logstash to elasticsearch. I don't know if i better place it in logstash or elasticsearch forum. So I start here :wink: I installed logstash and elasticsearc… --- ## [Adding fields to configuration file](https://discuss.elastic.co/t/adding-fields-to-configuration-file/215110)
**Author:** [@amin224](https://discuss.elastic.co/u/amin224)\ **Replies:** 7\ **Last updated:** [January 15, 2020, 11:30am UTC](https://discuss.elastic.co/t/adding-fields-to-configuration-file/215110 "2020-01-15T11:30:28Z")
Hi, I am running multiple java applications and creating logging files for each one, so i decided to use elastic stack for Centralised Log Management. My question is how can i add port and IP address fields to my index … --- ## [How to edit data coming in from kafka within logstash and ingest into elasticsearch](https://discuss.elastic.co/t/how-to-edit-data-coming-in-from-kafka-within-logstash-and-ingest-into-elasticsearch/215123)
**Author:** [@fallicious](https://discuss.elastic.co/u/fallicious)\ **Replies:** 0\ **Last updated:** [January 15, 2020, 11:23am UTC](https://discuss.elastic.co/t/how-to-edit-data-coming-in-from-kafka-within-logstash-and-ingest-into-elasticsearch/215123 "2020-01-15T11:23:06Z")
i have two indexes coming in elasticsearch that are being fed through kafka, (indexA) and (indexB), i want to take some fields from the (indexB) and a few from (indexA) and make a third (indexC) of of those fields. i ha… --- ## [How to get certifiates checked on elasticsearch-output-plugin?](https://discuss.elastic.co/t/how-to-get-certifiates-checked-on-elasticsearch-output-plugin/215100)
**Author:** [@asp](https://discuss.elastic.co/u/asp)\ **Replies:** 0\ **Last updated:** [January 15, 2020, 9:08am UTC](https://discuss.elastic.co/t/how-to-get-certifiates-checked-on-elasticsearch-output-plugin/215100 "2020-01-15T09:08:35Z")
Hi, I installed logstash on centos via rpm and installed via system-install as service. When starting logstash I get the following warning: \*\* WARNING \*\* You have enabled encryption but DISABLED certificate verificati… --- ## [Pipeline Error](https://discuss.elastic.co/t/pipeline-error/214973)
**Author:** [@jogoinar10](https://discuss.elastic.co/u/jogoinar10)\ **Replies:** 2\ **Last updated:** [January 15, 2020, 8:22am UTC](https://discuss.elastic.co/t/pipeline-error/214973 "2020-01-15T08:22:13Z")
\[2020-01-14T18:39:45,304\]\[INFO \]\[org.logstash.beats.BeatsHandler\]\[main\] \[local: 192.168.0.79:5044, remote: 192.168.0.97:55045\] Handling exception: Connection reset \[2020-01-14T18:39:45,314\]\[WARN \]\[io.netty.channel.Defaul… --- ## [Logstash SSL Error with http and https](https://discuss.elastic.co/t/logstash-ssl-error-with-http-and-https/214759)
**Author:** [@DawitCh](https://discuss.elastic.co/u/DawitCh)\ **Replies:** 2\ **Last updated:** [January 15, 2020, 8:08am UTC](https://discuss.elastic.co/t/logstash-ssl-error-with-http-and-https/214759 "2020-01-15T08:08:37Z")
\[ 2018-09-18T11:01:23,207\]\[ERROR\]\[logstash.licensechecker.licensemanager\] Unable to retrieve license information from license server {:message=\>"Explicit value for 'scheme' was declared, but it is different in one of the… --- ## [Problem Sorting Unique Values in CSV](https://discuss.elastic.co/t/problem-sorting-unique-values-in-csv/215057)
**Author:** [@niall](https://discuss.elastic.co/u/niall)\ **Replies:** 1\ **Last updated:** [January 14, 2020, 11:10pm UTC](https://discuss.elastic.co/t/problem-sorting-unique-values-in-csv/215057 "2020-01-14T23:10:28Z")
Hi, I've been trying to solve this issue and came to a dead end. I will need some advice for this. I do have a CSV file. Below is just sample data, but the file has like up to thousand lines. Below is the sample file: … --- ## [Jdbc problem when tracking column is fixed decimal](https://discuss.elastic.co/t/jdbc-problem-when-tracking-column-is-fixed-decimal/215051)
**Author:** [@Peter\_Li](https://discuss.elastic.co/u/Peter_Li)\ **Replies:** 0\ **Last updated:** [January 14, 2020, 9:38pm UTC](https://discuss.elastic.co/t/jdbc-problem-when-tracking-column-is-fixed-decimal/215051 "2020-01-14T21:38:53Z")
I have a table where the tracking value has to be constructed from two integers: batch and index. So, I created a fixed decimal "bistamp" like 123.0034, 124.0003, etc. using: select ... to\_number(to\_char ( batch … --- ## [Plotting geoIP Data in Kibana Maps via Beats Output to Logstash](https://discuss.elastic.co/t/plotting-geoip-data-in-kibana-maps-via-beats-output-to-logstash/214673)
**Author:** [@Log\_Gobbler](https://discuss.elastic.co/u/Log_Gobbler)\ **Replies:** 3\ **Last updated:** [January 14, 2020, 8:12pm UTC](https://discuss.elastic.co/t/plotting-geoip-data-in-kibana-maps-via-beats-output-to-logstash/214673 "2020-01-14T20:12:01Z")
We're running on Elastic Stack 7.5 and have all of our Beats output being sent to Logstash. Then from Logstash, output is sent to Elasticsearch and Kibana. Using this flow, we have been unsuccessful in implementing geoI… --- ## [How to make CSV output filter close file (not append) when pipeline runs?](https://discuss.elastic.co/t/how-to-make-csv-output-filter-close-file-not-append-when-pipeline-runs/215031)
**Author:** [@tomj](https://discuss.elastic.co/u/tomj)\ **Replies:** 0\ **Last updated:** [January 14, 2020, 6:31pm UTC](https://discuss.elastic.co/t/how-to-make-csv-output-filter-close-file-not-append-when-pipeline-runs/215031 "2020-01-14T18:31:00Z")
I have a pipeline set up that I wish to "refresh" a CSV file (to be used later in a "translate" step) from a database. What I'm seeing is that every time this pipeline runs, the entire CSV output gets appended to the na… --- ## [Can't install ruby gem into logstash environment](https://discuss.elastic.co/t/cant-install-ruby-gem-into-logstash-environment/214926)
**Author:** [@Brian\_Seel](https://discuss.elastic.co/u/Brian_Seel)\ **Replies:** 3\ **Last updated:** [January 14, 2020, 5:57pm UTC](https://discuss.elastic.co/t/cant-install-ruby-gem-into-logstash-environment/214926 "2020-01-14T17:57:36Z")
I need to install a gem into the ruby instance that logstash uses. Specifically, I need to install jdbc-sqlite3 to support database calls that can't be done as an input plugin. I have tried doing it remote or local, wit… --- ## [Logstash won't show output on console](https://discuss.elastic.co/t/logstash-wont-show-output-on-console/214941)
**Author:** [@Anindita\_Chavan](https://discuss.elastic.co/u/Anindita_Chavan)\ **Replies:** 8\ **Last updated:** [January 14, 2020, 5:40pm UTC](https://discuss.elastic.co/t/logstash-wont-show-output-on-console/214941 "2020-01-14T17:40:34Z")
This is my logstash configuration file: \< input { file { path =\> "/home/trimslogs/trims.log" sincedb\_path =\> "NUL" ignore\_older =\> 0 } } filter { grok { match =\> { "message" =\> "(\[(?%{YEAR}-%{MONTHNUM}-%{MONTH… --- ## [How to monitor individual pipelines for memory usage](https://discuss.elastic.co/t/how-to-monitor-individual-pipelines-for-memory-usage/215024)
**Author:** [@analog\_memories](https://discuss.elastic.co/u/analog_memories)\ **Replies:** 0\ **Last updated:** [January 14, 2020, 5:21pm UTC](https://discuss.elastic.co/t/how-to-monitor-individual-pipelines-for-memory-usage/215024 "2020-01-14T17:21:15Z")
Hello, Logstash 7.4.2 RHEL 7 16 Pipeline workers 8GB JVM heap, 16GB total system memory I have Logstash monitoring enabled and send data to my cluster, however, I need more in depth view into the memory usage for ea… --- ## [WID Extractions with Logstash](https://discuss.elastic.co/t/wid-extractions-with-logstash/215011)
**Author:** [@John\_Guzman](https://discuss.elastic.co/u/John_Guzman)\ **Replies:** 0\ **Last updated:** [January 14, 2020, 3:57pm UTC](https://discuss.elastic.co/t/wid-extractions-with-logstash/215011 "2020-01-14T15:57:34Z")
Hello! I need to extract some data from a Windows internal Database from WSUS. I already tried using the JDBC and Exec input plugins, buy no one worked using this string connection \\.\\pipe\\MICROSOFT##WID\\tsql\\query. Th… --- ## [Azure module setup with elastic https security](https://discuss.elastic.co/t/azure-module-setup-with-elastic-https-security/215009)
**Author:** [@rugenl](https://discuss.elastic.co/u/rugenl)\ **Replies:** 0\ **Last updated:** [January 14, 2020, 3:54pm UTC](https://discuss.elastic.co/t/azure-module-setup-with-elastic-https-security/215009 "2020-01-14T15:54:37Z")
I'm trying to run setup for the logstash (7.5.0) azure module. All examples are without security but we are using elasticsearch https. I'm getting errors: \[2020-01-14T09:42:45,431\]\[ERROR\]\[logstash.licensechecker.lice… --- ## [Disable/enable multiline codec when using clone filter](https://discuss.elastic.co/t/disable-enable-multiline-codec-when-using-clone-filter/214942)
**Author:** [@nino](https://discuss.elastic.co/u/nino)\ **Replies:** 4\ **Last updated:** [January 14, 2020, 3:45pm UTC](https://discuss.elastic.co/t/disable-enable-multiline-codec-when-using-clone-filter/214942 "2020-01-14T15:45:46Z")
Hello all, is it posible to disable multiline when using clone filter?. For instance: input{ codec =\> multiline { pattern =\> "^ImposibleMatch" negate =\> true what =\> previous … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=418) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=420)