# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=242 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 243 --- ## [I want to insert elasticsearch with multi time with range](https://discuss.elastic.co/t/i-want-to-insert-elasticsearch-with-multi-time-with-range/269088)
**Author:** [@gmcom2011](https://discuss.elastic.co/u/gmcom2011)\ **Replies:** 0\ **Last updated:** [April 2, 2021, 7:05am UTC](https://discuss.elastic.co/t/i-want-to-insert-elasticsearch-with-multi-time-with-range/269088 "2021-04-02T07:05:52Z")
If I have database table 100,000++ record and I want to insert to elasticsearch multi time. How to use logstash insert to elasticserch 1,000 records per request? example round 1 input record 1 - 1000 round 2 input … --- ## [Is the multiline codec supposed to work with the JDBC input plugin?](https://discuss.elastic.co/t/is-the-multiline-codec-supposed-to-work-with-the-jdbc-input-plugin/268852)
**Author:** [@ehanft](https://discuss.elastic.co/u/ehanft)\ **Replies:** 5\ **Last updated:** [April 1, 2021, 10:06pm UTC](https://discuss.elastic.co/t/is-the-multiline-codec-supposed-to-work-with-the-jdbc-input-plugin/268852 "2021-04-01T22:06:08Z")
I been having a hard time processing log files stored in an Oracle database table. The logs are stored as CLOB, but converted to VARCHAR2 using a pipelined table function. This works great with the JDBC input plugin in L… --- ## [Filter elements based on array constant or array environment variable](https://discuss.elastic.co/t/filter-elements-based-on-array-constant-or-array-environment-variable/269074)
**Author:** [@Automation\_Scripts](https://discuss.elastic.co/u/Automation_Scripts)\ **Replies:** 1\ **Last updated:** [April 1, 2021, 7:41pm UTC](https://discuss.elastic.co/t/filter-elements-based-on-array-constant-or-array-environment-variable/269074 "2021-04-01T19:41:53Z")
Hi guys, How can I use an array constant in Logstash's filter? In my .env file I have a constant BLACKLIST=192.168.3.131, 10.0.0.1 if \[id.orig\_h\] in \[${BLACKLIST}\] { drop {} } How can we declare… --- ## [S3 input getting stuck and not deleting SQS messages](https://discuss.elastic.co/t/s3-input-getting-stuck-and-not-deleting-sqs-messages/267090)
**Author:** [@buzzdeee](https://discuss.elastic.co/u/buzzdeee)\ **Replies:** 5\ **Last updated:** [April 1, 2021, 5:28pm UTC](https://discuss.elastic.co/t/s3-input-getting-stuck-and-not-deleting-sqs-messages/267090 "2021-04-01T17:28:50Z")
Hi, I'm with filebeat 7.10.0 on Ubuntu. I've configured filebeat to retrieve files from S3 bucket alike: filebeat.inputs: - type: s3 enabled: true queue\_url: "https://sqs.us-west-2.amazonaws.com/… --- ## [Logstash listening](https://discuss.elastic.co/t/logstash-listening/269063)
**Author:** [@nunex\_17](https://discuss.elastic.co/u/nunex_17)\ **Replies:** 3\ **Last updated:** [April 1, 2021, 5:28pm UTC](https://discuss.elastic.co/t/logstash-listening/269063 "2021-04-01T17:28:06Z")
Hi! Assuming filebeat -\> ELK are on different machines. On the filebeat output i have to specify the IP address of the Logstash host. However I can´t see any option to bind my server IP address to Logstash. All the tut… --- ## [Logstash SQS input to accept sqs url](https://discuss.elastic.co/t/logstash-sqs-input-to-accept-sqs-url/269059)
**Author:** [@rachel.takeuchi](https://discuss.elastic.co/u/rachel.takeuchi)\ **Replies:** 0\ **Last updated:** [April 1, 2021, 4:45pm UTC](https://discuss.elastic.co/t/logstash-sqs-input-to-accept-sqs-url/269059 "2021-04-01T16:45:51Z")
I'm using a third party that generates AWS credentials, a bucket and a queue for me to read logs from. I don't have the ability to change the permissions they've supplied to these credentials and unfortunately these cred… --- ## [Pipeline stopped due to database recovery](https://discuss.elastic.co/t/pipeline-stopped-due-to-database-recovery/268983)
**Author:** [@Dhruv\_Rank](https://discuss.elastic.co/u/Dhruv_Rank)\ **Replies:** 1\ **Last updated:** [April 1, 2021, 3:05pm UTC](https://discuss.elastic.co/t/pipeline-stopped-due-to-database-recovery/268983 "2021-04-01T15:05:52Z")
Hello,I am new to this and everyday I got same issue \[2021-03-31T19:28:00,364\]\[ERROR\]\[logstash.inputs.jdbc \]\[add-jobs\] Unable to connect to database. Tried 1 times {:error\_message=\>"Java::OrgPostgresqlUtil::PSQLExcepti… --- ## [Dynamic input for logstash](https://discuss.elastic.co/t/dynamic-input-for-logstash/269004)
**Author:** [@igorid70](https://discuss.elastic.co/u/igorid70)\ **Replies:** 1\ **Last updated:** [April 1, 2021, 2:55pm UTC](https://discuss.elastic.co/t/dynamic-input-for-logstash/269004 "2021-04-01T14:55:09Z")
Hi I have a use case when my logstash pipeline generates output based on which I want to initiate another logstash pipeline with input based on that output. In more details: the current pipeline uses a command input th… --- ## [Changing logstash jdbc statement](https://discuss.elastic.co/t/changing-logstash-jdbc-statement/269037)
**Author:** [@Gregorkosmina](https://discuss.elastic.co/u/Gregorkosmina)\ **Replies:** 1\ **Last updated:** [April 1, 2021, 2:53pm UTC](https://discuss.elastic.co/t/changing-logstash-jdbc-statement/269037 "2021-04-01T14:53:13Z")
I want to change SQL statement in my logstash, so I would fill index with newely queried data. I re-run logstash with new statement but my index still contains old data. Am I missing something? --- ## [Geoip cache\_size # relates to + flush frequency?](https://discuss.elastic.co/t/geoip-cache-size-relates-to-flush-frequency/269007)
**Author:** [@probson](https://discuss.elastic.co/u/probson)\ **Replies:** 2\ **Last updated:** [April 1, 2021, 2:52pm UTC](https://discuss.elastic.co/t/geoip-cache-size-relates-to-flush-frequency/269007 "2021-04-01T14:52:05Z")
Hi, The default is cache\_size for geoip is documented at 1000. Is that 1000 IPs, bytes/kb? Also with pereodic\_flush:false (default), when does this flush? Tracing down a mismatch IP where going back a month up until … --- ## [How to filter logs coming from filebeat and apply a filter on message](https://discuss.elastic.co/t/how-to-filter-logs-coming-from-filebeat-and-apply-a-filter-on-message/267694)
**Author:** [@muralikrishna](https://discuss.elastic.co/u/muralikrishna)\ **Replies:** 3\ **Last updated:** [April 1, 2021, 2:32pm UTC](https://discuss.elastic.co/t/how-to-filter-logs-coming-from-filebeat-and-apply-a-filter-on-message/267694 "2021-04-01T14:32:49Z")
Hi Folks, I am using Logstash 7.8 version and i will be getting logs from multiple data sources as part of my requirement. I wanted to apply a filter on my log message coming from filebeat and containing the string "pa… --- ## [Logstash](https://discuss.elastic.co/t/logstash/269018)
**Author:** [@dddddddddddddddd](https://discuss.elastic.co/u/dddddddddddddddd)\ **Replies:** 6\ **Last updated:** [April 1, 2021, 1:06pm UTC](https://discuss.elastic.co/t/logstash/269018 "2021-04-01T13:06:58Z")
Hi i hope this message find you well , how could to send filtred logs to elasticsearch and in the same time send raw logs to another server (I need this logs for investigation and forensics task) regards and thanks --- ## [Find values inside parenthesees and put it into an array](https://discuss.elastic.co/t/find-values-inside-parenthesees-and-put-it-into-an-array/268935)
**Author:** [@Roberto\_B](https://discuss.elastic.co/u/Roberto_B)\ **Replies:** 2\ **Last updated:** [April 1, 2021, 8:26am UTC](https://discuss.elastic.co/t/find-values-inside-parenthesees-and-put-it-into-an-array/268935 "2021-04-01T08:26:06Z")
Hi all, i have this problem , i have this kind of value into a field: Update for Windows Server 2012 R2 (KB3013410)"", Update for Windows Server 2012 R2 (KB3033446), Update for Windows Server 2012 R2 (KB3024751), Updat… --- ## [Filter Logstash based on field from Filebeat tags](https://discuss.elastic.co/t/filter-logstash-based-on-field-from-filebeat-tags/268965)
**Author:** [@Automation\_Scripts](https://discuss.elastic.co/u/Automation_Scripts)\ **Replies:** 2\ **Last updated:** [April 1, 2021, 5:33am UTC](https://discuss.elastic.co/t/filter-logstash-based-on-field-from-filebeat-tags/268965 "2021-04-01T05:33:09Z")
Hi guys, I need some help filtering some data from Filebeat in Logstash. My original filebeat log looks liks this: \*\*\* This is the log from docker from filebeat container ssl.log "@timestamp": "2021-03-31T23:14:55.7… --- ## [Split filebeat log with a variable length into several new events](https://discuss.elastic.co/t/split-filebeat-log-with-a-variable-length-into-several-new-events/268957)
**Author:** [@rmartinez.rv](https://discuss.elastic.co/u/rmartinez.rv)\ **Replies:** 2\ **Last updated:** [March 31, 2021, 9:49pm UTC](https://discuss.elastic.co/t/split-filebeat-log-with-a-variable-length-into-several-new-events/268957 "2021-03-31T21:49:48Z")
Hi. I'm fairly new with ELK, and i been struggling a lot trying to make this work, with no luck, so i'm asking for a way to solve this... I'm using Filebeat to get logs into my elasticsearc server and, the line logs ar… --- ## [Logstash importing hostname incorrectly](https://discuss.elastic.co/t/logstash-importing-hostname-incorrectly/268496)
**Author:** [@ASA01](https://discuss.elastic.co/u/ASA01)\ **Replies:** 6\ **Last updated:** [March 31, 2021, 9:02pm UTC](https://discuss.elastic.co/t/logstash-importing-hostname-incorrectly/268496 "2021-03-31T21:02:39Z")
Running across something I don't understand after building a new stack with the latest release of Logstash (7.11.1). It appears to be entering the hostname wrong or I understand it wrong. The system hostname is cmscd h… --- ## [Filter postgresql ERROR, FATAL, PANIC messages from postgresql logs](https://discuss.elastic.co/t/filter-postgresql-error-fatal-panic-messages-from-postgresql-logs/268843)
**Author:** [@elk\_newbie](https://discuss.elastic.co/u/elk_newbie)\ **Replies:** 4\ **Last updated:** [March 31, 2021, 6:46pm UTC](https://discuss.elastic.co/t/filter-postgresql-error-fatal-panic-messages-from-postgresql-logs/268843 "2021-03-31T18:46:21Z")
Hi community, Is there any way to filter only ERROR, FATAL and PANIC messages from Postgresql logs? appreciate any help! Regards Patrick --- ## [Time in logstash is getting different due to daylight saving time](https://discuss.elastic.co/t/time-in-logstash-is-getting-different-due-to-daylight-saving-time/268934)
**Author:** [@anjilinga](https://discuss.elastic.co/u/anjilinga)\ **Replies:** 1\ **Last updated:** [March 31, 2021, 5:35pm UTC](https://discuss.elastic.co/t/time-in-logstash-is-getting-different-due-to-daylight-saving-time/268934 "2021-03-31T17:35:16Z")
Hi i am using ruby code in logstash as below and getting the time one hour less. ruby { code =\> "event.set('current\_window\_end', Time.now());" } if current time is 10 :00 (in London) getting the time as 9:00. i have … --- ## [Preventing specific events from being indexed](https://discuss.elastic.co/t/preventing-specific-events-from-being-indexed/268522)
**Author:** [@tsmori](https://discuss.elastic.co/u/tsmori)\ **Replies:** 2\ **Last updated:** [March 31, 2021, 3:48pm UTC](https://discuss.elastic.co/t/preventing-specific-events-from-being-indexed/268522 "2021-03-31T15:48:30Z")
Is there a way to prevent specific events from being indexed? I don't mean specific fields, but conditionals for a field. For example, we're ingesting netflow traffic data and I want to exclude all events that have both… --- ## [Logstash Kafka input : conditonnal consuming?](https://discuss.elastic.co/t/logstash-kafka-input-conditonnal-consuming/268664)
**Author:** [@Travis](https://discuss.elastic.co/u/Travis)\ **Replies:** 12\ **Last updated:** [March 31, 2021, 3:27pm UTC](https://discuss.elastic.co/t/logstash-kafka-input-conditonnal-consuming/268664 "2021-03-31T15:27:08Z")
Hello ! I have multiple firewall log sources. All these sources push the logs in a dedicated topic named "firewall". In logstash, I would like to have a different pipeline for each of these sources to apply different p… --- ## [How to assign one value to another](https://discuss.elastic.co/t/how-to-assign-one-value-to-another/268824)
**Author:** [@Pallavibhushan](https://discuss.elastic.co/u/Pallavibhushan)\ **Replies:** 5\ **Last updated:** [March 31, 2021, 8:52am UTC](https://discuss.elastic.co/t/how-to-assign-one-value-to-another/268824 "2021-03-31T08:52:13Z")
Hi , I am new to logstash I have below data in JSON. "attributes": \[ { "name": "test1", "value": "test1\_value" }, { "name": "test2", "value": "test2\_value" } \] I want this data in below format : "attributes": … --- ## [How to increase the number of filebeat servers in stages](https://discuss.elastic.co/t/how-to-increase-the-number-of-filebeat-servers-in-stages/268853)
**Author:** [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\ **Replies:** 8\ **Last updated:** [March 31, 2021, 3:44am UTC](https://discuss.elastic.co/t/how-to-increase-the-number-of-filebeat-servers-in-stages/268853 "2021-03-31T03:44:45Z")
We are currently trying to increase the number of servers we log using filebeat and logstash in stages. Filebeat can be installed on a new logging server, and logstash can send logs to the installed server. logstash is… --- ## [How to break long condition in logstash config](https://discuss.elastic.co/t/how-to-break-long-condition-in-logstash-config/268709)
**Author:** [@alicango](https://discuss.elastic.co/u/alicango)\ **Replies:** 2\ **Last updated:** [March 30, 2021, 10:34pm UTC](https://discuss.elastic.co/t/how-to-break-long-condition-in-logstash-config/268709 "2021-03-30T22:34:11Z")
I have a filter section that removes a field if the event matches with regex. So my if condition is very long. if \[type\] == "something" { if \[displayName\] =~ /(\[M-m\]as|MAS)\[\\d\]\*/ or \[displayName\] =~ /(\[E-e\]lk|ELK)/ … --- ## [Logstash doesn't log to elastic](https://discuss.elastic.co/t/logstash-doesnt-log-to-elastic/268025)
**Author:** [@ShadwDrgn](https://discuss.elastic.co/u/ShadwDrgn)\ **Replies:** 5\ **Last updated:** [March 30, 2021, 9:43pm UTC](https://discuss.elastic.co/t/logstash-doesnt-log-to-elastic/268025 "2021-03-30T21:43:02Z")
I've been trying to troubleshoot this for quite some time and have managed to find out the following: filebeat logs everything seemingly fine turning on logstash after deleting the indexes i'm logging to SEEMINGLY wor… --- ## [Logstash unexpected message - crypto/rsa verification error](https://discuss.elastic.co/t/logstash-unexpected-message-crypto-rsa-verification-error/267484)
**Author:** [@dkdlv38](https://discuss.elastic.co/u/dkdlv38)\ **Replies:** 3\ **Last updated:** [March 30, 2021, 12:29pm UTC](https://discuss.elastic.co/t/logstash-unexpected-message-crypto-rsa-verification-error/267484 "2021-03-30T12:29:49Z")
Hi all, I'm new here and I have an issue with the configuration of my Windows machines sending logs to a Logstash server through Winlogbeat (for what I understand as I recovered this installation undocumented from a col… --- ## [How to add count value to each csv value in logstash](https://discuss.elastic.co/t/how-to-add-count-value-to-each-csv-value-in-logstash/268770)
**Author:** [@ofitz](https://discuss.elastic.co/u/ofitz)\ **Replies:** 0\ **Last updated:** [March 30, 2021, 10:22am UTC](https://discuss.elastic.co/t/how-to-add-count-value-to-each-csv-value-in-logstash/268770 "2021-03-30T10:22:50Z")
I have some csv, and with ruby i convert it to array , like this: 11;22;33;44;55 66;77;88;99;100 ruby { code =\> ' m = event.get("message").split(";") m.each\_index { |x| m\[x\] = m\[x\].to\_f } … --- ## [Logstash - output csv plugin - all fields are strings](https://discuss.elastic.co/t/logstash-output-csv-plugin-all-fields-are-strings/268513)
**Author:** [@YaronB](https://discuss.elastic.co/u/YaronB)\ **Replies:** 6\ **Last updated:** [March 30, 2021, 10:11am UTC](https://discuss.elastic.co/t/logstash-output-csv-plugin-all-fields-are-strings/268513 "2021-03-30T10:11:55Z")
i'm trying to move some data to mongodb from elastic using elastic input plugin and csv ouput plugin and then mongoimport. how can i define the fields types , including nested fields? everything is showing up as string… --- ## [Logstash failed to start after plugin update](https://discuss.elastic.co/t/logstash-failed-to-start-after-plugin-update/268472)
**Author:** [@anandd4](https://discuss.elastic.co/u/anandd4)\ **Replies:** 2\ **Last updated:** [March 30, 2021, 8:00am UTC](https://discuss.elastic.co/t/logstash-failed-to-start-after-plugin-update/268472 "2021-03-30T08:00:42Z")
After updating beats input plugin from 6.0.5 to 6.1.0 on logstash(7.5.2), the service stopped to work & keeps on throwing the following error - \`logstash\[22854\]: \[ERROR\] 2021-03-26 10:35:57.423 \[main\] Logstash - java.la… --- ## ["Content-Type header \[text/plain; charset=ISO-8859-1\] is not supported","status":406](https://discuss.elastic.co/t/content-type-header-text-plain-charset-iso-8859-1-is-not-supported-status-406/268309)
**Author:** [@Dan4](https://discuss.elastic.co/u/Dan4)\ **Replies:** 2\ **Last updated:** [March 30, 2021, 7:47am UTC](https://discuss.elastic.co/t/content-type-header-text-plain-charset-iso-8859-1-is-not-supported-status-406/268309 "2021-03-30T07:47:33Z")
Hello together, I am new to the elastic stack and have no further experience than just google a solution. In my environment I have to use Logstash on a remote server. I have no control and influence over the version whi… --- ## [Logstash persistent queue empty but all events coming in fine](https://discuss.elastic.co/t/logstash-persistent-queue-empty-but-all-events-coming-in-fine/268619)
**Author:** [@pk.241011](https://discuss.elastic.co/u/pk.241011)\ **Replies:** 5\ **Last updated:** [March 30, 2021, 4:11am UTC](https://discuss.elastic.co/t/logstash-persistent-queue-empty-but-all-events-coming-in-fine/268619 "2021-03-30T04:11:49Z")
I have configured logstash to use persistent queue. As per documentation, "When an input has events ready to process, it writes them to the queue." I was expecting to see some files in the folder '/Data/logstash/data/… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=241) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=243)