# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=196 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 197 --- ## [Logstash Ruby Tmp Folders - Temporary Solution](https://discuss.elastic.co/t/logstash-ruby-tmp-folders-temporary-solution/283793)
**Author:** [@Tydorius](https://discuss.elastic.co/u/Tydorius)\ **Replies:** 0\ **Last updated:** [September 9, 2021, 2:38pm UTC](https://discuss.elastic.co/t/logstash-ruby-tmp-folders-temporary-solution/283793 "2021-09-09T14:38:55Z")
I am currently experiencing the known issue where JRuby files are not cleaned up in /tmp on CentOS 7. My temporary resolution has been to set up a cron job to remove these folders. To save others trouble, here is my set… --- ## [Logstash - based on filed type create a new field](https://discuss.elastic.co/t/logstash-based-on-filed-type-create-a-new-field/283684)
**Author:** [@antonisnyc94](https://discuss.elastic.co/u/antonisnyc94)\ **Replies:** 5\ **Last updated:** [September 9, 2021, 10:43am UTC](https://discuss.elastic.co/t/logstash-based-on-filed-type-create-a-new-field/283684 "2021-09-09T10:43:42Z")
Hello, I've been trying for 1-2 days to figure out why to create new fields when the type of the event is object but I cant seem to make it work.. This is the error im getting: Could not index event to Elasticsearch. … --- ## [Logstash support snmptrap v3?](https://discuss.elastic.co/t/logstash-support-snmptrap-v3/283722)
**Author:** [@T\_Mashimo](https://discuss.elastic.co/u/T_Mashimo)\ **Replies:** 1\ **Last updated:** [September 9, 2021, 3:00am UTC](https://discuss.elastic.co/t/logstash-support-snmptrap-v3/283722 "2021-09-09T03:00:57Z")
Hello. Does snmptrap input plugin for logstash support v3? The official documentation does not specify the supported versions. Someone is asking the same question here, but there is no answer. Thank you. --- ## [Json parse with logstash](https://discuss.elastic.co/t/json-parse-with-logstash/283714)
**Author:** [@hiba](https://discuss.elastic.co/u/hiba)\ **Replies:** 4\ **Last updated:** [September 8, 2021, 11:49pm UTC](https://discuss.elastic.co/t/json-parse-with-logstash/283714 "2021-09-08T23:49:49Z")
Hi team, I want to laod data from sql server to elasticsearch via logstash 6.5 I have a table in sql server which has json objects as a value in one of the columns. My requirement is to read those objects as it is and … --- ## [How to get only deltas in elasticsearch input](https://discuss.elastic.co/t/how-to-get-only-deltas-in-elasticsearch-input/283708)
**Author:** [@francieliton\_araujo](https://discuss.elastic.co/u/francieliton_araujo)\ **Replies:** 1\ **Last updated:** [September 8, 2021, 8:41pm UTC](https://discuss.elastic.co/t/how-to-get-only-deltas-in-elasticsearch-input/283708 "2021-09-08T20:41:07Z")
I would like to know how I do something like jdbc-input, that the query is always the most recent, but this using elaticsearch\_input. if it is not possible, please help me to build a query that returns everything from t… --- ## [Running as service, continuously giving "No source loaders matched! This shouldn't happen"](https://discuss.elastic.co/t/running-as-service-continuously-giving-no-source-loaders-matched-this-shouldnt-happen/283487)
**Author:** [@matt.snyder](https://discuss.elastic.co/u/matt.snyder)\ **Replies:** 1\ **Last updated:** [September 8, 2021, 5:57pm UTC](https://discuss.elastic.co/t/running-as-service-continuously-giving-no-source-loaders-matched-this-shouldnt-happen/283487 "2021-09-08T17:57:26Z")
I am running logstash as a service, and it seems that an erroneous job is stuck in a queue somewhere, though I cannot find it. The error in the subject appears over and over in logstash-plain.log, no matter what files I… --- ## [Ruby time.strftime Stripping hours from the time](https://discuss.elastic.co/t/ruby-time-strftime-stripping-hours-from-the-time/283655)
**Author:** [@Bryan\_Hamilton](https://discuss.elastic.co/u/Bryan_Hamilton)\ **Replies:** 3\ **Last updated:** [September 8, 2021, 3:46pm UTC](https://discuss.elastic.co/t/ruby-time-strftime-stripping-hours-from-the-time/283655 "2021-09-08T15:46:17Z")
Hi all, I am using the following filter to extract only the time from the @timestamp field. ruby { code =\> "event.set('\[datetime\]\[time\]',event.get('@timestamp').time.strftime('%H:%M:%S'))" } Which works exept … --- ## [Logstash - not update a document even if document\_id is specified](https://discuss.elastic.co/t/logstash-not-update-a-document-even-if-document-id-is-specified/283660)
**Author:** [@reed](https://discuss.elastic.co/u/reed)\ **Replies:** 3\ **Last updated:** [September 8, 2021, 2:23pm UTC](https://discuss.elastic.co/t/logstash-not-update-a-document-even-if-document-id-is-specified/283660 "2021-09-08T14:23:15Z")
Hi all, I'm new with elastic technology and I wold ask this: I have my pipeline with output { elasticsearch { hosts =\> "localhost:9200" index =\> "crsi-index-template-%{+yyyy.MM.dd}" document\_id … --- ## [Logstash processing only number of events 2 times to pipeline.max\_inflight](https://discuss.elastic.co/t/logstash-processing-only-number-of-events-2-times-to-pipeline-max-inflight/283664)
**Author:** [@amar.molaka](https://discuss.elastic.co/u/amar.molaka)\ **Replies:** 0\ **Last updated:** [September 8, 2021, 1:21pm UTC](https://discuss.elastic.co/t/logstash-processing-only-number-of-events-2-times-to-pipeline-max-inflight/283664 "2021-09-08T13:21:17Z")
My pipeline configuration as below - pipeline.id: staticmetrics queue.type: persisted pipeline.workers: 10 pipeline.batch.size: 2 pipeline.batch.delay: 1000 path.config: "/Users/James/Desktop/metrics.con… --- ## [Updating Logstash Keystore Key-Values Externally](https://discuss.elastic.co/t/updating-logstash-keystore-key-values-externally/283611)
**Author:** [@nitvakhare](https://discuss.elastic.co/u/nitvakhare)\ **Replies:** 0\ **Last updated:** [September 8, 2021, 5:52am UTC](https://discuss.elastic.co/t/updating-logstash-keystore-key-values-externally/283611 "2021-09-08T05:52:43Z")
Hi, I wanted to know that if there is any option to update the existing key-value pair in the Logstash keystore externally. I am storing 'username' and 'password' fields in this keystore manually for now. But, I want to… --- ## [Error in Netty pipeline: java.io.IOException: Connection reset by peer](https://discuss.elastic.co/t/error-in-netty-pipeline-java-io-ioexception-connection-reset-by-peer/283606)
**Author:** [@smm](https://discuss.elastic.co/u/smm)\ **Replies:** 0\ **Last updated:** [September 8, 2021, 5:04am UTC](https://discuss.elastic.co/t/error-in-netty-pipeline-java-io-ioexception-connection-reset-by-peer/283606 "2021-09-08T05:04:23Z")
Dear community, in my logstash 7.8.1 logs I get this error message quite a lot: \[2021-09-07T08:36:56,031\]\[ERROR\]\[logstash.inputs.tcp \]\[main\] Error in Netty pipeline: java.io.IOException: Connection reset by peer It on… --- ## [Filter to change the field name to be more user friendly](https://discuss.elastic.co/t/filter-to-change-the-field-name-to-be-more-user-friendly/283543)
**Author:** [@Shreesh\_Narayanan](https://discuss.elastic.co/u/Shreesh_Narayanan)\ **Replies:** 2\ **Last updated:** [September 8, 2021, 4:38am UTC](https://discuss.elastic.co/t/filter-to-change-the-field-name-to-be-more-user-friendly/283543 "2021-09-08T04:38:16Z")
Hello, I have a log line for diskusage such as disk-usage: \[1\]=76% \[2\]=26% \[3\]=26% \[4\]=24% \[5\]=24% \[6\]=28% i used dissect and kv to get the values as needed . { "@timestamp" =\> 2021-09-06T08:22:46.944Z, … --- ## [How to filter data while migrating from SQL to Elasticsearch using Logstash to obtain a nested strings array in a JSON object](https://discuss.elastic.co/t/how-to-filter-data-while-migrating-from-sql-to-elasticsearch-using-logstash-to-obtain-a-nested-strings-array-in-a-json-object/283585)
**Author:** [@maklil](https://discuss.elastic.co/u/maklil)\ **Replies:** 0\ **Last updated:** [September 7, 2021, 10:25pm UTC](https://discuss.elastic.co/t/how-to-filter-data-while-migrating-from-sql-to-elasticsearch-using-logstash-to-obtain-a-nested-strings-array-in-a-json-object/283585 "2021-09-07T22:25:59Z")
Hi, I'm trying to migrate data from SQL Server to Elasticsearch using Logstash into a certain format. My data is normalized and essentially involves a central table which is linked to other tables by a foreign key, and m… --- ## [Logastash Exec plugin Cannot allocate memory](https://discuss.elastic.co/t/logastash-exec-plugin-cannot-allocate-memory/283561)
**Author:** [@vikasgurlinka](https://discuss.elastic.co/u/vikasgurlinka)\ **Replies:** 2\ **Last updated:** [September 7, 2021, 6:28pm UTC](https://discuss.elastic.co/t/logastash-exec-plugin-cannot-allocate-memory/283561 "2021-09-07T18:28:31Z")
I am running Logstash 7.9.1 with 24 GB of maximum heap size configured in a Linux machine with 30 GB Memory and configured a pipeline to run a python program with Logstash exec plugin I see the below error message in log… --- ## [Logstash pipelines failing as Logstash failed to create queue](https://discuss.elastic.co/t/logstash-pipelines-failing-as-logstash-failed-to-create-queue/283461)
**Author:** [@Rahul\_Dey](https://discuss.elastic.co/u/Rahul_Dey)\ **Replies:** 0\ **Last updated:** [September 6, 2021, 3:21pm UTC](https://discuss.elastic.co/t/logstash-pipelines-failing-as-logstash-failed-to-create-queue/283461 "2021-09-06T15:21:16Z")
I am running logstash in a docker container. There are 7-8 logstash pipelines defined under pipelines.yml file. Queue type for every pipeline is "persisted". Everything was working fine but recently whenever I am startin… --- ## [Else if condition is not working?](https://discuss.elastic.co/t/else-if-condition-is-not-working/283544)
**Author:** [@padamrai](https://discuss.elastic.co/u/padamrai)\ **Replies:** 4\ **Last updated:** [September 7, 2021, 2:46pm UTC](https://discuss.elastic.co/t/else-if-condition-is-not-working/283544 "2021-09-07T14:46:14Z")
I want if SubmittedOn is null then add New field with value 1 mutate{ add\_field=\>{"Queue"=\>1} } if \[SubmittedOn\]{ if \[null\] not in \[SubmittedOn\]{ date { m… --- ## [Logstash unable to connect to elasticsearch](https://discuss.elastic.co/t/logstash-unable-to-connect-to-elasticsearch/283548)
**Author:** [@geetika\_gopi](https://discuss.elastic.co/u/geetika_gopi)\ **Replies:** 0\ **Last updated:** [September 7, 2021, 2:39pm UTC](https://discuss.elastic.co/t/logstash-unable-to-connect-to-elasticsearch/283548 "2021-09-07T14:39:14Z")
I have SSL enabled on all my elastic nodes, using a self signed certificate. However logstash is unable to connect to elasticsearch. Gives the following error \[2021-09-07T10:34:50,283\]\[WARN \]\[logstash.outputs.elasticsea… --- ## [Does logstash support having a list/array in root (top level) of the event](https://discuss.elastic.co/t/does-logstash-support-having-a-list-array-in-root-top-level-of-the-event/283524)
**Author:** [@Renjith\_PK](https://discuss.elastic.co/u/Renjith_PK)\ **Replies:** 4\ **Last updated:** [September 7, 2021, 2:35pm UTC](https://discuss.elastic.co/t/does-logstash-support-having-a-list-array-in-root-top-level-of-the-event/283524 "2021-09-07T14:35:29Z")
We have input data like this message:\[{key1:value1},{key2:value2}\] and we need value of message, which is a list, directly on root of event (to use as zipkin storage) without any additional field. I tried using json fi… --- ## [Fingerprint duplicate hash for different events](https://discuss.elastic.co/t/fingerprint-duplicate-hash-for-different-events/282462)
**Author:** [@Tuckson](https://discuss.elastic.co/u/Tuckson)\ **Replies:** 3\ **Last updated:** [September 7, 2021, 10:51am UTC](https://discuss.elastic.co/t/fingerprint-duplicate-hash-for-different-events/282462 "2021-09-07T10:51:11Z")
Hi, I have a strange issue which came to light after starting to use datastreams (and thus create events instead of updates). The following fingerprint config we have in logstash: ### Add a fingerprint to… --- ## [Fast ingest](https://discuss.elastic.co/t/fast-ingest/283518)
**Author:** [@zozo6015](https://discuss.elastic.co/u/zozo6015)\ **Replies:** 0\ **Last updated:** [September 7, 2021, 9:52am UTC](https://discuss.elastic.co/t/fast-ingest/283518 "2021-09-07T09:52:36Z")
Hello, I have a project where we are considering using elasticsearch for some very high amount of data around 150billion documents/day is there a way we can ingest that amount of data in a real life scenario? I underst… --- ## [Rate\_1min keeps dropping for my kafka to s3 batching on logstash](https://discuss.elastic.co/t/rate-1min-keeps-dropping-for-my-kafka-to-s3-batching-on-logstash/283508)
**Author:** [@Varsha1](https://discuss.elastic.co/u/Varsha1)\ **Replies:** 0\ **Last updated:** [September 7, 2021, 8:45am UTC](https://discuss.elastic.co/t/rate-1min-keeps-dropping-for-my-kafka-to-s3-batching-on-logstash/283508 "2021-09-07T08:45:22Z")
I'm trying to transfer data from kafka to s3, without any data transformation filters. I added metrics filter plugin to calculate the rate of transfer, which keeps falling from ~16k to as low as ~3k. Is this expected an… --- ## [Compare/Merge 2 type of dataset](https://discuss.elastic.co/t/compare-merge-2-type-of-dataset/282624)
**Author:** [@Peter\_Ch](https://discuss.elastic.co/u/Peter_Ch)\ **Replies:** 4\ **Last updated:** [September 7, 2021, 8:22am UTC](https://discuss.elastic.co/t/compare-merge-2-type-of-dataset/282624 "2021-09-07T08:22:37Z")
Dear All, I user winlogbeat to collet my window log which include username. And I have a csv file which contain the username and user department. May I know is it possible to compare those datasets username field, if th… --- ## [Logstash udp input listen on port 5044 but doesn't receive syslog messages](https://discuss.elastic.co/t/logstash-udp-input-listen-on-port-5044-but-doesnt-receive-syslog-messages/283496)
**Author:** [@kiran80511](https://discuss.elastic.co/u/kiran80511)\ **Replies:** 0\ **Last updated:** [September 7, 2021, 7:17am UTC](https://discuss.elastic.co/t/logstash-udp-input-listen-on-port-5044-but-doesnt-receive-syslog-messages/283496 "2021-09-07T07:17:57Z")
I have tried with this command tcpdump -vvv -A -i any port 5044 It receive the syslog messages 12:41:34.438831 IP (tos 0x0, ttl 128, id 37582, offset 0, flags \[none\], proto UDP (17), length 374) gateway.lxi-evntsv… --- ## [Elasticsearch data to mysql](https://discuss.elastic.co/t/elasticsearch-data-to-mysql/283488)
**Author:** [@zhyp](https://discuss.elastic.co/u/zhyp)\ **Replies:** 0\ **Last updated:** [September 7, 2021, 5:13am UTC](https://discuss.elastic.co/t/elasticsearch-data-to-mysql/283488 "2021-09-07T05:13:35Z")
Hi :grinning: My data is in elasticsearch. I want to use logstash to synchronize to MySQL, or use other tools. What can I do to achieve it？ --- ## [Logstash OSS License after 7.10 version](https://discuss.elastic.co/t/logstash-oss-license-after-7-10-version/283429)
**Author:** [@YashC](https://discuss.elastic.co/u/YashC)\ **Replies:** 3\ **Last updated:** [September 6, 2021, 10:57pm UTC](https://discuss.elastic.co/t/logstash-oss-license-after-7-10-version/283429 "2021-09-06T22:57:07Z")
Hello, I am using logstash-oss docker image version 7.10.2, as it is based on Apache 2.0 license. Since Jan 2021 Elastic introduced a different license model starting from versions 7.11. However, there are still logst… --- ## [Date filter plugin doesn't take the timezone into account when using ISO8601](https://discuss.elastic.co/t/date-filter-plugin-doesnt-take-the-timezone-into-account-when-using-iso8601/283306)
**Author:** [@Jose\_E](https://discuss.elastic.co/u/Jose_E)\ **Replies:** 3\ **Last updated:** [September 6, 2021, 9:14am UTC](https://discuss.elastic.co/t/date-filter-plugin-doesnt-take-the-timezone-into-account-when-using-iso8601/283306 "2021-09-06T09:14:37Z")
Hi, I'm trying to normalize some timestamps into UTC from CEST, so far I've had no issues whatsoever except for when the parsed timestamp is already in ISO8601. At first, I wasn't sure whether the date-filter-plugin was… --- ## [Only one Logstash pipeline doesn't have uuid and appear in the Standalone Cluster](https://discuss.elastic.co/t/only-one-logstash-pipeline-doesnt-have-uuid-and-appear-in-the-standalone-cluster/283417)
**Author:** [@Musketeer7](https://discuss.elastic.co/u/Musketeer7)\ **Replies:** 0\ **Last updated:** [September 6, 2021, 7:44am UTC](https://discuss.elastic.co/t/only-one-logstash-pipeline-doesnt-have-uuid-and-appear-in-the-standalone-cluster/283417 "2021-09-06T07:44:26Z")
Hello everyone Firstly I should mention that I already know about the whole xpack.monitoring.cluster\_uuid thing that was fixed in this PR. However this doesn't seem to be the case here. I have a production cluster, wh… --- ## [Using memcached for logstash](https://discuss.elastic.co/t/using-memcached-for-logstash/283381)
**Author:** [@kriss332](https://discuss.elastic.co/u/kriss332)\ **Replies:** 0\ **Last updated:** [September 5, 2021, 5:57pm UTC](https://discuss.elastic.co/t/using-memcached-for-logstash/283381 "2021-09-05T17:57:12Z")
Hi all. I have to work with memcached for logstash to get the alerts about the destination IPs (belonging to C2 IPs) from my firewall's destination IPs. I have MISP for finding malicious/suspicious IPs about the traffic. … --- ## [Ruby filter plugin does not read new field values](https://discuss.elastic.co/t/ruby-filter-plugin-does-not-read-new-field-values/283343)
**Author:** [@priyaankaa](https://discuss.elastic.co/u/priyaankaa)\ **Replies:** 8\ **Last updated:** [September 5, 2021, 2:22pm UTC](https://discuss.elastic.co/t/ruby-filter-plugin-does-not-read-new-field-values/283343 "2021-09-05T14:22:25Z")
How to get correct value of the field inside ruby code in logstash pipeline? sample 1: input { elasticsearch { hosts =\> "http://localhost:9200" index =\> "test1" } } filter { mutate { add\_field =\>… --- ## [Using Kafka Stream for data transform and enrichment](https://discuss.elastic.co/t/using-kafka-stream-for-data-transform-and-enrichment/283370)
**Author:** [@mruthyu](https://discuss.elastic.co/u/mruthyu)\ **Replies:** 0\ **Last updated:** [September 5, 2021, 11:40am UTC](https://discuss.elastic.co/t/using-kafka-stream-for-data-transform-and-enrichment/283370 "2021-09-05T11:40:04Z")
Is it possible to use Kafka Streaming for data transform and enrichment instead of logstash? What are the advantages and disadvantages? [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=195) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=197)