# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=383 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 384 --- ## [Execute program and assign a field its output through ruby filter](https://discuss.elastic.co/t/execute-program-and-assign-a-field-its-output-through-ruby-filter/225484)
**Author:** [@deostroll](https://discuss.elastic.co/u/deostroll)\ **Replies:** 4\ **Last updated:** [March 30, 2020, 2:46am UTC](https://discuss.elastic.co/t/execute-program-and-assign-a-field-its-output-through-ruby-filter/225484 "2020-03-30T02:46:07Z")
My logstash receives JSON messages from a kafka topic. During development phase it is useful if I can read the json in a flattened way. There exists a program called catj which does this. It is an npm binary. You need to… --- ## [Limit down logstash data ingestion rate](https://discuss.elastic.co/t/limit-down-logstash-data-ingestion-rate/225595)
**Author:** [@rsuper\_6616](https://discuss.elastic.co/u/rsuper_6616)\ **Replies:** 0\ **Last updated:** [March 29, 2020, 9:56pm UTC](https://discuss.elastic.co/t/limit-down-logstash-data-ingestion-rate/225595 "2020-03-29T21:56:32Z")
I am reading data from csv and loading it to elastic cloud v7 using logstash I need to limit down the data ingestion rate of logstash as it afect the production cluster --- ## [Load Several CSV Files to several indices in Elastic cloud based on date field](https://discuss.elastic.co/t/load-several-csv-files-to-several-indices-in-elastic-cloud-based-on-date-field/225382)
**Author:** [@rsuper\_6616](https://discuss.elastic.co/u/rsuper_6616)\ **Replies:** 6\ **Last updated:** [March 29, 2020, 9:53pm UTC](https://discuss.elastic.co/t/load-several-csv-files-to-several-indices-in-elastic-cloud-based-on-date-field/225382 "2020-03-29T21:53:11Z")
I have several CSV I need to load them to elastic cloud based on date field such that each index will contain only one day. is it possible? --- ## [Logstash XML Filter - xpath won't select element value](https://discuss.elastic.co/t/logstash-xml-filter-xpath-wont-select-element-value/225564)
**Author:** [@Paul\_Offord](https://discuss.elastic.co/u/Paul_Offord)\ **Replies:** 1\ **Last updated:** [March 29, 2020, 3:56pm UTC](https://discuss.elastic.co/t/logstash-xml-filter-xpath-wont-select-element-value/225564 "2020-03-29T15:56:26Z")
I am using Logstash 7.6.1 to ingest an XML file that was generated using a standard Microsoft Windows tool call WinInet Trace. Although I am able to ingest the entire XML, I want to generate some hashes from the values … --- ## [Ruby filter error](https://discuss.elastic.co/t/ruby-filter-error/225306)
**Author:** [@Stefano\_Bossi](https://discuss.elastic.co/u/Stefano_Bossi)\ **Replies:** 2\ **Last updated:** [March 29, 2020, 2:41pm UTC](https://discuss.elastic.co/t/ruby-filter-error/225306 "2020-03-29T14:41:48Z")
Dear Elastic Forum, I am trying to understand how the ruby filter works in order to apply a complicated reindexing of some data in my cluster. I wrote a simple ruby filter example which doesn't work: input { ela… --- ## [\[logstash-input-file\] when I rename file and restart , logstash will read duplication](https://discuss.elastic.co/t/logstash-input-file-when-i-rename-file-and-restart-logstash-will-read-duplication/225559)
**Author:** [@Antido-L](https://discuss.elastic.co/u/Antido-L)\ **Replies:** 0\ **Last updated:** [March 29, 2020, 2:37pm UTC](https://discuss.elastic.co/t/logstash-input-file-when-i-rename-file-and-restart-logstash-will-read-duplication/225559 "2020-03-29T14:37:35Z")
I use logstash-input-file-plugin. when I rename file and restart , logstash will read duplication. I rotate log in the way renaming method. The scene is as follows： During the logstash shutdown, the service.log is re… --- ## [Use Logstash persistent queue, or use Redis/Kafka/RabbitMQ? (On a server that's offline most of the time)](https://discuss.elastic.co/t/use-logstash-persistent-queue-or-use-redis-kafka-rabbitmq-on-a-server-thats-offline-most-of-the-time/225502)
**Author:** [@dreamspy](https://discuss.elastic.co/u/dreamspy)\ **Replies:** 1\ **Last updated:** [March 28, 2020, 5:29pm UTC](https://discuss.elastic.co/t/use-logstash-persistent-queue-or-use-redis-kafka-rabbitmq-on-a-server-thats-offline-most-of-the-time/225502 "2020-03-28T17:29:09Z")
Hi there I'm in the situation where I'll have robots/machines that are offline most of the time, since they'll be out in the field doing measurements in situations where one can't have access to the internet. But when … --- ## [Logstash Ruby Filter Not Removing Fields](https://discuss.elastic.co/t/logstash-ruby-filter-not-removing-fields/225297)
**Author:** [@Rahul\_Kumar4](https://discuss.elastic.co/u/Rahul_Kumar4)\ **Replies:** 3\ **Last updated:** [March 28, 2020, 11:06am UTC](https://discuss.elastic.co/t/logstash-ruby-filter-not-removing-fields/225297 "2020-03-28T11:06:33Z")
Hello, I have logs that look like this { "attachments" : \[ { "field1" : "rkredux", "inner\_value" : \[ "check" \] }, { "field1" … --- ## [Grok Matches but getting \_grokparsefailure in Kibana](https://discuss.elastic.co/t/grok-matches-but-getting-grokparsefailure-in-kibana/225476)
**Author:** [@Harsh\_Sharma](https://discuss.elastic.co/u/Harsh_Sharma)\ **Replies:** 0\ **Last updated:** [March 28, 2020, 8:58am UTC](https://discuss.elastic.co/t/grok-matches-but-getting-grokparsefailure-in-kibana/225476 "2020-03-28T08:58:26Z")
I don't know what's wrong here. Grok matches for access.log but when creating index pattern in KIBANA logs showing \_grokparsefailure I'm using ELK version of 6.5.4. Sample log - 157.50.50.50 - - \[28/Mar/2020:13:36:28 … --- ## [Enhance ELK data records with an external source?](https://discuss.elastic.co/t/enhance-elk-data-records-with-an-external-source/225291)
**Author:** [@redapplesonly](https://discuss.elastic.co/u/redapplesonly)\ **Replies:** 2\ **Last updated:** [March 27, 2020, 8:45pm UTC](https://discuss.elastic.co/t/enhance-elk-data-records-with-an-external-source/225291 "2020-03-27T20:45:32Z")
Hello Logstash Gurus, I have built a server that receives network data from external sources. That data goes into Logstash, then Elasticsearch, then is displayed on Kibana. Everything works great. (I’m using ELK vers… --- ## [Two different aggregate blocks in one logstash filter](https://discuss.elastic.co/t/two-different-aggregate-blocks-in-one-logstash-filter/225205)
**Author:** [@dsv](https://discuss.elastic.co/u/dsv)\ **Replies:** 3\ **Last updated:** [March 27, 2020, 5:47pm UTC](https://discuss.elastic.co/t/two-different-aggregate-blocks-in-one-logstash-filter/225205 "2020-03-27T17:47:26Z")
Need to aggregate two different sets of Cisco ASA logs. One - for a cert verification, other - for a firewall. ### TOKEN PRESENSE CHECK (717022 - cert validated THAN 722051 - IP assigned, # To match task\_id user.nam… --- ## [Logstash only receiving DNS log while ignoring DHCP using filebeat](https://discuss.elastic.co/t/logstash-only-receiving-dns-log-while-ignoring-dhcp-using-filebeat/225422)
**Author:** [@data101](https://discuss.elastic.co/u/data101)\ **Replies:** 6\ **Last updated:** [March 27, 2020, 5:30pm UTC](https://discuss.elastic.co/t/logstash-only-receiving-dns-log-while-ignoring-dhcp-using-filebeat/225422 "2020-03-27T17:30:42Z")
Hello, I am trying to ship windows server DNS and DHCP logs to my testing ELK environment (Ubuntu). I can see Filebeat is sending the logs but for some reason logstash is only receiving dns logs, not dhcp. I am using la… --- ## [Filtering on nested json fields of varying length](https://discuss.elastic.co/t/filtering-on-nested-json-fields-of-varying-length/225373)
**Author:** [@cyoez](https://discuss.elastic.co/u/cyoez)\ **Replies:** 3\ **Last updated:** [March 27, 2020, 4:44pm UTC](https://discuss.elastic.co/t/filtering-on-nested-json-fields-of-varying-length/225373 "2020-03-27T16:44:05Z")
Hello everyone, First of all, I am new to Logstash, so I would not be surprised if what I am asking is not within the scope of what Logstash can do. So, I am working on a project where I need to do multiple GeoIP looku… --- ## [Is it possible to pass "nil" in a @metadata field to "pipeline =\>" on elasticsearch output](https://discuss.elastic.co/t/is-it-possible-to-pass-nil-in-a-metadata-field-to-pipeline-on-elasticsearch-output/225424)
**Author:** [@A\_B](https://discuss.elastic.co/u/A_B)\ **Replies:** 0\ **Last updated:** [March 27, 2020, 2:38pm UTC](https://discuss.elastic.co/t/is-it-possible-to-pass-nil-in-a-metadata-field-to-pipeline-on-elasticsearch-output/225424 "2020-03-27T14:38:04Z")
Hello, this might be a slightly odd question. I have been successfully gotten by with no conditionals in the output section of my Logstash config for quite some time now. I would however like to start to make use of Ela… --- ## [Randomly crashes due to a TypeError - no implicit conversion of nil into Integer](https://discuss.elastic.co/t/randomly-crashes-due-to-a-typeerror-no-implicit-conversion-of-nil-into-integer/225420)
**Author:** [@rohe](https://discuss.elastic.co/u/rohe)\ **Replies:** 0\ **Last updated:** [March 27, 2020, 2:05pm UTC](https://discuss.elastic.co/t/randomly-crashes-due-to-a-typeerror-no-implicit-conversion-of-nil-into-integer/225420 "2020-03-27T14:05:23Z")
Hello I am using logstash 7.3.0, which is configured to read input from Kafka and output to elasticsearch or a different Kafka topic in certain cases. My config is as follows: input { kafka { bootstra… --- ## [JT400.jar on Logstash to AS400](https://discuss.elastic.co/t/jt400-jar-on-logstash-to-as400/225364)
**Author:** [@HerveSavard](https://discuss.elastic.co/u/HerveSavard)\ **Replies:** 1\ **Last updated:** [March 27, 2020, 1:44pm UTC](https://discuss.elastic.co/t/jt400-jar-on-logstash-to-as400/225364 "2020-03-27T13:44:45Z")
'Hi, Maybe somebody used JT400.jar. I received always this message: Unable to connect to database. Tried 1 times {:error\_message=\>"Java::JavaSql::SQLException: No suitable driver found for jdbc:as400 Some idea? Herv… --- ## [Multiple geoip target for source and destination](https://discuss.elastic.co/t/multiple-geoip-target-for-source-and-destination/225417)
**Author:** [@fadjar340](https://discuss.elastic.co/u/fadjar340)\ **Replies:** 0\ **Last updated:** [March 27, 2020, 1:37pm UTC](https://discuss.elastic.co/t/multiple-geoip-target-for-source-and-destination/225417 "2020-03-27T13:37:43Z")
Hi, I just create logstash filter and need multiple target of geoip, for source and destination. The source already have from the filter but for destination target not create the source.geoip.location and destination.g… --- ## [Two year old issue with persistent queues](https://discuss.elastic.co/t/two-year-old-issue-with-persistent-queues/225378)
**Author:** [@ceekay](https://discuss.elastic.co/u/ceekay)\ **Replies:** 0\ **Last updated:** [March 27, 2020, 9:46am UTC](https://discuss.elastic.co/t/two-year-old-issue-with-persistent-queues/225378 "2020-03-27T09:46:29Z")
Can someone from Elastic please look at this issue from 2018? It's never been assigned or commented on by anyone from Elastic, and it's an ongoing issue. There seem to be a number that slip under the radar, but this … --- ## [Logstash Performance comparison \_update\_by\_query v/s pulling data from DB via SQL](https://discuss.elastic.co/t/logstash-performance-comparison-update-by-query-v-s-pulling-data-from-db-via-sql/225308)
**Author:** [@MChat](https://discuss.elastic.co/u/MChat)\ **Replies:** 0\ **Last updated:** [March 26, 2020, 11:13pm UTC](https://discuss.elastic.co/t/logstash-performance-comparison-update-by-query-v-s-pulling-data-from-db-via-sql/225308 "2020-03-26T23:13:30Z")
I have created 2 logstash pipelines to index docs in Index A and Index B. I am pulling data from 2 tables in Oracle DB and indexing it in Index A & B respectively. The requirement is to update all docs in Index B for a … --- ## [Batch input data and send parallel http calls](https://discuss.elastic.co/t/batch-input-data-and-send-parallel-http-calls/225315)
**Author:** [@mikes](https://discuss.elastic.co/u/mikes)\ **Replies:** 0\ **Last updated:** [March 27, 2020, 12:21am UTC](https://discuss.elastic.co/t/batch-input-data-and-send-parallel-http-calls/225315 "2020-03-27T00:21:49Z")
Hello, I am trying to do the following: Take input on tcp and send it to a http endpoint. I need to send batches of events to the http endpoint. I am trying to use the Aggregate filter to batch events and then sending … --- ## [Will variables in ruby filter break with multiple workers?](https://discuss.elastic.co/t/will-variables-in-ruby-filter-break-with-multiple-workers/225285)
**Author:** [@seanziee](https://discuss.elastic.co/u/seanziee)\ **Replies:** 2\ **Last updated:** [March 26, 2020, 10:19pm UTC](https://discuss.elastic.co/t/will-variables-in-ruby-filter-break-with-multiple-workers/225285 "2020-03-26T22:19:14Z")
I'm using the ruby filter to clean up some data and during that cleanup, I create temporary variables and I have a few question on how variables may interfere with each other in a pipeline Case) If I have multiple work… --- ## [Hex Character removal using logstash](https://discuss.elastic.co/t/hex-character-removal-using-logstash/225278)
**Author:** [@waqas\_rasheed](https://discuss.elastic.co/u/waqas_rasheed)\ **Replies:** 5\ **Last updated:** [March 26, 2020, 9:13pm UTC](https://discuss.elastic.co/t/hex-character-removal-using-logstash/225278 "2020-03-26T21:13:53Z")
Hi, I am transferring NGINX logs from filebeat to logstash and than to Kibana but i am not getting desired output as response\_body contains hex characters and i don't know how to remove them using logstash. Current re… --- ## [Logstash geoip filter not working properly for certain public IPs when pushing data to es](https://discuss.elastic.co/t/logstash-geoip-filter-not-working-properly-for-certain-public-ips-when-pushing-data-to-es/225279)
**Author:** [@arunporwal](https://discuss.elastic.co/u/arunporwal)\ **Replies:** 1\ **Last updated:** [March 26, 2020, 9:12pm UTC](https://discuss.elastic.co/t/logstash-geoip-filter-not-working-properly-for-certain-public-ips-when-pushing-data-to-es/225279 "2020-03-26T21:12:03Z")
I am pushing aws ELB logs to elasticsearch with using below config as: input { s3 { type =\> "elb-logs" bucket =\> "\" region =\> "us-east-1" sincedb\_path =\> "/var/… --- ## [Logstash not logging data](https://discuss.elastic.co/t/logstash-not-logging-data/225276)
**Author:** [@adwaitjoshi](https://discuss.elastic.co/u/adwaitjoshi)\ **Replies:** 0\ **Last updated:** [March 26, 2020, 7:54pm UTC](https://discuss.elastic.co/t/logstash-not-logging-data/225276 "2020-03-26T19:54:11Z")
I am having a weird error. I have a logstash pipeline that works from command line but when I set it up in pipelines.yml it wont work. And whats weird is that logstash is not logging anything in the /var/log/ folder. And… --- ## [Same uuid or other id type when field content is equal](https://discuss.elastic.co/t/same-uuid-or-other-id-type-when-field-content-is-equal/225086)
**Author:** [@ebarros29](https://discuss.elastic.co/u/ebarros29)\ **Replies:** 3\ **Last updated:** [March 26, 2020, 6:53pm UTC](https://discuss.elastic.co/t/same-uuid-or-other-id-type-when-field-content-is-equal/225086 "2020-03-26T18:53:09Z")
Hi, I'm beginner using Logstash and I have two messages on my Firewall logs that are important to me, follows below: Mar 20 2020 15:38:00 1.1.1.1 : %ASA-6-113039: Group \ User IP \ AnyConnec… --- ## [Is it possible to do nested calls of event fields?](https://discuss.elastic.co/t/is-it-possible-to-do-nested-calls-of-event-fields/225219)
**Author:** [@aidanoc15](https://discuss.elastic.co/u/aidanoc15)\ **Replies:** 1\ **Last updated:** [March 26, 2020, 5:44pm UTC](https://discuss.elastic.co/t/is-it-possible-to-do-nested-calls-of-event-fields/225219 "2020-03-26T17:44:04Z")
I'm not sure if I am really phrasing the issue I am having correctly, so let me show you an example of what I am trying to do and maybe you can tell me if this is possible in logstash. add\_field =\> { "\[variables\]\[compan… --- ## [Unable to execute logstash conf for Azure Eventhub](https://discuss.elastic.co/t/unable-to-execute-logstash-conf-for-azure-eventhub/225253)
**Author:** [@Nithani25](https://discuss.elastic.co/u/Nithani25)\ **Replies:** 0\ **Last updated:** [March 26, 2020, 5:30pm UTC](https://discuss.elastic.co/t/unable-to-execute-logstash-conf-for-azure-eventhub/225253 "2020-03-26T17:30:43Z")
Hi Team, My Company is now working towards the microservices in azure platform and i have been assigned to work on Azure Eventhub integration to my on prem ELK. I have completed required conf changes and updated the lo… --- ## [Logstash create index only first time after install](https://discuss.elastic.co/t/logstash-create-index-only-first-time-after-install/225246)
**Author:** [@DenisCiobanuAC](https://discuss.elastic.co/u/DenisCiobanuAC)\ **Replies:** 0\ **Last updated:** [March 26, 2020, 4:57pm UTC](https://discuss.elastic.co/t/logstash-create-index-only-first-time-after-install/225246 "2020-03-26T16:57:49Z")
I install logstash. It created new index first time, but when i update logstash.conf it does neither create another index nor update existent logs. Actually nothing happens, even if i restart logstash. I also set run --… --- ## [Logstash flat files integartion](https://discuss.elastic.co/t/logstash-flat-files-integartion/225233)
**Author:** [@Goryoun](https://discuss.elastic.co/u/Goryoun)\ **Replies:** 0\ **Last updated:** [March 26, 2020, 3:54pm UTC](https://discuss.elastic.co/t/logstash-flat-files-integartion/225233 "2020-03-26T15:54:22Z")
How logstash collects flat files like SMB? --- ## [Persistent queue question](https://discuss.elastic.co/t/persistent-queue-question/225230)
**Author:** [@Ian\_Boje](https://discuss.elastic.co/u/Ian_Boje)\ **Replies:** 0\ **Last updated:** [March 26, 2020, 3:46pm UTC](https://discuss.elastic.co/t/persistent-queue-question/225230 "2020-03-26T15:46:38Z")
Hi everybody, I think I'm having an issue similar to this post: https://discuss.elastic.co/t/persistent-queue-configuration-question/73408/11 What seems to happen is logstash performance will degrade the longer it's pr… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=382) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=384)