# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=317 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 318 --- ## [How to connect Django and Elasticsearch by using Logstash?](https://discuss.elastic.co/t/how-to-connect-django-and-elasticsearch-by-using-logstash/244645)
**Author:** [@Jun.8](https://discuss.elastic.co/u/Jun.8)\ **Replies:** 1\ **Last updated:** [August 12, 2020, 4:33am UTC](https://discuss.elastic.co/t/how-to-connect-django-and-elasticsearch-by-using-logstash/244645 "2020-08-12T04:33:16Z")
Hi, I'm trying to send data in Django sqlite DB to Elasticsearch by using Logstash. I made two tables in Django and they are in sqlite DB basically embedded in Django. And I want send these datas to elasticsearch by usi… --- ## [Query on getting the complete event data in output plugin](https://discuss.elastic.co/t/query-on-getting-the-complete-event-data-in-output-plugin/244608)
**Author:** [@bimanarajesh](https://discuss.elastic.co/u/bimanarajesh)\ **Replies:** 2\ **Last updated:** [August 12, 2020, 3:49am UTC](https://discuss.elastic.co/t/query-on-getting-the-complete-event-data-in-output-plugin/244608 "2020-08-12T03:49:46Z")
Hi, In the logstash output plugin, i would like to get the complete event data. Could you please advice on how we can achieve that,. Once the event is received, I would like to wrap it around another json object. This w… --- ## [Ruby - First few characters (substring)](https://discuss.elastic.co/t/ruby-first-few-characters-substring/244493)
**Author:** [@bevano](https://discuss.elastic.co/u/bevano)\ **Replies:** 4\ **Last updated:** [August 12, 2020, 2:09am UTC](https://discuss.elastic.co/t/ruby-first-few-characters-substring/244493 "2020-08-12T02:09:46Z")
Hey All, Trying to get the first 4 characters of a field that is obtained via XPATH. This is what i currently have, but it copies everything rather then the first 4 characters. Been stumped on it for the past couple ho… --- ## [Supported kafka version with logstash 7.5.1](https://discuss.elastic.co/t/supported-kafka-version-with-logstash-7-5-1/244632)
**Author:** [@satish92](https://discuss.elastic.co/u/satish92)\ **Replies:** 1\ **Last updated:** [August 11, 2020, 9:35pm UTC](https://discuss.elastic.co/t/supported-kafka-version-with-logstash-7-5-1/244632 "2020-08-11T21:35:45Z")
I upgraded the kafka from 0.10.0 to 2.5 and also logstash from 2.4 to 7.5.1 after upgrade logstash is sending empty messages to Kafka( i can see this in kafka UI) but I can see messages in kafka UI which is forwarded fro… --- ## [Logstash filtr strange behavior](https://discuss.elastic.co/t/logstash-filtr-strange-behavior/244228)
**Author:** [@masech](https://discuss.elastic.co/u/masech)\ **Replies:** 16\ **Last updated:** [August 11, 2020, 9:09pm UTC](https://discuss.elastic.co/t/logstash-filtr-strange-behavior/244228 "2020-08-11T21:09:49Z")
Hi guys. I noticed that the convert\_datatype option of the Dissect filter has a strange result. Pipeline: input { stdin { } } filter { # split IPv4 address into octets dissect { mapping =\> { … --- ## [Logstash sending same message 3 times](https://discuss.elastic.co/t/logstash-sending-same-message-3-times/244610)
**Author:** [@srbhklkrn](https://discuss.elastic.co/u/srbhklkrn)\ **Replies:** 0\ **Last updated:** [August 11, 2020, 5:45pm UTC](https://discuss.elastic.co/t/logstash-sending-same-message-3-times/244610 "2020-08-11T17:45:40Z")
We have Pivotal Cloud Foundry as a platform and we are sending logs to ELK. So our setup is like we have an F5 load balancer which sends logs to 3 Logstash servers and these 3 Logstash servers send logs to 3 Elastic serv… --- ## [Query metrics of VMS through API call via HTTP Poller](https://discuss.elastic.co/t/query-metrics-of-vms-through-api-call-via-http-poller/244569)
**Author:** [@Ayesha](https://discuss.elastic.co/u/Ayesha)\ **Replies:** 3\ **Last updated:** [August 11, 2020, 6:32pm UTC](https://discuss.elastic.co/t/query-metrics-of-vms-through-api-call-via-http-poller/244569 "2020-08-11T18:32:48Z")
Hi I want to grab specific metrics of VMs by API calls Currently I am using ///suite-api/api/resources?resourceKind=VirtualMachine to call all VMs and their resource IDs in postman Next I am using below to get metri… --- ## [Create temp metadata field using regex in message field and parse json](https://discuss.elastic.co/t/create-temp-metadata-field-using-regex-in-message-field-and-parse-json/244594)
**Author:** [@srbhklkrn](https://discuss.elastic.co/u/srbhklkrn)\ **Replies:** 6\ **Last updated:** [August 11, 2020, 6:15pm UTC](https://discuss.elastic.co/t/create-temp-metadata-field-using-regex-in-message-field-and-parse-json/244594 "2020-08-11T18:15:12Z")
I need to extract the JSON object from the message field filter { grok { match =\> { "message" =\> "(?\<\[@metadata\]\[tempjson\]\>{.+})" } } json { source =\> "\[@metadata\]\[tempjson\]" } } Above… --- ## [An exception caught while applying mutate filter - Could not set field 'log' on object '' to value '...'](https://discuss.elastic.co/t/an-exception-caught-while-applying-mutate-filter-could-not-set-field-log-on-object-to-value/244583)
**Author:** [@Yuval\_Khalifa](https://discuss.elastic.co/u/Yuval_Khalifa)\ **Replies:** 5\ **Last updated:** [August 11, 2020, 4:02pm UTC](https://discuss.elastic.co/t/an-exception-caught-while-applying-mutate-filter-could-not-set-field-log-on-object-to-value/244583 "2020-08-11T16:02:03Z")
Hi, I'm getting the following error message in Logstash logs: \[2020-08-11T15:07:11,248\]\[WARN \]\[logstash.filters.mutate \]\[main\]\[51aae80b265c32978f64e0aace0ece634a29ae2a08ef228095111afd72bfdd0b\] Exception caught while a… --- ## [Beats can not connect to Logstash](https://discuss.elastic.co/t/beats-can-not-connect-to-logstash/244584)
**Author:** [@Ben\_said](https://discuss.elastic.co/u/Ben_said)\ **Replies:** 1\ **Last updated:** [August 11, 2020, 3:52pm UTC](https://discuss.elastic.co/t/beats-can-not-connect-to-logstash/244584 "2020-08-11T15:52:33Z")
\[2020-08-12T12:53:29,146\]\[INFO \]\[org.logstash.beats.BeatsHandler\]\[main\]\[2ab0e765b84268c7156e67dc3b10bd35b876b27ad91871edfa9d3b1669f4fea7\] \[local: 10.100.80.40:5044, remote: 10.100.80.40:39494\] Handling exception: org.log… --- ## [JDBC Driver to be used to crawl the data from IBM db2 database](https://discuss.elastic.co/t/jdbc-driver-to-be-used-to-crawl-the-data-from-ibm-db2-database/243338)
**Author:** [@Mohan\_vel](https://discuss.elastic.co/u/Mohan_vel)\ **Replies:** 1\ **Last updated:** [August 11, 2020, 2:45pm UTC](https://discuss.elastic.co/t/jdbc-driver-to-be-used-to-crawl-the-data-from-ibm-db2-database/243338 "2020-08-11T14:45:26Z")
Hi All, I have created a logstash configuration file to crawl the data from IBM Db2. I am getting an issue that JDBC driver libirary is not loaded. Here i am using logstash version 7.7.1. Configuration file in… --- ## [Using a codec with the BigQuery output plugin](https://discuss.elastic.co/t/using-a-codec-with-the-bigquery-output-plugin/241510)
**Author:** [@James\_Sherwood\_Jones](https://discuss.elastic.co/u/James_Sherwood_Jones)\ **Replies:** 1\ **Last updated:** [August 11, 2020, 11:15am UTC](https://discuss.elastic.co/t/using-a-codec-with-the-bigquery-output-plugin/241510 "2020-08-11T11:15:56Z")
I've set a codec on the BigQuery output plugin in my pipeline: google\_bigquery { codec =\> "my-custom-codec" ... } The codec itself works great with an output like "stdout" but I've spent a considerable amount of ti… --- ## [Aggregate filter cache](https://discuss.elastic.co/t/aggregate-filter-cache/244432)
**Author:** [@kvtang](https://discuss.elastic.co/u/kvtang)\ **Replies:** 2\ **Last updated:** [August 11, 2020, 6:38am UTC](https://discuss.elastic.co/t/aggregate-filter-cache/244432 "2020-08-11T06:38:00Z")
Hi guys, If I'm using aggregate filter plugin, and after a start event, my collector node restarted for whatever reason. When my end event comes, is it able to find my start event? Is there any cache storing task\_id? T… --- ## [Logstash in the cloud](https://discuss.elastic.co/t/logstash-in-the-cloud/244492)
**Author:** [@mykael](https://discuss.elastic.co/u/mykael)\ **Replies:** 1\ **Last updated:** [August 11, 2020, 6:25am UTC](https://discuss.elastic.co/t/logstash-in-the-cloud/244492 "2020-08-11T06:25:24Z")
Am I right in thinking that if I want to use a customized logstash to ingest data into elastic search running under elastic cloud, then I need to get a separate cloud machine to run the logstash instance and connect it, … --- ## [What is the tools/features in Logstash which are related to Integrations part](https://discuss.elastic.co/t/what-is-the-tools-features-in-logstash-which-are-related-to-integrations-part/244487)
**Author:** [@Ismaeel\_Enjreny](https://discuss.elastic.co/u/Ismaeel_Enjreny)\ **Replies:** 0\ **Last updated:** [August 11, 2020, 5:28am UTC](https://discuss.elastic.co/t/what-is-the-tools-features-in-logstash-which-are-related-to-integrations-part/244487 "2020-08-11T05:28:46Z")
Hello dears, What is the tools/features in Logstash which are related to Integrations part in enterprise license Based on the following link: https://www.elastic.co/subscriptions --- ## [Enable Logstash to Talk with Outside Server?](https://discuss.elastic.co/t/enable-logstash-to-talk-with-outside-server/244321)
**Author:** [@redapplesonly](https://discuss.elastic.co/u/redapplesonly)\ **Replies:** 2\ **Last updated:** [August 10, 2020, 10:01pm UTC](https://discuss.elastic.co/t/enable-logstash-to-talk-with-outside-server/244321 "2020-08-10T22:01:42Z")
Hi Logstash, While processing data records, I need my Logstash instance (ver 7.7.1) to send a copy of the entire data record to an outside server. The server does some processing, then sends back a one-string response.… --- ## [Ingest rate](https://discuss.elastic.co/t/ingest-rate/244465)
**Author:** [@dietrich](https://discuss.elastic.co/u/dietrich)\ **Replies:** 0\ **Last updated:** [August 10, 2020, 9:19pm UTC](https://discuss.elastic.co/t/ingest-rate/244465 "2020-08-10T21:19:01Z")
How do I calculate the number of documents ingested per day? Thread 157245 suggested it's possible to get the number of index requests per second, but the thread did not give any API call(s) for getting this number, and… --- ## [Blank message field](https://discuss.elastic.co/t/blank-message-field/244441)
**Author:** [@matheus.santoro](https://discuss.elastic.co/u/matheus.santoro)\ **Replies:** 5\ **Last updated:** [August 10, 2020, 9:17pm UTC](https://discuss.elastic.co/t/blank-message-field/244441 "2020-08-10T21:17:23Z")
Trying to create a simple, non-filtered pipeline, but Kibana shows the message field as blank Pipeline conf is input { tcp { port =\> 9600 codec =\> json mode =\> server } } output { elasticsearch { … --- ## [Replace field value when equals "-"](https://discuss.elastic.co/t/replace-field-value-when-equals/244453)
**Author:** [@gbasic](https://discuss.elastic.co/u/gbasic)\ **Replies:** 1\ **Last updated:** [August 10, 2020, 9:00pm UTC](https://discuss.elastic.co/t/replace-field-value-when-equals/244453 "2020-08-10T21:00:33Z")
Hi I would need to replace a field value (WindDirection,Temp,CityName,WindSpeed,Humidity) to some other value when it equals "-" This is my data: { "@timestamp" =\> 2020-08-10T18:54:08.055Z, "Europe" =\> { … --- ## [Number\_format\_exception: Elasticsearch Query Template - check if value of int field present inside array\[int\] field](https://discuss.elastic.co/t/number-format-exception-elasticsearch-query-template-check-if-value-of-int-field-present-inside-array-int-field/244220)
**Author:** [@tinhn](https://discuss.elastic.co/u/tinhn)\ **Replies:** 2\ **Last updated:** [August 10, 2020, 6:07pm UTC](https://discuss.elastic.co/t/number-format-exception-elasticsearch-query-template-check-if-value-of-int-field-present-inside-array-int-field/244220 "2020-08-10T18:07:44Z")
Hello, I'm trying to write a logstash filter that reads from an elasticsearch index using a template. I want to check if an integer matches a value stored in an array of ints. // incoming record { "id": 2 } // query… --- ## [How to merge two different data source into single index](https://discuss.elastic.co/t/how-to-merge-two-different-data-source-into-single-index/244442)
**Author:** [@Hassaan\_mustafa](https://discuss.elastic.co/u/Hassaan_mustafa)\ **Replies:** 0\ **Last updated:** [August 10, 2020, 4:39pm UTC](https://discuss.elastic.co/t/how-to-merge-two-different-data-source-into-single-index/244442 "2020-08-10T16:39:44Z")
I want to Update certain field of index data coming for metricbeat but the fields that need to updated its data is in different data source its in radius how to achieve this --- ## [Whats the difference as well as the preferred way to?](https://discuss.elastic.co/t/whats-the-difference-as-well-as-the-preferred-way-to/244415)
**Author:** [@kopacko](https://discuss.elastic.co/u/kopacko)\ **Replies:** 2\ **Last updated:** [August 10, 2020, 2:55pm UTC](https://discuss.elastic.co/t/whats-the-difference-as-well-as-the-preferred-way-to/244415 "2020-08-10T14:55:36Z")
I am trying to update my Logstash configs to be the best they can be. If I am looking for a string within a field, which way is the "best" way, in terms of performance? Option #1 if \[message\] =~ /(?i)(QNAME)/ { Optio… --- ## [Help me to refine this query](https://discuss.elastic.co/t/help-me-to-refine-this-query/244023)
**Author:** [@pk.241011](https://discuss.elastic.co/u/pk.241011)\ **Replies:** 4\ **Last updated:** [August 10, 2020, 1:37pm UTC](https://discuss.elastic.co/t/help-me-to-refine-this-query/244023 "2020-08-10T13:37:21Z")
Context: Logstash Elasticsearch Input plugin does not maintain a state like JDBC input plugin. However it does provide scheduling support. So I decided to pull data daily from the previous day records, e.g. from say 4t… --- ## [Logstash File input filter, ingest twice same file name](https://discuss.elastic.co/t/logstash-file-input-filter-ingest-twice-same-file-name/244385)
**Author:** [@nahiko2000](https://discuss.elastic.co/u/nahiko2000)\ **Replies:** 1\ **Last updated:** [August 10, 2020, 1:27pm UTC](https://discuss.elastic.co/t/logstash-file-input-filter-ingest-twice-same-file-name/244385 "2020-08-10T13:27:36Z")
Hi! Using Logstash File Input Filter, I will ingest text files (actually they will be .gz files) that are "finished" (I mean, they will not be written anymore) When ingest gets done, file must be automatically deleted … --- ## [Logstash move subfield to top field](https://discuss.elastic.co/t/logstash-move-subfield-to-top-field/244309)
**Author:** [@gbasic](https://discuss.elastic.co/u/gbasic)\ **Replies:** 6\ **Last updated:** [August 10, 2020, 11:52am UTC](https://discuss.elastic.co/t/logstash-move-subfield-to-top-field/244309 "2020-08-10T11:52:09Z")
"\_score" : 1.0, "\_source" : { "Europe" : { "DateTime" : { "Date" : "08.08.2020", "Time" : "15" }, "City" : \[ { … --- ## [Logstash to elasticsearch](https://discuss.elastic.co/t/logstash-to-elasticsearch/244306)
**Author:** [@headtea](https://discuss.elastic.co/u/headtea)\ **Replies:** 6\ **Last updated:** [August 10, 2020, 9:52am UTC](https://discuss.elastic.co/t/logstash-to-elasticsearch/244306 "2020-08-10T09:52:47Z")
I think logstash is having trouble sending data to elasticsearch. To secure beats communications, I've set elasticsearch to use https. I think because of that logstash is having some trouble sending data to elasticsearc… --- ## [X-Path Wildcard](https://discuss.elastic.co/t/x-path-wildcard/244352)
**Author:** [@bevano](https://discuss.elastic.co/u/bevano)\ **Replies:** 3\ **Last updated:** [August 10, 2020, 8:48am UTC](https://discuss.elastic.co/t/x-path-wildcard/244352 "2020-08-10T08:48:48Z")
Hey, Was wondering whether someone could help me with the following xpaths since I dont have much experience. I am trying to get the value of /eCloud1/CloudMsg/\*/name() but this field can be 16 different values. I don't… --- ## [Logstash Error](https://discuss.elastic.co/t/logstash-error/244343)
**Author:** [@Amantha](https://discuss.elastic.co/u/Amantha)\ **Replies:** 1\ **Last updated:** [August 10, 2020, 6:26am UTC](https://discuss.elastic.co/t/logstash-error/244343 "2020-08-10T06:26:54Z")
I run bin/logstash -e 'input { stdin { } } output { stdout {} }' command in linux terminal. But it give the following error. OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and … --- ## [Convert epoch time columns to date in lostash config file](https://discuss.elastic.co/t/convert-epoch-time-columns-to-date-in-lostash-config-file/244122)
**Author:** [@Bucky101](https://discuss.elastic.co/u/Bucky101)\ **Replies:** 2\ **Last updated:** [August 10, 2020, 4:35am UTC](https://discuss.elastic.co/t/convert-epoch-time-columns-to-date-in-lostash-config-file/244122 "2020-08-10T04:35:26Z")
Hey as per below, i have am polling a http and passing through the csv filter pulling date which works fine. However I need to convert the epoch timestamp to readable date and time. I have tried a few different ways wi… --- ## [Dockerizing a working ELK project, 'Dead ES instance'](https://discuss.elastic.co/t/dockerizing-a-working-elk-project-dead-es-instance/243639)
**Author:** [@kl29](https://discuss.elastic.co/u/kl29)\ **Replies:** 6\ **Last updated:** [August 9, 2020, 10:18pm UTC](https://discuss.elastic.co/t/dockerizing-a-working-elk-project-dead-es-instance/243639 "2020-08-09T22:18:09Z")
My setup works without Dockerizing. Now, making a Dockerized project, logstash cannot connect to the ES instance. I set http.host: "0.0.0.0" in logstash.yml, and my pipeline.conf is: input { exec { command =\> "loc… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=316) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=318)