# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=47 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 48 --- ## [Logstash with ouput clickhouse plugin more than 80% logs are missing](https://discuss.elastic.co/t/logstash-with-ouput-clickhouse-plugin-more-than-80-logs-are-missing/349724)
**Author:** [@Anandh\_Kumar1](https://discuss.elastic.co/u/Anandh_Kumar1)\ **Replies:** 1\ **Last updated:** [December 20, 2023, 2:58pm UTC](https://discuss.elastic.co/t/logstash-with-ouput-clickhouse-plugin-more-than-80-logs-are-missing/349724 "2023-12-20T14:58:11Z")
I am using logstash version 7.17.15 in production environment, In that i am using the ouput plugin is clickhouse and the version is 20.8.3.18. Using the filebeat I am moving the logs into logstash which is there is remo… --- ## [Installation of logstash-output-opensearch in an airgap environment(no internet access on server)](https://discuss.elastic.co/t/installation-of-logstash-output-opensearch-in-an-airgap-environment-no-internet-access-on-server/349717)
**Author:** [@kushak\_kain](https://discuss.elastic.co/u/kushak_kain)\ **Replies:** 2\ **Last updated:** [December 20, 2023, 12:34pm UTC](https://discuss.elastic.co/t/installation-of-logstash-output-opensearch-in-an-airgap-environment-no-internet-access-on-server/349717 "2023-12-20T12:34:06Z")
Hello Experts, I need your assistance in installing logstash-output-opensearch in an airgap environment(our servers can not connect to internet) We are receiving the following error while executing the command : ./log… --- ## [Sql query returns nothing , but output file udpated](https://discuss.elastic.co/t/sql-query-returns-nothing-but-output-file-udpated/349622)
**Author:** [@gayatri\_SN](https://discuss.elastic.co/u/gayatri_SN)\ **Replies:** 3\ **Last updated:** [December 20, 2023, 5:28am UTC](https://discuss.elastic.co/t/sql-query-returns-nothing-but-output-file-udpated/349622 "2023-12-20T05:28:21Z")
Hi, I'm using jdbc input streaming filter to get the data from query. but in some cases query returns empty or null value. \< last\_run\_metadata\_path =\> \ \ **Author:** [@CodeMonky](https://discuss.elastic.co/u/CodeMonky)\ **Replies:** 4\ **Last updated:** [December 19, 2023, 10:10pm UTC](https://discuss.elastic.co/t/custom-tcp-integration-with-tls/349679 "2023-12-19T22:10:42Z") Good day all. I hope this is a simple question, but I've not been able to find any info. For the custom TCP integration that can be used when there's not an Elastic provided integration to use for data ingestion: does i… --- ## [What happens on shutdown if queue.drain is true but output is unavailable?](https://discuss.elastic.co/t/what-happens-on-shutdown-if-queue-drain-is-true-but-output-is-unavailable/349680)
**Author:** [@noobiewan](https://discuss.elastic.co/u/noobiewan)\ **Replies:** 0\ **Last updated:** [December 19, 2023, 7:29pm UTC](https://discuss.elastic.co/t/what-happens-on-shutdown-if-queue-drain-is-true-but-output-is-unavailable/349680 "2023-12-19T19:29:44Z")
Hi there, I'm currently running Logstash on Kubernetes and have configured a pipeline with a Persistent Queue and the setting queue.drain: true. In this setup, I'm curious about Logstash's behavior when it receives a SI… --- ## [IF Regex not working](https://discuss.elastic.co/t/if-regex-not-working/349648)
**Author:** [@marcowiskhy](https://discuss.elastic.co/u/marcowiskhy)\ **Replies:** 5\ **Last updated:** [December 19, 2023, 5:51pm UTC](https://discuss.elastic.co/t/if-regex-not-working/349648 "2023-12-19T17:51:51Z")
Hey guys, In my pipeline I use a dictionary to enrich internal IPs and, to capture, I use the following regex: if \[source\] =~ "^10\\." or \[source\] =~ "^127\\.0\\." or \[source\] =~ "^192\\.168\\." or \[source\] =~ "^172\\.(1\[678… --- ## [About Logstash configuration using ssl](https://discuss.elastic.co/t/about-logstash-configuration-using-ssl/349628)
**Author:** [@Hamada](https://discuss.elastic.co/u/Hamada)\ **Replies:** 1\ **Last updated:** [December 19, 2023, 12:59pm UTC](https://discuss.elastic.co/t/about-logstash-configuration-using-ssl/349628 "2023-12-19T12:59:12Z")
I have a question regarding Logstash configuration. In order to connect to Elasticsearch from Logstash, enter the following into the Logstash configuration and execute. == output{ elasticsearch { hosts =\> \["https://… --- ## [I configure rsyslog for my linux server now I want to send logs to LOGSTASH. How could I achieve that](https://discuss.elastic.co/t/i-configure-rsyslog-for-my-linux-server-now-i-want-to-send-logs-to-logstash-how-could-i-achieve-that/349566)
**Author:** [@Subrato1](https://discuss.elastic.co/u/Subrato1)\ **Replies:** 4\ **Last updated:** [December 19, 2023, 12:52pm UTC](https://discuss.elastic.co/t/i-configure-rsyslog-for-my-linux-server-now-i-want-to-send-logs-to-logstash-how-could-i-achieve-that/349566 "2023-12-19T12:52:08Z")
Below is my rsyslog conf. My audit logs are generating in syslogs only. --- ## [Logstash split log base on space and =](https://discuss.elastic.co/t/logstash-split-log-base-on-space-and/348526)
**Author:** [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Replies:** 2\ **Last updated:** [December 19, 2023, 11:51am UTC](https://discuss.elastic.co/t/logstash-split-log-base-on-space-and/348526 "2023-12-19T11:51:39Z")
I want to separate below log in Logstash, I know that we can do it by grok filter, but is there any way to do it without grok? Log: date=2023-12-04 time=11:26:01 my\_id=5646875 dir="D" type=ML severety=info mtype="my lo… --- ## [Dissect in logstash and tabs](https://discuss.elastic.co/t/dissect-in-logstash-and-tabs/349595)
**Author:** [@astateofmind](https://discuss.elastic.co/u/astateofmind)\ **Replies:** 3\ **Last updated:** [December 19, 2023, 2:29am UTC](https://discuss.elastic.co/t/dissect-in-logstash-and-tabs/349595 "2023-12-19T02:29:56Z")
Trying to use dissect to add log.level field to some beats. Using filebeat to send the data and some logs have their fields separated by tabs instead of spaces. The logs with space work ok with this filter: "%{} %{log… --- ## [Elasticsearch Input on Logstash](https://discuss.elastic.co/t/elasticsearch-input-on-logstash/349609)
**Author:** [@Leonadius](https://discuss.elastic.co/u/Leonadius)\ **Replies:** 0\ **Last updated:** [December 19, 2023, 3:04am UTC](https://discuss.elastic.co/t/elasticsearch-input-on-logstash/349609 "2023-12-19T03:04:23Z")
Dear Elastic Team, I have a case where i need to sync all of the documents from 1 index to another elastic cluster with near real-time. I'm thinking using logstash elasticsearch input to read all of the documents conti… --- ## [Logstash not connecting to Elasticsearch - using Docker-Compose](https://discuss.elastic.co/t/logstash-not-connecting-to-elasticsearch-using-docker-compose/349461)
**Author:** [@zewcro](https://discuss.elastic.co/u/zewcro)\ **Replies:** 12\ **Last updated:** [December 18, 2023, 6:19pm UTC](https://discuss.elastic.co/t/logstash-not-connecting-to-elasticsearch-using-docker-compose/349461 "2023-12-18T18:19:57Z")
Hello, I'm trying to create indexes in elasticsearch from a postgresql database. So I set up docker compose: version: '3.8' services: postgres: image: postgres:latest volumes: - C:\\Users\\theor\\desktop… --- ## [Logstash error Cpu.cfs\_period\_us cannot be found](https://discuss.elastic.co/t/logstash-error-cpu-cfs-period-us-cannot-be-found/349515)
**Author:** [@Lena\_Yoon](https://discuss.elastic.co/u/Lena_Yoon)\ **Replies:** 9\ **Last updated:** [December 18, 2023, 11:43am UTC](https://discuss.elastic.co/t/logstash-error-cpu-cfs-period-us-cannot-be-found/349515 "2023-12-18T11:43:11Z")
Hello, I have been working with Logstash this week but stuck with below error. The error occurs when retrieving data from Oracle DB using the JDBC input plugin, filtering it in the pipeline, and despite the index being… --- ## [One logstash instance per kubernetes cluster](https://discuss.elastic.co/t/one-logstash-instance-per-kubernetes-cluster/349546)
**Author:** [@codedoings](https://discuss.elastic.co/u/codedoings)\ **Replies:** 0\ **Last updated:** [December 18, 2023, 10:11am UTC](https://discuss.elastic.co/t/one-logstash-instance-per-kubernetes-cluster/349546 "2023-12-18T10:11:01Z")
Hi, What would be the best approach for configuring logstash in an environment where: Elasticsearch and Kibana are running in their own kubernetes cluster (deployed with ECK). Elasticsearch and Kibana instance is shar… --- ## [Logstash error(no data ) while ingesting CSV data with ELK version 8.9.2](https://discuss.elastic.co/t/logstash-error-no-data-while-ingesting-csv-data-with-elk-version-8-9-2/349510)
**Author:** [@raemonx](https://discuss.elastic.co/u/raemonx)\ **Replies:** 2\ **Last updated:** [December 17, 2023, 10:08pm UTC](https://discuss.elastic.co/t/logstash-error-no-data-while-ingesting-csv-data-with-elk-version-8-9-2/349510 "2023-12-17T22:08:44Z")
I was facing an issue while ingesting an csv file called housing\_price\_data.csv using logstash. I was using ELK with docker. I was using ELK version 8.11 I did not want to add any security so there is no SSL, passwords o… --- ## [Logstash ran as service won't read logs only when ran through the command line](https://discuss.elastic.co/t/logstash-ran-as-service-wont-read-logs-only-when-ran-through-the-command-line/349403)
**Author:** [@ELI\_MA](https://discuss.elastic.co/u/ELI_MA)\ **Replies:** 14\ **Last updated:** [December 17, 2023, 5:10am UTC](https://discuss.elastic.co/t/logstash-ran-as-service-wont-read-logs-only-when-ran-through-the-command-line/349403 "2023-12-17T05:10:27Z")
Hi, I’m running Logstash on SUSE Linux where I’ve installed the RPM package for compatibility. Currently, When I start logstash as a service sudo systemctl stop logstash.service and check service status it seems to be r… --- ## [Some fields are missing after rename.](https://discuss.elastic.co/t/some-fields-are-missing-after-rename/349459)
**Author:** [@JHub-Wei](https://discuss.elastic.co/u/JHub-Wei)\ **Replies:** 11\ **Last updated:** [December 16, 2023, 2:22am UTC](https://discuss.elastic.co/t/some-fields-are-missing-after-rename/349459 "2023-12-16T02:22:57Z")
After logstash-oss is upgraded from 7.6.0 to 7.12.1, some fields are lost after parsing the nested JSON data of Kafka. Kafka JSON example data: {"timestamp":1702630468791,"region":"cn-north-3","eventId":"QER\_INFO","args… --- ## [What could be the cause of error ""](https://discuss.elastic.co/t/what-could-be-the-cause-of-error/349436)
**Author:** [@Chen\_Wei](https://discuss.elastic.co/u/Chen_Wei)\ **Replies:** 1\ **Last updated:** [December 15, 2023, 7:07pm UTC](https://discuss.elastic.co/t/what-could-be-the-cause-of-error/349436 "2023-12-15T19:07:29Z")
I recently met a error about timestamp, the error appears after the logstash pipeline start and work for a while. I could not reproduce it. But I wonder where the error could happen. Does it happen in the input plugin?… --- ## [Drop complete row or message](https://discuss.elastic.co/t/drop-complete-row-or-message/349415)
**Author:** [@kundan](https://discuss.elastic.co/u/kundan)\ **Replies:** 1\ **Last updated:** [December 15, 2023, 7:08pm UTC](https://discuss.elastic.co/t/drop-complete-row-or-message/349415 "2023-12-15T19:08:38Z")
Hi, I want to drop full row based on one of the field. I am using following in filter. filter { grok { match =\> {"message" =\> \["%{IP:ip} %{SPACE}\\{user:%{USERNAME:UserId}\\}"\]} } date… --- ## [Logstash 8.10.3 ERROR Badly formatted index, after interpolation still contains placeholder](https://discuss.elastic.co/t/logstash-8-10-3-error-badly-formatted-index-after-interpolation-still-contains-placeholder/349377)
**Author:** [@efrainMZ](https://discuss.elastic.co/u/efrainMZ)\ **Replies:** 5\ **Last updated:** [December 15, 2023, 4:33pm UTC](https://discuss.elastic.co/t/logstash-8-10-3-error-badly-formatted-index-after-interpolation-still-contains-placeholder/349377 "2023-12-15T16:33:43Z")
good day! I am trying to extract data from redis using logstash, the data comes from an apm version 8.10.3 but I receive a warining that does not allow me to see the data in kibana. The log I receive is the following: … --- ## [SystemCallError, :message=\>"Unknown error (SystemCallError)](https://discuss.elastic.co/t/systemcallerror-message-unknown-error-systemcallerror/347909)
**Author:** [@berta](https://discuss.elastic.co/u/berta)\ **Replies:** 5\ **Last updated:** [December 15, 2023, 3:49pm UTC](https://discuss.elastic.co/t/systemcallerror-message-unknown-error-systemcallerror/347909 "2023-12-15T15:49:09Z")
Hello, We have a server RHEL7 running with logstash, sending the logs to an opensearch with elasticsearch engine, and every few seconds the files are not send with this error: 2023-09-28T06:33:37,702\]\[ERROR\]\[filewatch.… --- ## [File sharing between multiple logstash instance](https://discuss.elastic.co/t/file-sharing-between-multiple-logstash-instance/349445)
**Author:** [@kishan\_vadalia](https://discuss.elastic.co/u/kishan_vadalia)\ **Replies:** 1\ **Last updated:** [December 15, 2023, 2:49pm UTC](https://discuss.elastic.co/t/file-sharing-between-multiple-logstash-instance/349445 "2023-12-15T14:49:47Z")
I have 3 logstash instance running on same machine and putting data to same ES index. all 3 are reading file input from same location (/etc/logstash/conf.d). If there are 500 files in that location than on ES index numbe… --- ## [I Want to remove the duplicate events inside Logstash filter how could I do that? I mention the events below please have a look and suggest](https://discuss.elastic.co/t/i-want-to-remove-the-duplicate-events-inside-logstash-filter-how-could-i-do-that-i-mention-the-events-below-please-have-a-look-and-suggest/349175)
**Author:** [@Subrato1](https://discuss.elastic.co/u/Subrato1)\ **Replies:** 6\ **Last updated:** [December 15, 2023, 1:35pm UTC](https://discuss.elastic.co/t/i-want-to-remove-the-duplicate-events-inside-logstash-filter-how-could-i-do-that-i-mention-the-events-below-please-have-a-look-and-suggest/349175 "2023-12-15T13:35:00Z")
{ "date" =\> 2023-12-12T00:00:00.000Z, "category" =\> "AUTH", "username" =\> "cassandra", "event\_time" =\> "ab390a7b-98e7-11ee-af20-4b75abbb029d", "node" =\> "172.31.57.239",… --- ## [Logstash runs on the linux container and extremely slow](https://discuss.elastic.co/t/logstash-runs-on-the-linux-container-and-extremely-slow/349249)
**Author:** [@ranjini](https://discuss.elastic.co/u/ranjini)\ **Replies:** 3\ **Last updated:** [December 14, 2023, 6:22pm UTC](https://discuss.elastic.co/t/logstash-runs-on-the-linux-container-and-extremely-slow/349249 "2023-12-14T18:22:56Z")
logstash runs on linux container. Below is my configuration. It is very slow. Sharing my configuration for reference. This is my service configuration. file { path =\> "/common/logs/\*\*/\*.log" start\_posit… --- ## [Not able to read the data from external json file in logstash config](https://discuss.elastic.co/t/not-able-to-read-the-data-from-external-json-file-in-logstash-config/349257)
**Author:** [@subash\_k](https://discuss.elastic.co/u/subash_k)\ **Replies:** 10\ **Last updated:** [December 14, 2023, 1:26pm UTC](https://discuss.elastic.co/t/not-able-to-read-the-data-from-external-json-file-in-logstash-config/349257 "2023-12-14T13:26:32Z")
I'm trying to search the host value from current event and looking for same value in json file. If Json block has the host value I'm just converting the block into struct value and inserting as a new column in index. ou… --- ## [The Persistent Volume Claim (PVC) persists even after scaling in the Logstash deployment](https://discuss.elastic.co/t/the-persistent-volume-claim-pvc-persists-even-after-scaling-in-the-logstash-deployment/349323)
**Author:** [@Vignesh\_M](https://discuss.elastic.co/u/Vignesh_M)\ **Replies:** 0\ **Last updated:** [December 14, 2023, 5:57am UTC](https://discuss.elastic.co/t/the-persistent-volume-claim-pvc-persists-even-after-scaling-in-the-logstash-deployment/349323 "2023-12-14T05:57:32Z")
Hi All, We are using the elastic/logstash Helm chart to deploy Logstash (StatefulSet) with persistent volume enabled in one of our Kubernetes clusters. While attempting to downscale the Logstash pod count, we observed t… --- ## [Cloudwatch input plugin configuration details for fetching AWS/ECS metrics](https://discuss.elastic.co/t/cloudwatch-input-plugin-configuration-details-for-fetching-aws-ecs-metrics/349321)
**Author:** [@mittal\_rawal1](https://discuss.elastic.co/u/mittal_rawal1)\ **Replies:** 0\ **Last updated:** [December 14, 2023, 5:39am UTC](https://discuss.elastic.co/t/cloudwatch-input-plugin-configuration-details-for-fetching-aws-ecs-metrics/349321 "2023-12-14T05:39:30Z")
input { cloudwatch { namespace =\> "AWS/ECS" period =\> 6000 interval =\> 6000000 metrics =\> \["Average", "Minimum", "Maximum", "Sum", "Sample Count","CPUUtilization"\] filters =\> { "ClusterName" =\> "microservices" "S… --- ## [Files too big for Sentinel plugin](https://discuss.elastic.co/t/files-too-big-for-sentinel-plugin/348530)
**Author:** [@Joseph\_Leiber](https://discuss.elastic.co/u/Joseph_Leiber)\ **Replies:** 1\ **Last updated:** [December 13, 2023, 11:41pm UTC](https://discuss.elastic.co/t/files-too-big-for-sentinel-plugin/348530 "2023-12-13T23:41:01Z")
Hi Logstash Experts - This is my first time dealing with Logstash, so I'm not quite sure why the logs are being formatted like this, whether this is expected/normal, or how to handle them. I'm hitting an issue with log… --- ## [How to know the Acknowledge of message on logstash](https://discuss.elastic.co/t/how-to-know-the-acknowledge-of-message-on-logstash/349225)
**Author:** [@pradeep.kumar](https://discuss.elastic.co/u/pradeep.kumar)\ **Replies:** 2\ **Last updated:** [December 13, 2023, 8:06pm UTC](https://discuss.elastic.co/t/how-to-know-the-acknowledge-of-message-on-logstash/349225 "2023-12-13T20:06:50Z")
Hi Team, we want to know whether the messages in the queue are acknowledged or not. We use a persistent queue on the logstash. Please let us know is there any way to get the status of it. --- ## [Logstash regex expression in conf xpath](https://discuss.elastic.co/t/logstash-regex-expression-in-conf-xpath/349252)
**Author:** [@d.silwon](https://discuss.elastic.co/u/d.silwon)\ **Replies:** 2\ **Last updated:** [December 13, 2023, 5:48pm UTC](https://discuss.elastic.co/t/logstash-regex-expression-in-conf-xpath/349252 "2023-12-13T17:48:31Z")
Dears, Can we use regex expression in logstash configuration in case of filter and xpath? There is right now such to conditions: ... filter { if "xmlapps" in \[tags\] { xml { source =\> "message" store\_xml =\> … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=46) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=48)