# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=85 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 86 --- ## [Overwrite data VS data duplication](https://discuss.elastic.co/t/overwrite-data-vs-data-duplication/328143)
**Author:** [@m3bgwad](https://discuss.elastic.co/u/m3bgwad)\ **Replies:** 0\ **Last updated:** [March 21, 2023, 8:53am UTC](https://discuss.elastic.co/t/overwrite-data-vs-data-duplication/328143 "2023-03-21T08:53:18Z")
Hello, every all, What is the goal of handling the data duplication using a fingerprint filter, In my opinion, this is overwritten data, not preventing the duplication. Let's say the overwrite is removing the oldest an… --- ## [What are logstash-plain-YYYY-MM-DD.log.gz and logstash-deprecation-YYYY-MM-DD.log.gz?](https://discuss.elastic.co/t/what-are-logstash-plain-yyyy-mm-dd-log-gz-and-logstash-deprecation-yyyy-mm-dd-log-gz/328125)
**Author:** [@ohaya](https://discuss.elastic.co/u/ohaya)\ **Replies:** 1\ **Last updated:** [March 21, 2023, 8:21am UTC](https://discuss.elastic.co/t/what-are-logstash-plain-yyyy-mm-dd-log-gz-and-logstash-deprecation-yyyy-mm-dd-log-gz/328125 "2023-03-21T08:21:37Z")
Hi, I'm fairly new working with logstash (and actually the entire ELK components), but am trying to determine why some logs are not being ingested and indexed. While I was investigating this, I noticed that on the mach… --- ## [How to specify "bulk\_path" in elasticsearch output on logstash config](https://discuss.elastic.co/t/how-to-specify-bulk-path-in-elasticsearch-output-on-logstash-config/328130)
**Author:** [@Shreesh\_Narayanan](https://discuss.elastic.co/u/Shreesh_Narayanan)\ **Replies:** 0\ **Last updated:** [March 21, 2023, 5:49am UTC](https://discuss.elastic.co/t/how-to-specify-bulk-path-in-elasticsearch-output-on-logstash-config/328130 "2023-03-21T05:49:19Z")
Hello, how do i mention "bulk\_path" in the ES output in logstash ? this is how it's currently implemented (testing) , but i wanted to confirm if i'm doing is correct I have 2 elasticsearch nodes ( a 3rd one will be pr… --- ## [Json parsin](https://discuss.elastic.co/t/json-parsin/326849)
**Author:** [@chrispos](https://discuss.elastic.co/u/chrispos)\ **Replies:** 10\ **Last updated:** [March 20, 2023, 4:34pm UTC](https://discuss.elastic.co/t/json-parsin/326849 "2023-03-20T16:34:54Z")
Hello, I have a question. We are trying to set up a logging system for a java application running on Jboss. The goal is to be able to filter for certain errors. We've done the following Server.log converted to server.… --- ## [How to convert one filed date to string in Logstash](https://discuss.elastic.co/t/how-to-convert-one-filed-date-to-string-in-logstash/328075)
**Author:** [@upreddy](https://discuss.elastic.co/u/upreddy)\ **Replies:** 0\ **Last updated:** [March 20, 2023, 12:47pm UTC](https://discuss.elastic.co/t/how-to-convert-one-filed-date-to-string-in-logstash/328075 "2023-03-20T12:47:06Z")
Hi, I have below log pattern, 2023-03-15T11:11:59.341602012Z stdout F \[INFO \] {"logtype":"msg","trackingid":"XXXXXXX","abcid":"XXXXXXX","operation":{"name":"XXXXX","version":"105"} ,"usecase":"ABC","runtimes":{"output… --- ## [Logstash multiple pipeline Openshift/kubernetes no traffic](https://discuss.elastic.co/t/logstash-multiple-pipeline-openshift-kubernetes-no-traffic/327776)
**Author:** [@splitmessage88](https://discuss.elastic.co/u/splitmessage88)\ **Replies:** 3\ **Last updated:** [March 20, 2023, 10:52am UTC](https://discuss.elastic.co/t/logstash-multiple-pipeline-openshift-kubernetes-no-traffic/327776 "2023-03-20T10:52:59Z")
EDIT I finally get traffic into the cluster, but I only receive logs that have the string "FMC\_AUDIT\_LOG", nothing else get past. If I receive a syslog message with the string "test", it gets dropped. I want the message… --- ## [Kibana server is not ready yet and logstash 401 error](https://discuss.elastic.co/t/kibana-server-is-not-ready-yet-and-logstash-401-error/327247)
**Author:** [@Antony-m](https://discuss.elastic.co/u/Antony-m)\ **Replies:** 2\ **Last updated:** [March 20, 2023, 4:52am UTC](https://discuss.elastic.co/t/kibana-server-is-not-ready-yet-and-logstash-401-error/327247 "2023-03-20T04:52:41Z")
@adityasinghal26 @renangenova I'm new to the ELK stack I watched a youtube tutorial and setup the ELK in my local using docker, here the youtube video link This is my docker-compose.yml file version: '3.6' services: … --- ## [Can i strip different values from "message" field](https://discuss.elastic.co/t/can-i-strip-different-values-from-message-field/328018)
**Author:** [@Hramoff](https://discuss.elastic.co/u/Hramoff)\ **Replies:** 1\ **Last updated:** [March 20, 2023, 4:07am UTC](https://discuss.elastic.co/t/can-i-strip-different-values-from-message-field/328018 "2023-03-20T04:07:28Z")
I am using logstash with syslog plugin to collect logs from vsphere. The problem is that I get a lot of unnecessary entries, over 12,000 different rows per minute. I want to whitelist only the values ​​that I want in th… --- ## [Decompress a gzip compressed string in logstash and push to es](https://discuss.elastic.co/t/decompress-a-gzip-compressed-string-in-logstash-and-push-to-es/327720)
**Author:** [@shdasgupta](https://discuss.elastic.co/u/shdasgupta)\ **Replies:** 2\ **Last updated:** [March 20, 2023, 2:18am UTC](https://discuss.elastic.co/t/decompress-a-gzip-compressed-string-in-logstash-and-push-to-es/327720 "2023-03-20T02:18:45Z")
Team, I am trying to decompress a gzip compressed data using logstash - am not able to figure out how to do this. Input json: (this is fed through a file in this example. In actual, it is consuming a kafka message whic… --- ## [Logstash plugins unit tests](https://discuss.elastic.co/t/logstash-plugins-unit-tests/327686)
**Author:** [@mabourgeot](https://discuss.elastic.co/u/mabourgeot)\ **Replies:** 0\ **Last updated:** [March 14, 2023, 3:43pm UTC](https://discuss.elastic.co/t/logstash-plugins-unit-tests/327686 "2023-03-14T15:43:26Z")
Hi there ! I'm very new to JRuby unit tests and I'm struggling a bit to run existing unit tests in GitHub - logstash-plugins/logstash-integration-aws. I've opened a PR to add a feature and would like to add some tests … --- ## [How to address json objects in a json array in jdbc-output-plugin logstash conf?](https://discuss.elastic.co/t/how-to-address-json-objects-in-a-json-array-in-jdbc-output-plugin-logstash-conf/327980)
**Author:** [@alex\_petrov](https://discuss.elastic.co/u/alex_petrov)\ **Replies:** 0\ **Last updated:** [March 18, 2023, 6:56am UTC](https://discuss.elastic.co/t/how-to-address-json-objects-in-a-json-array-in-jdbc-output-plugin-logstash-conf/327980 "2023-03-18T06:56:16Z")
for example you have this json array : { "accounting" : \[ { "firstName" : "John", "lastName" : "Doe", "age" : 23 }, … --- ## [Question about the logstash plugin version](https://discuss.elastic.co/t/question-about-the-logstash-plugin-version/328022)
**Author:** [@wensionblao](https://discuss.elastic.co/u/wensionblao)\ **Replies:** 1\ **Last updated:** [March 19, 2023, 2:28pm UTC](https://discuss.elastic.co/t/question-about-the-logstash-plugin-version/328022 "2023-03-19T14:28:31Z")
When I use logstash-integration-kafka at version 10.4.0, I find that the corresponding kafka-clients version is 2.4 but logstash-input-kafka and logstash-output-kafka of version 10.4.0 have kafka-clients of version 2… --- ## [Logging from script within Ruby Filter](https://discuss.elastic.co/t/logging-from-script-within-ruby-filter/328012)
**Author:** [@16318a22907f3cbfa04b](https://discuss.elastic.co/u/16318a22907f3cbfa04b)\ **Replies:** 1\ **Last updated:** [March 19, 2023, 9:48am UTC](https://discuss.elastic.co/t/logging-from-script-within-ruby-filter/328012 "2023-03-19T09:48:42Z")
Hello, I can write to the logfile of Logstash from code within ruby filter. https://discuss.elastic.co/t/logging-from-within-ruby-filter/127983/2 Is it possible to do the same from script (not code)? And when yes how? … --- ## [No config files found in path](https://discuss.elastic.co/t/no-config-files-found-in-path/327879)
**Author:** [@Mxnita](https://discuss.elastic.co/u/Mxnita)\ **Replies:** 16\ **Last updated:** [March 18, 2023, 6:56pm UTC](https://discuss.elastic.co/t/no-config-files-found-in-path/327879 "2023-03-18T18:56:27Z")
Hello everyone I am new with Logstash and i trying to start Logstash 8.6.2 on a Windows Server 2019 Server to forward syslogs from a Firewall to Wazuh. When I try to run as administrator in PS the command C:\\logstash-8… --- ## [Logstash filter: a field pointing to many format of the same field](https://discuss.elastic.co/t/logstash-filter-a-field-pointing-to-many-format-of-the-same-field/327918)
**Author:** [@Farah\_Bhr](https://discuss.elastic.co/u/Farah_Bhr)\ **Replies:** 3\ **Last updated:** [March 18, 2023, 10:51am UTC](https://discuss.elastic.co/t/logstash-filter-a-field-pointing-to-many-format-of-the-same-field/327918 "2023-03-18T10:51:33Z")
I have a lot of logs and I want to search through these log lines a callID flow and visualize everything related to that callID into kibana with logstash I made a filter that detects an hexadecimal format for that CallI… --- ## [Simple example for using curl to log a message to Logstash using the HTTP input plugin?](https://discuss.elastic.co/t/simple-example-for-using-curl-to-log-a-message-to-logstash-using-the-http-input-plugin/327964)
**Author:** [@jba](https://discuss.elastic.co/u/jba)\ **Replies:** 1\ **Last updated:** [March 17, 2023, 11:03pm UTC](https://discuss.elastic.co/t/simple-example-for-using-curl-to-log-a-message-to-logstash-using-the-http-input-plugin/327964 "2023-03-17T23:03:39Z")
I am trying to use curl to to log something, just something, to an index, any index, on a ELK 8.4.1 cluster. On my Logstash node I have the following as part of the configuration in the conf.d directory: input { beat… --- ## [Logstash on Windows](https://discuss.elastic.co/t/logstash-on-windows/327906)
**Author:** [@Rakesh\_Mukherjee](https://discuss.elastic.co/u/Rakesh_Mukherjee)\ **Replies:** 1\ **Last updated:** [March 17, 2023, 4:49pm UTC](https://discuss.elastic.co/t/logstash-on-windows/327906 "2023-03-17T16:49:15Z")
I installed Logstash on Windows and find that my output plugin showing error, please help, the error message is here, Unable to configure plugins: (pluginloading error) couldn't find any output plugin named 'microsoft-se… --- ## [Logstash filter help pleas](https://discuss.elastic.co/t/logstash-filter-help-pleas/327718)
**Author:** [@alexsamad](https://discuss.elastic.co/u/alexsamad)\ **Replies:** 2\ **Last updated:** [March 17, 2023, 12:10pm UTC](https://discuss.elastic.co/t/logstash-filter-help-pleas/327718 "2023-03-17T12:10:27Z")
Hi I'm in the process of trying to migrate my config from 6.7 to 8.x I have found I have to rewrite my logstash rules - okay probably a good time to do that. On that note - my filebeat 6.7 client worked fine, when i u… --- ## [Add new fields based on hostname substring](https://discuss.elastic.co/t/add-new-fields-based-on-hostname-substring/327845)
**Author:** [@lemospt](https://discuss.elastic.co/u/lemospt)\ **Replies:** 4\ **Last updated:** [March 16, 2023, 7:41pm UTC](https://discuss.elastic.co/t/add-new-fields-based-on-hostname-substring/327845 "2023-03-16T19:41:53Z")
Hi guys, i want to add new fields based on the information in hostname. Below the examples of what i need, hostname=alfrdnsresolverfixed01 new fields: site=alfr, dns\_type=fixed hostname=boavdnsresolvermbbnat01 new f… --- ## [Logstash performance issues](https://discuss.elastic.co/t/logstash-performance-issues/327679)
**Author:** [@AKAM14](https://discuss.elastic.co/u/AKAM14)\ **Replies:** 9\ **Last updated:** [March 16, 2023, 1:50pm UTC](https://discuss.elastic.co/t/logstash-performance-issues/327679 "2023-03-16T13:50:28Z")
Hi , I Have a Logstash 7.17 version , i have two logstash servers in my Setup that gets connected to a 3 node ELK Cluster. One Kibana server We are experiencing less data getting populated in the kibana dashboards. wh… --- ## [Lag in Logs](https://discuss.elastic.co/t/lag-in-logs/327840)
**Author:** [@kriti\_dabas](https://discuss.elastic.co/u/kriti_dabas)\ **Replies:** 0\ **Last updated:** [March 16, 2023, 10:23am UTC](https://discuss.elastic.co/t/lag-in-logs/327840 "2023-03-16T10:23:29Z")
Why is there a lag in my logs that too for a particular group? I have four kafka consumer groups out of which one group is not giving real-time logs. There is a one hour delay in the working hours whenever I monitor th… --- ## [Duplicate entries in elastic for the same message](https://discuss.elastic.co/t/duplicate-entries-in-elastic-for-the-same-message/327732)
**Author:** [@aks03](https://discuss.elastic.co/u/aks03)\ **Replies:** 3\ **Last updated:** [March 16, 2023, 8:58am UTC](https://discuss.elastic.co/t/duplicate-entries-in-elastic-for-the-same-message/327732 "2023-03-16T08:58:46Z")
Hey everyone, I have been trying to fix this error of duplicate entries into Elasticsearch through logstash Below is the example of the two entries Entry 1: { "\_index": "test-index", "\_type": "doc", "\_id": "3044350… --- ## [Is it possible to call a python script in logstash?](https://discuss.elastic.co/t/is-it-possible-to-call-a-python-script-in-logstash/327825)
**Author:** [@mruthyu](https://discuss.elastic.co/u/mruthyu)\ **Replies:** 1\ **Last updated:** [March 16, 2023, 8:44am UTC](https://discuss.elastic.co/t/is-it-possible-to-call-a-python-script-in-logstash/327825 "2023-03-16T08:44:11Z")
Is it possible to call a python script in logstash pipeline? or only the RUBY language is supported? --- ## [Logstash - elapsed plugin](https://discuss.elastic.co/t/logstash-elapsed-plugin/326879)
**Author:** [@Rnx](https://discuss.elastic.co/u/Rnx)\ **Replies:** 1\ **Last updated:** [March 16, 2023, 7:24am UTC](https://discuss.elastic.co/t/logstash-elapsed-plugin/326879 "2023-03-16T07:24:43Z")
How to install filter plugin (Elapsed) into Logstash, which is already running as a pod in K8s? If I run Logstash from the image docker.elastic.co/logstash/logstash:8.6.1, I can't use "Elapsed" filter, as it's not insta… --- ## [How to check length of an array field in logstash](https://discuss.elastic.co/t/how-to-check-length-of-an-array-field-in-logstash/327625)
**Author:** [@yuswanul](https://discuss.elastic.co/u/yuswanul)\ **Replies:** 8\ **Last updated:** [March 16, 2023, 1:12am UTC](https://discuss.elastic.co/t/how-to-check-length-of-an-array-field-in-logstash/327625 "2023-03-16T01:12:14Z")
Hey everyone, can you give me some example about pipeline config to check if an array field is not null? and how i combine it with if else? i already made config like this and i want to create some if statement under… --- ## [Logstash date filter truncating milliseconds when they are set to 000](https://discuss.elastic.co/t/logstash-date-filter-truncating-milliseconds-when-they-are-set-to-000/327770)
**Author:** [@Samuel\_Delepiere](https://discuss.elastic.co/u/Samuel_Delepiere)\ **Replies:** 2\ **Last updated:** [March 15, 2023, 6:05pm UTC](https://discuss.elastic.co/t/logstash-date-filter-truncating-milliseconds-when-they-are-set-to-000/327770 "2023-03-15T18:05:44Z")
We use the following filter date { match =\> \[ "timestampInUtc" , "UNIX\_MS" \] target =\> "timestamp" timezone =\> "UTC" } This works correctly except when the milliseconds are set to 000. In that case, they get trun… --- ## [Metricbeat - unable to retrieve license information from license server no available connection](https://discuss.elastic.co/t/metricbeat-unable-to-retrieve-license-information-from-license-server-no-available-connection/327113)
**Author:** [@Mary2022](https://discuss.elastic.co/u/Mary2022)\ **Replies:** 5\ **Last updated:** [March 15, 2023, 5:50pm UTC](https://discuss.elastic.co/t/metricbeat-unable-to-retrieve-license-information-from-license-server-no-available-connection/327113 "2023-03-15T17:50:05Z")
This group has always been helpful and hopefully they can help me again. We created a new cluster and installed 8.6 We have a 3 nodes cluster for Elasticsearch, 2 Logstash and 2 Kibana. We completed the configuring in… --- ## [Mapping directly in Logstash Pipeline](https://discuss.elastic.co/t/mapping-directly-in-logstash-pipeline/327757)
**Author:** [@\_Thomas](https://discuss.elastic.co/u/_Thomas)\ **Replies:** 2\ **Last updated:** [March 15, 2023, 1:49pm UTC](https://discuss.elastic.co/t/mapping-directly-in-logstash-pipeline/327757 "2023-03-15T13:49:24Z")
Hi Guys, since I didn't find anything helpful on the net - I need to ask here: Is it possible to do the Mapping of fields directly in the Logstash Pipeline - either in the Input or in the Filter section? As I do have … --- ## [Logstash ruby filter hash class exception](https://discuss.elastic.co/t/logstash-ruby-filter-hash-class-exception/327747)
**Author:** [@onuruzun](https://discuss.elastic.co/u/onuruzun)\ **Replies:** 0\ **Last updated:** [March 15, 2023, 11:47am UTC](https://discuss.elastic.co/t/logstash-ruby-filter-hash-class-exception/327747 "2023-03-15T11:47:29Z")
I tried to get the difference between these two JSON objects coming from JDBC in Logstash. example JDBC JSON: before = '{"heroes":\[{"id":1,"name":"pudge"},{"id":2,"name":"slark"},{"id":3,"name":"techies"}\]}' after = '… --- ## [Useragent filter not working as expected after enabling ECS](https://discuss.elastic.co/t/useragent-filter-not-working-as-expected-after-enabling-ecs/327662)
**Author:** [@flalar](https://discuss.elastic.co/u/flalar)\ **Replies:** 2\ **Last updated:** [March 14, 2023, 8:13pm UTC](https://discuss.elastic.co/t/useragent-filter-not-working-as-expected-after-enabling-ecs/327662 "2023-03-14T20:13:13Z")
We're having trouble with the useragent filter not adding the data to the document sent to Elasticsearch or stdout. Seems this happend after enabling support for ECS. Upgrading from Logstash 7.17.9 to 8.6.2 did not solv… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=84) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=86)