# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=434 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 435 --- ## [Need to do a calculation in logstash](https://discuss.elastic.co/t/need-to-do-a-calculation-in-logstash/211117)
**Author:** [@namali](https://discuss.elastic.co/u/namali)\ **Replies:** 2\ **Last updated:** [December 10, 2019, 3:43am UTC](https://discuss.elastic.co/t/need-to-do-a-calculation-in-logstash/211117 "2019-12-10T03:43:25Z")
logstash input is elasticsearch and there is field called 'check' in input index. when ever read the documents in this index need to increment check by 1. I used ruby filter for this. ruby{code =\> "event.set('check',\[… --- ## [Elasticsearch filter plugin not working](https://discuss.elastic.co/t/elasticsearch-filter-plugin-not-working/210532)
**Author:** [@namali](https://discuss.elastic.co/u/namali)\ **Replies:** 4\ **Last updated:** [December 10, 2019, 3:19am UTC](https://discuss.elastic.co/t/elasticsearch-filter-plugin-not-working/210532 "2019-12-10T03:19:16Z")
running multiple configs in logstash server. Last config, querying the previous logs in elasticsearch which have transferred by other config files in logstash server. filter{ ... elasticsearch { hosts =\> \["XXXXXX/"\] … --- ## [Getting an error with Logstash/filebeat](https://discuss.elastic.co/t/getting-an-error-with-logstash-filebeat/208758)
**Author:** [@seanziee](https://discuss.elastic.co/u/seanziee)\ **Replies:** 4\ **Last updated:** [December 10, 2019, 3:04am UTC](https://discuss.elastic.co/t/getting-an-error-with-logstash-filebeat/208758 "2019-12-10T03:04:33Z")
Hi there, I have a Logstash configuration sitting on an EC2 machine where I push a certain set of data from a machine on port 5044, and a different set on 5045. I never get this error for port 5044 but always for port 50… --- ## [RabbitMQ Input - Logstash (Json Parse Error)](https://discuss.elastic.co/t/rabbitmq-input-logstash-json-parse-error/211213)
**Author:** [@IshanP1](https://discuss.elastic.co/u/IshanP1)\ **Replies:** 0\ **Last updated:** [December 10, 2019, 2:27am UTC](https://discuss.elastic.co/t/rabbitmq-input-logstash-json-parse-error/211213 "2019-12-10T02:27:47Z")
input { rabbitmq { host =\> "localhost" vhost =\> "Test" queue =\> "Ishan" durable =\> true key =\> "NLog.\*" exchange =\> "np.fibre.esb" threads =\> 3 prefetch\_count =\> 50 port =\> 5672 user =\> "xxxx" password =\> "xx… --- ## [Logstash TLS/SSL fails to connect to Elasticsearch with CN=instance does not match the certificate subject provided](https://discuss.elastic.co/t/logstash-tls-ssl-fails-to-connect-to-elasticsearch-with-cn-instance-does-not-match-the-certificate-subject-provided/211181)
**Author:** [@Bharath\_venkat](https://discuss.elastic.co/u/Bharath_venkat)\ **Replies:** 0\ **Last updated:** [December 9, 2019, 7:37pm UTC](https://discuss.elastic.co/t/logstash-tls-ssl-fails-to-connect-to-elasticsearch-with-cn-instance-does-not-match-the-certificate-subject-provided/211181 "2019-12-09T19:37:09Z")
Hi Trying to enable logstash TLS/SSL communication on kubernetes. Logstash fails to communicate with Elasticsearch . ried several ways to fix the CN instance issue . below are my methods Method 1 Created cert bin/elas… --- ## [Logstash Ingesting Daily JSON Files](https://discuss.elastic.co/t/logstash-ingesting-daily-json-files/211153)
**Author:** [@Booooooo](https://discuss.elastic.co/u/Booooooo)\ **Replies:** 0\ **Last updated:** [December 9, 2019, 4:07pm UTC](https://discuss.elastic.co/t/logstash-ingesting-daily-json-files/211153 "2019-12-09T16:07:10Z")
Hello, I am working in a setup on which Logstash will look into a local specific directory for JSON files to parse and forward to Elasticsearch. Those files will be generated daily and placed on the directory where Logs… --- ## [Unable to Remove Plugin standard / custom](https://discuss.elastic.co/t/unable-to-remove-plugin-standard-custom/211027)
**Author:** [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Replies:** 1\ **Last updated:** [December 9, 2019, 3:30pm UTC](https://discuss.elastic.co/t/unable-to-remove-plugin-standard-custom/211027 "2019-12-09T15:30:05Z")
Hi All I am not able to remove the plugin with any 6.x version. I have installed the custom plugin \` bin/logstash-plugin remove logstash-output-xxx \` ERROR: Operation aborted, cannot remove plugin, message: The pa… --- ## [Iterating over certain field extracted from xml to do few operations like remove and replace](https://discuss.elastic.co/t/iterating-over-certain-field-extracted-from-xml-to-do-few-operations-like-remove-and-replace/211126)
**Author:** [@saif3r](https://discuss.elastic.co/u/saif3r)\ **Replies:** 0\ **Last updated:** [December 9, 2019, 1:35pm UTC](https://discuss.elastic.co/t/iterating-over-certain-field-extracted-from-xml-to-do-few-operations-like-remove-and-replace/211126 "2019-12-09T13:35:11Z")
Hi, I have an XML with multiple events. Each event can contain 0 - 6 entries. I would like to make sure that Logstash will extract all parameters, add it to corresponding param field and also enrich one of the fields … --- ## [How to reference http poller data](https://discuss.elastic.co/t/how-to-reference-http-poller-data/210915)
**Author:** [@raged](https://discuss.elastic.co/u/raged)\ **Replies:** 1\ **Last updated:** [December 9, 2019, 1:51pm UTC](https://discuss.elastic.co/t/how-to-reference-http-poller-data/210915 "2019-12-09T13:51:08Z")
I have logstash rigged up for two inputs, one for beats and the other for an http poller to further enrich beats data. How do I reference the data that comes from the http\_poller input? Here is my config for the inputs… --- ## [Path.config and config.string mixed](https://discuss.elastic.co/t/path-config-and-config-string-mixed/211120)
**Author:** [@devopstt](https://discuss.elastic.co/u/devopstt)\ **Replies:** 0\ **Last updated:** [December 9, 2019, 1:26pm UTC](https://discuss.elastic.co/t/path-config-and-config-string-mixed/211120 "2019-12-09T13:26:40Z")
Hi, we want to seperate some information in our logstash configurations, to let teams configure the filter and index setting, but not the input ports or ES output settings, like ES hosts, or users. So we created a conf… --- ## [How to use Type string in a multiple input Http\_Poller for different Grok pattern](https://discuss.elastic.co/t/how-to-use-type-string-in-a-multiple-input-http-poller-for-different-grok-pattern/210759)
**Author:** [@kamalesh](https://discuss.elastic.co/u/kamalesh)\ **Replies:** 1\ **Last updated:** [December 9, 2019, 12:25pm UTC](https://discuss.elastic.co/t/how-to-use-type-string-in-a-multiple-input-http-poller-for-different-grok-pattern/210759 "2019-12-09T12:25:25Z")
Hi All, I am trying to using multiple Url's for the http\_poller and need to use a different Grok patterns for each of the Url's I am aware that i can make use of type string in the Input plugins and make Grok patterns… --- ## [Logstash queue is not draining](https://discuss.elastic.co/t/logstash-queue-is-not-draining/211098)
**Author:** [@74db36a597f21b891b3f](https://discuss.elastic.co/u/74db36a597f21b891b3f)\ **Replies:** 0\ **Last updated:** [December 9, 2019, 11:31am UTC](https://discuss.elastic.co/t/logstash-queue-is-not-draining/211098 "2019-12-09T11:31:48Z")
Hello. Right after the issue described here (https://discuss.elastic.co/t/problem-in-output-pipeline-affected-monitoring-pipeline/209321) i changed logstash pipeline configs and restarted it. But one of servers did not… --- ## [About DB listener / Logstash](https://discuss.elastic.co/t/about-db-listener-logstash/211095)
**Author:** [@eogredici](https://discuss.elastic.co/u/eogredici)\ **Replies:** 0\ **Last updated:** [December 9, 2019, 10:55am UTC](https://discuss.elastic.co/t/about-db-listener-logstash/211095 "2019-12-09T10:55:48Z")
Hi, i want to create a new db listener with join sql and also 2 tables have a same column name. Column names get mixed up when I try to create a new index in Kibana. i don't want to use the alias for column name. Log… --- ## [Failing to create pipeline using RabbitMQ ouput - Fresh install](https://discuss.elastic.co/t/failing-to-create-pipeline-using-rabbitmq-ouput-fresh-install/211029)
**Author:** [@KevSex](https://discuss.elastic.co/u/KevSex)\ **Replies:** 1\ **Last updated:** [December 9, 2019, 9:23am UTC](https://discuss.elastic.co/t/failing-to-create-pipeline-using-rabbitmq-ouput-fresh-install/211029 "2019-12-09T09:23:51Z")
I've installed that latest version of Logstash (at the time of writing 7.5.0), and attempting to use the RabbitMQ output within the pipeline. When I start logstash, it connects to RabbitMQ as seen in the logs but receive… --- ## [REST API using http\_poller and http filter](https://discuss.elastic.co/t/rest-api-using-http-poller-and-http-filter/211076)
**Author:** [@akhilsharma.in](https://discuss.elastic.co/u/akhilsharma.in)\ **Replies:** 0\ **Last updated:** [December 9, 2019, 9:19am UTC](https://discuss.elastic.co/t/rest-api-using-http-poller-and-http-filter/211076 "2019-12-09T09:19:29Z")
Hi, I'm using http\_poller in the input block and http in the filter block to do REST API calls, they are working fine. But recently we have observed variations in the output. I'm running the calls every 5 minutes. I ha… --- ## [Problem in output pipeline affected monitoring pipeline](https://discuss.elastic.co/t/problem-in-output-pipeline-affected-monitoring-pipeline/209321)
**Author:** [@74db36a597f21b891b3f](https://discuss.elastic.co/u/74db36a597f21b891b3f)\ **Replies:** 1\ **Last updated:** [December 9, 2019, 9:12am UTC](https://discuss.elastic.co/t/problem-in-output-pipeline-affected-monitoring-pipeline/209321 "2019-12-09T09:12:49Z")
Hi. I'd like to talk about one case, that had been today and maybe get and advice in pipeline handling. Today I added new ES output to my logstash config. I define outputs in separate pipelines to avoid problem when one… --- ## [Replace @timestamp or add new field with time from log file](https://discuss.elastic.co/t/replace-timestamp-or-add-new-field-with-time-from-log-file/211007)
**Author:** [@pashang](https://discuss.elastic.co/u/pashang)\ **Replies:** 3\ **Last updated:** [December 9, 2019, 6:27am UTC](https://discuss.elastic.co/t/replace-timestamp-or-add-new-field-with-time-from-log-file/211007 "2019-12-09T06:27:33Z")
I have below sample log and I need to replace @timestamp or add new field from 15:38:43.335 segment. -15:38:43.335 U888 ExecuteThread: '3' for queue ... I used below filter but not worked. filter { date { … --- ## [Error occurred after xpack setting](https://discuss.elastic.co/t/error-occurred-after-xpack-setting/207625)
**Author:** [@Takeshi\_Ko](https://discuss.elastic.co/u/Takeshi_Ko)\ **Replies:** 24\ **Last updated:** [December 9, 2019, 12:02am UTC](https://discuss.elastic.co/t/error-occurred-after-xpack-setting/207625 "2019-12-09T00:02:52Z")
After xpack.security enabling, I received error below when I start logstash. Nov 13 05:32:26 ip-172-31-60-170 logstash\[25575\]: \[2019-11-13T14:32:26,597\]\[WARN \]\[logstash.outputs.elasticsearch\]\[main\] Attempted to resurrec… --- ## [Consuming messages from Rabbit while output is not responsive](https://discuss.elastic.co/t/consuming-messages-from-rabbit-while-output-is-not-responsive/210997)
**Author:** [@athlan](https://discuss.elastic.co/u/athlan)\ **Replies:** 1\ **Last updated:** [December 8, 2019, 2:34pm UTC](https://discuss.elastic.co/t/consuming-messages-from-rabbit-while-output-is-not-responsive/210997 "2019-12-08T14:34:03Z")
Hello, I've noticed that using RabbitMQ (with ACKing) as Input and Elasticsearch as Output in Logstash might fall into situation, when Output is not responsive from some reason (beeing restarted, connection loss), then … --- ## [Logstash - Avoid duplicated events](https://discuss.elastic.co/t/logstash-avoid-duplicated-events/210999)
**Author:** [@slawek](https://discuss.elastic.co/u/slawek)\ **Replies:** 0\ **Last updated:** [December 8, 2019, 9:28am UTC](https://discuss.elastic.co/t/logstash-avoid-duplicated-events/210999 "2019-12-08T09:28:53Z")
Every 30 minutes, the log file is replaced with a new log file (also containing old events). Is it possible to configure the logstash to sent the event only the most recent events? I want avoid duplicated events in othe… --- ## [Hot-Warm Arch, without ILM](https://discuss.elastic.co/t/hot-warm-arch-without-ilm/210959)
**Author:** [@Securitybits](https://discuss.elastic.co/u/Securitybits)\ **Replies:** 3\ **Last updated:** [December 8, 2019, 8:09am UTC](https://discuss.elastic.co/t/hot-warm-arch-without-ilm/210959 "2019-12-08T08:09:22Z")
Long time lurker, but first time poster! So lets see if this works! Currently implementing a Hot-Warm architecture. Which consists of 3 dedicated Master Nodes 3 Hot Nodes 3 Warm Nodes The Attrib is set by node.attr… --- ## [Logstash Elasticsearch Input - Query in @timestamp order](https://discuss.elastic.co/t/logstash-elasticsearch-input-query-in-timestamp-order/210607)
**Author:** [@Paul\_Ainslie](https://discuss.elastic.co/u/Paul_Ainslie)\ **Replies:** 11\ **Last updated:** [December 7, 2019, 9:33pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-input-query-in-timestamp-order/210607 "2019-12-07T21:33:15Z")
I'm querying an index with Nginx access records, running it thru the Aggregate filter plugin and writing it to a file. However, the elasticsearch input plugin is not reading the records in @timestamp order - they are wr… --- ## [Reference variable in grok match pattern](https://discuss.elastic.co/t/reference-variable-in-grok-match-pattern/210968)
**Author:** [@marekful](https://discuss.elastic.co/u/marekful)\ **Replies:** 1\ **Last updated:** [December 7, 2019, 3:58pm UTC](https://discuss.elastic.co/t/reference-variable-in-grok-match-pattern/210968 "2019-12-07T15:58:23Z")
I define fields in filebeat.yml as below: ... - paths: - /var/log/\*-\*/a-b-c.log\* document\_type: ... enabled: true fields: serviceNam: a-b-c - paths: - /var/log/\*-\*/d-e-f.log\* document\_type: ... … --- ## [Logstash "stateful" store](https://discuss.elastic.co/t/logstash-stateful-store/210962)
**Author:** [@xvzf](https://discuss.elastic.co/u/xvzf)\ **Replies:** 0\ **Last updated:** [December 7, 2019, 10:54am UTC](https://discuss.elastic.co/t/logstash-stateful-store/210962 "2019-12-07T10:54:03Z")
Hi there, I'm currently learning about Logstash and am wondering if I can keep a "stateful" (let's just call it like that) object which I can use for enhancements. E.g. a HTTP Request pushes a dictionary which is used … --- ## [Aggregate filter - stop after N events](https://discuss.elastic.co/t/aggregate-filter-stop-after-n-events/210950)
**Author:** [@tomr](https://discuss.elastic.co/u/tomr)\ **Replies:** 0\ **Last updated:** [December 7, 2019, 4:13am UTC](https://discuss.elastic.co/t/aggregate-filter-stop-after-n-events/210950 "2019-12-07T04:13:36Z")
Short version: can the aggregate filter be made to stop after it receives 2 events for a given task ID? I'm using the aggregate filter to successfully combine two very similar apache log lines, one of which contains a v… --- ## [Installing Logstash 7.5 on Windows server](https://discuss.elastic.co/t/installing-logstash-7-5-on-windows-server/210945)
**Author:** [@kaushik\_thakkar](https://discuss.elastic.co/u/kaushik_thakkar)\ **Replies:** 0\ **Last updated:** [December 6, 2019, 11:15pm UTC](https://discuss.elastic.co/t/installing-logstash-7-5-on-windows-server/210945 "2019-12-06T23:15:03Z")
Need assistance for installing Logstash 7.5 on Windows server facing issue to unzip the file --- ## [Extract Field from Windows EventLog using Logstash](https://discuss.elastic.co/t/extract-field-from-windows-eventlog-using-logstash/210803)
**Author:** [@nmoham](https://discuss.elastic.co/u/nmoham)\ **Replies:** 3\ **Last updated:** [December 6, 2019, 8:00pm UTC](https://discuss.elastic.co/t/extract-field-from-windows-eventlog-using-logstash/210803 "2019-12-06T20:00:20Z")
Hi, I'm new to using Logstash for parsing data. We've windows Event Logs coming though Winbeats to Logstash and then sent to Elastic Search . I am trying to extract different fields using Logstash, but it doesn't work., … --- ## [Logstash error of strict dynamic mapping exception](https://discuss.elastic.co/t/logstash-error-of-strict-dynamic-mapping-exception/210883)
**Author:** [@karlm](https://discuss.elastic.co/u/karlm)\ **Replies:** 1\ **Last updated:** [December 6, 2019, 1:29pm UTC](https://discuss.elastic.co/t/logstash-error-of-strict-dynamic-mapping-exception/210883 "2019-12-06T13:29:34Z")
Im using logstash 6.7 and when trying to create an index i receive the following error in the logs; 'strict\_dynamic\_mapping\_exception'. mapping set to strict, dynamic introduction of \[dof\] within \[doc\] is not allowed" … --- ## [Logstash elasticsearch output fault tolerance](https://discuss.elastic.co/t/logstash-elasticsearch-output-fault-tolerance/210908)
**Author:** [@mtudisco](https://discuss.elastic.co/u/mtudisco)\ **Replies:** 0\ **Last updated:** [December 6, 2019, 2:54pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-output-fault-tolerance/210908 "2019-12-06T14:54:39Z")
Hi, I'm investigating on high availability options with ELK, for instance Beats -------\> Logstash --------\> Elasticsearch ----- \>Kibana And i have two options: Load balancing: when you have more than one resource f… --- ## [Two files =\> one output](https://discuss.elastic.co/t/two-files-one-output/210904)
**Author:** [@slawek](https://discuss.elastic.co/u/slawek)\ **Replies:** 0\ **Last updated:** [December 6, 2019, 2:23pm UTC](https://discuss.elastic.co/t/two-files-one-output/210904 "2019-12-06T14:23:19Z")
Can logstash be configured to process logs? Scenario below: File1 with logs is copied to path/file1 File2 with logs is copied to path/file2. File2 containing events from file1 and new events. Task for logstash:) Only… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=433) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=435)