# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=321 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 322 --- ## [How to send logs to http output plugins conditionally](https://discuss.elastic.co/t/how-to-send-logs-to-http-output-plugins-conditionally/243403)
**Author:** [@srbhklkrn](https://discuss.elastic.co/u/srbhklkrn)\ **Replies:** 3\ **Last updated:** [July 31, 2020, 10:30pm UTC](https://discuss.elastic.co/t/how-to-send-logs-to-http-output-plugins-conditionally/243403 "2020-07-31T22:30:27Z")
I have one application that I need to monitor which sends the same logs until that error is not resolved. Now, I need to send a notification to Slack when this particular error occurs. Consider these two sample logs: {… --- ## [Ingest Email Bounces](https://discuss.elastic.co/t/ingest-email-bounces/243396)
**Author:** [@xzelkworld](https://discuss.elastic.co/u/xzelkworld)\ **Replies:** 1\ **Last updated:** [July 31, 2020, 10:15pm UTC](https://discuss.elastic.co/t/ingest-email-bounces/243396 "2020-07-31T22:15:44Z")
We have Email bounce. They are in Email Mailbox format, but they are stored on Linux. Currently we have to set up complicate infrastructure to handle such Email Mailbox format ingestion processing. Is there a modern way … --- ## [Logstash xml input configuration: for multiple documents](https://discuss.elastic.co/t/logstash-xml-input-configuration-for-multiple-documents/243119)
**Author:** [@rahulnama](https://discuss.elastic.co/u/rahulnama)\ **Replies:** 17\ **Last updated:** [July 31, 2020, 7:35pm UTC](https://discuss.elastic.co/t/logstash-xml-input-configuration-for-multiple-documents/243119 "2020-07-31T19:35:30Z")
Hi Team I'm using http\_poller to poll an end point which gives xml data as response. I'm trying to send this xml data to elasticsearch. But when I run logstash, I see logstash is failing. Please have a look at below co… --- ## [Not able to run logstash first stash event](https://discuss.elastic.co/t/not-able-to-run-logstash-first-stash-event/243383)
**Author:** [@xzelkworld](https://discuss.elastic.co/u/xzelkworld)\ **Replies:** 1\ **Last updated:** [July 31, 2020, 7:11pm UTC](https://discuss.elastic.co/t/not-able-to-run-logstash-first-stash-event/243383 "2020-07-31T19:11:16Z")
I installed latest logstash 7.8.1 to CentOS 7.8. I am not able to run first stash event: \[root@logstash\]# bin/logstash -e 'input { stdin { } } output { stdout {} }' hello world ^C I don't know why. Please help! Thanks! … --- ## [How to use fields in http output plugin?](https://discuss.elastic.co/t/how-to-use-fields-in-http-output-plugin/243379)
**Author:** [@srbhklkrn](https://discuss.elastic.co/u/srbhklkrn)\ **Replies:** 5\ **Last updated:** [July 31, 2020, 6:10pm UTC](https://discuss.elastic.co/t/how-to-use-fields-in-http-output-plugin/243379 "2020-07-31T18:10:36Z")
I want to attach syslog5424\_msg in message field in HTTP output plugin here is my current implementation, but it's erroring Failed to execute action I'm not sure if we can interpolate fields in this plugin, any input… --- ## [Event routing to indexes not working](https://discuss.elastic.co/t/event-routing-to-indexes-not-working/243349)
**Author:** [@bijender\_kr](https://discuss.elastic.co/u/bijender_kr)\ **Replies:** 2\ **Last updated:** [July 31, 2020, 5:58pm UTC](https://discuss.elastic.co/t/event-routing-to-indexes-not-working/243349 "2020-07-31T17:58:59Z")
Hi, I am taking windows and unix logs onto logstash and wanted these event logs to be ingested on two different indexes. Below is by config file. input { beats { port =\> "5042" } } filter { if "winlogbea… --- ## [Best practice for shipping logs from large amount of clients using logstash](https://discuss.elastic.co/t/best-practice-for-shipping-logs-from-large-amount-of-clients-using-logstash/243207)
**Author:** [@Kosodrom](https://discuss.elastic.co/u/Kosodrom)\ **Replies:** 3\ **Last updated:** [July 31, 2020, 2:24pm UTC](https://discuss.elastic.co/t/best-practice-for-shipping-logs-from-large-amount-of-clients-using-logstash/243207 "2020-07-31T14:24:37Z")
Hi guys, I am looking for best practices for following use case: I have a lot of clients (linux vms) which going to ship logs (and maybe metrics) over logstash to elasticseach cluster. My idea is to create one index pe… --- ## [Optimum number of workers](https://discuss.elastic.co/t/optimum-number-of-workers/243324)
**Author:** [@Darshan\_Parab](https://discuss.elastic.co/u/Darshan_Parab)\ **Replies:** 1\ **Last updated:** [July 31, 2020, 2:22pm UTC](https://discuss.elastic.co/t/optimum-number-of-workers/243324 "2020-07-31T14:22:41Z")
Hello guys, I have a Kafka topic with 5 partitions. Assume I setup a 2 node Logstash cluster each with a pipeline to read from a Kafka topic, and each pipeline is assigned 4 worker nodes. Now what would be the number … --- ## [Does Snmptrap input plugin of logstash support IPV6?](https://discuss.elastic.co/t/does-snmptrap-input-plugin-of-logstash-support-ipv6/243357)
**Author:** [@viveknagar](https://discuss.elastic.co/u/viveknagar)\ **Replies:** 0\ **Last updated:** [July 31, 2020, 1:13pm UTC](https://discuss.elastic.co/t/does-snmptrap-input-plugin-of-logstash-support-ipv6/243357 "2020-07-31T13:13:44Z")
Hello Everyone, I am using Snmptrap plugin of logstash to receive snmp traps. My configurations of snmptrap are below type =\> snmptrap host =\> '\[XXX.XXX.XXX.XXX.XXX.XXX.XXX.XXX\]' … --- ## [Issue with replacing @timestamp field with the timestamp from logs using date filter in logstash](https://discuss.elastic.co/t/issue-with-replacing-timestamp-field-with-the-timestamp-from-logs-using-date-filter-in-logstash/243176)
**Author:** [@saurabhsule82](https://discuss.elastic.co/u/saurabhsule82)\ **Replies:** 3\ **Last updated:** [July 31, 2020, 12:43pm UTC](https://discuss.elastic.co/t/issue-with-replacing-timestamp-field-with-the-timestamp-from-logs-using-date-filter-in-logstash/243176 "2020-07-31T12:43:37Z")
Hi, We are just learning to setup the ELK stack that includes Filebeats --\> Logstash --\> Elasticsearch \<-- Kibana. Filebeats on our server sends the logs to Logstash, which is configured to send it to Elasticsearch and … --- ## [Logstash wants JVM to use /proc/](https://discuss.elastic.co/t/logstash-wants-jvm-to-use-proc/243313)
**Author:** [@bn1980](https://discuss.elastic.co/u/bn1980)\ **Replies:** 0\ **Last updated:** [July 31, 2020, 6:41am UTC](https://discuss.elastic.co/t/logstash-wants-jvm-to-use-proc/243313 "2020-07-31T06:41:35Z")
Logstash wants JVM to access /procfs but it is not available in a FreeBSD Jail \> \[2020-07-31T14:32:10,897\]\[ERROR\]\[org.logstash.Logstash \] java.lang.InternalError: errno: 2 error: Unable to open /proc/curproc/map I… --- ## [Logstash s3 input plugin specify dynamic prefix](https://discuss.elastic.co/t/logstash-s3-input-plugin-specify-dynamic-prefix/243353)
**Author:** [@Orest\_Gulman](https://discuss.elastic.co/u/Orest_Gulman)\ **Replies:** 0\ **Last updated:** [July 31, 2020, 12:35pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-specify-dynamic-prefix/243353 "2020-07-31T12:35:52Z")
Hi. I would like to use a dynamicaly changing path in prefix based on the current month and year, prefix =\> "elasticloadbalancing/region/YYYY/MM" input { s3 { access\_key\_id =\> "" secret\_access\_key =\> "38/" … --- ## [Logstash s3 input plugin file extension in prefix not working](https://discuss.elastic.co/t/logstash-s3-input-plugin-file-extension-in-prefix-not-working/243352)
**Author:** [@Orest\_Gulman](https://discuss.elastic.co/u/Orest_Gulman)\ **Replies:** 0\ **Last updated:** [July 31, 2020, 12:24pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-file-extension-in-prefix-not-working/243352 "2020-07-31T12:24:30Z")
Hi. I would like to specify a file extension that should be read by logstash. Below my config that I've tried and it's not working. input { s3 { access\_key\_id =\> "" secret\_access\_key =\> "" bucket =\> "log… --- ## [Number of events in logstash node stats on pipelines](https://discuss.elastic.co/t/number-of-events-in-logstash-node-stats-on-pipelines/243345)
**Author:** [@benvl](https://discuss.elastic.co/u/benvl)\ **Replies:** 0\ **Last updated:** [July 31, 2020, 11:48am UTC](https://discuss.elastic.co/t/number-of-events-in-logstash-node-stats-on-pipelines/243345 "2020-07-31T11:48:55Z")
Hi, when viewing logstash node stats, in particular the number of events. then I get reasonable high numbers on the 'in' and 'out' values. Eg. on a logstash 6.7 # curl -s -XGET 'localhost:9600/\_node/stats/pipelines'… --- ## [How do I add a new field based on the contents of a file?](https://discuss.elastic.co/t/how-do-i-add-a-new-field-based-on-the-contents-of-a-file/243340)
**Author:** [@blueren](https://discuss.elastic.co/u/blueren)\ **Replies:** 1\ **Last updated:** [July 31, 2020, 11:14am UTC](https://discuss.elastic.co/t/how-do-i-add-a-new-field-based-on-the-contents-of-a-file/243340 "2020-07-31T11:14:38Z")
Hi, I'm currently using the netflow codec to decode netflow data and store in elastic. I want to introduce a new field called "site\_id". The site ID is going to be present in a file called /tmp/site.id This file will co… --- ## [Logstash jdbc input schedule](https://discuss.elastic.co/t/logstash-jdbc-input-schedule/243311)
**Author:** [@Luismg](https://discuss.elastic.co/u/Luismg)\ **Replies:** 2\ **Last updated:** [July 31, 2020, 9:57am UTC](https://discuss.elastic.co/t/logstash-jdbc-input-schedule/243311 "2020-07-31T09:57:55Z")
Hi, I'm currently trying to import data from a DB into ES using the jdbc input plugin. I want to run the pipeline during the night between 9pm and 4am. How can I define my schedule? I've tried '\*/1 21-04 \* \* \*' but th… --- ## [JDBC input type "store output into single field"](https://discuss.elastic.co/t/jdbc-input-type-store-output-into-single-field/243169)
**Author:** [@rguptarg](https://discuss.elastic.co/u/rguptarg)\ **Replies:** 2\ **Last updated:** [July 31, 2020, 7:29am UTC](https://discuss.elastic.co/t/jdbc-input-type-store-output-into-single-field/243169 "2020-07-31T07:29:39Z")
Hi, I am using jdbc input type and execute the oracle query, is it possible to store query output into a single variable/field ? Query Output : - ID 100 101 102 Above output store into three different JSON into ela… --- ## [Logstash - if - indexOf](https://discuss.elastic.co/t/logstash-if-indexof/243314)
**Author:** [@daemon](https://discuss.elastic.co/u/daemon)\ **Replies:** 1\ **Last updated:** [July 31, 2020, 7:14am UTC](https://discuss.elastic.co/t/logstash-if-indexof/243314 "2020-07-31T07:14:05Z")
Java if("str||aaa").indexOf("||") logstash filter if? ps. What should I do? --- ## [Logstash - Filter - Error message(LogStash::ConfigurationError)](https://discuss.elastic.co/t/logstash-filter-error-message-logstash-configurationerror/243307)
**Author:** [@daemon](https://discuss.elastic.co/u/daemon)\ **Replies:** 1\ **Last updated:** [July 31, 2020, 6:06am UTC](https://discuss.elastic.co/t/logstash-filter-error-message-logstash-configurationerror/243307 "2020-07-31T06:06:09Z")
When I start Logstash I get the below error. filter{ mutate { } grok { match =\> { "message" =\> "%{TIMESTAMP\_ISO8601:requestTime}||%{WORD:type}||%{WORD:jobId}||%{DATA:edgeId}||%{GREEDYDATA:other}"} #match =\> { "mess… --- ## [To exclude around 350 fields in JSON in Logstash](https://discuss.elastic.co/t/to-exclude-around-350-fields-in-json-in-logstash/239019)
**Author:** [@tamilarasanbravo](https://discuss.elastic.co/u/tamilarasanbravo)\ **Replies:** 15\ **Last updated:** [July 31, 2020, 5:12am UTC](https://discuss.elastic.co/t/to-exclude-around-350-fields-in-json-in-logstash/239019 "2020-07-31T05:12:43Z")
Hi, I am trying to exclude many Keys in the JSON field. The data is actually of a Github payload. As the contributors are bit high, the details it carries is also high. For example, the total fields/kvp are around 720 b… --- ## [Can I filter logstash logs with my own log file fields?](https://discuss.elastic.co/t/can-i-filter-logstash-logs-with-my-own-log-file-fields/243300)
**Author:** [@Bhavin\_Varsur](https://discuss.elastic.co/u/Bhavin_Varsur)\ **Replies:** 0\ **Last updated:** [July 31, 2020, 4:02am UTC](https://discuss.elastic.co/t/can-i-filter-logstash-logs-with-my-own-log-file-fields/243300 "2020-07-31T04:02:52Z")
I have my own log file layout so i want to filter logstash filter in pipepines so that i can see my fields and view my own layout logs data . Is that Possible? if yes then how can i filterize ? --- ## [Capturing all messages in a log](https://discuss.elastic.co/t/capturing-all-messages-in-a-log/243261)
**Author:** [@eyts](https://discuss.elastic.co/u/eyts)\ **Replies:** 2\ **Last updated:** [July 31, 2020, 2:13am UTC](https://discuss.elastic.co/t/capturing-all-messages-in-a-log/243261 "2020-07-31T02:13:24Z")
I'm trying to capture all the messages in a log. For example, we get this log that contains 6 messages. "message":\["18.59.43 STC47873 +LRGPSEVRI\\u000007/29/20 18:59:43 I West Logged on to Tandem/Base24.","18.59.43 STC4… --- ## [Converting TimeStamp to epoch in LogStash](https://discuss.elastic.co/t/converting-timestamp-to-epoch-in-logstash/243168)
**Author:** [@Shanmuka\_Chowdary](https://discuss.elastic.co/u/Shanmuka_Chowdary)\ **Replies:** 9\ **Last updated:** [July 31, 2020, 12:22am UTC](https://discuss.elastic.co/t/converting-timestamp-to-epoch-in-logstash/243168 "2020-07-31T00:22:24Z")
Hi Team, I am trying to get some records from SQLServer through logstash. In the database the record is like this : 2002-09-03 04:00:00.000 , But it's saving in elastic like : "fileddate" : "2002-09-03T04:00:00.000Z" … --- ## [How to find last element of array in logstash](https://discuss.elastic.co/t/how-to-find-last-element-of-array-in-logstash/243276)
**Author:** [@Rakesh\_Katakam](https://discuss.elastic.co/u/Rakesh_Katakam)\ **Replies:** 4\ **Last updated:** [July 30, 2020, 8:44pm UTC](https://discuss.elastic.co/t/how-to-find-last-element-of-array-in-logstash/243276 "2020-07-30T20:44:10Z")
Hi, I am trying to split string and add a new event with last index as value. Here below is code snippet which I am trying. Could someone please help me out on the same. Input : eqdomain/eqlogs/abc/hyper.log Script: … --- ## [Logstash multi-pipeline output failure breaks other pipelines](https://discuss.elastic.co/t/logstash-multi-pipeline-output-failure-breaks-other-pipelines/243246)
**Author:** [@Dave\_Waters](https://discuss.elastic.co/u/Dave_Waters)\ **Replies:** 2\ **Last updated:** [July 30, 2020, 6:15pm UTC](https://discuss.elastic.co/t/logstash-multi-pipeline-output-failure-breaks-other-pipelines/243246 "2020-07-30T18:15:15Z")
Running Logstash 7.8.1 Multi pipelines Pipeline A output to B and C B output to ES cluster 1 C output to ES cluster 2 Shutdown ES cluster 1 Error from output attempt to send bulk request... normal error when ES is … --- ## [Logstash csv filter plugin, with mariadb audit plugin logs](https://discuss.elastic.co/t/logstash-csv-filter-plugin-with-mariadb-audit-plugin-logs/242953)
**Author:** [@fabrizio73](https://discuss.elastic.co/u/fabrizio73)\ **Replies:** 6\ **Last updated:** [July 30, 2020, 5:29pm UTC](https://discuss.elastic.co/t/logstash-csv-filter-plugin-with-mariadb-audit-plugin-logs/242953 "2020-07-30T17:29:32Z")
Hi, we are develeping our log management ELK Platform, and we managed almost everything we need in order to get logs with filebeat from percona, mariadb, ssh and SAMBA4 ADC access. The only issue we are facing is that lo… --- ## [Log rotation script for logstash/elasticsearch logs](https://discuss.elastic.co/t/log-rotation-script-for-logstash-elasticsearch-logs/243214)
**Author:** [@pkshara](https://discuss.elastic.co/u/pkshara)\ **Replies:** 1\ **Last updated:** [July 30, 2020, 5:25pm UTC](https://discuss.elastic.co/t/log-rotation-script-for-logstash-elasticsearch-logs/243214 "2020-07-30T17:25:34Z")
Hi Team, I am trying to understand the log rotation scripts for logstash/elasticsearch enabled by default. however i cannot find any logrotate script in servers. Can you please explain, what is the default log rotation… --- ## [Logstash - Filter.Aggregate - push\_map\_as\_event\_on\_timeout not happening](https://discuss.elastic.co/t/logstash-filter-aggregate-push-map-as-event-on-timeout-not-happening/243044)
**Author:** [@GitsBdr](https://discuss.elastic.co/u/GitsBdr)\ **Replies:** 3\ **Last updated:** [July 30, 2020, 5:17pm UTC](https://discuss.elastic.co/t/logstash-filter-aggregate-push-map-as-event-on-timeout-not-happening/243044 "2020-07-30T17:17:53Z")
Hi there, Thanks in advance for your time I'm trying to use filter { aggregate { } } while reading/parsing an old log file Into this log file, every second, there is a block of statistics generated that I'd like to ag… --- ## [Logstash - Syslog Output - Custom message](https://discuss.elastic.co/t/logstash-syslog-output-custom-message/240979)
**Author:** [@vasek](https://discuss.elastic.co/u/vasek)\ **Replies:** 7\ **Last updated:** [July 30, 2020, 2:37pm UTC](https://discuss.elastic.co/t/logstash-syslog-output-custom-message/240979 "2020-07-30T14:37:23Z")
Hi, I'm working with Logstash - Syslog Output and I've found problem with custom field message. I'm using Elasticstack 7.8.0. I've installed logstash syslog-output plugin. /usr/share/logstash/bin/logstash-plugin insta… --- ## [Elapsed filter - keeps crashing](https://discuss.elastic.co/t/elapsed-filter-keeps-crashing/243215)
**Author:** [@Tommie](https://discuss.elastic.co/u/Tommie)\ **Replies:** 0\ **Last updated:** [July 30, 2020, 12:19pm UTC](https://discuss.elastic.co/t/elapsed-filter-keeps-crashing/243215 "2020-07-30T12:19:52Z")
Can anyone please help me with the following: This logstash config is working fine (only with the elapsed part commented): filter { csv { separator =\> "|" columns =\> \[ "datum","level","uniek","mes… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=320) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=322)