# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=446 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 447 --- ## [Logstash ssl verification mode](https://discuss.elastic.co/t/logstash-ssl-verification-mode/207858)
**Author:** [@safderali5](https://discuss.elastic.co/u/safderali5)\ **Replies:** 0\ **Last updated:** [November 14, 2019, 10:20am UTC](https://discuss.elastic.co/t/logstash-ssl-verification-mode/207858 "2019-11-14T10:20:45Z")
does logstash 7.4 support following setting? xpack.monitoring.elasticsearch.ssl.verificationmode If yes, what modes are supported, full, certificate, none ? --- ## [Transforming data array to multiple event copies](https://discuss.elastic.co/t/transforming-data-array-to-multiple-event-copies/207929)
**Author:** [@clasyc](https://discuss.elastic.co/u/clasyc)\ **Replies:** 2\ **Last updated:** [November 15, 2019, 7:53am UTC](https://discuss.elastic.co/t/transforming-data-array-to-multiple-event-copies/207929 "2019-11-15T07:53:08Z")
Hello there, I'm new with all elasticsearch stack so it may sound easy and simple task, but I can't find information on how to achieve what I want. I'm using SQS input to the Logstash and output to the elasticsearch. S… --- ## ["logstash output exec " no output](https://discuss.elastic.co/t/logstash-output-exec-no-output/206289)
**Author:** [@pegasus-zhang](https://discuss.elastic.co/u/pegasus-zhang)\ **Replies:** 3\ **Last updated:** [November 15, 2019, 7:17am UTC](https://discuss.elastic.co/t/logstash-output-exec-no-output/206289 "2019-11-15T07:17:53Z")
i tried the exec plugin of logstash: output { exec { codec =\> line { format =\> "%{message}"} command =\> "echo %{message}" } } but nothing print out , only shown: \[2019-10-30T22:18:42,715\]\[INFO \]\[logstash.outputs.… --- ## [Setting the best config in logstash.yml for my Environment](https://discuss.elastic.co/t/setting-the-best-config-in-logstash-yml-for-my-environment/207780)
**Author:** [@Thiago\_Paiva](https://discuss.elastic.co/u/Thiago_Paiva)\ **Replies:** 2\ **Last updated:** [November 15, 2019, 4:33am UTC](https://discuss.elastic.co/t/setting-the-best-config-in-logstash-yml-for-my-environment/207780 "2019-11-15T04:33:15Z")
Hi community. Recently in my environment, I'm feeling that a input data that a pipeline is too slow. Before explain how is mine solution ELK. Following info’s about my machine: OS: Linux version 4.1.12-124.25.1 Red Hat … --- ## [Parse single line multi object json with logstash](https://discuss.elastic.co/t/parse-single-line-multi-object-json-with-logstash/207608)
**Author:** [@sili.redwork](https://discuss.elastic.co/u/sili.redwork)\ **Replies:** 12\ **Last updated:** [November 14, 2019, 11:59pm UTC](https://discuss.elastic.co/t/parse-single-line-multi-object-json-with-logstash/207608 "2019-11-14T23:59:44Z")
Hello, We are sending collecting cloudwatch logs to central log account's S3 bucket using Cloudwatch --\> Log Destination --\> Kinesis --\> S3 S3 file has multiple json objects in single line, below is the sample format {… --- ## [Logstash UDP to InfluxDB](https://discuss.elastic.co/t/logstash-udp-to-influxdb/207966)
**Author:** [@nck8791](https://discuss.elastic.co/u/nck8791)\ **Replies:** 0\ **Last updated:** [November 14, 2019, 10:39pm UTC](https://discuss.elastic.co/t/logstash-udp-to-influxdb/207966 "2019-11-14T22:39:09Z")
I am trying to push data from a UDP to InfluxDB pipeline in Logstash. The logstash configuration is as follows Sample Logstash configuration for creating a simple UDP -\> Logstash -\> InfluxDB pipeline. input { udp { po… --- ## [Extract string from a field in logstash](https://discuss.elastic.co/t/extract-string-from-a-field-in-logstash/207736)
**Author:** [@aasoft](https://discuss.elastic.co/u/aasoft)\ **Replies:** 7\ **Last updated:** [November 14, 2019, 6:24pm UTC](https://discuss.elastic.co/t/extract-string-from-a-field-in-logstash/207736 "2019-11-14T18:24:40Z")
I have a field in my index as: "url" : "http://myaddress:myport/mydoc-3.docx" I would like to create a new field ("newid") and put the extracted value mydoc to it. basically I want to capture the document name with ex… --- ## [List variable reference - is it possible?](https://discuss.elastic.co/t/list-variable-reference-is-it-possible/207940)
**Author:** [@sdndude](https://discuss.elastic.co/u/sdndude)\ **Replies:** 0\ **Last updated:** [November 14, 2019, 5:08pm UTC](https://discuss.elastic.co/t/list-variable-reference-is-it-possible/207940 "2019-11-14T17:08:56Z")
I have a scenario where I could have firewalls (all the same vendor) that are of differing OS versions sending data to logstash. The only difference in the messages is the number of fields it sends. It's all comma del… --- ## [LogStash Docker just stops after start](https://discuss.elastic.co/t/logstash-docker-just-stops-after-start/207790)
**Author:** [@aravindpillai](https://discuss.elastic.co/u/aravindpillai)\ **Replies:** 4\ **Last updated:** [November 14, 2019, 3:29pm UTC](https://discuss.elastic.co/t/logstash-docker-just-stops-after-start/207790 "2019-11-14T15:29:08Z")
HI Here is a simple Docker file and configuration file for Elastic Search and Log stash. The plugin wont start version: '3.2' services: elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:7.4.0 … --- ## [Creating a new field based on the input and send it to the output](https://discuss.elastic.co/t/creating-a-new-field-based-on-the-input-and-send-it-to-the-output/207915)
**Author:** [@Avinaba\_Ghosh\_Hajra](https://discuss.elastic.co/u/Avinaba_Ghosh_Hajra)\ **Replies:** 1\ **Last updated:** [November 14, 2019, 3:02pm UTC](https://discuss.elastic.co/t/creating-a-new-field-based-on-the-input-and-send-it-to-the-output/207915 "2019-11-14T15:02:00Z")
I have a simple query that fetch data from database through logstash jdbc plugin. Query : Select emp\_id, department\_id from employee; Now I need an addition field department\_name in the output. department\_name can be … --- ## [Create structure in painless update script](https://discuss.elastic.co/t/create-structure-in-painless-update-script/207865)
**Author:** [@MMH](https://discuss.elastic.co/u/MMH)\ **Replies:** 2\ **Last updated:** [November 14, 2019, 2:53pm UTC](https://discuss.elastic.co/t/create-structure-in-painless-update-script/207865 "2019-11-14T14:53:10Z")
I am using Elasticsearch 6.5. I want to use an update API with the painless script. Let's say, that my target document has an address structure inside of it. Update script may look something like this: ctx.\_source.addr… --- ## [Indexing txt file that has more than 1gb](https://discuss.elastic.co/t/indexing-txt-file-that-has-more-than-1gb/207908)
**Author:** [@haythem](https://discuss.elastic.co/u/haythem)\ **Replies:** 0\ **Last updated:** [November 14, 2019, 1:59pm UTC](https://discuss.elastic.co/t/indexing-txt-file-that-has-more-than-1gb/207908 "2019-11-14T13:59:01Z")
Hello guys , im facing a problem here indexing txt file that has 8 million lines and 1.5 gb size , logstash only insert 4 million lines !! is it possible to indext txt file in elasticsearch with more than 1.5 gb in size … --- ## [Conditional check grok and mutate a grok field](https://discuss.elastic.co/t/conditional-check-grok-and-mutate-a-grok-field/207895)
**Author:** [@Incauto](https://discuss.elastic.co/u/Incauto)\ **Replies:** 0\ **Last updated:** [November 14, 2019, 12:54pm UTC](https://discuss.elastic.co/t/conditional-check-grok-and-mutate-a-grok-field/207895 "2019-11-14T12:54:19Z")
Hi, I need to check if the first column of data "os" contains the string "lnx" or "win", if contains the string "lnx" in the first column do this grok: grok { match =\> { "message" =\> "%{DATA:os}%{NUMBER:virt\_mem\_used}… --- ## [Logstash not able to create index on elastic search](https://discuss.elastic.co/t/logstash-not-able-to-create-index-on-elastic-search/207665)
**Author:** [@deepak9976](https://discuss.elastic.co/u/deepak9976)\ **Replies:** 1\ **Last updated:** [November 14, 2019, 9:32am UTC](https://discuss.elastic.co/t/logstash-not-able-to-create-index-on-elastic-search/207665 "2019-11-14T09:32:57Z")
I am not able to create index on the elasticsearch by logstash . I was able to create the index when I used it first time , but then I deleted the index, modified by logstash pipeline , and now I am not able to create. … --- ## [Logstash syslog output: Weird message processing](https://discuss.elastic.co/t/logstash-syslog-output-weird-message-processing/207838)
**Author:** [@froto](https://discuss.elastic.co/u/froto)\ **Replies:** 2\ **Last updated:** [November 14, 2019, 9:29am UTC](https://discuss.elastic.co/t/logstash-syslog-output-weird-message-processing/207838 "2019-11-14T09:29:05Z")
Hey there! I've been trying to send a part of my events to another syslog receiver for further processing. I use the logstash-output-syslog plugin for this purpose. This is my configuration: \> output { \> if "flow" in… --- ## [Configuring filebeat to transport csv](https://discuss.elastic.co/t/configuring-filebeat-to-transport-csv/207655)
**Author:** [@Charles\_Yuliansen](https://discuss.elastic.co/u/Charles_Yuliansen)\ **Replies:** 2\ **Last updated:** [November 14, 2019, 7:49am UTC](https://discuss.elastic.co/t/configuring-filebeat-to-transport-csv/207655 "2019-11-14T07:49:23Z")
Im trying to learn using elastic, for my first case case im trying to push csv data to elasticsearch so i can read it in kibana. elasticsearch has been installed on the serverside. Im trying to push it with filebeat f… --- ## [Run logstah pipeline continously in Amazon EC2(Amazon Linux 2 AMI)](https://discuss.elastic.co/t/run-logstah-pipeline-continously-in-amazon-ec2-amazon-linux-2-ami/207823)
**Author:** [@Sunil\_Murali](https://discuss.elastic.co/u/Sunil_Murali)\ **Replies:** 0\ **Last updated:** [November 14, 2019, 6:58am UTC](https://discuss.elastic.co/t/run-logstah-pipeline-continously-in-amazon-ec2-amazon-linux-2-ami/207823 "2019-11-14T06:58:10Z")
Hi All, I have installed logstash in Amazon EC2(Amazon Linux 2 AMI). I am able to execute the filename.conf file by running the following command sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/filename.… --- ## [Using Logstash to upload to S3 log line by line](https://discuss.elastic.co/t/using-logstash-to-upload-to-s3-log-line-by-line/207807)
**Author:** [@Ranika\_Madurawe](https://discuss.elastic.co/u/Ranika_Madurawe)\ **Replies:** 0\ **Last updated:** [November 14, 2019, 5:12am UTC](https://discuss.elastic.co/t/using-logstash-to-upload-to-s3-log-line-by-line/207807 "2019-11-14T05:12:47Z")
I am using LogStash to upload log files in a Volatile K8S system currently running on GCP. I am aiming to upload log files to an AWS s3 bucket using LogStash s3 output plugin. As I understand from the documentation the … --- ## [How to combine two numerical fields](https://discuss.elastic.co/t/how-to-combine-two-numerical-fields/207817)
**Author:** [@mastinder](https://discuss.elastic.co/u/mastinder)\ **Replies:** 0\ **Last updated:** [November 14, 2019, 6:06am UTC](https://discuss.elastic.co/t/how-to-combine-two-numerical-fields/207817 "2019-11-14T06:06:41Z")
I want to combine these two fields | (?\(-|%{INT})) | (?\(-|%{INT})) | and want one field like tota\_bytes=byte\_read+byte\_written , can someone guide me how to do it. %{IP:client-ip}+ | %{WORD:p… --- ## [Change the logfilepath for internal logstash files](https://discuss.elastic.co/t/change-the-logfilepath-for-internal-logstash-files/207809)
**Author:** [@Kamesh\_M](https://discuss.elastic.co/u/Kamesh_M)\ **Replies:** 0\ **Last updated:** [November 14, 2019, 5:36am UTC](https://discuss.elastic.co/t/change-the-logfilepath-for-internal-logstash-files/207809 "2019-11-14T05:36:52Z")
Need to change the logpath for the below internal logstash files logstash.err logstash.log logstash.stdout Currently, its writing in /var/log/logstash and this needs to be changed to different path. Version : logsta… --- ## [Warning message seen at the logs "InvalidFrameProtocolException: Invalid version of beats protocol: 71"](https://discuss.elastic.co/t/warning-message-seen-at-the-logs-invalidframeprotocolexception-invalid-version-of-beats-protocol-71/205789)
**Author:** [@Joseph\_John](https://discuss.elastic.co/u/Joseph_John)\ **Replies:** 1\ **Last updated:** [November 14, 2019, 1:49am UTC](https://discuss.elastic.co/t/warning-message-seen-at-the-logs-invalidframeprotocolexception-invalid-version-of-beats-protocol-71/205789 "2019-11-14T01:49:00Z")
Hi All, Good morning at the logstash logs I can see some warning message, since it is warning nothing wrong , but I am posting the message here for giving the feedback 8.3.13:21374\] Handling exception: org.logstash.b… --- ## [Persist data between log lines with ruby filter](https://discuss.elastic.co/t/persist-data-between-log-lines-with-ruby-filter/206068)
**Author:** [@jong99](https://discuss.elastic.co/u/jong99)\ **Replies:** 4\ **Last updated:** [November 13, 2019, 7:16pm UTC](https://discuss.elastic.co/t/persist-data-between-log-lines-with-ruby-filter/206068 "2019-11-13T19:16:19Z")
What is the best way to persist data between two messages using the ruby filter? For example to include the previous message parsed I can use global variables like this: input { generator { lines =\> \[ "line… --- ## [Logstash file input plugin duplicate events after log file rollover](https://discuss.elastic.co/t/logstash-file-input-plugin-duplicate-events-after-log-file-rollover/207750)
**Author:** [@Cristi\_Ruja](https://discuss.elastic.co/u/Cristi_Ruja)\ **Replies:** 0\ **Last updated:** [November 13, 2019, 4:58pm UTC](https://discuss.elastic.co/t/logstash-file-input-plugin-duplicate-events-after-log-file-rollover/207750 "2019-11-13T16:58:20Z")
I have an application writing a log file, with log4j through RollingFileAppender. Logstash listen to that file (control-engine.log) in order to retrieve and generate events: When log file rollover is happening, the fil… --- ## [Wrong values pass to index](https://discuss.elastic.co/t/wrong-values-pass-to-index/207564)
**Author:** [@namali](https://discuss.elastic.co/u/namali)\ **Replies:** 3\ **Last updated:** [November 13, 2019, 4:03pm UTC](https://discuss.elastic.co/t/wrong-values-pass-to-index/207564 "2019-11-13T16:03:40Z")
there are multiple config files in logstash server. all configs files transfer data to different indexes. values pass from other config files as well to particular index. Please let me know how to over come from to pass … --- ## [Is it possible to set jdbc input plugin retry counts?](https://discuss.elastic.co/t/is-it-possible-to-set-jdbc-input-plugin-retry-counts/207733)
**Author:** [@Jerry\_Zheng](https://discuss.elastic.co/u/Jerry_Zheng)\ **Replies:** 0\ **Last updated:** [November 13, 2019, 3:48pm UTC](https://discuss.elastic.co/t/is-it-possible-to-set-jdbc-input-plugin-retry-counts/207733 "2019-11-13T15:48:23Z")
As title, are there any jdbc input configuration options to set retry counts? Say, when jdbc unable connect to sql server, it will retry 3 times and exit. I have tried connection\_retry\_attempts with value 3, it is expec… --- ## [Does logstash has a size limit for events?](https://discuss.elastic.co/t/does-logstash-has-a-size-limit-for-events/207728)
**Author:** [@asp](https://discuss.elastic.co/u/asp)\ **Replies:** 0\ **Last updated:** [November 13, 2019, 3:12pm UTC](https://discuss.elastic.co/t/does-logstash-has-a-size-limit-for-events/207728 "2019-11-13T15:12:03Z")
Hi, I have payloads (soap) in my logfiles which want to save within the events in elastic stack. These payloads can get big. I've seen about 25MB but I've heard about 100MB+. Is logstash capable of inserting such big … --- ## [How to reference value of field in translate?](https://discuss.elastic.co/t/how-to-reference-value-of-field-in-translate/207561)
**Author:** [@Andrew22](https://discuss.elastic.co/u/Andrew22)\ **Replies:** 16\ **Last updated:** [November 13, 2019, 1:28pm UTC](https://discuss.elastic.co/t/how-to-reference-value-of-field-in-translate/207561 "2019-11-13T13:28:57Z")
I would like to use the device name I have in the dictionary\_path in the translate filter but I am having issues with it. is it even possible? something like translate { field =\> "name" destination =\> "lookUp" dicti… --- ## [Field with duplicate values, keep only one](https://discuss.elastic.co/t/field-with-duplicate-values-keep-only-one/207709)
**Author:** [@victor.nilsson](https://discuss.elastic.co/u/victor.nilsson)\ **Replies:** 0\ **Last updated:** [November 13, 2019, 1:11pm UTC](https://discuss.elastic.co/t/field-with-duplicate-values-keep-only-one/207709 "2019-11-13T13:11:50Z")
Hi, We're parsing logs with the GROK filter and have set the option "brean\_on\_match" to false. This allows us to be absolutely sure that we don't miss a field but at the same time allows for multiple matches in a single… --- ## [Logstash 7.2 monitoring uses deprecated config and unsafe options](https://discuss.elastic.co/t/logstash-7-2-monitoring-uses-deprecated-config-and-unsafe-options/207708)
**Author:** [@YvorL](https://discuss.elastic.co/u/YvorL)\ **Replies:** 0\ **Last updated:** [November 13, 2019, 1:11pm UTC](https://discuss.elastic.co/t/logstash-7-2-monitoring-uses-deprecated-config-and-unsafe-options/207708 "2019-11-13T13:11:30Z")
Hi! I've set up an ES cluster (all parts are 7.2.x) and without any custom configuration I'm already getting errors from the built-in monitoring process: \[WARN \]\[logstash.outputs.elasticsearch\] You are using a depreca… --- ## [How to set Index name in conf.d .conf File to adjust to ILM](https://discuss.elastic.co/t/how-to-set-index-name-in-conf-d-conf-file-to-adjust-to-ilm/207544)
**Author:** [@Moritz\_Kiesewetter](https://discuss.elastic.co/u/Moritz_Kiesewetter)\ **Replies:** 4\ **Last updated:** [November 13, 2019, 12:17pm UTC](https://discuss.elastic.co/t/how-to-set-index-name-in-conf-d-conf-file-to-adjust-to-ilm/207544 "2019-11-13T12:17:10Z")
Hello Guys, i've got the following set in my /logstash/conf.d/10-syslog-XXXXX.conf file: output { if \[type\] == "syslog" { elasticsearch { hosts =\> \["000.000.000.000"\] index =\> "logstash-vmw-000002" I've also … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=445) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=447)