# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=197 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 198 --- ## [S3 output plugin "upload\_worker\_count" logs](https://discuss.elastic.co/t/s3-output-plugin-upload-worker-count-logs/283357)
**Author:** [@Varsha1](https://discuss.elastic.co/u/Varsha1)\ **Replies:** 0\ **Last updated:** [September 5, 2021, 5:15am UTC](https://discuss.elastic.co/t/s3-output-plugin-upload-worker-count-logs/283357 "2021-09-05T05:15:32Z")
I've realized I don't see the detailed s3 log in console when running logstash for kafka input and s3 output. Also I've configured the S3 output plugin to run with multiple workers, how could I make sure the 5 workers ar… --- ## [I am trying to pass a fieldname in jdbc\_streaming plugin to the jdbc\_user parameter, but it fails to read the value of the field](https://discuss.elastic.co/t/i-am-trying-to-pass-a-fieldname-in-jdbc-streaming-plugin-to-the-jdbc-user-parameter-but-it-fails-to-read-the-value-of-the-field/283241)
**Author:** [@Varsha1](https://discuss.elastic.co/u/Varsha1)\ **Replies:** 4\ **Last updated:** [September 5, 2021, 5:07am UTC](https://discuss.elastic.co/t/i-am-trying-to-pass-a-fieldname-in-jdbc-streaming-plugin-to-the-jdbc-user-parameter-but-it-fails-to-read-the-value-of-the-field/283241 "2021-09-05T05:07:31Z")
jdbc\_streaming { jdbc\_driver\_class =\> "Java::oracle.jdbc.driver.OracleDriver" jdbc\_driver\_library =\> "/home/user/logstash-7.14.0/logstash-core/lib/jars/ojdbc8.jar" jdbc\_connection\_string =\> "jdbc:oracle:thin:@XXXXX" … --- ## [Mixed kinds of rename option in single mutate filter won't work](https://discuss.elastic.co/t/mixed-kinds-of-rename-option-in-single-mutate-filter-wont-work/283278)
**Author:** [@tasneem-hyder](https://discuss.elastic.co/u/tasneem-hyder)\ **Replies:** 4\ **Last updated:** [September 4, 2021, 6:23pm UTC](https://discuss.elastic.co/t/mixed-kinds-of-rename-option-in-single-mutate-filter-wont-work/283278 "2021-09-04T18:23:11Z")
In a single mutate filter, we can't use the mixed kinds of rename option. This is not configuration issue but a runtime issue so it is hard to debug. As this kind of config setup doesn't compile to an expected config so … --- ## [How to calculate old date from today by subtracting given years, months, days in logstash?](https://discuss.elastic.co/t/how-to-calculate-old-date-from-today-by-subtracting-given-years-months-days-in-logstash/283349)
**Author:** [@priyaankaa](https://discuss.elastic.co/u/priyaankaa)\ **Replies:** 2\ **Last updated:** [September 4, 2021, 5:36pm UTC](https://discuss.elastic.co/t/how-to-calculate-old-date-from-today-by-subtracting-given-years-months-days-in-logstash/283349 "2021-09-04T17:36:53Z")
I want to subtract current date with given years, months, days using logstash pipeline. For example : Today = 1 Aug 2021 Given values : Years=25, Months=10, Days=10 Expected date : 22 Sept 1995 My logstash filter sec… --- ## [Logstash multiline issue - limit of size in single event](https://discuss.elastic.co/t/logstash-multiline-issue-limit-of-size-in-single-event/283335)
**Author:** [@ErSumit](https://discuss.elastic.co/u/ErSumit)\ **Replies:** 2\ **Last updated:** [September 4, 2021, 5:15pm UTC](https://discuss.elastic.co/t/logstash-multiline-issue-limit-of-size-in-single-event/283335 "2021-09-04T17:15:40Z")
Logstash file input configured with multiline pattern as codec =\> multiline { pattern =\> '^ {' negate =\> true what =\> previous } } Input file consist of multiple json objects. This mu… --- ## [Logstash add field default type is string but check field in an array field must match type](https://discuss.elastic.co/t/logstash-add-field-default-type-is-string-but-check-field-in-an-array-field-must-match-type/283334)
**Author:** [@qinkkai](https://discuss.elastic.co/u/qinkkai)\ **Replies:** 1\ **Last updated:** [September 4, 2021, 7:56am UTC](https://discuss.elastic.co/t/logstash-add-field-default-type-is-string-but-check-field-in-an-array-field-must-match-type/283334 "2021-09-04T07:56:24Z")
My logstash config as follow，I want to output only when the field a in the arr. I think the condition in the output is always true. input { generator { message =\> '{"a":111}' codec =\> json add\_field =\>… --- ## [Dynamic naming of data-streams](https://discuss.elastic.co/t/dynamic-naming-of-data-streams/283281)
**Author:** [@marcus\_lhisp](https://discuss.elastic.co/u/marcus_lhisp)\ **Replies:** 2\ **Last updated:** [September 3, 2021, 10:49pm UTC](https://discuss.elastic.co/t/dynamic-naming-of-data-streams/283281 "2021-09-03T22:49:32Z")
Hello, Currently I'm trying to simplify my pipelines but unfortunately the sprintf format for field reference according to Field References Deep Dive | Logstash Reference \[7.14\] | Elastic does not work. Am I doing some… --- ## [Getting \_grokparsefailure when passing full log files](https://discuss.elastic.co/t/getting-grokparsefailure-when-passing-full-log-files/283207)
**Author:** [@tarabhavi](https://discuss.elastic.co/u/tarabhavi)\ **Replies:** 4\ **Last updated:** [September 3, 2021, 3:42pm UTC](https://discuss.elastic.co/t/getting-grokparsefailure-when-passing-full-log-files/283207 "2021-09-03T15:42:18Z")
have ping logs files used the excerpt of it as a sample for grok filter and tested with grok \_debugger and all good When I pass the full file will all logs instead of excerpt get \_grokparsefailure though was working w… --- ## [Extact costum field from log](https://discuss.elastic.co/t/extact-costum-field-from-log/283288)
**Author:** [@lemahdois](https://discuss.elastic.co/u/lemahdois)\ **Replies:** 1\ **Last updated:** [September 3, 2021, 3:13pm UTC](https://discuss.elastic.co/t/extact-costum-field-from-log/283288 "2021-09-03T15:13:26Z")
Hello, I have a compicated log that does not follow commun format ......... RType: XXX ....... I want to find an easy way to extract the RType field and the value that corresponds to it XXX (for example). note that i… --- ## [How to transfer data to a host with a username and password](https://discuss.elastic.co/t/how-to-transfer-data-to-a-host-with-a-username-and-password/283161)
**Author:** [@Busra\_Duygu](https://discuss.elastic.co/u/Busra_Duygu)\ **Replies:** 8\ **Last updated:** [September 3, 2021, 11:05am UTC](https://discuss.elastic.co/t/how-to-transfer-data-to-a-host-with-a-username-and-password/283161 "2021-09-03T11:05:13Z")
Hello dear friends. I have a problem. I have been helped by this website in many ways before. First of all, thank you very much for all the help for helping me improve myself on elasticsearch. :slight\_smile: Current pro… --- ## [Can we parse XML data without giving xpath details using logstash?](https://discuss.elastic.co/t/can-we-parse-xml-data-without-giving-xpath-details-using-logstash/283128)
**Author:** [@Shruti\_Gupta1](https://discuss.elastic.co/u/Shruti_Gupta1)\ **Replies:** 2\ **Last updated:** [September 3, 2021, 4:03am UTC](https://discuss.elastic.co/t/can-we-parse-xml-data-without-giving-xpath-details-using-logstash/283128 "2021-09-03T04:03:53Z")
Hi Team, We are monitoring google apigee apiProxy messaging logs with ELK Stack. As we are not aware which fields will be sent by workflows in the request and response body of XML, We need help to understand how can we … --- ## [Does logstash can automatically consume new kafka topic?](https://discuss.elastic.co/t/does-logstash-can-automatically-consume-new-kafka-topic/283149)
**Author:** [@a1114066506](https://discuss.elastic.co/u/a1114066506)\ **Replies:** 2\ **Last updated:** [September 3, 2021, 1:24am UTC](https://discuss.elastic.co/t/does-logstash-can-automatically-consume-new-kafka-topic/283149 "2021-09-03T01:24:02Z")
I use input-kafka,configured by below code. input{ kafka{ bootstrap\_servers =\> "127.0.0.1:9092,127.0.0.1:9093,127.0.0.1:9094" topics\_pattern =\> "elk.\*" consumer\_threads =\> 3 decorate\_events =\> true … --- ## [JDBC Logstash Plugin issue](https://discuss.elastic.co/t/jdbc-logstash-plugin-issue/283224)
**Author:** [@valleram](https://discuss.elastic.co/u/valleram)\ **Replies:** 1\ **Last updated:** [September 2, 2021, 11:19pm UTC](https://discuss.elastic.co/t/jdbc-logstash-plugin-issue/283224 "2021-09-02T23:19:23Z")
Hi All, I'm trying to impletement a Logstash pipeline using the JDBC plugin to fetch events from an Oracle database, but I'm receiving following errors. Java::JavaSql::SQLSyntaxErrorException: ORA-00936: missing expres… --- ## [Logstash: how to concatenate static field with dynamic field?](https://discuss.elastic.co/t/logstash-how-to-concatenate-static-field-with-dynamic-field/283183)
**Author:** [@kelk](https://discuss.elastic.co/u/kelk)\ **Replies:** 1\ **Last updated:** [September 2, 2021, 2:47pm UTC](https://discuss.elastic.co/t/logstash-how-to-concatenate-static-field-with-dynamic-field/283183 "2021-09-02T14:47:37Z")
I'm trying to prepend a static variable to identify certain hosts mutate { add\_field =\> { "\[host\]\[hostname\]" =\> "UK\_\_\[app\]\[host\]" } } I need to preprend "UK\_\_" to all fields of app.host field. But the a… --- ## [Create Multiple Indexes from filebeat](https://discuss.elastic.co/t/create-multiple-indexes-from-filebeat/283163)
**Author:** [@rahul\_sirugudi](https://discuss.elastic.co/u/rahul_sirugudi)\ **Replies:** 2\ **Last updated:** [September 2, 2021, 1:54pm UTC](https://discuss.elastic.co/t/create-multiple-indexes-from-filebeat/283163 "2021-09-02T13:54:57Z")
Hi, I am new to ELK stack. I am trying to set up logs for applications. In total we have 29 applications (deb). running in both (autoscaling and spot-fleet in aws). My idea is to use one filebeat configuration on all the… --- ## [Multiple workers fail to process CSV files when auto detect column names is set](https://discuss.elastic.co/t/multiple-workers-fail-to-process-csv-files-when-auto-detect-column-names-is-set/283121)
**Author:** [@ylmp](https://discuss.elastic.co/u/ylmp)\ **Replies:** 2\ **Last updated:** [September 2, 2021, 1:37pm UTC](https://discuss.elastic.co/t/multiple-workers-fail-to-process-csv-files-when-auto-detect-column-names-is-set/283121 "2021-09-02T13:37:10Z")
The issue described here is related to CSV filter plugin with autodetect\_column\_names set. I experience inconsistent behavior when processing CSV files with multiple workers. Seems that the first line in the file is re… --- ## [Parse nested JSON data into logstash](https://discuss.elastic.co/t/parse-nested-json-data-into-logstash/283039)
**Author:** [@Magesh\_02](https://discuss.elastic.co/u/Magesh_02)\ **Replies:** 3\ **Last updated:** [September 2, 2021, 1:05pm UTC](https://discuss.elastic.co/t/parse-nested-json-data-into-logstash/283039 "2021-09-02T13:05:31Z")
New to ELK stack, trying to import my JSON data into elastic serach, but its fails, since my json input is nested types. I have tried many solution but still i could not able to get. here is my input { "Folders": \[ … --- ## [Logstash CSV Output all fields are not working](https://discuss.elastic.co/t/logstash-csv-output-all-fields-are-not-working/282985)
**Author:** [@yogicd](https://discuss.elastic.co/u/yogicd)\ **Replies:** 2\ **Last updated:** [September 2, 2021, 5:55am UTC](https://discuss.elastic.co/t/logstash-csv-output-all-fields-are-not-working/282985 "2021-09-02T05:55:32Z")
Hello, I am testing to get the all the data in .CSV file and unable to get the all the fields and nested in the.CSV file,My config file as below input { stdin {} beats { port =\> 5044 } } output { csv { #fields … --- ## [Attempted to resurrect connection to dead ES instance in secure ELK](https://discuss.elastic.co/t/attempted-to-resurrect-connection-to-dead-es-instance-in-secure-elk/282989)
**Author:** [@kiran80511](https://discuss.elastic.co/u/kiran80511)\ **Replies:** 2\ **Last updated:** [September 2, 2021, 4:56am UTC](https://discuss.elastic.co/t/attempted-to-resurrect-connection-to-dead-es-instance-in-secure-elk/282989 "2021-09-02T04:56:48Z")
Hi, I am using version 7.14 for elasticsearch, logstash and kibana . In logstash log I got this error: Sep 01 10:43:09 elastic.sys logstash\[2975\]: \[2021-09-01T10:43:09,352\]\[WARN \]\[logstash.outputs.elasticsearch\]\[main\]… --- ## [Logstash is not able to connect to ELasticsearch](https://discuss.elastic.co/t/logstash-is-not-able-to-connect-to-elasticsearch/282206)
**Author:** [@Aboli\_77](https://discuss.elastic.co/u/Aboli_77)\ **Replies:** 12\ **Last updated:** [September 2, 2021, 1:07am UTC](https://discuss.elastic.co/t/logstash-is-not-able-to-connect-to-elasticsearch/282206 "2021-09-02T01:07:40Z")
I am trying to run logstash but getting below error: ./logstash -f /etc/logstash/conf.d/abc.conf WARNING: Could not find logstash.yml which is typically located in $LS\_HOME/config or /etc/logstash. You can specify the p… --- ## [Multiple Elasticsearch Filters in one Logstash Pipeline](https://discuss.elastic.co/t/multiple-elasticsearch-filters-in-one-logstash-pipeline/283087)
**Author:** [@pkward](https://discuss.elastic.co/u/pkward)\ **Replies:** 5\ **Last updated:** [September 1, 2021, 6:49pm UTC](https://discuss.elastic.co/t/multiple-elasticsearch-filters-in-one-logstash-pipeline/283087 "2021-09-01T18:49:29Z")
Is it possible to do multiple Elasticsearch lookup filters to search multiple indices in a single Logstash pipeline? --- ## [Troubleshooting logstash configs](https://discuss.elastic.co/t/troubleshooting-logstash-configs/282842)
**Author:** [@kriss332](https://discuss.elastic.co/u/kriss332)\ **Replies:** 8\ **Last updated:** [September 1, 2021, 6:14pm UTC](https://discuss.elastic.co/t/troubleshooting-logstash-configs/282842 "2021-09-01T18:14:11Z")
Hi all.. I've been working on ELK for about a couple of months now and I've been really liking it apart from the troubelshooting steps. I've just integrated a syslog for trial before going heavily onto production phase. … --- ## [Kafka connectors/streaming for enriching the data](https://discuss.elastic.co/t/kafka-connectors-streaming-for-enriching-the-data/283083)
**Author:** [@mruthyu](https://discuss.elastic.co/u/mruthyu)\ **Replies:** 0\ **Last updated:** [September 1, 2021, 4:56pm UTC](https://discuss.elastic.co/t/kafka-connectors-streaming-for-enriching-the-data/283083 "2021-09-01T16:56:38Z")
Is it possible to enrich/transform the data in Kafka connect/Streams and feed it to elasticsearch, without using logstash for data transformation/enrichment? --- ## [Multiple Grok patterns help](https://discuss.elastic.co/t/multiple-grok-patterns-help/282965)
**Author:** [@Zorkmid](https://discuss.elastic.co/u/Zorkmid)\ **Replies:** 4\ **Last updated:** [September 1, 2021, 3:17pm UTC](https://discuss.elastic.co/t/multiple-grok-patterns-help/282965 "2021-09-01T15:17:19Z")
Hi All, I'm doing something wrong with the following. The goal is parse 5 basic events from a PulseSecure VPN syslog feed. Is the following the correct?? filter { if "PulseSecure" in \[message\] { grok { … --- ## [From logstash how to lookup on a data in elastic index?](https://discuss.elastic.co/t/from-logstash-how-to-lookup-on-a-data-in-elastic-index/281387)
**Author:** [@sathish.mtech01](https://discuss.elastic.co/u/sathish.mtech01)\ **Replies:** 10\ **Last updated:** [August 18, 2021, 6:36pm UTC](https://discuss.elastic.co/t/from-logstash-how-to-lookup-on-a-data-in-elastic-index/281387 "2021-08-18T18:36:15Z")
Need a support, i have created a lookup index in elastic search index - lookup Username;Identifier;First name;Last name booker12;l9012;Rachel;Booker grey07;l2070;Laura;Grey johnson81;l4081;Craig;Johnson jenkins46;… --- ## [Logstash Pipeline CSV Date Filter Issue](https://discuss.elastic.co/t/logstash-pipeline-csv-date-filter-issue/282994)
**Author:** [@piotr.krupinski](https://discuss.elastic.co/u/piotr.krupinski)\ **Replies:** 1\ **Last updated:** [September 1, 2021, 10:40am UTC](https://discuss.elastic.co/t/logstash-pipeline-csv-date-filter-issue/282994 "2021-09-01T10:40:53Z")
Hello, I have an issue with pipeline which should fetch data from CSV file. When I'm adding a "Data" to filter section because I need to have it in Elastic as a date it does not want to work. The config is below: inpu… --- ## [Logstash filter file issue](https://discuss.elastic.co/t/logstash-filter-file-issue/282993)
**Author:** [@Nazakat](https://discuss.elastic.co/u/Nazakat)\ **Replies:** 0\ **Last updated:** [September 1, 2021, 6:25am UTC](https://discuss.elastic.co/t/logstash-filter-file-issue/282993 "2021-09-01T06:25:17Z")
My winlogbeat new version is not parsing. new version is 7.14.0 and old version is same filter files is parse. --- ## [Parsing JSON objects within Square Brackets](https://discuss.elastic.co/t/parsing-json-objects-within-square-brackets/282953)
**Author:** [@pkward](https://discuss.elastic.co/u/pkward)\ **Replies:** 1\ **Last updated:** [August 31, 2021, 9:46pm UTC](https://discuss.elastic.co/t/parsing-json-objects-within-square-brackets/282953 "2021-08-31T21:46:59Z")
I'm using an API to get JSON objects, but the API formats the JSON objects within square brackets as an array and splits the objects within the array by commas. Can anyone help me with parsing this JSON with Logstash. I'… --- ## [Did not find the logs in google-cloud-storage pushed by logstash](https://discuss.elastic.co/t/did-not-find-the-logs-in-google-cloud-storage-pushed-by-logstash/282950)
**Author:** [@Xiaochi\_Weng](https://discuss.elastic.co/u/Xiaochi_Weng)\ **Replies:** 0\ **Last updated:** [August 31, 2021, 5:43pm UTC](https://discuss.elastic.co/t/did-not-find-the-logs-in-google-cloud-storage-pushed-by-logstash/282950 "2021-08-31T17:43:07Z")
Hello, I recently opened a google cloud account and installed the plugins. Here is my code: input{ udp{ port=\>55514 type=\>"syslog" } } output{ google\_cloud\_storage { bucket =\> "test-logstash-gcp-send-function" … --- ## [Logstash plugin exec error: Port number ended with '\\'](https://discuss.elastic.co/t/logstash-plugin-exec-error-port-number-ended-with/282947)
**Author:** [@nayeli\_hernandez](https://discuss.elastic.co/u/nayeli_hernandez)\ **Replies:** 0\ **Last updated:** [August 31, 2021, 4:46pm UTC](https://discuss.elastic.co/t/logstash-plugin-exec-error-port-number-ended-with/282947 "2021-08-31T16:46:48Z")
Hi, I want to execute a CURL aggreation in Logstash & schedule it every 1 mins and store the data in elastic search. I'm using exec logstash plugin to do, here is my code: input{ exec{ codec =\> json command =\>"cu… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=196) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=198)