# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=243 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 244 --- ## [Error in Logstash Pipeline](https://discuss.elastic.co/t/error-in-logstash-pipeline/267930)
**Author:** [@caitlyn.carlisle](https://discuss.elastic.co/u/caitlyn.carlisle)\ **Replies:** 4\ **Last updated:** [March 30, 2021, 3:32am UTC](https://discuss.elastic.co/t/error-in-logstash-pipeline/267930 "2021-03-30T03:32:55Z")
I am getting this error when trying to run my Logstash pipeline. I would appreciate any help!! Thanks so much! \[2021-03-21T22:02:58,827\]\[WARN \]\[logstash.config.source.multilocal\] Ignoring the 'pipelines.yml' file becaus… --- ## [Logstash to elasticsearch service](https://discuss.elastic.co/t/logstash-to-elasticsearch-service/268626)
**Author:** [@charles97](https://discuss.elastic.co/u/charles97)\ **Replies:** 12\ **Last updated:** [March 30, 2021, 2:50am UTC](https://discuss.elastic.co/t/logstash-to-elasticsearch-service/268626 "2021-03-30T02:50:50Z")
Hi everyone, I'm on my trial to test elasticcloud. But now I got problem to create pipeline from logstash to elasticcloud. Here is my logstash.conf output output { stdout{codec=\>rubydebug} elasticsearch… --- ## [Parsing Logstash Message Fields](https://discuss.elastic.co/t/parsing-logstash-message-fields/268509)
**Author:** [@Micah\_Barsness](https://discuss.elastic.co/u/Micah_Barsness)\ **Replies:** 5\ **Last updated:** [March 29, 2021, 7:26pm UTC](https://discuss.elastic.co/t/parsing-logstash-message-fields/268509 "2021-03-29T19:26:13Z")
I currently have logstash sending to Elastic cloud - I'd like to get that message section parsed out a bit so that its not one giant section. Could someone help me with that? Here's my filter section - it is just the "o… --- ## [Logstash on linux machine to read files from remote Windows folder](https://discuss.elastic.co/t/logstash-on-linux-machine-to-read-files-from-remote-windows-folder/268458)
**Author:** [@gukutini](https://discuss.elastic.co/u/gukutini)\ **Replies:** 3\ **Last updated:** [March 29, 2021, 4:48pm UTC](https://discuss.elastic.co/t/logstash-on-linux-machine-to-read-files-from-remote-windows-folder/268458 "2021-03-29T16:48:31Z")
I have logstash installed on ubuntu machine and use file input to read log files. I need read files from remote folder of Windows machine. For more convenience I mounted remote folder of Windows machine to my Ubuntu f… --- ## [Dynamic index names](https://discuss.elastic.co/t/dynamic-index-names/268645)
**Author:** [@neilt](https://discuss.elastic.co/u/neilt)\ **Replies:** 0\ **Last updated:** [March 29, 2021, 10:36am UTC](https://discuss.elastic.co/t/dynamic-index-names/268645 "2021-03-29T10:36:02Z")
I've lifted the below pattern from the logstash docs and modified for my use case. Deffo have a field service.region in my log messages that is populated with values dev1 dev2 or sit. But i am not seeing the indexes wi… --- ## [Invalid value "persisted", Options are : \["memory","persisted"\] in 7.9.2 logstash](https://discuss.elastic.co/t/invalid-value-persisted-options-are-memory-persisted-in-7-9-2-logstash/268633)
**Author:** [@Kshema](https://discuss.elastic.co/u/Kshema)\ **Replies:** 0\ **Last updated:** [March 29, 2021, 7:01am UTC](https://discuss.elastic.co/t/invalid-value-persisted-options-are-memory-persisted-in-7-9-2-logstash/268633 "2021-03-29T07:01:52Z")
In 7.9.2 version of logstash, when logstash is run using pipelines.yml, an error occurs saying "Invalid value \\"persisted#\\". Options are : \\"persisted\\", \\"memory\\" ". The error occurs in settings.rb file validate(val… --- ## [Writing Kafka message headers using Logstash's Kafka output plugin](https://discuss.elastic.co/t/writing-kafka-message-headers-using-logstashs-kafka-output-plugin/268625)
**Author:** [@vigneshr35](https://discuss.elastic.co/u/vigneshr35)\ **Replies:** 0\ **Last updated:** [March 29, 2021, 5:25am UTC](https://discuss.elastic.co/t/writing-kafka-message-headers-using-logstashs-kafka-output-plugin/268625 "2021-03-29T05:25:04Z")
Hi All, I am looking to use Logstash's Kafka output plugin. This plugin allows us to post message to Kafka topic, but I don't see any configuration in the documentation that allows to post kafka message headers. Thank … --- ## [Logsash pipeline terminating before timeout duration of aggregation filter plugin](https://discuss.elastic.co/t/logsash-pipeline-terminating-before-timeout-duration-of-aggregation-filter-plugin/268112)
**Author:** [@vigneshr35](https://discuss.elastic.co/u/vigneshr35)\ **Replies:** 2\ **Last updated:** [March 29, 2021, 5:06am UTC](https://discuss.elastic.co/t/logsash-pipeline-terminating-before-timeout-duration-of-aggregation-filter-plugin/268112 "2021-03-29T05:06:23Z")
Hi All, I am having a logstash pipeline that has an elasticsearch plugin for input and aggregation filter plugin. The aggregation filter plugin has a timeout duration of 15 seconds. Refer to the filter plugin configurat… --- ## [Logstash aggregation to get total memory available at a timestamp](https://discuss.elastic.co/t/logstash-aggregation-to-get-total-memory-available-at-a-timestamp/268431)
**Author:** [@D\_R](https://discuss.elastic.co/u/D_R)\ **Replies:** 2\ **Last updated:** [March 29, 2021, 4:26am UTC](https://discuss.elastic.co/t/logstash-aggregation-to-get-total-memory-available-at-a-timestamp/268431 "2021-03-29T04:26:50Z")
Hi, I am new to logstash filters. My input files looks something like below ''' { "@version" =\> "1", "collectd\_type" =\> "memory", "host" =\> "ubuntues", "type\_instance" =\> "slab\_unrecl", "plugin" =\> "memory", "va… --- ## [Cant index field with ingest pipeline](https://discuss.elastic.co/t/cant-index-field-with-ingest-pipeline/268586)
**Author:** [@Alexandros888](https://discuss.elastic.co/u/Alexandros888)\ **Replies:** 2\ **Last updated:** [March 28, 2021, 6:25pm UTC](https://discuss.elastic.co/t/cant-index-field-with-ingest-pipeline/268586 "2021-03-28T18:25:05Z")
Hello, I am facing the following issue Steps: I am creating without problem the following ingest pipeline: PUT \_ingest/pipeline/alexpipeline { "description" : "calculate for the field http.request.body.origina… --- ## [LogStash not sending data to ElasticSearch](https://discuss.elastic.co/t/logstash-not-sending-data-to-elasticsearch/268597)
**Author:** [@ritika3799](https://discuss.elastic.co/u/ritika3799)\ **Replies:** 4\ **Last updated:** [March 28, 2021, 4:46pm UTC](https://discuss.elastic.co/t/logstash-not-sending-data-to-elasticsearch/268597 "2021-03-28T16:46:02Z")
input { file{ path =\> "C:/elastic\_stack/\*.csv" start\_position =\> "beginning" sincedb\_path =\> "null" } } filter { csv { separator =\> "," columns =\> \["name","mfr","type","calori… --- ## [Logstash parsen logfiles](https://discuss.elastic.co/t/logstash-parsen-logfiles/268596)
**Author:** [@bab](https://discuss.elastic.co/u/bab)\ **Replies:** 0\ **Last updated:** [March 28, 2021, 3:51pm UTC](https://discuss.elastic.co/t/logstash-parsen-logfiles/268596 "2021-03-28T15:51:25Z")
Hi all, i'm new on ELK Stack, i need to Filter some log files using Logstash Filter features, so i need some input and tips how i can filter and parse my logs correctly, here is an example of a line from die log file: … --- ## [How to use logstash to collect docker logs cleanly](https://discuss.elastic.co/t/how-to-use-logstash-to-collect-docker-logs-cleanly/268548)
**Author:** [@wajika](https://discuss.elastic.co/u/wajika)\ **Replies:** 6\ **Last updated:** [March 28, 2021, 2:10pm UTC](https://discuss.elastic.co/t/how-to-use-logstash-to-collect-docker-logs-cleanly/268548 "2021-03-28T14:10:19Z")
The default output of docker is in json format, which will contain many special characters (such as \\n \\r). The following log sample, I need to merge multiple lines and remove all \\n \\r. For "\\r \\n", should Line Feed a… --- ## [Logstash multiline codec not working](https://discuss.elastic.co/t/logstash-multiline-codec-not-working/268174)
**Author:** [@111435](https://discuss.elastic.co/u/111435)\ **Replies:** 1\ **Last updated:** [March 28, 2021, 9:34am UTC](https://discuss.elastic.co/t/logstash-multiline-codec-not-working/268174 "2021-03-28T09:34:53Z")
Hello, I have multilines logs: Trace file /gh/app Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production With the Partitioning option ORACLE\_HOME System name: SunOS Node name: kz-cdb Release:… --- ## [Logstash not reading the environment variable properly](https://discuss.elastic.co/t/logstash-not-reading-the-environment-variable-properly/268574)
**Author:** [@mastersmit](https://discuss.elastic.co/u/mastersmit)\ **Replies:** 0\ **Last updated:** [March 28, 2021, 4:43am UTC](https://discuss.elastic.co/t/logstash-not-reading-the-environment-variable-properly/268574 "2021-03-28T04:43:39Z")
As we are connecting to a secure elasticsearch via logstash, I have decided to have the password of the elasticsearch configured as a env variable for example ${ELASTIC\_SEARCH\_PASSWORD} but it doesnt work as expected wh… --- ## [Unsupported symbol \[-\] in geohash](https://discuss.elastic.co/t/unsupported-symbol-in-geohash/268563)
**Author:** [@Aniket\_Pant](https://discuss.elastic.co/u/Aniket_Pant)\ **Replies:** 2\ **Last updated:** [March 28, 2021, 3:06am UTC](https://discuss.elastic.co/t/unsupported-symbol-in-geohash/268563 "2021-03-28T03:06:57Z")
I am facing some issue while defining mapping for an index \[2021-03-27T23:27:36,733\]\[WARN \]\[logstash.outputs.elasticsearch\]\[packetbeat\]\[7bb6e349f13f74fc053ef83bdac5d84b4de038193ed7d008068e4432f88bd27d\] Could not index e… --- ## [Logstash not starting properly](https://discuss.elastic.co/t/logstash-not-starting-properly/268549)
**Author:** [@snobysmake](https://discuss.elastic.co/u/snobysmake)\ **Replies:** 2\ **Last updated:** [March 27, 2021, 4:34pm UTC](https://discuss.elastic.co/t/logstash-not-starting-properly/268549 "2021-03-27T16:34:49Z")
For some strange reason, starting logstash manually seems to work like, bin/logstash -f proj.conf It doesn't start if I run, systemctl start logstash Nothing on the logs since the 25th as shown below, \[2021-03-25T20… --- ## [Filebeat - Logstash SSL not working](https://discuss.elastic.co/t/filebeat-logstash-ssl-not-working/268476)
**Author:** [@anandd4](https://discuss.elastic.co/u/anandd4)\ **Replies:** 1\ **Last updated:** [March 27, 2021, 5:06am UTC](https://discuss.elastic.co/t/filebeat-logstash-ssl-not-working/268476 "2021-03-27T05:06:08Z")
Problem Statement - Unable to transfer filebeat logs to logstash when SSL is enabled. Logstash keeps on complaining about incorrect OpenSSL version number PS - Filebeat works fine when SSL is disabled & able to transfer… --- ## [Logstash http output basic auth base64 encode not parsed](https://discuss.elastic.co/t/logstash-http-output-basic-auth-base64-encode-not-parsed/268432)
**Author:** [@lemoncodes91](https://discuss.elastic.co/u/lemoncodes91)\ **Replies:** 1\ **Last updated:** [March 26, 2021, 5:02pm UTC](https://discuss.elastic.co/t/logstash-http-output-basic-auth-base64-encode-not-parsed/268432 "2021-03-26T17:02:53Z")
Hello i followed some guides on how to do basic auth in logstash. However, when i check the outgoing packet from logstash, the HTTP protocol contains "Authorization: Basic %{password}" instead of "Authorization: Basic \ **Author:** [@GrahamHannington](https://discuss.elastic.co/u/GrahamHannington)\ **Replies:** 5\ **Last updated:** [March 26, 2021, 4:58pm UTC](https://discuss.elastic.co/t/replay-old-events-as-if-they-were-new/268218 "2021-03-26T16:58:01Z") Suppose I have: A Kibana dashboard that shows the last few minutes, automatically refreshing every few seconds A Logstash config that reads JSON Lines from a file (via stdin), sets the event timestamp to the value of a… --- ## [Elasticsearch Output Plugin - Batching, How?](https://discuss.elastic.co/t/elasticsearch-output-plugin-batching-how/268469)
**Author:** [@Liam\_B](https://discuss.elastic.co/u/Liam_B)\ **Replies:** 1\ **Last updated:** [March 26, 2021, 4:51pm UTC](https://discuss.elastic.co/t/elasticsearch-output-plugin-batching-how/268469 "2021-03-26T16:51:51Z")
In the Logstash elastic search output documentation it mentions... Batch Sizes: This plugin attempts to send batches of events to the \[Elasticsearch Bulk API\] I can't find any information on how the batches are buffere… --- ## [Logstash does not pick up all gzip log files](https://discuss.elastic.co/t/logstash-does-not-pick-up-all-gzip-log-files/268446)
**Author:** [@erictung](https://discuss.elastic.co/u/erictung)\ **Replies:** 1\ **Last updated:** [March 26, 2021, 4:39pm UTC](https://discuss.elastic.co/t/logstash-does-not-pick-up-all-gzip-log-files/268446 "2021-03-26T16:39:25Z")
Hi community, this is my first post in this forum and I would like to seek help regarding Logstash. So here's the scenario: I have a Python script to pull logs from Cloudflare ELS and then save it into local folder as … --- ## [Extract value from regex with parentheses in if condition](https://discuss.elastic.co/t/extract-value-from-regex-with-parentheses-in-if-condition/268190)
**Author:** [@Romanian\_Coder](https://discuss.elastic.co/u/Romanian_Coder)\ **Replies:** 8\ **Last updated:** [March 26, 2021, 4:14pm UTC](https://discuss.elastic.co/t/extract-value-from-regex-with-parentheses-in-if-condition/268190 "2021-03-26T16:14:24Z")
I have next input in logstash { logstash | "sourceRecordPosition" =\> 11, logstash | "rejectionReason" =\> "", logstash | "key" =\> 2251799813685255, logstash … --- ## [Data from SQL to Azure Data Lake Gen2](https://discuss.elastic.co/t/data-from-sql-to-azure-data-lake-gen2/268500)
**Author:** [@\_Louw](https://discuss.elastic.co/u/_Louw)\ **Replies:** 0\ **Last updated:** [March 26, 2021, 1:34pm UTC](https://discuss.elastic.co/t/data-from-sql-to-azure-data-lake-gen2/268500 "2021-03-26T13:34:34Z")
What I want to do: Input: Query from a SQL MariaDB Output: Put everything into an Azure Data Lake Gen2 What Azure wants in order to put a file on the data lake: A precise count of all characters in the input My pro… --- ## [I want to combine field](https://discuss.elastic.co/t/i-want-to-combine-field/268300)
**Author:** [@BlackCat](https://discuss.elastic.co/u/BlackCat)\ **Replies:** 2\ **Last updated:** [March 26, 2021, 1:46am UTC](https://discuss.elastic.co/t/i-want-to-combine-field/268300 "2021-03-26T01:46:38Z")
I want to new create "Message" field by "Thread" + "Loglevel" + "Content". Please teach me how to do it. Raw log. \[13172\] \[INFO\] 2021-03-25 09:18:21.150 +0900 : Java class name: org.apache.catalina.startup.Bootstrap; … --- ## [Remove multiple similar prefix events from a json variable](https://discuss.elastic.co/t/remove-multiple-similar-prefix-events-from-a-json-variable/268402)
**Author:** [@rgreen](https://discuss.elastic.co/u/rgreen)\ **Replies:** 1\ **Last updated:** [March 25, 2021, 6:46pm UTC](https://discuss.elastic.co/t/remove-multiple-similar-prefix-events-from-a-json-variable/268402 "2021-03-25T18:46:01Z")
How can I remove multiple events from a json variable with a prefix such as "customer\_" ? I have used the json plugin to pull the data into a variable json { source =\> "message" target =\> "\[@metadata\]\[json\]" } Now i a… --- ## [How to split index by field value?](https://discuss.elastic.co/t/how-to-split-index-by-field-value/268055)
**Author:** [@Rahmat\_Agung\_W](https://discuss.elastic.co/u/Rahmat_Agung_W)\ **Replies:** 9\ **Last updated:** [March 25, 2021, 6:15pm UTC](https://discuss.elastic.co/t/how-to-split-index-by-field-value/268055 "2021-03-25T18:15:07Z")
I have a VM. There are 2 similiar apps (docker apps) inside it, staging and development. They use 1 filebeat to push log to logstash. On these log, I have field container.labels.com\_docker\_compose\_project. I want to crea… --- ## [Should I run multiple pipelines with http input or a single one?](https://discuss.elastic.co/t/should-i-run-multiple-pipelines-with-http-input-or-a-single-one/268337)
**Author:** [@pk.241011](https://discuss.elastic.co/u/pk.241011)\ **Replies:** 1\ **Last updated:** [March 25, 2021, 5:53pm UTC](https://discuss.elastic.co/t/should-i-run-multiple-pipelines-with-http-input-or-a-single-one/268337 "2021-03-25T17:53:48Z")
I have a 16GB machine solely dedicated for running logstash. I will be using the http input. I will have different sources whose data will be stored in different indices. I have a choice of running separate pipelines f… --- ## [How to connect Sybase Sql Anywhere 9 with logstash](https://discuss.elastic.co/t/how-to-connect-sybase-sql-anywhere-9-with-logstash/268390)
**Author:** [@AltC](https://discuss.elastic.co/u/AltC)\ **Replies:** 1\ **Last updated:** [March 25, 2021, 5:45pm UTC](https://discuss.elastic.co/t/how-to-connect-sybase-sql-anywhere-9-with-logstash/268390 "2021-03-25T17:45:24Z")
I'm new to using Elasticsearh components, and I'm having trouble connecting the Sybase SqlAnywhere database to LogStash. Below the configuration file to connect LogStash to SqlAnywhere is showing an error, I have already… --- ## [Can logstash parse same type of multiple log files?](https://discuss.elastic.co/t/can-logstash-parse-same-type-of-multiple-log-files/268024)
**Author:** [@sourabhjain104](https://discuss.elastic.co/u/sourabhjain104)\ **Replies:** 4\ **Last updated:** [March 25, 2021, 4:40pm UTC](https://discuss.elastic.co/t/can-logstash-parse-same-type-of-multiple-log-files/268024 "2021-03-25T16:40:11Z")
Hello, I have a question regarding the multiple log files (of similar type) parsing by logstash. Scenario :- My application generate log files every day,(2 log files in a day) by the name of intranet-YYYY-MM-DD-PORT.l… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=242) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=244)