# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=319 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 320 --- ## [Need help using "Logstash multiline codec" to parse below oswatcher log lines](https://discuss.elastic.co/t/need-help-using-logstash-multiline-codec-to-parse-below-oswatcher-log-lines/244040)
**Author:** [@saurabhc](https://discuss.elastic.co/u/saurabhc)\ **Replies:** 3\ **Last updated:** [August 6, 2020, 4:18pm UTC](https://discuss.elastic.co/t/need-help-using-logstash-multiline-codec-to-parse-below-oswatcher-log-lines/244040 "2020-08-06T16:18:39Z")
I have logs from oswatcher which looks like below. zzz \*\*\*Thu Jul 23 14:00:31 UTC 2020 MemoryTotal: 100831888 kB MemoryFree: 1212388 kB MemoryAvailable: 49997588 kB zzz \*\*\*Thu Jul 23 14:01:02 UTC 2020 Me… --- ## [Logstash indexing data to elasticsearch cluster](https://discuss.elastic.co/t/logstash-indexing-data-to-elasticsearch-cluster/244033)
**Author:** [@FranB](https://discuss.elastic.co/u/FranB)\ **Replies:** 1\ **Last updated:** [August 6, 2020, 3:16pm UTC](https://discuss.elastic.co/t/logstash-indexing-data-to-elasticsearch-cluster/244033 "2020-08-06T15:16:17Z")
hello guys, i need some help here. i was indexing data to a cluster at a 2500~/s and then suddenly it deprecated to 100~/s any idea what might have been or what do i have to do? logs seems fine, no errors. i can a… --- ## [Best Practice to Make sure clients Logs are written in correct clients Index](https://discuss.elastic.co/t/best-practice-to-make-sure-clients-logs-are-written-in-correct-clients-index/244006)
**Author:** [@Kosodrom](https://discuss.elastic.co/u/Kosodrom)\ **Replies:** 2\ **Last updated:** [August 6, 2020, 2:00pm UTC](https://discuss.elastic.co/t/best-practice-to-make-sure-clients-logs-are-written-in-correct-clients-index/244006 "2020-08-06T14:00:12Z")
Hi, I want to ask for best practice for following problem: You have different clients and you ship your logs using filebeat to logstash. You want to seperate the clients in different indeces and you have a filter … --- ## [How to send laravel logs to elk server](https://discuss.elastic.co/t/how-to-send-laravel-logs-to-elk-server/243931)
**Author:** [@mycloud2040](https://discuss.elastic.co/u/mycloud2040)\ **Replies:** 3\ **Last updated:** [August 6, 2020, 1:57pm UTC](https://discuss.elastic.co/t/how-to-send-laravel-logs-to-elk-server/243931 "2020-08-06T13:57:56Z")
Hi , I am new to ELK, can i have any document to send laravel logs to elk. my ELK server running on ubuntu 18.04. and laraval application is running on same ubuntu18.04. Thanks --- ## [Add header row to csv output file](https://discuss.elastic.co/t/add-header-row-to-csv-output-file/244010)
**Author:** [@akhilsharma.in](https://discuss.elastic.co/u/akhilsharma.in)\ **Replies:** 1\ **Last updated:** [August 6, 2020, 1:47pm UTC](https://discuss.elastic.co/t/add-header-row-to-csv-output-file/244010 "2020-08-06T13:47:33Z")
Using csv output plugin to store JSON field data. But there are no column names; wondering how to give names to columns or how to add header row. --- ## [Problem with "cloning" (filebeat -\> logstash -\> elasticsearch) - more logstash pipelines](https://discuss.elastic.co/t/problem-with-cloning-filebeat-logstash-elasticsearch-more-logstash-pipelines/244018)
**Author:** [@Rysiu](https://discuss.elastic.co/u/Rysiu)\ **Replies:** 5\ **Last updated:** [August 6, 2020, 1:22pm UTC](https://discuss.elastic.co/t/problem-with-cloning-filebeat-logstash-elasticsearch-more-logstash-pipelines/244018 "2020-08-06T13:22:47Z")
Hi, I have a problem. In simple terms, the problem looks like this: If I add one pipeline in logstash: input { beats { ... } } filter { ... } output { elasticsearch { ... } } Everything wor… --- ## [With a small ES cluster that shares master duties, where do I send my Logstash output?](https://discuss.elastic.co/t/with-a-small-es-cluster-that-shares-master-duties-where-do-i-send-my-logstash-output/244028)
**Author:** [@brandondash](https://discuss.elastic.co/u/brandondash)\ **Replies:** 0\ **Last updated:** [August 6, 2020, 1:14pm UTC](https://discuss.elastic.co/t/with-a-small-es-cluster-that-shares-master-duties-where-do-i-send-my-logstash-output/244028 "2020-08-06T13:14:18Z")
Do I send to all ES nodes? Just pick one? output { elasticsearch { index =\> "test-%{+YYYY.MM.dd}" hosts =\> \["10.128.172.112","10.128.172.113","10.128.172.114"\] } } --- ## [Not able to load custom data using csv file in logstash](https://discuss.elastic.co/t/not-able-to-load-custom-data-using-csv-file-in-logstash/243891)
**Author:** [@Divya\_Mathur\_ELK](https://discuss.elastic.co/u/Divya_Mathur_ELK)\ **Replies:** 3\ **Last updated:** [August 6, 2020, 1:02pm UTC](https://discuss.elastic.co/t/not-able-to-load-custom-data-using-csv-file-in-logstash/243891 "2020-08-06T13:02:20Z")
Hi Team- I am really new to elk and looking for the help. i am trying importing data from csv to elasticsearch using logstash but that is getting failed. Below is my CSV file. \`\`\` Sl.No InstanceName InstanceID … --- ## [Logstash 7.8 beats input with SSL throws error](https://discuss.elastic.co/t/logstash-7-8-beats-input-with-ssl-throws-error/243273)
**Author:** [@frankfoti](https://discuss.elastic.co/u/frankfoti)\ **Replies:** 3\ **Last updated:** [August 6, 2020, 11:55am UTC](https://discuss.elastic.co/t/logstash-7-8-beats-input-with-ssl-throws-error/243273 "2020-08-06T11:55:59Z")
I added the SSL to the beats input and now the pipline throws an error. Cert and key are verified. See below 1) Config, 2) Cert and Key verify and 3) error in pipeline log. input { beats { port =\> 5046 ssl =\>… --- ## [Logstash output plugin not working without any error](https://discuss.elastic.co/t/logstash-output-plugin-not-working-without-any-error/243532)
**Author:** [@ksarkar](https://discuss.elastic.co/u/ksarkar)\ **Replies:** 2\ **Last updated:** [August 6, 2020, 8:28am UTC](https://discuss.elastic.co/t/logstash-output-plugin-not-working-without-any-error/243532 "2020-08-06T08:28:37Z")
I have confugured Logstash to feed Elastic search indexes. Here we have integrated with postgresql DB and reading data using timestamp field. There are multiple (10/12) pipeline has been configured to read latest data … --- ## [Parsing Nested JSON with Array](https://discuss.elastic.co/t/parsing-nested-json-with-array/243794)
**Author:** [@nikhilsuryan](https://discuss.elastic.co/u/nikhilsuryan)\ **Replies:** 16\ **Last updated:** [August 6, 2020, 8:05am UTC](https://discuss.elastic.co/t/parsing-nested-json-with-array/243794 "2020-08-06T08:05:50Z")
Hi Team, Can anyone help me in parsing the below and store the same in elastic Search. I am new to Logstash. Wanted to store the metric in Elastic search. { "collectionTime": { "timeStamp": "2020-08-04T17:27:16Z", … --- ## [\[s3 Logstash\] Setting bucket prefix to be specific range of days](https://discuss.elastic.co/t/s3-logstash-setting-bucket-prefix-to-be-specific-range-of-days/243787)
**Author:** [@ian2](https://discuss.elastic.co/u/ian2)\ **Replies:** 2\ **Last updated:** [August 5, 2020, 7:37pm UTC](https://discuss.elastic.co/t/s3-logstash-setting-bucket-prefix-to-be-specific-range-of-days/243787 "2020-08-05T19:37:14Z")
We're pushing logs into an s3 bucket. We've setup the our indexer configuration and tested with a single folder within our bucket - which naming convention is the date (e.g. my-bucket-of-logs/20200804) however we want to… --- ## [Logstash does not create index on elasticsearch](https://discuss.elastic.co/t/logstash-does-not-create-index-on-elasticsearch/243773)
**Author:** [@asuarez](https://discuss.elastic.co/u/asuarez)\ **Replies:** 4\ **Last updated:** [August 5, 2020, 6:38pm UTC](https://discuss.elastic.co/t/logstash-does-not-create-index-on-elasticsearch/243773 "2020-08-05T18:38:07Z")
Hello, I am new to Elastic, I need help with the development of Logstash, I could not get any logstash to automatically create indexes in ElasticSearch. I have created 2 logstast .conf and none create the indexes, I lea… --- ## [Illegal character in authority at index 8:](https://discuss.elastic.co/t/illegal-character-in-authority-at-index-8/243795)
**Author:** [@elaair](https://discuss.elastic.co/u/elaair)\ **Replies:** 12\ **Last updated:** [August 5, 2020, 6:37pm UTC](https://discuss.elastic.co/t/illegal-character-in-authority-at-index-8/243795 "2020-08-05T18:37:09Z")
I am using logstash 7.8 and am seeing this error while using the elasticsearch filter: :error=\>"Illegal character in authority at index 8: https://{:host=\>\\"fqdn.here.com\\", :scheme=\>\\"https\\", :protocol=\>\\"https\\", :po… --- ## [Drop filter is not dropping the logs](https://discuss.elastic.co/t/drop-filter-is-not-dropping-the-logs/243781)
**Author:** [@srbhklkrn](https://discuss.elastic.co/u/srbhklkrn)\ **Replies:** 4\ **Last updated:** [August 5, 2020, 4:19pm UTC](https://discuss.elastic.co/t/drop-filter-is-not-dropping-the-logs/243781 "2020-08-05T16:19:54Z")
I need to drop logs when syslog5424\_host is e.g dev.dev.sample-app-109, the number at the end will change but I have regex to accommodate that. This is my current configuration but it's not dropping the logs. The field… --- ## [GROK pattern and Debugger](https://discuss.elastic.co/t/grok-pattern-and-debugger/242905)
**Author:** [@dmalchikov](https://discuss.elastic.co/u/dmalchikov)\ **Replies:** 2\ **Last updated:** [August 5, 2020, 3:34pm UTC](https://discuss.elastic.co/t/grok-pattern-and-debugger/242905 "2020-08-05T15:34:22Z")
Hi there! Could not find a problem and supported index I have pattern which pass test in Grok Debugger but shows "The input is not valid." in Processor field while Create pipeline Grok pattern: ^%{TIMESTAMP\_ISO8601:t… --- ## [Logstash docker container growing](https://discuss.elastic.co/t/logstash-docker-container-growing/243886)
**Author:** [@MartiM](https://discuss.elastic.co/u/MartiM)\ **Replies:** 0\ **Last updated:** [August 5, 2020, 3:02pm UTC](https://discuss.elastic.co/t/logstash-docker-container-growing/243886 "2020-08-05T15:02:04Z")
We are having an issue with Logstash docker container . The container grows up to 1 TB in 2 or 5 days and when we explore the inside the container in the filesystem there are not folders containing near this space usage … --- ## [Application of grok pattern to the actual YAML](https://discuss.elastic.co/t/application-of-grok-pattern-to-the-actual-yaml/243877)
**Author:** [@eyts](https://discuss.elastic.co/u/eyts)\ **Replies:** 3\ **Last updated:** [August 5, 2020, 2:52pm UTC](https://discuss.elastic.co/t/application-of-grok-pattern-to-the-actual-yaml/243877 "2020-08-05T14:52:25Z")
This is related to topic https://discuss.elastic.co/t/capturing-all-messages-in-a-log/243261. I was able to parse the messages inside the log with (\[%{QS:user\_msg1},)(?:|%{QS:user\_msg2},)(?:|%{QS:user\_msg3},).....(?:|%{… --- ## [Logstash path.data](https://discuss.elastic.co/t/logstash-path-data/243844)
**Author:** [@a11an](https://discuss.elastic.co/u/a11an)\ **Replies:** 2\ **Last updated:** [August 5, 2020, 10:48am UTC](https://discuss.elastic.co/t/logstash-path-data/243844 "2020-08-05T10:48:37Z")
Hi, I am wondering what the path.data compared to path.settings. I was deploying a couple of .conf files under /etc/logstash/conf.d/ directory via cmd line and it complained that it cannot find the --path.settings and s… --- ## [Logstash Grok JSON error - mapper of different type](https://discuss.elastic.co/t/logstash-grok-json-error-mapper-of-different-type/243840)
**Author:** [@nnikolay](https://discuss.elastic.co/u/nnikolay)\ **Replies:** 3\ **Last updated:** [August 5, 2020, 11:08am UTC](https://discuss.elastic.co/t/logstash-grok-json-error-mapper-of-different-type/243840 "2020-08-05T11:08:55Z")
I have this log file: \`2020-08-05 09:11:19 INFO-flask.model-{"version": "1.2.1", "time": 0.651745080947876, "output": {...}}\` This is my logstash filter setting \`grok{ match =\> { "message… --- ## [Logstash dosen't support avro date type?](https://discuss.elastic.co/t/logstash-dosent-support-avro-date-type/243836)
**Author:** [@hwi](https://discuss.elastic.co/u/hwi)\ **Replies:** 0\ **Last updated:** [August 5, 2020, 9:30am UTC](https://discuss.elastic.co/t/logstash-dosent-support-avro-date-type/243836 "2020-08-05T09:30:57Z")
I use logstash to stream data from kafka topic to elasticsearch index. (Input plugin kafka, output plugin elasticsearch) logstash version is 7.8 kafka topic schema include date type. example) name startDate type int… --- ## [Specifies the date format](https://discuss.elastic.co/t/specifies-the-date-format/243178)
**Author:** [@Champion\_Xie](https://discuss.elastic.co/u/Champion_Xie)\ **Replies:** 9\ **Last updated:** [August 5, 2020, 7:33am UTC](https://discuss.elastic.co/t/specifies-the-date-format/243178 "2020-08-05T07:33:07Z")
A timestamp that specifies the format 07/20/20 00:00:08 match =\> \["timestamp","mm/dd/yy HH:mm:ss"\] after test result grokparsefailure how to slove? --- ## [Unable to send logs to remote Logstash server. working fine in localhost](https://discuss.elastic.co/t/unable-to-send-logs-to-remote-logstash-server-working-fine-in-localhost/243628)
**Author:** [@mycloud2040](https://discuss.elastic.co/u/mycloud2040)\ **Replies:** 2\ **Last updated:** [August 5, 2020, 6:42am UTC](https://discuss.elastic.co/t/unable-to-send-logs-to-remote-logstash-server-working-fine-in-localhost/243628 "2020-08-05T06:42:23Z")
Hi, i am new to Elasticstack i configure filebeat in client machine, to send logs to logstash- it is not sending logs to logstash server. Here when I locally send logs to logstash server using filebeat its working fin… --- ## [Logstash grok pattern validator](https://discuss.elastic.co/t/logstash-grok-pattern-validator/243807)
**Author:** [@ash2](https://discuss.elastic.co/u/ash2)\ **Replies:** 0\ **Last updated:** [August 5, 2020, 4:52am UTC](https://discuss.elastic.co/t/logstash-grok-pattern-validator/243807 "2020-08-05T04:52:27Z")
I am working on an application in which I am taking grok pattern as input and want to validate if the pattern is correct or not. Is there any library or any other way to validate the grok pattern in Java Spring backend? --- ## [Cannot log to logstash when xpack security is enabled](https://discuss.elastic.co/t/cannot-log-to-logstash-when-xpack-security-is-enabled/243348)
**Author:** [@Beefstew](https://discuss.elastic.co/u/Beefstew)\ **Replies:** 8\ **Last updated:** [August 4, 2020, 4:04pm UTC](https://discuss.elastic.co/t/cannot-log-to-logstash-when-xpack-security-is-enabled/243348 "2020-08-04T16:04:30Z")
Hi I am running Elk 7.0.0 on Windows server 2016 As soon as I set xpack.security.enabled: true in Elasticsearch.yml logging fails from wionlogbeat clients. What config changes do I need to make in the winlogbeat.yml f… --- ## [Offline Logstash install \[Unable to download data from https://rubygems.org\]](https://discuss.elastic.co/t/offline-logstash-install-unable-to-download-data-from-https-rubygems-org/243706)
**Author:** [@Arethusa](https://discuss.elastic.co/u/Arethusa)\ **Replies:** 2\ **Last updated:** [August 4, 2020, 3:54pm UTC](https://discuss.elastic.co/t/offline-logstash-install-unable-to-download-data-from-https-rubygems-org/243706 "2020-08-04T15:54:48Z")
Hello, I must install logstash and his plugin on offline VM. First on online VM I do: Install Java and Logstash yum install --downloadonly --downloaddir=/home/logstash/ logstash yum install --downloadonly --download… --- ## [Unable to capture some syslog with socket input](https://discuss.elastic.co/t/unable-to-capture-some-syslog-with-socket-input/243735)
**Author:** [@elk2](https://discuss.elastic.co/u/elk2)\ **Replies:** 0\ **Last updated:** [August 4, 2020, 2:47pm UTC](https://discuss.elastic.co/t/unable-to-capture-some-syslog-with-socket-input/243735 "2020-08-04T14:47:14Z")
I'm unable to capture some syslog from different sources: this is my conf: input { udp { port =\> 514 } tcp { port =\> 514 } } filter { } output { file { path … --- ## [JSON parse error, original data now in message field](https://discuss.elastic.co/t/json-parse-error-original-data-now-in-message-field/242459)
**Author:** [@nisaxena](https://discuss.elastic.co/u/nisaxena)\ **Replies:** 7\ **Last updated:** [August 4, 2020, 2:12pm UTC](https://discuss.elastic.co/t/json-parse-error-original-data-now-in-message-field/242459 "2020-08-04T14:12:13Z")
Hi There, I am new to ELK, and need support. I have deployed a new ELK stack \[7.8.0\] & I am trying to ingest my rsyslogs to logstash. I am able to retrieve the logs to logstash but with following errors, logstash-plain… --- ## [Tls handshake timeout tcp input plugin](https://discuss.elastic.co/t/tls-handshake-timeout-tcp-input-plugin/243715)
**Author:** [@f3lang](https://discuss.elastic.co/u/f3lang)\ **Replies:** 0\ **Last updated:** [August 4, 2020, 1:34pm UTC](https://discuss.elastic.co/t/tls-handshake-timeout-tcp-input-plugin/243715 "2020-08-04T13:34:14Z")
Hi Everyone, I hope, you can help me with my problem :slight\_smile: I have configured a logstash server with the tcp input plugin and client tls authentication enabled. So far everything works great and as long as th… --- ## [Logstash to Logstash Certificate Error](https://discuss.elastic.co/t/logstash-to-logstash-certificate-error/243713)
**Author:** [@Yawn](https://discuss.elastic.co/u/Yawn)\ **Replies:** 0\ **Last updated:** [August 4, 2020, 1:21pm UTC](https://discuss.elastic.co/t/logstash-to-logstash-certificate-error/243713 "2020-08-04T13:21:51Z")
Hello everyone Would really appreciate anyone's input on this.. I've been struggling with a certificate problem between 2 logstash servers. I have the following setup basically: agents \> LogstashA \> LogstashB \> Elast… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=318) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=320)