# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=543 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 544 --- ## [Logstash Filter to combine patterns together](https://discuss.elastic.co/t/logstash-filter-to-combine-patterns-together/182184)
**Author:** [@unicxs\_support](https://discuss.elastic.co/u/unicxs_support)\ **Replies:** 2\ **Last updated:** [May 22, 2019, 1:47pm UTC](https://discuss.elastic.co/t/logstash-filter-to-combine-patterns-together/182184 "2019-05-22T13:47:46Z")
Hello Experts, I've Just started evaluating the ELK and working great , we are trying to get this product to be used. I'm using 6.5 Version. Below are log patterns.. oradb001,10/03/22,00:29,00-mins oradb002,11/03/21,1… --- ## [Logstash elasticsearch output plugin](https://discuss.elastic.co/t/logstash-elasticsearch-output-plugin/182230)
**Author:** [@safderali5](https://discuss.elastic.co/u/safderali5)\ **Replies:** 0\ **Last updated:** [May 22, 2019, 1:08pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-output-plugin/182230 "2019-05-22T13:08:06Z")
when using tls between logstash and elasticsearch, is there a way to set certificate validation mode to "certificate" in output { elasticsearch {} } the default seems to be set to full. --- ## [Text based log format, need help in parsing](https://discuss.elastic.co/t/text-based-log-format-need-help-in-parsing/182149)
**Author:** [@Abhilash\_B](https://discuss.elastic.co/u/Abhilash_B)\ **Replies:** 2\ **Last updated:** [May 22, 2019, 12:28pm UTC](https://discuss.elastic.co/t/text-based-log-format-need-help-in-parsing/182149 "2019-05-22T12:28:11Z")
Hey All, I have a log format which looks like this. I need to extract all the key value based pairs. My current configuration looks like: filter{ if \[message\] =~ /^#/ { drop {} } else if \[message\] =~ /^\\s\*$/ { dro… --- ## [Logstash randomly stopped sending data to ES/Kibana](https://discuss.elastic.co/t/logstash-randomly-stopped-sending-data-to-es-kibana/182222)
**Author:** [@jamesp220291](https://discuss.elastic.co/u/jamesp220291)\ **Replies:** 0\ **Last updated:** [May 22, 2019, 12:27pm UTC](https://discuss.elastic.co/t/logstash-randomly-stopped-sending-data-to-es-kibana/182222 "2019-05-22T12:27:05Z")
Hi I have been using a ELK stack now for nearly a year. I have noticed that my logstash is no longer sending data to ES/Kibana, It looks like the issue started about a month ago (worked fine until then) I am using a s… --- ## [Need help on logstash running in docker container](https://discuss.elastic.co/t/need-help-on-logstash-running-in-docker-container/182219)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 0\ **Last updated:** [May 22, 2019, 12:06pm UTC](https://discuss.elastic.co/t/need-help-on-logstash-running-in-docker-container/182219 "2019-05-22T12:06:28Z")
Hi Team, I have queries about logstash docker and would really appreciate if someone can help me on this? I have logstash running as docker and I need output writtten in CSV format Host server /var/ partition no in do… --- ## [Parsing date and forwarding to Elastic](https://discuss.elastic.co/t/parsing-date-and-forwarding-to-elastic/182200)
**Author:** [@WarriorHarb](https://discuss.elastic.co/u/WarriorHarb)\ **Replies:** 0\ **Last updated:** [May 22, 2019, 10:30am UTC](https://discuss.elastic.co/t/parsing-date-and-forwarding-to-elastic/182200 "2019-05-22T10:30:03Z")
Hello, I have basicly the same problem mentionned in this topic : https://discuss.elastic.co/t/parsing-the-date-and-sending-to-elasticsearch/83883 I have this log format : 2019-04-19 12:00:05.225 FCC7-VCS01-B VCS: \[… --- ## [Logstash couldn't connect to to elasticsearch](https://discuss.elastic.co/t/logstash-couldnt-connect-to-to-elasticsearch/182193)
**Author:** [@85acb6d03dd51a41bbc7](https://discuss.elastic.co/u/85acb6d03dd51a41bbc7)\ **Replies:** 0\ **Last updated:** [May 22, 2019, 9:57am UTC](https://discuss.elastic.co/t/logstash-couldnt-connect-to-to-elasticsearch/182193 "2019-05-22T09:57:59Z")
Hi guys, I'm configured ELK stack. ElasticSearch in ubuntu server 18.04 Kibana in docker Logstash in docker FileBeat in web-server (IIS) Logstash don't adding data in elasticsearch. If i'm install logstash on windo… --- ## [Auto - install plugin after upgrade](https://discuss.elastic.co/t/auto-install-plugin-after-upgrade/182178)
**Author:** [@thomas.heuberger](https://discuss.elastic.co/u/thomas.heuberger)\ **Replies:** 0\ **Last updated:** [May 22, 2019, 8:57am UTC](https://discuss.elastic.co/t/auto-install-plugin-after-upgrade/182178 "2019-05-22T08:57:42Z")
I'm using the sflow plugin for logstash, which has to be installed after every single upgrade, or else logstash won't start. Now the problem is that our systems are automatically upgraded. So, either I disable automatic … --- ## [Secure Elasticsearch 7.1.0](https://discuss.elastic.co/t/secure-elasticsearch-7-1-0/182163)
**Author:** [@andreas\_renggli](https://discuss.elastic.co/u/andreas_renggli)\ **Replies:** 0\ **Last updated:** [May 22, 2019, 7:49am UTC](https://discuss.elastic.co/t/secure-elasticsearch-7-1-0/182163 "2019-05-22T07:49:15Z")
Hi all, I tested the security feature according: Getting started with Elasticsearch security And it works great. Unfortunately I cannot get Logstash running. output { if(\[type\] =~ /SQL-/) { elasticsear… --- ## [How to segregate logs from different clients remotely from winlogbeat and using index in log stash](https://discuss.elastic.co/t/how-to-segregate-logs-from-different-clients-remotely-from-winlogbeat-and-using-index-in-log-stash/182156)
**Author:** [@Adithya](https://discuss.elastic.co/u/Adithya)\ **Replies:** 0\ **Last updated:** [May 22, 2019, 7:27am UTC](https://discuss.elastic.co/t/how-to-segregate-logs-from-different-clients-remotely-from-winlogbeat-and-using-index-in-log-stash/182156 "2019-05-22T07:27:42Z")
I have configured ELk and pulling log files from three different client using winlogbeat into a logstash how can i segregate those in logstash,.. Thanks in advance --- ## [Generate a URL for a search query from Kibana](https://discuss.elastic.co/t/generate-a-url-for-a-search-query-from-kibana/182150)
**Author:** [@Praneet\_Khandelwal](https://discuss.elastic.co/u/Praneet_Khandelwal)\ **Replies:** 0\ **Last updated:** [May 22, 2019, 7:05am UTC](https://discuss.elastic.co/t/generate-a-url-for-a-search-query-from-kibana/182150 "2019-05-22T07:05:54Z")
I am working for a specific use case. I want to know if there is a way to generate a URL for a particular search query using a python script. And the generated URL when opened in a browser would display the results of th… --- ## [InvalidFrameProtocolException and Connection reset by peer error](https://discuss.elastic.co/t/invalidframeprotocolexception-and-connection-reset-by-peer-error/181753)
**Author:** [@Eva](https://discuss.elastic.co/u/Eva)\ **Replies:** 6\ **Last updated:** [May 22, 2019, 7:05am UTC](https://discuss.elastic.co/t/invalidframeprotocolexception-and-connection-reset-by-peer-error/181753 "2019-05-22T07:05:26Z")
Hi there I have been trying to use Filebeat to push the logs to logstash and then output it to AWS ES domain. The logs are shown up on AWS Elasticsearch domain when I do not use SSL authentication between Filebeat and L… --- ## [Configuring ELK in Windows](https://discuss.elastic.co/t/configuring-elk-in-windows/180744)
**Author:** [@Adithya](https://discuss.elastic.co/u/Adithya)\ **Replies:** 2\ **Last updated:** [May 22, 2019, 7:00am UTC](https://discuss.elastic.co/t/configuring-elk-in-windows/180744 "2019-05-22T07:00:05Z")
Hi All Can you please help me with configuring ELK to pull logs remotely from different servers or computers, Thank you --- ## [Parsing composite log format only with text](https://discuss.elastic.co/t/parsing-composite-log-format-only-with-text/182148)
**Author:** [@Abhilash\_B](https://discuss.elastic.co/u/Abhilash_B)\ **Replies:** 1\ **Last updated:** [May 22, 2019, 6:58am UTC](https://discuss.elastic.co/t/parsing-composite-log-format-only-with-text/182148 "2019-05-22T06:58:37Z")
Hi All, I have a log file which looks like this. My filter config looks like this: filter{ if \[message\] =~ /(.\*): (.\*) - (.\*)/ { mutate { strip =\> \[ "message" \] } ruby { code =\> ' msg = event… --- ## [MySQL data stops working through logstash](https://discuss.elastic.co/t/mysql-data-stops-working-through-logstash/182145)
**Author:** [@bikingbadger](https://discuss.elastic.co/u/bikingbadger)\ **Replies:** 0\ **Last updated:** [May 22, 2019, 6:52am UTC](https://discuss.elastic.co/t/mysql-data-stops-working-through-logstash/182145 "2019-05-22T06:52:44Z")
Hi there, I've setup logstash to query data from a MySQL db and it is working fine up to a point. The last run file contains the correct index and i see in the log that the query is being run: The issue is that in … --- ## [Can someone help with filter logic of my log file](https://discuss.elastic.co/t/can-someone-help-with-filter-logic-of-my-log-file/181983)
**Author:** [@anon69830709](https://discuss.elastic.co/u/anon69830709)\ **Replies:** 2\ **Last updated:** [May 22, 2019, 4:33am UTC](https://discuss.elastic.co/t/can-someone-help-with-filter-logic-of-my-log-file/181983 "2019-05-22T04:33:33Z")
This is my sample log. May 21 13:21:48 Network Log: NSD FAIL WAN\[1\] May 21 13:21:58 Network Log: NSD SUCCESS WAN\[1\] May 21 13:32:45 User Log: User cisco login failed. May 21 13:33:03 User Log: User cisco login succ… --- ## [Logstash 7 error - ssl trustore path must be string](https://discuss.elastic.co/t/logstash-7-error-ssl-trustore-path-must-be-string/181962)
**Author:** [@hari.v](https://discuss.elastic.co/u/hari.v)\ **Replies:** 6\ **Last updated:** [May 22, 2019, 3:27am UTC](https://discuss.elastic.co/t/logstash-7-error-ssl-trustore-path-must-be-string/181962 "2019-05-22T03:27:11Z")
Hi, i tried to configure logstash nodes. i followed this document : https://www.elastic.co/guide/en/logstash/current/configuring-logstash.html i copied the ca.p12 cert from elasticsearch cluster nodes. my logstash.ym… --- ## [Logstash 7 not showing any o/p](https://discuss.elastic.co/t/logstash-7-not-showing-any-o-p/182116)
**Author:** [@hari.v](https://discuss.elastic.co/u/hari.v)\ **Replies:** 0\ **Last updated:** [May 22, 2019, 2:54am UTC](https://discuss.elastic.co/t/logstash-7-not-showing-any-o-p/182116 "2019-05-22T02:54:46Z")
Hi My logstash service is running ... but log file is not updating ... showing past messages... i tried to restart logstash service many times but not updating in the logs. any reason ... even i killed the processes … --- ## [Can Output elasticsearch plugin credential use @metadata?](https://discuss.elastic.co/t/can-output-elasticsearch-plugin-credential-use-metadata/181980)
**Author:** [@mmk1995](https://discuss.elastic.co/u/mmk1995)\ **Replies:** 6\ **Last updated:** [May 22, 2019, 1:49am UTC](https://discuss.elastic.co/t/can-output-elasticsearch-plugin-credential-use-metadata/181980 "2019-05-22T01:49:20Z")
Dear All, I am trying to use @metadata that has defined in filter for the user and password fields in output, is that possible or is there another way? Please see below configuration: output { elasticsearch { hosts=\>… --- ## [Defining ES index pattern with ILM in logstash](https://discuss.elastic.co/t/defining-es-index-pattern-with-ilm-in-logstash/182082)
**Author:** [@xyz2](https://discuss.elastic.co/u/xyz2)\ **Replies:** 0\ **Last updated:** [May 21, 2019, 8:00pm UTC](https://discuss.elastic.co/t/defining-es-index-pattern-with-ilm-in-logstash/182082 "2019-05-21T20:00:54Z")
HI , I have ILM for an index say test-000001. In order to define the output ES index in logstash, do i just need to mention "test\*" and no other parameter related to ILM ? Is there any regex support also in index patte… --- ## [Logstash not picking the file after placing in the input path](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703)
**Author:** [@syed\_faheem\_Hussain](https://discuss.elastic.co/u/syed_faheem_Hussain)\ **Replies:** 44\ **Last updated:** [May 21, 2019, 6:50pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703 "2019-05-21T18:50:55Z")
I have placed the config.properties file with input of the log file path but whenever a new file is put in that location it is not fetching the latest file my conf as below input { file { path =\> "/mnt/storage/logs/\*" … --- ## [Failed to publish events: write tcp 127.0.0.1:59906-\>127.0.0.1:5044: write: connection reset by peer](https://discuss.elastic.co/t/failed-to-publish-events-write-tcp-127-0-0-1-59906-127-0-0-1-write-connection-reset-by-peer/182068)
**Author:** [@ankitachow](https://discuss.elastic.co/u/ankitachow)\ **Replies:** 0\ **Last updated:** [May 21, 2019, 5:08pm UTC](https://discuss.elastic.co/t/failed-to-publish-events-write-tcp-127-0-0-1-59906-127-0-0-1-write-connection-reset-by-peer/182068 "2019-05-21T17:08:40Z")
Using Logstash & filbeat 6.4. I'm trying to write to ES/stdout from Filebeat through Logstash. I use stdout for debugging. When I write output to elastisearch it's working fine. But when I try stdout, I get below error. … --- ## [ELK - Kibana showing data that was already deleted, won't show new data](https://discuss.elastic.co/t/elk-kibana-showing-data-that-was-already-deleted-wont-show-new-data/181631)
**Author:** [@Mikael\_Brenner\_Vieir](https://discuss.elastic.co/u/Mikael_Brenner_Vieir)\ **Replies:** 3\ **Last updated:** [May 21, 2019, 4:24pm UTC](https://discuss.elastic.co/t/elk-kibana-showing-data-that-was-already-deleted-wont-show-new-data/181631 "2019-05-21T16:24:50Z")
Hi, I'm currently using the ELK stack, and I'm running into this problem quite often. My Logstash config file is looking like this: input { file{ path =\> "/home/me/logs/\*.log" sta… --- ## [Logstash applying grok to only specific log lines](https://discuss.elastic.co/t/logstash-applying-grok-to-only-specific-log-lines/181645)
**Author:** [@chandukreddi](https://discuss.elastic.co/u/chandukreddi)\ **Replies:** 12\ **Last updated:** [May 21, 2019, 4:20pm UTC](https://discuss.elastic.co/t/logstash-applying-grok-to-only-specific-log-lines/181645 "2019-05-21T16:20:20Z")
Hi, why logstash applying grok for only only few log lines even though I have grok patterns for all log lines and I can run through same logs using grok debugger and its working fine, I think its behaving like unpredict… --- ## [SSL Error when using SSL with TCP output plugin](https://discuss.elastic.co/t/ssl-error-when-using-ssl-with-tcp-output-plugin/181277)
**Author:** [@Tieis](https://discuss.elastic.co/u/Tieis)\ **Replies:** 1\ **Last updated:** [May 21, 2019, 4:14pm UTC](https://discuss.elastic.co/t/ssl-error-when-using-ssl-with-tcp-output-plugin/181277 "2019-05-21T16:14:30Z")
Hi, I'm trying to use ssl for tcp output plugin for logstash. The workflow goes to a nginx-proxy ssl termination point for logstash that will forward the logs to a logstash pod that takes in tcp input. My output part f… --- ## [Logstash cannot index data from weather API to elasticsearch](https://discuss.elastic.co/t/logstash-cannot-index-data-from-weather-api-to-elasticsearch/182057)
**Author:** [@decodeit](https://discuss.elastic.co/u/decodeit)\ **Replies:** 0\ **Last updated:** [May 21, 2019, 4:00pm UTC](https://discuss.elastic.co/t/logstash-cannot-index-data-from-weather-api-to-elasticsearch/182057 "2019-05-21T16:00:09Z")
I started with docker-compose up I can reach localhost:9200 localhost:5601 localhost:9600 but there is no data in elasticsearch. The error I get is: error=\>"Elasticsearch Unreachable: \[http://localhost:9200/\]\[Mant… --- ## [Cron Schedule](https://discuss.elastic.co/t/cron-schedule/182000)
**Author:** [@decodeit](https://discuss.elastic.co/u/decodeit)\ **Replies:** 4\ **Last updated:** [May 21, 2019, 3:49pm UTC](https://discuss.elastic.co/t/cron-schedule/182000 "2019-05-21T15:49:04Z")
can any one explain what this line does. schedule =\> { cron =\> "\* \* \* \* \* \*" } I am trying to get the weather data pulled from openweathermap.org Thanks --- ## [Extract field from url?](https://discuss.elastic.co/t/extract-field-from-url/181782)
**Author:** [@Orest\_Gulman](https://discuss.elastic.co/u/Orest_Gulman)\ **Replies:** 10\ **Last updated:** [May 21, 2019, 3:46pm UTC](https://discuss.elastic.co/t/extract-field-from-url/181782 "2019-05-21T15:46:29Z")
I want to analyze AWS ELB logs, and I need to grep "instance" word from URL into a separate field, could you please provide an example, how to do it in filter section in logstash. Thanks logs: "GET https://cloude.compa… --- ## [Logstash grok issue while filtering data](https://discuss.elastic.co/t/logstash-grok-issue-while-filtering-data/181941)
**Author:** [@Rocky\_RK](https://discuss.elastic.co/u/Rocky_RK)\ **Replies:** 2\ **Last updated:** [May 21, 2019, 3:35pm UTC](https://discuss.elastic.co/t/logstash-grok-issue-while-filtering-data/181941 "2019-05-21T15:35:01Z")
I have a data that's basically for data deletion via rm command which looks as follows. ttmv516,19/05/21,03:59,00-mins,dvcm,dvcm 166820 4.1 0.0 4212 736 ? DN 03:59 0:01 rm -rf /dv/project/agile/mce\_dev\_folic/test/instal… --- ## [Again this issue : Reached open files limit: 4095, set by the 'max\_open\_files' option or default, files yet to open: 4298](https://discuss.elastic.co/t/again-this-issue-reached-open-files-limit-4095-set-by-the-max-open-files-option-or-default-files-yet-to-open-4298/181363)
**Author:** [@samyo](https://discuss.elastic.co/u/samyo)\ **Replies:** 14\ **Last updated:** [May 21, 2019, 1:13pm UTC](https://discuss.elastic.co/t/again-this-issue-reached-open-files-limit-4095-set-by-the-max-open-files-option-or-default-files-yet-to-open-4298/181363 "2019-05-21T13:13:33Z")
Hello folks, i am using Elasticsearch and Logstash Version 6.3.1 , and Java version 8 , on Ubunto Virtualmachine , with 8GB RAM and 50GB for the Memory . I am trying to index 8000 textfiles after filter them using Logs… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=542) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=544)