# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=630 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 631 --- ## [Help with the grok filter on the custom application log](https://discuss.elastic.co/t/help-with-the-grok-filter-on-the-custom-application-log/161277)
**Author:** [@email2vimalraj](https://discuss.elastic.co/u/email2vimalraj)\ **Replies:** 0\ **Last updated:** [December 18, 2018, 8:14am UTC](https://discuss.elastic.co/t/help-with-the-grok-filter-on-the-custom-application-log/161277 "2018-12-18T08:14:14Z")
Hi, I'm getting logs like this: 2018-12-17 15:10:58,806 INFO AS="234\_sample-app" REQ="1234" com.example.Event - COMP="my-sample-app" Status=200 Sometimes like this also: 2018-12-17 15:10:58,806 INFO AS="" REQ="" … --- ## [Help with nested fields](https://discuss.elastic.co/t/help-with-nested-fields/161255)
**Author:** [@LoZio](https://discuss.elastic.co/u/LoZio)\ **Replies:** 2\ **Last updated:** [December 18, 2018, 7:53am UTC](https://discuss.elastic.co/t/help-with-nested-fields/161255 "2018-12-18T07:53:40Z")
Hi, I searched the forums with no success, hope someone can help me. I have a CSV from a feedback system, that gives me a CSV like this: Field1; Field2; Field3; Field4; Value1; Val2-1,Val2-2; Val3; Val4-1, Val4-2, Val4… --- ## [Hi there, how to prevent lost data when connection lost in streaming data to logstash?](https://discuss.elastic.co/t/hi-there-how-to-prevent-lost-data-when-connection-lost-in-streaming-data-to-logstash/161245)
**Author:** [@shwesinhan](https://discuss.elastic.co/u/shwesinhan)\ **Replies:** 1\ **Last updated:** [December 18, 2018, 7:26am UTC](https://discuss.elastic.co/t/hi-there-how-to-prevent-lost-data-when-connection-lost-in-streaming-data-to-logstash/161245 "2018-12-18T07:26:22Z")
hi there, how to prevent lost data when connection lost in streaming data to logstash? --- ## [How to know in which configuration file we are geeting ERROR](https://discuss.elastic.co/t/how-to-know-in-which-configuration-file-we-are-geeting-error/161256)
**Author:** [@Dan](https://discuss.elastic.co/u/Dan)\ **Replies:** 0\ **Last updated:** [December 18, 2018, 7:21am UTC](https://discuss.elastic.co/t/how-to-know-in-which-configuration-file-we-are-geeting-error/161256 "2018-12-18T07:21:34Z")
Hi All, I am running 2configuration file with input as JDBC.. In one of configuration file I am getting below error due to logon denied. "exception": "Java::JavaSql::SQLException: ORA-01017: invalid username/password;… --- ## [Unable to match DateTime Format in Logstash filter](https://discuss.elastic.co/t/unable-to-match-datetime-format-in-logstash-filter/161253)
**Author:** [@HARSH\_GOYAL](https://discuss.elastic.co/u/HARSH_GOYAL)\ **Replies:** 0\ **Last updated:** [December 18, 2018, 6:59am UTC](https://discuss.elastic.co/t/unable-to-match-datetime-format-in-logstash-filter/161253 "2018-12-18T06:59:44Z")
I am trying to parse the following date using LogStash Date Filter 2018-05-14T16:55:10.613Z date{ match =\> \["time","yyyy-MM-dd'T'HH:mm:ss.SSS'Z'","ISO8601"\] timezone=\>"UTC" target =\> "@timestamp" } Please help on … --- ## [Logstash Custom Patterns is not not working](https://discuss.elastic.co/t/logstash-custom-patterns-is-not-not-working/161242)
**Author:** [@Surya\_Vamsi\_Pappu](https://discuss.elastic.co/u/Surya_Vamsi_Pappu)\ **Replies:** 0\ **Last updated:** [December 18, 2018, 5:49am UTC](https://discuss.elastic.co/t/logstash-custom-patterns-is-not-not-working/161242 "2018-12-18T05:49:02Z")
I have defined the following grok filter : filter{ if\[message\] !~ ".\*sub-page-type=\[-a-zA-Z0-9\_\]+" { drop{} } else { patterns\_dir =\> \["{env path}/patternCustom"\] match =\>{ "message" =\> ".\*StartTime=%{NUMBER:Sta… --- ## [Logstash broken logs when log message contains \ \ tags](https://discuss.elastic.co/t/logstash-broken-logs-when-log-message-contains-div-div-tags/161151)
**Author:** [@Kent\_Ridge](https://discuss.elastic.co/u/Kent_Ridge)\ **Replies:** 0\ **Last updated:** [December 17, 2018, 12:59pm UTC](https://discuss.elastic.co/t/logstash-broken-logs-when-log-message-contains-div-div-tags/161151 "2018-12-17T12:59:16Z")
Hi, I have a HTML string that contains "hello\< div \>world\< /div \>". This string is supposed to be picked up at logstash and stored into Elastic Search. Noticed that logstash picks up this string in two different messag… --- ## [Logstash not pushing past Queued data to AWS elasticsearch but is able to send current logs being pushed into](https://discuss.elastic.co/t/logstash-not-pushing-past-queued-data-to-aws-elasticsearch-but-is-able-to-send-current-logs-being-pushed-into/161232)
**Author:** [@ravikumaEdifecs](https://discuss.elastic.co/u/ravikumaEdifecs)\ **Replies:** 0\ **Last updated:** [December 18, 2018, 2:02am UTC](https://discuss.elastic.co/t/logstash-not-pushing-past-queued-data-to-aws-elasticsearch-but-is-able-to-send-current-logs-being-pushed-into/161232 "2018-12-18T02:02:04Z")
ELS versions : ES 5.5 as service on AWS and Logstash 5.5.3 on EC2 instance ES had issue with heapspace due to which Queue in Logstash got filled up, After we upgraded ES from small to medium instance Logstash is able to… --- ## [\[JRuby\] Additional ruby module](https://discuss.elastic.co/t/jruby-additional-ruby-module/161222)
**Author:** [@drosys](https://discuss.elastic.co/u/drosys)\ **Replies:** 0\ **Last updated:** [December 17, 2018, 11:16pm UTC](https://discuss.elastic.co/t/jruby-additional-ruby-module/161222 "2018-12-17T23:16:45Z")
Dear All, I d like to calculate hiversine distance (nbr of kilometer between 2 geo points) To implement this I need to use "Math" module. I have tried to check the availability with: $ /usr/share/logstash/bin/logsta… --- ## [Logstash grokparsefailure when multiple IF ELSE in use](https://discuss.elastic.co/t/logstash-grokparsefailure-when-multiple-if-else-in-use/161206)
**Author:** [@nazgul](https://discuss.elastic.co/u/nazgul)\ **Replies:** 1\ **Last updated:** [December 17, 2018, 9:58pm UTC](https://discuss.elastic.co/t/logstash-grokparsefailure-when-multiple-if-else-in-use/161206 "2018-12-17T21:58:38Z")
Hello experts, Please help me to figure out where the problem is and what can I try more. filter { if "...." in \[......\] { ... } if "test" in \[tags\] \<--- true in my case { if "syl1dsaws" in \[message\] { grok { … --- ## [Timestamp field with local timezone!](https://discuss.elastic.co/t/timestamp-field-with-local-timezone/160238)
**Author:** [@alexwbai](https://discuss.elastic.co/u/alexwbai)\ **Replies:** 3\ **Last updated:** [December 17, 2018, 6:04pm UTC](https://discuss.elastic.co/t/timestamp-field-with-local-timezone/160238 "2018-12-17T18:04:18Z")
Sorry I know these timezone questions have been beaten to death but I'm really having trouble finding the solution to this problem: This is what I want to do: Take existing epoch time integer field (eg. 1544471514) an… --- ## [Using Logstash to index MQTT from Mosquitto](https://discuss.elastic.co/t/using-logstash-to-index-mqtt-from-mosquitto/161181)
**Author:** [@stornian](https://discuss.elastic.co/u/stornian)\ **Replies:** 0\ **Last updated:** [December 17, 2018, 2:50pm UTC](https://discuss.elastic.co/t/using-logstash-to-index-mqtt-from-mosquitto/161181 "2018-12-17T14:50:45Z")
Hello, this is my first time to use MQTT. My goal is to use the mqtt input plugin (https://github.com/jurek7/logstash-input-mqtt) My logstash configuration is: I installed Mosquitto and run it with default settings. … --- ## [Logstash appends port 9200 to Elasticsearch URL](https://discuss.elastic.co/t/logstash-appends-port-9200-to-elasticsearch-url/161154)
**Author:** [@patrick007](https://discuss.elastic.co/u/patrick007)\ **Replies:** 4\ **Last updated:** [December 17, 2018, 2:38pm UTC](https://discuss.elastic.co/t/logstash-appends-port-9200-to-elasticsearch-url/161154 "2018-12-17T14:38:25Z")
I am using AWS elasticsearch service , which return me a URL for Elasticsearch. Example : https:// www.es-aws.com // ES Url When I try to use the URL in logstash, it tries to connect https:// www.es-aws.com:9200 inst… --- ## [Logstash JSON GROK Parsing](https://discuss.elastic.co/t/logstash-json-grok-parsing/161162)
**Author:** [@Kenny\_H](https://discuss.elastic.co/u/Kenny_H)\ **Replies:** 0\ **Last updated:** [December 17, 2018, 1:53pm UTC](https://discuss.elastic.co/t/logstash-json-grok-parsing/161162 "2018-12-17T13:53:10Z")
I am new to Logstash and Grok with Non Developer background. Can anyone help me with GROK parsing for below. { "source": "logs", "id": "9ab3ce4e-fff7-11e8-be78-06da4fe50a90", "recorded": "2018-14-12T23:25:55.228Z", … --- ## [Logstash6.5: It seems your ruby installation is missing psych](https://discuss.elastic.co/t/logstash6-5-it-seems-your-ruby-installation-is-missing-psych/160062)
**Author:** [@kevink00](https://discuss.elastic.co/u/kevink00)\ **Replies:** 1\ **Last updated:** [December 17, 2018, 1:34pm UTC](https://discuss.elastic.co/t/logstash6-5-it-seems-your-ruby-installation-is-missing-psych/160062 "2018-12-17T13:34:57Z")
Following steps exactly as noted here: https://www.elastic.co/guide/en/logstash/current/getting-started-with-logstash.html I'm on CentOS 7.5 (3.10.0-862.14.4.el7.x86\_64). New OS installed. I've installed elasticsearch,… --- ## [Aggregate syslog with Logstash](https://discuss.elastic.co/t/aggregate-syslog-with-logstash/161157)
**Author:** [@drjackal](https://discuss.elastic.co/u/drjackal)\ **Replies:** 0\ **Last updated:** [December 17, 2018, 1:32pm UTC](https://discuss.elastic.co/t/aggregate-syslog-with-logstash/161157 "2018-12-17T13:32:23Z")
Hi All, I am working on getting the logs from our smtp gateway into elasticsearch and use logstash to parse the separate syslog events. Now, unfortunately the gateway events are kind of tricky. Each event has a differe… --- ## [How to store Database date format is same as to Elastic Index through logstash](https://discuss.elastic.co/t/how-to-store-database-date-format-is-same-as-to-elastic-index-through-logstash/161150)
**Author:** [@karthick\_tgi](https://discuss.elastic.co/u/karthick_tgi)\ **Replies:** 0\ **Last updated:** [December 17, 2018, 12:58pm UTC](https://discuss.elastic.co/t/how-to-store-database-date-format-is-same-as-to-elastic-index-through-logstash/161150 "2018-12-17T12:58:51Z")
How to store Database date format in same as to Elastic Index through logstash. My Database date format : (YYYY-MM-DD) What ever i change the date format it could stored only GMT time zone. I changed the timezone li… --- ## [Logstash CPU utilisation - Aure Bolb Plugin](https://discuss.elastic.co/t/logstash-cpu-utilisation-aure-bolb-plugin/161140)
**Author:** [@charan.gandra](https://discuss.elastic.co/u/charan.gandra)\ **Replies:** 0\ **Last updated:** [December 17, 2018, 12:16pm UTC](https://discuss.elastic.co/t/logstash-cpu-utilisation-aure-bolb-plugin/161140 "2018-12-17T12:16:49Z")
Hi, I am using LS to read NSG flow logs form Azure using Azure blob plugin. My Azure VM is F8 Size i.e. 64GB RAM and 8 CPUs. However CPU utilization is always over 100%. My filter is exactly the same as provided in Azur… --- ## [Pipeline Raise Exception from Filter](https://discuss.elastic.co/t/pipeline-raise-exception-from-filter/160013)
**Author:** [@Belly](https://discuss.elastic.co/u/Belly)\ **Replies:** 4\ **Last updated:** [December 17, 2018, 12:03pm UTC](https://discuss.elastic.co/t/pipeline-raise-exception-from-filter/160013 "2018-12-17T12:03:01Z")
Dear All, I've an issue with the pipeline that it keep raise on exception because of filter issue. If I restarted the Logstash it work fine until it raise exception after sometime I'm not sure exactly what could be th… --- ## [Logstash drop data](https://discuss.elastic.co/t/logstash-drop-data/161123)
**Author:** [@jmukhtar](https://discuss.elastic.co/u/jmukhtar)\ **Replies:** 0\ **Last updated:** [December 17, 2018, 10:47am UTC](https://discuss.elastic.co/t/logstash-drop-data/161123 "2018-12-17T10:47:21Z")
Hi Guys I am having an issue with the logstash drop filter. I am getting 2 types of data via filebeat from couple of servers, and i want to drop anything that is with json type. In filebeat, I have following config -… --- ## [Can build logstash input are API?](https://discuss.elastic.co/t/can-build-logstash-input-are-api/160500)
**Author:** [@keep](https://discuss.elastic.co/u/keep)\ **Replies:** 6\ **Last updated:** [December 17, 2018, 9:39am UTC](https://discuss.elastic.co/t/can-build-logstash-input-are-api/160500 "2018-12-17T09:39:43Z")
Can build logstash input are API for waiting log there are samples? --- ## [SNMP(network, storage) - CPU, Fan status information](https://discuss.elastic.co/t/snmp-network-storage-cpu-fan-status-information/161090)
**Author:** [@dh77hd](https://discuss.elastic.co/u/dh77hd)\ **Replies:** 0\ **Last updated:** [December 17, 2018, 8:22am UTC](https://discuss.elastic.co/t/snmp-network-storage-cpu-fan-status-information/161090 "2018-12-17T08:22:39Z")
Hi, I have a question about snmp not snmp trap. I want to use snmp input to collect network, storage cpu, fan status. Network is CISCO n9k, n3k, mds and Storage is Netapp. (That system is already set snmp.) I made MI… --- ## [Why my log gets to another index](https://discuss.elastic.co/t/why-my-log-gets-to-another-index/160851)
**Author:** [@04535f91a01ed79f8d98](https://discuss.elastic.co/u/04535f91a01ed79f8d98)\ **Replies:** 2\ **Last updated:** [December 17, 2018, 8:29am UTC](https://discuss.elastic.co/t/why-my-log-gets-to-another-index/160851 "2018-12-17T08:29:06Z")
Hi everyone, can someone tell me why my log in the "terminal" type is in the index fr- \* ? Filebeat: https://pastebin.com/zB1cefRe Logstash conf files: https://pastebin.com/BWUBVuLy https://pastebin.com/hms3fMGz T… --- ## [Logstash Startup&Configuration error](https://discuss.elastic.co/t/logstash-startup-configuration-error/160829)
**Author:** [@Kavin\_Kumaran](https://discuss.elastic.co/u/Kavin_Kumaran)\ **Replies:** 1\ **Last updated:** [December 17, 2018, 8:20am UTC](https://discuss.elastic.co/t/logstash-startup-configuration-error/160829 "2018-12-17T08:20:06Z")
Hi, Getting the below mentioned error while starting the logstash can anyone help to resolve this issue.Thanks in advance Filter.conf: // filter { \*\* if \[type\] == "tomcat"{\*\* \*\* codec =\> multiline {\*\* … --- ## [Nexsus log grok parse](https://discuss.elastic.co/t/nexsus-log-grok-parse/161065)
**Author:** [@dh77hd](https://discuss.elastic.co/u/dh77hd)\ **Replies:** 2\ **Last updated:** [December 17, 2018, 7:57am UTC](https://discuss.elastic.co/t/nexsus-log-grok-parse/161065 "2018-12-17T07:57:25Z")
Hi. I wanna parse Nexsus log using grok filter. But, it doesn't work. please help me. Here is my log \<189\>: 2018 Dec 17 11:33:04 KST: %ETHPORT-5-IF\_ADMIN\_UP: Interface Ethernet1/40 is admin up . and I want to parse … --- ## [Why second grok fails?](https://discuss.elastic.co/t/why-second-grok-fails/161086)
**Author:** [@jamespb](https://discuss.elastic.co/u/jamespb)\ **Replies:** 0\ **Last updated:** [December 17, 2018, 7:46am UTC](https://discuss.elastic.co/t/why-second-grok-fails/161086 "2018-12-17T07:46:21Z")
Tried many way still fail. Thanks a lot for helping filter { grok { match =\> {"message" =\> \["(?\^\[0-9\]+-\[0-9\]+-\[0-9\]+)%{SPACE}(?\\[0-9\]+:\[0-9\]+:\[0-9\]+)%{SPACE}(?\\[0-9\]+-\[0-9\]+)%… --- ## [Reindex multiple matchind documents into a single document?](https://discuss.elastic.co/t/reindex-multiple-matchind-documents-into-a-single-document/161077)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 1\ **Last updated:** [December 17, 2018, 6:33am UTC](https://discuss.elastic.co/t/reindex-multiple-matchind-documents-into-a-single-document/161077 "2018-12-17T06:33:34Z")
Hi, I was wondering whether there is a practical way of querying existing documents in an index, find matching documents, and than write that that as a single document to a new index? Example: I'm collecting network t… --- ## [Please help, I cannot add two tags without replacing the tag that added first](https://discuss.elastic.co/t/please-help-i-cannot-add-two-tags-without-replacing-the-tag-that-added-first/160911)
**Author:** [@Harshet\_Jain](https://discuss.elastic.co/u/Harshet_Jain)\ **Replies:** 2\ **Last updated:** [December 17, 2018, 5:26am UTC](https://discuss.elastic.co/t/please-help-i-cannot-add-two-tags-without-replacing-the-tag-that-added-first/160911 "2018-12-17T05:26:24Z")
I have tried the following code but it does not work. event.set("\[tags\]", event.get("\[tags\]").push("PSY")) if event.get("\[NAME\]") != "ALPHA" event.set("\[tags\]", event.get("\[tags\]").push("OUR")) if event.get("\[NAME\]"… --- ## [Log Parsing](https://discuss.elastic.co/t/log-parsing/159098)
**Author:** [@Vineeth\_Ananthula](https://discuss.elastic.co/u/Vineeth_Ananthula)\ **Replies:** 7\ **Last updated:** [December 17, 2018, 3:44am UTC](https://discuss.elastic.co/t/log-parsing/159098 "2018-12-17T03:44:12Z")
I am collecting hundred types of router logs from a syslog server . I was able to write grok pattern for few of them and feed them into elasticsearch and kibana. But its becoming a tedious job to write for each type of l… --- ## [Connecting Logstash to MSSQL without hardcording the password in configuration file](https://discuss.elastic.co/t/connecting-logstash-to-mssql-without-hardcording-the-password-in-configuration-file/160567)
**Author:** [@Lal\_Prasad\_Rajendrap](https://discuss.elastic.co/u/Lal_Prasad_Rajendrap)\ **Replies:** 1\ **Last updated:** [December 16, 2018, 8:52pm UTC](https://discuss.elastic.co/t/connecting-logstash-to-mssql-without-hardcording-the-password-in-configuration-file/160567 "2018-12-16T20:52:26Z")
Hi All, Is there any way to connect Logstash with to MSSQL DB, without hard-cording the password in configuration file, since this is a security risk. Are there any options of encrypting the password etc to overcome th… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=629) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=631)