# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=594 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 595 --- ## [Send Logstash internal logs to Syslog](https://discuss.elastic.co/t/send-logstash-internal-logs-to-syslog/169299)
**Author:** [@petrik](https://discuss.elastic.co/u/petrik)\ **Replies:** 2\ **Last updated:** [February 25, 2019, 7:00pm UTC](https://discuss.elastic.co/t/send-logstash-internal-logs-to-syslog/169299 "2019-02-25T19:00:41Z")
We're wanting to send Logstash's own logs to syslog instead of to a file unfortunately there is very little information about what the log configuration should look like for this approach. So far what we've come up with … --- ## [Mongo Output Plugin For Logstash](https://discuss.elastic.co/t/mongo-output-plugin-for-logstash/169891)
**Author:** [@darsan55](https://discuss.elastic.co/u/darsan55)\ **Replies:** 0\ **Last updated:** [February 25, 2019, 6:15pm UTC](https://discuss.elastic.co/t/mongo-output-plugin-for-logstash/169891 "2019-02-25T18:15:04Z")
Hi, I am using Mongo Output Plugin in Logstash. I want to know is there anyway i can update the Doc in Mongo coll or delete using some condition. I am getting data using JDBC plugin and mongo is my output plugin. Its… --- ## [How to Parse JSON Input using Filter Plugins?](https://discuss.elastic.co/t/how-to-parse-json-input-using-filter-plugins/169836)
**Author:** [@pavan1991](https://discuss.elastic.co/u/pavan1991)\ **Replies:** 5\ **Last updated:** [February 25, 2019, 5:35pm UTC](https://discuss.elastic.co/t/how-to-parse-json-input-using-filter-plugins/169836 "2019-02-25T17:35:09Z")
Team, I am New to Elastic Stack and need your help to setup the same in my environment. Setup : We are planning to disable Stackdriver logging in our Google Cloud Platform and instead we want to use ELK to display thes… --- ## [Logstash parsed the date, and saved as string](https://discuss.elastic.co/t/logstash-parsed-the-date-and-saved-as-string/169883)
**Author:** [@scorpioy1](https://discuss.elastic.co/u/scorpioy1)\ **Replies:** 2\ **Last updated:** [February 25, 2019, 5:35pm UTC](https://discuss.elastic.co/t/logstash-parsed-the-date-and-saved-as-string/169883 "2019-02-25T17:35:04Z")
Hi, here is my logstash filter { csv { columns =\> \["nowdate","Total","DIFF"\] convert =\> { "Total" =\> "integer" "DIFF" =\> "integer" } } date { match =\> \[ "nowdate", "yyyy-mm-dd HH:mm:ss.SSS", "ISO8601" \] } W… --- ## [Can logstash lookup an ID from ingested logs and replace it with its corresponding data from a database table?](https://discuss.elastic.co/t/can-logstash-lookup-an-id-from-ingested-logs-and-replace-it-with-its-corresponding-data-from-a-database-table/169868)
**Author:** [@jrplus](https://discuss.elastic.co/u/jrplus)\ **Replies:** 1\ **Last updated:** [February 25, 2019, 4:46pm UTC](https://discuss.elastic.co/t/can-logstash-lookup-an-id-from-ingested-logs-and-replace-it-with-its-corresponding-data-from-a-database-table/169868 "2019-02-25T16:46:29Z")
as above. can logstash do this kind of relational database FK=PK kind of thing? --- ## [Transform each array item (nested object) in a new document - Splitting a field before removing it](https://discuss.elastic.co/t/transform-each-array-item-nested-object-in-a-new-document-splitting-a-field-before-removing-it/169851)
**Author:** [@Lucas\_Porto](https://discuss.elastic.co/u/Lucas_Porto)\ **Replies:** 7\ **Last updated:** [February 25, 2019, 4:43pm UTC](https://discuss.elastic.co/t/transform-each-array-item-nested-object-in-a-new-document-splitting-a-field-before-removing-it/169851 "2019-02-25T16:43:54Z")
Hi all, I have a index with a nested field like this: { "id\_single\_profile": "14e98c40-d5ed-11e8-aa2a-d991a7f7e009", "id\_audience": "3c9b46e4-1b10-4ad4-9a68-ae371034adfe", "activities": \[ … --- ## [FB \> Logstash \> Kafka](https://discuss.elastic.co/t/fb-logstash-kafka/169635)
**Author:** [@Kevin\_Comer](https://discuss.elastic.co/u/Kevin_Comer)\ **Replies:** 5\ **Last updated:** [February 25, 2019, 3:23pm UTC](https://discuss.elastic.co/t/fb-logstash-kafka/169635 "2019-02-25T15:23:19Z")
We have an issue where Kafka does not receive full json message, we can see the json messages and tags in output to file in logstash but when we view the message in kafka it is missing the filebeat header appended. We h… --- ## [Filter by timestamp - re-ingesting logs from Kafka](https://discuss.elastic.co/t/filter-by-timestamp-re-ingesting-logs-from-kafka/169844)
**Author:** [@carrot](https://discuss.elastic.co/u/carrot)\ **Replies:** 1\ **Last updated:** [February 25, 2019, 2:53pm UTC](https://discuss.elastic.co/t/filter-by-timestamp-re-ingesting-logs-from-kafka/169844 "2019-02-25T14:53:36Z")
We had a problem with our cluster this weekend (index went read-only) and looks like the OFFSET expired in Kafka or something similar. The following message appeared in our logstash logs regarding KafkaINVALID\_FETCH\_SESS… --- ## [Usage of grok filter](https://discuss.elastic.co/t/usage-of-grok-filter/169850)
**Author:** [@erdo](https://discuss.elastic.co/u/erdo)\ **Replies:** 2\ **Last updated:** [February 25, 2019, 2:47pm UTC](https://discuss.elastic.co/t/usage-of-grok-filter/169850 "2019-02-25T14:47:46Z")
Tying to use grok to add regex part of the XML as a field. grok { patterns\_dir =\> \["tns:.+?\>"\] match =\> { "message" =\> "methodName:"} } } My config: input { file { path =\> "C:\\Users… --- ## [Parse syslog from Syslog](https://discuss.elastic.co/t/parse-syslog-from-syslog/168291)
**Author:** [@Digital](https://discuss.elastic.co/u/Digital)\ **Replies:** 11\ **Last updated:** [February 25, 2019, 2:42pm UTC](https://discuss.elastic.co/t/parse-syslog-from-syslog/168291 "2019-02-25T14:42:53Z")
Hi all, I am new in ELK solution :smiley: and currently I am working on Logstash -\> Elasticsearch -\> Kibana. I send the syslog from a checkpoint to logstash however I would like to parse correctly the syslog with logs… --- ## [Grok custom patterns](https://discuss.elastic.co/t/grok-custom-patterns/169778)
**Author:** [@Mayurbiw](https://discuss.elastic.co/u/Mayurbiw)\ **Replies:** 1\ **Last updated:** [February 25, 2019, 2:14pm UTC](https://discuss.elastic.co/t/grok-custom-patterns/169778 "2019-02-25T14:14:44Z")
when I define the grok pattern like this one (?someoperation) is the string someoperation is case sensitive or case insensitive. In other words, would it match if a file contains something like someOPERATION ? --- ## [How to parse mix json logs](https://discuss.elastic.co/t/how-to-parse-mix-json-logs/167594)
**Author:** [@amolp](https://discuss.elastic.co/u/amolp)\ **Replies:** 27\ **Last updated:** [February 25, 2019, 1:59pm UTC](https://discuss.elastic.co/t/how-to-parse-mix-json-logs/167594 "2019-02-25T13:59:54Z")
hi, i am very new to elk.so i do not know how to parse the mixed json logs here is sample log that i want parse,i getting these logs from filebeat. 2019-02-03 23:51:54,263 | {" MACID":"00009934","ID":"1","SS":"26","FW"… --- ## [How to use custom field in dns filter?](https://discuss.elastic.co/t/how-to-use-custom-field-in-dns-filter/169809)
**Author:** [@Wind2dk](https://discuss.elastic.co/u/Wind2dk)\ **Replies:** 1\ **Last updated:** [February 25, 2019, 1:55pm UTC](https://discuss.elastic.co/t/how-to-use-custom-field-in-dns-filter/169809 "2019-02-25T13:55:26Z")
I'm working on a filter to resolve hostnames in ip fields. Since there is multiple ip fields, i've added a metadata field, that i can later use to resolve the actual field. If i use resolve =\> "ip\_field\_1" it correctly … --- ## [Logstash Netflow Module on Debian - Elasticsearch index not found: netflow-\*](https://discuss.elastic.co/t/logstash-netflow-module-on-debian-elasticsearch-index-not-found-netflow/169425)
**Author:** [@falc410](https://discuss.elastic.co/u/falc410)\ **Replies:** 1\ **Last updated:** [February 25, 2019, 1:45pm UTC](https://discuss.elastic.co/t/logstash-netflow-module-on-debian-elasticsearch-index-not-found-netflow/169425 "2019-02-25T13:45:06Z")
I installed a Debian 9 Machine, installed OpenJDK8 and then followed the Quick Start Guide on how to setup Elasticsearch, then Kibana and finally wanted to install Logstash to pump some Netflow data in ELK and see what i… --- ## [How to migrate parent child relation data from RDBMS, through logstash to Elastic?](https://discuss.elastic.co/t/how-to-migrate-parent-child-relation-data-from-rdbms-through-logstash-to-elastic/169819)
**Author:** [@prasad1986](https://discuss.elastic.co/u/prasad1986)\ **Replies:** 0\ **Last updated:** [February 25, 2019, 11:48am UTC](https://discuss.elastic.co/t/how-to-migrate-parent-child-relation-data-from-rdbms-through-logstash-to-elastic/169819 "2019-02-25T11:48:56Z")
I created Parent and child relation through Kibana tool. Index :demosampleindex documenttype: itemdoctype Parent Relation:itemjoin child Relation :itemjoin Syntax: PUT demosampleindex { "settings": { "number\_of\_… --- ## [Log clean up on logstash servers](https://discuss.elastic.co/t/log-clean-up-on-logstash-servers/169810)
**Author:** [@npalanis](https://discuss.elastic.co/u/npalanis)\ **Replies:** 0\ **Last updated:** [February 25, 2019, 11:20am UTC](https://discuss.elastic.co/t/log-clean-up-on-logstash-servers/169810 "2019-02-25T11:20:35Z")
Hi team, Here I come up with some serious problem that we mainly facing on our daily devops task related to ELK stack. Recent days few of our ELK machine's disk space level getting filled very often and every time cle… --- ## [Logstash output to nginx](https://discuss.elastic.co/t/logstash-output-to-nginx/169785)
**Author:** [@arp220](https://discuss.elastic.co/u/arp220)\ **Replies:** 1\ **Last updated:** [February 25, 2019, 11:19am UTC](https://discuss.elastic.co/t/logstash-output-to-nginx/169785 "2019-02-25T11:19:28Z")
hi everybody. I have 3 node and run cluster graylog-server. I use nginx for load balancer and on nginx open port (12301). i want to send log JSON format from logstash to this nginx then nginx, load balancer and send t… --- ## [How can I make logstash work on folders named with VM hostnames on latest 5 log files every one hour?](https://discuss.elastic.co/t/how-can-i-make-logstash-work-on-folders-named-with-vm-hostnames-on-latest-5-log-files-every-one-hour/169792)
**Author:** [@Neeraj\_Jain](https://discuss.elastic.co/u/Neeraj_Jain)\ **Replies:** 0\ **Last updated:** [February 25, 2019, 9:48am UTC](https://discuss.elastic.co/t/how-can-i-make-logstash-work-on-folders-named-with-vm-hostnames-on-latest-5-log-files-every-one-hour/169792 "2019-02-25T09:48:51Z")
I plan to use ELK stack for analytics of my application logs. My application runs on openstack environment which has different VMs handling application logic. The application logs and counters from each of the VMs is a… --- ## [Elastic search and logstash performance tuning](https://discuss.elastic.co/t/elastic-search-and-logstash-performance-tuning/169422)
**Author:** [@fadihaddad](https://discuss.elastic.co/u/fadihaddad)\ **Replies:** 1\ **Last updated:** [February 25, 2019, 9:32am UTC](https://discuss.elastic.co/t/elastic-search-and-logstash-performance-tuning/169422 "2019-02-25T09:32:37Z")
hello, i have been using configuring logstash for tuning and i need help. i set the JVM heap of elasticsearch to 8gb and logstash JVM heap to 6gb my server has 4 cpus, 4 cores each 2.2ghz and 16gb RAM. i set the number… --- ## [LS-ES connection issue](https://discuss.elastic.co/t/ls-es-connection-issue/169205)
**Author:** [@Kanthasamyraja](https://discuss.elastic.co/u/Kanthasamyraja)\ **Replies:** 17\ **Last updated:** [February 25, 2019, 8:56am UTC](https://discuss.elastic.co/t/ls-es-connection-issue/169205 "2019-02-25T08:56:20Z")
Hi, Unable to start Logstash after xpack enabled. Exception Unable to retrieve license information from license server {:message=\>"Got response code '401' contacting Elasticsearch at URL 'http://HOSTNAME:9200/\_xpack'"… --- ## [Unacceptable Logstash startup times](https://discuss.elastic.co/t/unacceptable-logstash-startup-times/169708)
**Author:** [@rcowart](https://discuss.elastic.co/u/rcowart)\ **Replies:** 5\ **Last updated:** [February 24, 2019, 9:26pm UTC](https://discuss.elastic.co/t/unacceptable-logstash-startup-times/169708 "2019-02-24T21:26:11Z")
There has been a disturbing trend with Logstash performance and stability. Including stability issues (6.1.3 is the last version that can run a long time on MacOS without crashing), the 6.6.x containers crash within minu… --- ## [Logstash unable to index due to multiple doc types in mapping](https://discuss.elastic.co/t/logstash-unable-to-index-due-to-multiple-doc-types-in-mapping/168944)
**Author:** [@bbking](https://discuss.elastic.co/u/bbking)\ **Replies:** 1\ **Last updated:** [February 24, 2019, 8:18pm UTC](https://discuss.elastic.co/t/logstash-unable-to-index-due-to-multiple-doc-types-in-mapping/168944 "2019-02-24T20:18:44Z")
Hi, I manually put an index template in Elasticsearch with doc\_type doc. But when Logstash tries to index an event using this template. it shows failed to index due to multiple doc types. I think the template I defined i… --- ## [Logstash jdbc\_streaming, sql statement](https://discuss.elastic.co/t/logstash-jdbc-streaming-sql-statement/169736)
**Author:** [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Replies:** 0\ **Last updated:** [February 24, 2019, 5:36pm UTC](https://discuss.elastic.co/t/logstash-jdbc-streaming-sql-statement/169736 "2019-02-24T17:36:41Z")
hi all, I am using following logstash.conf file: input { jdbc { jdbc\_driver\_library =\> "F:\\driver\\sqljdbc\_6.0\\enu\\jre8\\sqljdbc42.jar" jdbc\_driver\_class =\> "com.microsoft.sqlserver.jdbc.SQLServerDriver" jdbc… --- ## [Logstash jdbc plugin sql\_last\_value](https://discuss.elastic.co/t/logstash-jdbc-plugin-sql-last-value/169704)
**Author:** [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Replies:** 1\ **Last updated:** [February 24, 2019, 3:57pm UTC](https://discuss.elastic.co/t/logstash-jdbc-plugin-sql-last-value/169704 "2019-02-24T15:57:53Z")
hi all, I am using logstash jdbc plugin as following: input { jdbc { jdbc\_driver\_library =\> "F:\\driver\\sqljdbc\_6.0\\enu\\jre8\\sqljdbc42.jar" jdbc\_driver\_class =\> "com.microsoft.sqlserver.jdbc.SQLServerDriver" … --- ## [Relational inputs of logstash in jdbc plugin](https://discuss.elastic.co/t/relational-inputs-of-logstash-in-jdbc-plugin/169724)
**Author:** [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Replies:** 6\ **Last updated:** [February 24, 2019, 3:40pm UTC](https://discuss.elastic.co/t/relational-inputs-of-logstash-in-jdbc-plugin/169724 "2019-02-24T15:40:55Z")
hi all, i am using following configuration in logstash: input { jdbc { jdbc\_driver\_library =\> "F:\\driver\\sqljdbc\_6.0\\enu\\jre8\\sqljdbc42.jar" jdbc\_driver\_class =\> "com.microsoft.sqlserver.jdbc.SQLServerDriver" … --- ## [Trying to add two extra fields to my indexed documents](https://discuss.elastic.co/t/trying-to-add-two-extra-fields-to-my-indexed-documents/169731)
**Author:** [@rwl](https://discuss.elastic.co/u/rwl)\ **Replies:** 0\ **Last updated:** [February 24, 2019, 3:00pm UTC](https://discuss.elastic.co/t/trying-to-add-two-extra-fields-to-my-indexed-documents/169731 "2019-02-24T15:00:16Z")
Hey guys, I haven't been using the elastic stack for long, I'm still very much learning the ropes. I use an elasticsearch+logstash+kibana configuration and filebeat for monitoring a server log. I want to add two fields… --- ## [Logstash Pipelines - logstash.yml reloadable?](https://discuss.elastic.co/t/logstash-pipelines-logstash-yml-reloadable/169727)
**Author:** [@Wayne\_Taylor](https://discuss.elastic.co/u/Wayne_Taylor)\ **Replies:** 0\ **Last updated:** [February 24, 2019, 2:05pm UTC](https://discuss.elastic.co/t/logstash-pipelines-logstash-yml-reloadable/169727 "2019-02-24T14:05:36Z")
Hi Logstash Users, I have started playing with Logstash Pipeline and love the feature. I've got it working and understand that certain elements are not reloabable e.g. if you use stdin. However, I noticed that this req… --- ## [Dissecting an array of variable length](https://discuss.elastic.co/t/dissecting-an-array-of-variable-length/169681)
**Author:** [@Michael.Swiss](https://discuss.elastic.co/u/Michael.Swiss)\ **Replies:** 2\ **Last updated:** [February 24, 2019, 10:54am UTC](https://discuss.elastic.co/t/dissecting-an-array-of-variable-length/169681 "2019-02-24T10:54:30Z")
Hi there I am trying to dissect an array of variable length. That means it can contain up to 5 entries. Using the dissect plugin however requires me to indicate a fixed length and results in an \[0\] "\_dissectfailure" re… --- ## [CIDR filter network\_path error](https://discuss.elastic.co/t/cidr-filter-network-path-error/169684)
**Author:** [@jpbourke](https://discuss.elastic.co/u/jpbourke)\ **Replies:** 3\ **Last updated:** [February 24, 2019, 8:35am UTC](https://discuss.elastic.co/t/cidr-filter-network-path-error/169684 "2019-02-24T08:35:50Z")
Hi, logstash-6.5.3-1.noarch on RHEL I get the following error when using the cidr filter with network\_path \[2019-02-23T21:30:16,297\]\[ERROR\]\[logstash.pipeline \] Exception in pipelineworker, the pipeline stopped … --- ## [Match a string and then assign this string to the output field](https://discuss.elastic.co/t/match-a-string-and-then-assign-this-string-to-the-output-field/169698)
**Author:** [@ozmhsh](https://discuss.elastic.co/u/ozmhsh)\ **Replies:** 0\ **Last updated:** [February 24, 2019, 3:29am UTC](https://discuss.elastic.co/t/match-a-string-and-then-assign-this-string-to-the-output-field/169698 "2019-02-24T03:29:33Z")
I have a requirement to match a string out the log, say 'ABC123' or 'XYZ456'. Once this is identified, I want to assign this string into the output field. How can I do it? I can obviously use add\_field with the hardcode… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=593) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=595)