# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=691 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 692 --- ## [How to create index in output by message content](https://discuss.elastic.co/t/how-to-create-index-in-output-by-message-content/146812)
**Author:** [@yangmengxiang](https://discuss.elastic.co/u/yangmengxiang)\ **Replies:** 1\ **Last updated:** [September 4, 2018, 6:46am UTC](https://discuss.elastic.co/t/how-to-create-index-in-output-by-message-content/146812 "2018-09-04T06:46:03Z")
how to create index in output by message content --- ## [Metricbeat - Whitelist Processes from include\_top\_n](https://discuss.elastic.co/t/metricbeat-whitelist-processes-from-include-top-n/146885)
**Author:** [@KRasekhi](https://discuss.elastic.co/u/KRasekhi)\ **Replies:** 1\ **Last updated:** [September 4, 2018, 6:44am UTC](https://discuss.elastic.co/t/metricbeat-whitelist-processes-from-include-top-n/146885 "2018-09-04T06:44:21Z")
Is there anyway to whitelist processes to be collected in addition to - process.include\_top\_n: by\_cpu: 10 by\_memory: 10 without having to create an additional system module? --- ## [Logstash not fetching conf file during startup](https://discuss.elastic.co/t/logstash-not-fetching-conf-file-during-startup/147051)
**Author:** [@anokpe](https://discuss.elastic.co/u/anokpe)\ **Replies:** 4\ **Last updated:** [September 4, 2018, 6:12am UTC](https://discuss.elastic.co/t/logstash-not-fetching-conf-file-during-startup/147051 "2018-09-04T06:12:36Z")
OS : ubuntu 16.04 Logstash version : 6.4 I have recently installed logstash with apt package repository. Every thing is as default, I have added /etc/logstash/conf.d/filebeat.conf where it should listen to port 5044 f… --- ## [JSON Parse an already parsed field](https://discuss.elastic.co/t/json-parse-an-already-parsed-field/147006)
**Author:** [@jibsonline](https://discuss.elastic.co/u/jibsonline)\ **Replies:** 10\ **Last updated:** [September 4, 2018, 6:11am UTC](https://discuss.elastic.co/t/json-parse-an-already-parsed-field/147006 "2018-09-04T06:11:05Z")
I have the following field parsed using JSON filter json { source =\> "log" target =\> "request" skip\_on\_invalid\_json =\> true } But there is a field in "request" which still prints as JSON S… --- ## [Date format not working in logstash 6.3](https://discuss.elastic.co/t/date-format-not-working-in-logstash-6-3/147111)
**Author:** [@sachinvy](https://discuss.elastic.co/u/sachinvy)\ **Replies:** 3\ **Last updated:** [September 3, 2018, 10:02pm UTC](https://discuss.elastic.co/t/date-format-not-working-in-logstash-6-3/147111 "2018-09-03T22:02:27Z")
I am using logstash 6.3 and following is my input and message. For some reason it is not converting String to Date. input { stdin { codec =\> json } } output { stdout { codec =\> rubydebug } } filter { date { ma… --- ## [Exec input for command in another directory](https://discuss.elastic.co/t/exec-input-for-command-in-another-directory/147109)
**Author:** [@cowensel](https://discuss.elastic.co/u/cowensel)\ **Replies:** 1\ **Last updated:** [September 3, 2018, 9:04pm UTC](https://discuss.elastic.co/t/exec-input-for-command-in-another-directory/147109 "2018-09-03T21:04:21Z")
I am looking into ways of being able to make monitoring of redis easier by collecting the slow log from redis and parsing it to elk to easily see trends and protential issues. It looks like this is possible by using the… --- ## [Getting error while running logstash with multiple beats data](https://discuss.elastic.co/t/getting-error-while-running-logstash-with-multiple-beats-data/146401)
**Author:** [@ravikt](https://discuss.elastic.co/u/ravikt)\ **Replies:** 15\ **Last updated:** [September 3, 2018, 8:58pm UTC](https://discuss.elastic.co/t/getting-error-while-running-logstash-with-multiple-beats-data/146401 "2018-09-03T20:58:59Z")
Currently I am sending metricbeat and winlogbeat data from same source to same logstash instance A. Now I would like to send filebeat data from another source to same logstash instance A. When I restart logstash, am get… --- ## [\[logstash.codecs.netflow \] errors](https://discuss.elastic.co/t/logstash-codecs-netflow-errors/146761)
**Author:** [@drzbass](https://discuss.elastic.co/u/drzbass)\ **Replies:** 0\ **Last updated:** [August 30, 2018, 5:56pm UTC](https://discuss.elastic.co/t/logstash-codecs-netflow-errors/146761 "2018-08-30T17:56:44Z")
Hello, I'm trying to deploy Logstash+ElasticSearch+Kibana+Elastiflow and testing it trying to get flow from 2 Juniper routers. I noticed that the flow are too low (around 1.4Mbits) when I'm expecting something like 50… --- ## [Csv date automatically converts to timestamp](https://discuss.elastic.co/t/csv-date-automatically-converts-to-timestamp/147108)
**Author:** [@Divit\_Sharma](https://discuss.elastic.co/u/Divit_Sharma)\ **Replies:** 0\ **Last updated:** [September 3, 2018, 6:06pm UTC](https://discuss.elastic.co/t/csv-date-automatically-converts-to-timestamp/147108 "2018-09-03T18:06:55Z")
When parsing the date field from csv file it automatically converts to date time. I just want the date without any timestamp. How to write the code in logstash ? One solution is to edit the field in kibana index as per… --- ## [Dead letter queue encoding issues / binary format](https://discuss.elastic.co/t/dead-letter-queue-encoding-issues-binary-format/146617)
**Author:** [@jgzurano](https://discuss.elastic.co/u/jgzurano)\ **Replies:** 4\ **Last updated:** [September 3, 2018, 2:59pm UTC](https://discuss.elastic.co/t/dead-letter-queue-encoding-issues-binary-format/146617 "2018-09-03T14:59:32Z")
Environment: ELK Stack 6.2.3 Version: Logstash 6.2.3 Docker image: docker.elastic.co/logstash/logstash-oss:6.2.3 I have this stack running: filebeat -\> logstash -\> elasticsearch I enabled the DLQ after noticing that … --- ## [Logstash as a service, it does not connect to the network drives to read files](https://discuss.elastic.co/t/logstash-as-a-service-it-does-not-connect-to-the-network-drives-to-read-files/147083)
**Author:** [@emmanuel792](https://discuss.elastic.co/u/emmanuel792)\ **Replies:** 2\ **Last updated:** [September 3, 2018, 2:22pm UTC](https://discuss.elastic.co/t/logstash-as-a-service-it-does-not-connect-to-the-network-drives-to-read-files/147083 "2018-09-03T14:22:44Z")
I am configuring logstash as a windows service, with the procrun tool. Everything is configured, is problent when I run logstash as a service, it does not connect to the network drives to read files. but when I run it … --- ## [Unable to load csv file through logstash](https://discuss.elastic.co/t/unable-to-load-csv-file-through-logstash/146853)
**Author:** [@Prabhakaran\_D](https://discuss.elastic.co/u/Prabhakaran_D)\ **Replies:** 2\ **Last updated:** [September 3, 2018, 2:15pm UTC](https://discuss.elastic.co/t/unable-to-load-csv-file-through-logstash/146853 "2018-09-03T14:15:15Z")
I am trying to load an csv file through logstash. Please find the config file below input { file { path =\> "D:\\summer.csv" start\_position =\> "beginning" sincedb\_path =\> "nul" } } filter … --- ## [Kv filter values is always returned as string](https://discuss.elastic.co/t/kv-filter-values-is-always-returned-as-string/147073)
**Author:** [@kgdi](https://discuss.elastic.co/u/kgdi)\ **Replies:** 5\ **Last updated:** [September 3, 2018, 1:39pm UTC](https://discuss.elastic.co/t/kv-filter-values-is-always-returned-as-string/147073 "2018-09-03T13:39:59Z")
Hi, I am using the kv filter to get values from extra\_fields and it works fine, except it shows up as string in the index and not number. Is there any limitations in using mutate on kv generated values? One entry of t… --- ## [Unable to parse timestamp](https://discuss.elastic.co/t/unable-to-parse-timestamp/147067)
**Author:** [@Nithani25](https://discuss.elastic.co/u/Nithani25)\ **Replies:** 2\ **Last updated:** [September 3, 2018, 12:31pm UTC](https://discuss.elastic.co/t/unable-to-parse-timestamp/147067 "2018-09-03T12:31:38Z")
HI Team, I am working on some logs which looks something like below 2018-09-02 23:02:18,358|\[\[ACTIVE\] ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)'\]|ERROR|GlobalExceptionMapper|Unknown|Unknown… --- ## [Unable to install logstash-input-jdbc to connect to Oracle Db](https://discuss.elastic.co/t/unable-to-install-logstash-input-jdbc-to-connect-to-oracle-db/147035)
**Author:** [@groverjatin17](https://discuss.elastic.co/u/groverjatin17)\ **Replies:** 3\ **Last updated:** [September 3, 2018, 11:48am UTC](https://discuss.elastic.co/t/unable-to-install-logstash-input-jdbc-to-connect-to-oracle-db/147035 "2018-09-03T11:48:05Z")
Hi, I am using ELK 6.2.4 for a while. Till now i have been uploading .csv files to create index in ES and create dashboards. But recently my team asked to make it more dynamic and automated as in connect to a database … --- ## [Kafka input multiple logstash](https://discuss.elastic.co/t/kafka-input-multiple-logstash/146631)
**Author:** [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\ **Replies:** 0\ **Last updated:** [August 30, 2018, 7:22am UTC](https://discuss.elastic.co/t/kafka-input-multiple-logstash/146631 "2018-08-30T07:22:55Z")
HI Team, I'm very happy to post back question hear and here my doubt, I going to receive data from kakfa to logstash. For example: i have log in kakfa called server A and i want to index that log into 2 separate serv… --- ## [Logstash and MongoDB using logstash-input-mongodb Problem](https://discuss.elastic.co/t/logstash-and-mongodb-using-logstash-input-mongodb-problem/147047)
**Author:** [@Sami\_Hilaneh](https://discuss.elastic.co/u/Sami_Hilaneh)\ **Replies:** 0\ **Last updated:** [September 3, 2018, 10:24am UTC](https://discuss.elastic.co/t/logstash-and-mongodb-using-logstash-input-mongodb-problem/147047 "2018-09-03T10:24:05Z")
Hi, tried using the unofficial logstash-input-mongodb, but im having problem importing MongoDB to ES. Is there something that I am missing in the config file? TIA MongoDB Name: test Table: restaurants Configuration: … --- ## [How can I export data in CEF format using logstash](https://discuss.elastic.co/t/how-can-i-export-data-in-cef-format-using-logstash/147046)
**Author:** [@yash\_mangla](https://discuss.elastic.co/u/yash_mangla)\ **Replies:** 0\ **Last updated:** [September 3, 2018, 10:18am UTC](https://discuss.elastic.co/t/how-can-i-export-data-in-cef-format-using-logstash/147046 "2018-09-03T10:18:50Z")
Hi folk, i'm trying to dump data using Logstash in CEF format. I'm not sure it is possible or not. Is anyone here who done this before then give me some direction to perform this task Thanks, Yash M. --- ## [Logstash filter, how to find first and last log?](https://discuss.elastic.co/t/logstash-filter-how-to-find-first-and-last-log/147037)
**Author:** [@zxc654951](https://discuss.elastic.co/u/zxc654951)\ **Replies:** 0\ **Last updated:** [September 3, 2018, 8:51am UTC](https://discuss.elastic.co/t/logstash-filter-how-to-find-first-and-last-log/147037 "2018-09-03T08:51:56Z")
Hi, I have some log with timestamp, keyID, message, or more. Those logs don't have start\_tag and end\_tag, however messages don't. Can I use mutate filter find the first and last log by keyID(machineID)? Below are some … --- ## [Working with brackets in regex](https://discuss.elastic.co/t/working-with-brackets-in-regex/146958)
**Author:** [@fishfox](https://discuss.elastic.co/u/fishfox)\ **Replies:** 3\ **Last updated:** [September 3, 2018, 8:37am UTC](https://discuss.elastic.co/t/working-with-brackets-in-regex/146958 "2018-09-03T08:37:33Z")
I have a regex if statement that includes brackets: if \[prog\] =~ /^openvpn\[\[0-9\]\*\]$/ { grok { match =\> \[ "message", "%{IP:remote\_ip}:%{POSINT:remote\_port} \\\[%{USERNAME:username}\\\] %{GREEDYDATA:event}"\] }… --- ## [Logs not parsed as per splunk](https://discuss.elastic.co/t/logs-not-parsed-as-per-splunk/146111)
**Author:** [@mav](https://discuss.elastic.co/u/mav)\ **Replies:** 8\ **Last updated:** [September 3, 2018, 6:25am UTC](https://discuss.elastic.co/t/logs-not-parsed-as-per-splunk/146111 "2018-09-03T06:25:30Z")
Hi, I am a newbie to ELK stack and I am trying to read logs from an application, so please pardon my ignorance. This is a Citrix based application from a vendor and have logs in the format as shown below. ============… --- ## [Logstash conf issue](https://discuss.elastic.co/t/logstash-conf-issue/146986)
**Author:** [@gopisamineni99](https://discuss.elastic.co/u/gopisamineni99)\ **Replies:** 1\ **Last updated:** [September 3, 2018, 6:12am UTC](https://discuss.elastic.co/t/logstash-conf-issue/146986 "2018-09-03T06:12:07Z")
Hi All, i am panning to implement log stash for my application , my log files look like ( json code) } userid : "..." group : ...." mail : "..." time : "..." level : ".." msg : "..." } } userid : "..." grou… --- ## [Logstash data won't read new template](https://discuss.elastic.co/t/logstash-data-wont-read-new-template/146988)
**Author:** [@Yash\_Sharma](https://discuss.elastic.co/u/Yash_Sharma)\ **Replies:** 1\ **Last updated:** [September 3, 2018, 6:10am UTC](https://discuss.elastic.co/t/logstash-data-wont-read-new-template/146988 "2018-09-03T06:10:59Z")
I am creating a new template in my elastic search using kibana dev console. PUT \_template/template\_1 { "index\_patterns": \["ubq-\*"\], "settings": { "number\_of\_shards": 1 }, "mappings": { "\_doc": { … --- ## [Parsing mixing of json and kv filter in logstash](https://discuss.elastic.co/t/parsing-mixing-of-json-and-kv-filter-in-logstash/146878)
**Author:** [@ramesh87](https://discuss.elastic.co/u/ramesh87)\ **Replies:** 5\ **Last updated:** [September 3, 2018, 6:06am UTC](https://discuss.elastic.co/t/parsing-mixing-of-json-and-kv-filter-in-logstash/146878 "2018-09-03T06:06:36Z")
Hi, I am using the logstash configuration version 6.3. I am trying to parsing my log files with conjunction of json and kv filter in the logstash configuration. Logstash Configuration: input { beats { port =\> 5990 … --- ## [Filter don't filter and don't drop events in MongoDB log](https://discuss.elastic.co/t/filter-dont-filter-and-dont-drop-events-in-mongodb-log/146061)
**Author:** [@gabberoid](https://discuss.elastic.co/u/gabberoid)\ **Replies:** 5\ **Last updated:** [September 3, 2018, 6:03am UTC](https://discuss.elastic.co/t/filter-dont-filter-and-dont-drop-events-in-mongodb-log/146061 "2018-09-03T06:03:29Z")
Hi. I want to filter all events from MongoDB log except ACCESS component. My config file is: filter { if \[document\_type\] == "windows-mongodb" { if \[mongodb.log.component\] != "ACCESS" { drop{} } grok { match =\> { … --- ## [Grok pattern for custome logs](https://discuss.elastic.co/t/grok-pattern-for-custome-logs/146873)
**Author:** [@learnhub17](https://discuss.elastic.co/u/learnhub17)\ **Replies:** 0\ **Last updated:** [August 31, 2018, 2:00pm UTC](https://discuss.elastic.co/t/grok-pattern-for-custome-logs/146873 "2018-08-31T14:00:25Z")
Hi, version of logstash is: 6.3.2 I am parsing my nginx logs, logging format is: log\_format timed\_combined '$remote\_addr - $remote\_user \[$time\_local\] "$request" ' '$status $body\_bytes\_sent "$http\_referer" ' '"$http\_… --- ## [How does elasticsearch output affect the logstash pipeline performance?](https://discuss.elastic.co/t/how-does-elasticsearch-output-affect-the-logstash-pipeline-performance/147003)
**Author:** [@YuWatanabe](https://discuss.elastic.co/u/YuWatanabe)\ **Replies:** 0\ **Last updated:** [September 3, 2018, 12:16am UTC](https://discuss.elastic.co/t/how-does-elasticsearch-output-affect-the-logstash-pipeline-performance/147003 "2018-09-03T00:16:34Z")
I have done a benchmarking for logstash v 6.3.2 and ask for help interpreting the result. Benchmarking in logstash under couple of conditions and saw drastic decrease with Events Emitted Rate . All logs were sent from f… --- ## [Multiple inputs and multiple outputs](https://discuss.elastic.co/t/multiple-inputs-and-multiple-outputs/87788)
**Author:** [@jgarbora](https://discuss.elastic.co/u/jgarbora)\ **Replies:** 3\ **Last updated:** [September 2, 2018, 1:09pm UTC](https://discuss.elastic.co/t/multiple-inputs-and-multiple-outputs/87788 "2018-09-02T13:09:56Z")
Hi there, I'm trying to set multiple inputs and multiple outputs to different elasticsearch indexes ... and something I am doing wrong because everything is going to the else. Here is my config: input { jdbc … --- ## [How does connection pooling internally work with logstash http plugin](https://discuss.elastic.co/t/how-does-connection-pooling-internally-work-with-logstash-http-plugin/143891)
**Author:** [@Sruthi\_Grace](https://discuss.elastic.co/u/Sruthi_Grace)\ **Replies:** 2\ **Last updated:** [September 2, 2018, 10:27am UTC](https://discuss.elastic.co/t/how-does-connection-pooling-internally-work-with-logstash-http-plugin/143891 "2018-09-02T10:27:38Z")
The default pool max for http output plugin is 50 connections . What happens if all the 50 connections to send out are occupied and there are more logs lines coming into the logstash but all the 50 out connections are st… --- ## [Creating same index using java ESrepository and logstash](https://discuss.elastic.co/t/creating-same-index-using-java-esrepository-and-logstash/146946)
**Author:** [@LQss](https://discuss.elastic.co/u/LQss)\ **Replies:** 0\ **Last updated:** [September 1, 2018, 4:59pm UTC](https://discuss.elastic.co/t/creating-same-index-using-java-esrepository-and-logstash/146946 "2018-09-01T16:59:04Z")
My application works when i try to post something using my URIs, ES automatically create a new index. And now when i try to put some more data over that index with same name i get an indexing issue.. even tho I am posti… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=690) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=692)