# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1138 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1139 --- ## [Access a network share for logs](https://discuss.elastic.co/t/access-a-network-share-for-logs/51391)
**Author:** [@rashidmirza](https://discuss.elastic.co/u/rashidmirza)\ **Replies:** 4\ **Last updated:** [June 21, 2016, 5:10am UTC](https://discuss.elastic.co/t/access-a-network-share-for-logs/51391 "2016-06-21T05:10:39Z")
i have recently installed ELK on a windows machine. I would like to parse and create some sort of statistics out of a text based log file that resides on a Linux machine, but as that folder is made as a share, it is acce… --- ## [Dynamic Index in ElasticSearch](https://discuss.elastic.co/t/dynamic-index-in-elasticsearch/53267)
**Author:** [@supercoder](https://discuss.elastic.co/u/supercoder)\ **Replies:** 3\ **Last updated:** [June 21, 2016, 4:18am UTC](https://discuss.elastic.co/t/dynamic-index-in-elasticsearch/53267 "2016-06-21T04:18:59Z")
I have following configuration in logstash whereby I am able to create dynamic "document\_type" into ES based on input JSON received: elasticsearch { hosts =\> \["localhost:9200"\] index =\> "queuelogs" … --- ## [Help with GeoLocation please!](https://discuss.elastic.co/t/help-with-geolocation-please/53119)
**Author:** [@gott](https://discuss.elastic.co/u/gott)\ **Replies:** 7\ **Last updated:** [June 20, 2016, 9:30pm UTC](https://discuss.elastic.co/t/help-with-geolocation-please/53119 "2016-06-20T21:30:58Z")
Looking for some help please. Im new to Elastic and im finding it very enjoyable!! My elk stack is running on a windows server ok so my GEO situation is... Filebeat on a IIS webserver sending iis logs to logstash serv… --- ## [Logstash: Translation missing](https://discuss.elastic.co/t/logstash-translation-missing/53028)
**Author:** [@urbanninja](https://discuss.elastic.co/u/urbanninja)\ **Replies:** 4\ **Last updated:** [June 20, 2016, 8:44pm UTC](https://discuss.elastic.co/t/logstash-translation-missing/53028 "2016-06-20T20:44:23Z")
Hi, I'm new to ELK and have recently got the ELK stack to work on an Amazon Web Services t2.medium instance. My problem is that I can't get Logstash run my configuration file, which is throwing an error. I get thi… --- ## [No Data after Pipeline main started](https://discuss.elastic.co/t/no-data-after-pipeline-main-started/52891)
**Author:** [@spraveenjd](https://discuss.elastic.co/u/spraveenjd)\ **Replies:** 4\ **Last updated:** [June 20, 2016, 8:17pm UTC](https://discuss.elastic.co/t/no-data-after-pipeline-main-started/52891 "2016-06-20T20:17:35Z")
I have created logstsh-order.conf and started the instance with that. I do see the logs Pipeline main started after that nothing happened. No Error and no data logs. Please help me out what going wrong here --- ## [New to ELK, do we need a queue for ELK stack](https://discuss.elastic.co/t/new-to-elk-do-we-need-a-queue-for-elk-stack/48829)
**Author:** [@nanshan](https://discuss.elastic.co/u/nanshan)\ **Replies:** 6\ **Last updated:** [June 20, 2016, 8:16pm UTC](https://discuss.elastic.co/t/new-to-elk-do-we-need-a-queue-for-elk-stack/48829 "2016-06-20T20:16:53Z")
Thanks --- ## [Conflicting logstash timestaps system vs UTC](https://discuss.elastic.co/t/conflicting-logstash-timestaps-system-vs-utc/52097)
**Author:** [@PepeK](https://discuss.elastic.co/u/PepeK)\ **Replies:** 5\ **Last updated:** [June 20, 2016, 7:30pm UTC](https://discuss.elastic.co/t/conflicting-logstash-timestaps-system-vs-utc/52097 "2016-06-20T19:30:29Z")
Hi, I am posting this from Chile and have patched java with the latest timezone data from iana.org: I see conflicting timestamp data in logstash-2.3.2-1(receiving from rsyslog) which is causing kibana to display the lo… --- ## [Bad link on Logstash download page](https://discuss.elastic.co/t/bad-link-on-logstash-download-page/53362)
**Author:** [@jelammers](https://discuss.elastic.co/u/jelammers)\ **Replies:** 0\ **Last updated:** [June 20, 2016, 7:04pm UTC](https://discuss.elastic.co/t/bad-link-on-logstash-download-page/53362 "2016-06-20T19:04:19Z")
On page https://www.elastic.co/downloads/logstash, the RPM sha1 link points to logstash-2.3.3-1.noarch.rpm instead of logstash-2.3.3-1.noarch.rpm.sha1.txt Once I figured out the pattern, I was able to add .sha1.txt to… --- ## [Using regex grouping and logical OR in a Grok Pattern](https://discuss.elastic.co/t/using-regex-grouping-and-logical-or-in-a-grok-pattern/53069)
**Author:** [@cb2015](https://discuss.elastic.co/u/cb2015)\ **Replies:** 2\ **Last updated:** [June 20, 2016, 6:15pm UTC](https://discuss.elastic.co/t/using-regex-grouping-and-logical-or-in-a-grok-pattern/53069 "2016-06-20T18:15:15Z")
Hi Everyone, I am trying to parse a log which has two formats in it. They are: Timestamp loglevel thread blahblahblah Timestamp thread loglevel blahblahblah I am currently trying to use a | and ()'s to cover for b… --- ## [Logstash issues in connecting to database](https://discuss.elastic.co/t/logstash-issues-in-connecting-to-database/53351)
**Author:** [@KiranG](https://discuss.elastic.co/u/KiranG)\ **Replies:** 0\ **Last updated:** [June 20, 2016, 6:04pm UTC](https://discuss.elastic.co/t/logstash-issues-in-connecting-to-database/53351 "2016-06-20T18:04:09Z")
Hi, I am using logstash-2.1.0 and a novice user of it. I have multiple config files which also implies multiple sql queries for each to run. The scheduling times are different from one query to another. Could not f… --- ## [Docker gelf and logstash to logstash](https://discuss.elastic.co/t/docker-gelf-and-logstash-to-logstash/52740)
**Author:** [@elmamazo](https://discuss.elastic.co/u/elmamazo)\ **Replies:** 4\ **Last updated:** [June 20, 2016, 4:01pm UTC](https://discuss.elastic.co/t/docker-gelf-and-logstash-to-logstash/52740 "2016-06-20T16:01:41Z")
Hello guys Im using the dock gelf logging driver and I have the gelf input plugin and elastic search as output this works fine. What I want to is is docker -\> gelf\_input -\> lumberjack\_out -\> lunberjack\_in -\> es the p… --- ## [Multiple logstash instance running](https://discuss.elastic.co/t/multiple-logstash-instance-running/53173)
**Author:** [@jay224](https://discuss.elastic.co/u/jay224)\ **Replies:** 2\ **Last updated:** [June 20, 2016, 3:52pm UTC](https://discuss.elastic.co/t/multiple-logstash-instance-running/53173 "2016-06-20T15:52:43Z")
In our environment we have multiple logstash harvestor running at same time. PS on commna line looks like this "logstash 1668 129 3.9 10291044 1302464 ? SNl 12:07 27:08 /usr/bin/java -XX:+UseParNewGC -XX… --- ## [How can we run logstash conf files in sequence?](https://discuss.elastic.co/t/how-can-we-run-logstash-conf-files-in-sequence/53318)
**Author:** [@rupal](https://discuss.elastic.co/u/rupal)\ **Replies:** 0\ **Last updated:** [June 20, 2016, 1:37pm UTC](https://discuss.elastic.co/t/how-can-we-run-logstash-conf-files-in-sequence/53318 "2016-06-20T13:37:22Z")
I have following scenario. I have 4 different logstash files and 4 different document\_type. Conf files Document Type CSV file used to get input 1) doc1.conf doc1 doc1.c… --- ## [Mutate on inexistent field creates them](https://discuss.elastic.co/t/mutate-on-inexistent-field-creates-them/52743)
**Author:** [@Bruno\_Lavoie](https://discuss.elastic.co/u/Bruno_Lavoie)\ **Replies:** 0\ **Last updated:** [June 14, 2016, 1:45pm UTC](https://discuss.elastic.co/t/mutate-on-inexistent-field-creates-them/52743 "2016-06-14T13:45:15Z")
Hello, We're on putting a central logging service with many clients and different products. Depending on source, some standard fields are set at times, at other times not. We're putting some standard filters to normal… --- ## [Sqs output errors cause blocked input pipelines](https://discuss.elastic.co/t/sqs-output-errors-cause-blocked-input-pipelines/52741)
**Author:** [@NoumanSaleem](https://discuss.elastic.co/u/NoumanSaleem)\ **Replies:** 2\ **Last updated:** [June 20, 2016, 1:30pm UTC](https://discuss.elastic.co/t/sqs-output-errors-cause-blocked-input-pipelines/52741 "2016-06-20T13:30:34Z")
It's become a daily occurrence in the last week where we see a drop in logs in Elasticsearch caused by blocked pipelines in our logstash stack. We utilize a cluster of logstash for accepting and pushing logs into an SQ… --- ## [Add a counter field to each message](https://discuss.elastic.co/t/add-a-counter-field-to-each-message/53312)
**Author:** [@Ben\_Ryan](https://discuss.elastic.co/u/Ben_Ryan)\ **Replies:** 0\ **Last updated:** [June 20, 2016, 1:12pm UTC](https://discuss.elastic.co/t/add-a-counter-field-to-each-message/53312 "2016-06-20T13:12:21Z")
This should be simple but I cannot find a solution - I want each message that is output to have a counter added that is incremented each time is new message is output --- ## [Create index on pdf file input using logstash](https://discuss.elastic.co/t/create-index-on-pdf-file-input-using-logstash/52724)
**Author:** [@Tamanna](https://discuss.elastic.co/u/Tamanna)\ **Replies:** 2\ **Last updated:** [June 20, 2016, 11:06am UTC](https://discuss.elastic.co/t/create-index-on-pdf-file-input-using-logstash/52724 "2016-06-20T11:06:15Z")
Hi All, I am new to Logstash and have little idea about it. I need to create an index in ElasticSearch on a pdf file. Ingesting this pdf file, i heard, can be done with Logstash. But i have no idea how it could be don… --- ## [Kakfa is reading off the wrong schema id](https://discuss.elastic.co/t/kakfa-is-reading-off-the-wrong-schema-id/53298)
**Author:** [@Nir\_Sharony](https://discuss.elastic.co/u/Nir_Sharony)\ **Replies:** 0\ **Last updated:** [June 20, 2016, 9:36am UTC](https://discuss.elastic.co/t/kakfa-is-reading-off-the-wrong-schema-id/53298 "2016-06-20T09:36:32Z")
Hi, I am reading a JSON input file and feeds it to a an Json-to-Avro codec converter before sending it to a Kafka topic. the codec takes the schema, writes to the schema registry and obtains a schema id. It then writes… --- ## [Logstash filter config to parse JSON](https://discuss.elastic.co/t/logstash-filter-config-to-parse-json/53269)
**Author:** [@gugansankar](https://discuss.elastic.co/u/gugansankar)\ **Replies:** 3\ **Last updated:** [June 20, 2016, 7:28am UTC](https://discuss.elastic.co/t/logstash-filter-config-to-parse-json/53269 "2016-06-20T07:28:27Z")
I tried to parse json input file. But logstash doesn't parse the logs and not adding custom field "add\_field". I think, my filter configuration is wrong. Someone please help me on this. Logs file: {"web1":{"stat1":{… --- ## [Help To parse xml](https://discuss.elastic.co/t/help-to-parse-xml/53273)
**Author:** [@hearvishwas](https://discuss.elastic.co/u/hearvishwas)\ **Replies:** 0\ **Last updated:** [June 20, 2016, 6:55am UTC](https://discuss.elastic.co/t/help-to-parse-xml/53273 "2016-06-20T06:55:20Z")
Help required to parse xml: \\14618.806\\2016-04-25\\04:25:28.806\\kjdasjhsajhkjahdk.com\\10.23.232.32323\\managed\_server\ **Author:** [@sumeshms](https://discuss.elastic.co/u/sumeshms)\ **Replies:** 0\ **Last updated:** [June 20, 2016, 6:30am UTC](https://discuss.elastic.co/t/default-config-parsers-for-logstash/53270 "2016-06-20T06:30:50Z") Hello Is there any default config\_parsers for logstash? Am getting this line on logstash.log -- :config\_parsers=\>"MMM d HH:mm:ss,MMM dd HH:mm:ss,ISO8601" for parsing timestamp field as the default one. Any idea why? … --- ## [Logstash 2.3.3 on windows "=1.0 was unexpected at this time" error](https://discuss.elastic.co/t/logstash-2-3-3-on-windows-1-0-was-unexpected-at-this-time-error/53135)
**Author:** [@SKumarMN](https://discuss.elastic.co/u/SKumarMN)\ **Replies:** 2\ **Last updated:** [June 20, 2016, 6:21am UTC](https://discuss.elastic.co/t/logstash-2-3-3-on-windows-1-0-was-unexpected-at-this-time-error/53135 "2016-06-20T06:21:41Z")
I have logstash 2.3.3 on windows server 2008 R2. When i go try to excute any command getting into bin folder of logsatsh., I get "=1.0 was unexpected at this time" as the error I have java 1.8\_0.92 set as java\_home --- ## [Configtest does not support using environment variables](https://discuss.elastic.co/t/configtest-does-not-support-using-environment-variables/52931)
**Author:** [@JPvRiel](https://discuss.elastic.co/u/JPvRiel)\ **Replies:** 1\ **Last updated:** [June 20, 2016, 5:38am UTC](https://discuss.elastic.co/t/configtest-does-not-support-using-environment-variables/52931 "2016-06-20T05:38:51Z")
As per https://www.elastic.co/guide/en/logstash/current/environment-variables.html Tried v2.3 and v5.0 E.g. $ export LS\_BEATS\_IN\_PORT=5044 sample config file input { beats { port =\> ${LS\_BEAT… --- ## [Can we pass a directory including multiples configuration files from different sources?](https://discuss.elastic.co/t/can-we-pass-a-directory-including-multiples-configuration-files-from-different-sources/53057)
**Author:** [@nanshan](https://discuss.elastic.co/u/nanshan)\ **Replies:** 3\ **Last updated:** [June 20, 2016, 5:35am UTC](https://discuss.elastic.co/t/can-we-pass-a-directory-including-multiples-configuration-files-from-different-sources/53057 "2016-06-20T05:35:31Z")
I am trying to build a elk stack. Under /etc/logstash/conf.d, I have several configration files, a.conf, b.conf. How to start logstash and pass both of the files, or can I just pass the direcory ? --- ## [Using ELK to create a unified logging system with large log input](https://discuss.elastic.co/t/using-elk-to-create-a-unified-logging-system-with-large-log-input/53261)
**Author:** [@rvu95](https://discuss.elastic.co/u/rvu95)\ **Replies:** 3\ **Last updated:** [June 20, 2016, 4:48am UTC](https://discuss.elastic.co/t/using-elk-to-create-a-unified-logging-system-with-large-log-input/53261 "2016-06-20T04:48:21Z")
Hello, I'm still a newbie, I want to ask about the tradeoff of using ELK for unified logging syste, with large log input. I'm planning on using HAProxy, redis, and ELK. What wanna I ask is how's the performance of this k… --- ## [Single CPU is pegged](https://discuss.elastic.co/t/single-cpu-is-pegged/53220)
**Author:** [@carrot](https://discuss.elastic.co/u/carrot)\ **Replies:** 2\ **Last updated:** [June 19, 2016, 9:50pm UTC](https://discuss.elastic.co/t/single-cpu-is-pegged/53220 "2016-06-19T21:50:07Z")
I'm getting about 600-1000% CPU usage on a 12 core machine running logstash @6000 eps. Although I wonder what is this 100% usage on 1 cpu with 25% user and 75% system. Seems to be related to all the java thread running. … --- ## [\[SOLVED\] Stuck on getting netflow data into logstash](https://discuss.elastic.co/t/solved-stuck-on-getting-netflow-data-into-logstash/52909)
**Author:** [@DRW](https://discuss.elastic.co/u/DRW)\ **Replies:** 3\ **Last updated:** [June 19, 2016, 8:14pm UTC](https://discuss.elastic.co/t/solved-stuck-on-getting-netflow-data-into-logstash/52909 "2016-06-19T20:14:14Z")
Hello, I've been playing with this for longer than I would care to admit(weeks). I am trying to setup up an nProbe/Netflow into ELK and have it in one or more docker containers. I have tried the existing projects, … --- ## [Remove field value in logstash](https://discuss.elastic.co/t/remove-field-value-in-logstash/53226)
**Author:** [@shan](https://discuss.elastic.co/u/shan)\ **Replies:** 1\ **Last updated:** [June 19, 2016, 6:21pm UTC](https://discuss.elastic.co/t/remove-field-value-in-logstash/53226 "2016-06-19T18:21:49Z")
Hi, I want to remove the field value which has the domain name like \*.abc.com. Is it possible in logstash ? Anyone help me on this. Thanks, --- ## [Logstash -Using Schedular with JDBC Input plugin results in Ram being locked for scheduler and high CPU usage](https://discuss.elastic.co/t/logstash-using-schedular-with-jdbc-input-plugin-results-in-ram-being-locked-for-scheduler-and-high-cpu-usage/53200)
**Author:** [@amitpalsaluja](https://discuss.elastic.co/u/amitpalsaluja)\ **Replies:** 2\ **Last updated:** [June 19, 2016, 1:41pm UTC](https://discuss.elastic.co/t/logstash-using-schedular-with-jdbc-input-plugin-results-in-ram-being-locked-for-scheduler-and-high-cpu-usage/53200 "2016-06-19T13:41:11Z")
Hi , Using JDBC Input plugin with Schedule option , results in a. RAM(around 150+ MB) being Locked for the process. b. High CPU Usage. Version details logstash-2.2.0 --- ## [How to add new fields from specific set of strings in the message?](https://discuss.elastic.co/t/how-to-add-new-fields-from-specific-set-of-strings-in-the-message/52799)
**Author:** [@khizaralvi](https://discuss.elastic.co/u/khizaralvi)\ **Replies:** 1\ **Last updated:** [June 19, 2016, 10:23am UTC](https://discuss.elastic.co/t/how-to-add-new-fields-from-specific-set-of-strings-in-the-message/52799 "2016-06-19T10:23:07Z")
Hi everyone, I have a log file which records the query stats of a MongoDB server. 2016-06-06T00:31:14.848-0700 I COMMAND \[conn55216\] query ums.broadcasts query: { $readPreference: { mode: "secondary" }, $query: { s… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1137) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1139)