# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1237 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1238 --- ## [Logatash input file from URL](https://discuss.elastic.co/t/logatash-input-file-from-url/29241)
**Author:** [@Navneet\_Mathpal](https://discuss.elastic.co/u/Navneet_Mathpal)\ **Replies:** 5\ **Last updated:** [September 15, 2015, 10:38am UTC](https://discuss.elastic.co/t/logatash-input-file-from-url/29241 "2015-09-15T10:38:01Z")
Hi, I have a requirement where I need to put the file path as URL . Is it possible in logatash to put the file path as a URL ? and will it be real time ? ex: input { file { path =\> https://www.google.com/jsonfile… --- ## [Logstash AWS GeoIP Issue?](https://discuss.elastic.co/t/logstash-aws-geoip-issue/29274)
**Author:** [@russK](https://discuss.elastic.co/u/russK)\ **Replies:** 7\ **Last updated:** [September 14, 2015, 10:46pm UTC](https://discuss.elastic.co/t/logstash-aws-geoip-issue/29274 "2015-09-14T22:46:11Z")
Hello, We're running an ELK cluster on AWS. We also run a single box with all of ELK on it to verify changes locally before pushing to the cloud. We're running into an issue where the local box can run the GeoIP2-City.… --- ## [Logstash crashes after sometime and freezes the system](https://discuss.elastic.co/t/logstash-crashes-after-sometime-and-freezes-the-system/28848)
**Author:** [@imad](https://discuss.elastic.co/u/imad)\ **Replies:** 20\ **Last updated:** [September 14, 2015, 10:44pm UTC](https://discuss.elastic.co/t/logstash-crashes-after-sometime-and-freezes-the-system/28848 "2015-09-14T22:44:18Z")
I am using logstash to feed elasticsearch indices data from my couchdb. I have about 20 indices. At this time I have very simple setup. All my data is in different databases in couchDB, e.g., users has its own db, roles … --- ## [How to send windows 2012 r2 log file to logstash](https://discuss.elastic.co/t/how-to-send-windows-2012-r2-log-file-to-logstash/28708)
**Author:** [@Kaufusihm](https://discuss.elastic.co/u/Kaufusihm)\ **Replies:** 17\ **Last updated:** [September 14, 2015, 10:09pm UTC](https://discuss.elastic.co/t/how-to-send-windows-2012-r2-log-file-to-logstash/28708 "2015-09-14T22:09:13Z")
I currently have a logstash server, Elasticsearch server, and kibana server. I am able to send logstash local data using stdin from LS to my ES instance and I can interact with Kibana, but now I am trying to use a window… --- ## [Logstash wont add date to logs](https://discuss.elastic.co/t/logstash-wont-add-date-to-logs/29267)
**Author:** [@mgirndt](https://discuss.elastic.co/u/mgirndt)\ **Replies:** 3\ **Last updated:** [September 14, 2015, 7:26pm UTC](https://discuss.elastic.co/t/logstash-wont-add-date-to-logs/29267 "2015-09-14T19:26:20Z")
How can I get logstah add the date field that I have added to my jboss logs? The log looks like: August 10 2015 10:07:01,048 INFO \[org.jboss.as.naming\] (ServerService Thread Pool -- 38) JBAS011800: Activating Naming Subs… --- ## [ELK + redis query](https://discuss.elastic.co/t/elk-redis-query/28870)
**Author:** [@knightsg](https://discuss.elastic.co/u/knightsg)\ **Replies:** 5\ **Last updated:** [September 14, 2015, 5:36pm UTC](https://discuss.elastic.co/t/elk-redis-query/28870 "2015-09-14T17:36:04Z")
Hi all, I've set up ELK on AWS in a fault-tolerant configuration (multi-AZ), and have been looking at integrating redis into the stack to ease the load on logstash, as is commonly recommended. However, it seems to me t… --- ## [Logstash twitter output to mysql database](https://discuss.elastic.co/t/logstash-twitter-output-to-mysql-database/29213)
**Author:** [@Dan](https://discuss.elastic.co/u/Dan)\ **Replies:** 2\ **Last updated:** [September 14, 2015, 6:28am UTC](https://discuss.elastic.co/t/logstash-twitter-output-to-mysql-database/29213 "2015-09-14T06:28:16Z")
Hi All, i want to store twitter data into mysql database. Here can we able to use logstash output plugin. If we are not able to use that please give me if we have any alternate. Thanks & Regards, Dinesh. --- ## [Customer id](https://discuss.elastic.co/t/customer-id/29020)
**Author:** [@scaarup](https://discuss.elastic.co/u/scaarup)\ **Replies:** 2\ **Last updated:** [September 13, 2015, 2:51pm UTC](https://discuss.elastic.co/t/customer-id/29020 "2015-09-13T14:51:53Z")
I am indexing logdata in elasticsearch with a string like "customerid=1234". I have a csv/database with relations between a customer-name and the customerid. Could I add a field automatically so the customers name gets s… --- ## [Dumping logs to a file based on timestamp field](https://discuss.elastic.co/t/dumping-logs-to-a-file-based-on-timestamp-field/29165)
**Author:** [@Deb](https://discuss.elastic.co/u/Deb)\ **Replies:** 5\ **Last updated:** [September 13, 2015, 2:36pm UTC](https://discuss.elastic.co/t/dumping-logs-to-a-file-based-on-timestamp-field/29165 "2015-09-13T14:36:44Z")
I am trying to dump log events to a file having name of the format, hello-YYYY-MM-DD.log using the file output plugin. I want to get YYYY-MM-DD from a field named timestamp contained in the log event (not @timestamp fiel… --- ## [Failed to tls handshake with logstash](https://discuss.elastic.co/t/failed-to-tls-handshake-with-logstash/29184)
**Author:** [@sumeet\_dembra](https://discuss.elastic.co/u/sumeet_dembra)\ **Replies:** 0\ **Last updated:** [September 13, 2015, 2:22am UTC](https://discuss.elastic.co/t/failed-to-tls-handshake-with-logstash/29184 "2015-09-13T02:22:53Z")
Hi, I have a setup in which there are 7 machines with logstash-forwarder connecting to logstash server. 6 of these are RHEL and 1 is debian. I use same certificate on all servers to connect to logstash. All RHEL ma… --- ## [Can't parse simple xml](https://discuss.elastic.co/t/cant-parse-simple-xml/29179)
**Author:** [@gogators](https://discuss.elastic.co/u/gogators)\ **Replies:** 1\ **Last updated:** [September 12, 2015, 8:16pm UTC](https://discuss.elastic.co/t/cant-parse-simple-xml/29179 "2015-09-12T20:16:01Z")
Why is this simple test failing? I really don't understand the error message: # echo "\123\" | logstash -e "input { stdin {} } filter { xml { source =\> "message" } } output { stdout {} }" … --- ## [\[kafka\] \[java.lang.ClassNotFoundException\] overriding partitioner\_class](https://discuss.elastic.co/t/kafka-java-lang-classnotfoundexception-overriding-partitioner-class/28527)
**Author:** [@steotia](https://discuss.elastic.co/u/steotia)\ **Replies:** 1\ **Last updated:** [September 12, 2015, 7:03pm UTC](https://discuss.elastic.co/t/kafka-java-lang-classnotfoundexception-overriding-partitioner-class/28527 "2015-09-12T19:03:40Z")
I created a partitioner\_class and replaced the partitioner\_class appropriately in the conf file. The jar containing the partitioner\_class was added to the CLASSPATH. However, when I started logstash, logstash errored o… --- ## [Log parse pattern for Kafka & ZooKeeper](https://discuss.elastic.co/t/log-parse-pattern-for-kafka-zookeeper/28442)
**Author:** [@Anirban\_Mukherjee](https://discuss.elastic.co/u/Anirban_Mukherjee)\ **Replies:** 1\ **Last updated:** [September 12, 2015, 6:53pm UTC](https://discuss.elastic.co/t/log-parse-pattern-for-kafka-zookeeper/28442 "2015-09-12T18:53:48Z")
Hi , Currently I am using a cluster of Zookeeper and Kafka . I would like to parse these logs through logstash . Is there any available pattern for these two types logs parsing ? Thanks, AM --- ## [Prioritising logs Using Kafka & Logstash](https://discuss.elastic.co/t/prioritising-logs-using-kafka-logstash/29057)
**Author:** [@Deb](https://discuss.elastic.co/u/Deb)\ **Replies:** 5\ **Last updated:** [September 12, 2015, 6:48pm UTC](https://discuss.elastic.co/t/prioritising-logs-using-kafka-logstash/29057 "2015-09-12T18:48:39Z")
We have a deployment in which different types of logs are getting pushed to Kafka. Then logstash is pulling these logs from Kafka and pushing it to ES. We want to prioritise the way logs getting indexed in ES. For exampl… --- ## [Logstash 1.5.4 + Lumberjack 1.0.5 = Mutex Errors](https://discuss.elastic.co/t/logstash-1-5-4-lumberjack-1-0-5-mutex-errors/29173)
**Author:** [@zonywhoop](https://discuss.elastic.co/u/zonywhoop)\ **Replies:** 0\ **Last updated:** [September 12, 2015, 7:58am UTC](https://discuss.elastic.co/t/logstash-1-5-4-lumberjack-1-0-5-mutex-errors/29173 "2015-09-12T07:58:51Z")
Just upgraded from Logstash 1.4.5 to 1.5.4 and have been beating my head against the wall trying to get lumberjack to work again. My issue is that if tcp/5000 is open (the port I have lumberjack listing on) then logstas… --- ## [Input HTTP\_Poller, HTTPS on the API / untrusted cert](https://discuss.elastic.co/t/input-http-poller-https-on-the-api-untrusted-cert/25492)
**Author:** [@CharlesLabri](https://discuss.elastic.co/u/CharlesLabri)\ **Replies:** 4\ **Last updated:** [September 11, 2015, 6:32pm UTC](https://discuss.elastic.co/t/input-http-poller-https-on-the-api-untrusted-cert/25492 "2015-09-11T18:32:07Z")
Anyone have any luck with HTTPS on untrusted cert with this plugin? I am getting a java error "PKIX path building failed" due to an untrusted cert. PS. trying to use the nagios NCPA plugin to gather perf metrics from… --- ## [Formating field dependent output](https://discuss.elastic.co/t/formating-field-dependent-output/29151)
**Author:** [@jfairchild](https://discuss.elastic.co/u/jfairchild)\ **Replies:** 0\ **Last updated:** [September 11, 2015, 4:21pm UTC](https://discuss.elastic.co/t/formating-field-dependent-output/29151 "2015-09-11T16:21:45Z")
tl;dr: Can sprintf format in logstash do more than just date formatting based on @timestamp? Long Version: I'm in the process of moving the log collection for my application from essentially a proprietary mishmash of… --- ## [Way to sort walltime field with the format of hh:mm:ss](https://discuss.elastic.co/t/way-to-sort-walltime-field-with-the-format-of-hh-mm-ss/29086)
**Author:** [@angelsmile](https://discuss.elastic.co/u/angelsmile)\ **Replies:** 2\ **Last updated:** [September 11, 2015, 12:02pm UTC](https://discuss.elastic.co/t/way-to-sort-walltime-field-with-the-format-of-hh-mm-ss/29086 "2015-09-11T12:02:28Z")
Hi, There is a walltime field in our logs with the format of hh:mm:ss. What is the best format/way for this field in logstash so that we can have a panel in Kibana to sort these walltime data? Currently the field is st… --- ## [Elasticsearch filter not working due to 1 sec refresh?](https://discuss.elastic.co/t/elasticsearch-filter-not-working-due-to-1-sec-refresh/1820)
**Author:** [@niksfirefly](https://discuss.elastic.co/u/niksfirefly)\ **Replies:** 3\ **Last updated:** [September 11, 2015, 10:43am UTC](https://discuss.elastic.co/t/elasticsearch-filter-not-working-due-to-1-sec-refresh/1820 "2015-09-11T10:43:28Z")
I am trying to use elasticsearch filter but with no success. I want to search logs to find past events and add fields to actual event. I think this is due to 1 sec refresh of indices so when past events are too close to … --- ## [Output NetFlow and Syslog to Elasticsearch](https://discuss.elastic.co/t/output-netflow-and-syslog-to-elasticsearch/29046)
**Author:** [@Ben19](https://discuss.elastic.co/u/Ben19)\ **Replies:** 3\ **Last updated:** [September 11, 2015, 10:24am UTC](https://discuss.elastic.co/t/output-netflow-and-syslog-to-elasticsearch/29046 "2015-09-11T10:24:17Z")
Hello, I have an ELK server currently taking NetFlow traffic from a single host and sending this to a custom index name (logstash\_netflow-\*) etc and I’m now wanting to take syslogs on UDP port 5000. Currently NetFlow is c… --- ## [Help Me Filter this strange timestamp](https://discuss.elastic.co/t/help-me-filter-this-strange-timestamp/29104)
**Author:** [@Ibrahim\_Lubis](https://discuss.elastic.co/u/Ibrahim_Lubis)\ **Replies:** 2\ **Last updated:** [September 11, 2015, 10:09am UTC](https://discuss.elastic.co/t/help-me-filter-this-strange-timestamp/29104 "2015-09-11T10:09:05Z")
Hi, I got raw log from a wireless controller, and it show strange timestamp, i can filter the log after timestamp, but i cant filter the timestamp, this is the raw log, \<190\>2015-09-11T15:40:37.05+07:00 MAP-NUS03-PR… --- ## [Errors with Logstash bulk load using CSV](https://discuss.elastic.co/t/errors-with-logstash-bulk-load-using-csv/29059)
**Author:** [@cburkins](https://discuss.elastic.co/u/cburkins)\ **Replies:** 3\ **Last updated:** [September 11, 2015, 9:58am UTC](https://discuss.elastic.co/t/errors-with-logstash-bulk-load-using-csv/29059 "2015-09-11T09:58:52Z")
I'm using Logstash to bulk load into ElasticSearch using a CSV template I'm getting lots of warnings like: retrying failed action with response code: 429 {:level=\>:warn} And lots of errors like: too many attempts … --- ## [Basic-parsing XML into logsash](https://discuss.elastic.co/t/basic-parsing-xml-into-logsash/29093)
**Author:** [@ushadatt](https://discuss.elastic.co/u/ushadatt)\ **Replies:** 3\ **Last updated:** [September 11, 2015, 9:56am UTC](https://discuss.elastic.co/t/basic-parsing-xml-into-logsash/29093 "2015-09-11T09:56:56Z")
I want to parse my XML document through logstash so that each document gets parsed differently..How to achieve this? For example: book id 101 becomes one document with parsed fields and book102 becomes another document w… --- ## [Logstash reads my entry 2 times (or more)](https://discuss.elastic.co/t/logstash-reads-my-entry-2-times-or-more/29039)
**Author:** [@Hornov](https://discuss.elastic.co/u/Hornov)\ **Replies:** 2\ **Last updated:** [September 11, 2015, 8:44am UTC](https://discuss.elastic.co/t/logstash-reads-my-entry-2-times-or-more/29039 "2015-09-11T08:44:23Z")
When I use elasticsearch input, sometimes logstash refers several times my result in succession. I don't understand why. Sometimes it works, sometimes not and nevertheless neither my configuration nor my data changed. --- ## [Failed action with response of 400, dropping action](https://discuss.elastic.co/t/failed-action-with-response-of-400-dropping-action/28802)
**Author:** [@angelsmile](https://discuss.elastic.co/u/angelsmile)\ **Replies:** 6\ **Last updated:** [September 11, 2015, 5:08am UTC](https://discuss.elastic.co/t/failed-action-with-response-of-400-dropping-action/28802 "2015-09-11T05:08:20Z")
Hi, For the following log: Job Start Summary: jobid=2291646.r-man2 project=p08 user=rnt651@raijin1 queue=express-node ncpus=16 mem\_requested=8gb walltime=16:00:00 cpu\_hrs=768.0 jobfs\_local=8gb jobfs\_global=None When… --- ## [Logstash Output CSV Plugin produces no content in the output file](https://discuss.elastic.co/t/logstash-output-csv-plugin-produces-no-content-in-the-output-file/29078)
**Author:** [@Bill.T](https://discuss.elastic.co/u/Bill.T)\ **Replies:** 0\ **Last updated:** [September 10, 2015, 10:33pm UTC](https://discuss.elastic.co/t/logstash-output-csv-plugin-produces-no-content-in-the-output-file/29078 "2015-09-10T22:33:55Z")
I am trying to write an output CSV file that contains two of the four columns from an input CSV file. It appears that the input file is correctly parsed and mapped. The output file is created but nothing gets written to… --- ## [Logstash logging options (rotating, removing old logs, etc)](https://discuss.elastic.co/t/logstash-logging-options-rotating-removing-old-logs-etc/28966)
**Author:** [@edouglass](https://discuss.elastic.co/u/edouglass)\ **Replies:** 5\ **Last updated:** [September 10, 2015, 6:48pm UTC](https://discuss.elastic.co/t/logstash-logging-options-rotating-removing-old-logs-etc/28966 "2015-09-10T18:48:39Z")
Hello! Thanks for any help! When running logstash, I am aware of the --log option that allows you to specify a log directory for logstash to log to. However, I do not know what logstash's settings are for rotating log fi… --- ## [Using Unicast discovery with logstash & elasticsearch](https://discuss.elastic.co/t/using-unicast-discovery-with-logstash-elasticsearch/28897)
**Author:** [@Deb](https://discuss.elastic.co/u/Deb)\ **Replies:** 2\ **Last updated:** [September 10, 2015, 6:11pm UTC](https://discuss.elastic.co/t/using-unicast-discovery-with-logstash-elasticsearch/28897 "2015-09-10T18:11:31Z")
I want to use unicast discovery in one of my logstash deployments as mentioned here in the best practices. I have gone through this link. My ES Cluster looks like below:- 3 Master-Data Nodes - es-masterdata1, es-m… --- ## [How logstash reads a log file from a subfolder](https://discuss.elastic.co/t/how-logstash-reads-a-log-file-from-a-subfolder/29042)
**Author:** [@vikas\_gopal](https://discuss.elastic.co/u/vikas_gopal)\ **Replies:** 2\ **Last updated:** [September 10, 2015, 4:57pm UTC](https://discuss.elastic.co/t/how-logstash-reads-a-log-file-from-a-subfolder/29042 "2015-09-10T16:57:31Z")
HI Experts, My logs are placed in the following directory structure logs/ Cust1/2015/Sep-09/cisco.log ................. ................. ........... Cust1/2015/Sep-10/cisco.log Here log is the m… --- ## [Logstash on Windows requires drive letter in the path variable of the file input plugin](https://discuss.elastic.co/t/logstash-on-windows-requires-drive-letter-in-the-path-variable-of-the-file-input-plugin/28969)
**Author:** [@jamo3](https://discuss.elastic.co/u/jamo3)\ **Replies:** 3\ **Last updated:** [September 10, 2015, 1:44pm UTC](https://discuss.elastic.co/t/logstash-on-windows-requires-drive-letter-in-the-path-variable-of-the-file-input-plugin/28969 "2015-09-10T13:44:26Z")
If you leave of the drive letter logstash won't process files on windows, i.e: input { file { type =\> "cfcidnelog" path =\> \["c:/Temp/logstash-cidne/REDNE.log"\] start\_position =\> "beginning" … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1236) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1238)