# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1076 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1077 --- ## [Help with Date](https://discuss.elastic.co/t/help-with-date/66240)
**Author:** [@debarun](https://discuss.elastic.co/u/debarun)\ **Replies:** 5\ **Last updated:** [November 18, 2016, 11:07am UTC](https://discuss.elastic.co/t/help-with-date/66240 "2016-11-18T11:07:12Z")
Hi, I need to replace the time coming from the logs to timestamp field. But i keep getting a parse failure. The loglines look like Time Stamp 11/14/16 15:14:25 Event Number 2004 I am using the following filters:- … --- ## [Issue while installing custom logstash gem filter with logstash 2.3.1](https://discuss.elastic.co/t/issue-while-installing-custom-logstash-gem-filter-with-logstash-2-3-1/48296)
**Author:** [@asgawali](https://discuss.elastic.co/u/asgawali)\ **Replies:** 1\ **Last updated:** [November 18, 2016, 10:47am UTC](https://discuss.elastic.co/t/issue-while-installing-custom-logstash-gem-filter-with-logstash-2-3-1/48296 "2016-11-18T10:47:08Z")
I was using logstash 1.5.4 and was able to install custom logstash filter received from third party team, now I have updated the logstash from 1.5.4 to 2.3.1 and when I try to install logstash filter it gives me followin… --- ## [Logstash filter "metrics" is not working!](https://discuss.elastic.co/t/logstash-filter-metrics-is-not-working/66482)
**Author:** [@Ritesh\_Sharma](https://discuss.elastic.co/u/Ritesh_Sharma)\ **Replies:** 5\ **Last updated:** [November 18, 2016, 10:21am UTC](https://discuss.elastic.co/t/logstash-filter-metrics-is-not-working/66482 "2016-11-18T10:21:42Z")
Hey Guys , I am trying to get response per second for some of my APIs from NGINX Access logs and after some googling i came across "metrics" filter of logstash which claims to provide per duration (/sec , /minute etc … --- ## [2.4.0: Too Many Open Files](https://discuss.elastic.co/t/2-4-0-too-many-open-files/65611)
**Author:** [@dawiro](https://discuss.elastic.co/u/dawiro)\ **Replies:** 1\ **Last updated:** [November 18, 2016, 9:19am UTC](https://discuss.elastic.co/t/2-4-0-too-many-open-files/65611 "2016-11-18T09:19:50Z")
Hi, After upgrading to logstash 2.4.0 we're seeing our logstashes run out of file handles: :error\_message=\>"Too many open files", :class=\>"Manticore::SocketException", :level=\>:error The cause is open socket connect… --- ## [Logstash is not processing files from input plugin Azure Storage blob, even though new files are available to process](https://discuss.elastic.co/t/logstash-is-not-processing-files-from-input-plugin-azure-storage-blob-even-though-new-files-are-available-to-process/65999)
**Author:** [@nethis](https://discuss.elastic.co/u/nethis)\ **Replies:** 2\ **Last updated:** [November 18, 2016, 8:54am UTC](https://discuss.elastic.co/t/logstash-is-not-processing-files-from-input-plugin-azure-storage-blob-even-though-new-files-are-available-to-process/65999 "2016-11-18T08:54:47Z")
I have been using Azure Storage Input Plugin , for json files. After an improper shutdown of pipeline, the restart of the logstash process is not picking up the files from Azure Storage blob, even there are so many pendi… --- ## [How to improve performance of Re-indexing from logstash?](https://discuss.elastic.co/t/how-to-improve-performance-of-re-indexing-from-logstash/65966)
**Author:** [@nethis](https://discuss.elastic.co/u/nethis)\ **Replies:** 2\ **Last updated:** [November 18, 2016, 8:49am UTC](https://discuss.elastic.co/t/how-to-improve-performance-of-re-indexing-from-logstash/65966 "2016-11-18T08:49:27Z")
Hi There, We have 460 GB of index, which is to be re-index , to make all fields not\_analyzed. We started re-indexed around 8 days back, and still in progress.(though there are environment issues on shards unavailabl… --- ## [Logstash imap input plugin. Want to trash mail. Error: undefined local variable or method \`id\_set'](https://discuss.elastic.co/t/logstash-imap-input-plugin-want-to-trash-mail-error-undefined-local-variable-or-method-id-set/66502)
**Author:** [@kjellgrb](https://discuss.elastic.co/u/kjellgrb)\ **Replies:** 0\ **Last updated:** [November 18, 2016, 7:58am UTC](https://discuss.elastic.co/t/logstash-imap-input-plugin-want-to-trash-mail-error-undefined-local-variable-or-method-id-set/66502 "2016-11-18T07:58:38Z")
Hi I want the IMAP plugin to delete mail. I found undocumented settings delete =\> true expunge =\> true But then I get Plugin: "xxxxxx", password=\>, user=\>"xxxxxx", secure=\>false, type=\>"imap", delete=\>true, exp… --- ## [Logstash Input Eventlog](https://discuss.elastic.co/t/logstash-input-eventlog/66364)
**Author:** [@RdrgPorto](https://discuss.elastic.co/u/RdrgPorto)\ **Replies:** 5\ **Last updated:** [November 18, 2016, 7:47am UTC](https://discuss.elastic.co/t/logstash-input-eventlog/66364 "2016-11-18T07:47:28Z")
Hi everyone, I have a trouble with Input Eventlog with Logstash 2.3.4 in Windows 2008 R2. My configuration: input { eventlog { type =\> "test" logfile =\> "Test Application Log" } } I am trying to obtain Test App… --- ## [Help needed for reading Logs](https://discuss.elastic.co/t/help-needed-for-reading-logs/65959)
**Author:** [@Haribaskar](https://discuss.elastic.co/u/Haribaskar)\ **Replies:** 9\ **Last updated:** [November 18, 2016, 7:11am UTC](https://discuss.elastic.co/t/help-needed-for-reading-logs/65959 "2016-11-18T07:11:51Z")
Hi there, Could someone guide me on how to use pipeline(|) as separator for the Logs. I tried gsub. But everytime I use gsub, it says non ascii character found in the log. Any help would be appreciated.. Regards, Hari --- ## [Logstash to elasticsearch output error](https://discuss.elastic.co/t/logstash-to-elasticsearch-output-error/66083)
**Author:** [@shashant](https://discuss.elastic.co/u/shashant)\ **Replies:** 1\ **Last updated:** [November 18, 2016, 6:51am UTC](https://discuss.elastic.co/t/logstash-to-elasticsearch-output-error/66083 "2016-11-18T06:51:16Z")
Got error to send bulk of actions: blocked by: \[SERVICE\_UNAVAILABLE/1/state not recovered / initialized\];\[SERVICE\_UNAVAILABLE/2/no master\]; {:level=\>:error} Failed to flush outgoing items {:outgoing\_count=\>2146, :excepti… --- ## [Reindexing Tweets changing a field type in Elasticsearch using Logstash](https://discuss.elastic.co/t/reindexing-tweets-changing-a-field-type-in-elasticsearch-using-logstash/65991)
**Author:** [@Luzcka](https://discuss.elastic.co/u/Luzcka)\ **Replies:** 3\ **Last updated:** [November 18, 2016, 6:38am UTC](https://discuss.elastic.co/t/reindexing-tweets-changing-a-field-type-in-elasticsearch-using-logstash/65991 "2016-11-18T06:38:41Z")
Hi guys, I know twitter / tweets are a type of subject with many doubts and answers, but I don't find the solution. Well, the problem is the same than others guys find: coordinates isn't a geo\_poin type. I want to u… --- ## [Must logstash-input-wmi be run on a Windows host, or can it be run on linux?](https://discuss.elastic.co/t/must-logstash-input-wmi-be-run-on-a-windows-host-or-can-it-be-run-on-linux/66455)
**Author:** [@Kevin\_Mullet](https://discuss.elastic.co/u/Kevin_Mullet)\ **Replies:** 1\ **Last updated:** [November 18, 2016, 4:22am UTC](https://discuss.elastic.co/t/must-logstash-input-wmi-be-run-on-a-windows-host-or-can-it-be-run-on-linux/66455 "2016-11-18T04:22:40Z")
Obviously, logstash-input-wmi is only useful to collect data FROM windows hosts, but must the logstash instance running the logstash-input-wmi plugin necessarily be a windows host? I've done WMI in the past from linux,… --- ## [Bug with symbolic links given as input files ? \[Logstash 2.3.4\]](https://discuss.elastic.co/t/bug-with-symbolic-links-given-as-input-files-logstash-2-3-4/66459)
**Author:** [@oristano](https://discuss.elastic.co/u/oristano)\ **Replies:** 0\ **Last updated:** [November 17, 2016, 9:43pm UTC](https://discuss.elastic.co/t/bug-with-symbolic-links-given-as-input-files-logstash-2-3-4/66459 "2016-11-17T21:43:05Z")
Hello all :slight\_smile: I'm working with Logstash 2.3.4 and I'm facing a strange problem regarding the behaviour of the input feature. Here's the situation : I work with a very simple input configuration : input… --- ## [Logtash is not accepting/recieving syslog sent by linux server](https://discuss.elastic.co/t/logtash-is-not-accepting-recieving-syslog-sent-by-linux-server/66333)
**Author:** [@vivekcts](https://discuss.elastic.co/u/vivekcts)\ **Replies:** 9\ **Last updated:** [November 17, 2016, 5:30pm UTC](https://discuss.elastic.co/t/logtash-is-not-accepting-recieving-syslog-sent-by-linux-server/66333 "2016-11-17T17:30:08Z")
I have configure syslogng to sent syslog to server1 and port 514. but the logtash installed on server1 is not recieving the syslogs on port 514. i have defined input also for the port 514. Pls assist --- ## [Logstash : Cant filter syslog from Pfsense firewall](https://discuss.elastic.co/t/logstash-cant-filter-syslog-from-pfsense-firewall/66404)
**Author:** [@Patriote](https://discuss.elastic.co/u/Patriote)\ **Replies:** 2\ **Last updated:** [November 17, 2016, 4:54pm UTC](https://discuss.elastic.co/t/logstash-cant-filter-syslog-from-pfsense-firewall/66404 "2016-11-17T16:54:58Z")
hi everyone, i have an hard time filtering logs comming from a pfsense firewall i tried a lot of filters and groks but nothing's working it seems that logstash isnt reading filters or something doesnt any one have an id… --- ## [ElasticSearch/Logstash input to CSV output mapping question](https://discuss.elastic.co/t/elasticsearch-logstash-input-to-csv-output-mapping-question/65837)
**Author:** [@Thermocatalyst](https://discuss.elastic.co/u/Thermocatalyst)\ **Replies:** 4\ **Last updated:** [November 17, 2016, 4:21pm UTC](https://discuss.elastic.co/t/elasticsearch-logstash-input-to-csv-output-mapping-question/65837 "2016-11-17T16:21:51Z")
I'm attempting to read JSON data from ES and convert it to a CSV file, but I'm getting an unexpected result. I'd like to take selected fields from the JSON document, and add write them to a CSV. Here's my configura… --- ## [Logstash 2.3 with Elasticsearch 5.0](https://discuss.elastic.co/t/logstash-2-3-with-elasticsearch-5-0/66342)
**Author:** [@Jay\_Dihenkar](https://discuss.elastic.co/u/Jay_Dihenkar)\ **Replies:** 5\ **Last updated:** [November 17, 2016, 2:29pm UTC](https://discuss.elastic.co/t/logstash-2-3-with-elasticsearch-5-0/66342 "2016-11-17T14:29:56Z")
Can we use Logstash 2.3 with Elasticsearch 5.0? Compatibility matrix says : It's not supported. If not what can be the breaking changes to take note of? """ Matrix """ Supported Logstash Versions Logstash Version… --- ## [Logstash Too small initial heap](https://discuss.elastic.co/t/logstash-too-small-initial-heap/66068)
**Author:** [@manojvenkat](https://discuss.elastic.co/u/manojvenkat)\ **Replies:** 10\ **Last updated:** [November 17, 2016, 2:28pm UTC](https://discuss.elastic.co/t/logstash-too-small-initial-heap/66068 "2016-11-17T14:28:59Z")
Team, I have installed logstash 2.2 on my AWS linux instance . When I executed the command to test the config file: service logstash configtest I get the heap error as below, \[root@ip-10-34-77-105 conf.d\]# service lo… --- ## [Logstash Twitter plugin proxy issue](https://discuss.elastic.co/t/logstash-twitter-plugin-proxy-issue/66408)
**Author:** [@Vinoj\_Mathew](https://discuss.elastic.co/u/Vinoj_Mathew)\ **Replies:** 0\ **Last updated:** [November 17, 2016, 2:19pm UTC](https://discuss.elastic.co/t/logstash-twitter-plugin-proxy-issue/66408 "2016-11-17T14:19:45Z")
I am using Logstash5.0 with Twitter Plugin and it works fine with out any issue, if there is no proxy. Some of our machines we have to set the proxy to get connected to the outside world. When I ran with proxy getting… --- ## [Check if Logstash is Stuck](https://discuss.elastic.co/t/check-if-logstash-is-stuck/66303)
**Author:** [@chromechris](https://discuss.elastic.co/u/chromechris)\ **Replies:** 1\ **Last updated:** [November 17, 2016, 2:01pm UTC](https://discuss.elastic.co/t/check-if-logstash-is-stuck/66303 "2016-11-17T14:01:09Z")
Is there any way to check if Logstash is stuck besides restarting it and seeing that the Logstash process failed to terminate? --- ## [Loadbalancing with syslog](https://discuss.elastic.co/t/loadbalancing-with-syslog/66361)
**Author:** [@trenzalore](https://discuss.elastic.co/u/trenzalore)\ **Replies:** 3\ **Last updated:** [November 17, 2016, 1:04pm UTC](https://discuss.elastic.co/t/loadbalancing-with-syslog/66361 "2016-11-17T13:04:14Z")
Hi ! I use a syslog server which sends log to my cluster (3 servers cluster). Each server has a logstash instance. The syslog server send only to 2 servers. I found that my entries are doubled and I think it's becau… --- ## [Issue with file input multiline codec](https://discuss.elastic.co/t/issue-with-file-input-multiline-codec/66269)
**Author:** [@sjivan](https://discuss.elastic.co/u/sjivan)\ **Replies:** 1\ **Last updated:** [November 17, 2016, 12:48pm UTC](https://discuss.elastic.co/t/issue-with-file-input-multiline-codec/66269 "2016-11-17T12:48:41Z")
Hi, I have a log file that has I need to support multiline parsing and I'm in the process of migrating from multiline filter to multiline codec My input configuration is file { path =\> "c:/temp/test.log" sta… --- ## [Logstash 2.4.1 issue](https://discuss.elastic.co/t/logstash-2-4-1-issue/66383)
**Author:** [@pankug](https://discuss.elastic.co/u/pankug)\ **Replies:** 0\ **Last updated:** [November 17, 2016, 12:40pm UTC](https://discuss.elastic.co/t/logstash-2-4-1-issue/66383 "2016-11-17T12:40:31Z")
My Logstash process exits every hour after 100% CPU uses. Need to change JVM heap size for logstash 2.4.1. If anyone can tell me how can I change JVM heap size in logstash 2.4.1 --- ## [Mutate add\_field value null](https://discuss.elastic.co/t/mutate-add-field-value-null/42296)
**Author:** [@rss87](https://discuss.elastic.co/u/rss87)\ **Replies:** 3\ **Last updated:** [November 17, 2016, 12:01pm UTC](https://discuss.elastic.co/t/mutate-add-field-value-null/42296 "2016-11-17T12:01:44Z")
Hi, Is possible to add a field with value null, for example something like this: Mutate{ add\_field{"field1"=\> null}} I tried this, but added null like a string Thanks!! --- ## [Logstash big files](https://discuss.elastic.co/t/logstash-big-files/66273)
**Author:** [@txisme](https://discuss.elastic.co/u/txisme)\ **Replies:** 9\ **Last updated:** [November 17, 2016, 10:45am UTC](https://discuss.elastic.co/t/logstash-big-files/66273 "2016-11-17T10:45:47Z")
Hey, recently i sent to logstash with filebeat a 5GB log text data and it just broke logstash. Is there any problem with sending big files at once? --- ## [Greedydata does not work](https://discuss.elastic.co/t/greedydata-does-not-work/64032)
**Author:** [@ally](https://discuss.elastic.co/u/ally)\ **Replies:** 10\ **Last updated:** [November 17, 2016, 10:28am UTC](https://discuss.elastic.co/t/greedydata-does-not-work/64032 "2016-11-17T10:28:21Z")
Hello, I have Logstash v2.3 with the following configuration: \> input { \> beats { \> port =\> 5044 \> } \> } \> filter { \> grok { \> match =\> \["message", "(?m)\[%{HTTPDATE:msg\_timestamp}\] \[%… --- ## [The Architecture of entperise log management system based on Elastic Stack 5.0.1 and Kafka 0.10.1.0](https://discuss.elastic.co/t/the-architecture-of-entperise-log-management-system-based-on-elastic-stack-5-0-1-and-kafka-0-10-1-0/66356)
**Author:** [@obsicn](https://discuss.elastic.co/u/obsicn)\ **Replies:** 3\ **Last updated:** [November 17, 2016, 10:26am UTC](https://discuss.elastic.co/t/the-architecture-of-entperise-log-management-system-based-on-elastic-stack-5-0-1-and-kafka-0-10-1-0/66356 "2016-11-17T10:26:28Z")
In blog "Just Enough Kafka for the Elastic Stack（https://www.elastic.co/blog/just-enough-kafka-for-the-elastic-stack-part1）, Logstash takes the role of Shipper. But Filebeat 5.0.1 can support Kafka Output Configurati… --- ## [Confirmation about the load balance mechanism when using elasticsearch output](https://discuss.elastic.co/t/confirmation-about-the-load-balance-mechanism-when-using-elasticsearch-output/65581)
**Author:** [@YuWatanabe](https://discuss.elastic.co/u/YuWatanabe)\ **Replies:** 2\ **Last updated:** [November 17, 2016, 10:24am UTC](https://discuss.elastic.co/t/confirmation-about-the-load-balance-mechanism-when-using-elasticsearch-output/65581 "2016-11-17T10:24:26Z")
According to document elasticsearch output load balances the requests to elasticsearch when multiple hosts are set in hosts. Sets the host(s) of the remote instance. If given an array it will load balance requests acro… --- ## [How do you merge the syslogs from network device into message queue when logstash scales out?](https://discuss.elastic.co/t/how-do-you-merge-the-syslogs-from-network-device-into-message-queue-when-logstash-scales-out/66032)
**Author:** [@YuWatanabe](https://discuss.elastic.co/u/YuWatanabe)\ **Replies:** 2\ **Last updated:** [November 17, 2016, 10:24am UTC](https://discuss.elastic.co/t/how-do-you-merge-the-syslogs-from-network-device-into-message-queue-when-logstash-scales-out/66032 "2016-11-17T10:24:01Z")
I'd like to ask question about the theory of scale out process for logstash. I am planning to have 2300 evt/sec events into logstash. This number includes both from filebeat and syslog of network device. But number of… --- ## [How to filter tweets of logstash\_input\_twiiter based on timestapm?](https://discuss.elastic.co/t/how-to-filter-tweets-of-logstash-input-twiiter-based-on-timestapm/66340)
**Author:** [@jamal](https://discuss.elastic.co/u/jamal)\ **Replies:** 0\ **Last updated:** [November 17, 2016, 7:28am UTC](https://discuss.elastic.co/t/how-to-filter-tweets-of-logstash-input-twiiter-based-on-timestapm/66340 "2016-11-17T07:28:46Z")
here is my twitter input tweets "\_source": { "created\_at": "Wed Aug 10 06:42:48 +0000 2016", "id": 763264318242783200, "timestamp\_ms": "1470811368891", "@version": "1", "@timestamp": "2016-08-10T06:42:48.000Z" } and … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1075) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1077)