# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1032 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1033 --- ## [Help with simple grok filter for getting date](https://discuss.elastic.co/t/help-with-simple-grok-filter-for-getting-date/75951)
**Author:** [@Pete\_Del\_Rosso](https://discuss.elastic.co/u/Pete_Del_Rosso)\ **Replies:** 4\ **Last updated:** [February 22, 2017, 6:46am UTC](https://discuss.elastic.co/t/help-with-simple-grok-filter-for-getting-date/75951 "2017-02-22T06:46:20Z")
i have tried to get this working for a couple of days now. it seems so simple i must be missing something. I have read all the topics and tried to find examples to help me get it working, no luck. I just want to have the… --- ## [Convert HashMap to Json in Logstash](https://discuss.elastic.co/t/convert-hashmap-to-json-in-logstash/75776)
**Author:** [@sachiiit](https://discuss.elastic.co/u/sachiiit)\ **Replies:** 1\ **Last updated:** [February 22, 2017, 6:39am UTC](https://discuss.elastic.co/t/convert-hashmap-to-json-in-logstash/75776 "2017-02-22T06:39:04Z")
I am using Logstash in which the input is kafka topic. Through java code we are pushing the HashMap to the topic. According to my requirements, I can't convert the HashMap to json string on my java code side. The example… --- ## [Input file start\_position =\> "beginning" doesn't work even after deleting .sincedb files](https://discuss.elastic.co/t/input-file-start-position-beginning-doesnt-work-even-after-deleting-sincedb-files/75627)
**Author:** [@Prathamesh](https://discuss.elastic.co/u/Prathamesh)\ **Replies:** 1\ **Last updated:** [February 22, 2017, 6:37am UTC](https://discuss.elastic.co/t/input-file-start-position-beginning-doesnt-work-even-after-deleting-sincedb-files/75627 "2017-02-22T06:37:48Z")
Hello World Version: ElasticSearch-5.2.1/Logstash-5.2.1/Kibana-5.2.1 OS: Windows 2008 I've just started working on the ELK Stack & am facing some problems loading data I've got the following .json code input { fi… --- ## [Configure Logstash to accept data from one host location to another host location](https://discuss.elastic.co/t/configure-logstash-to-accept-data-from-one-host-location-to-another-host-location/75563)
**Author:** [@Mani2451](https://discuss.elastic.co/u/Mani2451)\ **Replies:** 2\ **Last updated:** [February 22, 2017, 6:36am UTC](https://discuss.elastic.co/t/configure-logstash-to-accept-data-from-one-host-location-to-another-host-location/75563 "2017-02-22T06:36:21Z")
Hi, i am trying to configure logstash by accepting data from one host location i.e xyz1.com to xyz2.com and the xyz 1 has a log which needs to be sent to xyz2 and then it should be parsed . Since i'm able to see that inp… --- ## [Unexpected pool error](https://discuss.elastic.co/t/unexpected-pool-error/74340)
**Author:** [@111126](https://discuss.elastic.co/u/111126)\ **Replies:** 3\ **Last updated:** [February 22, 2017, 5:53am UTC](https://discuss.elastic.co/t/unexpected-pool-error/74340 "2017-02-22T05:53:09Z")
hello! I have logstash, elastic verison 5.2 My elastic - there is cluster with two nodes (ip adress x.x.x.x and y.y.y.y). My output logstash config: \> elasticsearch { \> hosts =\> … --- ## [Streamset vs Logstash](https://discuss.elastic.co/t/streamset-vs-logstash/75976)
**Author:** [@gringo](https://discuss.elastic.co/u/gringo)\ **Replies:** 0\ **Last updated:** [February 22, 2017, 3:31am UTC](https://discuss.elastic.co/t/streamset-vs-logstash/75976 "2017-02-22T03:31:07Z")
Anyone know what is difference between streamset and logstash. Do they complement each other? --- ## [Loads of geoip errors](https://discuss.elastic.co/t/loads-of-geoip-errors/75829)
**Author:** [@PSi\_AU](https://discuss.elastic.co/u/PSi_AU)\ **Replies:** 5\ **Last updated:** [February 22, 2017, 1:18am UTC](https://discuss.elastic.co/t/loads-of-geoip-errors/75829 "2017-02-22T01:18:58Z")
I'm getting loads of the following types of error: \[2017-02-21T18:22:02,599\]\[WARN \]\[logstash.outputs.elasticsearch\] Failed action. {:status=\>400, :action=\>\["index", {:id=\>nil, :index=\>"filebeat-2017.02.21", :type=\>"s… --- ## [GeoIP Fields != desired geo\_point type](https://discuss.elastic.co/t/geoip-fields-desired-geo-point-type/75954)
**Author:** [@jc034240](https://discuss.elastic.co/u/jc034240)\ **Replies:** 3\ **Last updated:** [February 21, 2017, 9:27pm UTC](https://discuss.elastic.co/t/geoip-fields-desired-geo-point-type/75954 "2017-02-21T21:27:02Z")
Elastic 2.3.1, Kibana 4.5 Two fields src\_geoip.location, dest\_geoip.location are being indexed are not the desired geo\_point type, but instead as number. Despite the below: Logstash Conf file filter section: geoip { … --- ## [Cloudtrail logs parsing with logstash](https://discuss.elastic.co/t/cloudtrail-logs-parsing-with-logstash/75806)
**Author:** [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\ **Replies:** 1\ **Last updated:** [February 21, 2017, 8:09pm UTC](https://discuss.elastic.co/t/cloudtrail-logs-parsing-with-logstash/75806 "2017-02-21T20:09:11Z")
Hi All, I am trying to parse Amazon Cloudtrail logs to Elasticsearch using logstash. I have my s3 plugin configured and output been set to Elasticsearch but i find out that there is no grok pattern defined as to pass t… --- ## [Logstash with Cloudtrail](https://discuss.elastic.co/t/logstash-with-cloudtrail/75930)
**Author:** [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\ **Replies:** 1\ **Last updated:** [February 21, 2017, 7:47pm UTC](https://discuss.elastic.co/t/logstash-with-cloudtrail/75930 "2017-02-21T19:47:24Z")
Hi, I have configure a logstash to pull data from cloudtrail data from amazon s3. I am running it as command line on Centos 7. /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf Below is my logs… --- ## [Unable to install logstash plugin offline](https://discuss.elastic.co/t/unable-to-install-logstash-plugin-offline/75676)
**Author:** [@TinTin](https://discuss.elastic.co/u/TinTin)\ **Replies:** 1\ **Last updated:** [February 21, 2017, 7:08pm UTC](https://discuss.elastic.co/t/unable-to-install-logstash-plugin-offline/75676 "2017-02-21T19:08:41Z")
Hi, I have built a logstash-output-custom plugin, which I can install offline in my local logstash setup. But, if I try to install the same when trying to create a docker image, it fails with the following error: t-m… --- ## [How to read JSON files stored in HDFS via Logstash](https://discuss.elastic.co/t/how-to-read-json-files-stored-in-hdfs-via-logstash/46954)
**Author:** [@gowthamsadasivam](https://discuss.elastic.co/u/gowthamsadasivam)\ **Replies:** 3\ **Last updated:** [February 21, 2017, 5:52pm UTC](https://discuss.elastic.co/t/how-to-read-json-files-stored-in-hdfs-via-logstash/46954 "2017-02-21T17:52:30Z")
I have my data (JSON Files) stored in HDFS and would like to push these data to ElasticSearch. I have seen output plugin for HDFS but not any input plugin. Is there a way to read the HDFS files via Logstash? or any other… --- ## [Custom pattern - Telephone number and others](https://discuss.elastic.co/t/custom-pattern-telephone-number-and-others/75712)
**Author:** [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\ **Replies:** 7\ **Last updated:** [February 21, 2017, 4:55pm UTC](https://discuss.elastic.co/t/custom-pattern-telephone-number-and-others/75712 "2017-02-21T16:55:38Z")
Hello, I trying to capture specific fields in log like a phone\_number. In grok debbuger, i try this (regular expression found) : {'/^(+33|0)\[67\]\[0-9\]{8}$/':PHONE\_NUMBER} Entry : 0614151612 Result : Not found. … --- ## [How to increase Java heap space for Logstash running on Windows?](https://discuss.elastic.co/t/how-to-increase-java-heap-space-for-logstash-running-on-windows/75509)
**Author:** [@Adyne](https://discuss.elastic.co/u/Adyne)\ **Replies:** 12\ **Last updated:** [February 21, 2017, 4:43pm UTC](https://discuss.elastic.co/t/how-to-increase-java-heap-space-for-logstash-running-on-windows/75509 "2017-02-21T16:43:09Z")
After about 4-5 hours of running Logstash 5.2.1 on Windows I get this error: java.lang.OutOfMemoryError: Java heap space Dumping heap to C:\\logstash-5.2.1/heapdump.hprof ... Unable to create C:\\logstash-5.2.1/heapdump.… --- ## [Problem with elapsed plugin](https://discuss.elastic.co/t/problem-with-elapsed-plugin/52689)
**Author:** [@zpp](https://discuss.elastic.co/u/zpp)\ **Replies:** 7\ **Last updated:** [February 21, 2017, 4:12pm UTC](https://discuss.elastic.co/t/problem-with-elapsed-plugin/52689 "2017-02-21T16:12:29Z")
Hi, Just hit a problem with the Elapsed plugin, referring to the data below, not sure why it reported the elapsed\_end\_without\_match error though the start event is there. I have checked that the unique\_field is the sam… --- ## [Multiline Java Stack Traces with Syslog](https://discuss.elastic.co/t/multiline-java-stack-traces-with-syslog/75379)
**Author:** [@taytos](https://discuss.elastic.co/u/taytos)\ **Replies:** 1\ **Last updated:** [February 21, 2017, 3:14pm UTC](https://discuss.elastic.co/t/multiline-java-stack-traces-with-syslog/75379 "2017-02-21T15:14:22Z")
Hello there! I have researched this problem I'm having for a long time and can't seem to figure it out. I'm trying to avoid using the multiline filter plug-in as that is being deprecated so the multiline input codec is… --- ## [Logstash use two query in the statement](https://discuss.elastic.co/t/logstash-use-two-query-in-the-statement/75900)
**Author:** [@Catello\_Sabatino](https://discuss.elastic.co/u/Catello_Sabatino)\ **Replies:** 0\ **Last updated:** [February 21, 2017, 3:10pm UTC](https://discuss.elastic.co/t/logstash-use-two-query-in-the-statement/75900 "2017-02-21T15:10:39Z")
Hi all, I'm newbie of Elasticsearch and logstash, but i must know if is possible have this feature: have a file .sql with two istructions, for example: select package.SET\_CODE\_FUNCTION('XXX') from dual select \* from … --- ## [Ruby plugin getting 'undefined method'](https://discuss.elastic.co/t/ruby-plugin-getting-undefined-method/75888)
**Author:** [@seanziee](https://discuss.elastic.co/u/seanziee)\ **Replies:** 2\ **Last updated:** [February 21, 2017, 2:58pm UTC](https://discuss.elastic.co/t/ruby-plugin-getting-undefined-method/75888 "2017-02-21T14:58:21Z")
Hi, I was wondering if anyone knew why I'm getting this exception for the following ruby code. It seems to not recognize the gsub method. Do I have to initialize it in order for LS to understand it? ruby… --- ## [Performance Issues with Logstash UDP/IPFIX](https://discuss.elastic.co/t/performance-issues-with-logstash-udp-ipfix/75069)
**Author:** [@beshbesh](https://discuss.elastic.co/u/beshbesh)\ **Replies:** 1\ **Last updated:** [February 21, 2017, 1:04pm UTC](https://discuss.elastic.co/t/performance-issues-with-logstash-udp-ipfix/75069 "2017-02-21T13:04:33Z")
Hello, I have been trying to get Logstash to be able to process as many IPFIX (Netflow v10) packets as possible. I have seen some cases where Logstash users easily reached 40k events per second, or even up to 90k: I… --- ## [Input Plugins for Google Drive, Sharepoint, JIRA etc](https://discuss.elastic.co/t/input-plugins-for-google-drive-sharepoint-jira-etc/75862)
**Author:** [@Maheshwar\_SL](https://discuss.elastic.co/u/Maheshwar_SL)\ **Replies:** 1\ **Last updated:** [February 21, 2017, 11:30am UTC](https://discuss.elastic.co/t/input-plugins-for-google-drive-sharepoint-jira-etc/75862 "2017-02-21T11:30:50Z")
I want to fetch the data from various repositories. In the available plugins for logstash I did not find plugins for Google Drive, Sharepoint, JIRA, Yammer. I want to know whether they are in development or already avail… --- ## [Is logstash input stage multithreaded?](https://discuss.elastic.co/t/is-logstash-input-stage-multithreaded/75858)
**Author:** [@Zeeshan\_Alam](https://discuss.elastic.co/u/Zeeshan_Alam)\ **Replies:** 0\ **Last updated:** [February 21, 2017, 10:27am UTC](https://discuss.elastic.co/t/is-logstash-input-stage-multithreaded/75858 "2017-02-21T10:27:50Z")
I was looking at the logstash pipeline.workers option which states that -w, --pipeline.workers COUNT Sets the number of pipeline workers to run. This option sets the number of workers that will, in parallel, execute… --- ## [Multiple Insert Statements Example for theangryangel/logstash-output-jdbc](https://discuss.elastic.co/t/multiple-insert-statements-example-for-theangryangel-logstash-output-jdbc/75856)
**Author:** [@gringo](https://discuss.elastic.co/u/gringo)\ **Replies:** 0\ **Last updated:** [February 21, 2017, 10:13am UTC](https://discuss.elastic.co/t/multiple-insert-statements-example-for-theangryangel-logstash-output-jdbc/75856 "2017-02-21T10:13:07Z")
I am currently using the theangryangel/logstash-output-jdbc. I am trying to have a statement field that contains two insert statement so that I can write once together into the DB. Does anyone has any example? I am… --- ## [\_grokparsefailure and \_geoip\_lookup\_failure when upgrading to ELK 5.0](https://discuss.elastic.co/t/-grokparsefailure-and--geoip-lookup-failure-when-upgrading-to-elk-5-0/75742)
**Author:** [@Jemli\_Fathi](https://discuss.elastic.co/u/Jemli_Fathi)\ **Replies:** 2\ **Last updated:** [February 21, 2017, 9:15am UTC](https://discuss.elastic.co/t/-grokparsefailure-and--geoip-lookup-failure-when-upgrading-to-elk-5-0/75742 "2017-02-21T09:15:04Z")
I have installed an ELK Stack log parsing pipeline and I was using ES 2.4, LS 2.4 and Kibana 4, and everything woks fine. Now I am trying to upgrade my Stack into ELK 5.0, unfortunately, I am getting \_grokparsefailure a… --- ## [Logstash if statement with regex example](https://discuss.elastic.co/t/logstash-if-statement-with-regex-example/75706)
**Author:** [@gringo](https://discuss.elastic.co/u/gringo)\ **Replies:** 6\ **Last updated:** [February 21, 2017, 8:32am UTC](https://discuss.elastic.co/t/logstash-if-statement-with-regex-example/75706 "2017-02-21T08:32:35Z")
Can anyone show me how an if statement with a regex look like in logstash? my statement s if \[fieldname\] =~ /^\[0-9\]\*$/ if \[fieldname\] =~ "^\[0-9\]\*$" does not work What I intend to do is to check if the "fieldname" con… --- ## [How to use HTTP\_POLLER through proxy server](https://discuss.elastic.co/t/how-to-use-http-poller-through-proxy-server/54524)
**Author:** [@webvisor](https://discuss.elastic.co/u/webvisor)\ **Replies:** 1\ **Last updated:** [February 20, 2017, 3:01pm UTC](https://discuss.elastic.co/t/how-to-use-http-poller-through-proxy-server/54524 "2017-02-20T15:01:20Z")
Anyone knows how to get a webpage using HTTP\_POLLER and using a proxyserver using NTLM authentication? I noticed the 'proxy' setting, but documentation only shows this: Value type is \<\> \* There is no default value fo… --- ## [Wrong Long vlaue when store long value in es](https://discuss.elastic.co/t/wrong-long-vlaue-when-store-long-value-in-es/53882)
**Author:** [@yzhang226](https://discuss.elastic.co/u/yzhang226)\ **Replies:** 5\ **Last updated:** [February 21, 2017, 7:22am UTC](https://discuss.elastic.co/t/wrong-long-vlaue-when-store-long-value-in-es/53882 "2017-02-21T07:22:16Z")
today i met one long value convert issue, for clear seeing this issue, i capture one image, as following: logstash: 2.1.1 elasticsearch: 2.1.1 completely logstash config file is shown as following, input { stdin … --- ## [Undefined method \`items' for nil:NilClass\>](https://discuss.elastic.co/t/undefined-method-items-for-nil-nilclass/74792)
**Author:** [@jamal](https://discuss.elastic.co/u/jamal)\ **Replies:** 5\ **Last updated:** [February 21, 2017, 7:11am UTC](https://discuss.elastic.co/t/undefined-method-items-for-nil-nilclass/74792 "2017-02-21T07:11:35Z")
here is the configuration of logstash input { rss { interval =\> 60000 url =\> "http://internews.org/rss.xml" type =\> "tribune" } } filter { if \[type\] == "tribune" { ruby … --- ## [Is it possible to increment or decrement a time field based on seconds and store it as another attribute?](https://discuss.elastic.co/t/is-it-possible-to-increment-or-decrement-a-time-field-based-on-seconds-and-store-it-as-another-attribute/75583)
**Author:** [@elasticheart](https://discuss.elastic.co/u/elasticheart)\ **Replies:** 6\ **Last updated:** [February 21, 2017, 4:46am UTC](https://discuss.elastic.co/t/is-it-possible-to-increment-or-decrement-a-time-field-based-on-seconds-and-store-it-as-another-attribute/75583 "2017-02-21T04:46:17Z")
Hi, I am using ELK GA 5.0.0. I have a time entry like 06:52:02 and a delay entry (in seconds) like 1.016. I want to push another attribure to elasticsearch, by subtracting seconds from timefield, say 06:52:01 in this c… --- ## [Confusion about the plugin about logstash-input-exec and logstash-output-tcp](https://discuss.elastic.co/t/confusion-about-the-plugin-about-logstash-input-exec-and-logstash-output-tcp/75466)
**Author:** [@intelyu0921](https://discuss.elastic.co/u/intelyu0921)\ **Replies:** 6\ **Last updated:** [February 21, 2017, 2:51am UTC](https://discuss.elastic.co/t/confusion-about-the-plugin-about-logstash-input-exec-and-logstash-output-tcp/75466 "2017-02-21T02:51:19Z")
As most friends in our forum, I am also a newbie about our ELK stack. Here I get the Logstash 2.4.0 environment, then I receive the request: we want to collect contents of the file /home/yhh/VMState.3 and empty the file … --- ## [CPU use grows to 100%, Index rate to almost 0](https://discuss.elastic.co/t/cpu-use-grows-to-100-index-rate-to-almost-0/75710)
**Author:** [@andreas\_at\_work](https://discuss.elastic.co/u/andreas_at_work)\ **Replies:** 1\ **Last updated:** [February 20, 2017, 9:41pm UTC](https://discuss.elastic.co/t/cpu-use-grows-to-100-index-rate-to-almost-0/75710 "2017-02-20T21:41:31Z")
Hey, Configuration Elastic Cluster (3 master nodes, 2 coordination nodes, 6 data nodes) + 2 Logstash (5.2.1) nodes Elastic Cluster status is green: Nodes: 11 Indices: 23 Memory: 57GB / 249GB Total Shards: 150 Unassign… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1031) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1033)