# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=759 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 760 --- ## [Logstash crashing Elastic](https://discuss.elastic.co/t/logstash-crashing-elastic/132043)
**Author:** [@safhw91](https://discuss.elastic.co/u/safhw91)\ **Replies:** 2\ **Last updated:** [May 16, 2018, 7:33am UTC](https://discuss.elastic.co/t/logstash-crashing-elastic/132043 "2018-05-16T07:33:38Z")
I have been parsing logs using logstash's grok parser log by log, and recently one of my pipelines.yml files which contains 100+ input files has been crashing my elastic server (out of memory). After reading this, my gu… --- ## [%{host} %{message} error logstash](https://discuss.elastic.co/t/host-message-error-logstash/132040)
**Author:** [@Alok](https://discuss.elastic.co/u/Alok)\ **Replies:** 1\ **Last updated:** [May 16, 2018, 7:10am UTC](https://discuss.elastic.co/t/host-message-error-logstash/132040 "2018-05-16T07:10:31Z")
I am trying to put some data from protgres to Elasticsearch through jdbc. When I execute the command /bin/logstash -f \ the pipeline gets started , the execution reaches to the point where select query… --- ## [Separate Message into different event](https://discuss.elastic.co/t/separate-message-into-different-event/132015)
**Author:** [@peterch](https://discuss.elastic.co/u/peterch)\ **Replies:** 4\ **Last updated:** [May 16, 2018, 6:53am UTC](https://discuss.elastic.co/t/separate-message-into-different-event/132015 "2018-05-16T06:53:22Z")
Dear All, I tried to forward some log to ELK from a device. I find all event merge into one message. May I know is it possible to separate into different event? here's the log example. Thanks \<51\>May 16 10:10:35 xp SYS… --- ## [Logstash xml parsing problems](https://discuss.elastic.co/t/logstash-xml-parsing-problems/128115)
**Author:** [@povisx1](https://discuss.elastic.co/u/povisx1)\ **Replies:** 8\ **Last updated:** [May 16, 2018, 6:13am UTC](https://discuss.elastic.co/t/logstash-xml-parsing-problems/128115 "2018-05-16T06:13:06Z")
Hello, I have the problem with xml parsing, my parsed result looks like : {"newMSISDN":"%{parsedMSISDN}","offset":52307874,"host":"miram.int.bite.lt","prospector":{"type":"log"},"@version":"1","@timestamp":"2018-04-16T… --- ## [How do I install logstash ver 6.2.3 by installing from package Repositories](https://discuss.elastic.co/t/how-do-i-install-logstash-ver-6-2-3-by-installing-from-package-repositories/132030)
**Author:** [@seungtae](https://discuss.elastic.co/u/seungtae)\ **Replies:** 4\ **Last updated:** [May 16, 2018, 6:08am UTC](https://discuss.elastic.co/t/how-do-i-install-logstash-ver-6-2-3-by-installing-from-package-repositories/132030 "2018-05-16T06:08:35Z")
Hi. I installed logstash by package repositories. It was ver 6.2.4. But, if logstash version 6.2.4, there is an Error like this: \[ERROR\] 2018-05-16 14:00:23.797 \[Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jru… --- ## [Grok-Filter not working](https://discuss.elastic.co/t/grok-filter-not-working/131009)
**Author:** [@Dante91](https://discuss.elastic.co/u/Dante91)\ **Replies:** 10\ **Last updated:** [May 16, 2018, 5:33am UTC](https://discuss.elastic.co/t/grok-filter-not-working/131009 "2018-05-16T05:33:22Z")
Hey, I want to log my MikroTik-Router. Thatfor I get for example the following log: May/03/2018 17:37:01 system,info,account user admin logged out from 192.168.88.1 via telnet With help from the grok debugger at gro… --- ## [Stream data from ELK to google cloud services](https://discuss.elastic.co/t/stream-data-from-elk-to-google-cloud-services/130909)
**Author:** [@mayurshah](https://discuss.elastic.co/u/mayurshah)\ **Replies:** 1\ **Last updated:** [May 16, 2018, 2:00am UTC](https://discuss.elastic.co/t/stream-data-from-elk-to-google-cloud-services/130909 "2018-05-16T02:00:41Z")
Hi, I need some suggestion/options to stream data out of our ELK stack to Google Cloud services like PubSub, Dataflow. Currently we have Filebeat-\>Logstash-\>Elastic-\>Kibana setup. I found option to send data from Fileb… --- ## [Logstash with Elasticsearch: Update the field which involves json object on the script section](https://discuss.elastic.co/t/logstash-with-elasticsearch-update-the-field-which-involves-json-object-on-the-script-section/132012)
**Author:** [@sivaa2017](https://discuss.elastic.co/u/sivaa2017)\ **Replies:** 0\ **Last updated:** [May 16, 2018, 1:52am UTC](https://discuss.elastic.co/t/logstash-with-elasticsearch-update-the-field-which-involves-json-object-on-the-script-section/132012 "2018-05-16T01:52:25Z")
Initially, i have inserted the below data into elasticsearch {"id":1234,"name":"Siva","City":"Chennai"} Currently, I am trying to update the above record by adding additional columns, {"phoneType":"Personal","PhoneNum… --- ## [Best practice for log filter for multiple log](https://discuss.elastic.co/t/best-practice-for-log-filter-for-multiple-log/131860)
**Author:** [@Parthgandhi](https://discuss.elastic.co/u/Parthgandhi)\ **Replies:** 1\ **Last updated:** [May 15, 2018, 9:31pm UTC](https://discuss.elastic.co/t/best-practice-for-log-filter-for-multiple-log/131860 "2018-05-15T21:31:55Z")
HI, I am working on setting up ELK stack for aggregating my IIS and IVR logs. Both services run on different servers. I have already set up logstash pipeline for IIS logs. i need to set it up for IVR logs. Shall i use t… --- ## [Duplicate content in indexes](https://discuss.elastic.co/t/duplicate-content-in-indexes/131821)
**Author:** [@xternal](https://discuss.elastic.co/u/xternal)\ **Replies:** 4\ **Last updated:** [May 15, 2018, 9:00pm UTC](https://discuss.elastic.co/t/duplicate-content-in-indexes/131821 "2018-05-15T21:00:15Z")
Hi, I was hoping someone might be able to help me with this logstash.conf. Everything from beats is ending up in the wazuh index and the winlogbeat index. However the wazuh data does not end up in the winglogbeat index.… --- ## [Logstash Alter filter not found](https://discuss.elastic.co/t/logstash-alter-filter-not-found/125278)
**Author:** [@motts](https://discuss.elastic.co/u/motts)\ **Replies:** 6\ **Last updated:** [May 15, 2018, 5:59pm UTC](https://discuss.elastic.co/t/logstash-alter-filter-not-found/125278 "2018-05-15T17:59:20Z")
Hello, I am on CentOS 6.9 and running LS 6.x. I have the Alter filter plugin installed, but when running LS, there is an error and LS will not start: \[2018-03-22T18:41:50,722\]\[ERROR\]\[logstash.agent \] Failed … --- ## [Grok filter server logs with kv plugin](https://discuss.elastic.co/t/grok-filter-server-logs-with-kv-plugin/131336)
**Author:** [@ram\_kriz](https://discuss.elastic.co/u/ram_kriz)\ **Replies:** 4\ **Last updated:** [May 15, 2018, 4:55pm UTC](https://discuss.elastic.co/t/grok-filter-server-logs-with-kv-plugin/131336 "2018-05-15T16:55:43Z")
Hi All, I have started using the ELK stack(6.2) version and trying to parse the below logs. Input: 018-04-20T21:51:06.365164500-0400 level=7 \[Proc1\] \[Proc2\] \[ID=22\] \[0001232800011fe8\] \[01285 01388\] \[Proc3\] "\[key1:N/A … --- ## [Output Sample for HTTP](https://discuss.elastic.co/t/output-sample-for-http/131810)
**Author:** [@sm00thindian](https://discuss.elastic.co/u/sm00thindian)\ **Replies:** 2\ **Last updated:** [May 15, 2018, 3:50pm UTC](https://discuss.elastic.co/t/output-sample-for-http/131810 "2018-05-15T15:50:57Z")
I'd like to see the data going to one of my http outputs, is there a simple way to see this. I do not have control of the http destination. I have created another output to stdout and that is fine however I want to con… --- ## [Increment @timestamp with a float field](https://discuss.elastic.co/t/increment-timestamp-with-a-float-field/131915)
**Author:** [@Federico\_Foschini](https://discuss.elastic.co/u/Federico_Foschini)\ **Replies:** 2\ **Last updated:** [May 15, 2018, 3:44pm UTC](https://discuss.elastic.co/t/increment-timestamp-with-a-float-field/131915 "2018-05-15T15:44:21Z")
Hello, I'm parsing some logs and I have a timestamp field (that goes in @timestamp) and a float field mapped in duration. What I'm trying to do is to copy the @timestamp to another field, and then add the duration to t… --- ## [Logstash grok - How to match not(negate) the pattern in grok](https://discuss.elastic.co/t/logstash-grok-how-to-match-not-negate-the-pattern-in-grok/131922)
**Author:** [@arun\_prasath1](https://discuss.elastic.co/u/arun_prasath1)\ **Replies:** 2\ **Last updated:** [May 15, 2018, 3:19pm UTC](https://discuss.elastic.co/t/logstash-grok-how-to-match-not-negate-the-pattern-in-grok/131922 "2018-05-15T15:19:08Z")
I have bunch of log files which are named using its version numbers like below. To match the filename having compA and 2-0/3-0 version, I have like below projectA-compA-test.log projectA-compA-2-0-test.log projectA-co… --- ## [Consume data from sharded RabbitMQ queue](https://discuss.elastic.co/t/consume-data-from-sharded-rabbitmq-queue/130248)
**Author:** [@peter.ridzi](https://discuss.elastic.co/u/peter.ridzi)\ **Replies:** 6\ **Last updated:** [May 15, 2018, 3:06pm UTC](https://discuss.elastic.co/t/consume-data-from-sharded-rabbitmq-queue/130248 "2018-05-15T15:06:49Z")
Hi, I would like to set up a Logstash instance to consume data from a sharded RabbitMQ queue. I am using RabbitMQ 3.3.5 and Logstash 5.2 right now. I installed the rabbitmq-sharding plugin (https://github.com/rabbitmq/… --- ## [Email attachements](https://discuss.elastic.co/t/email-attachements/131917)
**Author:** [@rouchad\_rouchad](https://discuss.elastic.co/u/rouchad_rouchad)\ **Replies:** 2\ **Last updated:** [May 15, 2018, 2:45pm UTC](https://discuss.elastic.co/t/email-attachements/131917 "2018-05-15T14:45:16Z")
i use elasticSearch 6.2 i want transfer my database(emails) from my gmail to elasticsearch the probleme is that elastcisearch can't read the emails which contains attachements like PDF , XML files i found that i need… --- ## [An issue on jdbc input seems to hang logstash](https://discuss.elastic.co/t/an-issue-on-jdbc-input-seems-to-hang-logstash/131957)
**Author:** [@Nicolas\_Labrot](https://discuss.elastic.co/u/Nicolas_Labrot)\ **Replies:** 0\ **Last updated:** [May 15, 2018, 2:34pm UTC](https://discuss.elastic.co/t/an-issue-on-jdbc-input-seems-to-hang-logstash/131957 "2018-05-15T14:34:07Z")
Hello, Logstash: 6.2.4 Oracle driver: 12.1.0.2.0 JDK 8 101-b13 A JDBC plugin pulls rows from an Oracle table. Once a day, the DB is restarted and during the restart, Logstash throws this error: 30574 rufus-scheduler… --- ## [Logstash Configuration Error (Failed to execute action)](https://discuss.elastic.co/t/logstash-configuration-error-failed-to-execute-action/131475)
**Author:** [@NotPete](https://discuss.elastic.co/u/NotPete)\ **Replies:** 3\ **Last updated:** [May 15, 2018, 2:31pm UTC](https://discuss.elastic.co/t/logstash-configuration-error-failed-to-execute-action/131475 "2018-05-15T14:31:53Z")
Hi, I am literally 2 days old with ELK Stack and I know I have missed something. I'm running version 6.2.4 on RHEL 7.4. I'm getting the following error looping in /var/log/logstash/logstash-plain.log: \[2018-05-11T07:… --- ## [Dissect FILEPATH](https://discuss.elastic.co/t/dissect-filepath/131857)
**Author:** [@frnklnfrancis](https://discuss.elastic.co/u/frnklnfrancis)\ **Replies:** 1\ **Last updated:** [May 15, 2018, 1:55pm UTC](https://discuss.elastic.co/t/dissect-filepath/131857 "2018-05-15T13:55:50Z")
Hi, I'm pretty new to this, can someone help me to dissect the below file path? (\_LQ3zMFgBEee5Sr7bPizxug) /com.ti.b.ld.nr.ee.kb.tiontion.jar/src/main/java/com/nk/fiswoa/sb/desk/del/SBBdDkWhLtDO.java I only want the la… --- ## [Handle Blank data in CSV - Ingest By Using Logstash](https://discuss.elastic.co/t/handle-blank-data-in-csv-ingest-by-using-logstash/131867)
**Author:** [@nandan\_bigdata](https://discuss.elastic.co/u/nandan_bigdata)\ **Replies:** 1\ **Last updated:** [May 15, 2018, 1:50pm UTC](https://discuss.elastic.co/t/handle-blank-data-in-csv-ingest-by-using-logstash/131867 "2018-05-15T13:50:07Z")
Hi, I have CSV file as below col1,col2,col3,col4 A,B,C,D ,,E,F ,G,H ,,,I J,K,,, More then 10 milion records, I am using below config code as :- input { file { path =\> "/home/nandan/data/data.csv" start\_p… --- ## [Filter firewall : Fortigat](https://discuss.elastic.co/t/filter-firewall-fortigat/131769)
**Author:** [@asalma](https://discuss.elastic.co/u/asalma)\ **Replies:** 4\ **Last updated:** [May 15, 2018, 1:45pm UTC](https://discuss.elastic.co/t/filter-firewall-fortigat/131769 "2018-05-15T13:45:45Z")
Hi, Please, can someone help me to create a filter grok to separate the fields of logs that came from a Fortigat firewall. This is an example of log : May 9 09:40:15 frkeofwglo01 date=2018-05-09 time=09:40:15 devnam… --- ## [Starting Logstash as a service](https://discuss.elastic.co/t/starting-logstash-as-a-service/131940)
**Author:** [@asalma](https://discuss.elastic.co/u/asalma)\ **Replies:** 0\ **Last updated:** [May 15, 2018, 1:07pm UTC](https://discuss.elastic.co/t/starting-logstash-as-a-service/131940 "2018-05-15T13:07:58Z")
Hi, I have a probleme, my logstash can't start as a service, it starts only with config.reload.automatic !!! Help !! --- ## [Require example/tutorial for Logstash S3 Input Plugin](https://discuss.elastic.co/t/require-example-tutorial-for-logstash-s3-input-plugin/131907)
**Author:** [@Anilkumar\_Patel](https://discuss.elastic.co/u/Anilkumar_Patel)\ **Replies:** 2\ **Last updated:** [May 15, 2018, 12:10pm UTC](https://discuss.elastic.co/t/require-example-tutorial-for-logstash-s3-input-plugin/131907 "2018-05-15T12:10:45Z")
I am facing some issues in configuring the logstash s3 input plugin. I tried searching for a tutorial for this plugin but couldn't find any. Could you please provide a link to the tutorial/examples for this plugin if the… --- ## [How to convert nanoseconds to milliseconds](https://discuss.elastic.co/t/how-to-convert-nanoseconds-to-milliseconds/131895)
**Author:** [@stevebeattie](https://discuss.elastic.co/u/stevebeattie)\ **Replies:** 2\ **Last updated:** [May 15, 2018, 11:56am UTC](https://discuss.elastic.co/t/how-to-convert-nanoseconds-to-milliseconds/131895 "2018-05-15T11:56:40Z")
Hi, I have a timestamp as nanoseconds in an access log that I want to store in ES. I understand that the timestamp in ES is stored as milliseconds and that I'll need to convert. Will I have to use mutate and gsub to t… --- ## [Logstash configuration error while reading data from s3](https://discuss.elastic.co/t/logstash-configuration-error-while-reading-data-from-s3/131918)
**Author:** [@Anilkumar\_Patel](https://discuss.elastic.co/u/Anilkumar_Patel)\ **Replies:** 0\ **Last updated:** [May 15, 2018, 11:20am UTC](https://discuss.elastic.co/t/logstash-configuration-error-while-reading-data-from-s3/131918 "2018-05-15T11:20:20Z")
I am facing an error while configuring logstash to fetch data from s3 below is the configuration of logstash s3 input plugin. input { s3 { access\_key\_id =\> "################" bucket =\> "anil-data" region =\> "######… --- ## [Logstash filter no longer works after upgrading](https://discuss.elastic.co/t/logstash-filter-no-longer-works-after-upgrading/131899)
**Author:** [@kernelpanic](https://discuss.elastic.co/u/kernelpanic)\ **Replies:** 1\ **Last updated:** [May 15, 2018, 9:55am UTC](https://discuss.elastic.co/t/logstash-filter-no-longer-works-after-upgrading/131899 "2018-05-15T09:55:17Z")
Hello all, when I was using Logstash 5.3.0 I had the following filter which worked without issue: if \[event\_id\] == 257 and \[computer\_name\] == "EXCHANGE-SERVER-1.DOMAIN.COM" or \[computer\_name\] == "EXCHANGE-SERVER-2.DOMAI… --- ## [Connection refused logstash](https://discuss.elastic.co/t/connection-refused-logstash/131897)
**Author:** [@asalma](https://discuss.elastic.co/u/asalma)\ **Replies:** 1\ **Last updated:** [May 15, 2018, 9:54am UTC](https://discuss.elastic.co/t/connection-refused-logstash/131897 "2018-05-15T09:54:59Z")
Hi, Please, I have a problem with my logstash, he can't start. I have this error : 2018-05-15T11:43:38.172+0200 ERROR pipeline/output.go:74 Failed to connect: dial tcp \[::1\]:5044: getsockopt: connection refused … --- ## [Logstash dissect offline installation](https://discuss.elastic.co/t/logstash-dissect-offline-installation/131876)
**Author:** [@solo](https://discuss.elastic.co/u/solo)\ **Replies:** 0\ **Last updated:** [May 15, 2018, 7:50am UTC](https://discuss.elastic.co/t/logstash-dissect-offline-installation/131876 "2018-05-15T07:50:10Z")
Im trying to install dissect offline using the local option. But it times out. cd /usr/share/logstash/ sudo bin/logstash-plugin install --local /usr/share/logstash/logstash/logstash-filter-dissect-1.1.4.gem ERROR: Some… --- ## [Automatic create index at 12 AM IST](https://discuss.elastic.co/t/automatic-create-index-at-12-am-ist/131880)
**Author:** [@Arpit\_Gulati](https://discuss.elastic.co/u/Arpit_Gulati)\ **Replies:** 2\ **Last updated:** [May 15, 2018, 8:16am UTC](https://discuss.elastic.co/t/automatic-create-index-at-12-am-ist/131880 "2018-05-15T08:16:52Z")
Hi Team, I have a problem that automatic index is created at 5:30 according to UTC . But I want that index should be created at 12 AM according to IST ( INDIAN Standard time). What should I do? Kindly, help me out . I … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=758) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=760)