# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=774 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 775 --- ## [Output to file, codec for windows events](https://discuss.elastic.co/t/output-to-file-codec-for-windows-events/128696)
**Author:** [@Rakkijake](https://discuss.elastic.co/u/Rakkijake)\ **Replies:** 5\ **Last updated:** [April 23, 2018, 11:56am UTC](https://discuss.elastic.co/t/output-to-file-codec-for-windows-events/128696 "2018-04-23T11:56:36Z")
Hello, Setting up my first Elastic, and ran to an issue with logstash output to file. I am trying to get output both on elastic and to file, and both are working otherwise fine; however, I dont get any timestamp on the… --- ## [Gsub remplace double double ("") quotes by space](https://discuss.elastic.co/t/gsub-remplace-double-double-quotes-by-space/129070)
**Author:** [@Pierre2](https://discuss.elastic.co/u/Pierre2)\ **Replies:** 5\ **Last updated:** [April 23, 2018, 11:54am UTC](https://discuss.elastic.co/t/gsub-remplace-double-double-quotes-by-space/129070 "2018-04-23T11:54:29Z")
I use the csv filter but some field are like that : ""toto toto" sip:+4999999999@10.58.12.00;user=phone;tag=8b1adde9f755210b19546545b056e2c555504" The csv filter can't parse fields if there is more that 2 double quotes… --- ## [Compare two datasets](https://discuss.elastic.co/t/compare-two-datasets/129006)
**Author:** [@counterf](https://discuss.elastic.co/u/counterf)\ **Replies:** 6\ **Last updated:** [April 23, 2018, 11:40am UTC](https://discuss.elastic.co/t/compare-two-datasets/129006 "2018-04-23T11:40:52Z")
Hello, I have a list containing millions of malicious IP addresses, and I am using ELK to get and analyze data from multiple firewalls. I received millions of logs per days from the firewall and my malicious IP address… --- ## [Need help Flattening the json array](https://discuss.elastic.co/t/need-help-flattening-the-json-array/128625)
**Author:** [@karnamonkster](https://discuss.elastic.co/u/karnamonkster)\ **Replies:** 4\ **Last updated:** [April 23, 2018, 10:23am UTC](https://discuss.elastic.co/t/need-help-flattening-the-json-array/128625 "2018-04-23T10:23:36Z")
Hi, I am trying to flatten the following json: { "beans" : \[ { "name" : "Hadoop:service=NameNode,name=NameNodeInfo", "modelerType" : "org.apache.hadoop.hdfs.server.namenode.FSNamesystem", "Total" : 976194928640, "Upg… --- ## [How can i get aggregated count of particular errors](https://discuss.elastic.co/t/how-can-i-get-aggregated-count-of-particular-errors/128538)
**Author:** [@Utkarsh\_Tripathi](https://discuss.elastic.co/u/Utkarsh_Tripathi)\ **Replies:** 7\ **Last updated:** [April 23, 2018, 10:01am UTC](https://discuss.elastic.co/t/how-can-i-get-aggregated-count-of-particular-errors/128538 "2018-04-23T10:01:13Z")
hi, I want to get number of times particular error occured on my server.After parsing logs look as below mentioned. \< { "path" =\> "/home/utkarsh/sample.log2016\_07\_21.txt", "time" =\> "00:00:05,626", "host" =\> "Lapto… --- ## [Logstash RabbitMQ Output](https://discuss.elastic.co/t/logstash-rabbitmq-output/129051)
**Author:** [@paul.rapp](https://discuss.elastic.co/u/paul.rapp)\ **Replies:** 4\ **Last updated:** [April 23, 2018, 9:49am UTC](https://discuss.elastic.co/t/logstash-rabbitmq-output/129051 "2018-04-23T09:49:38Z")
I am having trouble getting logstash to output to rabbit on a non-default port. My output configuration for logstash is ... rabbitmq { durable =\> "true" persistent =\> "true" exchange =\> "secure-logs-exch" exchange\_… --- ## [Logstash-input-mongodb configuration file](https://discuss.elastic.co/t/logstash-input-mongodb-configuration-file/129002)
**Author:** [@Nfekih](https://discuss.elastic.co/u/Nfekih)\ **Replies:** 6\ **Last updated:** [April 22, 2018, 10:42pm UTC](https://discuss.elastic.co/t/logstash-input-mongodb-configuration-file/129002 "2018-04-22T22:42:45Z")
Hi all, I am trying to pull data out of mongodb and to process it with Logstash. Know that I want to pull all the data from a DB named "test" and so I started with the collection named "student". This is my configurati… --- ## [Two outputs with different data](https://discuss.elastic.co/t/two-outputs-with-different-data/128966)
**Author:** [@AlexH](https://discuss.elastic.co/u/AlexH)\ **Replies:** 3\ **Last updated:** [April 22, 2018, 10:31pm UTC](https://discuss.elastic.co/t/two-outputs-with-different-data/128966 "2018-04-22T22:31:22Z")
I want to take one input, do some filters, send output to one ES index, do more filters, then send to a second ES index. The result should be one index with a set of fields, and the second index has a mostly different s… --- ## [Index Name not expected](https://discuss.elastic.co/t/index-name-not-expected/128573)
**Author:** [@dhenshaw](https://discuss.elastic.co/u/dhenshaw)\ **Replies:** 1\ **Last updated:** [April 22, 2018, 8:14pm UTC](https://discuss.elastic.co/t/index-name-not-expected/128573 "2018-04-22T20:14:22Z")
Hello, I have an ElasticStack setup, and am looking at simplifying the footprint. As such I have setup an rsyslog server, which will ship the logs to logstash, store in elasticsearch, and viewable in Kibana. Using guid… --- ## [Filebeat - Failed to publish events caused by: read tcp x.x.x.x:36196-\>x.x.x.x:5045: i/o timeout](https://discuss.elastic.co/t/filebeat-failed-to-publish-events-caused-by-read-tcp-x-x-x-x-36196-x-x-x-x-5045-i-o-timeout/128994)
**Author:** [@Tomer\_Lev1](https://discuss.elastic.co/u/Tomer_Lev1)\ **Replies:** 0\ **Last updated:** [April 22, 2018, 9:08am UTC](https://discuss.elastic.co/t/filebeat-failed-to-publish-events-caused-by-read-tcp-x-x-x-x-36196-x-x-x-x-5045-i-o-timeout/128994 "2018-04-22T09:08:46Z")
Hi i'm running into a problem while sending logs via filebeat to logstash: In short - Can't see logs in kibana - when tailing the filebeat log I see a lot of these: ERROR logstash/async.go:235 Failed to publish events … --- ## [Logstash syslog input malformed haproxy grok lines](https://discuss.elastic.co/t/logstash-syslog-input-malformed-haproxy-grok-lines/128990)
**Author:** [@asaf400](https://discuss.elastic.co/u/asaf400)\ **Replies:** 0\ **Last updated:** [April 22, 2018, 7:18am UTC](https://discuss.elastic.co/t/logstash-syslog-input-malformed-haproxy-grok-lines/128990 "2018-04-22T07:18:22Z")
Hello, I'll explain first what are the symptoms I am seeing, and then describe the attempts made to remedy that.. I'm using haproxy, and have configuered it to send it's logs, with the UDP protocol in syslog format (I … --- ## [Logstash after installation not working in windows 7](https://discuss.elastic.co/t/logstash-after-installation-not-working-in-windows-7/128950)
**Author:** [@Raghavendra\_Bhat1](https://discuss.elastic.co/u/Raghavendra_Bhat1)\ **Replies:** 2\ **Last updated:** [April 21, 2018, 5:30pm UTC](https://discuss.elastic.co/t/logstash-after-installation-not-working-in-windows-7/128950 "2018-04-21T17:30:07Z")
Hi, I am new to elastic and logstash. I have my Oracle database and want to index documents in to elastic search. I installed elastic search and Kibana locally in my machine Windows 7 and are running fine. I downloaded… --- ## [Extract json field and overwrite document](https://discuss.elastic.co/t/extract-json-field-and-overwrite-document/128738)
**Author:** [@Sneijky](https://discuss.elastic.co/u/Sneijky)\ **Replies:** 1\ **Last updated:** [April 19, 2018, 5:34pm UTC](https://discuss.elastic.co/t/extract-json-field-and-overwrite-document/128738 "2018-04-19T17:34:26Z")
Hello, I have the following json: \[ { "key": "Key-1", "field1": "hello", "field2": "world" }, { "key": "Key-2", "field1": "hello", "field2": "world" } \] and the following logstash file: in… --- ## [Grok filter and split message](https://discuss.elastic.co/t/grok-filter-and-split-message/128654)
**Author:** [@arkady\_renko](https://discuss.elastic.co/u/arkady_renko)\ **Replies:** 2\ **Last updated:** [April 20, 2018, 8:52pm UTC](https://discuss.elastic.co/t/grok-filter-and-split-message/128654 "2018-04-20T20:52:11Z")
Hello, I want to put my java app logs into elasticsearch. Filebeat sends logs to logstash reads the log file. I checked my grok filter at http://grokdebug.herokuapp.com/ and looks OK. But in elastic search there is onl… --- ## [UDP listener died (SYSLOG) when running as a service](https://discuss.elastic.co/t/udp-listener-died-syslog-when-running-as-a-service/128890)
**Author:** [@packetferret](https://discuss.elastic.co/u/packetferret)\ **Replies:** 5\ **Last updated:** [April 20, 2018, 8:18pm UTC](https://discuss.elastic.co/t/udp-listener-died-syslog-when-running-as-a-service/128890 "2018-04-20T20:18:30Z")
Hello and thanks for taking a moment of your time to read over this. I am running Logstash as a collector for SYSLOG messages from networking equipment. Executing the binary while pointing to the configuration file with… --- ## [Advise for logstash filter](https://discuss.elastic.co/t/advise-for-logstash-filter/128604)
**Author:** [@Maddy\_M](https://discuss.elastic.co/u/Maddy_M)\ **Replies:** 9\ **Last updated:** [April 20, 2018, 7:34pm UTC](https://discuss.elastic.co/t/advise-for-logstash-filter/128604 "2018-04-20T19:34:35Z")
Hi, I am trying to parse below line and due to time stamp and varying delimiter I am unable to use it. Could you please advise? 18/09/2016 11:09:12:AYS2345:2 dcs:0 meta-\>info:0 … --- ## [Logstash - Using a VIP output - hosts](https://discuss.elastic.co/t/logstash-using-a-vip-output-hosts/128888)
**Author:** [@sabharadwaj](https://discuss.elastic.co/u/sabharadwaj)\ **Replies:** 3\ **Last updated:** [April 20, 2018, 7:01pm UTC](https://discuss.elastic.co/t/logstash-using-a-vip-output-hosts/128888 "2018-04-20T19:01:10Z")
I have tried searching existing threads but couldnt find any solution to this problem We are trying to use a Elasticsearch cluster VIP in Logstash but it appends 9200 to the VIP.. It doesnt work even if use globalogs.w… --- ## [Client Certificate Revocation](https://discuss.elastic.co/t/client-certificate-revocation/128919)
**Author:** [@holdenkilbride](https://discuss.elastic.co/u/holdenkilbride)\ **Replies:** 1\ **Last updated:** [April 20, 2018, 6:33pm UTC](https://discuss.elastic.co/t/client-certificate-revocation/128919 "2018-04-20T18:33:47Z")
I have a scenario where we have multiple distributed Filebeat instances sending logs to Logstash. I would like to use force\_peer authentication to verify the client certificates that were signed by a self signed authori… --- ## [Issue parsing XML File with logstash](https://discuss.elastic.co/t/issue-parsing-xml-file-with-logstash/128924)
**Author:** [@rodher](https://discuss.elastic.co/u/rodher)\ **Replies:** 0\ **Last updated:** [April 20, 2018, 6:15pm UTC](https://discuss.elastic.co/t/issue-parsing-xml-file-with-logstash/128924 "2018-04-20T18:15:40Z")
Hi to all I have an XML file, and I'm parsing it with Logstash This is my filter section: xml { source =\> message target =\> GLTData store\_xml =\> true } However, my xml file has only 2 fields: "Date", and… --- ## [Grok parser not working](https://discuss.elastic.co/t/grok-parser-not-working/128910)
**Author:** [@Maddy\_M](https://discuss.elastic.co/u/Maddy_M)\ **Replies:** 6\ **Last updated:** [April 20, 2018, 4:39pm UTC](https://discuss.elastic.co/t/grok-parser-not-working/128910 "2018-04-20T16:39:03Z")
Actual Data \ 02/05/2016 10:56:50:8125C2453:502014 dcs:1 dcs-\>info:1 info:40 info-\>tina:50 \ Actual syntax \ %{DATESTAMP:Time}:%{USER:u… --- ## [How do i get elasticsearch logs into elasticsearch?](https://discuss.elastic.co/t/how-do-i-get-elasticsearch-logs-into-elasticsearch/128758)
**Author:** [@elasticated](https://discuss.elastic.co/u/elasticated)\ **Replies:** 3\ **Last updated:** [April 20, 2018, 2:53pm UTC](https://discuss.elastic.co/t/how-do-i-get-elasticsearch-logs-into-elasticsearch/128758 "2018-04-20T14:53:32Z")
Completely new to ELK, and very confused right now, and in the asking idiotic questions phase, so bear with me... I've installed ELK on linux and the first thing I wanted to try was getting some logs into elastic and th… --- ## [Logstash does not send all my data](https://discuss.elastic.co/t/logstash-does-not-send-all-my-data/128901)
**Author:** [@Jose\_Campos](https://discuss.elastic.co/u/Jose_Campos)\ **Replies:** 0\ **Last updated:** [April 20, 2018, 2:49pm UTC](https://discuss.elastic.co/t/logstash-does-not-send-all-my-data/128901 "2018-04-20T14:49:11Z")
Hello I have a problem with logstash 6.2.4, when I run my logstash manually, elasticsearch if it shows me my full information, for example a full hour. But if I let it run on systemctl in this case I use centos, it only… --- ## [Logstash to elasticsearch](https://discuss.elastic.co/t/logstash-to-elasticsearch/128021)
**Author:** [@Senthiles](https://discuss.elastic.co/u/Senthiles)\ **Replies:** 3\ **Last updated:** [April 20, 2018, 1:56pm UTC](https://discuss.elastic.co/t/logstash-to-elasticsearch/128021 "2018-04-20T13:56:18Z")
Hi, while inserting data from database to elasticsearch index. the query statement run to fetch the no of count. Can we alter the query which runs to get the count. --- ## [Remove keys and avoid "Direct event field references have been disabled" error message](https://discuss.elastic.co/t/remove-keys-and-avoid-direct-event-field-references-have-been-disabled-error-message/128853)
**Author:** [@juan.domenech](https://discuss.elastic.co/u/juan.domenech)\ **Replies:** 6\ **Last updated:** [April 20, 2018, 1:06pm UTC](https://discuss.elastic.co/t/remove-keys-and-avoid-direct-event-field-references-have-been-disabled-error-message/128853 "2018-04-20T13:06:33Z")
Hi, I need to remove a nasty field with keys and after reading this forum I've decided to use this: ruby { code =\> "event.get('\[field\_with\_keys\]').keys.each { |k| event\['field\_with\_keys'\].remove(k) }" } mutate { remove… --- ## [Parsing Emails with logstash](https://discuss.elastic.co/t/parsing-emails-with-logstash/128567)
**Author:** [@M\_B\_I](https://discuss.elastic.co/u/M_B_I)\ **Replies:** 9\ **Last updated:** [April 20, 2018, 12:16pm UTC](https://discuss.elastic.co/t/parsing-emails-with-logstash/128567 "2018-04-20T12:16:29Z")
Hi, everyone! I'm newest in ELK, so I need help =) Could some one help me? I try to search some data from my email message. I used imap input plugin , but what should I use in filter? For example if i use rubydebug in… --- ## [Multine: Divide an email with a number of jobs in multiple events](https://discuss.elastic.co/t/multine-divide-an-email-with-a-number-of-jobs-in-multiple-events/128864)
**Author:** [@Swantje](https://discuss.elastic.co/u/Swantje)\ **Replies:** 0\ **Last updated:** [April 20, 2018, 11:26am UTC](https://discuss.elastic.co/t/multine-divide-an-email-with-a-number-of-jobs-in-multiple-events/128864 "2018-04-20T11:26:50Z")
Hey everyone, I have got some questions about grok pattern and multiline. My Input is an email from an Mailbox. This email is composed of multiple Jobs, but logstash takes it as one single event. I like to divide thi… --- ## [Pipelines YAML file is empty](https://discuss.elastic.co/t/pipelines-yaml-file-is-empty/128858)
**Author:** [@Nfekih](https://discuss.elastic.co/u/Nfekih)\ **Replies:** 2\ **Last updated:** [April 20, 2018, 11:25am UTC](https://discuss.elastic.co/t/pipelines-yaml-file-is-empty/128858 "2018-04-20T11:25:50Z")
Hi, I am trying to load a csv file to Elasticsearch using Logstash. I ran this command: bin/logstash --path.data file\_name.conf I get this error: ERROR: Pipelines YAML file is empty. Location: /home/nerifkh/Downl… --- ## [Regex on matched field](https://discuss.elastic.co/t/regex-on-matched-field/128839)
**Author:** [@prateek\_shrivastava](https://discuss.elastic.co/u/prateek_shrivastava)\ **Replies:** 1\ **Last updated:** [April 20, 2018, 10:41am UTC](https://discuss.elastic.co/t/regex-on-matched-field/128839 "2018-04-20T10:41:26Z")
I have a grok filter which is: grok { match =\> \[ "message", "^(\\s+)?%{TIMESTAMP\_ISO8601:logtime}%{SPACE}\\|%{SPACE}\\\[%{NUMBER:thread\_id}\\\]%{SPACE}\\|%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}\\|%{SPACE}%{GREEDYDATA:classname… --- ## [Extracting a specific data from a field](https://discuss.elastic.co/t/extracting-a-specific-data-from-a-field/128826)
**Author:** [@Yuvaraja\_kannan](https://discuss.elastic.co/u/Yuvaraja_kannan)\ **Replies:** 5\ **Last updated:** [April 20, 2018, 9:55am UTC](https://discuss.elastic.co/t/extracting-a-specific-data-from-a-field/128826 "2018-04-20T09:55:30Z")
Hi All, I have extracted a field from a log file which has the value \[ROBO LOG A1-B\] , is there a way to extract or trim only 'A1-B' into a separate new field. --- ## [Merge separate json objects output to an array](https://discuss.elastic.co/t/merge-separate-json-objects-output-to-an-array/128393)
**Author:** [@SKing](https://discuss.elastic.co/u/SKing)\ **Replies:** 7\ **Last updated:** [April 20, 2018, 9:20am UTC](https://discuss.elastic.co/t/merge-separate-json-objects-output-to-an-array/128393 "2018-04-20T09:20:51Z")
I am querying an elastic search instance. I end up with a series of json objects. {"business\_id": "1", "Accepts Credit Cards": true, "Price Range": 1, "type": "food"} {"business\_id": "2", "Accepts Credit Cards": tr… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=773) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=775)