# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=655 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 656 --- ## [Logstash: Sent formatted metrics to an Elasticsearch index](https://discuss.elastic.co/t/logstash-sent-formatted-metrics-to-an-elasticsearch-index/155130)
**Author:** [@pkaramol](https://discuss.elastic.co/u/pkaramol)\ **Replies:** 0\ **Last updated:** [November 2, 2018, 8:13am UTC](https://discuss.elastic.co/t/logstash-sent-formatted-metrics-to-an-elasticsearch-index/155130 "2018-11-02T08:13:59Z")
I am able to capture some metrics in stdout from events in logstash using the relevant plugin as follows: in my filter file: if "somekeyword" in \[tags\] { metrics { meter =\> "events\_somekeyword" flush\_… --- ## [Getting open\_file OPEN\_WARN\_INTERVAL is '300' error in logstash](https://discuss.elastic.co/t/getting-open-file-open-warn-interval-is-300-error-in-logstash/155155)
**Author:** [@yash\_mangla](https://discuss.elastic.co/u/yash_mangla)\ **Replies:** 0\ **Last updated:** [November 2, 2018, 11:12am UTC](https://discuss.elastic.co/t/getting-open-file-open-warn-interval-is-300-error-in-logstash/155155 "2018-11-02T11:12:24Z")
Hi Folks, I'm running technology cluster in production. where there is alert.json file which will update if any new events is coming across. I already allow that file permission using chmod -R 777 \… --- ## [Unable to parse only one conf file present in conf.d](https://discuss.elastic.co/t/unable-to-parse-only-one-conf-file-present-in-conf-d/155143)
**Author:** [@Nithani25](https://discuss.elastic.co/u/Nithani25)\ **Replies:** 1\ **Last updated:** [November 2, 2018, 11:03am UTC](https://discuss.elastic.co/t/unable-to-parse-only-one-conf-file-present-in-conf-d/155143 "2018-11-02T11:03:48Z")
HI Team, I have installed logstash rpm package on to my prodcution environment. I am able to start the service OK and was able to check the new conf file with bin/logstash --debug option ' but starting it as a service… --- ## [How can I start logstash on windows machine and any instructions to ingest csv data on elasticsearch using logstash](https://discuss.elastic.co/t/how-can-i-start-logstash-on-windows-machine-and-any-instructions-to-ingest-csv-data-on-elasticsearch-using-logstash/155141)
**Author:** [@sachi79](https://discuss.elastic.co/u/sachi79)\ **Replies:** 0\ **Last updated:** [November 2, 2018, 9:54am UTC](https://discuss.elastic.co/t/how-can-i-start-logstash-on-windows-machine-and-any-instructions-to-ingest-csv-data-on-elasticsearch-using-logstash/155141 "2018-11-02T09:54:23Z")
I am trying to ingest a csv file using logstash into my elasticsearch on a local windows machine, if someone can guide me it will be of great help. --- ## [Covert the date format to "TIMESTAMP\_ISO8601"](https://discuss.elastic.co/t/covert-the-date-format-to-timestamp-iso8601/153599)
**Author:** [@vishnuduttpv](https://discuss.elastic.co/u/vishnuduttpv)\ **Replies:** 7\ **Last updated:** [November 2, 2018, 7:53am UTC](https://discuss.elastic.co/t/covert-the-date-format-to-timestamp-iso8601/153599 "2018-11-02T07:53:05Z")
I wish to parse the tomcat logs. The tomcat logs start with different time formats as follows: 1.23-Oct-2018 13:19:41.677 INFO \[main\] org.apache.catalina.startup.VersionLoggerListener.log Server version: Apa… --- ## [How to parse the json field with grok](https://discuss.elastic.co/t/how-to-parse-the-json-field-with-grok/155103)
**Author:** [@talon](https://discuss.elastic.co/u/talon)\ **Replies:** 2\ **Last updated:** [November 2, 2018, 7:04am UTC](https://discuss.elastic.co/t/how-to-parse-the-json-field-with-grok/155103 "2018-11-02T07:04:45Z")
HI here is my message {"@timestamp":"2018-11-01T03:05:28.391Z","records":\[{"properties":{"Keywords":0,"Message":"w3wp.exe Information: 0 :\\r\\n| 2018-10-31 02:43:16.396 | AuthorizeActivityAttributeBase:IsAuthorizedAsync… --- ## [Unable to connect from kafka as input and output as elasticsearch](https://discuss.elastic.co/t/unable-to-connect-from-kafka-as-input-and-output-as-elasticsearch/154997)
**Author:** [@vasanthprthv](https://discuss.elastic.co/u/vasanthprthv)\ **Replies:** 0\ **Last updated:** [November 1, 2018, 10:58am UTC](https://discuss.elastic.co/t/unable-to-connect-from-kafka-as-input-and-output-as-elasticsearch/154997 "2018-11-01T10:58:34Z")
Iam trying to use logstash to connect data from kafka to elasticsearch. when I use stdin {} in the input , output is coming in the elasticsearch but when i give kafka { } as input , the output was not coming to elastics… --- ## [Can I pass data between 2 input plugin of logstash](https://discuss.elastic.co/t/can-i-pass-data-between-2-input-plugin-of-logstash/154851)
**Author:** [@patrick007](https://discuss.elastic.co/u/patrick007)\ **Replies:** 11\ **Last updated:** [November 2, 2018, 5:18am UTC](https://discuss.elastic.co/t/can-i-pass-data-between-2-input-plugin-of-logstash/154851 "2018-11-02T05:18:34Z")
input { kafka { // Pass a value from kafka to jdbc input plugin } jdbc { } } --- ## [API call to find frequency of occurrence of values of a given field](https://discuss.elastic.co/t/api-call-to-find-frequency-of-occurrence-of-values-of-a-given-field/154836)
**Author:** [@GitSpree23](https://discuss.elastic.co/u/GitSpree23)\ **Replies:** 1\ **Last updated:** [October 31, 2018, 1:19pm UTC](https://discuss.elastic.co/t/api-call-to-find-frequency-of-occurrence-of-values-of-a-given-field/154836 "2018-10-31T13:19:46Z")
My index contains the "IP" field. While there are ~4000 records in my index, there are only ~10 unique IPs in it. I want an API call which will return the frequency of occurrence of each IP in the index. I have develope… --- ## [Merging results of similar aggregation buckets to perform union and intersect](https://discuss.elastic.co/t/merging-results-of-similar-aggregation-buckets-to-perform-union-and-intersect/155099)
**Author:** [@dx123](https://discuss.elastic.co/u/dx123)\ **Replies:** 0\ **Last updated:** [November 2, 2018, 1:41am UTC](https://discuss.elastic.co/t/merging-results-of-similar-aggregation-buckets-to-perform-union-and-intersect/155099 "2018-11-02T01:41:35Z")
Here is an example of the aggregation bucket results that I have now: "aggregations":{ "by ip":{ "buckets":\[ { "key":"192.168.0.1", "doc\_count":2, "by date":{ "buckets":\[ { "key\_as\_string":"2018-01-01T00:00:00.… --- ## [Authentication using non xpack logstash](https://discuss.elastic.co/t/authentication-using-non-xpack-logstash/155090)
**Author:** [@rjconroy](https://discuss.elastic.co/u/rjconroy)\ **Replies:** 0\ **Last updated:** [November 1, 2018, 11:27pm UTC](https://discuss.elastic.co/t/authentication-using-non-xpack-logstash/155090 "2018-11-01T23:27:08Z")
Forgive what is im sure a dumb question but if I using straight logstash, without xpack or other commercial things just straight ELK, what authentications are available, if any? Its there any roles or control to indices… --- ## [Configure Logstash to send MS SQL ErrorLog](https://discuss.elastic.co/t/configure-logstash-to-send-ms-sql-errorlog/154794)
**Author:** [@Mustapha\_MJ](https://discuss.elastic.co/u/Mustapha_MJ)\ **Replies:** 1\ **Last updated:** [November 1, 2018, 8:38pm UTC](https://discuss.elastic.co/t/configure-logstash-to-send-ms-sql-errorlog/154794 "2018-11-01T20:38:10Z")
Hi guys i spend almost a week without result trying to find out how to send ErrorLogs of my MS SQL Server to Elasticsearch via logstash, in another word i want a logstash config file sample --- ## [Xml Filter Not Extracting All Values From XML Document](https://discuss.elastic.co/t/xml-filter-not-extracting-all-values-from-xml-document/155074)
**Author:** [@Sankalan\_Banerjee](https://discuss.elastic.co/u/Sankalan_Banerjee)\ **Replies:** 0\ **Last updated:** [November 1, 2018, 8:00pm UTC](https://discuss.elastic.co/t/xml-filter-not-extracting-all-values-from-xml-document/155074 "2018-11-01T20:00:10Z")
I have an xml document having nested array elements for Test Case Run Summary . Since a Test Case contains multiple Test Steps, so in the document you will see multiple node and each node contains corresponding step inf… --- ## [Timestamp from filename](https://discuss.elastic.co/t/timestamp-from-filename/154938)
**Author:** [@SusantaBN](https://discuss.elastic.co/u/SusantaBN)\ **Replies:** 2\ **Last updated:** [November 1, 2018, 7:58pm UTC](https://discuss.elastic.co/t/timestamp-from-filename/154938 "2018-11-01T19:58:18Z")
I have a requirement to put the timestamp on log file name and subsequently extract the timestamp from filename and add as a field in the elastic search index. e.g. file name: 2018-10-31\_13-11-25.33.csv My custom grok … --- ## [Catching exceptions from output plugins](https://discuss.elastic.co/t/catching-exceptions-from-output-plugins/154743)
**Author:** [@Layne\_nz](https://discuss.elastic.co/u/Layne_nz)\ **Replies:** 2\ **Last updated:** [November 1, 2018, 7:52pm UTC](https://discuss.elastic.co/t/catching-exceptions-from-output-plugins/154743 "2018-11-01T19:52:27Z")
Hi All, Like in this thread I'm trying to catch exceptions from output plugins so we can handle the failed input. It's been a few years, is there a better way than forking the output plugin? Thanks, Layne --- ## [Custom patterns\_dir for all groks?](https://discuss.elastic.co/t/custom-patterns-dir-for-all-groks/155069)
**Author:** [@ryannelle](https://discuss.elastic.co/u/ryannelle)\ **Replies:** 0\ **Last updated:** [November 1, 2018, 7:35pm UTC](https://discuss.elastic.co/t/custom-patterns-dir-for-all-groks/155069 "2018-11-01T19:35:33Z")
:male\_detective: I have multiple grok filters but I point all of them to a single patterns\_dir like below. :male\_detective: grok { patterns\_dir =\> \["/usr/share/logstash/patterns" \] } grok { patterns\_dir =\> \["/usr/sh… --- ## [Support for alternate 'endpoint' and 'protocol' parameters in Logstash S3 output plugin?](https://discuss.elastic.co/t/support-for-alternate-endpoint-and-protocol-parameters-in-logstash-s3-output-plugin/155047)
**Author:** [@tad](https://discuss.elastic.co/u/tad)\ **Replies:** 0\ **Last updated:** [November 1, 2018, 4:26pm UTC](https://discuss.elastic.co/t/support-for-alternate-endpoint-and-protocol-parameters-in-logstash-s3-output-plugin/155047 "2018-11-01T16:26:21Z")
I'm hoping to find out if there's any support for specifying an alternate 'endpoint' and 'protocol' in the Logstash S3 output plugin, similar to what is provided in the following S3 Repository plugin? We have an interna… --- ## [WARNING: Could not find logstash.yml which is typically located in $LS\_HOME/config or /etc/logstash. Problem](https://discuss.elastic.co/t/warning-could-not-find-logstash-yml-which-is-typically-located-in-ls-home-config-or-etc-logstash-problem/155053)
**Author:** [@vnallamothu](https://discuss.elastic.co/u/vnallamothu)\ **Replies:** 0\ **Last updated:** [November 1, 2018, 5:13pm UTC](https://discuss.elastic.co/t/warning-could-not-find-logstash-yml-which-is-typically-located-in-ls-home-config-or-etc-logstash-problem/155053 "2018-11-01T17:13:49Z")
I've installed Logstash on Amazon Linux 64 bit server. I've installed java version "1.8.0\_191" on the same server and followed following steps: Download and install the public signing key: rpm --import https://artifac… --- ## [Can\`t start up logstash at installed xpack](https://discuss.elastic.co/t/can-t-start-up-logstash-at-installed-xpack/154373)
**Author:** [@KevinLiu](https://discuss.elastic.co/u/KevinLiu)\ **Replies:** 1\ **Last updated:** [November 1, 2018, 4:28pm UTC](https://discuss.elastic.co/t/can-t-start-up-logstash-at-installed-xpack/154373 "2018-11-01T16:28:54Z")
use offical document config logstash\_internal grant logstash\_writer role,modifyed logstash.yml write logstash\_system username and password,but logstash still can\`t start up,log file like this: \[2018-10-29T04:44:57,702\]\[W… --- ## [Error while connecting Logstash to Elastic search](https://discuss.elastic.co/t/error-while-connecting-logstash-to-elastic-search/154965)
**Author:** [@saya](https://discuss.elastic.co/u/saya)\ **Replies:** 2\ **Last updated:** [November 1, 2018, 2:36pm UTC](https://discuss.elastic.co/t/error-while-connecting-logstash-to-elastic-search/154965 "2018-11-01T14:36:42Z")
I am very very new to ELK, I installed ELK version 5.6.12 on CentOS sever. Elasticsearch and Kibana works fine. But I cannot connect Logstash to Elastic search. I have set environment variable as export JAVA\_HOME=/usr/l… --- ## [Store XML file on S3 and send specific XML TAGs to ElasticSearch to index](https://discuss.elastic.co/t/store-xml-file-on-s3-and-send-specific-xml-tags-to-elasticsearch-to-index/154941)
**Author:** [@costamarcelo](https://discuss.elastic.co/u/costamarcelo)\ **Replies:** 1\ **Last updated:** [November 1, 2018, 2:20pm UTC](https://discuss.elastic.co/t/store-xml-file-on-s3-and-send-specific-xml-tags-to-elasticsearch-to-index/154941 "2018-11-01T14:20:00Z")
Hi folks Can I use LogStash to sent only some TAGs from an XML file and store the physical XML file on an S3? |----\> ElasticSearch | XML --\> LogStash… --- ## [Issue with logstash configuration](https://discuss.elastic.co/t/issue-with-logstash-configuration/155031)
**Author:** [@jamesp220291](https://discuss.elastic.co/u/jamesp220291)\ **Replies:** 0\ **Last updated:** [November 1, 2018, 2:17pm UTC](https://discuss.elastic.co/t/issue-with-logstash-configuration/155031 "2018-11-01T14:17:17Z")
Hi All New to the ELK Stack, I am wanting to use it to parse my nginx access logs. Having two issue, removing client ip field breaks geoip and overwriting the message..doesnt work.. it just gives the full message. M… --- ## [Setting @timestamp once MM YYYY etc are parsed](https://discuss.elastic.co/t/setting-timestamp-once-mm-yyyy-etc-are-parsed/155029)
**Author:** [@P\_Larsen](https://discuss.elastic.co/u/P_Larsen)\ **Replies:** 0\ **Last updated:** [November 1, 2018, 2:07pm UTC](https://discuss.elastic.co/t/setting-timestamp-once-mm-yyyy-etc-are-parsed/155029 "2018-11-01T14:07:19Z")
I am able to grok out MM YYYY etc, the part I can't get a handle on is the logstash conf files for non-standard time formats and setting @timestamp Sample log \\ Nov 01 2018 07:42:25: %ASA-6-106100: access-list global-i… --- ## [ERROR logstash.inputs.udp Exception in inputworker exception ZeroDivisionError divided by 0](https://discuss.elastic.co/t/error-logstash-inputs-udp-exception-in-inputworker-exception-zerodivisionerror-divided-by-0/155022)
**Author:** [@Muruga](https://discuss.elastic.co/u/Muruga)\ **Replies:** 0\ **Last updated:** [November 1, 2018, 12:56pm UTC](https://discuss.elastic.co/t/error-logstash-inputs-udp-exception-in-inputworker-exception-zerodivisionerror-divided-by-0/155022 "2018-11-01T12:56:34Z")
Logstash shutsdown in less than 3 minutes with the below error: \[2018-11-01T23:49:50,288\]\[ERROR\]\[logstash.inputs.udp \] Exception in inputworker {"exception"=\>#\, "backtrace"=\>\["org/j… --- ## [Remove objects in arrays](https://discuss.elastic.co/t/remove-objects-in-arrays/154870)
**Author:** [@shiv94](https://discuss.elastic.co/u/shiv94)\ **Replies:** 1\ **Last updated:** [October 31, 2018, 6:22pm UTC](https://discuss.elastic.co/t/remove-objects-in-arrays/154870 "2018-10-31T18:22:13Z")
Hello, using json filter in logstash and trying to remove the field device.vendor and also split the field of similar kind I tried to do remove\_field =\> "\[device\]\[vendor\]\[0\]" but it did not work. how do I remove or spli… --- ## [Parse source field](https://discuss.elastic.co/t/parse-source-field/154987)
**Author:** [@luggo](https://discuss.elastic.co/u/luggo)\ **Replies:** 3\ **Last updated:** [November 1, 2018, 10:40am UTC](https://discuss.elastic.co/t/parse-source-field/154987 "2018-11-01T10:40:29Z")
Hey community, my source field (provided by filebeat) contains information about the host, the application and the version. Those are information that need to be in my log entries in Elasticsearch. Example source: C:\\… --- ## [File input adds escaping](https://discuss.elastic.co/t/file-input-adds-escaping/154992)
**Author:** [@sapirkallmann](https://discuss.elastic.co/u/sapirkallmann)\ **Replies:** 0\ **Last updated:** [November 1, 2018, 10:39am UTC](https://discuss.elastic.co/t/file-input-adds-escaping/154992 "2018-11-01T10:39:27Z")
I would like to disable escaping when receiving events from a file. For instance, here is the event (input is a file): a="hello" b="world" This is the output: a=\\"hello\\" b=\\"world\\"\\r" While using other inputs such… --- ## [Logslash splits files unexpectedly](https://discuss.elastic.co/t/logslash-splits-files-unexpectedly/154990)
**Author:** [@L\_Y](https://discuss.elastic.co/u/L_Y)\ **Replies:** 0\ **Last updated:** [November 1, 2018, 10:33am UTC](https://discuss.elastic.co/t/logslash-splits-files-unexpectedly/154990 "2018-11-01T10:33:55Z")
hi there I am using logstash as a tool to store files on elasticsearch the is no manipulation on the file content (message) every day logstash splits a small number of files at an unexpected line the rest of the file… --- ## [XML parsing using logstash](https://discuss.elastic.co/t/xml-parsing-using-logstash/154811)
**Author:** [@ramanna\_hk](https://discuss.elastic.co/u/ramanna_hk)\ **Replies:** 2\ **Last updated:** [November 1, 2018, 6:30am UTC](https://discuss.elastic.co/t/xml-parsing-using-logstash/154811 "2018-11-01T06:30:18Z")
Hi, I am working on Nessus logs ex: https://gist.github.com/najmisyahir/7a1d2c17257ced61257e3033cd55d090 But i am getting results in array for each fields, please help me to convert to json records. FYI: i am using lo… --- ## [Parse json in "logstash or filebeat from a text file line?](https://discuss.elastic.co/t/parse-json-in-logstash-or-filebeat-from-a-text-file-line/154956)
**Author:** [@hgaxifodu](https://discuss.elastic.co/u/hgaxifodu)\ **Replies:** 0\ **Last updated:** [November 1, 2018, 4:40am UTC](https://discuss.elastic.co/t/parse-json-in-logstash-or-filebeat-from-a-text-file-line/154956 "2018-11-01T04:40:57Z")
"message": "update请求\[{"UserName":"fullwang","CardType":"1","IdNum":"1000001","AccountId":"10000128","PassengerState":1,"FailMsg":"over"}\]net/http: request canceled (Client.Timeout exceeded while awaiting headers)", "mes… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=654) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=656)