# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1077 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1078 --- ## [Use Logstash and Syslog-ng server](https://discuss.elastic.co/t/use-logstash-and-syslog-ng-server/40901)
**Author:** [@Miguel\_Bessa](https://discuss.elastic.co/u/Miguel_Bessa)\ **Replies:** 7\ **Last updated:** [November 17, 2016, 7:23am UTC](https://discuss.elastic.co/t/use-logstash-and-syslog-ng-server/40901 "2016-11-17T07:23:40Z")
Hi, I have in machine logstash and elasticsearch installed and I have in other server configured syslog-ng to collect logs. How I can use logstash to connect syslog-ng server to get logs and parse data locally to ela… --- ## [Sync up changes from mongodb to elasticsearch](https://discuss.elastic.co/t/sync-up-changes-from-mongodb-to-elasticsearch/66194)
**Author:** [@sonpv1](https://discuss.elastic.co/u/sonpv1)\ **Replies:** 3\ **Last updated:** [November 17, 2016, 6:53am UTC](https://discuss.elastic.co/t/sync-up-changes-from-mongodb-to-elasticsearch/66194 "2016-11-17T06:53:51Z")
Hi experts, I'm working on syncing up data from mongodb including changes to ES. Could you please help me to provide me with your idea. Thank you so much! --- ## [One grok pattern for Messages,Secure ,cron and yum.log](https://discuss.elastic.co/t/one-grok-pattern-for-messages-secure-cron-and-yum-log/65902)
**Author:** [@NITIN-BHAISARE](https://discuss.elastic.co/u/NITIN-BHAISARE)\ **Replies:** 1\ **Last updated:** [November 17, 2016, 6:38am UTC](https://discuss.elastic.co/t/one-grok-pattern-for-messages-secure-cron-and-yum-log/65902 "2016-11-17T06:38:29Z")
Hello, I would need some help in generating a grok pattern. I am trying to build a grok pattern for Messages,Secure ,cron and yum.log. Although we have a grok pattern for secure and messages file,but it says grok pars… --- ## [Logstash doesn't use all CPU available](https://discuss.elastic.co/t/logstash-doesnt-use-all-cpu-available/66189)
**Author:** [@Oliver\_Hernandez](https://discuss.elastic.co/u/Oliver_Hernandez)\ **Replies:** 13\ **Last updated:** [November 17, 2016, 6:16am UTC](https://discuss.elastic.co/t/logstash-doesnt-use-all-cpu-available/66189 "2016-11-17T06:16:00Z")
Hi guys, Hope you can help me. i'm running logstash 5.0.1 on a Blade server with 40 cores and 256GB of RAM. I'm getting a LOT of logs into a single file (around 50.000 lines per minute) from a service we're running i… --- ## [Logstash to ignore beat.hostname to set as host , and use logstash server hostname as o/p](https://discuss.elastic.co/t/logstash-to-ignore-beat-hostname-to-set-as-host-and-use-logstash-server-hostname-as-o-p/66325)
**Author:** [@Prashant\_Agrawal](https://discuss.elastic.co/u/Prashant_Agrawal)\ **Replies:** 0\ **Last updated:** [November 17, 2016, 5:20am UTC](https://discuss.elastic.co/t/logstash-to-ignore-beat-hostname-to-set-as-host-and-use-logstash-server-hostname-as-o-p/66325 "2016-11-17T05:20:18Z")
Hi , we have a use case where we send data from beat to logstash to ES. so while doing this logstash adds host as per recieved in beats.hostname , so how can I make logstash to output the host value as per logstash m… --- ## [Init.d missing - CentOS 6.8 - 5.0.0rc1](https://discuss.elastic.co/t/init-d-missing-centos-6-8-5-0-0rc1/63674)
**Author:** [@jaaplaya](https://discuss.elastic.co/u/jaaplaya)\ **Replies:** 6\ **Last updated:** [November 16, 2016, 8:44pm UTC](https://discuss.elastic.co/t/init-d-missing-centos-6-8-5-0-0rc1/63674 "2016-11-16T20:44:43Z")
Running CentOS 6.8, installing 5.0.0 RC1 through yum following the instructions in the docs (https://www.elastic.co/guide/en/logstash/5.0/installing-logstash.html) and the init.d scripts are missing for starting/stopping… --- ## [Why logstash-output-jms isn't part of the pluggins in the official documentation?](https://discuss.elastic.co/t/why-logstash-output-jms-isnt-part-of-the-pluggins-in-the-official-documentation/53874)
**Author:** [@vmcano](https://discuss.elastic.co/u/vmcano)\ **Replies:** 4\ **Last updated:** [November 16, 2016, 7:15pm UTC](https://discuss.elastic.co/t/why-logstash-output-jms-isnt-part-of-the-pluggins-in-the-official-documentation/53874 "2016-11-16T19:15:35Z")
I need to use JMS as a Logstash output and, after looking for some information, I found there is a plugin called logstash-output-jms. However, I found it in GitHub (https://github.com/logstash-plugins/logstash-output-jms… --- ## [How to break up long strings without embeded newlines appearing?](https://discuss.elastic.co/t/how-to-break-up-long-strings-without-embeded-newlines-appearing/66288)
**Author:** [@gyre](https://discuss.elastic.co/u/gyre)\ **Replies:** 0\ **Last updated:** [November 16, 2016, 6:53pm UTC](https://discuss.elastic.co/t/how-to-break-up-long-strings-without-embeded-newlines-appearing/66288 "2016-11-16T18:53:03Z")
I have the following snippet of logstash config that contains a long string. (the actual string is around 200 characters long as it contains a lot of %{blah} references.) output { file { path =\> "/tmp/foo… --- ## [Filter tweets based on date?](https://discuss.elastic.co/t/filter-tweets-based-on-date/66277)
**Author:** [@jamal](https://discuss.elastic.co/u/jamal)\ **Replies:** 0\ **Last updated:** [November 16, 2016, 5:40pm UTC](https://discuss.elastic.co/t/filter-tweets-based-on-date/66277 "2016-11-16T17:40:31Z")
i use logstash\_twitter\_input plugin to get data i have a field by the name @timestam =\> 2011-11-16T03:44:01.103Z now i want to filter it with logstash\_date\_filter plugin to get new data not old data for exampe today … --- ## [Fields parsed by Grok show up as unknown type](https://discuss.elastic.co/t/fields-parsed-by-grok-show-up-as-unknown-type/66254)
**Author:** [@adrianp](https://discuss.elastic.co/u/adrianp)\ **Replies:** 4\ **Last updated:** [November 16, 2016, 5:07pm UTC](https://discuss.elastic.co/t/fields-parsed-by-grok-show-up-as-unknown-type/66254 "2016-11-16T17:07:09Z")
So my fields are being parsed and I see them in Kibana when I run queries, but all my parsed fields are listed as "unknown" for their data types. Also, if I try to create a visualization, it's not listing any of my parse… --- ## [Logstash adds extra new line characters](https://discuss.elastic.co/t/logstash-adds-extra-new-line-characters/60660)
**Author:** [@andreys](https://discuss.elastic.co/u/andreys)\ **Replies:** 6\ **Last updated:** [November 16, 2016, 4:47pm UTC](https://discuss.elastic.co/t/logstash-adds-extra-new-line-characters/60660 "2016-11-16T16:47:46Z")
I am using next configuration file input { stdin { codec =\> multiline { pattern =\> "%{APEX\_TIMESTAMP}" negate =\> true what =\> "previous" } } } filter { grok { patterns\_dir =\> \["/etc/… --- ## [Elasticsearch index issue with logstash-netflow in 5.x](https://discuss.elastic.co/t/elasticsearch-index-issue-with-logstash-netflow-in-5-x/65854)
**Author:** [@Anantha\_Ravi](https://discuss.elastic.co/u/Anantha_Ravi)\ **Replies:** 7\ **Last updated:** [November 16, 2016, 3:56pm UTC](https://discuss.elastic.co/t/elasticsearch-index-issue-with-logstash-netflow-in-5-x/65854 "2016-11-16T15:56:44Z")
I find elastcisearch is not creating the index based on the below logstash configuration . have installed netflow-codec through logstash-plugin . logstash config: \[root@SERVER logstash\]# cat /etc/logstash/conf.d/logs… --- ## [Different headers - logstash csv filter](https://discuss.elastic.co/t/different-headers-logstash-csv-filter/37588)
**Author:** [@rfplourenco](https://discuss.elastic.co/u/rfplourenco)\ **Replies:** 1\ **Last updated:** [November 16, 2016, 3:38pm UTC](https://discuss.elastic.co/t/different-headers-logstash-csv-filter/37588 "2016-11-16T15:38:48Z")
Hello, I'm having issues when feeding CSV data into elastic. My logstash .conf file monitors a results folder and injects new .csv data into elastic with the file input plugin. The issue is that the .csv file contains… --- ## [Correlation of two log (sources)](https://discuss.elastic.co/t/correlation-of-two-log-sources/41752)
**Author:** [@mr\_rob](https://discuss.elastic.co/u/mr_rob)\ **Replies:** 6\ **Last updated:** [November 16, 2016, 3:38pm UTC](https://discuss.elastic.co/t/correlation-of-two-log-sources/41752 "2016-11-16T15:38:34Z")
Hello Newbie here, can someone help me out please? 2 log servers producing: connection logs (user ip etc) (generated every 10 mins) http logs (user actions) (generated on the fly) the connection logs have username an… --- ## [Join two logs by common field](https://discuss.elastic.co/t/join-two-logs-by-common-field/65622)
**Author:** [@Basem\_Mohammed](https://discuss.elastic.co/u/Basem_Mohammed)\ **Replies:** 2\ **Last updated:** [November 16, 2016, 2:51pm UTC](https://discuss.elastic.co/t/join-two-logs-by-common-field/65622 "2016-11-16T14:51:46Z")
Hello, I am trying to get a useful information from two logs format. Let me Start the call flow. Msgs:- dhcp,info dhcp 414 assigned 10.101.193.4 to F0:::::40 hotspot,account,info,debug cb8xgh (10.101.193.4): logge… --- ## [Problem with LS 5.0.1](https://discuss.elastic.co/t/problem-with-ls-5-0-1/66251)
**Author:** [@Kryten](https://discuss.elastic.co/u/Kryten)\ **Replies:** 0\ **Last updated:** [November 16, 2016, 1:43pm UTC](https://discuss.elastic.co/t/problem-with-ls-5-0-1/66251 "2016-11-16T13:43:52Z")
No longer needed. Please delete. --- ## [Grok pattern is not working](https://discuss.elastic.co/t/grok-pattern-is-not-working/65976)
**Author:** [@Ramesh\_Janagam](https://discuss.elastic.co/u/Ramesh_Janagam)\ **Replies:** 3\ **Last updated:** [November 16, 2016, 1:01pm UTC](https://discuss.elastic.co/t/grok-pattern-is-not-working/65976 "2016-11-16T13:01:06Z")
Hi All, I need help on grok patterns. I have a log file where few fields are missing(several fields actually), based on that I have come up with below grok patterns but its not working. In the grok contructor - it sho… --- ## [(Logstash newbie) Logstash 5.0. 150k flows/min limit](https://discuss.elastic.co/t/logstash-newbie-logstash-5-0-150k-flows-min-limit/65690)
**Author:** [@Simon2](https://discuss.elastic.co/u/Simon2)\ **Replies:** 4\ **Last updated:** [November 16, 2016, 12:44pm UTC](https://discuss.elastic.co/t/logstash-newbie-logstash-5-0-150k-flows-min-limit/65690 "2016-11-16T12:44:34Z")
Hello. I just started using Logstash 5, but I am stuck, and cannot figure out why. So if anybody have any idea/suggestions please let me know. I use Logstash to capture netflow V9 from UDP ports and forward it toward… --- ## [Logstash jdbc plugin: SELECT count(\*) on startup](https://discuss.elastic.co/t/logstash-jdbc-plugin-select-count-on-startup/56376)
**Author:** [@lorello](https://discuss.elastic.co/u/lorello)\ **Replies:** 1\ **Last updated:** [November 16, 2016, 12:38pm UTC](https://discuss.elastic.co/t/logstash-jdbc-plugin-select-count-on-startup/56376 "2016-11-16T12:38:10Z")
I have a table on MySQL server that I want to push to ES. I can accomplish this with the jdbc plugin, but I found it a little heavy on the mysql server, each time it starts it run a SELECT count(\*) AS \`count\` FROM … --- ## [Issue with XML filter - xmlparsefailure](https://discuss.elastic.co/t/issue-with-xml-filter-xmlparsefailure/66159)
**Author:** [@Lazypete](https://discuss.elastic.co/u/Lazypete)\ **Replies:** 2\ **Last updated:** [November 16, 2016, 12:27pm UTC](https://discuss.elastic.co/t/issue-with-xml-filter-xmlparsefailure/66159 "2016-11-16T12:27:25Z")
Ok I'm trying to use LEK (logstash, elasticsearch, kibana) to parse and display logs from a IBM datapower appliance. My problem is that whatever XML I send to logstash all I get is The unformatted data and this beauty … --- ## [Multiple hosts output](https://discuss.elastic.co/t/multiple-hosts-output/66226)
**Author:** [@elastic-fan](https://discuss.elastic.co/u/elastic-fan)\ **Replies:** 6\ **Last updated:** [November 16, 2016, 12:26pm UTC](https://discuss.elastic.co/t/multiple-hosts-output/66226 "2016-11-16T12:26:16Z")
Hi, i have a local ES and also elastic cloud ES, the local ES dont have uid and pwd where as the elastic cloud ES has , how can i specify this in logstash output output { if \[type\] == "output" { elasticsear… --- ## [If Conditional around input filter doesn't appear to be working](https://discuss.elastic.co/t/if-conditional-around-input-filter-doesnt-appear-to-be-working/66166)
**Author:** [@sjivan](https://discuss.elastic.co/u/sjivan)\ **Replies:** 2\ **Last updated:** [November 16, 2016, 10:27am UTC](https://discuss.elastic.co/t/if-conditional-around-input-filter-doesnt-appear-to-be-working/66166 "2016-11-16T10:27:00Z")
I'm trying to setup conditional file input sources based on an environment variable and it was reading the file even when the condition was false. To simplify the test I setup input { if 1 == 2 { fil… --- ## [Input JSON file and change type of fields](https://discuss.elastic.co/t/input-json-file-and-change-type-of-fields/66228)
**Author:** [@Jorge1](https://discuss.elastic.co/u/Jorge1)\ **Replies:** 0\ **Last updated:** [November 16, 2016, 9:32am UTC](https://discuss.elastic.co/t/input-json-file-and-change-type-of-fields/66228 "2016-11-16T09:32:41Z")
Hi, I have a json file with a fixed structure and dynamic structure. In the fixed struture I changed the type of data using a template, this part is easy. Logstash.conf input { stdin { codec =\> json … --- ## [How to run the schedule every five minutes in Logstash 5.0?](https://discuss.elastic.co/t/how-to-run-the-schedule-every-five-minutes-in-logstash-5-0/66222)
**Author:** [@Kulasangar\_Gowrisang](https://discuss.elastic.co/u/Kulasangar_Gowrisang)\ **Replies:** 2\ **Last updated:** [November 16, 2016, 9:47am UTC](https://discuss.elastic.co/t/how-to-run-the-schedule-every-five-minutes-in-logstash-5-0/66222 "2016-11-16T09:47:26Z")
I'm having a jdbc connection within my input in logstash where as I'm trying to execute the query according to the schedule property. I went through jdbc and rufus-scheduler, but still pretty unclear of what those five s… --- ## [Drop log messages that don't contain absolute path to application data](https://discuss.elastic.co/t/drop-log-messages-that-dont-contain-absolute-path-to-application-data/65930)
**Author:** [@triceras](https://discuss.elastic.co/u/triceras)\ **Replies:** 5\ **Last updated:** [November 16, 2016, 7:53am UTC](https://discuss.elastic.co/t/drop-log-messages-that-dont-contain-absolute-path-to-application-data/65930 "2016-11-16T07:53:45Z")
Hi Forum, The application I am working with outputs several log messages as below. The message logged by the application displays the root path until the absolute path where the application resides in reverse order. I w… --- ## [How to check if there is some specific field under fields](https://discuss.elastic.co/t/how-to-check-if-there-is-some-specific-field-under-fields/66201)
**Author:** [@elasticheart](https://discuss.elastic.co/u/elasticheart)\ **Replies:** 1\ **Last updated:** [November 16, 2016, 5:55am UTC](https://discuss.elastic.co/t/how-to-check-if-there-is-some-specific-field-under-fields/66201 "2016-11-16T05:55:50Z")
Hi, I have a kafka input and sample data is something like; { "@timestamp": "2016-11-16T05:26:16.137Z", "beat": { "hostname": "localhost", "name": "server", "version": "5.0.0" }, "fields": { "… --- ## [Index only specific data in elasticsearch from kafka input](https://discuss.elastic.co/t/index-only-specific-data-in-elasticsearch-from-kafka-input/65970)
**Author:** [@scireon](https://discuss.elastic.co/u/scireon)\ **Replies:** 1\ **Last updated:** [November 16, 2016, 5:14am UTC](https://discuss.elastic.co/t/index-only-specific-data-in-elasticsearch-from-kafka-input/65970 "2016-11-16T05:14:03Z")
I am using kafka as my input and put it in elasticsearch (output) input { kafka { topics =\> \["maxwell"\] codec =\> json } } filter { } output { stdout { codec =\> rubydebug } elasticsearch… --- ## [Help on date filter plugin to parse 5/24/2016 6:10:20 AM](https://discuss.elastic.co/t/help-on-date-filter-plugin-to-parse-5-24-2016-6-10-20-am/54578)
**Author:** [@tianlinzx](https://discuss.elastic.co/u/tianlinzx)\ **Replies:** 2\ **Last updated:** [November 16, 2016, 4:40am UTC](https://discuss.elastic.co/t/help-on-date-filter-plugin-to-parse-5-24-2016-6-10-20-am/54578 "2016-11-16T04:40:52Z")
I tried to use date plugin to parase date as 5/24/2016 6:10:20 AM and 5/24/2016 1:00:15 AM. My configuration is as below. but it doesn't work. Could anyone help me? date { match =\> \[ "ACS\_Timestamp", "M d YYYY H:m… --- ## [Referrer Inconsistencies](https://discuss.elastic.co/t/referrer-inconsistencies/66149)
**Author:** [@cjs6891](https://discuss.elastic.co/u/cjs6891)\ **Replies:** 2\ **Last updated:** [November 15, 2016, 9:55pm UTC](https://discuss.elastic.co/t/referrer-inconsistencies/66149 "2016-11-15T21:55:32Z")
I'm using logstash to ingest nginx logs and have found inconsistencies with the referrer field. For example, a google referrer is correct while other referrers aren't NGINX Log Format: log\_format main\_ext '$remote\_… --- ## [Why am I getting GrokTimeout for a my simple log?](https://discuss.elastic.co/t/why-am-i-getting-groktimeout-for-a-my-simple-log/65847)
**Author:** [@seanziee](https://discuss.elastic.co/u/seanziee)\ **Replies:** 3\ **Last updated:** [November 15, 2016, 9:20pm UTC](https://discuss.elastic.co/t/why-am-i-getting-groktimeout-for-a-my-simple-log/65847 "2016-11-15T21:20:26Z")
Hi, My grok keeps timing out while trying to filter a not so complicated/long log file. I have been trying for days now to no avail. I have tested all the patterns in the grok constructor with no issue so the main probl… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1076) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1078)