# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1030 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1031 --- ## [Logstash high CPU usage with no running config](https://discuss.elastic.co/t/logstash-high-cpu-usage-with-no-running-config/76272)
**Author:** [@orhiee](https://discuss.elastic.co/u/orhiee)\ **Replies:** 2\ **Last updated:** [February 24, 2017, 10:43am UTC](https://discuss.elastic.co/t/logstash-high-cpu-usage-with-no-running-config/76272 "2017-02-24T10:43:10Z")
hello, Yesterday i installed elk on my laptop as a vm, so i have a local server to play with when i go home from work :slight\_smile: its centos 7 x64 with elk all 5.2.1 and the latest java as of yesterday. i alrea… --- ## [How to add new index pattern in kibana 4 without using another client server?](https://discuss.elastic.co/t/how-to-add-new-index-pattern-in-kibana-4-without-using-another-client-server/75994)
**Author:** [@paoloyniguez](https://discuss.elastic.co/u/paoloyniguez)\ **Replies:** 1\ **Last updated:** [February 24, 2017, 10:10am UTC](https://discuss.elastic.co/t/how-to-add-new-index-pattern-in-kibana-4-without-using-another-client-server/75994 "2017-02-24T10:10:20Z")
Hi! We have an existing ELK server which we monitor logs from a csv format file that logrotates everynight. We are using a logstash-forwarder to do this. It's using the default index pattern in kibana. I would like … --- ## [\_grokparsefailure for matching patterns](https://discuss.elastic.co/t/-grokparsefailure-for-matching-patterns/76212)
**Author:** [@Rajdeep\_Mukherjee](https://discuss.elastic.co/u/Rajdeep_Mukherjee)\ **Replies:** 2\ **Last updated:** [February 24, 2017, 10:04am UTC](https://discuss.elastic.co/t/-grokparsefailure-for-matching-patterns/76212 "2017-02-24T10:04:45Z")
contents of pattern file: ZX\_ERROR \\A!%{SPACE}%{DATE:date}%{SPACE}%{TIME:time}%{SPACE}:%{SPACE}(%{JAVACLASS:caused\_by})?(%{JAVAFILE:caused\_by})?%{GREEDYDATA:error\_message} Logstash config file: filter { if "tzx" in \[… --- ## [Logstash output to cassandra](https://discuss.elastic.co/t/logstash-output-to-cassandra/76188)
**Author:** [@agrawalsaurabh](https://discuss.elastic.co/u/agrawalsaurabh)\ **Replies:** 3\ **Last updated:** [February 24, 2017, 9:46am UTC](https://discuss.elastic.co/t/logstash-output-to-cassandra/76188 "2017-02-24T09:46:12Z")
Hi, I am looking forward to dump the output of LOGSTASH to cassandra instead of elastic search. The idea is to have cassandra as a single source of truth when it coms to ingesting data from multiple sources and use cas… --- ## [Does logstash work with java 1.7](https://discuss.elastic.co/t/does-logstash-work-with-java-1-7/76366)
**Author:** [@daaakeee](https://discuss.elastic.co/u/daaakeee)\ **Replies:** 2\ **Last updated:** [February 24, 2017, 8:59am UTC](https://discuss.elastic.co/t/does-logstash-work-with-java-1-7/76366 "2017-02-24T08:59:52Z")
I am using java 1.7.0, which version of logstash is compatible with this version of java? --- ## [Line break into log file](https://discuss.elastic.co/t/line-break-into-log-file/75777)
**Author:** [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\ **Replies:** 5\ **Last updated:** [February 24, 2017, 6:43am UTC](https://discuss.elastic.co/t/line-break-into-log-file/75777 "2017-02-24T06:43:45Z")
Hello, I would like to know how to parse log line who have many empty field and many useless field. One line have 60 values but only 20 are really usefull. I have create a grok with sixty %{DATA:name} (I would impr… --- ## [Logstash Select All in Elasticsearch Input missing documents](https://discuss.elastic.co/t/logstash-select-all-in-elasticsearch-input-missing-documents/76349)
**Author:** [@gringo](https://discuss.elastic.co/u/gringo)\ **Replies:** 0\ **Last updated:** [February 24, 2017, 5:48am UTC](https://discuss.elastic.co/t/logstash-select-all-in-elasticsearch-input-missing-documents/76349 "2017-02-24T05:48:02Z")
I did the following input { elasticsearch { .............. query =\> '{"query": { "match\_all": {} }}' } } After running the script I noticed that the number of documents in the target index is… --- ## [Write csv to elastic search by using logstash](https://discuss.elastic.co/t/write-csv-to-elastic-search-by-using-logstash/76338)
**Author:** [@Ximeng\_Zhao](https://discuss.elastic.co/u/Ximeng_Zhao)\ **Replies:** 5\ **Last updated:** [February 24, 2017, 4:10am UTC](https://discuss.elastic.co/t/write-csv-to-elastic-search-by-using-logstash/76338 "2017-02-24T04:10:24Z")
Hello All, I want to write a csv file to the elasticsearch, but after running ./logstash -f logstash.conf, es and kibana, you can check the indices is created properly but only with ten records. It doesn't import all d… --- ## [To a rsyslog+ELK user(s), can I IM you with a specific rsyslog question?](https://discuss.elastic.co/t/to-a-rsyslog-elk-user-s-can-i-im-you-with-a-specific-rsyslog-question/76333)
**Author:** [@ZillaG](https://discuss.elastic.co/u/ZillaG)\ **Replies:** 0\ **Last updated:** [February 24, 2017, 2:12am UTC](https://discuss.elastic.co/t/to-a-rsyslog-elk-user-s-can-i-im-you-with-a-specific-rsyslog-question/76333 "2017-02-24T02:12:17Z")
Though I see a lot of ELK users using rsyslog as their shipper, I won't ask rsyslog questions here since this is an ELK forum. However, can I bother one of you with a specific rsyslog configuration question, either throu… --- ## [Trouble with GeoIP mapping form logstash to elastic search](https://discuss.elastic.co/t/trouble-with-geoip-mapping-form-logstash-to-elastic-search/76045)
**Author:** [@cisaksen](https://discuss.elastic.co/u/cisaksen)\ **Replies:** 7\ **Last updated:** [February 24, 2017, 2:08am UTC](https://discuss.elastic.co/t/trouble-with-geoip-mapping-form-logstash-to-elastic-search/76045 "2017-02-24T02:08:38Z")
Brand new to this product, I have a nginx server send logs via filebeats, to logstash, which is working. I have the filter geoip which is also working as far as i can tell except that it appears that I'm missing somethi… --- ## [Are multiple grok's ok?](https://discuss.elastic.co/t/are-multiple-groks-ok/76283)
**Author:** [@mhooper](https://discuss.elastic.co/u/mhooper)\ **Replies:** 1\ **Last updated:** [February 24, 2017, 1:51am UTC](https://discuss.elastic.co/t/are-multiple-groks-ok/76283 "2017-02-24T01:51:01Z")
I am parsing generic log4net logs, so all have timestamp threadid level message but only a small percentage have something contained within the remaining message such as "User login = 'USER'". my question is can I have … --- ## [Logstash filter for multiple log files parsing](https://discuss.elastic.co/t/logstash-filter-for-multiple-log-files-parsing/76310)
**Author:** [@sam281](https://discuss.elastic.co/u/sam281)\ **Replies:** 2\ **Last updated:** [February 24, 2017, 1:50am UTC](https://discuss.elastic.co/t/logstash-filter-for-multiple-log-files-parsing/76310 "2017-02-24T01:50:16Z")
Hi, I have file beats sending data to logstash. I am trying to filter and parse the log records based on the source file. On kibana I see a field called "source" which has the source log file name for each record, … --- ## [LS 5.2: how to set different java for logstash process, installed by system-install?](https://discuss.elastic.co/t/ls-5-2-how-to-set-different-java-for-logstash-process-installed-by-system-install/76179)
**Author:** [@asp](https://discuss.elastic.co/u/asp)\ **Replies:** 1\ **Last updated:** [February 23, 2017, 5:50pm UTC](https://discuss.elastic.co/t/ls-5-2-how-to-set-different-java-for-logstash-process-installed-by-system-install/76179 "2017-02-23T17:50:47Z")
Hi, I am installing logstash from the tar. To install it as service I use the script $LS\_HOME/bin/system-install I changed the service name on $LS\_HOME/config/startup.options, which works fine. Additionally I would l… --- ## [Logstash-translate plugin](https://discuss.elastic.co/t/logstash-translate-plugin/73853)
**Author:** [@Raj\_Kumar](https://discuss.elastic.co/u/Raj_Kumar)\ **Replies:** 2\ **Last updated:** [February 23, 2017, 5:16pm UTC](https://discuss.elastic.co/t/logstash-translate-plugin/73853 "2017-02-23T17:16:50Z")
Hi All, Please let me know how to install the translate plugin in version 5.2.0 Elasticstack,i get this message. /usr/share/logstash# ./bin/logstash-plugin install logstash-filter-translate Validating logstash-filter… --- ## [Windows DNS incompatible character encodings](https://discuss.elastic.co/t/windows-dns-incompatible-character-encodings/48961)
**Author:** [@BongoEADGC6](https://discuss.elastic.co/u/BongoEADGC6)\ **Replies:** 14\ **Last updated:** [February 23, 2017, 2:30pm UTC](https://discuss.elastic.co/t/windows-dns-incompatible-character-encodings/48961 "2017-02-23T14:30:16Z")
I am having a constant issue while sending Windows DNS logs from filebeat to logstash. This is the config I use during debug: input { beats { port =\> XXXX } } filter { if \[type\] =="Win\_DNS" { gro… --- ## [Ingesting node](https://discuss.elastic.co/t/ingesting-node/76111)
**Author:** [@sharon.c](https://discuss.elastic.co/u/sharon.c)\ **Replies:** 2\ **Last updated:** [February 23, 2017, 1:58pm UTC](https://discuss.elastic.co/t/ingesting-node/76111 "2017-02-23T13:58:34Z")
I used to use Logstash to ingest data to Elasticsearch, now I plan to change it to logstash, but there is one functionality I have to rely on, I need to generate a hash to uniquely identify a document This is the logst… --- ## [Logstash conf file parser](https://discuss.elastic.co/t/logstash-conf-file-parser/76224)
**Author:** [@itzikHan](https://discuss.elastic.co/u/itzikHan)\ **Replies:** 0\ **Last updated:** [February 23, 2017, 1:29pm UTC](https://discuss.elastic.co/t/logstash-conf-file-parser/76224 "2017-02-23T13:29:04Z")
Is there any tool or something that allows me to get the logstash conf by JSON, Or some other format? (or any other .conf parsing related program?) I know it look a lot like JSON, But i was wondering if there is any ex… --- ## [Grok: identify start of multiline message inside a line](https://discuss.elastic.co/t/grok-identify-start-of-multiline-message-inside-a-line/76210)
**Author:** [@crebegea](https://discuss.elastic.co/u/crebegea)\ **Replies:** 3\ **Last updated:** [February 23, 2017, 12:44pm UTC](https://discuss.elastic.co/t/grok-identify-start-of-multiline-message-inside-a-line/76210 "2017-02-23T12:44:34Z")
Hi, I am having serious problems parsing some logs of ours. The Problem is that the messages, which are also multiline, begin in the middle of the line. There is a workable pattern for the beginning of the message but … --- ## [Logstash grok parsing not working](https://discuss.elastic.co/t/logstash-grok-parsing-not-working/76200)
**Author:** [@elasticheart](https://discuss.elastic.co/u/elasticheart)\ **Replies:** 1\ **Last updated:** [February 23, 2017, 12:17pm UTC](https://discuss.elastic.co/t/logstash-grok-parsing-not-working/76200 "2017-02-23T12:17:14Z")
Hi, I use ELK GA 5.0.0 and I read from Kafka topic. I have a log entry like below; 2017-02-22 10:23:53 - - 0.000 304 '425B39368AE7AF1B0D7230B6413ACC6B' GET /qwe/rty/ghj.js http-abc-1234-def-567 192.168.0.1 192.168.0.… --- ## [Extracting from Bind 9 log files](https://discuss.elastic.co/t/extracting-from-bind-9-log-files/75744)
**Author:** [@HeadScratcher](https://discuss.elastic.co/u/HeadScratcher)\ **Replies:** 7\ **Last updated:** [February 23, 2017, 11:35am UTC](https://discuss.elastic.co/t/extracting-from-bind-9-log-files/75744 "2017-02-23T11:35:24Z")
So I've been basically testing and experimenting with the whole ELK and i'm impressed with its versatility and already have a few things being displayed from various data sources which pleases my boss and my team quite n… --- ## [Unable to install logstash-output-mongodb on Ubuntu 16.04](https://discuss.elastic.co/t/unable-to-install-logstash-output-mongodb-on-ubuntu-16-04/75855)
**Author:** [@jstar](https://discuss.elastic.co/u/jstar)\ **Replies:** 2\ **Last updated:** [February 23, 2017, 11:11am UTC](https://discuss.elastic.co/t/unable-to-install-logstash-output-mongodb-on-ubuntu-16-04/75855 "2017-02-23T11:11:25Z")
Hi, I was able to install logstash-output-mongodb on ubuntu 14.04 but when I tried that on ubuntu 16.04, it fails with ERROR: Installation aborted, verification failed for logstash-output-mongodb. I'm using logstash 5.… --- ## [Failed to construct kafka producer](https://discuss.elastic.co/t/failed-to-construct-kafka-producer/76195)
**Author:** [@zpp](https://discuss.elastic.co/u/zpp)\ **Replies:** 0\ **Last updated:** [February 23, 2017, 10:44am UTC](https://discuss.elastic.co/t/failed-to-construct-kafka-producer/76195 "2017-02-23T10:44:27Z")
with Logstash 5.2, I'm testing sending data to kafka (using Kerberos). Below is the kakfa confiugration: kafka { bootstrap\_servers =\> "xxx" jaas\_path =\> "/usr/hdp/current/kafka-broker/config/kafka\_c… --- ## [Connect Logstash to remote FTP server](https://discuss.elastic.co/t/connect-logstash-to-remote-ftp-server/76189)
**Author:** [@Nejc\_Gutnik](https://discuss.elastic.co/u/Nejc_Gutnik)\ **Replies:** 1\ **Last updated:** [February 23, 2017, 10:27am UTC](https://discuss.elastic.co/t/connect-logstash-to-remote-ftp-server/76189 "2017-02-23T10:27:27Z")
Hi there, I'm trying to connect to password secured remote FTP server where I have a few log files. I want to get a steady stream of new events to Logstash. As far as I know, Logstash can't connect directly to FTP serv… --- ## [Grok Filter Parsing Failure](https://discuss.elastic.co/t/grok-filter-parsing-failure/76013)
**Author:** [@agrawalsaurabh](https://discuss.elastic.co/u/agrawalsaurabh)\ **Replies:** 5\ **Last updated:** [February 23, 2017, 10:23am UTC](https://discuss.elastic.co/t/grok-filter-parsing-failure/76013 "2017-02-23T10:23:50Z")
Hi, I was able to integrate Filebeat/logstash/Elasticsearch and kibana and able to parise logs using default APACHE LOG filter. Next I tried to parse specific information and push it to Elasticsearch. However, logstas… --- ## [Read a constantly overwritten file](https://discuss.elastic.co/t/read-a-constantly-overwritten-file/76177)
**Author:** [@gringo](https://discuss.elastic.co/u/gringo)\ **Replies:** 5\ **Last updated:** [February 23, 2017, 10:23am UTC](https://discuss.elastic.co/t/read-a-constantly-overwritten-file/76177 "2017-02-23T10:23:04Z")
I am currently reading a log file to parse it and send the parsed fields to elasticsearch. The problem that I am facing now is that new lines are constantly being appended to the log file. I have ran into situations wh… --- ## [Missing and disordered when using logstash elasticsearch input and file output](https://discuss.elastic.co/t/missing-and-disordered-when-using-logstash-elasticsearch-input-and-file-output/75012)
**Author:** [@zpp](https://discuss.elastic.co/u/zpp)\ **Replies:** 2\ **Last updated:** [February 23, 2017, 9:26am UTC](https://discuss.elastic.co/t/missing-and-disordered-when-using-logstash-elasticsearch-input-and-file-output/75012 "2017-02-23T09:26:50Z")
I'm trying to query elasticsearch with logstash elasticsearch input plugin and save the query result to files, messages in the file appears disordered, although I've put in a sort clause in the query string, and not all … --- ## [Logstash-plugin error: message: certificate verify failed](https://discuss.elastic.co/t/logstash-plugin-error-message-certificate-verify-failed/76175)
**Author:** [@gilisade](https://discuss.elastic.co/u/gilisade)\ **Replies:** 0\ **Last updated:** [February 23, 2017, 8:48am UTC](https://discuss.elastic.co/t/logstash-plugin-error-message-certificate-verify-failed/76175 "2017-02-23T08:48:47Z")
Hi trying to install perfmon input plugin on offline RHEL machine i installed it from an online machine , did pack then unpacked it at the offline machine trying to install it with bin/logstash-plugin install --no-verif… --- ## [Logstash Private IP Address - geoip](https://discuss.elastic.co/t/logstash-private-ip-address-geoip/75989)
**Author:** [@Bondo](https://discuss.elastic.co/u/Bondo)\ **Replies:** 8\ **Last updated:** [February 23, 2017, 8:26am UTC](https://discuss.elastic.co/t/logstash-private-ip-address-geoip/75989 "2017-02-23T08:26:24Z")
I can successfully get geo-location data from my IIS weblogs into Elastic if it's a public IP, but there are some private IP addresses that I want to explicitly set to a certain geo-locations and it's not working quite r… --- ## [Incremental Refresh](https://discuss.elastic.co/t/incremental-refresh/76102)
**Author:** [@tausif786](https://discuss.elastic.co/u/tausif786)\ **Replies:** 1\ **Last updated:** [February 23, 2017, 7:07am UTC](https://discuss.elastic.co/t/incremental-refresh/76102 "2017-02-23T07:07:26Z")
Hello Everyone, I am new to ELK Stack. I have data in postgresql which i need to do some visualizations in Kibana. Steps i have followed Creating a logstash config file which reads data from postgresql and pushes it… --- ## [Only import new or updated rows using JDBC](https://discuss.elastic.co/t/only-import-new-or-updated-rows-using-jdbc/76152)
**Author:** [@Reno](https://discuss.elastic.co/u/Reno)\ **Replies:** 1\ **Last updated:** [February 23, 2017, 6:52am UTC](https://discuss.elastic.co/t/only-import-new-or-updated-rows-using-jdbc/76152 "2017-02-23T06:52:10Z")
Right now I've got Logstash importing a miniature version of my MSSQL database, using the JDBC plugin. I've got each JDBC input scheduled to run every minute to update Elasticsearch. To update, I'm currently just re-impo… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1029) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1031)