# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1019 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1020 --- ## [Issue in Sending Sensu Events to Logstash](https://discuss.elastic.co/t/issue-in-sending-sensu-events-to-logstash/78534)
**Author:** [@keeshia](https://discuss.elastic.co/u/keeshia)\ **Replies:** 0\ **Last updated:** [March 14, 2017, 2:10pm UTC](https://discuss.elastic.co/t/issue-in-sending-sensu-events-to-logstash/78534 "2017-03-14T14:10:25Z")
Hi, We are trying to send Sensu events to Logstash using tcp handler but it doesn't seem to be working and it won't display the results to Kibana. We can't troubleshoot this properly because the logs in /var/log/logsta… --- ## [Need advices to ingest this type of data](https://discuss.elastic.co/t/need-advices-to-ingest-this-type-of-data/76636)
**Author:** [@florentmair](https://discuss.elastic.co/u/florentmair)\ **Replies:** 1\ **Last updated:** [March 14, 2017, 2:08pm UTC](https://discuss.elastic.co/t/need-advices-to-ingest-this-type-of-data/76636 "2017-03-14T14:08:22Z")
Hi I have daily results from an audit and i am not sure how i should manage the data. All i want is to visualise the evolution of the number of documents and keep the structure i want to be able to visualise the numb… --- ## [S3 plugin - unable to change file extension of data in bucket](https://discuss.elastic.co/t/s3-plugin-unable-to-change-file-extension-of-data-in-bucket/31264)
**Author:** [@trekr5](https://discuss.elastic.co/u/trekr5)\ **Replies:** 1\ **Last updated:** [March 14, 2017, 1:58pm UTC](https://discuss.elastic.co/t/s3-plugin-unable-to-change-file-extension-of-data-in-bucket/31264 "2017-03-14T13:58:25Z")
Hello, I'm currently using the s3 plugin to upload logs from logstash into an s3 bucket. I'd like the uploaded files to be of the type json with a file extension of .json My problem is that the uploaded files keep ap… --- ## [Logstash pipeline doesn't stop when processing is done (Xpack)](https://discuss.elastic.co/t/logstash-pipeline-doesnt-stop-when-processing-is-done-xpack/78532)
**Author:** [@Clem\_Bac](https://discuss.elastic.co/u/Clem_Bac)\ **Replies:** 0\ **Last updated:** [March 14, 2017, 1:57pm UTC](https://discuss.elastic.co/t/logstash-pipeline-doesnt-stop-when-processing-is-done-xpack/78532 "2017-03-14T13:57:36Z")
Hi everyone, I have the following conf. : an elasticsearch 5.2.2 cluster (2 data node, 1 master) a logstash pipeline 5.2.2 with a JDBC connection to an oracle db (ojdbc6.jar) xpack 5.2.2 Since I've installed xpac… --- ## [Jdbc plugin local devepment problem?](https://discuss.elastic.co/t/jdbc-plugin-local-devepment-problem/78469)
**Author:** [@copywrite](https://discuss.elastic.co/u/copywrite)\ **Replies:** 1\ **Last updated:** [March 14, 2017, 1:45pm UTC](https://discuss.elastic.co/t/jdbc-plugin-local-devepment-problem/78469 "2017-03-14T13:45:04Z")
hi :smile: environment: macos yosemite. use the following steps: rbenv install jruby-9.1.6.0 cd logstash-git-path jruby -S gem install bundler jruby -S bundle install jruby -S bundle exec rspec jruby -S bundle e… --- ## [Logstash Logs Not Updating](https://discuss.elastic.co/t/logstash-logs-not-updating/78518)
**Author:** [@peter.alfafara](https://discuss.elastic.co/u/peter.alfafara)\ **Replies:** 0\ **Last updated:** [March 14, 2017, 1:00pm UTC](https://discuss.elastic.co/t/logstash-logs-not-updating/78518 "2017-03-14T13:00:28Z")
Hi, We just installed logstash for the second time and we are unable to read the logstash logs for any errors/misconfigurations because the logs under /var/logs/logstash/logstash-plain.log hasn't been updating just afte… --- ## [Elasticsearch to Elasticsearch duplicating messages](https://discuss.elastic.co/t/elasticsearch-to-elasticsearch-duplicating-messages/78511)
**Author:** [@JeremyinNC](https://discuss.elastic.co/u/JeremyinNC)\ **Replies:** 0\ **Last updated:** [March 14, 2017, 12:18pm UTC](https://discuss.elastic.co/t/elasticsearch-to-elasticsearch-duplicating-messages/78511 "2017-03-14T12:18:44Z")
Using ES 2.4.4 and Logstash 2.4.1 with the following config logstash appears to "loop" though the query results, ending up with the same documents being written over and over. Can someone help me figure out what I'm doi… --- ## [Reading a properties file using logstash](https://discuss.elastic.co/t/reading-a-properties-file-using-logstash/78440)
**Author:** [@kiranilla](https://discuss.elastic.co/u/kiranilla)\ **Replies:** 2\ **Last updated:** [March 14, 2017, 11:19am UTC](https://discuss.elastic.co/t/reading-a-properties-file-using-logstash/78440 "2017-03-14T11:19:12Z")
I am trying to to read a properties file using log stash, but getting some error as shown below Could you please help me in resolving this issue. As i want to read a properties file (using ruby code as shown in below s… --- ## [Logstash jdbc type conflicts with statement field](https://discuss.elastic.co/t/logstash-jdbc-type-conflicts-with-statement-field/78121)
**Author:** [@copywrite](https://discuss.elastic.co/u/copywrite)\ **Replies:** 2\ **Last updated:** [March 14, 2017, 10:20am UTC](https://discuss.elastic.co/t/logstash-jdbc-type-conflicts-with-statement-field/78121 "2017-03-14T10:20:18Z")
i have a conf as follows input { jdbc { statement =\> 'select type from item' type =\> "item" } } output { elasticsearch { document\_type =\> '%{type}' } } the output type value … --- ## [Logstash 5.x fail to install logstash-output-zabbix](https://discuss.elastic.co/t/logstash-5-x-fail-to-install-logstash-output-zabbix/78080)
**Author:** [@focus\_zheng](https://discuss.elastic.co/u/focus_zheng)\ **Replies:** 6\ **Last updated:** [March 14, 2017, 9:52am UTC](https://discuss.elastic.co/t/logstash-5-x-fail-to-install-logstash-output-zabbix/78080 "2017-03-14T09:52:09Z")
Hi, Yesterday I had a problem when I tried to install logstash-output-zabbix on my logstash 5.2. I used the recommended command "logstash-plugin install logstash-output-zabbix".But it failed with the error messgae "… --- ## [Converting a hash of key and values to a array of objects](https://discuss.elastic.co/t/converting-a-hash-of-key-and-values-to-a-array-of-objects/76856)
**Author:** [@Saravanan\_Sathyamoor](https://discuss.elastic.co/u/Saravanan_Sathyamoor)\ **Replies:** 1\ **Last updated:** [March 14, 2017, 8:29am UTC](https://discuss.elastic.co/t/converting-a-hash-of-key-and-values-to-a-array-of-objects/76856 "2017-03-14T08:29:12Z")
I have a json like { "230":{"startDatetime":"2017-07-21T07:00:00","endDatetime":"2017-08-07T06:59:59"}, "232":{"startDatetime":"2017-07-21T07:00:00","endDatetime":"2017-08-07T06:59:59"} } and i need to convert to \[ {"l… --- ## [Reindex to change type with wildcard indexes](https://discuss.elastic.co/t/reindex-to-change-type-with-wildcard-indexes/78374)
**Author:** [@p\_kc](https://discuss.elastic.co/u/p_kc)\ **Replies:** 2\ **Last updated:** [March 14, 2017, 7:46am UTC](https://discuss.elastic.co/t/reindex-to-change-type-with-wildcard-indexes/78374 "2017-03-14T07:46:00Z")
Hello, When I started to send data from logstash to elasticsearch the format was set to string for several inputs. I'd like to change the format from string to integer, but it looks like it requires reindexing the wh… --- ## [Syslog data, not getting the "fields" I want](https://discuss.elastic.co/t/syslog-data-not-getting-the-fields-i-want/78152)
**Author:** [@perch](https://discuss.elastic.co/u/perch)\ **Replies:** 1\ **Last updated:** [March 14, 2017, 6:45am UTC](https://discuss.elastic.co/t/syslog-data-not-getting-the-fields-i-want/78152 "2017-03-14T06:45:47Z")
Getting Syslog data to my ELK stack. But the info I am interested in is in the Message field and not Analyzed, it seems?! The syslog looks like this: { "\_index": "logstash-2017.03.06", "\_type": "logs", "\_id": … --- ## [Problem installing plugin](https://discuss.elastic.co/t/problem-installing-plugin/78373)
**Author:** [@Lazypete](https://discuss.elastic.co/u/Lazypete)\ **Replies:** 2\ **Last updated:** [March 13, 2017, 6:09pm UTC](https://discuss.elastic.co/t/problem-installing-plugin/78373 "2017-03-13T18:09:32Z")
I just installed logstash from the latest tar.gz file 5.2.2 is the version number. I have searched the forum and seen this eror come up a few time but no solution were proposed. So I installed logstash and I need 2 p… --- ## [Issue while trying to put Twitter data into a file](https://discuss.elastic.co/t/issue-while-trying-to-put-twitter-data-into-a-file/78378)
**Author:** [@vincent2mots](https://discuss.elastic.co/u/vincent2mots)\ **Replies:** 1\ **Last updated:** [March 13, 2017, 4:29pm UTC](https://discuss.elastic.co/t/issue-while-trying-to-put-twitter-data-into-a-file/78378 "2017-03-13T16:29:28Z")
Hi there, Here's my issue : I try to put Twitter data into a file (to begin) using Logstash 5.2.2. Here is my config file : input { twitter { consumer\_key =\> "XXX" consumer\_secret =\> "XXX" oauth\_token =\> "XXX" o… --- ## [Telnet logstash not working](https://discuss.elastic.co/t/telnet-logstash-not-working/78281)
**Author:** [@rrs](https://discuss.elastic.co/u/rrs)\ **Replies:** 4\ **Last updated:** [March 13, 2017, 2:44pm UTC](https://discuss.elastic.co/t/telnet-logstash-not-working/78281 "2017-03-13T14:44:13Z")
Hello, I installed logstash as per documentation and service started successfully. But, when we trying telnet localhost 5044, it is not working and the netstat also showing nothing. Can you please suggest me how to fi… --- ## [Help me to convert hexa in decimal value](https://discuss.elastic.co/t/help-me-to-convert-hexa-in-decimal-value/77625)
**Author:** [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\ **Replies:** 2\ **Last updated:** [March 13, 2017, 2:31pm UTC](https://discuss.elastic.co/t/help-me-to-convert-hexa-in-decimal-value/77625 "2017-03-13T14:31:49Z")
Hello, i search to convert hexa value in decimal value for performing. I have found a ruby code in disucss but it's seems to be fail. I use it to upload logs : mutate { convert =\> { "hexa\_value" =\> "integer" } … --- ## [How to make make an if condition with a regular expression pattern](https://discuss.elastic.co/t/how-to-make-make-an-if-condition-with-a-regular-expression-pattern/78338)
**Author:** [@DannyM](https://discuss.elastic.co/u/DannyM)\ **Replies:** 1\ **Last updated:** [March 13, 2017, 2:17pm UTC](https://discuss.elastic.co/t/how-to-make-make-an-if-condition-with-a-regular-expression-pattern/78338 "2017-03-13T14:17:45Z")
Hi, i'm new to ELK and trying to filter some spam lines from being added to the elasticsearch server. to do this i made the following logstash filter: ## var\_log\_messages filter filter { if \[type\] == "var\_lo… --- ## [S3 output problem: server side encryption](https://discuss.elastic.co/t/s3-output-problem-server-side-encryption/78348)
**Author:** [@Vittorio](https://discuss.elastic.co/u/Vittorio)\ **Replies:** 0\ **Last updated:** [March 13, 2017, 1:47pm UTC](https://discuss.elastic.co/t/s3-output-problem-server-side-encryption/78348 "2017-03-13T13:47:52Z")
Hi all, I'm sure my question was already answered but I didn't find anything here ... I'm using s3 output and what I do not understand is the encrypting part... I specify server\_side\_encryption = true, server\_side… --- ## [Logstash nested conditional statement one with null check](https://discuss.elastic.co/t/logstash-nested-conditional-statement-one-with-null-check/77946)
**Author:** [@kiranilla](https://discuss.elastic.co/u/kiranilla)\ **Replies:** 3\ **Last updated:** [March 13, 2017, 12:39pm UTC](https://discuss.elastic.co/t/logstash-nested-conditional-statement-one-with-null-check/77946 "2017-03-13T12:39:04Z")
As i am trying to write a script that as to populate the "Category" based on "Knowledge Title" contains some text called "AGSS", but here the condition is that i need to check the "Category" is null then only i have to … --- ## [Logstash and Bro IDS - Hi-Volume Logs are not "picked up"](https://discuss.elastic.co/t/logstash-and-bro-ids-hi-volume-logs-are-not-picked-up/78278)
**Author:** [@sherifeldeeb](https://discuss.elastic.co/u/sherifeldeeb)\ **Replies:** 2\ **Last updated:** [March 13, 2017, 11:43am UTC](https://discuss.elastic.co/t/logstash-and-bro-ids-hi-volume-logs-are-not-picked-up/78278 "2017-03-13T11:43:30Z")
Bro IDS "bro.org" creates multiple log files (http, dns, conn ...etc.) I created one .conf file for each log type. Problem is: logstash doesn't consistently "pick up" the log files that has high volume (like http.log… --- ## [DateTime Function in Ruby logstash filter](https://discuss.elastic.co/t/datetime-function-in-ruby-logstash-filter/78126)
**Author:** [@Gurveer\_Singh](https://discuss.elastic.co/u/Gurveer_Singh)\ **Replies:** 1\ **Last updated:** [March 13, 2017, 8:42am UTC](https://discuss.elastic.co/t/datetime-function-in-ruby-logstash-filter/78126 "2017-03-13T08:42:49Z")
Hi, I am trying to add a new field which will have value, week of year. Following code is working fine. ruby { code =\> ' require "date" d = DateTime.new(2017,02,02) event\["Week\_of\_Year"\] = d.cweek … --- ## [Simple problem with grok](https://discuss.elastic.co/t/simple-problem-with-grok/78117)
**Author:** [@parrot](https://discuss.elastic.co/u/parrot)\ **Replies:** 1\ **Last updated:** [March 13, 2017, 8:39am UTC](https://discuss.elastic.co/t/simple-problem-with-grok/78117 "2017-03-13T08:39:36Z")
Hi, this is my conf file: input { stdin { } } filter { grok { patterns\_dir =\> \["./patterns"\] match =\> { "message" =\> "%{NOTSPACE:timestamp}%{SPACE}%{WORD:vendor}%{NOTSPACE}%{SPACE}%{TIMESTAMP\_ISO8601:localE… --- ## [Kafka Logstash Topics config best practice](https://discuss.elastic.co/t/kafka-logstash-topics-config-best-practice/78304)
**Author:** [@Ram\_Vittal](https://discuss.elastic.co/u/Ram_Vittal)\ **Replies:** 0\ **Last updated:** [March 13, 2017, 3:31am UTC](https://discuss.elastic.co/t/kafka-logstash-topics-config-best-practice/78304 "2017-03-13T03:31:57Z")
We are using latest Kafka and Logstash versions for ingesting several business apps log data into ELK. We have a standardized logging structure for business apps to log data into Kafka topic and able to ingest into ELK … --- ## [Ingesting & indexing text files while keeping formatting](https://discuss.elastic.co/t/ingesting-indexing-text-files-while-keeping-formatting/78279)
**Author:** [@Bob\_Metelsky](https://discuss.elastic.co/u/Bob_Metelsky)\ **Replies:** 15\ **Last updated:** [March 13, 2017, 2:01am UTC](https://discuss.elastic.co/t/ingesting-indexing-text-files-while-keeping-formatting/78279 "2017-03-13T02:01:36Z")
Greetings - I have a use case where We have plain text files in a very specific formatting These files are in a directory I want to do a search in ES or (another front end) for a specific id in those files I want to r… --- ## [Logstash \<- Kafka over WAN](https://discuss.elastic.co/t/logstash-kafka-over-wan/78251)
**Author:** [@shaharmor](https://discuss.elastic.co/u/shaharmor)\ **Replies:** 3\ **Last updated:** [March 12, 2017, 10:39pm UTC](https://discuss.elastic.co/t/logstash-kafka-over-wan/78251 "2017-03-12T22:39:15Z")
Hi, I'm looking for the best configuration for a high load system where Logstash is in the Azure cloud, and Kafka is in AWS. At the moment I have 3 Kafka servers with 20 partitions for the topic. I've created an au… --- ## [Gelf plugin logstash](https://discuss.elastic.co/t/gelf-plugin-logstash/77782)
**Author:** [@Shrawan\_Bhagwat](https://discuss.elastic.co/u/Shrawan_Bhagwat)\ **Replies:** 1\ **Last updated:** [March 12, 2017, 7:37pm UTC](https://discuss.elastic.co/t/gelf-plugin-logstash/77782 "2017-03-12T19:37:22Z")
Hi, i have installed logstash-5.2.1. I have found gelf output plugin missing in this logstash version. is there any way to output to graylog2 from logstash-5.2.1? --- ## [Logstash losing the logs](https://discuss.elastic.co/t/logstash-losing-the-logs/62629)
**Author:** [@kumar21000](https://discuss.elastic.co/u/kumar21000)\ **Replies:** 3\ **Last updated:** [March 12, 2017, 7:18pm UTC](https://discuss.elastic.co/t/logstash-losing-the-logs/62629 "2017-03-12T19:18:48Z")
Hi, I think my logstash eating the logs, I am not sure, where I am making the mistake. I am using logstash 2.4, elasticsearch 2.4.1, kibana 4.6.1. I am not an expert in ELK, but started to explore it. Many times I notice… --- ## [Merge JSON events](https://discuss.elastic.co/t/merge-json-events/78282)
**Author:** [@Dan\_Markhasin](https://discuss.elastic.co/u/Dan_Markhasin)\ **Replies:** 0\ **Last updated:** [March 12, 2017, 3:21pm UTC](https://discuss.elastic.co/t/merge-json-events/78282 "2017-03-12T15:21:39Z")
I'm looking for some way to merge JSON events that have the same ID field. For example, given the following incoming events: {"id":"5f3f1cd7-517d-4d33-b4d5-5a7f6bc7630e", "value1":"somevalue"} {"id":"5f3f1cd7-517d-4d33… --- ## [N00b: E on bro problem](https://discuss.elastic.co/t/n00b-e-on-bro-problem/78262)
**Author:** [@randy](https://discuss.elastic.co/u/randy)\ **Replies:** 4\ **Last updated:** [March 12, 2017, 9:05am UTC](https://discuss.elastic.co/t/n00b-e-on-bro-problem/78262 "2017-03-12T09:05:15Z")
ubuntu 16.04, following recipe in # /usr/share/logstash/bin/logstash -e output { stdout {} } WARNING: Could not find logstash.yml which is typically located in $LS\_HOME/config or /etc/logstash. You can specify the… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1018) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1020)