# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=859 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 860 --- ## [Logstash to cloudwatch](https://discuss.elastic.co/t/logstash-to-cloudwatch/110592)
**Author:** [@TTB](https://discuss.elastic.co/u/TTB)\ **Replies:** 1\ **Last updated:** [December 11, 2017, 5:36pm UTC](https://discuss.elastic.co/t/logstash-to-cloudwatch/110592 "2017-12-11T17:36:42Z")
Hi, I am trying to push metrics from Logstash to Cloudwatch, I am using the cloudwatch output plugin. Below is the config. \` input { beats { port =\> 5044 } } filter { grok{ match=\> {"message"=\>"%{IP:clientip} … --- ## [Logstash Keeps Restarting](https://discuss.elastic.co/t/logstash-keeps-restarting/111122)
**Author:** [@MultiplierMultiplier](https://discuss.elastic.co/u/MultiplierMultiplier)\ **Replies:** 9\ **Last updated:** [December 11, 2017, 5:07pm UTC](https://discuss.elastic.co/t/logstash-keeps-restarting/111122 "2017-12-11T17:07:01Z")
The only setting I have changed in the Logstash .yml is: log.level: debug. Logstash keeps restarting and there is absolutely nothing in the log file (/var/log/logstash/logstash-plaing.log) The only info I can find is … --- ## [Configure logstash to receive different csv files from filebeat and send it to different index](https://discuss.elastic.co/t/configure-logstash-to-receive-different-csv-files-from-filebeat-and-send-it-to-different-index/110224)
**Author:** [@ArunPhilip](https://discuss.elastic.co/u/ArunPhilip)\ **Replies:** 2\ **Last updated:** [December 11, 2017, 5:14pm UTC](https://discuss.elastic.co/t/configure-logstash-to-receive-different-csv-files-from-filebeat-and-send-it-to-different-index/110224 "2017-12-11T17:14:19Z")
i am trying to configure the logstash to receive different csv files (/var/ticket/adr/adr\_20171204101010.csv and /var/ticket/idr/idr\_20171204101012.csv) from another server using filebeat . how to configure the logsta… --- ## [Logstash Aggregate plugin don´t aggregate](https://discuss.elastic.co/t/logstash-aggregate-plugin-don-t-aggregate/110338)
**Author:** [@gleberrl](https://discuss.elastic.co/u/gleberrl)\ **Replies:** 2\ **Last updated:** [December 11, 2017, 3:12pm UTC](https://discuss.elastic.co/t/logstash-aggregate-plugin-don-t-aggregate/110338 "2017-12-11T15:12:17Z")
I have this logs and need aggregate to selected fields: \[image\] I don´t have error log, but I not see aggregated log´s My logstash aggregate config: if \[type\] == "Logs-Clientes" { aggr… --- ## [Update xml file content to prevent xml parse error](https://discuss.elastic.co/t/update-xml-file-content-to-prevent-xml-parse-error/111050)
**Author:** [@iam\_thadiyan](https://discuss.elastic.co/u/iam_thadiyan)\ **Replies:** 6\ **Last updated:** [December 11, 2017, 3:12pm UTC](https://discuss.elastic.co/t/update-xml-file-content-to-prevent-xml-parse-error/111050 "2017-12-11T15:12:10Z")
I have a number of (~10k) XML files that have a missing closing root tag. When I load this file to logstash with xml filter I get parse error. Is there a way to make logstash add a closing tag in filter before parsing t… --- ## [Remove or subst for passwords in commandline logging?](https://discuss.elastic.co/t/remove-or-subst-for-passwords-in-commandline-logging/111126)
**Author:** [@mathurin68](https://discuss.elastic.co/u/mathurin68)\ **Replies:** 0\ **Last updated:** [December 11, 2017, 3:11pm UTC](https://discuss.elastic.co/t/remove-or-subst-for-passwords-in-commandline-logging/111126 "2017-12-11T15:11:40Z")
Logging windows command line events; cmd and powershell. And while it's policy that no one should be entering passwords directly on the command line... well, you know. Curious if anyone has come up with a clever way to… --- ## [Adding S3 Bucket Name in Output Logs](https://discuss.elastic.co/t/adding-s3-bucket-name-in-output-logs/110385)
**Author:** [@jmukhtar](https://discuss.elastic.co/u/jmukhtar)\ **Replies:** 3\ **Last updated:** [December 11, 2017, 3:07pm UTC](https://discuss.elastic.co/t/adding-s3-bucket-name-in-output-logs/110385 "2017-12-11T15:07:27Z")
I am a newbie to Logstash and ES in general, and struggling to add S3 bucket name to logstash output. I have a setup where we have multiple s3 buckets with RDS instance logs (1 bucket per instance) and we want to harves… --- ## [Getting logs in other index](https://discuss.elastic.co/t/getting-logs-in-other-index/110016)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 9\ **Last updated:** [December 11, 2017, 3:03pm UTC](https://discuss.elastic.co/t/getting-logs-in-other-index/110016 "2017-12-11T15:03:49Z")
Hi Guys, I am kinda facing weird issue! I have configured two config file and have created two different indices which are indexing my bind and Apache logs. However my bind logs are getting successfully parsed and even… --- ## [JSON logstash plugin - loading in Elastic](https://discuss.elastic.co/t/json-logstash-plugin-loading-in-elastic/109606)
**Author:** [@RickT](https://discuss.elastic.co/u/RickT)\ **Replies:** 1\ **Last updated:** [December 11, 2017, 2:57pm UTC](https://discuss.elastic.co/t/json-logstash-plugin-loading-in-elastic/109606 "2017-12-11T14:57:08Z")
Hello ! the sun is shinning today, but.... I have a simple question about JSON logstash plugin. the log i want to parse is like this : {"corId":"zzz01","datas":\[{"Key":"TstName","Value":"GlobZAZ"},{"Key":"ActName","V… --- ## [Issue about logstash input snmptrap plugin](https://discuss.elastic.co/t/issue-about-logstash-input-snmptrap-plugin/110870)
**Author:** [@dangal](https://discuss.elastic.co/u/dangal)\ **Replies:** 0\ **Last updated:** [December 8, 2017, 3:48pm UTC](https://discuss.elastic.co/t/issue-about-logstash-input-snmptrap-plugin/110870 "2017-12-08T15:48:18Z")
Hi All.... Here i have an issue with an option yamlmibdir in input snmptrap plugin input { snmptrap { type =\> "snmptrap" community =\> "public" port =\> 162 yamlmibdir =\> "/opt/logstash-5.4.0/vendor/bundle/jruby/1.9/gems… --- ## [Droping entire document](https://discuss.elastic.co/t/droping-entire-document/111097)
**Author:** [@Raj\_Kumar](https://discuss.elastic.co/u/Raj_Kumar)\ **Replies:** 7\ **Last updated:** [December 11, 2017, 2:34pm UTC](https://discuss.elastic.co/t/droping-entire-document/111097 "2017-12-11T14:34:39Z")
Hi All, Iam monitoring the windows events using Beats , but i could find lot of events generated every second and its kind of overloaded I want to drop this event id event\_id : 5,516 Eventid is the field name and … --- ## [Logstash 6.0.1 plugin install error](https://discuss.elastic.co/t/logstash-6-0-1-plugin-install-error/111116)
**Author:** [@Zen\_Spartan1](https://discuss.elastic.co/u/Zen_Spartan1)\ **Replies:** 0\ **Last updated:** [December 11, 2017, 2:33pm UTC](https://discuss.elastic.co/t/logstash-6-0-1-plugin-install-error/111116 "2017-12-11T14:33:41Z")
Using ubuntu 16.04.3 with open jdk , trying to install the oui plugin but keep getting this error, any ideas? cheers Zen Validating logstash-filter-oui Installing logstash-filter-oui Plugin version conflict, abortin… --- ## [Can some one please tell me what this error means?](https://discuss.elastic.co/t/can-some-one-please-tell-me-what-this-error-means/111107)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 1\ **Last updated:** [December 11, 2017, 2:27pm UTC](https://discuss.elastic.co/t/can-some-one-please-tell-me-what-this-error-means/111107 "2017-12-11T14:27:14Z")
What could be wrong with my config. I am getting below error message in logstash.log file \[2017-12-11T19:12:10,130\]\[ERROR\]\[logstash.filters.ruby \] Ruby exception occurred: Direct event field references (i.e. event\['f… --- ## [Logstash filter plugin elasticsearch problem](https://discuss.elastic.co/t/logstash-filter-plugin-elasticsearch-problem/111106)
**Author:** [@rromanovic](https://discuss.elastic.co/u/rromanovic)\ **Replies:** 0\ **Last updated:** [December 11, 2017, 1:51pm UTC](https://discuss.elastic.co/t/logstash-filter-plugin-elasticsearch-problem/111106 "2017-12-11T13:51:20Z")
Hi, my configuration is as follows: input { beats { port =\> "5043" } } filter { if \[message\] =~ /IDRequestType \[Varchar2 \\d+\] = '(\[A-Z\_\])'/ { grok { match =\> \["message", "(?\\\S)\\s(?\\[0-9a-f\]{3… --- ## [NameError: cannot load Java class org.apache.http.client.methods.HttpRequestBase](https://discuss.elastic.co/t/nameerror-cannot-load-java-class-org-apache-http-client-methods-httprequestbase/111108)
**Author:** [@picj86](https://discuss.elastic.co/u/picj86)\ **Replies:** 0\ **Last updated:** [December 11, 2017, 1:59pm UTC](https://discuss.elastic.co/t/nameerror-cannot-load-java-class-org-apache-http-client-methods-httprequestbase/111108 "2017-12-11T13:59:12Z")
I have installed logstash-6.0.1 but while starting it from bin directory I am getting the below exception: D:\\SVS\\logstash-6.0.1\\bin\>logstash -f logstash-simple.conf NameError: cannot load Java class org.apache.http.cli… --- ## [@metadata get lost in logstash redis output/input](https://discuss.elastic.co/t/metadata-get-lost-in-logstash-redis-output-input/111080)
**Author:** [@fabian](https://discuss.elastic.co/u/fabian)\ **Replies:** 3\ **Last updated:** [December 11, 2017, 1:21pm UTC](https://discuss.elastic.co/t/metadata-get-lost-in-logstash-redis-output-input/111080 "2017-12-11T13:21:57Z")
Hey there, I'm trying to give the filebeat modules a try but stuck in process the @metadata field. My setup looks like filebeat \> logstash forwarder \> redis \> logstash processor \> elasticsearch Config logstash forwar… --- ## [How to convert a field boolean to integer in logstash](https://discuss.elastic.co/t/how-to-convert-a-field-boolean-to-integer-in-logstash/111051)
**Author:** [@saafianihan](https://discuss.elastic.co/u/saafianihan)\ **Replies:** 1\ **Last updated:** [December 11, 2017, 11:11am UTC](https://discuss.elastic.co/t/how-to-convert-a-field-boolean-to-integer-in-logstash/111051 "2017-12-11T11:11:42Z")
Hi, I have one boolean column with 0 and 1. I have to add those values to calculate average . so i need to convert boolean value into integer in logstash. I tried : filter { mutate { convert =\> { "is\_onsite" =\> "inte… --- ## [Duplicated log messages](https://discuss.elastic.co/t/duplicated-log-messages/111073)
**Author:** [@Nandor\_Soma\_Abonyi](https://discuss.elastic.co/u/Nandor_Soma_Abonyi)\ **Replies:** 0\ **Last updated:** [December 11, 2017, 11:06am UTC](https://discuss.elastic.co/t/duplicated-log-messages/111073 "2017-12-11T11:06:24Z")
Hello! I've recently found that there are duplicate messages in elasticsearch. Sometimes the same message duplicating 20 times but sometimes only 2 times. Of course in the original log file it only appears once. I saw … --- ## [S3 plugin file collision](https://discuss.elastic.co/t/s3-plugin-file-collision/111066)
**Author:** [@russellp](https://discuss.elastic.co/u/russellp)\ **Replies:** 0\ **Last updated:** [December 11, 2017, 10:36am UTC](https://discuss.elastic.co/t/s3-plugin-file-collision/111066 "2017-12-11T10:36:15Z")
Having a small issue with pipelines on version 6.0 of logstash. I have a number of logs that I am reshaping and placing in an s3 bucket using the the following config: input { file { path =\> "/opt/folder/lib/billing/… --- ## [JDBC Input Plugin with MySQL show commands](https://discuss.elastic.co/t/jdbc-input-plugin-with-mysql-show-commands/111027)
**Author:** [@tmoeller](https://discuss.elastic.co/u/tmoeller)\ **Replies:** 0\ **Last updated:** [December 11, 2017, 7:35am UTC](https://discuss.elastic.co/t/jdbc-input-plugin-with-mysql-show-commands/111027 "2017-12-11T07:35:03Z")
Hi, i am trying to to query a MySQL 5.7.19 with the "show variables" or the "show slave status" command but always receive a syntax error. All other select queries working fine. When using the jdbc plugin there seem to… --- ## [Google BigQuery output plugin](https://discuss.elastic.co/t/google-bigquery-output-plugin/111016)
**Author:** [@Vasily\_Bronsky](https://discuss.elastic.co/u/Vasily_Bronsky)\ **Replies:** 0\ **Last updated:** [December 11, 2017, 6:49am UTC](https://discuss.elastic.co/t/google-bigquery-output-plugin/111016 "2017-12-11T06:49:51Z")
Hello! I have logs with this structure: { "syslog\_program" : bla bla bla, "message" : \<14\>Dec 8 18:45:50 bla bla bla , "type" : syslog, "syslog\_message" : bla bla bla "@timestamp" : 2017-12-08T17:45:50.000Z, "sys… --- ## [Sphinx sql\_attr\_multi like logstash](https://discuss.elastic.co/t/sphinx-sql-attr-multi-like-logstash/110699)
**Author:** [@bariseser](https://discuss.elastic.co/u/bariseser)\ **Replies:** 1\ **Last updated:** [December 11, 2017, 6:40am UTC](https://discuss.elastic.co/t/sphinx-sql-attr-multi-like-logstash/110699 "2017-12-11T06:40:52Z")
Hi Everybody, Logstash is pretty new too. Now we are using sphinx and the sql\_attr\_multi property is available in our indexes. Do you have a similar feature in LogStash? --- ## [Migrating Multiple table from SQL Server to Elasticsearch](https://discuss.elastic.co/t/migrating-multiple-table-from-sql-server-to-elasticsearch/110919)
**Author:** [@Anshuman\_Kirty](https://discuss.elastic.co/u/Anshuman_Kirty)\ **Replies:** 1\ **Last updated:** [December 11, 2017, 6:38am UTC](https://discuss.elastic.co/t/migrating-multiple-table-from-sql-server-to-elasticsearch/110919 "2017-12-11T06:38:33Z")
I am trying to migrate all the Tables from the database. The output received gives only the 1st Table data values in the elasticsearch. Can anyone help me. Thanks in Advance. logstash.conf input { jdbc { jdbc\_driver\_… --- ## [ELK Indexes](https://discuss.elastic.co/t/elk-indexes/110967)
**Author:** [@poojajoshi](https://discuss.elastic.co/u/poojajoshi)\ **Replies:** 4\ **Last updated:** [December 11, 2017, 6:32am UTC](https://discuss.elastic.co/t/elk-indexes/110967 "2017-12-11T06:32:50Z")
Hello All, I am trying to log multiple log files in a single index. When i run it for the first time it works. If i try to log more log files to the same index it doesn't work even after deleting the .sincedb files. P… --- ## [Grok failure even after the fields were filtered out successfully](https://discuss.elastic.co/t/grok-failure-even-after-the-fields-were-filtered-out-successfully/110688)
**Author:** [@George\_Cherian](https://discuss.elastic.co/u/George_Cherian)\ **Replies:** 4\ **Last updated:** [December 11, 2017, 6:10am UTC](https://discuss.elastic.co/t/grok-failure-even-after-the-fields-were-filtered-out-successfully/110688 "2017-12-11T06:10:34Z")
Hi Everyone, I am encountering the following problem: I have been filtering out the "Via\_IP1" field from "Via1" which matches on both grok debugger and grok constructor.But when running on logstash ,i could observe gr… --- ## [Conditional Input based on Regex](https://discuss.elastic.co/t/conditional-input-based-on-regex/110834)
**Author:** [@scch](https://discuss.elastic.co/u/scch)\ **Replies:** 4\ **Last updated:** [December 11, 2017, 5:12am UTC](https://discuss.elastic.co/t/conditional-input-based-on-regex/110834 "2017-12-11T05:12:10Z")
Hi, can we do conditional input to a "field" based on regex. I am able to achieve same in painless scripting but want to achieve it in Logstash. below is example from painless. if (doc\['Mach.keyword'\].value =~ /^T\\d{4}… --- ## [Elastic filter plugin is not working](https://discuss.elastic.co/t/elastic-filter-plugin-is-not-working/110491)
**Author:** [@Sai\_Jana\_Selvakumar](https://discuss.elastic.co/u/Sai_Jana_Selvakumar)\ **Replies:** 1\ **Last updated:** [December 10, 2017, 9:06pm UTC](https://discuss.elastic.co/t/elastic-filter-plugin-is-not-working/110491 "2017-12-10T21:06:33Z")
Config file: input { http { host =\> "host" port =\> " port" } } filter { json { source =\> "message" } ruby { code =\>" hash= {'id' =\> 'id' }" } elasticsearch { hosts =\> \["host"\] index =\> "index\_name" query… --- ## [Date @timestamp](https://discuss.elastic.co/t/date-timestamp/110948)
**Author:** [@haylesr](https://discuss.elastic.co/u/haylesr)\ **Replies:** 1\ **Last updated:** [December 10, 2017, 9:05pm UTC](https://discuss.elastic.co/t/date-timestamp/110948 "2017-12-10T21:05:00Z")
I am new to ELK and am playing around with some basic data sources to learn how to set things up properly. The current CSV file that I'm parsing has two date fields in it: Modified Date and Published Date. These fields… --- ## [Help writing a streaming binary codec](https://discuss.elastic.co/t/help-writing-a-streaming-binary-codec/110986)
**Author:** [@zslayton](https://discuss.elastic.co/u/zslayton)\ **Replies:** 0\ **Last updated:** [December 10, 2017, 8:45pm UTC](https://discuss.elastic.co/t/help-writing-a-streaming-binary-codec/110986 "2017-12-10T20:45:32Z")
Hi all, I'd like to create a codec to decode Ion, a binary data format. I'm new to both LogStash and Ruby. The Codec how-to shows that I need to define a decode method that accepts a single parameter: a variable called… --- ## [Convert mytimestamp to @timestamp](https://discuss.elastic.co/t/convert-mytimestamp-to-timestamp/110921)
**Author:** [@Vivek\_Nigam](https://discuss.elastic.co/u/Vivek_Nigam)\ **Replies:** 3\ **Last updated:** [December 10, 2017, 2:54am UTC](https://discuss.elastic.co/t/convert-mytimestamp-to-timestamp/110921 "2017-12-10T02:54:14Z")
i want to convert mytimestamp to @timestamp i merge two string field to mytimestamp field. now i want to convert this field to date & target to @timestamp. My code: add\_field =\> { "mytimestamp" =\> "%{filedate} %{MyT… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=858) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=860)