# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1090 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1091 --- ## [Logstash write data to the elasticsearch how to remove duplication](https://discuss.elastic.co/t/logstash-write-data-to-the-elasticsearch-how-to-remove-duplication/63246)
**Author:** [@Robert\_Wong](https://discuss.elastic.co/u/Robert_Wong)\ **Replies:** 3\ **Last updated:** [October 19, 2016, 5:33am UTC](https://discuss.elastic.co/t/logstash-write-data-to-the-elasticsearch-how-to-remove-duplication/63246 "2016-10-19T05:33:29Z")
I would like to achieve when I write data through the logstash to elasticsearch when the first to elasticsearch Lane investigation or any other way to achieve the removal of duplication of function? --- ## [Logstash not creating an index in elasticsearch](https://discuss.elastic.co/t/logstash-not-creating-an-index-in-elasticsearch/24231)
**Author:** [@PavanRGowda](https://discuss.elastic.co/u/PavanRGowda)\ **Replies:** 11\ **Last updated:** [October 19, 2016, 5:25am UTC](https://discuss.elastic.co/t/logstash-not-creating-an-index-in-elasticsearch/24231 "2016-10-19T05:25:47Z")
Below is my config file - input { file { path =\> "/home/pavan/Downloads/sample.csv" type =\> "Sample" start\_position =\> "beginning" } } filter { csv { columns =\> \['cda… --- ## [ES5.0.0-rc1 - Index\_not\_found\_exception](https://discuss.elastic.co/t/es5-0-0-rc1-index-not-found-exception/63336)
**Author:** [@sps90](https://discuss.elastic.co/u/sps90)\ **Replies:** 0\ **Last updated:** [October 18, 2016, 11:30pm UTC](https://discuss.elastic.co/t/es5-0-0-rc1-index-not-found-exception/63336 "2016-10-18T23:30:06Z")
I am currently running 5.0.0-rc1 version of ELK stack in ubuntu 14.04 The error i face is this My config file is : input { stdin { } } output { elasticsearch { } stdout { codec =\> rubydebug } } I would really a… --- ## [Received fatal alert: bad\_certificate](https://discuss.elastic.co/t/received-fatal-alert-bad-certificate/63333)
**Author:** [@aadicodes](https://discuss.elastic.co/u/aadicodes)\ **Replies:** 0\ **Last updated:** [October 18, 2016, 10:35pm UTC](https://discuss.elastic.co/t/received-fatal-alert-bad-certificate/63333 "2016-10-18T22:35:41Z")
We are using logstash version 2.3.4 . A cluster of logstash instances gets events from log-courier, filebeat and lumberjack input plugins. Here are the versions of each input plugin. log-courier 1.9.0 filebeat 2.29 lum… --- ## [Field munging](https://discuss.elastic.co/t/field-munging/63128)
**Author:** [@DigiAngel](https://discuss.elastic.co/u/DigiAngel)\ **Replies:** 6\ **Last updated:** [October 18, 2016, 10:58am UTC](https://discuss.elastic.co/t/field-munging/63128 "2016-10-18T10:58:58Z")
So here's what I got. I am monitoring two files via logstash, each has a field "service". Now, I have the below combinations of results in Kibana, the dashes show no result: ntp,- -,ntp ntp,ntp\_google -,- I'd like t… --- ## [Logstash class and file and method display "?"](https://discuss.elastic.co/t/logstash-class-and-file-and-method-display/63271)
**Author:** [@slowlizard](https://discuss.elastic.co/u/slowlizard)\ **Replies:** 1\ **Last updated:** [October 18, 2016, 9:49am UTC](https://discuss.elastic.co/t/logstash-class-and-file-and-method-display/63271 "2016-10-18T09:49:15Z")
"\_index": "meipianapp-2016.10.18", "\_type": "log4j", "\_id": "AVfXDMl1egLz1PD\_qnMm", "\_score": null, "\_source": { "message": "用户查询时光祭异常:MjController.userMjActivityPast: aMapCityCode:010", "@version": "1"… --- ## [Combine Information of two Databases with one Config](https://discuss.elastic.co/t/combine-information-of-two-databases-with-one-config/62719)
**Author:** [@TWalter](https://discuss.elastic.co/u/TWalter)\ **Replies:** 2\ **Last updated:** [October 18, 2016, 8:50am UTC](https://discuss.elastic.co/t/combine-information-of-two-databases-with-one-config/62719 "2016-10-18T08:50:01Z")
Hello there, i use two databases. A relational and non relational one. I want to combine (join) the Information of the two databases with logstash. For instance i want the value of a sensor from the non relational… --- ## [Joining incoming Data with existing Elasticsearch documents](https://discuss.elastic.co/t/joining-incoming-data-with-existing-elasticsearch-documents/62816)
**Author:** [@TWalter](https://discuss.elastic.co/u/TWalter)\ **Replies:** 1\ **Last updated:** [October 18, 2016, 8:48am UTC](https://discuss.elastic.co/t/joining-incoming-data-with-existing-elasticsearch-documents/62816 "2016-10-18T08:48:14Z")
Hello there, i have the following Incoming data: "ID": "1234567" "Value": 20 I have the following documents in an Elasticsearch Index: "ID": "1234567" "Name": "Sensor" "Location": "New York" I want to join with Lo… --- ## [(seems) Doesn't read default configuration, logstash.conf](https://discuss.elastic.co/t/seems-doesnt-read-default-configuration-logstash-conf/63169)
**Author:** [@dstjahjono](https://discuss.elastic.co/u/dstjahjono)\ **Replies:** 4\ **Last updated:** [October 18, 2016, 8:14am UTC](https://discuss.elastic.co/t/seems-doesnt-read-default-configuration-logstash-conf/63169 "2016-10-18T08:14:06Z")
Hi, Newbie here, just installed Logstash, have 2 questions. 1) I follow this step, https://www.elastic.co/guide/en/logstash/5.0/config-examples.html, run in command line and working. \[/opt/logstash\]$ bin/logstash … --- ## [Handle logs comming only when Logstash is turned on](https://discuss.elastic.co/t/handle-logs-comming-only-when-logstash-is-turned-on/62794)
**Author:** [@krzysztof\_pl](https://discuss.elastic.co/u/krzysztof_pl)\ **Replies:** 4\ **Last updated:** [October 18, 2016, 7:56am UTC](https://discuss.elastic.co/t/handle-logs-comming-only-when-logstash-is-turned-on/62794 "2016-10-18T07:56:41Z")
Hi Everyone, I am new one here. I have got a question regarding to 'file input' plugin and sincedb file. Let me try to describe situation: Traces are coming all the time to the specific log file. Generally Logstash is… --- ## [:exception=\>java.lang.ClassCastException: org.jruby.RubyArray cannot be cast to org.jruby.RubyIO, :level=\>:warn](https://discuss.elastic.co/t/exception-java-lang-classcastexception-org-jruby-rubyarray-cannot-be-cast-to-org-jruby-rubyio-level-warn/62895)
**Author:** [@ltkdrop1](https://discuss.elastic.co/u/ltkdrop1)\ **Replies:** 3\ **Last updated:** [October 18, 2016, 6:09am UTC](https://discuss.elastic.co/t/exception-java-lang-classcastexception-org-jruby-rubyarray-cannot-be-cast-to-org-jruby-rubyio-level-warn/62895 "2016-10-18T06:09:11Z")
Hi, I am currently using HTTP\_POLLER to get inputs into the following JSON\_PARSER and I encountered the following errors :- Error parsing json {:source=\>"value", :raw=\>\[{"Type"=\>"Roadwork", "Latitude"=\>#, "Longitude"… --- ## [How to change +07:00 to +0700 in @timestamp?](https://discuss.elastic.co/t/how-to-change-07-00-to-0700-in-timestamp/63027)
**Author:** [@mgxcool](https://discuss.elastic.co/u/mgxcool)\ **Replies:** 1\ **Last updated:** [October 18, 2016, 6:07am UTC](https://discuss.elastic.co/t/how-to-change-07-00-to-0700-in-timestamp/63027 "2016-10-18T06:07:56Z")
Version: 2.3.4 Operating System: CentOS i want to change @timestamp format from 2016-10-09T11:08:57.636Z to 2016-10-09T11:08:57.636+0800 i found two ways using filter to solve this problem 1. filter { ruby { code =\> "e… --- ## [Logstash fails at logstash-filter-cipher plugin](https://discuss.elastic.co/t/logstash-fails-at-logstash-filter-cipher-plugin/63176)
**Author:** [@anandpathak](https://discuss.elastic.co/u/anandpathak)\ **Replies:** 1\ **Last updated:** [October 18, 2016, 6:01am UTC](https://discuss.elastic.co/t/logstash-fails-at-logstash-filter-cipher-plugin/63176 "2016-10-18T06:01:05Z")
Hi, I am using logstash : 2.2.4 and logstash-filter-cipher : 2.0.5 on linux ubuntu 14.04. I am taking input from kafka and filtering it with cipher plugin where I am decrypting message which is encrypted using aes-128-… --- ## [Logstash index creation issue](https://discuss.elastic.co/t/logstash-index-creation-issue/63196)
**Author:** [@srikanth\_ramineni](https://discuss.elastic.co/u/srikanth_ramineni)\ **Replies:** 1\ **Last updated:** [October 18, 2016, 5:59am UTC](https://discuss.elastic.co/t/logstash-index-creation-issue/63196 "2016-10-18T05:59:32Z")
Hi, our system generate netstat and vmstat every half hour and we are ingesting the data and we are overriding the timestamp with my log time stamp in logstash. We have below issue when we are ingesting the historic… --- ## [Modify logstash conf through script](https://discuss.elastic.co/t/modify-logstash-conf-through-script/63235)
**Author:** [@shankarbala](https://discuss.elastic.co/u/shankarbala)\ **Replies:** 1\ **Last updated:** [October 18, 2016, 5:57am UTC](https://discuss.elastic.co/t/modify-logstash-conf-through-script/63235 "2016-10-18T05:57:32Z")
hi we have logstash 2.4.0 running..i know that it can reload whenever the config file changes automatically.. but i wanted to know how to modify the logstash conf file using a script because i wanna do this in automate… --- ## [Logstash grok match pattern for message field](https://discuss.elastic.co/t/logstash-grok-match-pattern-for-message-field/25311)
**Author:** [@abathula](https://discuss.elastic.co/u/abathula)\ **Replies:** 33\ **Last updated:** [October 18, 2016, 5:52am UTC](https://discuss.elastic.co/t/logstash-grok-match-pattern-for-message-field/25311 "2016-10-18T05:52:24Z")
My log data is like, 2015-01-31 15:58:56,851 \[9\] DEBUG NCR.AKPOS.ShoppingCart.CartInvoicer - Setting offline returninvoice: c0000000-2144-04e2-f409-ffff08d20b85 2015-01-31 15:58:56,860 \[9\] DEBUG NCR.AKPOS.ShoppingCart.… --- ## [Help with elapsed time](https://discuss.elastic.co/t/help-with-elapsed-time/31571)
**Author:** [@Greppy](https://discuss.elastic.co/u/Greppy)\ **Replies:** 5\ **Last updated:** [October 18, 2016, 12:22am UTC](https://discuss.elastic.co/t/help-with-elapsed-time/31571 "2016-10-18T00:22:44Z")
For some reason, I can't get the Elapsed time filter to work just right. From the debug log it with logstash 1.5.4 it appears that my start and end event are getting picked up correctly along with the timestamps but the … --- ## [Drop source value - Problem with filter](https://discuss.elastic.co/t/drop-source-value-problem-with-filter/63221)
**Author:** [@sukesh2016](https://discuss.elastic.co/u/sukesh2016)\ **Replies:** 1\ **Last updated:** [October 17, 2016, 8:49pm UTC](https://discuss.elastic.co/t/drop-source-value-problem-with-filter/63221 "2016-10-17T20:49:14Z")
I am new to ES and logstash. I am getting some unwanted logs with source : "horizon-error" in Kibana. I want to drop those messages. Have tried below option, but no luck. Would really appreciate any help. if "horizon" … --- ## [How can I create new events in ruby filter?](https://discuss.elastic.co/t/how-can-i-create-new-events-in-ruby-filter/63199)
**Author:** [@CraigFoote](https://discuss.elastic.co/u/CraigFoote)\ **Replies:** 1\ **Last updated:** [October 17, 2016, 5:46pm UTC](https://discuss.elastic.co/t/how-can-i-create-new-events-in-ruby-filter/63199 "2016-10-17T17:46:02Z")
I want to create new events from the existing one. I'd rather not create, compile and install a new filter if possible rather I'd like to use a ruby filter, e.g. filter{ ruby{ code =\> ' host\_lis… --- ## [How to access newly created custom events from logstash filter?](https://discuss.elastic.co/t/how-to-access-newly-created-custom-events-from-logstash-filter/27955)
**Author:** [@ankmathur14](https://discuss.elastic.co/u/ankmathur14)\ **Replies:** 1\ **Last updated:** [October 17, 2016, 3:17pm UTC](https://discuss.elastic.co/t/how-to-access-newly-created-custom-events-from-logstash-filter/27955 "2016-10-17T15:17:48Z")
Hi, I have created four custom events from a single original event in the ruby { } plugin of filter { }. I dropped the original event ( event.cancel ). I used yield LogStash::Event.new(data) to create new events… --- ## [Very low CPU/Mem usage in logstash server](https://discuss.elastic.co/t/very-low-cpu-mem-usage-in-logstash-server/61667)
**Author:** [@caoping](https://discuss.elastic.co/u/caoping)\ **Replies:** 5\ **Last updated:** [October 17, 2016, 2:28pm UTC](https://discuss.elastic.co/t/very-low-cpu-mem-usage-in-logstash-server/61667 "2016-10-17T14:28:49Z")
Hi, My logstash server has 4 CPUs and 16g Mem, and 4 pipeline-workers (It's said the number of pipeline-worker should be the number of CPUs. ). However, after running more than 20 hours, I found it said: Beats input: … --- ## [Avoid overridding of default message field in logstash](https://discuss.elastic.co/t/avoid-overridding-of-default-message-field-in-logstash/63127)
**Author:** [@javatechy](https://discuss.elastic.co/u/javatechy)\ **Replies:** 2\ **Last updated:** [October 17, 2016, 2:09pm UTC](https://discuss.elastic.co/t/avoid-overridding-of-default-message-field-in-logstash/63127 "2016-10-17T14:09:33Z")
I am using logstash to parse my logs. when i am parsing the json (which contains a "message" field) overrides the default message field. I tried using remove\_field option of json{ } filter but that didn't work work for … --- ## [Convert from string to integer](https://discuss.elastic.co/t/convert-from-string-to-integer/63163)
**Author:** [@Lego](https://discuss.elastic.co/u/Lego)\ **Replies:** 4\ **Last updated:** [October 17, 2016, 12:31pm UTC](https://discuss.elastic.co/t/convert-from-string-to-integer/63163 "2016-10-17T12:31:53Z")
Hello. I need to convert data from string to integer. I use following syntax: filter { if \[type\] == "log1c" { csv { columns =\> \[ "ReportDate","ReportTime","UserName","ErpMode","Actio… --- ## [Aggregate Filter Timeout](https://discuss.elastic.co/t/aggregate-filter-timeout/63179)
**Author:** [@Gabry993](https://discuss.elastic.co/u/Gabry993)\ **Replies:** 0\ **Last updated:** [October 17, 2016, 10:31am UTC](https://discuss.elastic.co/t/aggregate-filter-timeout/63179 "2016-10-17T10:31:16Z")
Hi! I've got a couple of question about this plugin. I need to link a value from event A to an elapsed value from event B which does not depend on timestamp. I also want to retry in a second time if one of the two even… --- ## [How to use logstash S3 input plugin with AWS Mumbai (ap-south-1) region](https://discuss.elastic.co/t/how-to-use-logstash-s3-input-plugin-with-aws-mumbai-ap-south-1-region/63020)
**Author:** [@abhi\_chinchole](https://discuss.elastic.co/u/abhi_chinchole)\ **Replies:** 2\ **Last updated:** [October 17, 2016, 10:30am UTC](https://discuss.elastic.co/t/how-to-use-logstash-s3-input-plugin-with-aws-mumbai-ap-south-1-region/63020 "2016-10-17T10:30:31Z")
I am trying to use Logstash 2.4 to parse logfiles from AWS s3 Mumbai Region (ap-south-1). But Its not working due to compatibility issues of logstash with aws mumbai region. Is there any way to get it working ? --- ## [Cannot find grok pattern for my logs?](https://discuss.elastic.co/t/cannot-find-grok-pattern-for-my-logs/63166)
**Author:** [@amits](https://discuss.elastic.co/u/amits)\ **Replies:** 1\ **Last updated:** [October 17, 2016, 9:35am UTC](https://discuss.elastic.co/t/cannot-find-grok-pattern-for-my-logs/63166 "2016-10-17T09:35:40Z")
Hi, I am stuck in finding pattern for my logs. Please help. Logs are of type: "\[INFO \] \[19:37:08.327\] \[RSP\] |ADD|IOrd 8|EOrd 1100000000011338|ESeq 76038132743205|Strm 3|Error 0|ProcessExchAddResp" I have made grok … --- ## [Logstash Anonymize plugin](https://discuss.elastic.co/t/logstash-anonymize-plugin/63132)
**Author:** [@krlplm](https://discuss.elastic.co/u/krlplm)\ **Replies:** 2\ **Last updated:** [October 17, 2016, 6:29am UTC](https://discuss.elastic.co/t/logstash-anonymize-plugin/63132 "2016-10-17T06:29:32Z")
Hi, I am new to ELK and working on an assignment to anonymize few fields. I could successfully perform this action however is there a way if Logstash/Elasticsearch has a table having Field Value -\> Hash mapping in orde… --- ## [S3 output plugin behaviour on shutdown?](https://discuss.elastic.co/t/s3-output-plugin-behaviour-on-shutdown/2532)
**Author:** [@Sam\_Barham](https://discuss.elastic.co/u/Sam_Barham)\ **Replies:** 5\ **Last updated:** [October 17, 2016, 3:12am UTC](https://discuss.elastic.co/t/s3-output-plugin-behaviour-on-shutdown/2532 "2016-10-17T03:12:40Z")
I want to use the S3 plugin to archive all our logs in an S3 bucket. As I understand it, I can use the size\_file or time\_file options to make it upload to S3 when the file reaches a certain size or age. The issue is t… --- ## [Data Type Conversion for bytes](https://discuss.elastic.co/t/data-type-conversion-for-bytes/63061)
**Author:** [@Hans](https://discuss.elastic.co/u/Hans)\ **Replies:** 5\ **Last updated:** [October 16, 2016, 9:00pm UTC](https://discuss.elastic.co/t/data-type-conversion-for-bytes/63061 "2016-10-16T21:00:39Z")
Hi, I am importing a csv file and need to convert the fields that contain numbers to float to compute them in Kibana. However the bytes fields that contain volume of data should in Kibana be recognized as such. How can I… --- ## [Db streaming from last input](https://discuss.elastic.co/t/db-streaming-from-last-input/62905)
**Author:** [@higee](https://discuss.elastic.co/u/higee)\ **Replies:** 2\ **Last updated:** [October 16, 2016, 5:57pm UTC](https://discuss.elastic.co/t/db-streaming-from-last-input/62905 "2016-10-16T17:57:15Z")
Hi, I'm using logstash to extract data from database and send to elasticsearch and to kibana. What I want to do is I want elasticsesarch to store past data and just get newly inserted data to my oracle database. For ins… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1089) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1091)