# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1239 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1240 --- ## [Tie together events with subfields](https://discuss.elastic.co/t/tie-together-events-with-subfields/28673)
**Author:** [@sissi](https://discuss.elastic.co/u/sissi)\ **Replies:** 1\ **Last updated:** [September 6, 2015, 2:27pm UTC](https://discuss.elastic.co/t/tie-together-events-with-subfields/28673 "2015-09-06T14:27:27Z")
i have a question about tie events together with subfields,log file like following: ### LOG FILE ### id 1 name paris-1-university city paris country French id 1 namestu cici age 18 sex F id 1 name paris-3-university c… --- ## [Logstash output to https](https://discuss.elastic.co/t/logstash-output-to-https/27946)
**Author:** [@elitzur\_e](https://discuss.elastic.co/u/elitzur_e)\ **Replies:** 8\ **Last updated:** [September 6, 2015, 12:05pm UTC](https://discuss.elastic.co/t/logstash-output-to-https/27946 "2015-09-06T12:05:30Z")
Hi. been looking a while for some answer about this. is there support for output to https? this is my output: } http{ url =\> "https://\*\*\*\*/service.svc/log" http\_method =\>"post" } im running logstash on wi… --- ## [Add hours and replace @timestamp](https://discuss.elastic.co/t/add-hours-and-replace-timestamp/28730)
**Author:** [@colormegood](https://discuss.elastic.co/u/colormegood)\ **Replies:** 4\ **Last updated:** [September 6, 2015, 1:02am UTC](https://discuss.elastic.co/t/add-hours-and-replace-timestamp/28730 "2015-09-06T01:02:06Z")
Elasticsearch 1.7.1 Logstash 1.5.4 I use below syslog's logstash config. When i run logstash,error occurred. "logstash ruby exception occurred undefined method localtime". How can i fix it? configuration file for Lo… --- ## [Reposting type mapping ? error with logs and data](https://discuss.elastic.co/t/reposting-type-mapping-error-with-logs-and-data/28690)
**Author:** [@coderextreme](https://discuss.elastic.co/u/coderextreme)\ **Replies:** 1\ **Last updated:** [September 5, 2015, 11:30pm UTC](https://discuss.elastic.co/t/reposting-type-mapping-error-with-logs-and-data/28690 "2015-09-05T23:30:41Z")
Data file Logstash configuration for JSON files Logstash errors -- scroll to bottom I am getting errors loading the data file as detailed in the jsonerrors.log (error 400) I think it's a type mapping issue but not rea… --- ## [Using S3 as input plugin](https://discuss.elastic.co/t/using-s3-as-input-plugin/28725)
**Author:** [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\ **Replies:** 1\ **Last updated:** [September 5, 2015, 8:12pm UTC](https://discuss.elastic.co/t/using-s3-as-input-plugin/28725 "2015-09-05T20:12:14Z")
Hi, I am using s3 as the input plugin to download the logs in s3 bucket. The logs in my s3 bucket is organized in folders with a .tar.gz file in each of them. Can the S3 plugin help me achieve this. The .tar.gz file c… --- ## [Logstash error for datadog output plugin](https://discuss.elastic.co/t/logstash-error-for-datadog-output-plugin/28721)
**Author:** [@Deb](https://discuss.elastic.co/u/Deb)\ **Replies:** 0\ **Last updated:** [September 5, 2015, 5:05pm UTC](https://discuss.elastic.co/t/logstash-error-for-datadog-output-plugin/28721 "2015-09-05T17:05:40Z")
I am trying to submit some events to datadog from logstash using datadog output plugin. Below is my logstash config:- input { stdin{ type =\> "stdin-type" codec =\> json {} } } output { if "\_jsonparsefailure" i… --- ## [How can I start automatic Logstash? (SOLVED)](https://discuss.elastic.co/t/how-can-i-start-automatic-logstash-solved/28638)
**Author:** [@killmasta93](https://discuss.elastic.co/u/killmasta93)\ **Replies:** 4\ **Last updated:** [September 5, 2015, 12:05pm UTC](https://discuss.elastic.co/t/how-can-i-start-automatic-logstash-solved/28638 "2015-09-05T12:05:25Z")
Hi, I was wondering how can I put logstash to start automatic when it boots up I have had no luck on putting it as a daemon maybe as service? Operating system Lubuntu Every time Lubuntu restarts I have to run the comma… --- ## [How to split CSV file and filter CSV Files](https://discuss.elastic.co/t/how-to-split-csv-file-and-filter-csv-files/28710)
**Author:** [@Hans](https://discuss.elastic.co/u/Hans)\ **Replies:** 1\ **Last updated:** [September 5, 2015, 10:35am UTC](https://discuss.elastic.co/t/how-to-split-csv-file-and-filter-csv-files/28710 "2015-09-05T10:35:55Z")
Hi, I have a challenge where I am not able to filter a CSV file in logstash and pass this a separate files to elasticsearch. Original files: CLIENT\_IP,ISP,TEST\_DATE,SERVER\_NAME,DOWNLOAD\_KBPS,UPLOAD\_KBPS,LATENCY,LATITU… --- ## [Plugin couchdb\_changes not loading changes from specific databases](https://discuss.elastic.co/t/plugin-couchdb-changes-not-loading-changes-from-specific-databases/28701)
**Author:** [@micabot](https://discuss.elastic.co/u/micabot)\ **Replies:** 2\ **Last updated:** [September 5, 2015, 4:47am UTC](https://discuss.elastic.co/t/plugin-couchdb-changes-not-loading-changes-from-specific-databases/28701 "2015-09-05T04:47:07Z")
Hi there! I'm new to Logstash, kind of a newbie question as well. I just did a fresh install of Logstash 1.5.4 in my Archlinux machine. When I read changes from one of my DBs in Couch, I see the data output in stdout. Bu… --- ## [Multiline not removing \\n](https://discuss.elastic.co/t/multiline-not-removing-n/28659)
**Author:** [@lookingcloudy](https://discuss.elastic.co/u/lookingcloudy)\ **Replies:** 1\ **Last updated:** [September 5, 2015, 3:40am UTC](https://discuss.elastic.co/t/multiline-not-removing-n/28659 "2015-09-05T03:40:22Z")
My log looks like this: User Name,Transaction,Target,Transaction Time,Total Time PCGennes,POLICY LEVEL To INTEREST,PCG - TBA,01/09/2015 16:13:02:316 - 01/09/2015 16:13:04:100,1.78 ,,,Request Time:,0.39 ,,,Server Time:,… --- ## [Monitoring changes to application property files](https://discuss.elastic.co/t/monitoring-changes-to-application-property-files/28386)
**Author:** [@jjdepaul](https://discuss.elastic.co/u/jjdepaul)\ **Replies:** 8\ **Last updated:** [September 4, 2015, 4:12pm UTC](https://discuss.elastic.co/t/monitoring-changes-to-application-property-files/28386 "2015-09-04T16:12:15Z")
We have several important application property files that we would like to monitor with Logstash (app1.properties, app2.properties, etc). Ideally, whenever anything changes to one of these application property files, w… --- ## [How do I match a log pattern to different log files?](https://discuss.elastic.co/t/how-do-i-match-a-log-pattern-to-different-log-files/28661)
**Author:** [@ushadatt](https://discuss.elastic.co/u/ushadatt)\ **Replies:** 1\ **Last updated:** [September 4, 2015, 1:59pm UTC](https://discuss.elastic.co/t/how-do-i-match-a-log-pattern-to-different-log-files/28661 "2015-09-04T13:59:49Z")
I have two patterns defined for two different kind of logs in my config file.. example: simple\_logs and error\_logs So, now I am adding a new field, if the pattern matches simple-logs. But even if my log file contains t… --- ## [Matching two different patterns in one config](https://discuss.elastic.co/t/matching-two-different-patterns-in-one-config/27976)
**Author:** [@jjdepaul](https://discuss.elastic.co/u/jjdepaul)\ **Replies:** 4\ **Last updated:** [September 4, 2015, 1:23pm UTC](https://discuss.elastic.co/t/matching-two-different-patterns-in-one-config/27976 "2015-09-04T13:23:03Z")
I have been trying several things to no avail - need some help. My log file contains both successful and unsuccessful transactions - trouble is that I have to use a different match pattern to recognize Success and Fail… --- ## [Select fields for multiple outputs](https://discuss.elastic.co/t/select-fields-for-multiple-outputs/28531)
**Author:** [@dyermakov](https://discuss.elastic.co/u/dyermakov)\ **Replies:** 7\ **Last updated:** [September 4, 2015, 9:43am UTC](https://discuss.elastic.co/t/select-fields-for-multiple-outputs/28531 "2015-09-04T09:43:42Z")
Hi all! I have a config: input { rabbitmq { host =\> "x.x.x.x" port =\> 5672 vhost =\> "/" user =\> "guest" password =\> "guest" queue =\> "yyyy" durable =\> true codec =\> "es\_bulk" } } fil… --- ## [Grok filter if condition issue](https://discuss.elastic.co/t/grok-filter-if-condition-issue/28619)
**Author:** [@Navneet\_Mathpal](https://discuss.elastic.co/u/Navneet_Mathpal)\ **Replies:** 2\ **Last updated:** [September 4, 2015, 9:01am UTC](https://discuss.elastic.co/t/grok-filter-if-condition-issue/28619 "2015-09-04T09:01:47Z")
Hi All, how we can use if command in grok some logs contain tag method ex : "method" =\> "get" // I am able to parse the log I want to add another field , if I am getting the log which is having tag called method I … --- ## [ElasticSearch RAM requirement](https://discuss.elastic.co/t/elasticsearch-ram-requirement/28657)
**Author:** [@Hayder\_Abbass](https://discuss.elastic.co/u/Hayder_Abbass)\ **Replies:** 1\ **Last updated:** [September 4, 2015, 8:20am UTC](https://discuss.elastic.co/t/elasticsearch-ram-requirement/28657 "2015-09-04T08:20:19Z")
Hello, We are thinking of using ElasticSearch to store log data with the following simple schema: { "data" : { "properties" : { "date": { "type" : "date", "format" … --- ## [Filter specific Message with logstash before sending to ElasticSearch](https://discuss.elastic.co/t/filter-specific-message-with-logstash-before-sending-to-elasticsearch/28584)
**Author:** [@Hayder\_Abbass](https://discuss.elastic.co/u/Hayder_Abbass)\ **Replies:** 2\ **Last updated:** [September 4, 2015, 7:52am UTC](https://discuss.elastic.co/t/filter-specific-message-with-logstash-before-sending-to-elasticsearch/28584 "2015-09-04T07:52:02Z")
Hello, I had like to know if it is possible to send only specific log messages to elasticsearch via logstash? E.G let's say I have these messages in my log file: 2015-08-14 12:21:03 \[31946\] PASS level2 10.249.10.70… --- ## [Drop logs when there is a jsonparsefailure](https://discuss.elastic.co/t/drop-logs-when-there-is-a-jsonparsefailure/28655)
**Author:** [@Deb](https://discuss.elastic.co/u/Deb)\ **Replies:** 3\ **Last updated:** [September 4, 2015, 7:35am UTC](https://discuss.elastic.co/t/drop-logs-when-there-is-a-jsonparsefailure/28655 "2015-09-04T07:35:00Z")
Hi, I am trying to push some of our logs to elasticsearch via logstash. Can some one let me know how can I drop the logs in logstash itself if the logs are not in proper json format , before they are being pushed to el… --- ## [Sequel::PoolTimeout error when contacting Database](https://discuss.elastic.co/t/sequel-pooltimeout-error-when-contacting-database/28414)
**Author:** [@KarthikaAr](https://discuss.elastic.co/u/KarthikaAr)\ **Replies:** 1\ **Last updated:** [September 4, 2015, 6:28am UTC](https://discuss.elastic.co/t/sequel-pooltimeout-error-when-contacting-database/28414 "2015-09-04T06:28:08Z")
Can Somebody explains what this Error means. How it can be fixed, { 11754 rufus-scheduler intercepted an error: 11754 job: 11754 Rufus::Scheduler::CronJob "\*/14 \* \* \* \*" {} 11754 error: 11754 11754 … --- ## [Need help in configuring email alert in logstash 1.5.4](https://discuss.elastic.co/t/need-help-in-configuring-email-alert-in-logstash-1-5-4/28508)
**Author:** [@asgawali](https://discuss.elastic.co/u/asgawali)\ **Replies:** 2\ **Last updated:** [September 4, 2015, 5:46am UTC](https://discuss.elastic.co/t/need-help-in-configuring-email-alert-in-logstash-1-5-4/28508 "2015-09-04T05:46:53Z")
Hi, I was using logstash 1.4.2 and this email alert was working fine at that time. Now my system is upgraded to logstash 1.5.4 and logstash is failing because of older settings as below: email { from =\> "{{M… --- ## [Type woes loading X3D JSON](https://discuss.elastic.co/t/type-woes-loading-x3d-json/28555)
**Author:** [@coderextreme](https://discuss.elastic.co/u/coderextreme)\ **Replies:** 5\ **Last updated:** [September 4, 2015, 3:12am UTC](https://discuss.elastic.co/t/type-woes-loading-x3d-json/28555 "2015-09-04T03:12:44Z")
I am experiencing type woes loading X3D JSON. I think because it's a very hierarchical document, and there's no schema for it yet. There is an XML schema, which might be used to load metadata into elastic search, but … --- ## [Do we need jdk and jre to run logstash and elasticsearch or jre itself enough?](https://discuss.elastic.co/t/do-we-need-jdk-and-jre-to-run-logstash-and-elasticsearch-or-jre-itself-enough/28260)
**Author:** [@thiruna](https://discuss.elastic.co/u/thiruna)\ **Replies:** 4\ **Last updated:** [September 3, 2015, 10:42pm UTC](https://discuss.elastic.co/t/do-we-need-jdk-and-jre-to-run-logstash-and-elasticsearch-or-jre-itself-enough/28260 "2015-09-03T22:42:44Z")
Hi, We are using below version of Java for logstash and elasticsearch in our environment. java -version java version "1.7.0\_51" Java(TM) SE Runtime Environment (build 1.7.0\_51-b13) Java HotSpot(TM) 64-Bit Server VM (bu… --- ## [How do I read nested JSON?](https://discuss.elastic.co/t/how-do-i-read-nested-json/1775)
**Author:** [@John\_Yeuk\_Hon\_Wong](https://discuss.elastic.co/u/John_Yeuk_Hon_Wong)\ **Replies:** 2\ **Last updated:** [September 3, 2015, 8:37pm UTC](https://discuss.elastic.co/t/how-do-i-read-nested-json/1775 "2015-09-03T20:37:28Z")
Hi. Here is part of the JSON doc in Kibana after I use JSON filter to process incoming log. "OldStateValue": "INSUFFICIENT\_DATA", "Trigger": { "MetricName": "CPUUtilization", "Namespace": "AWS/EC2", "Statis… --- ## [Fingerprint issue](https://discuss.elastic.co/t/fingerprint-issue/25554)
**Author:** [@Khoa\_Nguyen1](https://discuss.elastic.co/u/Khoa_Nguyen1)\ **Replies:** 2\ **Last updated:** [September 3, 2015, 6:19pm UTC](https://discuss.elastic.co/t/fingerprint-issue/25554 "2015-09-03T18:19:18Z")
My CSV file has 26041 unique lines (I checked by manually generating sha1sum for each line). However, when using fingerprint as the document\_id, elastic only stores 25919 documents. One document has the wrong \_id: \_id:… --- ## [Logging in JSON format using custom log4j Patter Layout](https://discuss.elastic.co/t/logging-in-json-format-using-custom-log4j-patter-layout/28622)
**Author:** [@ravitandur](https://discuss.elastic.co/u/ravitandur)\ **Replies:** 1\ **Last updated:** [September 3, 2015, 6:05pm UTC](https://discuss.elastic.co/t/logging-in-json-format-using-custom-log4j-patter-layout/28622 "2015-09-03T18:05:43Z")
Hi All, I was using the ELK stack for some time and integrated with most of application servers which are developed in java. We are using log4j framework for logging and initially configured the log stash agent with in… --- ## [Selective indexing of fields](https://discuss.elastic.co/t/selective-indexing-of-fields/28609)
**Author:** [@astro](https://discuss.elastic.co/u/astro)\ **Replies:** 2\ **Last updated:** [September 3, 2015, 2:26pm UTC](https://discuss.elastic.co/t/selective-indexing-of-fields/28609 "2015-09-03T14:26:05Z")
Hi All , I have a specific problem related to json parsing and extracting particular field for indexing My sample json message is {"log\_time":"1\\/Sep\\/2015:12:35:50 +05:30","level":"INFO","log\_message":"test messag… --- ## [Want to Combine Field Values into New Field Value](https://discuss.elastic.co/t/want-to-combine-field-values-into-new-field-value/28564)
**Author:** [@bhutchinson](https://discuss.elastic.co/u/bhutchinson)\ **Replies:** 3\ **Last updated:** [September 3, 2015, 2:06pm UTC](https://discuss.elastic.co/t/want-to-combine-field-values-into-new-field-value/28564 "2015-09-03T14:06:08Z")
The useragent filter creates new field names in elasticsearch like "name", "major version", "minor version". Name is the browser name. Major version is the browser major version. Minor is the browser minor version. I… --- ## [InfluxDB output plugin - configuration for collectd metrics](https://discuss.elastic.co/t/influxdb-output-plugin-configuration-for-collectd-metrics/28589)
**Author:** [@asphaltsurfer](https://discuss.elastic.co/u/asphaltsurfer)\ **Replies:** 0\ **Last updated:** [September 3, 2015, 9:20am UTC](https://discuss.elastic.co/t/influxdb-output-plugin-configuration-for-collectd-metrics/28589 "2015-09-03T09:20:58Z")
Hi guys, I'm using standard input collecd plugin for logstash to collect CPU, memory and network interface bandwidth usage and I'm trying to understand how to properly put these metrics in InfluxDB and how to configure c… --- ## [How can i read seperated logs only when there is error occured current logs in Logstash?](https://discuss.elastic.co/t/how-can-i-read-seperated-logs-only-when-there-is-error-occured-current-logs-in-logstash/28580)
**Author:** [@RohanKumbhar](https://discuss.elastic.co/u/RohanKumbhar)\ **Replies:** 2\ **Last updated:** [September 3, 2015, 8:38am UTC](https://discuss.elastic.co/t/how-can-i-read-seperated-logs-only-when-there-is-error-occured-current-logs-in-logstash/28580 "2015-09-03T08:38:52Z")
Hi Team, I have one use case , where there two log files available in log folder and when there is error occured in1st log file i want to read the 2nd log file and then co-relate the information from the both file … --- ## [Cannot get the aggregate plugin to work](https://discuss.elastic.co/t/cannot-get-the-aggregate-plugin-to-work/28570)
**Author:** [@anoban](https://discuss.elastic.co/u/anoban)\ **Replies:** 1\ **Last updated:** [September 3, 2015, 8:28am UTC](https://discuss.elastic.co/t/cannot-get-the-aggregate-plugin-to-work/28570 "2015-09-03T08:28:18Z")
I am trying to compute response times and therefore do a line2 - line1 type operation. As I couldn't get this to work with real data, I wrote a tiny python program which logs in this format: The idea now is compare wh… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1238) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1240)