# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=583 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 584 --- ## [File in S3 has additional timestamp and host name as prefix to each message processed](https://discuss.elastic.co/t/file-in-s3-has-additional-timestamp-and-host-name-as-prefix-to-each-message-processed/172571)
**Author:** [@kaushik.vankayala](https://discuss.elastic.co/u/kaushik.vankayala)\ **Replies:** 1\ **Last updated:** [March 15, 2019, 4:45pm UTC](https://discuss.elastic.co/t/file-in-s3-has-additional-timestamp-and-host-name-as-prefix-to-each-message-processed/172571 "2019-03-15T16:45:56Z")
Hi There, I am having S3 as one of the output plugins in my logstash conf. Belos is my s3 plugin configuration details; s3 { id =\> "uat-s3-backup" bucket =\> "mt-uat-project" access\_key\_id =\> "BK7U8I90DGFN236BQ62A" … --- ## [If-else at Multiple Pipeline](https://discuss.elastic.co/t/if-else-at-multiple-pipeline/172235)
**Author:** [@xodn0812](https://discuss.elastic.co/u/xodn0812)\ **Replies:** 3\ **Last updated:** [March 15, 2019, 3:46pm UTC](https://discuss.elastic.co/t/if-else-at-multiple-pipeline/172235 "2019-03-15T15:46:58Z")
I want to use multiple pipelines. When i have aa.conf and bb.conf, each uses a different filter {}. So if-elase seems to have to be branched from the pipeline, is there a way? --- ## [Error parsing syslog input message in Logstash](https://discuss.elastic.co/t/error-parsing-syslog-input-message-in-logstash/172530)
**Author:** [@lcavator](https://discuss.elastic.co/u/lcavator)\ **Replies:** 2\ **Last updated:** [March 15, 2019, 2:43pm UTC](https://discuss.elastic.co/t/error-parsing-syslog-input-message-in-logstash/172530 "2019-03-15T14:43:17Z")
Hi to all, I'm trying to parse a syslog message (coming from vmWare Log Insight) to obtain additional fields added inside the message body string, and then send them in json format to a kafka broker. This is an example… --- ## [Error registering plugin for logstash output jdbc](https://discuss.elastic.co/t/error-registering-plugin-for-logstash-output-jdbc/172170)
**Author:** [@elkha](https://discuss.elastic.co/u/elkha)\ **Replies:** 4\ **Last updated:** [March 15, 2019, 1:51pm UTC](https://discuss.elastic.co/t/error-registering-plugin-for-logstash-output-jdbc/172170 "2019-03-15T13:51:15Z")
Hi , I am trying to pull data from file and send the result to JDBC oracle , i installed the plugin logstash-output-jdbc succefully but when i run the conf file i am getting the following error : \[2019-03-13T15:02:3… --- ## [Jdbc input plugin for influxdb](https://discuss.elastic.co/t/jdbc-input-plugin-for-influxdb/172454)
**Author:** [@jasony](https://discuss.elastic.co/u/jasony)\ **Replies:** 2\ **Last updated:** [March 15, 2019, 2:08pm UTC](https://discuss.elastic.co/t/jdbc-input-plugin-for-influxdb/172454 "2019-03-15T14:08:07Z")
hello, i am currently looking for way to transfer data from influxdb to elasticsearch. it should be performed every couple seconds likely realtime. i found below link but there is no explain about if it supports influx… --- ## [Multiple elasticsearch clusters for output](https://discuss.elastic.co/t/multiple-elasticsearch-clusters-for-output/172503)
**Author:** [@jasony](https://discuss.elastic.co/u/jasony)\ **Replies:** 2\ **Last updated:** [March 15, 2019, 2:07pm UTC](https://discuss.elastic.co/t/multiple-elasticsearch-clusters-for-output/172503 "2019-03-15T14:07:08Z")
Hello I am currently managing logstash-elasticsearch set. Per customer request, we need to add 2nd elasticsearch cluster but current logstashes will be used same and deliver logs to both clusters. Is it possible to set… --- ## [Parsing message with plain text and json](https://discuss.elastic.co/t/parsing-message-with-plain-text-and-json/172261)
**Author:** [@sunilmchaudhari](https://discuss.elastic.co/u/sunilmchaudhari)\ **Replies:** 8\ **Last updated:** [March 15, 2019, 2:00pm UTC](https://discuss.elastic.co/t/parsing-message-with-plain-text-and-json/172261 "2019-03-15T14:00:01Z")
Hello, I have to filter one log which is mix of plain text and the json message. So the sample is \[ERROR\] ABCD posting failed! { StatusCodeError: 422 - {"error":{"message":"Duplicated ticket","status\_code":422}} Can … --- ## [Logstash: How to mount a .conf file to elasticsearch](https://discuss.elastic.co/t/logstash-how-to-mount-a-conf-file-to-elasticsearch/172543)
**Author:** [@Robin020](https://discuss.elastic.co/u/Robin020)\ **Replies:** 0\ **Last updated:** [March 15, 2019, 1:53pm UTC](https://discuss.elastic.co/t/logstash-how-to-mount-a-conf-file-to-elasticsearch/172543 "2019-03-15T13:53:06Z")
Hello, I would like to mount my new made .conf file (located in /etc/logstash/conf.d/. Does somebody know the command for that? I can't find it on the internet (only old methods). Thanks alot! Robin --- ## [One or more Inputs to One or more Outputs](https://discuss.elastic.co/t/one-or-more-inputs-to-one-or-more-outputs/171906)
**Author:** [@samyo](https://discuss.elastic.co/u/samyo)\ **Replies:** 6\ **Last updated:** [March 15, 2019, 11:01am UTC](https://discuss.elastic.co/t/one-or-more-inputs-to-one-or-more-outputs/171906 "2019-03-15T11:01:23Z")
Hello Folks, i would like to ask, i have a lot of textfiels(log files) in one folder, each file has his name , and i matched them all in Logstash , is there any possibility after i matched all those textfiles(log file… --- ## [How to get offset value using Logstash Filter](https://discuss.elastic.co/t/how-to-get-offset-value-using-logstash-filter/172248)
**Author:** [@sana1](https://discuss.elastic.co/u/sana1)\ **Replies:** 4\ **Last updated:** [March 15, 2019, 12:55pm UTC](https://discuss.elastic.co/t/how-to-get-offset-value-using-logstash-filter/172248 "2019-03-15T12:55:02Z")
How to get offset value from a hex string "0b590b5900000000263efd95" say I want 0b printed in my new field "startmessage" --- ## [Multiple pipelines on Logstash 6.6](https://discuss.elastic.co/t/multiple-pipelines-on-logstash-6-6/172528)
**Author:** [@RafaelMontesinos](https://discuss.elastic.co/u/RafaelMontesinos)\ **Replies:** 0\ **Last updated:** [March 15, 2019, 12:43pm UTC](https://discuss.elastic.co/t/multiple-pipelines-on-logstash-6-6/172528 "2019-03-15T12:43:37Z")
Regards I have a problem when starting an instance of Logstash 6.6 with multiple pipelines, each pipeline will query an Oracle DB, these queries will generate a set of JSON data that will be showed at Kibana in a Dashbo… --- ## [Indexing is freeze any time while indexing the 5 crore data](https://discuss.elastic.co/t/indexing-is-freeze-any-time-while-indexing-the-5-crore-data/172245)
**Author:** [@Sweta\_Singh](https://discuss.elastic.co/u/Sweta_Singh)\ **Replies:** 8\ **Last updated:** [March 15, 2019, 11:58am UTC](https://discuss.elastic.co/t/indexing-is-freeze-any-time-while-indexing-the-5-crore-data/172245 "2019-03-15T11:58:30Z")
Hi, Actually i am indexing 5 crore data into my vm ware elastic server from SQL server using log-stash ,creating the 9 index with 1 primary shard and 1 replica shard and running the Multipipeline in pipeline.yml file … --- ## [Logstash stops when running as a service on window server 2012 r2](https://discuss.elastic.co/t/logstash-stops-when-running-as-a-service-on-window-server-2012-r2/172498)
**Author:** [@Rajesh\_Verma](https://discuss.elastic.co/u/Rajesh_Verma)\ **Replies:** 1\ **Last updated:** [March 15, 2019, 11:22am UTC](https://discuss.elastic.co/t/logstash-stops-when-running-as-a-service-on-window-server-2012-r2/172498 "2019-03-15T11:22:43Z")
I am running the logstash to index around 10M of document in the elastic using the logstash multi pipe line, Everthing going good but after 10% of data has been index then after the Logstash terminate. flashing the erro… --- ## [Unable to parse XML dataset, Failed to install template](https://discuss.elastic.co/t/unable-to-parse-xml-dataset-failed-to-install-template/172513)
**Author:** [@Rashmi\_Nagpal](https://discuss.elastic.co/u/Rashmi_Nagpal)\ **Replies:** 0\ **Last updated:** [March 15, 2019, 10:54am UTC](https://discuss.elastic.co/t/unable-to-parse-xml-dataset-failed-to-install-template/172513 "2019-03-15T10:54:15Z")
Can someone please help me here, I'm getting error as Failed to install template.{:message=\>"Template file ' ' could not be found!" on running below script. filter { ## interpret the message payload as XML xml … --- ## [Logstash problem with Logrotate: Missing out on a lot of logs](https://discuss.elastic.co/t/logstash-problem-with-logrotate-missing-out-on-a-lot-of-logs/172496)
**Author:** [@GiteshKhanna](https://discuss.elastic.co/u/GiteshKhanna)\ **Replies:** 0\ **Last updated:** [March 15, 2019, 9:35am UTC](https://discuss.elastic.co/t/logstash-problem-with-logrotate-missing-out-on-a-lot-of-logs/172496 "2019-03-15T09:35:04Z")
Hi, Problem: I am missing on alot of logs. My logrotate config looks like this: files: "/etc/logrotate.d/webapp": mode: "000755" owner: root group: root content: | /mnt/logs/xx.log { missingok maxsize 1G rota… --- ## [Expected json encoding for TCP input?](https://discuss.elastic.co/t/expected-json-encoding-for-tcp-input/172301)
**Author:** [@chmike](https://discuss.elastic.co/u/chmike)\ **Replies:** 4\ **Last updated:** [March 15, 2019, 10:38am UTC](https://discuss.elastic.co/t/expected-json-encoding-for-tcp-input/172301 "2019-03-15T10:38:32Z")
I need to implement a client that sends logging records to a logstash instance at a relative high rate (at least 2kHz). https can’t match this requirement. An option is to use the TCP input plugin, but I failed to find a… --- ## [Config JDBC input to only get new logs from MySQL](https://discuss.elastic.co/t/config-jdbc-input-to-only-get-new-logs-from-mysql/172509)
**Author:** [@thangld](https://discuss.elastic.co/u/thangld)\ **Replies:** 0\ **Last updated:** [March 15, 2019, 10:32am UTC](https://discuss.elastic.co/t/config-jdbc-input-to-only-get-new-logs-from-mysql/172509 "2019-03-15T10:32:15Z")
Hello everyone, I'm trying to use logstash to get logs from MySQL server. It keeps getting every logs every minute. This is my config for jdbc: jdbc { clean\_run =\> true jdbc\_connection\_string =\> "jdbc:… --- ## [How to convert the hrs and min to sec in the log file item](https://discuss.elastic.co/t/how-to-convert-the-hrs-and-min-to-sec-in-the-log-file-item/172507)
**Author:** [@Kharishma](https://discuss.elastic.co/u/Kharishma)\ **Replies:** 0\ **Last updated:** [March 15, 2019, 10:24am UTC](https://discuss.elastic.co/t/how-to-convert-the-hrs-and-min-to-sec-in-the-log-file-item/172507 "2019-03-15T10:24:49Z")
Hi all, 2018-12-18 08:25:56|Info|DEVAPPSVR016|11976|bgtTagger|total tagger time : 0.031 secs 2018-12-18 09:01:06|Debug|DEVAPPSVR016|7944|bgtProtocolAdapter|\[tag\] 2018-12-18 08:25:56|Stats|DEVAPPSVR016|11976|bgtTagger|… --- ## [Matching/Finding keywords in an already parsed field via GROK](https://discuss.elastic.co/t/matching-finding-keywords-in-an-already-parsed-field-via-grok/171999)
**Author:** [@WarriorHarb](https://discuss.elastic.co/u/WarriorHarb)\ **Replies:** 2\ **Last updated:** [March 15, 2019, 9:28am UTC](https://discuss.elastic.co/t/matching-finding-keywords-in-an-already-parsed-field-via-grok/171999 "2019-03-15T09:28:02Z")
Hello, So i have an unstructured log, after applying a GROK filter, i was able to parse it into seperate fields. The problem is that the last field which is {GREEDYDATA:Message} contains some keywords like "userID", "F… --- ## [Logstash failed to index data in elasticsearch](https://discuss.elastic.co/t/logstash-failed-to-index-data-in-elasticsearch/172495)
**Author:** [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\ **Replies:** 0\ **Last updated:** [March 15, 2019, 9:21am UTC](https://discuss.elastic.co/t/logstash-failed-to-index-data-in-elasticsearch/172495 "2019-03-15T09:21:41Z")
HI Team, I'm wondering with my configuration. Logstash is up and running but fail to parse data into elasticsearch, please find my conf below and correct me if am i doing any wrong, I'm running using windows input { … --- ## [Logstash split long json with beats input](https://discuss.elastic.co/t/logstash-split-long-json-with-beats-input/172489)
**Author:** [@Sergey.k](https://discuss.elastic.co/u/Sergey.k)\ **Replies:** 0\ **Last updated:** [March 15, 2019, 8:56am UTC](https://discuss.elastic.co/t/logstash-split-long-json-with-beats-input/172489 "2019-03-15T08:56:09Z")
When long json log with xml insertion being processed in logstash it split one line into two with \_jsonparefailure tag. logstash 6.2.4 CentOS Linux release 7.4.1708 (Core) input{ beats { port =\> 5001 } } filt… --- ## [Removing GREEDYDATA from Filter leaves empty "curly brace" at result?](https://discuss.elastic.co/t/removing-greedydata-from-filter-leaves-empty-curly-brace-at-result/172315)
**Author:** [@samyo](https://discuss.elastic.co/u/samyo)\ **Replies:** 4\ **Last updated:** [March 15, 2019, 7:50am UTC](https://discuss.elastic.co/t/removing-greedydata-from-filter-leaves-empty-curly-brace-at-result/172315 "2019-03-15T07:50:04Z")
hello Folks, i have following issue, i am using Grok filter with GREEDYDATA wich i want to remove(drop it of) out of my result, this is what i used , but at result it is still giving me an empty curly brace {} wich i do… --- ## [Way to delete ElasticSearch document using Logstash configuration](https://discuss.elastic.co/t/way-to-delete-elasticsearch-document-using-logstash-configuration/172272)
**Author:** [@keep](https://discuss.elastic.co/u/keep)\ **Replies:** 3\ **Last updated:** [March 15, 2019, 7:47am UTC](https://discuss.elastic.co/t/way-to-delete-elasticsearch-document-using-logstash-configuration/172272 "2019-03-15T07:47:46Z")
Way to delete ElasticSearch document using Logstash configuration. By delete with condition --- ## [Define time format as date prompt error](https://discuss.elastic.co/t/define-time-format-as-date-prompt-error/172050)
**Author:** [@shenshensinian](https://discuss.elastic.co/u/shenshensinian)\ **Replies:** 6\ **Last updated:** [March 15, 2019, 1:15am UTC](https://discuss.elastic.co/t/define-time-format-as-date-prompt-error/172050 "2019-03-15T01:15:38Z")
Error message：failed to parse field \[timestamp\] of type \[date\] Time sample format： mar 13 10:36:03 mar 13 2019 10:36:03 2019-03-13 11:11:11 I want to be able to convert different date formats into yyyy-MM-dd hh:m… --- ## [Very low cpu usage in logstash server](https://discuss.elastic.co/t/very-low-cpu-usage-in-logstash-server/172240)
**Author:** [@xiaochuan\_lin](https://discuss.elastic.co/u/xiaochuan_lin)\ **Replies:** 2\ **Last updated:** [March 15, 2019, 1:12am UTC](https://discuss.elastic.co/t/very-low-cpu-usage-in-logstash-server/172240 "2019-03-15T01:12:52Z")
Hi, My logstash server has 8 CPUs and 8g Men, and 4 pipeline-workers( 4 pipeline-workers is the best performance after testing), but no matter how i test it, the CPU usage has been 20-30%. The logstash input data come… --- ## [Error ArgumentError Setting "xpack.monitoring.elasticsearch.hosts" hasn't been registered](https://discuss.elastic.co/t/error-argumenterror-setting-xpack-monitoring-elasticsearch-hosts-hasnt-been-registered/172418)
**Author:** [@Adrian\_Hove](https://discuss.elastic.co/u/Adrian_Hove)\ **Replies:** 2\ **Last updated:** [March 14, 2019, 9:57pm UTC](https://discuss.elastic.co/t/error-argumenterror-setting-xpack-monitoring-elasticsearch-hosts-hasnt-been-registered/172418 "2019-03-14T21:57:06Z")
I get the error below: root@b48c5173f995:/var/www# /usr/share/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }' Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.pr… --- ## [Accounting for "field name cannot be an empty string"?](https://discuss.elastic.co/t/accounting-for-field-name-cannot-be-an-empty-string/172423)
**Author:** [@Kenneth\_M\_Kolano](https://discuss.elastic.co/u/Kenneth_M_Kolano)\ **Replies:** 1\ **Last updated:** [March 14, 2019, 9:52pm UTC](https://discuss.elastic.co/t/accounting-for-field-name-cannot-be-an-empty-string/172423 "2019-03-14T21:52:10Z")
I have incoming JSON logs with sections like... \> {"results":{"workers":{"trid": { "TRD": \[ { "likely": "50.0%", "type": "TrID defs package (14009/2/5)" } \], "": \[ { "likely": "14.2%", "type": "Gene… --- ## [Couldn't find any Elasticsearch data after deleting indexes](https://discuss.elastic.co/t/couldnt-find-any-elasticsearch-data-after-deleting-indexes/172416)
**Author:** [@Arthur19](https://discuss.elastic.co/u/Arthur19)\ **Replies:** 0\ **Last updated:** [March 14, 2019, 8:33pm UTC](https://discuss.elastic.co/t/couldnt-find-any-elasticsearch-data-after-deleting-indexes/172416 "2019-03-14T20:33:18Z")
Hi, Thanks in advance for your help. While I was trying to reload old logs, I deleted the existing indexes: DELETE filebeat-\* DELETE graylog\* In addition deleted the filebeat data folder which had the regisry and meta… --- ## [Help troubleshooting tcp input](https://discuss.elastic.co/t/help-troubleshooting-tcp-input/172214)
**Author:** [@reswob](https://discuss.elastic.co/u/reswob)\ **Replies:** 4\ **Last updated:** [March 14, 2019, 7:35pm UTC](https://discuss.elastic.co/t/help-troubleshooting-tcp-input/172214 "2019-03-14T19:35:35Z")
I have a simple conf file: input { tcp { port =\> 2000 } } filter { } output { stdout { coded =\> rubydebug } } That's it. I'm using nifi putTCP to send the logs. I see the logs come i… --- ## [Logstash using high CPU](https://discuss.elastic.co/t/logstash-using-high-cpu/172322)
**Author:** [@swilkey](https://discuss.elastic.co/u/swilkey)\ **Replies:** 3\ **Last updated:** [March 14, 2019, 6:00pm UTC](https://discuss.elastic.co/t/logstash-using-high-cpu/172322 "2019-03-14T18:00:05Z")
The other related topics didn't seem similar enough to this so I'm posting. I've installed kolide fleet, elasticsearch, kibana, and i've installed logstash all on the same server. Everything is fine (kolide fleet, elas… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=582) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=584)