# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=618 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 619 --- ## [\[ERROR\]\[logstash.inputs.metrics \] Failed to create monitoring event {:message=\>"For path: events. Map keys: \[:pipelines, :reloads\]", :error=\>"LogStash::Instrument::MetricStore::MetricNotFound"}](https://discuss.elastic.co/t/error-logstash-inputs-metrics-failed-to-create-monitoring-event-message-for-path-events-map-keys-pipelines-reloads-error-logstash-metricnotfound/163628)
**Author:** [@alexus](https://discuss.elastic.co/u/alexus)\ **Replies:** 1\ **Last updated:** [January 15, 2019, 4:28pm UTC](https://discuss.elastic.co/t/error-logstash-inputs-metrics-failed-to-create-monitoring-event-message-for-path-events-map-keys-pipelines-reloads-error-logstash-metricnotfound/163628 "2019-01-15T16:28:54Z")
I'm following Configuring Centralized Pipeline Management | Logstash Reference \[6.5\] | Elastic and as I enable following: # grep -i management docker-compose.override.yml - XPACK\_MANAGEMENT\_ELAS… --- ## [If make same head Multiline codec](https://discuss.elastic.co/t/if-make-same-head-multiline-codec/164227)
**Author:** [@jamespb](https://discuss.elastic.co/u/jamespb)\ **Replies:** 1\ **Last updated:** [January 15, 2019, 3:52pm UTC](https://discuss.elastic.co/t/if-make-same-head-multiline-codec/164227 "2019-01-15T15:52:46Z")
I would like to make if beginning of 63 chars the same to be same document. For example if following 8 lines start with "2018/11/14-15:14:14.518292-09855-09855-432638034320556033-004-" is one doc, if logstash find a … --- ## [Parsing tomcat hybris logs with logstash](https://discuss.elastic.co/t/parsing-tomcat-hybris-logs-with-logstash/164325)
**Author:** [@pavlin117](https://discuss.elastic.co/u/pavlin117)\ **Replies:** 0\ **Last updated:** [January 15, 2019, 2:53pm UTC](https://discuss.elastic.co/t/parsing-tomcat-hybris-logs-with-logstash/164325 "2019-01-15T14:53:28Z")
Hi all, I'm having trouble with creating a multiline message with hybris. The problem is that if the message in the console is: \` org.opensaml.saml2.metadata.provider.MetadataProviderException: org.opensaml.saml2.metad… --- ## [Logstash grok/mutate question](https://discuss.elastic.co/t/logstash-grok-mutate-question/164200)
**Author:** [@alexevon](https://discuss.elastic.co/u/alexevon)\ **Replies:** 6\ **Last updated:** [January 15, 2019, 2:19pm UTC](https://discuss.elastic.co/t/logstash-grok-mutate-question/164200 "2019-01-15T14:19:09Z")
Hi all- I have a fairly simple question which I hope not too many asked about: I have a sample JSON stream from kafka into logstash that looks like this example data: { "\_index": "kafka\_logstash", "\_type": "doc", "… --- ## [Logstash - list of codes to replace](https://discuss.elastic.co/t/logstash-list-of-codes-to-replace/163767)
**Author:** [@Francisca\_Lima](https://discuss.elastic.co/u/Francisca_Lima)\ **Replies:** 3\ **Last updated:** [January 15, 2019, 1:39pm UTC](https://discuss.elastic.co/t/logstash-list-of-codes-to-replace/163767 "2019-01-15T13:39:52Z")
Hello, I want to match my data with a set of codes. I tried to do that using translate plugin, having a file with those matches. But, it seems a bad option because I have various matches with lead me to have many auxili… --- ## [Aggregate filter plugin timeout weird behavior](https://discuss.elastic.co/t/aggregate-filter-plugin-timeout-weird-behavior/161847)
**Author:** [@Mickael\_Humphreys](https://discuss.elastic.co/u/Mickael_Humphreys)\ **Replies:** 2\ **Last updated:** [January 15, 2019, 12:14pm UTC](https://discuss.elastic.co/t/aggregate-filter-plugin-timeout-weird-behavior/161847 "2019-01-15T12:14:21Z")
Hello, I've tried to configure Aggregate filter plugin Timeout parametre above 10s. The problem is Logstash pipeline closed before this delay. It seems like while Logstash do not have any event send to output section … --- ## [Logstash translate filter throws error if dictionary file is empty](https://discuss.elastic.co/t/logstash-translate-filter-throws-error-if-dictionary-file-is-empty/164101)
**Author:** [@Riddhesh](https://discuss.elastic.co/u/Riddhesh)\ **Replies:** 0\ **Last updated:** [January 14, 2019, 7:45am UTC](https://discuss.elastic.co/t/logstash-translate-filter-throws-error-if-dictionary-file-is-empty/164101 "2019-01-14T07:45:05Z")
When using translate filter with yml file, if I have the file empty, logstash fails to start and throws error: Pipeline aborted due to error {:pipeline\_id=\>"main", :exception=\>#\ **Author:** [@fribse](https://discuss.elastic.co/u/fribse)\ **Replies:** 5\ **Last updated:** [January 15, 2019, 10:37am UTC](https://discuss.elastic.co/t/can-someone-help-with-the-grok-i-cant-figure-out-how-to-make-it-work/163905 "2019-01-15T10:37:04Z") I have this type of logfile: Events of MHSendMessagesService \* STOP \* 0320 \*\*\* START \*\*\* \* Module: C:\\Program Files (x86)\\PST\\M… --- ## [Logstash slow preprocessing of data](https://discuss.elastic.co/t/logstash-slow-preprocessing-of-data/164283)
**Author:** [@josephmanalo](https://discuss.elastic.co/u/josephmanalo)\ **Replies:** 0\ **Last updated:** [January 15, 2019, 9:52am UTC](https://discuss.elastic.co/t/logstash-slow-preprocessing-of-data/164283 "2019-01-15T09:52:29Z")
Hi. This is the result of my logstash statistics (curl localhost:9600/\_node/stats/events) I think it is running slow. It takes 1 doc/28 sec to process it. Here are some details: ~ Aprrox. 30M of data a day ~ Machine … --- ## [Logstash monitor file name](https://discuss.elastic.co/t/logstash-monitor-file-name/164282)
**Author:** [@fadihaddad](https://discuss.elastic.co/u/fadihaddad)\ **Replies:** 0\ **Last updated:** [January 15, 2019, 9:43am UTC](https://discuss.elastic.co/t/logstash-monitor-file-name/164282 "2019-01-15T09:43:56Z")
I have logstash as a service and I receive files from time to time to index to elasticsearch. Is there a way to see the name of the file being indexed on the monitor so I can confirm the files indexed to elastic --- ## [CSV: autodetect\_column\_names vs. autogenerate\_column\_names](https://discuss.elastic.co/t/csv-autodetect-column-names-vs-autogenerate-column-names/164182)
**Author:** [@Aralex](https://discuss.elastic.co/u/Aralex)\ **Replies:** 2\ **Last updated:** [January 15, 2019, 9:13am UTC](https://discuss.elastic.co/t/csv-autodetect-column-names-vs-autogenerate-column-names/164182 "2019-01-15T09:13:03Z")
Hi, What are the exact functional differences between autodetect\_column\_names and autogenerate\_column\_names in the csv filter? Is it bad/good using both (set to true)? IS there a recommended order for using both… --- ## [Convert utc format to local time](https://discuss.elastic.co/t/convert-utc-format-to-local-time/164237)
**Author:** [@wangqinghuan](https://discuss.elastic.co/u/wangqinghuan)\ **Replies:** 4\ **Last updated:** [January 15, 2019, 8:21am UTC](https://discuss.elastic.co/t/convert-utc-format-to-local-time/164237 "2019-01-15T08:21:09Z")
hi I am using logstash to output data from elasticsearch to file. A field named 'snapTime' in elasticsearch is represented by utc format '2018-09-10T15:05:43.000Z' and logstash outputs this field in utc format to file … --- ## [Logstash JMS Input Plugin - Oracle AQ buffered messages support](https://discuss.elastic.co/t/logstash-jms-input-plugin-oracle-aq-buffered-messages-support/164256)
**Author:** [@i-kovalyov](https://discuss.elastic.co/u/i-kovalyov)\ **Replies:** 0\ **Last updated:** [January 15, 2019, 7:47am UTC](https://discuss.elastic.co/t/logstash-jms-input-plugin-oracle-aq-buffered-messages-support/164256 "2019-01-15T07:47:30Z")
Hello. I want to support Oracle AQ buffered messages in jms input plugin and discuss here implementation details. What is appropriate way for this enhancement in configuration options? Maybe vendor\_options (or somethi… --- ## [Reduce logstash cloudwatch input logging info](https://discuss.elastic.co/t/reduce-logstash-cloudwatch-input-logging-info/164252)
**Author:** [@Izek](https://discuss.elastic.co/u/Izek)\ **Replies:** 0\ **Last updated:** [January 15, 2019, 7:25am UTC](https://discuss.elastic.co/t/reduce-logstash-cloudwatch-input-logging-info/164252 "2019-01-15T07:25:27Z")
When I use logstash-input-cloudwatch plugin, it will log info level information and for each query. It spam the log file. Can we enhance this plugin to reduce the size or setup different log value. \[2019-01-15T15:19:4… --- ## [Logstash does not start process files](https://discuss.elastic.co/t/logstash-does-not-start-process-files/164228)
**Author:** [@jamespb](https://discuss.elastic.co/u/jamespb)\ **Replies:** 0\ **Last updated:** [January 15, 2019, 3:20am UTC](https://discuss.elastic.co/t/logstash-does-not-start-process-files/164228 "2019-01-15T03:20:02Z")
when logstash start, it give out Rotation In Progress error. I just want it start process files \[2019-01-15T11:17:09,673\]\[WARN \]\[filewatch.tailmode.processor\] \>\>\> Rotation In Progress - inode change detected and origina… --- ## [Logstash Date Filter Not Recognizing the pattern](https://discuss.elastic.co/t/logstash-date-filter-not-recognizing-the-pattern/164221)
**Author:** [@dlimantoro](https://discuss.elastic.co/u/dlimantoro)\ **Replies:** 1\ **Last updated:** [January 15, 2019, 2:13am UTC](https://discuss.elastic.co/t/logstash-date-filter-not-recognizing-the-pattern/164221 "2019-01-15T02:13:16Z")
Hello there, I have a log file in CSV format (but semicolon separated instead of comma) that I generated from PowerShell, where one of the fields contain a date. Using grok, I managed to pull the date and store them int… --- ## [Logstash error when running .conf](https://discuss.elastic.co/t/logstash-error-when-running-conf/164079)
**Author:** [@OpSecMonkey](https://discuss.elastic.co/u/OpSecMonkey)\ **Replies:** 5\ **Last updated:** [January 14, 2019, 11:10pm UTC](https://discuss.elastic.co/t/logstash-error-when-running-conf/164079 "2019-01-14T23:10:00Z")
I get this error when running in centos 7 \] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of #, { at line 15, c… --- ## [Multiple matches in same line using grok](https://discuss.elastic.co/t/multiple-matches-in-same-line-using-grok/163637)
**Author:** [@ellje](https://discuss.elastic.co/u/ellje)\ **Replies:** 5\ **Last updated:** [January 14, 2019, 10:35pm UTC](https://discuss.elastic.co/t/multiple-matches-in-same-line-using-grok/163637 "2019-01-14T22:35:51Z")
Hey guys, new-ish to Elastic and was hoping to get help for the following question. To start off, I want to use logstash to parse a log file using grok statements. Say I have an input as follows: 'Test failed with the… --- ## [Logstash cannot extract json key](https://discuss.elastic.co/t/logstash-cannot-extract-json-key/162753)
**Author:** [@t.b](https://discuss.elastic.co/u/t.b)\ **Replies:** 3\ **Last updated:** [January 14, 2019, 8:35pm UTC](https://discuss.elastic.co/t/logstash-cannot-extract-json-key/162753 "2019-01-14T20:35:45Z")
Hi guys, I need help regarding logstash filter to extract json key/value to new\_field. The following is my logstash conf. input { tcp { port =\> 5044 } } filter { json { source =\> "message" add\_field =\> { "d… --- ## [Could not index event to Elasticsearch action=update](https://discuss.elastic.co/t/could-not-index-event-to-elasticsearch-action-update/164187)
**Author:** [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Replies:** 1\ **Last updated:** [January 14, 2019, 7:41pm UTC](https://discuss.elastic.co/t/could-not-index-event-to-elasticsearch-action-update/164187 "2019-01-14T19:41:39Z")
2019-01-14T12:42:10,098\]\[WARN \]\[logstash.outputs.elasticsearch\] Could not index event to Elasticsearch. {:status=\>404, :action=\>\["update", {:\_id=\>"417050708", :\_index=\>"houston\_jobs", :\_type=\>"doc", :routing=\>nil, :\_retr… --- ## [Logstash-elasticsearch input plugin, size not working as expeted](https://discuss.elastic.co/t/logstash-elasticsearch-input-plugin-size-not-working-as-expeted/163344)
**Author:** [@gauravsinghal3](https://discuss.elastic.co/u/gauravsinghal3)\ **Replies:** 1\ **Last updated:** [January 14, 2019, 6:46pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-input-plugin-size-not-working-as-expeted/163344 "2019-01-14T18:46:09Z")
Problem: Every 20 minutes logstash fetches all the records matching as per query in batches of 100(size=100) and waits for next schedule. in input plugin i have something like this: elasticsearch{ size =\> 100 schedul… --- ## [Is it possible to output to an elasticsearch instance via reverse proxy uri?](https://discuss.elastic.co/t/is-it-possible-to-output-to-an-elasticsearch-instance-via-reverse-proxy-uri/164180)
**Author:** [@Matt\_Vasquez](https://discuss.elastic.co/u/Matt_Vasquez)\ **Replies:** 0\ **Last updated:** [January 14, 2019, 5:38pm UTC](https://discuss.elastic.co/t/is-it-possible-to-output-to-an-elasticsearch-instance-via-reverse-proxy-uri/164180 "2019-01-14T17:38:58Z")
I am using Apache to reverse proxy to Elasticsearch via a uri location (https://reverse\_proxy/elastic). I am using a reverse proxy for external users due to firewall constraints which block 9200 but allow 443.. Here is … --- ## [Can I use replace to add a field?](https://discuss.elastic.co/t/can-i-use-replace-to-add-a-field/163625)
**Author:** [@Peter\_S](https://discuss.elastic.co/u/Peter_S)\ **Replies:** 3\ **Last updated:** [January 14, 2019, 4:53pm UTC](https://discuss.elastic.co/t/can-i-use-replace-to-add-a-field/163625 "2019-01-14T16:53:33Z")
Hello, I'm trying to write a mutation to describe a service name based on the port given. Suricata is feeding the data to logstash, and it currently doesn't have a very good way of describing services unless it's in the… --- ## [Trying to get Logstash to connect to a secured cluster. Where is the easiest connection to make?](https://discuss.elastic.co/t/trying-to-get-logstash-to-connect-to-a-secured-cluster-where-is-the-easiest-connection-to-make/164175)
**Author:** [@bigdamhero](https://discuss.elastic.co/u/bigdamhero)\ **Replies:** 0\ **Last updated:** [January 14, 2019, 4:30pm UTC](https://discuss.elastic.co/t/trying-to-get-logstash-to-connect-to-a-secured-cluster-where-is-the-easiest-connection-to-make/164175 "2019-01-14T16:30:07Z")
I was thinking that it might be easier to connect the monitoring to my secure cluster. I have not had any success, I am wondering if there needs to be an active pipline which is shipping logs to the same cluster or can … --- ## [Jdbc input - how to trigger immediately after pipeline is started / reloaded?](https://discuss.elastic.co/t/jdbc-input-how-to-trigger-immediately-after-pipeline-is-started-reloaded/163867)
**Author:** [@asp](https://discuss.elastic.co/u/asp)\ **Replies:** 3\ **Last updated:** [January 14, 2019, 3:00pm UTC](https://discuss.elastic.co/t/jdbc-input-how-to-trigger-immediately-after-pipeline-is-started-reloaded/163867 "2019-01-14T15:00:13Z")
Hi, I am currently inplementing a logstash pipeline which is using jdbc input plugin. Is there any way that logstash starts the jdbc input query immediately after the pipeline has been started or reloaded? When I do a… --- ## [Ruby filter for parsing deeply nested JSON in pipeline](https://discuss.elastic.co/t/ruby-filter-for-parsing-deeply-nested-json-in-pipeline/161050)
**Author:** [@Chris\_Lyons](https://discuss.elastic.co/u/Chris_Lyons)\ **Replies:** 5\ **Last updated:** [January 14, 2019, 2:30pm UTC](https://discuss.elastic.co/t/ruby-filter-for-parsing-deeply-nested-json-in-pipeline/161050 "2019-01-14T14:30:27Z")
We use Logstash extensively within my team as our ETL engine. Things started out simple enough with log file data, but we are at a point where our primary use case is centered around Business Intelligence. All our Inci… --- ## [LOGSTASH NOT EXECUTE SQL STORE PROCEDURE](https://discuss.elastic.co/t/logstash-not-execute-sql-store-procedure/164135)
**Author:** [@aidewuado](https://discuss.elastic.co/u/aidewuado)\ **Replies:** 0\ **Last updated:** [January 14, 2019, 12:00pm UTC](https://discuss.elastic.co/t/logstash-not-execute-sql-store-procedure/164135 "2019-01-14T12:00:55Z")
hello im try to execute store procedure in logstash and not work, im view de 1\[https://discuss.elastic.co/t/logstash-cannot-call-sql-stored-procedure/44271\] but not ecute my sp. that is my code: errorlog.conf: input { … --- ## [Stuck (~100 % CPU) pipeline workers](https://discuss.elastic.co/t/stuck-100-cpu-pipeline-workers/163650)
**Author:** [@maxh1](https://discuss.elastic.co/u/maxh1)\ **Replies:** 7\ **Last updated:** [January 14, 2019, 11:52am UTC](https://discuss.elastic.co/t/stuck-100-cpu-pipeline-workers/163650 "2019-01-14T11:52:43Z")
Hi! We suddenly noticed that we were no longer receiving events in Kibana/Elasticsearch, and upon further investigation we saw that the server running Logstash was using 100 % CPU. We've tried restarting both Logstash,… --- ## [Is there a solution available for issue mentioned in "153152"?](https://discuss.elastic.co/t/is-there-a-solution-available-for-issue-mentioned-in-153152/164098)
**Author:** [@girishpaik](https://discuss.elastic.co/u/girishpaik)\ **Replies:** 3\ **Last updated:** [January 14, 2019, 10:44am UTC](https://discuss.elastic.co/t/is-there-a-solution-available-for-issue-mentioned-in-153152/164098 "2019-01-14T10:44:44Z")
Hi Team, Issue mentioned in below discussion is solved? could you let me know version? I am still seeing this issue in latest GA'ed version, i.e., 6.5.4? Thanks in advance GPK --- ## [Send reports and logs from all devices to mcafee siem](https://discuss.elastic.co/t/send-reports-and-logs-from-all-devices-to-mcafee-siem/164090)
**Author:** [@mp28may](https://discuss.elastic.co/u/mp28may)\ **Replies:** 5\ **Last updated:** [January 14, 2019, 9:33am UTC](https://discuss.elastic.co/t/send-reports-and-logs-from-all-devices-to-mcafee-siem/164090 "2019-01-14T09:33:02Z")
HI Team, I am new to elkstack. I have configured ELKstack in my environment and it is working perfectly. I am able to get the reports and syslogs from all the devices on KIBANA interface. I also have Mcafee siem solu… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=617) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=619)