# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=593 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 594 --- ## [Logstash - No Cached mapping](https://discuss.elastic.co/t/logstash-no-cached-mapping/170060)
**Author:** [@mroell](https://discuss.elastic.co/u/mroell)\ **Replies:** 4\ **Last updated:** [February 26, 2019, 7:48pm UTC](https://discuss.elastic.co/t/logstash-no-cached-mapping/170060 "2019-02-26T19:48:26Z")
I'm using winlogbeat to ship event logs, and using a simple logstash configuration. It's not my configuration. https://www.syspanda.com/index.php/2018/05/03/monitoring-active-directory-elk/ From what I can find it's so… --- ## [Extract domains from a string of email address](https://discuss.elastic.co/t/extract-domains-from-a-string-of-email-address/170055)
**Author:** [@austin0918](https://discuss.elastic.co/u/austin0918)\ **Replies:** 2\ **Last updated:** [February 26, 2019, 7:37pm UTC](https://discuss.elastic.co/t/extract-domains-from-a-string-of-email-address/170055 "2019-02-26T19:37:40Z")
I have a field "to\_domains\_temp" (copied from email recipients list, amount of email addresses is unpredictable), let's say "akii.suzuki@gmail.com;machael.miki@hotmail.com;nobuo.miki@outlook.com;wakako.seki@gmail.com" h… --- ## [Logstash pipeline metric events not output under load](https://discuss.elastic.co/t/logstash-pipeline-metric-events-not-output-under-load/167451)
**Author:** [@mario\_wood](https://discuss.elastic.co/u/mario_wood)\ **Replies:** 2\ **Last updated:** [February 26, 2019, 5:49pm UTC](https://discuss.elastic.co/t/logstash-pipeline-metric-events-not-output-under-load/167451 "2019-02-26T17:49:10Z")
I have a pipeline configured with logstash 6.3.0 which should capture metrics using the metrics filter for the number of events then output them to statsd. My pipeline config is as follows (i've removed some bits of comp… --- ## [Logstash - Filebeat - Input not listening](https://discuss.elastic.co/t/logstash-filebeat-input-not-listening/169761)
**Author:** [@Kanthasamyraja](https://discuss.elastic.co/u/Kanthasamyraja)\ **Replies:** 5\ **Last updated:** [February 26, 2019, 5:39pm UTC](https://discuss.elastic.co/t/logstash-filebeat-input-not-listening/169761 "2019-02-26T17:39:50Z")
Hi, I am using 6.5.4 version of elastic stack. (Filebeat - Logstash - Elasticsearch - Kibana). filebeat was working fine. Enabled x-pack in ES and other related settings. ELK - Able to start the services. But Logstas… --- ## [Logstash - No mapping found for \[@timestamp\] in order to sort on](https://discuss.elastic.co/t/logstash-no-mapping-found-for-timestamp-in-order-to-sort-on/170021)
**Author:** [@Kanthasamyraja](https://discuss.elastic.co/u/Kanthasamyraja)\ **Replies:** 8\ **Last updated:** [February 26, 2019, 4:45pm UTC](https://discuss.elastic.co/t/logstash-no-mapping-found-for-timestamp-in-order-to-sort-on/170021 "2019-02-26T16:45:46Z")
Hi, I am using 6.5.4 version Logstash keeps printing the below line almost every second. Failed to query elasticsearch for previous event {:index=\>"", :query=\>"", :event=\>#LogStash::Event:0x535f580d, :error=\>"#\ **Author:** [@johann](https://discuss.elastic.co/u/johann)\ **Replies:** 6\ **Last updated:** [February 26, 2019, 4:08pm UTC](https://discuss.elastic.co/t/apply-index-if-available/170023 "2019-02-26T16:08:38Z") Hi, if I have an beats input from journalbeat to logstash, how can I use the index which is already set? The beats input is coming from journalbeat -\> logstash -\> elasticsearch. Journalbeat output: output.logstash wit… --- ## [Translate filter plugin](https://discuss.elastic.co/t/translate-filter-plugin/167971)
**Author:** [@Michael.Swiss](https://discuss.elastic.co/u/Michael.Swiss)\ **Replies:** 5\ **Last updated:** [February 26, 2019, 4:55pm UTC](https://discuss.elastic.co/t/translate-filter-plugin/167971 "2019-02-26T16:55:54Z")
Dear colleagues I am using many translate filter plugins to mutate data in logstash. The filter requires me to create a dictionary file (.yaml) that contains 2 columns. That way I need lots of .yaml files. For me it wo… --- ## [XML parsing- Filter to format a date field](https://discuss.elastic.co/t/xml-parsing-filter-to-format-a-date-field/170036)
**Author:** [@Skwilly](https://discuss.elastic.co/u/Skwilly)\ **Replies:** 4\ **Last updated:** [February 26, 2019, 4:30pm UTC](https://discuss.elastic.co/t/xml-parsing-filter-to-format-a-date-field/170036 "2019-02-26T16:30:07Z")
Hi, I have the following index: "hits" : \[ { "\_index" : "test", "\_type" : "doc", "\_id" : "OWZhKmkBDjX7TOqqZgwX", "\_score" : 1.0, "\_source" : { "@timestamp" : "2019-02-26T15:16:52.527Z" … --- ## [Set index according to input](https://discuss.elastic.co/t/set-index-according-to-input/170020)
**Author:** [@xduh](https://discuss.elastic.co/u/xduh)\ **Replies:** 4\ **Last updated:** [February 26, 2019, 4:21pm UTC](https://discuss.elastic.co/t/set-index-according-to-input/170020 "2019-02-26T16:21:19Z")
Hi, I'm trying to set the index according to the id of sqs input. Tried something like this but without success. input { sqs { id =\> "product queue =\> "live-product add\_f… --- ## [TLS/SSL for logstash 5.6](https://discuss.elastic.co/t/tls-ssl-for-logstash-5-6/168783)
**Author:** [@prabhakar\_talari](https://discuss.elastic.co/u/prabhakar_talari)\ **Replies:** 9\ **Last updated:** [February 26, 2019, 3:45pm UTC](https://discuss.elastic.co/t/tls-ssl-for-logstash-5-6/168783 "2019-02-26T15:45:36Z")
All, We have tried by adding TLS CA certificate which we already generated for ES cluster. When we add the parameters in logstash configuration file as per below doc https://www.elastic.co/guide/en/x-pack/5.6/logstash… --- ## [Splitting JSON arrays](https://discuss.elastic.co/t/splitting-json-arrays/168858)
**Author:** [@Skwilly](https://discuss.elastic.co/u/Skwilly)\ **Replies:** 2\ **Last updated:** [February 26, 2019, 2:26pm UTC](https://discuss.elastic.co/t/splitting-json-arrays/168858 "2019-02-26T14:26:46Z")
Hi, I have indexed a json file without any filter and queried the corresponding index to first explore the structure before splitting it in an appropriate way. I found the field of interest with the following query and… --- ## [Unable to create index in ElasticSearch from Filebeat - \> Logstash](https://discuss.elastic.co/t/unable-to-create-index-in-elasticsearch-from-filebeat-logstash/169983)
**Author:** [@Abinaya](https://discuss.elastic.co/u/Abinaya)\ **Replies:** 1\ **Last updated:** [February 26, 2019, 1:07pm UTC](https://discuss.elastic.co/t/unable-to-create-index-in-elasticsearch-from-filebeat-logstash/169983 "2019-02-26T13:07:21Z")
Hi, I am trying to read a sample apache log file from file beat and push it to Logstash. I am not seeing any error in debug mode. Index not being created in ES. Logstash.conf file input { beats { port =\> "5044" } … --- ## [Parsing Wazuh archives.json file with Logstash](https://discuss.elastic.co/t/parsing-wazuh-archives-json-file-with-logstash/169935)
**Author:** [@HaranKumar](https://discuss.elastic.co/u/HaranKumar)\ **Replies:** 2\ **Last updated:** [February 26, 2019, 12:43pm UTC](https://discuss.elastic.co/t/parsing-wazuh-archives-json-file-with-logstash/169935 "2019-02-26T12:43:58Z")
Hello Team, Might be json codec would have been discussed in the past , i tried googling and did not find relevant solution . so posting my query to get direct assistance from experts. I have a file alerts.json and it … --- ## [How to use Dissect to handle a newline](https://discuss.elastic.co/t/how-to-use-dissect-to-handle-a-newline/169977)
**Author:** [@akira](https://discuss.elastic.co/u/akira)\ **Replies:** 2\ **Last updated:** [February 26, 2019, 12:36pm UTC](https://discuss.elastic.co/t/how-to-use-dissect-to-handle-a-newline/169977 "2019-02-26T12:36:13Z")
I Just want split a message into two lines, it is a simple case, but i didn't get the result (according to: https://www.elastic.co/guide/en/logstash/5.6/plugins-filters-dissect.html#plugins-filters-dissect-mapping) a… --- ## [Throttle key?](https://discuss.elastic.co/t/throttle-key/169864)
**Author:** [@oscario](https://discuss.elastic.co/u/oscario)\ **Replies:** 3\ **Last updated:** [February 26, 2019, 12:11pm UTC](https://discuss.elastic.co/t/throttle-key/169864 "2019-02-26T12:11:20Z")
Dear all, I want the throttle plugin to group every line (not only lines matching a specific key value). Since a key is required and the only thing all lines have in common is at least one space I use a space for the k… --- ## [Pre-process message backslashes prior to JSON filter](https://discuss.elastic.co/t/pre-process-message-backslashes-prior-to-json-filter/169821)
**Author:** [@NathanBaulch](https://discuss.elastic.co/u/NathanBaulch)\ **Replies:** 3\ **Last updated:** [February 26, 2019, 1:30am UTC](https://discuss.elastic.co/t/pre-process-message-backslashes-prior-to-json-filter/169821 "2019-02-26T01:30:32Z")
I'm trying to replace the literal string "\\x" with "\\u00" just before parsing JSON to (somewhat naively) convert invalid hex escapes (coming from Nginx) into Unicode escapes. However no matter what I try I always get Log… --- ## [Regarding SMTP](https://discuss.elastic.co/t/regarding-smtp/169339)
**Author:** [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\ **Replies:** 4\ **Last updated:** [February 26, 2019, 11:51am UTC](https://discuss.elastic.co/t/regarding-smtp/169339 "2019-02-26T11:51:01Z")
can I get some help in writing grok for smtp logs? --- ## [Logstash jdbc input checkpoint](https://discuss.elastic.co/t/logstash-jdbc-input-checkpoint/168837)
**Author:** [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Replies:** 7\ **Last updated:** [February 26, 2019, 10:06am UTC](https://discuss.elastic.co/t/logstash-jdbc-input-checkpoint/168837 "2019-02-26T10:06:51Z")
hi all, I am using logstash to select data from a database and index them. my logstash configuration is as following: input { jdbc { jdbc\_driver\_library =\> "/home/srahimi/ojdbc6.jar" jdbc\_driver\_class =\> "Java::orac… --- ## [Maximum length of the CSV column parameter](https://discuss.elastic.co/t/maximum-length-of-the-csv-column-parameter/169961)
**Author:** [@CCH0124](https://discuss.elastic.co/u/CCH0124)\ **Replies:** 0\ **Last updated:** [February 26, 2019, 9:30am UTC](https://discuss.elastic.co/t/maximum-length-of-the-csv-column-parameter/169961 "2019-02-26T09:30:18Z")
I have 18 headers when I set columns, but the index for kibana only shows 13 why? My settings： filter { csv { separator =\> "," #autogenerate\_column\_names=\> true c… --- ## [Replace @timestamp with local machine timestamp](https://discuss.elastic.co/t/replace-timestamp-with-local-machine-timestamp/169948)
**Author:** [@Nikhil\_Jaiswal](https://discuss.elastic.co/u/Nikhil_Jaiswal)\ **Replies:** 1\ **Last updated:** [February 26, 2019, 8:17am UTC](https://discuss.elastic.co/t/replace-timestamp-with-local-machine-timestamp/169948 "2019-02-26T08:17:03Z")
Hello Folks, I am trying to replace @timestamp with local machine timestamp but no luck. Could anybody help me on the same. Thanks! --- ## [Twitter developer account application failed](https://discuss.elastic.co/t/twitter-developer-account-application-failed/169693)
**Author:** [@li\_jessen2016](https://discuss.elastic.co/u/li_jessen2016)\ **Replies:** 1\ **Last updated:** [February 26, 2019, 6:33am UTC](https://discuss.elastic.co/t/twitter-developer-account-application-failed/169693 "2019-02-26T06:33:16Z")
My application for Twitter developer account failed. Then how can I use Twitter feed as input of logstash? --- ## [Audit log](https://discuss.elastic.co/t/audit-log/167976)
**Author:** [@nandha\_88](https://discuss.elastic.co/u/nandha_88)\ **Replies:** 9\ **Last updated:** [February 26, 2019, 6:10am UTC](https://discuss.elastic.co/t/audit-log/167976 "2019-02-26T06:10:22Z")
Hi All, Please provide us a better solution. Our setup is below and right now we need to monitor user activities in the linux server. Filebeat -\> logstash -\> eS -\> kibana Please share whether we can use the same fi… --- ## [Invalid Frame error](https://discuss.elastic.co/t/invalid-frame-error/169813)
**Author:** [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\ **Replies:** 2\ **Last updated:** [February 26, 2019, 5:15am UTC](https://discuss.elastic.co/t/invalid-frame-error/169813 "2019-02-26T05:15:53Z")
Whenever I am trying to import logs using filebeat on the same server (logstash) I am getting error as: \[INFO \] 2019-02-25 16:59:35.337 \[defaultEventExecutorGroup-4-1\] BeatsHandler - \[local: 10.11.109.7:5044, remote: 10… --- ## [How can I import json data to elasticsearch?](https://discuss.elastic.co/t/how-can-i-import-json-data-to-elasticsearch/169207)
**Author:** [@YK\_Lee](https://discuss.elastic.co/u/YK_Lee)\ **Replies:** 2\ **Last updated:** [February 26, 2019, 3:34am UTC](https://discuss.elastic.co/t/how-can-i-import-json-data-to-elasticsearch/169207 "2019-02-26T03:34:08Z")
I wrote config file and excuted that but, I can't find the result in kibana(elasticsearch) I don't know what is wrong. help me ~ I need your help. this is my config file input { file { path =\> "D:\\2… --- ## [Logstash Input Plugins remote code execution code execution concerns](https://discuss.elastic.co/t/logstash-input-plugins-remote-code-execution-code-execution-concerns/166143)
**Author:** [@siddharthgoel88](https://discuss.elastic.co/u/siddharthgoel88)\ **Replies:** 1\ **Last updated:** [February 26, 2019, 3:19am UTC](https://discuss.elastic.co/t/logstash-input-plugins-remote-code-execution-code-execution-concerns/166143 "2019-02-26T03:19:50Z")
We are currently using Elastic Stack 6.2.4 (Elasticsearch, Logstash, Filebeat). In our setup we have two servers - Server A : This one has our publicly exposed applications for which we need to do log management. File… --- ## [Exception in pipeline worker and Java Out of Memory Errors](https://discuss.elastic.co/t/exception-in-pipeline-worker-and-java-out-of-memory-errors/169916)
**Author:** [@deepaknadig](https://discuss.elastic.co/u/deepaknadig)\ **Replies:** 1\ **Last updated:** [February 25, 2019, 11:52pm UTC](https://discuss.elastic.co/t/exception-in-pipeline-worker-and-java-out-of-memory-errors/169916 "2019-02-25T23:52:49Z")
Hello, I'm runnung ELK 6.6 and I'm trying to index about 2,000 json files and I'm getting the following error: java.lang.OutOfMemoryError: Java heap space Dumping heap to java\_pid6685.hprof ... Heap dump file created \[… --- ## [Logstash Pipeline to Pipeline issue and looking for more info](https://discuss.elastic.co/t/logstash-pipeline-to-pipeline-issue-and-looking-for-more-info/169888)
**Author:** [@analog\_memories](https://discuss.elastic.co/u/analog_memories)\ **Replies:** 2\ **Last updated:** [February 25, 2019, 10:24pm UTC](https://discuss.elastic.co/t/logstash-pipeline-to-pipeline-issue-and-looking-for-more-info/169888 "2019-02-25T22:24:59Z")
Hello, I have been looking for more in depth information on the Pipeline to Pipeline functionality in Logstash, specifically version 6.4.3. I know it is still in beta, but I have been trying to get it to work in a dev … --- ## [Dynamic field name with ruby](https://discuss.elastic.co/t/dynamic-field-name-with-ruby/169899)
**Author:** [@camilisette](https://discuss.elastic.co/u/camilisette)\ **Replies:** 7\ **Last updated:** [February 25, 2019, 10:06pm UTC](https://discuss.elastic.co/t/dynamic-field-name-with-ruby/169899 "2019-02-25T22:06:27Z")
I have a curl response that looks like this. { "tables": \[ { "name": "PrimaryResult", "columns": \[ { "name": "Category", "type": "string" }, { "name": "count\_", "type": "long" } \], "rows": \[ \[ "Administ… --- ## [Http filter - undefined local variable or method \`parameters\_sprintfed'](https://discuss.elastic.co/t/http-filter-undefined-local-variable-or-method-parameters-sprintfed/169905)
**Author:** [@DefensiveDepth](https://discuss.elastic.co/u/DefensiveDepth)\ **Replies:** 2\ **Last updated:** [February 25, 2019, 9:18pm UTC](https://discuss.elastic.co/t/http-filter-undefined-local-variable-or-method-parameters-sprintfed/169905 "2019-02-25T21:18:20Z")
I have a very simple pipeline using the following http filter, but I continue to get errors (see here: https://gist.github.com/defensivedepth/7b8d22aeec6dedac6438d8adeaed4b1d) I have tried different sites, different set… --- ## [Capturing Logstash version as a field?](https://discuss.elastic.co/t/capturing-logstash-version-as-a-field/169895)
**Author:** [@londonx](https://discuss.elastic.co/u/londonx)\ **Replies:** 0\ **Last updated:** [February 25, 2019, 7:22pm UTC](https://discuss.elastic.co/t/capturing-logstash-version-as-a-field/169895 "2019-02-25T19:22:22Z")
I notice on startup the Logstash version is reported, however I don't see this logged in a field like you see with Elastic beats. Is this some sort of environment variable that I can capture into a field instead? Logsta… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=592) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=594)