# Elastic Stack **URL:** https://discuss.elastic.co/c/elastic-stack/81.md?page=1280 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1281 --- ## [Match on Ngram field where the words are in order without scripting](https://discuss.elastic.co/t/match-on-ngram-field-where-the-words-are-in-order-without-scripting/281518)
**Author:** [@Ron\_lusha](https://discuss.elastic.co/u/Ron_lusha)\ **Replies:** 0\ **Last updated:** [August 16, 2021, 8:37am UTC](https://discuss.elastic.co/t/match-on-ngram-field-where-the-words-are-in-order-without-scripting/281518 "2021-08-16T08:37:37Z")
Hello, I have a question regarding auto-complete using Ngram search, Im using elasticsearch 7.3, so the new "wildcard" field type is not relevant for me This is my index: PUT ngram\_test { "mappings": { "propertie… --- ## [Trigger In elastic search](https://discuss.elastic.co/t/trigger-in-elastic-search/281458)
**Author:** [@ashurajchandra](https://discuss.elastic.co/u/ashurajchandra)\ **Replies:** 20\ **Last updated:** [August 16, 2021, 10:41am UTC](https://discuss.elastic.co/t/trigger-in-elastic-search/281458 "2021-08-16T10:41:38Z")
I'm working in startup and I got the task of writing trigger for error logs , but not when i'm using PUT \_watcher/watch/diagnostic\_monitor { "metadata" : { "color" : "red" }, "trigger" : { "schedule" : {… --- ## [Trigger an alert when Logstash pipeline gets less events than defined in threshold](https://discuss.elastic.co/t/trigger-an-alert-when-logstash-pipeline-gets-less-events-than-defined-in-threshold/281519)
**Author:** [@nemanja](https://discuss.elastic.co/u/nemanja)\ **Replies:** 0\ **Last updated:** [August 16, 2021, 8:49am UTC](https://discuss.elastic.co/t/trigger-an-alert-when-logstash-pipeline-gets-less-events-than-defined-in-threshold/281519 "2021-08-16T08:49:43Z")
Hello, Can we setup Logstash pipeline monitoring, that would for example trigger an alert when the number of events per second drops bellow certain threshold? in Kibana Pipeline Viewer there is nice visual overview on c… --- ## [Nested JSON flattened in Logstash Filter](https://discuss.elastic.co/t/nested-json-flattened-in-logstash-filter/281502)
**Author:** [@Divyank\_Mahalle](https://discuss.elastic.co/u/Divyank_Mahalle)\ **Replies:** 1\ **Last updated:** [August 16, 2021, 8:11am UTC](https://discuss.elastic.co/t/nested-json-flattened-in-logstash-filter/281502 "2021-08-16T08:11:47Z")
Hi, I am able to flatten at fields present in nested JSON excluding 3 fields that are present in double nested JSON. I need to flatten these fields so that I can create viz based on that such as Fields-"quantity", "rate… --- ## [Scripted fields - doc\[message.keyword\].value reading as empty - field is not empty](https://discuss.elastic.co/t/scripted-fields-doc-message-keyword-value-reading-as-empty-field-is-not-empty/280953)
**Author:** [@acastravet](https://discuss.elastic.co/u/acastravet)\ **Replies:** 15\ **Last updated:** [August 16, 2021, 7:05am UTC](https://discuss.elastic.co/t/scripted-fields-doc-message-keyword-value-reading-as-empty-field-is-not-empty/280953 "2021-08-16T07:05:48Z")
Hello everyone, I have a field that holds all the log information (multiple fields inside it). Field looks like this: logging.level="some info", @timestamp="some info", message="some info", trace\_id="some info" I am t… --- ## [How to Provide mapping to the API before injesting data to the elasticsearch using logstash conf file](https://discuss.elastic.co/t/how-to-provide-mapping-to-the-api-before-injesting-data-to-the-elasticsearch-using-logstash-conf-file/281061)
**Author:** [@Divyank\_Mahalle](https://discuss.elastic.co/u/Divyank_Mahalle)\ **Replies:** 1\ **Last updated:** [August 16, 2021, 6:58am UTC](https://discuss.elastic.co/t/how-to-provide-mapping-to-the-api-before-injesting-data-to-the-elasticsearch-using-logstash-conf-file/281061 "2021-08-16T06:58:45Z")
Hi, I am using Twilio API for ingesting data to elasticsearch. It is providing Output in the form of JSON. curl -G https://api.twilio.com/2010-04-01/Accounts/ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/Usage/Records/LastMonth.… --- ## [How configuration logstash 7.6.2 email in output](https://discuss.elastic.co/t/how-configuration-logstash-7-6-2-email-in-output/281148)
**Author:** [@arisxf](https://discuss.elastic.co/u/arisxf)\ **Replies:** 5\ **Last updated:** [August 16, 2021, 6:47am UTC](https://discuss.elastic.co/t/how-configuration-logstash-7-6-2-email-in-output/281148 "2021-08-16T06:47:28Z")
how to use logstash 7.6.2 email in output ? is this email plugin available by default? --- ## [Duplicate a document on elasticsearch](https://discuss.elastic.co/t/duplicate-a-document-on-elasticsearch/281504)
**Author:** [@Sandeep\_Raju](https://discuss.elastic.co/u/Sandeep_Raju)\ **Replies:** 1\ **Last updated:** [August 16, 2021, 6:22am UTC](https://discuss.elastic.co/t/duplicate-a-document-on-elasticsearch/281504 "2021-08-16T06:22:50Z")
Hi, I want to create a duplicate document entry in elasticsearch as it has 40 fields associated with it. Also i want to create another document entry with just 1 fields altered. Which command to go about ? --- ## [Jar file version for logstash jdbc input](https://discuss.elastic.co/t/jar-file-version-for-logstash-jdbc-input/281353)
**Author:** [@6igwig](https://discuss.elastic.co/u/6igwig)\ **Replies:** 1\ **Last updated:** [August 16, 2021, 5:20am UTC](https://discuss.elastic.co/t/jar-file-version-for-logstash-jdbc-input/281353 "2021-08-16T05:20:29Z")
I am trying to pull in data from a MS SQL Server database via logstash that is hosted on a linux server. It looks like this is a great job for the JDBC input plugin. I am going through the example config posted on the d… --- ## [Transferring log files via Filebeat to Logstash and then Elasticsearch](https://discuss.elastic.co/t/transferring-log-files-via-filebeat-to-logstash-and-then-elasticsearch/281471)
**Author:** [@Nisha2297](https://discuss.elastic.co/u/Nisha2297)\ **Replies:** 4\ **Last updated:** [August 16, 2021, 5:01am UTC](https://discuss.elastic.co/t/transferring-log-files-via-filebeat-to-logstash-and-then-elasticsearch/281471 "2021-08-16T05:01:53Z")
Hi, I am exploring on Elastic Stack and working on creating a pipeline to fetch logs via filbeat/metricbeat to logstash and then to Elasticsearch. I have updated the filebeat.yml/metricbeat.yml and logstash.conf file. C… --- ## [Search thread pool active and queue not full but rejected happened](https://discuss.elastic.co/t/search-thread-pool-active-and-queue-not-full-but-rejected-happened/281494)
**Author:** [@casterQ](https://discuss.elastic.co/u/casterQ)\ **Replies:** 7\ **Last updated:** [August 16, 2021, 4:20am UTC](https://discuss.elastic.co/t/search-thread-pool-active-and-queue-not-full-but-rejected-happened/281494 "2021-08-16T04:20:12Z")
i found a datanode search thread pool has rejected，but grafana monitor show that moment search threap\_pool ’s active(size=49) and queue(size=2000) is not full，I can't understand why ，please help me。thanks a lot... --- ## [Suggestions to use UltraWarm for the below scenario](https://discuss.elastic.co/t/suggestions-to-use-ultrawarm-for-the-below-scenario/281298)
**Author:** [@Kareemulla\_Dudekula](https://discuss.elastic.co/u/Kareemulla_Dudekula)\ **Replies:** 4\ **Last updated:** [August 16, 2021, 4:05am UTC](https://discuss.elastic.co/t/suggestions-to-use-ultrawarm-for-the-below-scenario/281298 "2021-08-16T04:05:16Z")
I have a scenario like, I want to move some ES index data to UltraWarm which we update/modify very rarely. So, when ever I want to update the particular ES UltraWarm index data through the app, I want to move that parti… --- ## [Elastic search configuration with kibana and nodes](https://discuss.elastic.co/t/elastic-search-configuration-with-kibana-and-nodes/281431)
**Author:** [@Dilnawaz\_Khan](https://discuss.elastic.co/u/Dilnawaz_Khan)\ **Replies:** 1\ **Last updated:** [August 16, 2021, 2:51am UTC](https://discuss.elastic.co/t/elastic-search-configuration-with-kibana-and-nodes/281431 "2021-08-16T02:51:33Z")
I have 2 master node of elasticsearch 7.5.2 and 3 data nodes & 1 seperate node for kibana following is the config files. Elasticsearch master1 & master2 ( elasticssearch.yml ) cluster.name: Es\_Cluster node.name: ES\_ma… --- ## [How to monitor Kubernetes Pod's status?](https://discuss.elastic.co/t/how-to-monitor-kubernetes-pods-status/281488)
**Author:** [@avi-devops](https://discuss.elastic.co/u/avi-devops)\ **Replies:** 1\ **Last updated:** [August 16, 2021, 2:41am UTC](https://discuss.elastic.co/t/how-to-monitor-kubernetes-pods-status/281488 "2021-08-16T02:41:24Z")
Hello Experts, I am very new to this wonderful platform, hence learning slowly to grab this technology. Meanwhile, I was wondering how to monitor and display Kubernetes pod status in Kibana ? Please let me know your v… --- ## [Custom index does NOT show in Kibana](https://discuss.elastic.co/t/custom-index-does-not-show-in-kibana/281490)
**Author:** [@panagiss](https://discuss.elastic.co/u/panagiss)\ **Replies:** 3\ **Last updated:** [August 15, 2021, 11:08pm UTC](https://discuss.elastic.co/t/custom-index-does-not-show-in-kibana/281490 "2021-08-15T23:08:56Z")
I'm outputting a custom index from Logstash to Elastisearch but it won't show in KIbana's main dashboards (e.g. Security panel / data sources). While it will show in index management in settings. Here is the how out… --- ## [Not getting the data](https://discuss.elastic.co/t/not-getting-the-data/281478)
**Author:** [@xubisid](https://discuss.elastic.co/u/xubisid)\ **Replies:** 1\ **Last updated:** [August 15, 2021, 10:53pm UTC](https://discuss.elastic.co/t/not-getting-the-data/281478 "2021-08-15T22:53:37Z")
Hi, Whenever I try to GET / POST through the following URL in Postman. I am able to successfully perform the operation XXXX.ent.eastus2.azure.elastic-cloud.com when try to make the similar call through the code, I am … --- ## [Matching Logstash and Elasticsearch data stream configuration](https://discuss.elastic.co/t/matching-logstash-and-elasticsearch-data-stream-configuration/281323)
**Author:** [@jze1](https://discuss.elastic.co/u/jze1)\ **Replies:** 2\ **Last updated:** [August 15, 2021, 10:44pm UTC](https://discuss.elastic.co/t/matching-logstash-and-elasticsearch-data-stream-configuration/281323 "2021-08-15T22:44:40Z")
Hello, can someone shed some light into configuration of logstash/data streams? I have a bunch of apps (kubernetes pods) and I need to store logs of some for one year and the rest just for one week. I found out that by… --- ## [Kibana 7.14 - fresh installation fails \[Starting saved objects migrations - action failed\]](https://discuss.elastic.co/t/kibana-7-14-fresh-installation-fails-starting-saved-objects-migrations-action-failed/281385)
**Author:** [@Dostoyevsky](https://discuss.elastic.co/u/Dostoyevsky)\ **Replies:** 1\ **Last updated:** [August 15, 2021, 10:43pm UTC](https://discuss.elastic.co/t/kibana-7-14-fresh-installation-fails-starting-saved-objects-migrations-action-failed/281385 "2021-08-15T22:43:53Z")
Hi everyone, note that this is a fresh install, not an upgrade, also elasticsearch is running i am attempting to run Kibana on my local win10 machine, however, am faced with the following message : log \[18:58:43.305… --- ## [How to Change Default Index Pattern on Filebeat dashboards on Kibana](https://discuss.elastic.co/t/how-to-change-default-index-pattern-on-filebeat-dashboards-on-kibana/281486)
**Author:** [@nunex\_17](https://discuss.elastic.co/u/nunex_17)\ **Replies:** 4\ **Last updated:** [August 15, 2021, 7:51pm UTC](https://discuss.elastic.co/t/how-to-change-default-index-pattern-on-filebeat-dashboards-on-kibana/281486 "2021-08-15T19:51:42Z")
Hi. I am having a hard time trying to figure this out. I changed the default index name to suricata-ids. Also, added this line in order to use this new index on the default dashboards. But still, default dashboa… --- ## [Understanding elasticsearch certiifcate](https://discuss.elastic.co/t/understanding-elasticsearch-certiifcate/281482)
**Author:** [@naaviin](https://discuss.elastic.co/u/naaviin)\ **Replies:** 1\ **Last updated:** [August 15, 2021, 5:00pm UTC](https://discuss.elastic.co/t/understanding-elasticsearch-certiifcate/281482 "2021-08-15T17:00:31Z")
i saw there 2 type of certificate which is elastic-stack-ca.p12 and elastic-certificates.p12. What are the differences between this 2 certificate https://www.elastic.co/guide/en/elasticsearch/reference/current/security-… --- ## [HTTPJson Input Request Transforms](https://discuss.elastic.co/t/httpjson-input-request-transforms/281481)
**Author:** [@Oliver2](https://discuss.elastic.co/u/Oliver2)\ **Replies:** 0\ **Last updated:** [August 15, 2021, 2:47pm UTC](https://discuss.elastic.co/t/httpjson-input-request-transforms/281481 "2021-08-15T14:47:45Z")
Im trying to create a custom input for: I can get it to produce events fine with the following config: - type: httpjson config\_version: 2 interval: 1h request.url: https://services.nvd.nist.gov/rest/json/cves/1.… --- ## [Cold data node is not starting after restart the ES service 7.7.0](https://discuss.elastic.co/t/cold-data-node-is-not-starting-after-restart-the-es-service-7-7-0/281474)
**Author:** [@alicango](https://discuss.elastic.co/u/alicango)\ **Replies:** 2\ **Last updated:** [August 15, 2021, 10:29am UTC](https://discuss.elastic.co/t/cold-data-node-is-not-starting-after-restart-the-es-service-7-7-0/281474 "2021-08-15T10:29:51Z")
Hi, I am getting below exception after restart the ES service; journalctl -xe systemd-entrypoint\[4842\]: Exception in thread "main" org.apache.lucene.index.CorruptIndexException: codec header mismatch: actual header=10… --- ## [The resident memory of the filebeat keep increasing if hitting the "harvester\_limit"](https://discuss.elastic.co/t/the-resident-memory-of-the-filebeat-keep-increasing-if-hitting-the-harvester-limit/281473)
**Author:** [@VinothNarasimhan](https://discuss.elastic.co/u/VinothNarasimhan)\ **Replies:** 0\ **Last updated:** [August 15, 2021, 7:19am UTC](https://discuss.elastic.co/t/the-resident-memory-of-the-filebeat-keep-increasing-if-hitting-the-harvester-limit/281473 "2021-08-15T07:19:40Z")
Hi, We used the filebeat to ship the logs to logstash. We used the filebeat version 7.3.1. We do set the harvester\_limit: 30 close\_timeout: 15 close\_inactive: 5m for the input. It works fine for a while . At certain t… --- ## [Harvester could not be started on new file & registry already stopped](https://discuss.elastic.co/t/harvester-could-not-be-started-on-new-file-registry-already-stopped/281444)
**Author:** [@bhavaniprasad\_reddy](https://discuss.elastic.co/u/bhavaniprasad_reddy)\ **Replies:** 0\ **Last updated:** [August 14, 2021, 2:50pm UTC](https://discuss.elastic.co/t/harvester-could-not-be-started-on-new-file-registry-already-stopped/281444 "2021-08-14T14:50:56Z")
I am using filebeat-oss v7.12.0 in order to collect events from kubernetes pods using autodiscover. It is expected to collect 5450 events from 5450 pods but fails to harvest for 150 pods and contains the following error… --- ## [Scripting - custom script to generate span query?](https://discuss.elastic.co/t/scripting-custom-script-to-generate-span-query/281470)
**Author:** [@byteandbit](https://discuss.elastic.co/u/byteandbit)\ **Replies:** 0\ **Last updated:** [August 15, 2021, 5:08am UTC](https://discuss.elastic.co/t/scripting-custom-script-to-generate-span-query/281470 "2021-08-15T05:08:23Z")
Our search use case uses Span Queries. Due to the nature of our data and Span Queries the "query" definition has to be always dynamic as we have to analyse the search text, eg: "quick brown fox" -\> { "query": { "s… --- ## [Tags for security-related encoding issues](https://discuss.elastic.co/t/tags-for-security-related-encoding-issues/281466)
**Author:** [@cknz](https://discuss.elastic.co/u/cknz)\ **Replies:** 0\ **Last updated:** [August 14, 2021, 11:58pm UTC](https://discuss.elastic.co/t/tags-for-security-related-encoding-issues/281466 "2021-08-14T23:58:42Z")
Let's say you have some logstash module or Ruby configuration that cleans up / parses / etc. logs, and it comes across something that would be illegal (Eg. illegal UTF-8 encoding sequence which if it were not sanitized c… --- ## [What Elasticsearch GUI admin and monitoring tools have you found that are awesome?](https://discuss.elastic.co/t/what-elasticsearch-gui-admin-and-monitoring-tools-have-you-found-that-are-awesome/281449)
**Author:** [@Jason\_Baumgartner](https://discuss.elastic.co/u/Jason_Baumgartner)\ **Replies:** 0\ **Last updated:** [August 14, 2021, 4:09pm UTC](https://discuss.elastic.co/t/what-elasticsearch-gui-admin-and-monitoring-tools-have-you-found-that-are-awesome/281449 "2021-08-14T16:09:44Z")
We all know that Kibana is an amazing product, but I'm curious what tools the rest of the ES community is using to monitor the health of their cluster and tools that help simplify administration of an Elasticsearch clust… --- ## [Unable to authenticate user \['elastic'\] in docker-compose](https://discuss.elastic.co/t/unable-to-authenticate-user-elastic-in-docker-compose/281445)
**Author:** [@naaviin](https://discuss.elastic.co/u/naaviin)\ **Replies:** 0\ **Last updated:** [August 14, 2021, 2:56pm UTC](https://discuss.elastic.co/t/unable-to-authenticate-user-elastic-in-docker-compose/281445 "2021-08-14T14:56:38Z")
I am hitting below error in Kibana during startup via docker compose. I am using same id elastic in my kibana below is my docker-compsoe code version: "3.9" services: kibana: image: docker.elastic.co/kibana/kiba… --- ## [Order of CTX operators?](https://discuss.elastic.co/t/order-of-ctx-operators/281442)
**Author:** [@Jason\_Baumgartner](https://discuss.elastic.co/u/Jason_Baumgartner)\ **Replies:** 0\ **Last updated:** [August 14, 2021, 2:36pm UTC](https://discuss.elastic.co/t/order-of-ctx-operators/281442 "2021-08-14T14:36:22Z")
I am trying to make an update script condition that specifies only updating a document if the retrieved\_utc is newer than the existing one and then after that, examine several other conditions. When I try to use parenthe… --- ## [Filter logs which are after a particular event](https://discuss.elastic.co/t/filter-logs-which-are-after-a-particular-event/281437)
**Author:** [@Pavan\_Kumar\_R](https://discuss.elastic.co/u/Pavan_Kumar_R)\ **Replies:** 0\ **Last updated:** [August 14, 2021, 11:12am UTC](https://discuss.elastic.co/t/filter-logs-which-are-after-a-particular-event/281437 "2021-08-14T11:12:30Z")
I have 4 applications each will write "Application started : App1", with application's name at the end. In logstash I used grok filter to extract applications name into a field 'appName', I also added tags '\_event', 'app… [Previous page](https://discuss.elastic.co/c/elastic-stack/81.md?page=1279) [Next page](https://discuss.elastic.co/c/elastic-stack/81.md?page=1281)