# Elastic Stack **URL:** https://discuss.elastic.co/c/elastic-stack/81.md?page=1252 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1253 --- ## [Elastic agent (metricbeat logs): Cannot index event publisher.Event](https://discuss.elastic.co/t/elastic-agent-metricbeat-logs-cannot-index-event-publisher-event/282364)
**Author:** [@Gomeisa](https://discuss.elastic.co/u/Gomeisa)\ **Replies:** 11\ **Last updated:** [September 6, 2021, 7:37am UTC](https://discuss.elastic.co/t/elastic-agent-metricbeat-logs-cannot-index-event-publisher-event/282364 "2021-09-06T07:37:14Z")
Hi, I have recently installed a fleet-managed Elastic Agent on my servers. my metricbeat\_monitor-json.log is flooded with this message: {"log.level":"warn","@timestamp":"2021-08-24T15:26:57.009+0430","log.logger":"ela… --- ## [Is there a way to just post a command using the low level or nest .net clients?](https://discuss.elastic.co/t/is-there-a-way-to-just-post-a-command-using-the-low-level-or-nest-net-clients/283392)
**Author:** [@astanton1978](https://discuss.elastic.co/u/astanton1978)\ **Replies:** 1\ **Last updated:** [September 6, 2021, 7:07am UTC](https://discuss.elastic.co/t/is-there-a-way-to-just-post-a-command-using-the-low-level-or-nest-net-clients/283392 "2021-09-06T07:07:03Z")
I dont see an appropriate category or tags for either of those .net clients, and the GitHub repo for them says to ask questions here, so sorry if this is the wrong category. I need to create an index mapping with some a… --- ## [Using Nest in AWS Lambda](https://discuss.elastic.co/t/using-nest-in-aws-lambda/283340)
**Author:** [@imranazad](https://discuss.elastic.co/u/imranazad)\ **Replies:** 1\ **Last updated:** [September 6, 2021, 6:59am UTC](https://discuss.elastic.co/t/using-nest-in-aws-lambda/283340 "2021-09-06T06:59:38Z")
Has anyone used Nest with AWS Lambda? How did you deal with cold starts? The size of dlls affects performance and Nests library is large, the version we're using is 4 megabytes. Is there a light weight version of the NE… --- ## [Using Scroll Api causes Out of Memory errors](https://discuss.elastic.co/t/using-scroll-api-causes-out-of-memory-errors/283388)
**Author:** [@lakshmiteja\_salapu](https://discuss.elastic.co/u/lakshmiteja_salapu)\ **Replies:** 8\ **Last updated:** [September 6, 2021, 6:54am UTC](https://discuss.elastic.co/t/using-scroll-api-causes-out-of-memory-errors/283388 "2021-09-06T06:54:06Z")
I am trying to use scroll for getting over 41 million documents (156 GB) but it fails with out of memory error after getting 410000 documents Using batch size of 5000 documents and 30s - scroll time (also tried changing… --- ## [Nested aggregation referencing outer aggregation](https://discuss.elastic.co/t/nested-aggregation-referencing-outer-aggregation/283164)
**Author:** [@Jakob](https://discuss.elastic.co/u/Jakob)\ **Replies:** 2\ **Last updated:** [September 6, 2021, 6:29am UTC](https://discuss.elastic.co/t/nested-aggregation-referencing-outer-aggregation/283164 "2021-09-06T06:29:14Z")
I am learning about aggregations by looking at the "kibana\_sample\_data\_flights" index and I'm trying to find how many flights have the same weather on departure and arrival. Here is what I have so far: GET /kibana\_samp… --- ## [How to reduce the memory usage of filebeat](https://discuss.elastic.co/t/how-to-reduce-the-memory-usage-of-filebeat/283401)
**Author:** [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\ **Replies:** 6\ **Last updated:** [September 6, 2021, 6:05am UTC](https://discuss.elastic.co/t/how-to-reduce-the-memory-usage-of-filebeat/283401 "2021-09-06T06:05:20Z")
I am using filebeat to collect log files. I'm using filebeat to collect log files, and on one of my servers, filebeat's memory usage is high. I would like to limit the memory usage of filebeat. I have set up queue.mem … --- ## [Cannot start transform job - Cannot set after key in the composite aggregation \[\_transform\]](https://discuss.elastic.co/t/cannot-start-transform-job-cannot-set-after-key-in-the-composite-aggregation-transform/283013)
**Author:** [@ciment](https://discuss.elastic.co/u/ciment)\ **Replies:** 4\ **Last updated:** [September 6, 2021, 6:02am UTC](https://discuss.elastic.co/t/cannot-start-transform-job-cannot-set-after-key-in-the-composite-aggregation-transform/283013 "2021-09-06T06:02:40Z")
Hi, We would like to setup transform job to create statistics - how many devices are connected per hour. In our code we have added for every request, deviceId and clientId to labels data \[labels.deviceId, labels.clientI… --- ## [\[mTLS\]\[filebeat-logstash\]\[beat-input-plugin\] Client certificate hostname verification](https://discuss.elastic.co/t/mtls-filebeat-logstash-beat-input-plugin-client-certificate-hostname-verification/282973)
**Author:** [@LotusD](https://discuss.elastic.co/u/LotusD)\ **Replies:** 6\ **Last updated:** [September 6, 2021, 5:58am UTC](https://discuss.elastic.co/t/mtls-filebeat-logstash-beat-input-plugin-client-certificate-hostname-verification/282973 "2021-09-06T05:58:31Z")
Dear All, I'm trying to set up mTLS between filebeat (client) and logstash (server). I refer to the documents Secure communication with Logstash | Filebeat Reference \[7.14\] | Elastic Filebeat config: output.logstash… --- ## [Why does es upgrade index data synchronously？](https://discuss.elastic.co/t/why-does-es-upgrade-index-data-synchronously/283226)
**Author:** [@jiankunking](https://discuss.elastic.co/u/jiankunking)\ **Replies:** 12\ **Last updated:** [September 6, 2021, 5:50am UTC](https://discuss.elastic.co/t/why-does-es-upgrade-index-data-synchronously/283226 "2021-09-06T05:50:32Z")
When I upgrade from 5.x to 7.x, I find that I cannot upgrade directly and need to reindex all index data. Why not update the index data automatically during the upgrade? --- ## [Index management](https://discuss.elastic.co/t/index-management/283393)
**Author:** [@alipujaistopo](https://discuss.elastic.co/u/alipujaistopo)\ **Replies:** 2\ **Last updated:** [September 6, 2021, 5:31am UTC](https://discuss.elastic.co/t/index-management/283393 "2021-09-06T05:31:53Z")
what does indices mean? and why do i have so many indices? and what is the difference between xxxx-000004 and xxxx-000005? here's my indices for example: \* winlogbeat-7.10.2-2021.08.22-000005 \* auditbeat-7.11.2-2021.0… --- ## [\[Beat Monitoring\] Beat name is differ with state](https://discuss.elastic.co/t/beat-monitoring-beat-name-is-differ-with-state/283239)
**Author:** [@InJong\_Park](https://discuss.elastic.co/u/InJong_Park)\ **Replies:** 2\ **Last updated:** [September 6, 2021, 2:35am UTC](https://discuss.elastic.co/t/beat-monitoring-beat-name-is-differ-with-state/283239 "2021-09-06T02:35:35Z")
Hi. Thanks for your attention. metricbeat.yml name: "sample" metricbeat.log 2021-09-03T14:46:12.876+0900 INFO \[publisher\] pipeline/module.go:113 Beat name: sample curl state \[root@ip-01-02-03-04 15:00:04… --- ## ["Unable to connect to Elasticsearch. Error: \[search\_phase\_execution\_exception\] all shards failed"](https://discuss.elastic.co/t/unable-to-connect-to-elasticsearch-error-search-phase-execution-exception-all-shards-failed/283344)
**Author:** [@ShashikanthKomandoor](https://discuss.elastic.co/u/ShashikanthKomandoor)\ **Replies:** 1\ **Last updated:** [September 5, 2021, 10:22pm UTC](https://discuss.elastic.co/t/unable-to-connect-to-elasticsearch-error-search-phase-execution-exception-all-shards-failed/283344 "2021-09-05T22:22:02Z")
Hi Team, I have been running ELK Cluster of version 7.8 with 3 master nodes and 5 data nodes. I am running this on RHEL 7.5. I have installed httpd to proxy the requests to Kibana. I installed httpd, Kibana, Elasticsear… --- ## [Fighting with ingest pipeline create](https://discuss.elastic.co/t/fighting-with-ingest-pipeline-create/283353)
**Author:** [@Russell\_Fulton](https://discuss.elastic.co/u/Russell_Fulton)\ **Replies:** 7\ **Last updated:** [September 5, 2021, 9:40pm UTC](https://discuss.elastic.co/t/fighting-with-ingest-pipeline-create/283353 "2021-09-05T21:40:43Z")
Version 7.14 I am trying to create a simple ingest pipeline in Kibana but can not figure out what syntax the condition will accept and there is no documentation that I can find. I know how to specify what I want in the… --- ## [Curl query to delete index patterns, and, parsefailures to stdout](https://discuss.elastic.co/t/curl-query-to-delete-index-patterns-and-parsefailures-to-stdout/280110)
**Author:** [@kriss332](https://discuss.elastic.co/u/kriss332)\ **Replies:** 6\ **Last updated:** [September 5, 2021, 6:35pm UTC](https://discuss.elastic.co/t/curl-query-to-delete-index-patterns-and-parsefailures-to-stdout/280110 "2021-09-05T18:35:38Z")
Hi all... I have 2 questions- I have created more than 150 index patterns based on the indexes I have. Yesterday I changed the names of indexes for better visibility. Now I need to delete all old index-patterns using… --- ## [Using memcached for logstash](https://discuss.elastic.co/t/using-memcached-for-logstash/283381)
**Author:** [@kriss332](https://discuss.elastic.co/u/kriss332)\ **Replies:** 0\ **Last updated:** [September 5, 2021, 5:57pm UTC](https://discuss.elastic.co/t/using-memcached-for-logstash/283381 "2021-09-05T17:57:12Z")
Hi all. I have to work with memcached for logstash to get the alerts about the destination IPs (belonging to C2 IPs) from my firewall's destination IPs. I have MISP for finding malicious/suspicious IPs about the traffic. … --- ## [Created simple filebeat module for ProxySQL](https://discuss.elastic.co/t/created-simple-filebeat-module-for-proxysql/283380)
**Author:** [@altuhovsu](https://discuss.elastic.co/u/altuhovsu)\ **Replies:** 0\ **Last updated:** [September 5, 2021, 5:16pm UTC](https://discuss.elastic.co/t/created-simple-filebeat-module-for-proxysql/283380 "2021-09-05T17:16:02Z")
If anyone needed, I have created (cloned from mysql module) a simple Filebeat module for parsing ProxySQL logs and ship them to ElasticSearch. https://github.com/altuhovsu/filebeat-proxysql-module --- ## [Ruby filter plugin does not read new field values](https://discuss.elastic.co/t/ruby-filter-plugin-does-not-read-new-field-values/283343)
**Author:** [@priyaankaa](https://discuss.elastic.co/u/priyaankaa)\ **Replies:** 8\ **Last updated:** [September 5, 2021, 2:22pm UTC](https://discuss.elastic.co/t/ruby-filter-plugin-does-not-read-new-field-values/283343 "2021-09-05T14:22:25Z")
How to get correct value of the field inside ruby code in logstash pipeline? sample 1: input { elasticsearch { hosts =\> "http://localhost:9200" index =\> "test1" } } filter { mutate { add\_field =\>… --- ## [Highlighting of exact search with simple\_query\_string](https://discuss.elastic.co/t/highlighting-of-exact-search-with-simple-query-string/283292)
**Author:** [@Wonder\_Garance](https://discuss.elastic.co/u/Wonder_Garance)\ **Replies:** 1\ **Last updated:** [September 5, 2021, 12:53pm UTC](https://discuss.elastic.co/t/highlighting-of-exact-search-with-simple-query-string/283292 "2021-09-05T12:53:23Z")
I use Elasticsearch 7.9 and I want to highlight an exact search filed with simple query string query, for a document. The following query doesn't highlight words in the document, although these words are in the document… --- ## [Influxdb and elasticsearch benchmarking for timeseries data](https://discuss.elastic.co/t/influxdb-and-elasticsearch-benchmarking-for-timeseries-data/283371)
**Author:** [@mruthyu](https://discuss.elastic.co/u/mruthyu)\ **Replies:** 0\ **Last updated:** [September 5, 2021, 11:48am UTC](https://discuss.elastic.co/t/influxdb-and-elasticsearch-benchmarking-for-timeseries-data/283371 "2021-09-05T11:48:48Z")
Do we have the benchmarking stats for Elasticsearch and Influxdb considering the time series data (Metrics) Looking into the performance aspects as recent releases of elasticsearch have the additional optimization strat… --- ## [Using Kafka Stream for data transform and enrichment](https://discuss.elastic.co/t/using-kafka-stream-for-data-transform-and-enrichment/283370)
**Author:** [@mruthyu](https://discuss.elastic.co/u/mruthyu)\ **Replies:** 0\ **Last updated:** [September 5, 2021, 11:40am UTC](https://discuss.elastic.co/t/using-kafka-stream-for-data-transform-and-enrichment/283370 "2021-09-05T11:40:04Z")
Is it possible to use Kafka Streaming for data transform and enrichment instead of logstash? What are the advantages and disadvantages? --- ## [Virustotal Module MISP attributes don't store in ELK](https://discuss.elastic.co/t/virustotal-module-misp-attributes-dont-store-in-elk/282834)
**Author:** [@Abanob\_Medhat](https://discuss.elastic.co/u/Abanob_Medhat)\ **Replies:** 3\ **Last updated:** [September 5, 2021, 11:20am UTC](https://discuss.elastic.co/t/virustotal-module-misp-attributes-dont-store-in-elk/282834 "2021-09-05T11:20:55Z")
hello, I integrated Virustotal with MISP and run virustotal on events and all virustotal attributes are saved in MISP. also , I integrated MISP with Elastic using Filebeats. the events are stored in Elastic and I can … --- ## [Elasticsearch is displaying correct result only after a space](https://discuss.elastic.co/t/elasticsearch-is-displaying-correct-result-only-after-a-space/283009)
**Author:** [@Drsaud](https://discuss.elastic.co/u/Drsaud)\ **Replies:** 9\ **Last updated:** [September 5, 2021, 10:43am UTC](https://discuss.elastic.co/t/elasticsearch-is-displaying-correct-result-only-after-a-space/283009 "2021-09-05T10:43:22Z")
I build react app the has search bar using elasticsearch, but the problem is when I search in my search bar earthier I have to write the word not complete or I write as complete word but after I click space. for example… --- ## [TSVB line chart has gaps](https://discuss.elastic.co/t/tsvb-line-chart-has-gaps/282529)
**Author:** [@vijay\_kaali](https://discuss.elastic.co/u/vijay_kaali)\ **Replies:** 3\ **Last updated:** [September 5, 2021, 6:45am UTC](https://discuss.elastic.co/t/tsvb-line-chart-has-gaps/282529 "2021-09-05T06:45:20Z")
i have create a line chart and tsvb . line chart. in visualization , tsvb has gaps while normal line chart looks fine . 1st chart tsvb 2nd chart normal line chart . with same data, same avg( runqueue) formula wh… --- ## [S3 output plugin "upload\_worker\_count" logs](https://discuss.elastic.co/t/s3-output-plugin-upload-worker-count-logs/283357)
**Author:** [@Varsha1](https://discuss.elastic.co/u/Varsha1)\ **Replies:** 0\ **Last updated:** [September 5, 2021, 5:15am UTC](https://discuss.elastic.co/t/s3-output-plugin-upload-worker-count-logs/283357 "2021-09-05T05:15:32Z")
I've realized I don't see the detailed s3 log in console when running logstash for kafka input and s3 output. Also I've configured the S3 output plugin to run with multiple workers, how could I make sure the 5 workers ar… --- ## [I am trying to pass a fieldname in jdbc\_streaming plugin to the jdbc\_user parameter, but it fails to read the value of the field](https://discuss.elastic.co/t/i-am-trying-to-pass-a-fieldname-in-jdbc-streaming-plugin-to-the-jdbc-user-parameter-but-it-fails-to-read-the-value-of-the-field/283241)
**Author:** [@Varsha1](https://discuss.elastic.co/u/Varsha1)\ **Replies:** 4\ **Last updated:** [September 5, 2021, 5:07am UTC](https://discuss.elastic.co/t/i-am-trying-to-pass-a-fieldname-in-jdbc-streaming-plugin-to-the-jdbc-user-parameter-but-it-fails-to-read-the-value-of-the-field/283241 "2021-09-05T05:07:31Z")
jdbc\_streaming { jdbc\_driver\_class =\> "Java::oracle.jdbc.driver.OracleDriver" jdbc\_driver\_library =\> "/home/user/logstash-7.14.0/logstash-core/lib/jars/ojdbc8.jar" jdbc\_connection\_string =\> "jdbc:oracle:thin:@XXXXX" … --- ## [Many deleted indices, no recovery of disk space?](https://discuss.elastic.co/t/many-deleted-indices-no-recovery-of-disk-space/283328)
**Author:** [@NetwarSystem](https://discuss.elastic.co/u/NetwarSystem)\ **Replies:** 3\ **Last updated:** [September 5, 2021, 3:51am UTC](https://discuss.elastic.co/t/many-deleted-indices-no-recovery-of-disk-space/283328 "2021-09-05T03:51:21Z")
I have a 1.2TB backup of Elasticsearch data that I have restored to a standalone machine running 7.14.0. The restore worked fine and I began grooming the data - about half of the indices in the snapshot are not needed fo… --- ## [Filebeat retrieve log with current time stamp](https://discuss.elastic.co/t/filebeat-retrieve-log-with-current-time-stamp/283355)
**Author:** [@josh12](https://discuss.elastic.co/u/josh12)\ **Replies:** 0\ **Last updated:** [September 5, 2021, 1:10am UTC](https://discuss.elastic.co/t/filebeat-retrieve-log-with-current-time-stamp/283355 "2021-09-05T01:10:36Z")
Can I know whether i can retrieve logs bsed on current date. I am planning for \/\*.log and set ignore\_older : 1h if this the case, how it works. will it scan all logs and take based on current date and time … --- ## [Why does every log use a different format for the Timestamp?](https://discuss.elastic.co/t/why-does-every-log-use-a-different-format-for-the-timestamp/283346)
**Author:** [@mhare](https://discuss.elastic.co/u/mhare)\ **Replies:** 10\ **Last updated:** [September 4, 2021, 9:19pm UTC](https://discuss.elastic.co/t/why-does-every-log-use-a-different-format-for-the-timestamp/283346 "2021-09-04T21:19:40Z")
I am using Stack and Filebeat version 7.13.4 on Windows 10 I have a new log to process and it has a new way to represent the timestamp. As soon as I figure one out, they throw me another This one is starts as: 2021/0… --- ## [Apply aggregation result to values in the document \[Elasticsearch\]](https://discuss.elastic.co/t/apply-aggregation-result-to-values-in-the-document-elasticsearch/283350)
**Author:** [@me.mohammed](https://discuss.elastic.co/u/me.mohammed)\ **Replies:** 0\ **Last updated:** [September 4, 2021, 6:59pm UTC](https://discuss.elastic.co/t/apply-aggregation-result-to-values-in-the-document-elasticsearch/283350 "2021-09-04T18:59:58Z")
so what I'm trying to do is change the actual values in the document depending on the aggregation result, I'm not sure if this is possible or not but here are some examples to what I'm trying to do { "deals": \[ { … --- ## [Mixed kinds of rename option in single mutate filter won't work](https://discuss.elastic.co/t/mixed-kinds-of-rename-option-in-single-mutate-filter-wont-work/283278)
**Author:** [@tasneem-hyder](https://discuss.elastic.co/u/tasneem-hyder)\ **Replies:** 4\ **Last updated:** [September 4, 2021, 6:23pm UTC](https://discuss.elastic.co/t/mixed-kinds-of-rename-option-in-single-mutate-filter-wont-work/283278 "2021-09-04T18:23:11Z")
In a single mutate filter, we can't use the mixed kinds of rename option. This is not configuration issue but a runtime issue so it is hard to debug. As this kind of config setup doesn't compile to an expected config so … [Previous page](https://discuss.elastic.co/c/elastic-stack/81.md?page=1251) [Next page](https://discuss.elastic.co/c/elastic-stack/81.md?page=1253)