# Latest **URL:** https://discuss.elastic.co/latest.md?page=1536 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1537 --- ## [Get the raw log which can't index to elastic search](https://discuss.elastic.co/t/get-the-raw-log-which-cant-index-to-elastic-search/280693)
**Author:** [@jamecollins](https://discuss.elastic.co/u/jamecollins)\ **Replies:** 1\ **Last updated:** [August 9, 2021, 8:40am UTC](https://discuss.elastic.co/t/get-the-raw-log-which-cant-index-to-elastic-search/280693 "2021-08-09T08:40:21Z")
I have error on logstash: \[2021-08-07T00:24:08,330\]\[WARN \]\[logstash.outputs.elasticsearch\] Could not index event to Elasticsearch. {:status=\>400, :action=\>\["index", {:\_id=\>nil, :\_index=\>"logstash-2021.08.07", :\_type=\>"… --- ## [Painless: Any easy way to define dates like now-1d, now-2d](https://discuss.elastic.co/t/painless-any-easy-way-to-define-dates-like-now-1d-now-2d/280773)
**Author:** [@kelk](https://discuss.elastic.co/u/kelk)\ **Replies:** 0\ **Last updated:** [August 9, 2021, 8:21am UTC](https://discuss.elastic.co/t/painless-any-easy-way-to-define-dates-like-now-1d-now-2d/280773 "2021-08-09T08:21:56Z")
We have a painless script to compare a logic before a date. Currently the user have to enter epoch value into the params, but was checking if you guys already have a script or logic, so we can use the "range" notations i… --- ## [Elastic agent in a docker container TLS error](https://discuss.elastic.co/t/elastic-agent-in-a-docker-container-tls-error/280534)
**Author:** [@NatSav](https://discuss.elastic.co/u/NatSav)\ **Replies:** 2\ **Last updated:** [August 9, 2021, 8:19am UTC](https://discuss.elastic.co/t/elastic-agent-in-a-docker-container-tls-error/280534 "2021-08-09T08:19:20Z")
Good afternoon, I am trying to make a docker container with elastic-agent, packetbeat and metricbeat. I am going to add other components later however I want to get these working first. When I try to install elastic-agen… --- ## [Setup logstash module netflow](https://discuss.elastic.co/t/setup-logstash-module-netflow/279528)
**Author:** [@Muhammed\_Ibrahim](https://discuss.elastic.co/u/Muhammed_Ibrahim)\ **Replies:** 1\ **Last updated:** [August 9, 2021, 8:16am UTC](https://discuss.elastic.co/t/setup-logstash-module-netflow/279528 "2021-08-09T08:16:46Z")
Dears, I need to install netflow dashboard as it's shown in elastic guide by this command : sudo bin/logstash --modules netflow --setup --path.settings /etc/logstash/ but appears this error below : Using bundled JDK… --- ## [Synchronize whole files (mostly PDFs) to ES from directory and e-mails](https://discuss.elastic.co/t/synchronize-whole-files-mostly-pdfs-to-es-from-directory-and-e-mails/279962)
**Author:** [@jporzelt](https://discuss.elastic.co/u/jporzelt)\ **Replies:** 3\ **Last updated:** [August 9, 2021, 8:08am UTC](https://discuss.elastic.co/t/synchronize-whole-files-mostly-pdfs-to-es-from-directory-and-e-mails/279962 "2021-08-09T08:08:54Z")
Hi all, we would like to setup a document synchronization from a file directory and from a e-mail imap folder to elasticsearch. For the e-mails we are interested in the attachments. Here the imap input plugin looks pro… --- ## [Logstash stopped to treat the incoming logs](https://discuss.elastic.co/t/logstash-stopped-to-treat-the-incoming-logs/280402)
**Author:** [@tarekilani](https://discuss.elastic.co/u/tarekilani)\ **Replies:** 1\ **Last updated:** [August 9, 2021, 7:59am UTC](https://discuss.elastic.co/t/logstash-stopped-to-treat-the-incoming-logs/280402 "2021-08-09T07:59:15Z")
Hello, I'm using ELK stack to correlate logs of network devices. The stack was perfectly working for a long period of time. But today logstash stopped treating the incoming logs. When i did an investigation i found thi… --- ## [Watcher action index](https://discuss.elastic.co/t/watcher-action-index/280078)
**Author:** [@bilel\_meddeb](https://discuss.elastic.co/u/bilel_meddeb)\ **Replies:** 3\ **Last updated:** [August 9, 2021, 7:54am UTC](https://discuss.elastic.co/t/watcher-action-index/280078 "2021-08-09T07:54:34Z")
hello i create a watcher with this action i want to store the result in index --- ## [How can I ...Compare two indices fields if data matches then create new index and store the matched data into it](https://discuss.elastic.co/t/how-can-i-compare-two-indices-fields-if-data-matches-then-create-new-index-and-store-the-matched-data-into-it/280130)
**Author:** [@sneha\_gupta](https://discuss.elastic.co/u/sneha_gupta)\ **Replies:** 2\ **Last updated:** [August 9, 2021, 7:52am UTC](https://discuss.elastic.co/t/how-can-i-compare-two-indices-fields-if-data-matches-then-create-new-index-and-store-the-matched-data-into-it/280130 "2021-08-09T07:52:52Z")
Was thinking if I can use sub queries either "union" but left with nothing as above is not supported --- ## [Logstash geoip databasemanager with outgoing proxy](https://discuss.elastic.co/t/logstash-geoip-databasemanager-with-outgoing-proxy/280751)
**Author:** [@cknz](https://discuss.elastic.co/u/cknz)\ **Replies:** 2\ **Last updated:** [August 9, 2021, 7:47am UTC](https://discuss.elastic.co/t/logstash-geoip-databasemanager-with-outgoing-proxy/280751 "2021-08-09T07:47:32Z")
Hi all, I've upgraded to Logstash 7.14 from 7.11, and now I'm having to deal with the geoip databasemanager. My servers do not have direct database access and need to go out via proxy. Current observed behaviour is that… --- ## [How logstash uses add\_field to add a JSON array](https://discuss.elastic.co/t/how-logstash-uses-add-field-to-add-a-json-array/280640)
**Author:** [@ktpktr0](https://discuss.elastic.co/u/ktpktr0)\ **Replies:** 2\ **Last updated:** [August 9, 2021, 7:27am UTC](https://discuss.elastic.co/t/how-logstash-uses-add-field-to-add-a-json-array/280640 "2021-08-09T07:27:11Z")
The raw data are as follows "money快照" =\> \[ \[0\] "money:14683689,value:15105589,time:2021-07-27 23:50:54", \[1\] "money:14683689,value:15105589,time:2021-07-27 23:50:56" \], The confi… --- ## [Script field .. shards failed](https://discuss.elastic.co/t/script-field-shards-failed/280290)
**Author:** [@djikon](https://discuss.elastic.co/u/djikon)\ **Replies:** 6\ **Last updated:** [August 9, 2021, 7:26am UTC](https://discuss.elastic.co/t/script-field-shards-failed/280290 "2021-08-09T07:26:24Z")
Hi, I create a script field and i have a lot of shards failed. my script field is just : doc\['evenement.@timestamp'\].value.getMillis() - doc\['debutSejour'\].value.getMillis() shards failed exist too fort : if(doc\['ev… --- ## [When ES updated from 7.3.1 to 7.10.2, search slow 2x in fetch phase](https://discuss.elastic.co/t/when-es-updated-from-7-3-1-to-7-10-2-search-slow-2x-in-fetch-phase/280765)
**Author:** [@jiangnanxue21](https://discuss.elastic.co/u/jiangnanxue21)\ **Replies:** 0\ **Last updated:** [August 9, 2021, 6:46am UTC](https://discuss.elastic.co/t/when-es-updated-from-7-3-1-to-7-10-2-search-slow-2x-in-fetch-phase/280765 "2021-08-09T06:46:30Z")
Our product environment had updated es from 7.3.1 to 7.10.2, the index mapping and search sentence has not changed anymore. We specify include and exclude fileds by using that sourceBuilder.fetchSource(getIncludeFe… --- ## [Elasticsearch search\_phase\_execution\_exception in Python](https://discuss.elastic.co/t/elasticsearch-search-phase-execution-exception-in-python/280763)
**Author:** [@rknd](https://discuss.elastic.co/u/rknd)\ **Replies:** 0\ **Last updated:** [August 9, 2021, 6:35am UTC](https://discuss.elastic.co/t/elasticsearch-search-phase-execution-exception-in-python/280763 "2021-08-09T06:35:42Z")
I fetch all my Elasticsearch documents via using scroll api with below code in Python. size = 100 data = es.search( index=index, scroll='25m', size=size, body=body … --- ## [Ibm mq module issue in metricbeat](https://discuss.elastic.co/t/ibm-mq-module-issue-in-metricbeat/280761)
**Author:** [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Replies:** 1\ **Last updated:** [August 9, 2021, 6:14am UTC](https://discuss.elastic.co/t/ibm-mq-module-issue-in-metricbeat/280761 "2021-08-09T06:14:52Z")
Hi all, I am trying to monitor my IBM WebSphere MQ using ibmmq module of metricbeat version 7.13.1. the metricbeat.yml setting is as following: # =========================== Modules configuration =====================… --- ## [Can I set default sorting for queries?](https://discuss.elastic.co/t/can-i-set-default-sorting-for-queries/280074)
**Author:** [@reka.gay](https://discuss.elastic.co/u/reka.gay)\ **Replies:** 4\ **Last updated:** [August 9, 2021, 6:04am UTC](https://discuss.elastic.co/t/can-i-set-default-sorting-for-queries/280074 "2021-08-09T06:04:34Z")
I know that sorting is based on relevance if no other sorting is defined - and that's how I would like to have it. However, when my search is facet-only, the score of each result is 1 and I'm not quite sure how the resu… --- ## [Discover: Request Timeout after 30000ms](https://discuss.elastic.co/t/discover-request-timeout-after-30000ms/280593)
**Author:** [@sha\_man](https://discuss.elastic.co/u/sha_man)\ **Replies:** 2\ **Last updated:** [August 9, 2021, 5:48am UTC](https://discuss.elastic.co/t/discover-request-timeout-after-30000ms/280593 "2021-08-09T05:48:13Z")
Hi, we are using Kibana 6.7.1 and Elasticsearch version 6.7 in our org. when I run a query to get the data i get timeout error as "Discover: Request Timeout after 30000ms". We have deployed kibana using helm charts in … --- ## [Find Correlation between fields in different indices](https://discuss.elastic.co/t/find-correlation-between-fields-in-different-indices/280498)
**Author:** [@WJ\_Y](https://discuss.elastic.co/u/WJ_Y)\ **Replies:** 1\ **Last updated:** [August 9, 2021, 5:24am UTC](https://discuss.elastic.co/t/find-correlation-between-fields-in-different-indices/280498 "2021-08-09T05:24:45Z")
Using Elasticsearch version 7.14.0, I want to find correlations betweet fields in different indices. An index named "A" has "name", "a-1", "a-2" fields. An index named "B" has "name", "b-1", "b-2" fields. The "name" f… --- ## [/\_security/api\_key - disable / enable](https://discuss.elastic.co/t/security-api-key-disable-enable/280755)
**Author:** [@byteandbit](https://discuss.elastic.co/u/byteandbit)\ **Replies:** 4\ **Last updated:** [August 9, 2021, 5:22am UTC](https://discuss.elastic.co/t/security-api-key-disable-enable/280755 "2021-08-09T05:22:17Z")
Hey, Just wondering if after creating API Key (/\_security/api\_key) there is a way to disable/enable it without invalidation? --- ## [Strange behaviour Index Pattern](https://discuss.elastic.co/t/strange-behaviour-index-pattern/280532)
**Author:** [@Jose\_Manuel\_Tirado](https://discuss.elastic.co/u/Jose_Manuel_Tirado)\ **Replies:** 1\ **Last updated:** [August 9, 2021, 5:03am UTC](https://discuss.elastic.co/t/strange-behaviour-index-pattern/280532 "2021-08-09T05:03:42Z")
HI! when I access index pattern I see these strange fields and I can't do anything with them What are they and how can I eliminate them? Are they occupying my memory? Thank you --- ## [Logstash not listening on beats input port](https://discuss.elastic.co/t/logstash-not-listening-on-beats-input-port/280559)
**Author:** [@r.saigiridhar](https://discuss.elastic.co/u/r.saigiridhar)\ **Replies:** 1\ **Last updated:** [August 9, 2021, 5:02am UTC](https://discuss.elastic.co/t/logstash-not-listening-on-beats-input-port/280559 "2021-08-09T05:02:27Z")
Logstash server is not listening on the beats input port specified. I have installed beats input plugin in logstash server, created a conf file with input port specified and pipelines file that has an entry for the conf. … --- ## [Handle Multiline logs, where number of lines are not sure?](https://discuss.elastic.co/t/handle-multiline-logs-where-number-of-lines-are-not-sure/280757)
**Author:** [@bpandit](https://discuss.elastic.co/u/bpandit)\ **Replies:** 0\ **Last updated:** [August 9, 2021, 3:47am UTC](https://discuss.elastic.co/t/handle-multiline-logs-where-number-of-lines-are-not-sure/280757 "2021-08-09T03:47:57Z")
I need a way to handle multi line logs, where number of lines are not sure, below is the log data, where start \[29-Apr-2018 16:46:32\] \[pool int-mobileapp\] pid 30064 script\_filename = /var/www/html/int-mobileapp/index.p… --- ## [Failed to verify server certificate - APM Go Agent with ECK](https://discuss.elastic.co/t/failed-to-verify-server-certificate-apm-go-agent-with-eck/280738)
**Author:** [@Hongbo-Miao](https://discuss.elastic.co/u/Hongbo-Miao)\ **Replies:** 12\ **Last updated:** [August 9, 2021, 1:05am UTC](https://discuss.elastic.co/t/failed-to-verify-server-certificate-apm-go-agent-with-eck/280738 "2021-08-09T01:05:52Z")
Kibana version: 7.14 Elasticsearch version: 7.14 APM Server version: 7.14 APM Agent language and version: Go (Gin) go.elastic.co/apm/module/apmgin v1.13.1 Browser version: None Original install method (e.g. do… --- ## [Remove index mappings in version 7.4](https://discuss.elastic.co/t/remove-index-mappings-in-version-7-4/280636)
**Author:** [@aldm](https://discuss.elastic.co/u/aldm)\ **Replies:** 3\ **Last updated:** [August 8, 2021, 11:26pm UTC](https://discuss.elastic.co/t/remove-index-mappings-in-version-7-4/280636 "2021-08-08T23:26:40Z")
Hi, We made a mistake to store details\_per\_category in this way { cat1: object, cat2: object, . .. } Now there are thousands of categories, which generated thousands of new fields in out index. We corrected it b… --- ## [FATAL UnableToReadKeystore: unable to read keystore](https://discuss.elastic.co/t/fatal-unabletoreadkeystore-unable-to-read-keystore/280710)
**Author:** [@AmirHossein\_Sohrabi](https://discuss.elastic.co/u/AmirHossein_Sohrabi)\ **Replies:** 2\ **Last updated:** [August 8, 2021, 11:25pm UTC](https://discuss.elastic.co/t/fatal-unabletoreadkeystore-unable-to-read-keystore/280710 "2021-08-08T23:25:54Z")
I'm using an ubuntu 16.04 server on a VM. and installed elasticsearch and kibana both version 7.14. elasticsearch works properly. but when I start the kibana service first it logs running on "systemctl status kibana" … --- ## [Unable to upload json file to elastic search](https://discuss.elastic.co/t/unable-to-upload-json-file-to-elastic-search/280712)
**Author:** [@cmunka](https://discuss.elastic.co/u/cmunka)\ **Replies:** 2\ **Last updated:** [August 8, 2021, 11:19pm UTC](https://discuss.elastic.co/t/unable-to-upload-json-file-to-elastic-search/280712 "2021-08-08T23:19:27Z")
I am trying to upload a jsonl file to elastic search via \_bulk api using Python. Below is my code.! Error: Data is having multiple json object: ##code## # -\*- coding: utf-8 -\*- import requests import json url … --- ## [Serilog and Filebeat Compatibility](https://discuss.elastic.co/t/serilog-and-filebeat-compatibility/280678)
**Author:** [@christamlyn](https://discuss.elastic.co/u/christamlyn)\ **Replies:** 1\ **Last updated:** [August 8, 2021, 11:14pm UTC](https://discuss.elastic.co/t/serilog-and-filebeat-compatibility/280678 "2021-08-08T23:14:48Z")
Quite specific to our setup but: we are currently using Filebeat to ship logs to Elasticsearch we want to slowly migrate to using Serilog sinks to ship to Elasticsearch we have a number of dashboards setup for the exis… --- ## [Parse a CSV file with grok logstash](https://discuss.elastic.co/t/parse-a-csv-file-with-grok-logstash/280726)
**Author:** [@Hind](https://discuss.elastic.co/u/Hind)\ **Replies:** 3\ **Last updated:** [August 8, 2021, 10:32pm UTC](https://discuss.elastic.co/t/parse-a-csv-file-with-grok-logstash/280726 "2021-08-08T22:32:19Z")
Hello! How could I parse a string that is between double quotes with grok logstash. "Kaathadimattam, Balacola Post, NEAR Siva Tea Factory, Ooty, 643203 Ooty, India – Great location -",ooty,India,..... thank you. --- ## [Array of objects - LogStash Conf](https://discuss.elastic.co/t/array-of-objects-logstash-conf/280750)
**Author:** [@me.mohammed](https://discuss.elastic.co/u/me.mohammed)\ **Replies:** 1\ **Last updated:** [August 8, 2021, 10:08pm UTC](https://discuss.elastic.co/t/array-of-objects-logstash-conf/280750 "2021-08-08T22:08:31Z")
I'm currently working on fetching my data from MySQL and pushing it to ElasticSearch through LogStash, although I'm kind of having an issue with the logstash configuration I basically have Products and each of those pro… --- ## [Issues with builtin network ML job high\_count\_network\_denies](https://discuss.elastic.co/t/issues-with-builtin-network-ml-job-high-count-network-denies/280635)
**Author:** [@willemdh](https://discuss.elastic.co/u/willemdh)\ **Replies:** 1\ **Last updated:** [August 8, 2021, 8:12pm UTC](https://discuss.elastic.co/t/issues-with-builtin-network-ml-job-high-count-network-denies/280635 "2021-08-08T20:12:09Z")
Hello, Just tried the new builtin network ML jobs.. First of all, our Elastic stack crashed after adding the ML jobs through the wizard. After recovering one of the four jobs, high\_count\_network\_denies, seemed to not pr… --- ## [Parsing a Raw JSON file "Invalid Setting for json filter"](https://discuss.elastic.co/t/parsing-a-raw-json-file-invalid-setting-for-json-filter/280735)
**Author:** [@Wilks](https://discuss.elastic.co/u/Wilks)\ **Replies:** 4\ **Last updated:** [August 8, 2021, 7:38pm UTC](https://discuss.elastic.co/t/parsing-a-raw-json-file-invalid-setting-for-json-filter/280735 "2021-08-08T19:38:58Z")
Hi, I am trying to parse a simple raw JSON file but I am getting an error about invalid setting for JSON filter. I am wondering if its the way my filter is setup and the fact that its nested json? \[ERROR\] 2021-08-08 1… [Previous page](https://discuss.elastic.co/latest.md?page=1535) [Next page](https://discuss.elastic.co/latest.md?page=1537)