# Elasticsearch **URL:** https://discuss.elastic.co/c/elastic-stack/elasticsearch/6.md?page=1609 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1610 --- ## [Date Invalid format too short](https://discuss.elastic.co/t/date-invalid-format-too-short/141046)
**Author:** [@magedmakled](https://discuss.elastic.co/u/magedmakled)\ **Replies:** 3\ **Last updated:** [July 22, 2018, 8:22pm UTC](https://discuss.elastic.co/t/date-invalid-format-too-short/141046 "2018-07-22T20:22:28Z")
I have an elasticsearch model with a field starts\_at with the mapping "starts\_at": { "type": "date", "format": "yyyy-MM-dd'T'HH:mm:ssZ" } When I try to enter 2018-07-29T13:00:… --- ## [Concept MS-SQL and Elasticsearch](https://discuss.elastic.co/t/concept-ms-sql-and-elasticsearch/141045)
**Author:** [@hispeed](https://discuss.elastic.co/u/hispeed)\ **Replies:** 1\ **Last updated:** [July 22, 2018, 7:45pm UTC](https://discuss.elastic.co/t/concept-ms-sql-and-elasticsearch/141045 "2018-07-22T19:45:09Z")
Hi @ all, First of all I thank you for the support you gave me. I have learned a lot and all was for private purpose. For my school I have to write something about: Dynamic analysis and I have to find and compare diffe… --- ## [Aggregation document count issue](https://discuss.elastic.co/t/aggregation-document-count-issue/141031)
**Author:** [@Kushal\_Goyal](https://discuss.elastic.co/u/Kushal_Goyal)\ **Replies:** 3\ **Last updated:** [July 22, 2018, 7:41pm UTC](https://discuss.elastic.co/t/aggregation-document-count-issue/141031 "2018-07-22T19:41:18Z")
Hi I am new in ELK stack, need your advice. I am using nested aggregation. GET account\_credit/person/\_search { "aggs": { "count\_by\_gender": { "terms": { "field": "creditcard\_number.keyword" }, "aggs": { … --- ## [Elasticsearch Index Completely Missing](https://discuss.elastic.co/t/elasticsearch-index-completely-missing/140640)
**Author:** [@hacker\_21](https://discuss.elastic.co/u/hacker_21)\ **Replies:** 13\ **Last updated:** [July 22, 2018, 4:34pm UTC](https://discuss.elastic.co/t/elasticsearch-index-completely-missing/140640 "2018-07-22T16:34:34Z")
Hey there, I've had an elasticsearch cluster running since late June. My search wasn't working today and when I looked at the app I was getting this error: elasticsearch.exceptions.NotFoundError: TransportError(404, '… --- ## [Is it possible to do disjoint queries or buckets?](https://discuss.elastic.co/t/is-it-possible-to-do-disjoint-queries-or-buckets/141033)
**Author:** [@pulse14](https://discuss.elastic.co/u/pulse14)\ **Replies:** 0\ **Last updated:** [July 22, 2018, 2:39pm UTC](https://discuss.elastic.co/t/is-it-possible-to-do-disjoint-queries-or-buckets/141033 "2018-07-22T14:39:09Z")
Hello, I have multiples simple bool queries Q1, Q2, .... Q100. (With 2 terms queries inside, and random scoring) Each query is associated to an integer N1, N2, ....N100. I need to query N1 docs matching Q1, N2 docs ma… --- ## [Hardware Configuration Setup for ELK](https://discuss.elastic.co/t/hardware-configuration-setup-for-elk/141019)
**Author:** [@sekharangadala](https://discuss.elastic.co/u/sekharangadala)\ **Replies:** 3\ **Last updated:** [July 22, 2018, 12:30pm UTC](https://discuss.elastic.co/t/hardware-configuration-setup-for-elk/141019 "2018-07-22T12:30:55Z")
Hi, We are trying to setup ELK in production to gather all our postfix-mail logs to show in an UI for the email delivery questions. Please let me know the hardware configuration required for the setup. #we have 300 se… --- ## [ELK It's broken, rerouting shards:high disk watermark exceeded on one or more nodes](https://discuss.elastic.co/t/elk-its-broken-rerouting-shards-high-disk-watermark-exceeded-on-one-or-more-nodes/140991)
**Author:** [@omid](https://discuss.elastic.co/u/omid)\ **Replies:** 6\ **Last updated:** [July 22, 2018, 12:28pm UTC](https://discuss.elastic.co/t/elk-its-broken-rerouting-shards-high-disk-watermark-exceeded-on-one-or-more-nodes/140991 "2018-07-22T12:28:03Z")
my elk elasticsearch, kibana, logstash, is up services but elasticsearch log is : \> \[2018-07-21T15:53:40,353\]\[WARN \]\[o.e.c.r.a.DiskThresholdMonitor\] \[\_POGvYv\] high disk watermark \[90%\] exceeded on \[\_POGvYvXQsK4h8… --- ## [How can i filter on my suggest list](https://discuss.elastic.co/t/how-can-i-filter-on-my-suggest-list/140989)
**Author:** [@Jacques\_du\_Plessis](https://discuss.elastic.co/u/Jacques_du_Plessis)\ **Replies:** 1\ **Last updated:** [July 22, 2018, 10:49am UTC](https://discuss.elastic.co/t/how-can-i-filter-on-my-suggest-list/140989 "2018-07-22T10:49:40Z")
I have read quite a few articles saying that you can not combine queries with suggesters. https://github.com/elastic/elasticsearch/issues/19143 is this still true? (old) If so, what would be the best way to filter on t… --- ## [Upgraded ES 6.3.1 cluster has only .kibana index](https://discuss.elastic.co/t/upgraded-es-6-3-1-cluster-has-only-kibana-index/140971)
**Author:** [@eswenson](https://discuss.elastic.co/u/eswenson)\ **Replies:** 2\ **Last updated:** [July 22, 2018, 5:29am UTC](https://discuss.elastic.co/t/upgraded-es-6-3-1-cluster-has-only-kibana-index/140971 "2018-07-22T05:29:03Z")
I upgraded a cluster from 2.4 to 5.6.10 and then from 5.6.10 to 6.3.1. After the upgrade of 5.6.10, I ran the upgrade helper and reindexed all the indices that had originally been created in 2.4 as per the instructions. … --- ## [Data loss ,bulk rejected high in elastic search](https://discuss.elastic.co/t/data-loss-bulk-rejected-high-in-elastic-search/140997)
**Author:** [@manish\_jaiswal](https://discuss.elastic.co/u/manish_jaiswal)\ **Replies:** 2\ **Last updated:** [July 22, 2018, 5:18am UTC](https://discuss.elastic.co/t/data-loss-bulk-rejected-high-in-elastic-search/140997 "2018-07-22T05:18:08Z")
I am sending data from filebeat (6.3) from 138 servers to elastic search 6.3.( 3 master and 17 data nodes with (6data node 30gb memory and 11 with 15gb memeory)) getting bulk failure in filebeat when checking in elastic… --- ## [Hardware requirements for Elasticsearch 6.3.1](https://discuss.elastic.co/t/hardware-requirements-for-elasticsearch-6-3-1/140919)
**Author:** [@adrwer](https://discuss.elastic.co/u/adrwer)\ **Replies:** 6\ **Last updated:** [July 21, 2018, 4:12pm UTC](https://discuss.elastic.co/t/hardware-requirements-for-elasticsearch-6-3-1/140919 "2018-07-21T16:12:40Z")
I'm new to elasticsearch and I've installed the .tar.gz package on a Rhel7.5 instance on Amazon Web Services with a RAM of 1GB and volume of 10GB but when I run it with this command "./bin/elasticsearch" I get this erro… --- ## [Elasticsearch 6.3.1 xpack password enabling shows error](https://discuss.elastic.co/t/elasticsearch-6-3-1-xpack-password-enabling-shows-error/140999)
**Author:** [@rijinmp](https://discuss.elastic.co/u/rijinmp)\ **Replies:** 1\ **Last updated:** [July 21, 2018, 3:55pm UTC](https://discuss.elastic.co/t/elasticsearch-6-3-1-xpack-password-enabling-shows-error/140999 "2018-07-21T15:55:10Z")
My ELK 6.3.1 working well logs indexed to elasticsearch and these logs are vailable in kibana. Then i am added this line in elasticsearch.yml as last line xpack.security.enabled: true Then tried to create passwords f… --- ## [Documents deletion in shards](https://discuss.elastic.co/t/documents-deletion-in-shards/140992)
**Author:** [@Itay\_Bittan](https://discuss.elastic.co/u/Itay_Bittan)\ **Replies:** 0\ **Last updated:** [July 21, 2018, 11:39am UTC](https://discuss.elastic.co/t/documents-deletion-in-shards/140992 "2018-07-21T11:39:40Z")
I have initial index with 5,655,272 documents, divided to 4 shards. My python module deletes most of the documents and leave ~1.4M (out of 5.6M). Then I also run "merge segments" to improve index serving time. Now if … --- ## [How to ingest/push multiple BIG attachments into indexed document's array attribute in ElasticSearch?](https://discuss.elastic.co/t/how-to-ingest-push-multiple-big-attachments-into-indexed-documents-array-attribute-in-elasticsearch/140781)
**Author:** [@mbm-rafal](https://discuss.elastic.co/u/mbm-rafal)\ **Replies:** 5\ **Last updated:** [July 21, 2018, 5:43am UTC](https://discuss.elastic.co/t/how-to-ingest-push-multiple-big-attachments-into-indexed-documents-array-attribute-in-elasticsearch/140781 "2018-07-21T05:43:26Z")
Hey, using: ES 5.1 I'm trying to figure out the solution for indexing multiple attachments for single document into Elasticsearch Index. I have some limitations around my service (using AWS) that limits my HTTP reques… --- ## [Shards Failing](https://discuss.elastic.co/t/shards-failing/140978)
**Author:** [@Sulaiman](https://discuss.elastic.co/u/Sulaiman)\ **Replies:** 1\ **Last updated:** [July 21, 2018, 5:33am UTC](https://discuss.elastic.co/t/shards-failing/140978 "2018-07-21T05:33:14Z")
Recently I have upgrade ElasticSearch version "6.3.1" but I am facing too many shards failing with old data. I am using Elastic search with kibana,logstash, wazuh curl -XGET localhost:9200/\_cat/shards?h=index,shard,pri… --- ## [How to deserialize avro file and use in elastic search](https://discuss.elastic.co/t/how-to-deserialize-avro-file-and-use-in-elastic-search/140980)
**Author:** [@Sakthivel\_G](https://discuss.elastic.co/u/Sakthivel_G)\ **Replies:** 0\ **Last updated:** [July 21, 2018, 5:26am UTC](https://discuss.elastic.co/t/how-to-deserialize-avro-file-and-use-in-elastic-search/140980 "2018-07-21T05:26:11Z")
Hi I am trying to use ELK stack as a datawarehouse solution to process user data coming in Avro format and take needed aggregated reports without waiting for Legacy ETL jobs. Can you help me to guide on processing Avro… --- ## [Sorting with variable names instead of fixed fields](https://discuss.elastic.co/t/sorting-with-variable-names-instead-of-fixed-fields/140976)
**Author:** [@vandanachadha](https://discuss.elastic.co/u/vandanachadha)\ **Replies:** 0\ **Last updated:** [July 21, 2018, 3:38am UTC](https://discuss.elastic.co/t/sorting-with-variable-names-instead-of-fixed-fields/140976 "2018-07-21T03:38:14Z")
Sorting with a specified fieldname works. But Is there any way I can use a variable name to provide the sorting field. For example: This works: "sort": \[ { "venue.price": { "order": "asc", "miss… --- ## [Saving ES data folder vs snapshotting](https://discuss.elastic.co/t/saving-es-data-folder-vs-snapshotting/140959)
**Author:** [@seanziee](https://discuss.elastic.co/u/seanziee)\ **Replies:** 1\ **Last updated:** [July 21, 2018, 1:08am UTC](https://discuss.elastic.co/t/saving-es-data-folder-vs-snapshotting/140959 "2018-07-21T01:08:40Z")
I'm debating the pros and cons of either storing a backup of the entire data folder of each node or snapshotting the cluster to save the data. if I save the data folder for each node, if that node burns down and I sti… --- ## [Integrating REST API call into role template](https://discuss.elastic.co/t/integrating-rest-api-call-into-role-template/139790)
**Author:** [@ewalters-cspace](https://discuss.elastic.co/u/ewalters-cspace)\ **Replies:** 2\ **Last updated:** [July 20, 2018, 10:02pm UTC](https://discuss.elastic.co/t/integrating-rest-api-call-into-role-template/139790 "2018-07-20T22:02:27Z")
I am trying to implement an ABAC scheme, similar to the link below. I would like to add on certain functionality - my users authenticate through PKI and their attributes, instead of being hardcoded into Elasticsearch… --- ## [Possible to use GeoIP without Filebeat shipping logs?](https://discuss.elastic.co/t/possible-to-use-geoip-without-filebeat-shipping-logs/140607)
**Author:** [@brolo](https://discuss.elastic.co/u/brolo)\ **Replies:** 5\ **Last updated:** [July 20, 2018, 9:31pm UTC](https://discuss.elastic.co/t/possible-to-use-geoip-without-filebeat-shipping-logs/140607 "2018-07-20T21:31:23Z")
I am working on adding logging functionality to an internal tool used as a build environment manager/build orchestrator. I am currently sending data to ES by formatting a JSON with relevant data and using a POST request … --- ## [Elasticsearch-py: query a unique count](https://discuss.elastic.co/t/elasticsearch-py-query-a-unique-count/140957)
**Author:** [@mksavic](https://discuss.elastic.co/u/mksavic)\ **Replies:** 0\ **Last updated:** [July 20, 2018, 8:01pm UTC](https://discuss.elastic.co/t/elasticsearch-py-query-a-unique-count/140957 "2018-07-20T20:01:49Z")
Hello, I am using elasticsearch-py for a project. Currently I have a filter that needs to be translated into python code. system.process.name:my\_process && system.process.cmdline:special\_flag In Kibana, I have this fi… --- ## [Client.search or .get \_source response just shows Object](https://discuss.elastic.co/t/client-search-or-get-source-response-just-shows-object/140950)
**Author:** [@ryannelle](https://discuss.elastic.co/u/ryannelle)\ **Replies:** 1\ **Last updated:** [July 20, 2018, 7:49pm UTC](https://discuss.elastic.co/t/client-search-or-get-source-response-just-shows-object/140950 "2018-07-20T19:49:25Z")
Running a query from a js file executed by node.js. The query executes correctly but objects like system.auth.program, system.auth.message, don't display correctly and only show up as \_source: { system: { auth {: \[Obje… --- ## [Nested aggregation empty bucket](https://discuss.elastic.co/t/nested-aggregation-empty-bucket/139722)
**Author:** [@zajca](https://discuss.elastic.co/u/zajca)\ **Replies:** 4\ **Last updated:** [July 20, 2018, 7:41pm UTC](https://discuss.elastic.co/t/nested-aggregation-empty-bucket/139722 "2018-07-20T19:41:39Z")
Hi, I have problem with aggregation, which return no bucket. (it's more complicated I did drop useless stuff for this example, thats why it's competence inside competence without any other field) my mapping: { "m… --- ## [Best Web Crawler For Elastisearch](https://discuss.elastic.co/t/best-web-crawler-for-elastisearch/140881)
**Author:** [@katiyar](https://discuss.elastic.co/u/katiyar)\ **Replies:** 1\ **Last updated:** [July 20, 2018, 5:29pm UTC](https://discuss.elastic.co/t/best-web-crawler-for-elastisearch/140881 "2018-07-20T17:29:37Z")
Hi, I want to create search functionality using Elastisearch for my on-premises website. Could you please suggest me best web Crawler for Elastisearch ? --- ## [Sum of distinct document fields](https://discuss.elastic.co/t/sum-of-distinct-document-fields/140944)
**Author:** [@Olexij\_Tkatchenko](https://discuss.elastic.co/u/Olexij_Tkatchenko)\ **Replies:** 0\ **Last updated:** [July 20, 2018, 4:18pm UTC](https://discuss.elastic.co/t/sum-of-distinct-document-fields/140944 "2018-07-20T16:18:28Z")
I have many documents with distinct field buildId (containing duplicates), common field buildType and a filed duration. How do I calculate a sum of durations for distinct documents in the sense of buildId with matching c… --- ## [{"match": {"region": "EAST"}}](https://discuss.elastic.co/t/match-region-east/140926)
**Author:** [@Mahesh\_Bablu](https://discuss.elastic.co/u/Mahesh_Bablu)\ **Replies:** 2\ **Last updated:** [July 20, 2018, 4:16pm UTC](https://discuss.elastic.co/t/match-region-east/140926 "2018-07-20T16:16:49Z")
reference Can anyone please tell me the query for not matching. Like I need {"match": {"region": "EAST"}} exactly opposite. Eg: match regions without EAST --- ## [Distance for GeoPolygon](https://discuss.elastic.co/t/distance-for-geopolygon/140936)
**Author:** [@Dean\_Clarke](https://discuss.elastic.co/u/Dean_Clarke)\ **Replies:** 0\ **Last updated:** [July 20, 2018, 3:34pm UTC](https://discuss.elastic.co/t/distance-for-geopolygon/140936 "2018-07-20T15:34:45Z")
I'm trying to get the distance between a geopoint (my coordinate) and GeoPolygon indexed property (serviceAreas). Are there any scripting method that does this? In the instance of GeoPoint, I'm aware I can use a Scripted… --- ## [Indexing suggestions using analyzer](https://discuss.elastic.co/t/indexing-suggestions-using-analyzer/140934)
**Author:** [@Dean\_Clarke](https://discuss.elastic.co/u/Dean_Clarke)\ **Replies:** 0\ **Last updated:** [July 20, 2018, 3:30pm UTC](https://discuss.elastic.co/t/indexing-suggestions-using-analyzer/140934 "2018-07-20T15:30:46Z")
Good day: I'm trying to figure out how to index suggestion without splitting my text using a delimiter and storing it in the CompletionField: List\ inputs = new List\() { facility.City, … --- ## [New setup - 2nd node not connecting](https://discuss.elastic.co/t/new-setup-2nd-node-not-connecting/140862)
**Author:** [@L33T](https://discuss.elastic.co/u/L33T)\ **Replies:** 2\ **Last updated:** [July 20, 2018, 1:51pm UTC](https://discuss.elastic.co/t/new-setup-2nd-node-not-connecting/140862 "2018-07-20T13:51:09Z")
Hi, I have put together a small cluster for metricbeat. All is well so far but i cannot get the 2nd node to connect to the cluster, my config on the second node is: ======================== Elasticsearch Configuration … --- ## [Determinizing ... would result in more than 10000 states](https://discuss.elastic.co/t/determinizing-would-result-in-more-than-10000-states/140718)
**Author:** [@Destroy666](https://discuss.elastic.co/u/Destroy666)\ **Replies:** 1\ **Last updated:** [July 20, 2018, 12:31pm UTC](https://discuss.elastic.co/t/determinizing-would-result-in-more-than-10000-states/140718 "2018-07-20T12:31:54Z")
https://pastebin.com/034F5M01 I'm getting this error when trying to do an aggregation based on some IDs in 5.6.x, it seems to appear only sometimes. Could anyone explain what it means and what could be done to fix it? [Previous page](https://discuss.elastic.co/c/elastic-stack/elasticsearch/6.md?page=1608) [Next page](https://discuss.elastic.co/c/elastic-stack/elasticsearch/6.md?page=1610)