# Elasticsearch **URL:** https://discuss.elastic.co/c/elastic-stack/elasticsearch/6.md?page=1332 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1333 --- ## [Shards Bulk exception](https://discuss.elastic.co/t/shards-bulk-exception/176610)
**Author:** [@aniaks](https://discuss.elastic.co/u/aniaks)\ **Replies:** 8\ **Last updated:** [April 14, 2019, 11:55am UTC](https://discuss.elastic.co/t/shards-bulk-exception/176610 "2019-04-14T11:55:18Z")
See loads of bulk exceptions .... could it be due to high number of shards ... we have 24 data nodes 10,800 shards ? and discovery.zen.minimum\_master\_nodes = 13 queue size = 410 let me know ? --- ## [@timestamp parsing failure](https://discuss.elastic.co/t/timestamp-parsing-failure/176779)
**Author:** [@bkasrai](https://discuss.elastic.co/u/bkasrai)\ **Replies:** 0\ **Last updated:** [April 14, 2019, 11:54am UTC](https://discuss.elastic.co/t/timestamp-parsing-failure/176779 "2019-04-14T11:54:23Z")
Hi All, I was going through trying to find something relevant to the issue I am experiencing however I have been unable to locate one to my specific use case-I was hoping for some guidance here as this is one I have not… --- ## [How data is getting populated in elasticsearch while new installation like kibana flight and all?](https://discuss.elastic.co/t/how-data-is-getting-populated-in-elasticsearch-while-new-installation-like-kibana-flight-and-all/176768)
**Author:** [@pathri](https://discuss.elastic.co/u/pathri)\ **Replies:** 0\ **Last updated:** [April 14, 2019, 8:48am UTC](https://discuss.elastic.co/t/how-data-is-getting-populated-in-elasticsearch-while-new-installation-like-kibana-flight-and-all/176768 "2019-04-14T08:48:03Z")
kibana sample data for ecommerce , flight and logs indexes, how they got created? is it using logstash or by curl commands or any other way?? --- ## [Want to upgrade Elasticsearch from 5.4.3 to 7](https://discuss.elastic.co/t/want-to-upgrade-elasticsearch-from-5-4-3-to-7/176757)
**Author:** [@amitmjadav](https://discuss.elastic.co/u/amitmjadav)\ **Replies:** 2\ **Last updated:** [April 14, 2019, 7:40am UTC](https://discuss.elastic.co/t/want-to-upgrade-elasticsearch-from-5-4-3-to-7/176757 "2019-04-14T07:40:52Z")
Hi, I am planning to upgrade my elasticsearch cluster from version 5.4.3 to the latest 7. I am running all my nodes through .bat file. So any thing specially regarding confuguration i have to worry about ?? Or just conf… --- ## [Object mapping](https://discuss.elastic.co/t/object-mapping/173411)
**Author:** [@pmarques](https://discuss.elastic.co/u/pmarques)\ **Replies:** 2\ **Last updated:** [April 14, 2019, 5:47am UTC](https://discuss.elastic.co/t/object-mapping/173411 "2019-04-14T05:47:44Z")
Hello All, I am using the version 6.3.2 instead of the latest stable one. The reason I did this was to use (test) the kibana-object-format plugin (https://github.com/istresearch/kibana-object-format). The plugin was in… --- ## [How to indexing realtime data in to Elasticsearch?](https://discuss.elastic.co/t/how-to-indexing-realtime-data-in-to-elasticsearch/176662)
**Author:** [@Mohamed\_eshaftri](https://discuss.elastic.co/u/Mohamed_eshaftri)\ **Replies:** 2\ **Last updated:** [April 13, 2019, 7:49pm UTC](https://discuss.elastic.co/t/how-to-indexing-realtime-data-in-to-elasticsearch/176662 "2019-04-13T19:49:29Z")
hello, i am new Elasticsearch user, i would like to know how i can indexing realtime data from webSocket API and index this data, using Javascript. even some link page to start, that will be helpful. thank you. --- ## [Unable to start elasticsearch on Ubuntu 18](https://discuss.elastic.co/t/unable-to-start-elasticsearch-on-ubuntu-18/176738)
**Author:** [@johndowe](https://discuss.elastic.co/u/johndowe)\ **Replies:** 2\ **Last updated:** [April 13, 2019, 8:08pm UTC](https://discuss.elastic.co/t/unable-to-start-elasticsearch-on-ubuntu-18/176738 "2019-04-13T20:08:05Z")
I have installed Elasticsearch 7 on Ubuntu VERSION="18.04.2 LTS (Bionic Beaver)" and I am unable to start the service after attempting a million options I always get the same error : root@PiEKStack:/var/log/elasticsear… --- ## [Inconsistent Behaviors between Different Runs of ES-Spark Connector](https://discuss.elastic.co/t/inconsistent-behaviors-between-different-runs-of-es-spark-connector/174649)
**Author:** [@qixinwm2010](https://discuss.elastic.co/u/qixinwm2010)\ **Replies:** 2\ **Last updated:** [April 12, 2019, 12:22am UTC](https://discuss.elastic.co/t/inconsistent-behaviors-between-different-runs-of-es-spark-connector/174649 "2019-04-12T00:22:10Z")
Hello Community, We are doing ES indexing using the ES-SPark Connector (6.6.1) in our daily spark job. Here is our API call we make in our spark code df.write .format("org.elasticsearch.spark.sql") .option("es.resourc… --- ## [Curator should be able to shrink indices by size](https://discuss.elastic.co/t/curator-should-be-able-to-shrink-indices-by-size/176740)
**Author:** [@DewinGoh](https://discuss.elastic.co/u/DewinGoh)\ **Replies:** 1\ **Last updated:** [April 13, 2019, 7:25pm UTC](https://discuss.elastic.co/t/curator-should-be-able-to-shrink-indices-by-size/176740 "2019-04-13T19:25:50Z")
Hi all, I've been experimenting with Elasticsearch and curator for a bit, but am stuck on figuring out how to shrink indices by their sizes. My current problem is as such: I have been handed a cluster with over 2000 sha… --- ## [How to listen for new index creation events?](https://discuss.elastic.co/t/how-to-listen-for-new-index-creation-events/176748)
**Author:** [@tomb23](https://discuss.elastic.co/u/tomb23)\ **Replies:** 0\ **Last updated:** [April 13, 2019, 7:15pm UTC](https://discuss.elastic.co/t/how-to-listen-for-new-index-creation-events/176748 "2019-04-13T19:15:56Z")
Hi, Creating a plugin that performs certain actions whenever an index is created and can't seem to find the best place to register a listener for that. I see onIndexModule which seems to happen for creation events, but … --- ## [Elasticsearch is asking for authentication after running .bat file](https://discuss.elastic.co/t/elasticsearch-is-asking-for-authentication-after-running-bat-file/174629)
**Author:** [@Ashwin\_srivatsa](https://discuss.elastic.co/u/Ashwin_srivatsa)\ **Replies:** 8\ **Last updated:** [April 13, 2019, 1:20pm UTC](https://discuss.elastic.co/t/elasticsearch-is-asking-for-authentication-after-running-bat-file/174629 "2019-04-13T13:20:45Z")
I am trying elasticsearch for the first time but when i hit http://localhost:9200 it is asking me to authenticate using username and password. how do i get rid of it. thanks --- ## [Extracting the higher level domains from the domain using pipeline](https://discuss.elastic.co/t/extracting-the-higher-level-domains-from-the-domain-using-pipeline/176734)
**Author:** [@Babak](https://discuss.elastic.co/u/Babak)\ **Replies:** 1\ **Last updated:** [April 13, 2019, 12:47pm UTC](https://discuss.elastic.co/t/extracting-the-higher-level-domains-from-the-domain-using-pipeline/176734 "2019-04-13T12:47:43Z")
how to construct higher level domains of a domain in elasticsearch pipeline. example: input data: { "domain": "test.main.site.com" } after extracting we have this: { "domain": "test.user.site.net", "higherLeve… --- ## [Searching domain and subdomain returns more hits than searching for just the domain itself](https://discuss.elastic.co/t/searching-domain-and-subdomain-returns-more-hits-than-searching-for-just-the-domain-itself/176579)
**Author:** [@aqiank](https://discuss.elastic.co/u/aqiank)\ **Replies:** 2\ **Last updated:** [April 13, 2019, 11:57am UTC](https://discuss.elastic.co/t/searching-domain-and-subdomain-returns-more-hits-than-searching-for-just-the-domain-itself/176579 "2019-04-13T11:57:49Z")
Hi guys, I'm trying to make the number of hits the same when I do the queries "example.com" and "example.com mail.example.com". However when I run "example.com", it returns less hits than "example.com mail.example.com" o… --- ## [Error while creating snapshot to s3 repository](https://discuss.elastic.co/t/error-while-creating-snapshot-to-s3-repository/176731)
**Author:** [@pranay\_sankpal](https://discuss.elastic.co/u/pranay_sankpal)\ **Replies:** 1\ **Last updated:** [April 13, 2019, 11:41am UTC](https://discuss.elastic.co/t/error-while-creating-snapshot-to-s3-repository/176731 "2019-04-13T11:41:07Z")
I've created s3 snapshot repository PUT \_snapshot/s3\_repository/snap1 { "indices": "pe-queue-jobs-new1-2019.03.30", "ignore\_unavailable": true, "include\_global\_state": false } I'm getting the following error … --- ## [Indexer cluster separate and Searcher cluster separate](https://discuss.elastic.co/t/indexer-cluster-separate-and-searcher-cluster-separate/175574)
**Author:** [@setprem](https://discuss.elastic.co/u/setprem)\ **Replies:** 8\ **Last updated:** [April 13, 2019, 11:05am UTC](https://discuss.elastic.co/t/indexer-cluster-separate-and-searcher-cluster-separate/175574 "2019-04-13T11:05:11Z")
Hi all, Want to try two different clusters sharing same data path (same index). The reason is, one cluster will keep on indexing streaming data. The other cluster will take care of all the searches and aggregations. So … --- ## [Minimum should match percentage not returning desired results](https://discuss.elastic.co/t/minimum-should-match-percentage-not-returning-desired-results/176723)
**Author:** [@ashit\_pupu](https://discuss.elastic.co/u/ashit_pupu)\ **Replies:** 5\ **Last updated:** [April 13, 2019, 10:45am UTC](https://discuss.elastic.co/t/minimum-should-match-percentage-not-returning-desired-results/176723 "2019-04-13T10:45:21Z")
I am looking at a weird behavior while doing a multimatch query with minimum\_should\_match percentage. Giving a sample use case, while searching for New York in multimatch query and then giving minimum\_should\_match as 90%… --- ## [Elasticsearch Cluster Monitoring](https://discuss.elastic.co/t/elasticsearch-cluster-monitoring/176699)
**Author:** [@Osama\_Tariq](https://discuss.elastic.co/u/Osama_Tariq)\ **Replies:** 1\ **Last updated:** [April 13, 2019, 8:05am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-monitoring/176699 "2019-04-13T08:05:10Z")
Hi, When I execute this script, It created the indices for each day e.g apm-2017-11-28 , apm-2017-11-29 the actual index is apm. I want to prevent them to create indices for each day. Actually I want to monitor elasti… --- ## [Benchmarking High Volumes](https://discuss.elastic.co/t/benchmarking-high-volumes/176690)
**Author:** [@rocksnslopes](https://discuss.elastic.co/u/rocksnslopes)\ **Replies:** 1\ **Last updated:** [April 13, 2019, 7:46am UTC](https://discuss.elastic.co/t/benchmarking-high-volumes/176690 "2019-04-13T07:46:15Z")
Hi - I would like to benchmark high volume requests using esrally. I'm trying to max somewhere between 500k-1M RPM. I've modified an existing challenge and changed the client and target-throughput values to match 500k… --- ## [How to clean field which contains a date value](https://discuss.elastic.co/t/how-to-clean-field-which-contains-a-date-value/175988)
**Author:** [@nsam](https://discuss.elastic.co/u/nsam)\ **Replies:** 1\ **Last updated:** [April 13, 2019, 7:23am UTC](https://discuss.elastic.co/t/how-to-clean-field-which-contains-a-date-value/175988 "2019-04-13T07:23:59Z")
I have documents like { "mydate" : "\\/Date(1553767871903)\\/", "myvalue" : 100 } in field mydate, date exists in epoch\_second (/Date(1553767871903)/). How can I clean up the characters I dont need and set field mydat… --- ## [Index from multi indexes with elasticsearch-PHP](https://discuss.elastic.co/t/index-from-multi-indexes-with-elasticsearch-php/176712)
**Author:** [@evahsiao](https://discuss.elastic.co/u/evahsiao)\ **Replies:** 1\ **Last updated:** [April 13, 2019, 3:36am UTC](https://discuss.elastic.co/t/index-from-multi-indexes-with-elasticsearch-php/176712 "2019-04-13T03:36:24Z")
here is my problem: I have many messages data in elasticsearch db, these data are stored in the same type but different index according to the date. for example, all the messages today are stored in the message\_2019-04-… --- ## [Log file rotation issue on Elasticsearch 6.7.0](https://discuss.elastic.co/t/log-file-rotation-issue-on-elasticsearch-6-7-0/176709)
**Author:** [@vijayakrishna.rg](https://discuss.elastic.co/u/vijayakrishna.rg)\ **Replies:** 0\ **Last updated:** [April 12, 2019, 11:24pm UTC](https://discuss.elastic.co/t/log-file-rotation-issue-on-elasticsearch-6-7-0/176709 "2019-04-12T23:24:44Z")
Elasticsearch version: 6.7.0 We have an issue with cluster log file rotation, it's creating log file every 10 mins and rolling the old one to \*.log.gz i have below setting in log4j2.properties appender.rolling.type = … --- ## [Performance optimization (indexing & search) for single node (master thesis)](https://discuss.elastic.co/t/performance-optimization-indexing-search-for-single-node-master-thesis/176621)
**Author:** [@nicolaipre](https://discuss.elastic.co/u/nicolaipre)\ **Replies:** 2\ **Last updated:** [April 12, 2019, 8:33pm UTC](https://discuss.elastic.co/t/performance-optimization-indexing-search-for-single-node-master-thesis/176621 "2019-04-12T20:33:13Z")
Hi. I am currently writing my master thesis on database comparions, and achieving rapid search results when dealing with large amounts of data (billions of records). The reason I chose this topic was because I was quit… --- ## [Substring and contains in painless](https://discuss.elastic.co/t/substring-and-contains-in-painless/176294)
**Author:** [@Jravella](https://discuss.elastic.co/u/Jravella)\ **Replies:** 8\ **Last updated:** [April 12, 2019, 8:30pm UTC](https://discuss.elastic.co/t/substring-and-contains-in-painless/176294 "2019-04-12T20:30:03Z")
i have a internal uri which has one of values like /appname/api/v4/bdq/rules/ i want to filter them a nd extract only field v4 from that url using substring function --- ## [Accessing \_id in ingest pipeline](https://discuss.elastic.co/t/accessing-id-in-ingest-pipeline/176503)
**Author:** [@Oleg-Arkhipov](https://discuss.elastic.co/u/Oleg-Arkhipov)\ **Replies:** 6\ **Last updated:** [April 12, 2019, 8:23pm UTC](https://discuss.elastic.co/t/accessing-id-in-ingest-pipeline/176503 "2019-04-12T20:23:55Z")
Search After documentation (as well as one or two other places in docs) state that aggregation and sorting on \_id field is inefficient, and it is better to create a duplicate of \_id in another ordinary field with doc\_va… --- ## [Truncate part of an existing url field value to a new field using painless?](https://discuss.elastic.co/t/truncate-part-of-an-existing-url-field-value-to-a-new-field-using-painless/176255)
**Author:** [@Jravella](https://discuss.elastic.co/u/Jravella)\ **Replies:** 3\ **Last updated:** [April 12, 2019, 7:25pm UTC](https://discuss.elastic.co/t/truncate-part-of-an-existing-url-field-value-to-a-new-field-using-painless/176255 "2019-04-12T19:25:58Z")
i have a requirement where i have to trim field based on url i have to trim until rules /someapi/api/v4/bdq/rules/16869 --- ## [Plugin Custom - Multi Source?](https://discuss.elastic.co/t/plugin-custom-multi-source/175439)
**Author:** [@EliuFlorez](https://discuss.elastic.co/u/EliuFlorez)\ **Replies:** 1\ **Last updated:** [April 12, 2019, 6:50pm UTC](https://discuss.elastic.co/t/plugin-custom-multi-source/175439 "2019-04-12T18:50:50Z")
Hi guys, I've been almost 1 month trying to migrate a plugin developed in 5.4 from elasticsearch to 6.6. What I want is to establish different sources of which in each one the dates entered are verified to show the avail… --- ## [Displaying Specific Fields in Search Results ES7](https://discuss.elastic.co/t/displaying-specific-fields-in-search-results-es7/176685)
**Author:** [@Vincent\_Stevenson](https://discuss.elastic.co/u/Vincent_Stevenson)\ **Replies:** 2\ **Last updated:** [April 12, 2019, 6:32pm UTC](https://discuss.elastic.co/t/displaying-specific-fields-in-search-results-es7/176685 "2019-04-12T18:32:07Z")
In ES 6.7, I could execute the following: GET /my\_index/\_search { "\_source": "name", "query": { "query\_string": { "query": "Columbia" } } } And my search results would only show me the Name field. When I do th… --- ## [ILM and indexing older logs after a period of downtime](https://discuss.elastic.co/t/ilm-and-indexing-older-logs-after-a-period-of-downtime/176526)
**Author:** [@justinw](https://discuss.elastic.co/u/justinw)\ **Replies:** 2\ **Last updated:** [April 12, 2019, 6:27pm UTC](https://discuss.elastic.co/t/ilm-and-indexing-older-logs-after-a-period-of-downtime/176526 "2019-04-12T18:27:10Z")
Hi, I'm about to transition our log collection fleet to using ILM. Looks promising, but I have a question. Currently we use time (by day) based indices. Filebeat sends to localhost logstash, and logstash bulk indexes t… --- ## [Querying Nested Object returns all objects](https://discuss.elastic.co/t/querying-nested-object-returns-all-objects/176632)
**Author:** [@bhaumik55231](https://discuss.elastic.co/u/bhaumik55231)\ **Replies:** 2\ **Last updated:** [April 12, 2019, 6:22pm UTC](https://discuss.elastic.co/t/querying-nested-object-returns-all-objects/176632 "2019-04-12T18:22:33Z")
I was testing the example given on elasticsearch nested object documentation. In that example, why it returns both records even though i'm querying for values that only matches with 1 record. Please see below: - Mapping… --- ## [How to query based on month name](https://discuss.elastic.co/t/how-to-query-based-on-month-name/176563)
**Author:** [@niraj\_pandey](https://discuss.elastic.co/u/niraj_pandey)\ **Replies:** 1\ **Last updated:** [April 12, 2019, 6:04pm UTC](https://discuss.elastic.co/t/how-to-query-based-on-month-name/176563 "2019-04-12T18:04:16Z")
Hi , Is there any way to pass month name for timestamp field to query the data for a specific month ? Like suppose I want to query some data for entire march month currently I am passing the timestamp in the following … [Previous page](https://discuss.elastic.co/c/elastic-stack/elasticsearch/6.md?page=1331) [Next page](https://discuss.elastic.co/c/elastic-stack/elasticsearch/6.md?page=1333)