# Kibana does not show documents updated via index API **URL:** \ **Category:** Kibana\ **Created:** [August 28, 2019, 4:41pm UTC](https://discuss.elastic.co/t/kibana-does-not-show-documents-updated-via-index-api/197175 "2019-08-28T16:41:56Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![cmelzer](https://avatars.discourse-cdn.com/v4/letter/c/b5e925/32.png) [@cmelzer](https://discuss.elastic.co/u/cmelzer)\ **Post date:** [August 28, 2019, 4:41pm UTC](https://discuss.elastic.co/t/kibana-does-not-show-documents-updated-via-index-api/197175/1 "2019-08-28T16:41:57Z")
Hey Guys, I encountered an issue on my elasic installation. While moving the elastic installation from one server to an other one, I had an error in my logstash pipeline so a field was not correctly parsed as json. Now I would like to update those documents. As far as I found out, there are basically two ways to do so: Either use the update API which has the disadvantage (for my use case) that the documents will be merged together or use the index API to replace a hole document. Sidenote: I'm doing this with the elasticsearch-js library I was actually going for the index API und started replacing 3 documents (one by one) with the new ones. While the elasticsearch API does show me the updated documents, Kibana does not show them at all. I tried to find a document by id (via kibana) and no result was returned. I double checked the selected timeframe, refreshed the index but with no success. The document replaced by the index API is missing. I updated an other document using the update API. This document is still displayed in Kibana and also contains all the new fields I passed to the document (and all the existing ones as well). What am I doing wrong? Why can't Kibana fetch or display those documents? Kind regards, Chris ---
**Author:** ![wylie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wylie/32/81794_2.png) [@wylie](https://discuss.elastic.co/u/wylie)\ **Post date:** [August 28, 2019, 6:01pm UTC](https://discuss.elastic.co/t/kibana-does-not-show-documents-updated-via-index-api/197175/2 "2019-08-28T18:01:11Z")
Let me check if I understand your scenario: 1. You are using the Discover app to see individual documents in Kibana 2. You reindexed 3 documents and they are no longer showing in Discover 3. You used the update API on another document, which is still shown in Discover Can you give more detail on: - How did you reindex? - What kind of query did you run to verify the documents are in Elasticsearch? - What kind of query are you running through Discover? ---
**Author:** ![cmelzer](https://avatars.discourse-cdn.com/v4/letter/c/b5e925/32.png) [@cmelzer](https://discuss.elastic.co/u/cmelzer)\ **Post date:** [August 28, 2019, 6:54pm UTC](https://discuss.elastic.co/t/kibana-does-not-show-documents-updated-via-index-api/197175/3 "2019-08-28T18:54:47Z")
Hi Wylie, thanks for your reply. You're right in your questions (1-3). That's exactly what happened. How I did reindex: In Kibana -\> Management -\> Index Management -\> choose the index the document is in by clicking the checkbox and choosing "Refresh Index". I also tried flush index and clear index cache with no success. What kind of query I run to verify documents are in Elasticsearch: I made a POST Request to `/logstash-application-2019.07/_search` with the following request body: ``` "query" : { "match": { "_id": "ZE_g9GsBPHUZVl-pfzao" } } ``` Since I made copy of the original document I double checked if they match which is the case. Response from the API: ``` { "took": 0, "timed_out": false, "_shards": { "total": 5, "successful": 5, "skipped": 0, "failed": 0 }, "hits": { "total": 1, "max_score": 1.0, "hits": [ { "_index": "logstash-application-2019.07", "_type": "doc", "_id": "ZE_g9GsBPHUZVl-pfzao", "_score": 1.0, "_source": { ... ``` What kind of query I run through Dicover: I tried by entering the search query manually (`_id: "ZE_g9GsBPHUZVl-pfzao"`) and by using the "Add a filter" button selecting \_id | is | ZE\_g9GsBPHUZVl-pfzao In both cases time frame was set to "This year" which should be correct since the document was added in July and still has a timestamp from July. By the way: I'm using kibana and elasticsearch version 6.8.1 ---
**Author:** ![wylie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wylie/32/81794_2.png) [@wylie](https://discuss.elastic.co/u/wylie)\ **Post date:** [August 28, 2019, 7:04pm UTC](https://discuss.elastic.co/t/kibana-does-not-show-documents-updated-via-index-api/197175/4 "2019-08-28T19:04:52Z")
Thanks for the followup. I'm still unclear on which part is not working. You've said that there are 3 documents that were updated, but are not showing up any more. Can you go over the steps you took for those 3 documents that are problematic? ---
**Author:** ![cmelzer](https://avatars.discourse-cdn.com/v4/letter/c/b5e925/32.png) [@cmelzer](https://discuss.elastic.co/u/cmelzer)\ **Post date:** [August 28, 2019, 7:56pm UTC](https://discuss.elastic.co/t/kibana-does-not-show-documents-updated-via-index-api/197175/5 "2019-08-28T19:56:19Z")
What do you mean by that? ---
**Author:** ![wylie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wylie/32/81794_2.png) [@wylie](https://discuss.elastic.co/u/wylie)\ **Post date:** [August 28, 2019, 8:11pm UTC](https://discuss.elastic.co/t/kibana-does-not-show-documents-updated-via-index-api/197175/6 "2019-08-28T20:11:16Z")
If you are looking for help with finding the right way to update documents, I would need to know what you tried that didn't work. ---
**Author:** ![cmelzer](https://avatars.discourse-cdn.com/v4/letter/c/b5e925/32.png) [@cmelzer](https://discuss.elastic.co/u/cmelzer)\ **Post date:** [August 28, 2019, 8:42pm UTC](https://discuss.elastic.co/t/kibana-does-not-show-documents-updated-via-index-api/197175/7 "2019-08-28T20:42:37Z")
Okay, got it. - First, i searched a document which wasn't parsed correctly due to the error in my logstash configuration. As I didn't wanted to update 33k documents without knowing if it will work out, I picked 3 (one at a time). The one in my example will be the document with id `ZE_g9GsBPHUZVl-pfzao` - Since I was not able to find a way to parse JSON with an `_update_by_query` call and `painless` scripts, I wrote a nodejs script to update the documents using the elasticsearch-js library ([https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/6.x/index.html](https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/6.x/index.html)) I initialized the client by `const client = new Client({ node: 'http://localhost:9200' })`, then searched for the document by id: ``` client.search({ index: 'game-of-thrones', body: { query: { match: { _id: 'ZE_g9GsBPHUZVl-pfzao' } } } }, (error, response) => { const document = response.hits.hits[0]['_source']; // logic to update the document client.index({ id: "ZE_g9GsBPHUZVl-pfzao", index: "logstash-application-2019.07, type: "doc", body: document }, (error, response) => { console.log(response); // this gave me a success and status 200 }); }); ``` When searching for the id with elasticsearch api, the api returned the updated document but kibana didn't. I just tried the exact same thing with another document but instead of using the javascrip library, i used the elasticsearch api. No problem there, everything went fine and the document is still in kibana. ---
**Author:** ![wylie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wylie/32/81794_2.png) [@wylie](https://discuss.elastic.co/u/wylie)\ **Post date:** [August 28, 2019, 10:18pm UTC](https://discuss.elastic.co/t/kibana-does-not-show-documents-updated-via-index-api/197175/8 "2019-08-28T22:18:51Z")
Okay, I think I see what you are trying to do, but let me confirm that you have already: 1. You previously used logstash to ingest data, but your index mapping was wrong 2. You changed the mapping of the index using the Management JSON editor, and it succeeded (there are cases where you are not allowed to remap existing indices) 3. Your documents already contain the right information in \_source, but need to be reindexed so that they are searchable and aggregatable. If that's your scenario, then \_update\_by\_query should have worked. \_update\_by\_query doesn't rewrite the \_source of your documents, but it does reindex and make searchable based on the new mappings. You can validate this by running an aggregation query against your new mapping, such as: ```auto POST logstash-application-2019.07/_search { "query": { "match": { "_id": "ZE_g9GsBPHUZVl-pfzao" } }, "aggs": { ... }, "size": 0 } ``` If that's not your scenario, then you probably need to change the \_source field of some documents. You could do this by overwriting documents using `PUT logstash-application-2019.07/_doc/ZE_g9GsBPHUZVl-pfzao`, or you could use a Painless script on the \_update\_by\_query request to rewrite the source. There are several examples of this in the Painless Update By Query docs: [https://www.elastic.co/guide/en/elasticsearch/painless/current/painless-update-by-query-context.html](https://www.elastic.co/guide/en/elasticsearch/painless/current/painless-update-by-query-context.html) ---
**Author:** ![cmelzer](https://avatars.discourse-cdn.com/v4/letter/c/b5e925/32.png) [@cmelzer](https://discuss.elastic.co/u/cmelzer)\ **Post date:** [August 29, 2019, 5:44am UTC](https://discuss.elastic.co/t/kibana-does-not-show-documents-updated-via-index-api/197175/9 "2019-08-29T05:44:57Z")
1. Yes, usually these data are mapped by logstash. 2. I wasn't able to update the index mapping. 3. Yes and no. My documents contain the information but the they are stored in a JSON string. So for example my document looks like this: ``` { "logContext": "{\"app\":\"my_app\",\"platform\":\"de\",\"requestId\":\"my_app.5d2c418738a4e1.01696328\",\"crossAppRequestId\":\"global.5d2c418738a451.59829866\",\"userId\":16584,\"order\":{\"id\": 1122334455,\"article\":{\"id\":12345, [...]}}}" } ``` Now what I need to do is to parse this string stored in `logContext` so I get the JSON object and this object then should be stored inside the `context` field. So after the update my document will look like this: ``` { "context": { "app: "my_app", "platform: "de", "requestId: "my_app.5d2c418738a4e1.01696328", [...] "order: { "id": 1122334455, "article": { "id": 12345 } } } } ``` When searching the documentation for any hint on how I would be able to parse a string as JSON (something like JSON.parse(jsonString) ) I was not able to find anything. I had a look at the `painless` language reference but without success. Thats why I was going for the JavaScript library. Anyway: As you suggested, I tried running an aggregation query. There was no result. My query: ``` "aggs": { "articles": { "terms": { "field": "context.order.id" } } } ``` returned me an empty bucket list. As mentioned: There are 2 other documents I updated. One by the updateApi of the JavaScript library and one via index API of elasticsearch (without the JavaScript involved) - same query works out for both documents and returned the buckets. ---
**Author:** ![cmelzer](https://avatars.discourse-cdn.com/v4/letter/c/b5e925/32.png) [@cmelzer](https://discuss.elastic.co/u/cmelzer)\ **Post date:** [August 29, 2019, 7:41am UTC](https://discuss.elastic.co/t/kibana-does-not-show-documents-updated-via-index-api/197175/10 "2019-08-29T07:41:35Z")
Hi Wylie, I was able to solve my problem. Because of the back and forth in my JavaScript with update and index Api I somehow mixed up the request body. While the following seems to works for an `client.update`: ``` client.update({ id: "ZE_g9GsBPHUZVl-pfzao", index: 'logstash-application-2019.07', type: 'doc', body: { doc: newDocument // note the 'doc' here } }) ``` With an `client.index({})` it must be ``` client.index({ id: "ZE_g9GsBPHUZVl-pfzao", index: "logstash-application-2019.07, type: "doc", body: newDocument // note that the document object is passed directly to the body }) ``` I missed this while switching between index and update. Because of this error, the documents now have an additional layer, so instead of: ``` { ..., "_source": { "app": "my_app", ... } } ``` I got: ``` { ..., "_source": { "document": { // note the "document" "app": "my_app", ... } } } ``` After the change, the documents will appear in Kibana and things work out as they should. I'm sorry for my mistake. Thank you so much for your help and your quick replies! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 26, 2019, 7:42am UTC](https://discuss.elastic.co/t/kibana-does-not-show-documents-updated-via-index-api/197175/11 "2019-09-26T07:42:11Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.