Kibana does not show documents updated via index API

cmelzer · August 28, 2019, 4:41pm

Hey Guys,

I encountered an issue on my elasic installation.
While moving the elastic installation from one server to an other one, I had an error in my logstash pipeline so a field was not correctly parsed as json.

Now I would like to update those documents. As far as I found out, there are basically two ways to do so: Either use the update API which has the disadvantage (for my use case) that the documents will be merged together or use the index API to replace a hole document. Sidenote: I'm doing this with the elasticsearch-js library

I was actually going for the index API und started replacing 3 documents (one by one) with the new ones. While the elasticsearch API does show me the updated documents, Kibana does not show them at all. I tried to find a document by id (via kibana) and no result was returned. I double checked the selected timeframe, refreshed the index but with no success. The document replaced by the index API is missing.

I updated an other document using the update API. This document is still displayed in Kibana and also contains all the new fields I passed to the document (and all the existing ones as well).

What am I doing wrong? Why can't Kibana fetch or display those documents?

Kind regards,
Chris

wylie · August 28, 2019, 6:01pm

Let me check if I understand your scenario:

You are using the Discover app to see individual documents in Kibana
You reindexed 3 documents and they are no longer showing in Discover
You used the update API on another document, which is still shown in Discover

Can you give more detail on:

How did you reindex?
What kind of query did you run to verify the documents are in Elasticsearch?
What kind of query are you running through Discover?

cmelzer · August 28, 2019, 6:54pm

Hi Wylie,

thanks for your reply. You're right in your questions (1-3). That's exactly what happened.

How I did reindex:
In Kibana -> Management -> Index Management -> choose the index the document is in by clicking the checkbox and choosing "Refresh Index". I also tried flush index and clear index cache with no success.

What kind of query I run to verify documents are in Elasticsearch:
I made a POST Request to /logstash-application-2019.07/_search with the following request body:

"query" : {
    "match": {
        "_id": "ZE_g9GsBPHUZVl-pfzao"
    }
}

Since I made copy of the original document I double checked if they match which is the case.
Response from the API:

{
    "took": 0,
    "timed_out": false,
    "_shards": {
        "total": 5,
        "successful": 5,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": 1,
        "max_score": 1.0,
        "hits": [
            {
                "_index": "logstash-application-2019.07",
                "_type": "doc",
                "_id": "ZE_g9GsBPHUZVl-pfzao",
                "_score": 1.0,
                "_source": { ...

What kind of query I run through Dicover:
I tried by entering the search query manually (_id: "ZE_g9GsBPHUZVl-pfzao") and by using the "Add a filter" button selecting _id | is | ZE_g9GsBPHUZVl-pfzao
In both cases time frame was set to "This year" which should be correct since the document was added in July and still has a timestamp from July.

By the way: I'm using kibana and elasticsearch version 6.8.1

wylie · August 28, 2019, 7:04pm

Thanks for the followup. I'm still unclear on which part is not working. You've said that there are 3 documents that were updated, but are not showing up any more. Can you go over the steps you took for those 3 documents that are problematic?

cmelzer · August 28, 2019, 7:56pm

What do you mean by that?

wylie · August 28, 2019, 8:11pm

If you are looking for help with finding the right way to update documents, I would need to know what you tried that didn't work.

cmelzer · August 28, 2019, 8:42pm

Okay, got it.

First, i searched a document which wasn't parsed correctly due to the error in my logstash configuration. As I didn't wanted to update 33k documents without knowing if it will work out, I picked 3 (one at a time). The one in my example will be the document with id ZE_g9GsBPHUZVl-pfzao
Since I was not able to find a way to parse JSON with an _update_by_query call and painless scripts, I wrote a nodejs script to update the documents using the elasticsearch-js library (https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/6.x/index.html)

I initialized the client by const client = new Client({ node: 'http://localhost:9200' }), then searched for the document by id:

client.search({
    index: 'game-of-thrones',
    body: {
        query: {
            match: {
                _id: 'ZE_g9GsBPHUZVl-pfzao'
            }
        }
    }
}, (error, response) => {
    const document = response.hits.hits[0]['_source'];
    // logic to update the document
    client.index({
        id: "ZE_g9GsBPHUZVl-pfzao",
        index: "logstash-application-2019.07,
        type: "doc",
        body: document
    }, (error, response) => {
        console.log(response); // this gave me a success and status 200
    });
});

When searching for the id with elasticsearch api, the api returned the updated document but kibana didn't.

I just tried the exact same thing with another document but instead of using the javascrip library, i used the elasticsearch api. No problem there, everything went fine and the document is still in kibana.

wylie · August 28, 2019, 10:18pm

Okay, I think I see what you are trying to do, but let me confirm that you have already:

You previously used logstash to ingest data, but your index mapping was wrong
You changed the mapping of the index using the Management JSON editor, and it succeeded (there are cases where you are not allowed to remap existing indices)
Your documents already contain the right information in _source, but need to be reindexed so that they are searchable and aggregatable.

If that's your scenario, then _update_by_query should have worked. _update_by_query doesn't rewrite the _source of your documents, but it does reindex and make searchable based on the new mappings. You can validate this by running an aggregation query against your new mapping, such as:

POST logstash-application-2019.07/_search
{
  "query": {
     "match": { "_id": "ZE_g9GsBPHUZVl-pfzao" }
  },
  "aggs": {
     ...
  },
  "size": 0
}

If that's not your scenario, then you probably need to change the _source field of some documents.
You could do this by overwriting documents using PUT logstash-application-2019.07/_doc/ZE_g9GsBPHUZVl-pfzao, or you could use a Painless script on the _update_by_query request to rewrite the source. There are several examples of this in the Painless Update By Query docs: https://www.elastic.co/guide/en/elasticsearch/painless/current/painless-update-by-query-context.html

cmelzer · August 29, 2019, 5:44am

Yes, usually these data are mapped by logstash.
I wasn't able to update the index mapping.
Yes and no. My documents contain the information but the they are stored in a JSON string.

So for example my document looks like this:

{
    "logContext": "{\"app\":\"my_app\",\"platform\":\"de\",\"requestId\":\"my_app.5d2c418738a4e1.01696328\",\"crossAppRequestId\":\"global.5d2c418738a451.59829866\",\"userId\":16584,\"order\":{\"id\": 1122334455,\"article\":{\"id\":12345, [...]}}}"
}

Now what I need to do is to parse this string stored in logContext so I get the JSON object and this object then should be stored inside the context field. So after the update my document will look like this:

{
    "context": {
        "app: "my_app",
        "platform: "de",
        "requestId: "my_app.5d2c418738a4e1.01696328",
        [...]
        "order: {
            "id": 1122334455,
            "article": {
                "id": 12345
            }
        }
    }
}

When searching the documentation for any hint on how I would be able to parse a string as JSON (something like JSON.parse(jsonString) ) I was not able to find anything. I had a look at the painless language reference but without success. Thats why I was going for the JavaScript library.

Anyway: As you suggested, I tried running an aggregation query. There was no result. My query:

"aggs": {
	"articles": {
		"terms": {
			"field": "context.order.id"
		}
	}
}

returned me an empty bucket list.

As mentioned: There are 2 other documents I updated. One by the updateApi of the JavaScript library and one via index API of elasticsearch (without the JavaScript involved) - same query works out for both documents and returned the buckets.

cmelzer · August 29, 2019, 7:41am

Hi Wylie,

I was able to solve my problem. Because of the back and forth in my JavaScript with update and index Api I somehow mixed up the request body. While the following seems to works for an client.update:

client.update({
    id: "ZE_g9GsBPHUZVl-pfzao",
    index: 'logstash-application-2019.07',
    type: 'doc',
    body: {
        doc: newDocument // note the 'doc' here
    }
})

With an client.index({}) it must be

client.index({
    id: "ZE_g9GsBPHUZVl-pfzao",
    index: "logstash-application-2019.07,
    type: "doc",
    body: newDocument // note that the document object is passed directly to the body
})

I missed this while switching between index and update. Because of this error, the documents now have an additional layer, so instead of:

{
    ...,
    "_source": {
        "app": "my_app",
        ...
    }
}

I got:

    {
    ...,
    "_source": {
        "document": {   // note the "document"
            "app": "my_app",
            ...
        }
    }
}

After the change, the documents will appear in Kibana and things work out as they should.
I'm sorry for my mistake. Thank you so much for your help and your quick replies!

system · September 26, 2019, 7:42am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
No docs showing up in discover after updating mappings and reindexing Kibana	2	289	September 24, 2019
Kibana no documents in index showing in discover Kibana	3	1609	November 20, 2018
I add documents to elastic indexes but they don't refresh in Kibana Kibana	2	210	June 10, 2021
Documents not showing Discover, show in queries and document counts Kibana	3	2193	June 8, 2021
Kibana not showing recent Elasticsearch data Kibana	13	25091	July 6, 2017

Kibana does not show documents updated via index API

Related topics