I encountered an issue on my elasic installation.
While moving the elastic installation from one server to an other one, I had an error in my logstash pipeline so a field was not correctly parsed as json.
Now I would like to update those documents. As far as I found out, there are basically two ways to do so: Either use the update API which has the disadvantage (for my use case) that the documents will be merged together or use the index API to replace a hole document. Sidenote: I'm doing this with the elasticsearch-js library
I was actually going for the index API und started replacing 3 documents (one by one) with the new ones. While the elasticsearch API does show me the updated documents, Kibana does not show them at all. I tried to find a document by id (via kibana) and no result was returned. I double checked the selected timeframe, refreshed the index but with no success. The document replaced by the index API is missing.
I updated an other document using the update API. This document is still displayed in Kibana and also contains all the new fields I passed to the document (and all the existing ones as well).
What am I doing wrong? Why can't Kibana fetch or display those documents?
thanks for your reply. You're right in your questions (1-3). That's exactly what happened.
How I did reindex:
In Kibana -> Management -> Index Management -> choose the index the document is in by clicking the checkbox and choosing "Refresh Index". I also tried flush index and clear index cache with no success.
What kind of query I run to verify documents are in Elasticsearch:
I made a POST Request to /logstash-application-2019.07/_search with the following request body:
What kind of query I run through Dicover:
I tried by entering the search query manually (_id: "ZE_g9GsBPHUZVl-pfzao") and by using the "Add a filter" button selecting _id | is | ZE_g9GsBPHUZVl-pfzao
In both cases time frame was set to "This year" which should be correct since the document was added in July and still has a timestamp from July.
By the way: I'm using kibana and elasticsearch version 6.8.1
Thanks for the followup. I'm still unclear on which part is not working. You've said that there are 3 documents that were updated, but are not showing up any more. Can you go over the steps you took for those 3 documents that are problematic?
First, i searched a document which wasn't parsed correctly due to the error in my logstash configuration. As I didn't wanted to update 33k documents without knowing if it will work out, I picked 3 (one at a time). The one in my example will be the document with id ZE_g9GsBPHUZVl-pfzao
I initialized the client by const client = new Client({ node: 'http://localhost:9200' }), then searched for the document by id:
client.search({
index: 'game-of-thrones',
body: {
query: {
match: {
_id: 'ZE_g9GsBPHUZVl-pfzao'
}
}
}
}, (error, response) => {
const document = response.hits.hits[0]['_source'];
// logic to update the document
client.index({
id: "ZE_g9GsBPHUZVl-pfzao",
index: "logstash-application-2019.07,
type: "doc",
body: document
}, (error, response) => {
console.log(response); // this gave me a success and status 200
});
});
When searching for the id with elasticsearch api, the api returned the updated document but kibana didn't.
I just tried the exact same thing with another document but instead of using the javascrip library, i used the elasticsearch api. No problem there, everything went fine and the document is still in kibana.
Okay, I think I see what you are trying to do, but let me confirm that you have already:
You previously used logstash to ingest data, but your index mapping was wrong
You changed the mapping of the index using the Management JSON editor, and it succeeded (there are cases where you are not allowed to remap existing indices)
Your documents already contain the right information in _source, but need to be reindexed so that they are searchable and aggregatable.
If that's your scenario, then _update_by_query should have worked. _update_by_query doesn't rewrite the _source of your documents, but it does reindex and make searchable based on the new mappings. You can validate this by running an aggregation query against your new mapping, such as:
If that's not your scenario, then you probably need to change the _source field of some documents.
You could do this by overwriting documents using PUT logstash-application-2019.07/_doc/ZE_g9GsBPHUZVl-pfzao, or you could use a Painless script on the _update_by_query request to rewrite the source. There are several examples of this in the Painless Update By Query docs: https://www.elastic.co/guide/en/elasticsearch/painless/current/painless-update-by-query-context.html
Now what I need to do is to parse this string stored in logContext so I get the JSON object and this object then should be stored inside the context field. So after the update my document will look like this:
When searching the documentation for any hint on how I would be able to parse a string as JSON (something like JSON.parse(jsonString) ) I was not able to find anything. I had a look at the painless language reference but without success. Thats why I was going for the JavaScript library.
Anyway: As you suggested, I tried running an aggregation query. There was no result. My query:
As mentioned: There are 2 other documents I updated. One by the updateApi of the JavaScript library and one via index API of elasticsearch (without the JavaScript involved) - same query works out for both documents and returned the buckets.
I was able to solve my problem. Because of the back and forth in my JavaScript with update and index Api I somehow mixed up the request body. While the following seems to works for an client.update:
client.update({
id: "ZE_g9GsBPHUZVl-pfzao",
index: 'logstash-application-2019.07',
type: 'doc',
body: {
doc: newDocument // note the 'doc' here
}
})
With an client.index({}) it must be
client.index({
id: "ZE_g9GsBPHUZVl-pfzao",
index: "logstash-application-2019.07,
type: "doc",
body: newDocument // note that the document object is passed directly to the body
})
I missed this while switching between index and update. Because of this error, the documents now have an additional layer, so instead of:
After the change, the documents will appear in Kibana and things work out as they should.
I'm sorry for my mistake. Thank you so much for your help and your quick replies!
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.