Determine a document's origin

Elastic Stack Kibana
1

Is there a way to glean additional information on what generated a document in elasticsearch? I'm working with an inherited reporting architecture and found some visualizations that are using an index pattern I'm not familiar with. It references documents that are not part of our typical logstash flow. They appear to be updated daily with summaries/totals but I haven't been able to figure out what job/system creates them.

  1. Is there any _search option I can perform in the dev tools for an index or specific document that might provide insight into what entity updated a document?
  2. Are there any likely sources for daily jobs that exist somewhere in Kibana and are creating these summary documents? I looked at Watcher (one system I have worked with before) but it's not coming from there.

I appreciate any guidance or advice on investigating this further.

2

Can you share an example of one of the documents? It might help us provide some advice.

3

Sure, I'm not sure what information would be most useful. Here is an example doc from the dev tools retrieved by calling GET index-name/_doc/QmRXULjl4SJCFBIfE7QWGXNJ7H4AAAAA.

  "_index" : "index-name",
  "_type" : "_doc",
  "_id" : "QmRXULjl4SJCFBIfE7QWGXNJ7H4AAAAA",
  "_version" : 5,
  "_seq_no" : 86,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "gameid" : "ddcfa91c-1c04-40f0-a97c-f1b9e84b679e",
    "value" : {
      "sum" : 69.0
    },
    "customerid" : {
      "value_count" : 11.0
    },
    "sessions" : {
      "value_count" : 23.0
    },
  }
}

Nothing jumps out in the settings for the index that's why I was wondering if there was a GET command I might be able to run for additional context or a source/origin on specific documents. I'm familiar with all of the logstash generated indices but this seems to exist somewhere outside of that flow.

Please let me know if there is a better command to run to get you more information on the specific index.

4

Damn, I was hoping you were ingesting with one of our products as that usually stores something useful :frowning:

In that case there's nothing natively enabled in the stack to do this. You can look at things like audit logging, or maybe adding in a "source" field via whatever code you are indexing these from.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.