Data in index not populating in the data view or kibana

Elastic Search
1

Hello,
I have a pipeline to update my documents with parsed url information via the uri_parts processor. However when I look at the datastream or in kibana those addition fields are absent. I checked the data in the backing indexes and it is present. I tried updating the index template in the datastream but that doesn't seem to help and I have mappings for the fields.

Any ideas on how to get that data to show up?

2

Thanks so much for reaching out, @Zachary_Joyner. Do you have a code sample you can provide? Can you also give some further details about how you set up your data view or dashboards?

Here are some similar posts that may be helpful, too:

3

This is my pipeline.

{
  "description": "Updates the data for use with watchdog.",
  "processors": [
    {
     "json": {
      "field": "message",
      "add_to_root": true
    },
      "uri_parts": {
        "field": "httpRequest.requestUrl",
        "target_field": "httpRequest.url",
        "keep_original": true,
        "remove_if_successful": false,
        "ignore_missing": true
      },
    "gsub": {
      "field": "httpRequest.latency",
      "pattern": "s$",
      "replacement": ""
    }
  }
  ]
}

I found an issue with my index management. I didn't update the pipeline there, which is now fixed with:

{
  "index": {
    "default_pipeline": "logs@default-pipeline",
    "final_pipeline": "watchdog_macro_pipeline"
  }
}

However I am seeing this new error in my ingress which is the elastic provided dataflow job:

java.io.IOException: Error writing to Elasticsearch, some elements could not be inserted: Document id tBzMxpABx8HgWJ22SKrq: [1:4025] failed to parse field [httpRequest.url] of type [text] in document with id 'tBzMxpABx8HgWJ22SKrq'. Preview of field's value: '{path=/api/sites/subdomains/gingafc, original=https://api.leagueapps.io/api/sites/subdomains/gingafc, scheme=https, domain=api.leagueapps.io}' (document_parsing_exception) Caused by: Can't get text on a START_OBJECT at 1:3872

I think the issue is with the indexes that back the data stream but I am still digging there.

4

Thanks for following up, @Zachary_Joyner. Typically this error occurs when new requests were disreaged because the queue was full.

Here are some further resources that could be helpful here:

5

I'm not sure this is a queue issue. Looking at the timing, this started when I created the new pipeline.

image
image1920×1080 375 KB

I also see this in the error message:

{path=/api/sites/subdomains/cityside208, original=https://api.leagueapps.io/api/sites/subdomains/cityside208, scheme=https, domain=api.leagueapps.io}

Which is the text I would expect to get added to the document.
I ran some tests with an index not managed by the data stream. And the pipeline was working as expected.

6

Also in the performance tab I am not seeing pegged CPUs or memory.

7

The issue was with the mapping I had an object field set as a text field in the index template.

1 Like
8

Thanks so much for your follow up here and sharing your solution, @Zachary_Joyner.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.