Kibana report not reading _source

[fidsamurai (Firdaus Aga)]

Hi Team,

Kibana version 7.13.4.
I've enabled read fields from _source in Kibana advanced settings.

Issue -
I've saved a search in Kibana and tried to generate the CSV, however only the Timestamp has a value.
_source simply shows "-" values.

I also tried adding a few fields and only date fields were showing values.

We have recently updated from 7.8 and the same process worked there without issues.

[monfera (Robert Monfera)]

What visualization / query tool are you using for your query and subsequent CSV export?

[fidsamurai (Firdaus Aga)]

@monfera,

This is using Discover.

[majagrubic (Maja)]

Hi,

_source is not meant to work when exporting CSV from Discover. Can just paste the mapping for the two remaining fields?

[fidsamurai (Firdaus Aga)]

@majagrubic,

In Kibana 7.8 even if I didn't have any fields added in my Discover search it would automatically create a CSV of the entire document.

And the question in the second screenshot is why does the _created_at column have values while the other two do not?

When I checked Inspect on the saved search I get the JSON of the Document correctly.
It's just not converting to a CSV.

Mappings -

APIName -

"APIName": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "norms": false
            },

errorCode -

"errorCode": {
                      "type": "text",
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "norms": false
                    },

errorMessage -

"errorMessage": {
                      "type": "text",
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "norms": false
                    },

[majagrubic (Maja)]

Sorry, I wasn't clear. Starting from 7.13 not showing _source in a CSV export is expected.

Do you have fieldsFromSource enabled in the Advanced settings? keyword fields won't have a value with fieldsFromSource enabled. If you have a saved search that is referencing those fields, it is expected that they would show up with empty values. Can you please recheck those two things?

[fidsamurai (Firdaus Aga)]

@majagrubic,

You're spot on fieldsFromSource enabled in Advanced Settings as the team wanted to view the documents in json.
And you're right again, the fields I added in the Saved Search were keyword fields.

Looks like if I want CSV exports I have to disable fieldsFromSource.

Is there any workaround to show the document response in raw json?

[majagrubic (Maja)]

We have a JSON view for each document, but I don't think you will ever be able to see the raw json in a CSV file. If you disable fieldsFromSource, you should be able to view all fields (with values ) in a CSV file.

[fidsamurai (Firdaus Aga)]

Sorry, I think even I wasn't clear enough.
Viewing the JSON and the CSV are two different requirements.

In Discover if we open a document we get two options -
i. Table view
ii. Json view

Now the problem is if we disable fieldsFromSource the JSON reads with each field in a separate line.
However if we do enable it then we are able to read json in the "Expanded document" view in the correct data structure.

So as you were saying in the previous post if we disable fieldsFromSource we'll be able to export CSVs however if we enable it we lose the raw Json view in Discover's "Expanded document" view.

If there is a workaround to view raw json in Discover with fieldsFromSource disabled that would be the ideal solution, otherwise I'll have to ask my team to prioritise one or the other.

[majagrubic (Maja)]

There is no workaround for this in 7.13. However, we implemented this in 7.14. Upgrading to 7.14 should give you what you need.

[fidsamurai (Firdaus Aga)]

That's great!! Thanks a ton!!