Can't see all fields after importing sample dataset but I can query


(Abdalla Sabri) #1

Hello,

I imported the bank accounts sample dataset (accounts.json) successfully into elastic 5.6.4 then I created my index pattern as ba*

Now in the discover tab, I can do queries like (account_number:<100 AND balance:>47500) and I get the correct amount of results but I dont see except the fields that begin with _

Here is a screenshot

Same result even when I do match queries in curl (totally independent of kibana). Please let me know what I'm doing wrong.. why is _source empty? Im sure all the fields/values got copied to ES but I can't view them.

Curl example
curl test:9200/bank/account/_search?q=Putnam
{"took":0,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":1,"max_score":4.6146765,"hits":[{"_index":"bank","_type":"account","_id":"25","_score":4.6146765,"_source":{}}]}}

GET /bank/_mapping


(Val Crettaz) #2

Can you refresh your index pattern? What happens?


(Abdalla Sabri) #3

Hi val,

Thanks for the quick reply. I tried to refresh the index pattern from the following page

And I also restarted elastic and kibana just to make sure... but that made no difference, I still have the same view in the screenshot in my first post. I've been trying to figure out what I'm doing wrong for the past 3 hours with no luck. Thank you.

edit: one thing to note, in the discover tab of kibana, under Available fields.. if i uncheck 'Hide Missing Fields' then it shows me the field names like balance, etc.. but still the _source is empty and no content is shown


(Val Crettaz) #4

Can you show how you imported the accounts.json file into ES?


(David Pilato) #5

You should not define _source settings in your mapping.


(Abdalla Sabri) #6

@val
curl -H "Content-Type: application/x-ndjson" -XPOST "test:9200/bank/account/_bulk?pretty" --data-binary @D:\accounts.json

result was successful with no errors

@dadoonet I only imported the accounts.json via the above command and then I created an index pattern with default options (no time-based events) called ba* .. which is exactly what is described by elastic guides.. can you elaborate more please? what can I do to fix this?


(Val Crettaz) #7

You should remove all _source sections in your mapping. Not sure why they are in there.


(Abdalla Sabri) #8

I've tried to delete all indices and index patterns and then re-imported the data.. checked the index pattern and _source is there exactly like the above screenshot. If you guys say that _source shouldn't be there then I don't know why it's automatically added to the mapping. Is there a way I can delete _source from the mapping manually?

edit: Also what about the other fields with _ like _id / _index / _score ... are these normal to appear in the mapping? Or should be removed same as _source?


(Val Crettaz) #9

maybe you have an index template installed. What do you get when running this:

curl localhost:9200/_cat/templates?v

(Abdalla Sabri) #10

You're right.. I totally forgot about index templates. Before beginning to experiment with the sample data I configured elasticsearch-transport-couchbase because my main objective is to ingest our companys couchbase data into elastic for statistical purposes.. and the couchbase plugin had me insert the following template, I didn't even know this template will apply to all data.

I removed the template and checked the mappings again, I no longer have _source there and the data is showing correctly like described in the guides.

I know this question is unrelated to elastic but maybe someone can shed some light, why this template is required when transferring data from couchbase? And should it be removed on a successful import?


(Val Crettaz) #11

Cool, glad we figured it out and you can now see your data.

As for your other question, I guess that the Couchbase transport plugin requires some specific fields mappings, which is what you can achieve by using a template.


(system) #12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.