Discover not showing all details

Elastic Stack Kibana
1

I have a script (python) that fetches AWS Instance state and pushes the data into ES. I am using ElasticSearch Python module and doing bulk upload

Data that pushed to elasticsearch (we use bulk upload):

{'_type': 'custom', 'data': {'name': x', 'Environment': 'y'}, '_index': 'test_index'}
{'_type': 'custom', 'data': {'name': y', 'Environment': 'y'}, '_index': 'test_index'}
{'_type': 'custom', 'data': {'name': z', 'Environment': 'y'}, '_index': 'test_index'}

The above works fine. However, when we run it again, and if the data hasn't changed, within Kibana > Discover, we would see the 'number of hits' go up by 3 but can't see the extra 3 items.

If any of the data changes like below

{'_type': 'custom', 'data': {'name': xxx', 'Environment': 'y'}, '_index': 'test_index'}
{'_type': 'custom', 'data': {'name': y', 'Environment': 'y'}, '_index': 'test_index'}
{'_type': 'custom', 'data': {'name': z', 'Environment': 'y'}, '_index': 'test_index'}

We see both the 'number of hits' go up by 3 and see the 3 news entries too.

We want to see all the added entries regardless if the data changes or not.

2

Hi Geek,

Do you mean to say here that you can never see the data in discover in Kibana unless otherwise something changes? What happens when you first create your index pattern in Kibana. Does discover show you results? I am confused about this line in particular?

but can't see the extra 3 items.

The above works fine. However, when we run it again, and if the data hasn't changed, within Kibana > Discover, we would see the 'number of hits' go up by 3 but can't see the extra 3 items.

Thanks,
Bhavya

3

Hi Bavya,

Do you mean to say here that you can never see the data in discover in Kibana unless otherwise something changes?

Yes.

When the index is created very first time, i can see the data. Then when I subsequently run the script again (that pushes more data), I do see the 'number of hits' going up every time but the 'discover' only shows the data when there is a change.

4

ah interesting. Let me ask around.

Thanks,
Bhavya

5

Can you please give us your Kibana version and a screenshot?

Thanks,
Bhavya

6

When you're only seeing the first three results but the hits are at 6, check the network tab in your browser's dev tools. Does the response to the _msearch request contain all 6 documents?

When bulk indexing, are you always creating new documents or are you also updating old ones?

7

Thanks @bhavyarm and @Bargs

Kibana is: Build 14566, Commit SHA 85a6f4da

network_second-time
network_second-time.png815×697 69 KB
second_run
second_run.png1681×447 48 KB
after_Few_more_runs
after_Few_more_runs.png1647×695 77.1 KB
first_run
first_run.png1840×492 60.9 KB
changed
changed.png1677×639 67.3 KB

See snapshots:

first_run: Very first run when new index is created. Events present. Notice the time stamp of 15:30 and event hits of 882

Second_run: Event Hits of 1764 but still the latest event shown is 15:30 (tried sorting as well in-case)

after_few_more_runs: Event Hit count goes up all time but latest event shown is still 15:30

changes: Now i change 'name: testingx'. Hit count up as well as latest even is now 15:45

Network_secon-time: Browser network snapshot during second run. It does say doc_count: 1764 which matches the Hit count.

My python code is as below:

instances = ec2.instances.all() # standard boto3 call
es = Elasticsearch() # ElasticSearch Python Library

for instance in instances: (iterate over all instances)
     data = {
            'name': 'testing',
            'department': 'finance',
            'timestamp': time_
     }
     doc.append(data)

items = ({'_type':'custom', '_index':index_name, 'data': instance}
                  for instance in doc )

bulk(es,items)

Above, I am filling in a static data but in real-world, it will be instance data like instance.name, instance.type etc.
Also, above when I change the data ex: name: testingx it then shows up within Discover otherwise no.

8

The screenshot of the network tab tells me that all 1764 hits in your second run have a timestamp between 15:30:30 and 15:31:00 (a single 30 second interval). I'm not very familiar with Python, are you sure your script is working correctly? I would suggest taking Discover out of the equation temporarily and investigate the data in ES using Curl or Kibana's Dev Tools app to make sure the data looks correct.

9

Thanks once again.

I will investigate it.

Meanwhile, can I ask if I 'HAVE TO' include timestamp when i push data to ES? When I run my script, I get around 800 events per run (running script every 15 min for now). I want to push all those 800 events to ES. TimeStamp is common for all those 800 events as they happened at the same time (when the script ran).

10

Timestamp isn't a required field, unless you want to be able to do timeseries in Kibana.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.