Elastic-Agent No Index/Data

Elastic Stack Elasticsearch
1

I am trying to test out this ingestion manager and i enrolled a agent that is doing metrics and nginx and i never seem to get any new indicies that have 7.9. I go through kibana -> Stack management -> Index Management -> search for 7.9 and i only see apm* indicies. Nothing new from metric beat or filebeat. Ingest Manager shows 1 agent in fleet, and it appears to have activity updated within the last 30 seconds. Viewing the agent in the UI shows the following.

image
image1247×216 39.3 KB

Everything appears to be running fine. If i restart elastic-agent i see everything stop (filebeat--7.9.0 and metricbeat--7.9.0) and then start right after. But I dont ever get any errors in journalctl and never any data on the elastic stack. Am i missing something?

In my settings for Ingest manager settings i have the Global output to kibana as my kibana.myDomain.com url (which is a how i access kibana), and my elasticsearch url (logs.myDomain.com) which is a nginx proxy to my 3 coordinator nodes. I am kind of stumped right now because i do not see any errors or issues, seems like everything is processing fine.

One thing to note, i am not sure how this data gets pushed into elasticsearch. Because my logs.myDomain.com is password protected. But i do not see any way to put a password in. Unless it pushes all the data through kibana and kibana forwards the data? If not then that could be an issue, but i would think i would see an error stating invalid credentials or something to that effect.

2

In Fleet mode (Agent receives config from Kibana) the Kibana authentication info will be included with the enrolment commands, but I guess if you have a custom setup Kibana might not know about the auth wall you've put in front.

Try following the Standalone mode setup. Step 4 is the username and password change. The credentials are very likely the source of your problem.

If that doesn't work, try the steps in the I can't see data FAQ, especially the installing Metricbeat yourself just to diagnose if that can get its data out (i.e. no network, firewall or other problems).

3

So something i do notice. And I am assuming this is the issue. If i run the following command.

elastic-agent inspect output --output default

I see all the configs and all the configs have an output of hosts: localhost:9200. Example below.

    [default] FLEET_MONITORING:
    agent:
      monitoring:
        enabled: true
        logs: true
        metrics: true
        use_output: default
    output:
      elasticsearch:
        api_key: RemovedApiKey
        hosts:
        - http://localhost:9200
        type: elasticsearch
    programs:
    - filebeat
    - metricbeat

How is this set? I assumed it would get it from fleet manager settings right?

1 Like
4

Yep

image
image3328×718 145 KB

image
image1234×634 38.5 KB

5

Ok so apparently to update that in the agent you have to unenroll. And then re-enroll. It does not appear to update the agent when you change it. I now appear to have the correct output in the default configs. Now i am going to try and dig into the routes and see if i can see the traffic. I still dont see the data in the index, but i will take a look and see if i can track down the network traffic and update after i have anymore details.

6

Thanks, I didn't know that myself.

What do you see in the Datasets tab? You should see index names and there's even links to relevant default dashboards for each dataset (if there are any).

image
image1068×228 17.3 KB

7

Hi, we are working on fix the issue with the update of the host.

For debbuging on what is going wrong, have a look at the logs inside the data directory. One common issue we currently see if with certificates.

8

I think my main issue now with not seeing the data was mostly not understanding that i wouldn't see an actual index with 7.9 within the name. Apparently they are all datastreams huh? So really the only issue is the update of the host which you said you are working on. Once i unenrolled the agent and reenrolled i could see the data coming across the nginx server.

2 Likes
9

Datastreams have backing indices, which you should be seeing in the Ingest Manager Datasets tab. More info how they're named and how they work: https://www.elastic.co/guide/en/elasticsearch/reference/master/data-streams.html#backing-indices . They are a bit different to usual indices but you should be able to at least see they exist if everything is working.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.