Hi,
I'm experiencing this issue on a fresh ELK 7.10 installation. I just deployed the Elastic-Agent with the only integration Endpoint , not even system . Is there a solution or workaround available or this is still under research?
Thank you
Hi,
Any update on this?
Thanks
I'm including the mappings, to help you with the troubleshooting
Hi, thanks for the ping back. I'm interested to confirm some questions that may be "too basic" or silly, but they would help build context.
-
Are you using self-managed or cloud based stack?
-
While you cite it was a 'fresh' deploy of 7.10 (which is helpful to know) was any data restored as part of the process, or upgraded and then migrated over to the 'fresh' install?
-
Are there any old Beats running on any hosts pointing to this cluster?
We aren't sure yet why you are seeing this, I see some chatter from team on it internally. Hopefully more will be known and we can post back! Thanks for posting the mappings!
Thank you,
Eric, a tester guy at Elastic
Hi,
Thank you for your interest on my report. Answering to your questions:
-
This would be a "self-managed" stack
-
The only beats shipping data to the stack are matching the stack version (7.10.0) and are also fresh installed. Just a couple of hosts are currently sending data to this new stack, so the data sources are controlled.
-
I did import some saved objects in Kibana, from other stack running v7.9.2. Mostly dashboards and visualizations. Most of theses pulls data from
winlogbeat-*,if this information helps somehow.
All indexes are new. I did not import any from other stack. -
Finally, in the other stack I implemented a the
geo_ipenrichment by following this documentation. I have not done so for current stack running v7.10.0. I wasn't sure if it was going to be necessary or not.
If you require more details, please let me know.
Thank you
Hi,
Any update about this issue?
Thank you
Hi Manual, thank you for the good info and reply. I don't have much experience deep diving on this, but we are building good info to bring it to the Fleet team to review - what you posted is very helpful. I do have one more idea.
I do think the import of the Kibana Saved-Objects is impacting your usage. Fleet relies heavily on SO in Kibana in 7.10. It seems likely something from the geo_ip setup done in 7.9.2 didn't get fully and correctly setup in 7.10 just by doing the Kibana Saved Objects import action.
From that, I can cite, if you just want to get it working again, you could probably clone a new 7.10 and start fresh without importing the saved objects and it should work. If you just want to wipe ES and Kibana, that should do it. I don't know how to guide you to more surgically try to delete the problems and the impact from them.
If you are ok to re-set the cluster and work then that's your call, please let us know how it goes!
But if you want to follow thru further with the current cluster and assess deeper still, we could review what Saved Objects you have. If you desire, you can this doc link and send us the capture for review.
Hi @EricDavisX,
Thank you for your help. Reset/rebuild the stack is not an option for me unfortunately. In fact this is a fresh ELK installation, because the last upgrade from v7.9.2 did not work as expected. The only thing I keep saving from the old version, are just the visualizations and dashboards, in which I spent a lot of time designing and building. Deploying a new stack from scratch it is also time consuming. Therefore It would be difficult for me to wipe out everything and start over one more time (perhaps when I switch over ELK 8, because it will introduce many deep changes).
On the other hand, none of the visualizations imported is reading data from Metricbeat, nor Filebeat, but from Winlogbeat. Winlogbeat was not mentioned in the errors. Do you still think it might be related?
Do you think I should try this first? Would it be OK to modify the configuration file generated by the Filebeat instance running under the Elastic-Agent to add the geoIP config line?: C:\Program Files\Elastic\Agent\data\elastic-agent-1428d5\install\filebeat-7.10.0-windows-x86_64
Hi, I don't know that the Elastic Agent controlled Beats will pick up the config changes you make, I'm not sure - you can try it tho.
I fully understand the time to get a cluster up and that re-setting it isn't an option, it was a long shot I proposed if you had spare resources to help confirm it was the import of the saved objects that was negatively impacting. About that... I'm not sure of the recommended way to export / import data across clusters - the Fleet product being pre-GA we are still trying to make upgrades work. I'm sorry to hear the original 7.9.x to 7.10 upgrade didn't work.
Regards,
Eric
Hi, one more time I really appreciate the time you are taking to help me on this. I don't think either that the Elastic Agent controlled Beats will pick up the config changes I make. I don't see any custom config to match my stack connection, so it may be using the connection config from Elastic-Agent in background.
I could try re-importing the objects in Kibana, but not build them from scratch. I can export them all and delete them after, then check Security and see if the error persist. Once the test have concluded I just have to import all objects one more time.
I'll try that tomorrow morning and let you know how it went.
Thank you
Hi @EricDavisX,
I removed all imported objects from my node and restarted ELK. The issue persist. My question is:
After you install ELK from scratch (clean installation, nothing more than ELK) do you need to enable or config anything so Security or Elastic Agent can collect geo location data?
What it looks like is that the geo-location data is not available.
Thank you
Is it safe If I delete index .ds-logs-elastic_agent-default-000001? Will this harm my ES, or ES will re-create the index?
@EricDavisX, Please I need to know if it is safe to remove index .ds-logs-elastic_agent-default-000001 and if ES will re-create it. Perhaps this helps to fix the error.
Thank you
System indices start with a dot ( . ) like .kibana or .security and others, so you should be very careful not to delete these since it will break your cluster.
However , indices that use data streams also use a dot syntax (starting with .ds* ) these you can delete, so you can delete older ones if need be.
If you go into the console, and paste this, this will lists the indices and sort them via creation date.
GET _cat/indices?v&h=h,s,i,id,p,r,dc,dd,ss,creation.date.string&s=creation.date