Deleted ingest pipeline, doh! - Can't get it Back

Elastic Stack Beats
1

Hey there, I configured Fleet in the console, installed/enrolled an agent down on my server and configured an integration.. Moments later i got logs and kibana was happy.

I'm an idiot and accidently deleted the ingest pipeline in Kibana and the logs are now going unstructured into logs-elastic_agent.filebeat.

I read that Fleet automatically installs the ingest pipelines when it first deploys a module to an agent, so i removed and re-added the agent, however i don't think it is re-installing the pipeline.

I looked for the pipeline json, but in official elastic github, can only find the yaml versions.

What can i do?

2

@ajhstn Yes Doh! but we all Doh!

Did you try to remove the integration from the policy ... apply it, let it all come back up.

Then re-add the integration to the policy and apply it?

Curious... Which integration?

3

Hi @stephenb yes, i have uninstalled the agent, reinstalled it, removed the integration and reconfigured it, applied it, unapplied it..

The Palo Alto integration. It has worked since i first installed, so i know it works..

4

Hmmmm.... Something besides the obvious seems a bit wrong it should be failing with a missing pipeline I would think (unless we added some safety logic for that).

What version are you on?

Do have the exact name of the pipeline that you deleted?

You could try installing the agent on another box?

I updated your Subject Line perhaps an agent specialist will chime in.

It probably thinks that integration is correctly installed....

5

I'll try a full integration remove, then apply, then uninstall, then hard delete /opt/Elastic on disk, then reboot, then re-install new agent.

6

What version are you on.

I can easily load the ingest pipeline and get it to you to help get you stable.

Then we can figure out what went wrong later.

7

7.10 - but im going to upgrade to 7.14.

I found this in /opt/Elastic/Agent/data/elastic-agent-1428d5/logs/default/filebeat-json.log i am unsure if it is relevant.

{"log.level":"debug","@timestamp":"2021-08-07T08:27:16.214+1000","log.logger":"processors","log.origin":{"file.name":"processing/processors.go","file.line":128},"message":"Fail to apply processor client{add_index_pattern=logs-panw.panos-default

8

Yeah 7.10 ... Looong ago early early agent....

I would get to 7.14 and then try.

I will get the 7.10 ingest pipeline as a back up

1 Like
9

@ajhstn Exactly which version 7.10.2?

Also that does NOT look like a good error... did you touch any of the agent side ingest / processors? Although OTH that is probably because the index pattern is already there... so maybe not so bad... hard to tell..

10

It is 7.10.0 specifically, but i am moving it up to 7.14.0 as we speak.

I haven't touched anything on agent side, except for default install procedures.

1 Like
11

Give that a Clean Whirl... see how it goes, stepping away!

Here is the gist with _ingest/pipeline/filebeat-7.10.0-panw-panos-pipeline

For clarity you don't really want your agent version to be ahead of your Elasticsearch Version

See order here

1 Like
12

Hi @stephenb thanks for your help today. I upgraded the cluster to 7.14.0, reinstalled everything and it IS gettting data into Elasticsearch. There is now a Grok error.. but i'll raise a new ticket for that.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.