Testing with Filebeat vs Custom Filestream Integration

Elastic Stack Elastic Agent
1

Hello,

I have custom filebeat logs and I am moving them to Elastic Agent's Custom File stream Integration. For testing everything from how the multiline is processed to the transformation to the data, was done by rinse and repeat. In other words, it will be a trial and error to get the data right.

Now comparing the differences with how I used to test with Filebeat and Custom Filestream Integration, is there way with the integration to delete the registry file, in order to resend the history after already ingesting?

2

Are you moving over and existing filebeat logs... they "migration" should be pretty straight forward... pretty one to one...

Are you asking how to delete the registry from Elastic Agent so it will reload file again?

Yes let me find it (it is not straight forward if I recall) ...

Should be in something like

# cd /opt/Elastic/Agent/data/elastic-agent-8.17.0-96f2b9/run/
# ls log-default/
meta.json  registry

CAREFULL ^^ :slight_smile:

another way is in the path to the log use /my/path/*.log

Then just
cp mylog.log mylog2.log

This is what I do and just clean up after myself....

1 Like
3

Hey @stephenb
yes that works, thanks!

I been doing this:

  1. I built the custom filestream integration and only ingest 1 or 2 sample logs.
  2. Ingest the raw sample logs
  3. Use those sample docs to drive building the ingest pipelines
  4. Ingest more raw sample logs, to continue to test out the ingest pipeline
  5. Delete all docs
  6. Rollover/Reindex or Delete datastream to be able to avoid mapping conflictings

Adding to this, would be removing the registry to fully ingest everything or just deleting the integration and rebuilding it.

Yeah I wish there was a way to test more easily.

1 Like