Ingesting Logs Without Integration Approach (SIEM)

Elastic Stack Elastic Agent
1

I have been taksed to ingest logs of APPS/WEBSITES which are hosted on cloud like SAP S/4 , paypac , tally etc etc (30+apps - mainly proprietary/internal apps like )

Q1. what approach should i be using each time i get into a situation where integrations are not present (free license so no creating a integration )

the solution i thought of was

  1. first we will check if in any documentation integration through api is possible like tally then ingestion through elastic agent with help of "Custom API integration " (if pushing machnism is in use then we can go with other integration for lisnting )(here manual parsing will be required as all the log will be in message feild with some meta data feilds )

  2. logstash with inputs like http poller , syslog and etc etc (but requires manual parsing )

so just wanted to know the approach and how things should be done(if anyone can ans with example of SAP S/4 , SAP SF)

2

It really depends on each application, on how they can generate the logs.

If the logs can be written into a file you may use an Elastic Agent running on the server to get the logs with the Custom Filestream integration, if the logs can be sent elsewhere using TCP or UDP you could use an Elastic Agent or Lgostash running on a different server to receive those logs, if you need to pull these logs you could use the Custom API integration or write some code using Python or another language to get these logs.

The main issue is that you will need to build an pipeline to parse your message, does not matter what is the source.

Also, you do not need a license to create a custom integration, with a license you could use the AI tools to help you create it, but you can create it manually, the integrations repository and the documentation have the steps.

First thing would be to check how those tools generate the logs you need, from previous experience I know that SAP is a nightmare regarding audit logs, not sure if it changed.