Use Logstash for access REST APIs and do complex queries or better Connector Clients

[sebastianboelling (Sebastian Bölling)]

Hi anybody,

has anybody experiences in using Logstash to gather data from a complex REST/JSON API.

The API delivers user specific data similar to OneDrive or SharePoint. That means I have to access the (1) users list and (2) for each user the stored content (documents with metadata), (3) the permissions for the content and the (4) fulltext of the documents.

Then I have to ingest that data to Elasticsearch and/or Enterprise Search with Search Applications to provide Document Level Security later.

We are currently evaluating the Python Connector Framework because of the complexity of the API access.

But: can Logstash also do such complex API access and transformation jobs ? (fetch user list, fetch documents/data/metadata for each user, fetch fulltext of each user). Of course I have to reindex the permissions quite often and I have to identify new, modified, deleted or blocked users.

Thanks for your experiences with Logstash and/or Connector Clients

Regards

Sebastian

[Sean_Story]

Not Logstash, but have you looked at HTTP JSON input | Filebeat Reference [8.11] | Elastic ? I've not used it personally, and am not certain you could do nested loop pagination like you're wanting. But this is the closest I'm aware of to a generic HTTP API ingestion tool.