Hi community,
I'm trying to fetch the Symantec's Event Stream (Broadcom Enterprise Security Group - API Documentation) with filebeat.
First of all, to check if Symantec's side works properly, I wrote this script to fetch the stream with curl:
#!/bin/sh
HOST=https://api.sep.eu.securitycloud.symantec.com
OAUTH="Basic Tz..."
BTOKEN=`curl -q -sS -X POST $HOST/v1/oauth2/tokens -H "accept: application/json" -H "authorization: $OAUTH" -H "content-type: application/x-www-form-urlencoded" | jq -r .access_token`
STREAM="v1/event-export/stream/1db8a6c9-7b60-4451-a698-f4d35a6c0271/0"
curl --location-trusted --request POST "$HOST/$STREAM" \
--header "Content-Type: application/json" \
--header "Accept: application/x-ndjson" \
--header "Accept-Encoding: gzip" \
--header "authorization: $BTOKEN" \
--data "{}" \
--compressed \
--output -
It works properly: gets the token, then connects and starts receiving an event stream.
To make it more robust, I want to use Filebeat.
As a starting point, I took the Symantec Endpoint Security integration for cloud.elastic.co (Symantec Endpoint Security | Elastic integrations | Elastic) and made necessary adjustments (url, post data, etc).
Here's my filebeat.yml:
filebeat.inputs:
- type: cel
auth:
oauth2:
client:
id: O2ID*****
secret: ******
endpoint_params:
grant_type: client_credentials
token_url: https://api.sep.eu.securitycloud.symantec.com/v1/oauth2/tokens
data_stream:
dataset: some_dummy_value
type: logs
id: id_a
interval: 10m
resource:
ssl: null
timeout: 4s
tracer:
filename: ./logs/cel/trace-*.ndjson
maxbackups: 3
maxsize: 4
url: https://api.sep.eu.securitycloud.symantec.com
state:
initial_interval: 24h
limit: 2000
next: 0
want_more: false
tags:
- preserve_original_event
- forwarded
- rl-cv161
program: |
(
state.want_more ?
state
:
state.with({
"limit": state.limit,
"start_date": state.?cursor.last_timestamp.orValue(string(now - duration(state.initial_interval))),
"end_date": now,
"next": state.next,
})
).as(state,
post_request(
state.url.trim_right("/") + "/v1/event-export/stream/1db8a6c9-7b60-4451-a698-f4d35a6c0271/0",
"application/json",
{}.encode_json()
).do_request().as(resp, resp.StatusCode == 200 ?
bytes(resp.Body).decode_json().as(body,
(body.?next.orValue(body.total) != body.total).as(want_more, {
"events": body.events.map(e, {
"message": e.encode_json(),
}),
"next": want_more ? body.next : 0,
"want_more": want_more,
"limit": state.limit,
"start_date": string(state.start_date),
"end_date": string(state.end_date),
"cursor": {
?"last_timestamp": want_more ?
state.?cursor.last_timestamp
:
optional.of(state.end_date),
}
})
)
:
{
"events": {
"error": {
"code": string(resp.StatusCode),
"id": string(resp.Status),
"message": "POST:"+(
size(resp.Body) != 0 ?
string(resp.Body)
:
string(resp.Status) + ' (' + string(resp.StatusCode) + ')'
),
},
},
"want_more": false,
}
)
)
output:
console:
pretty: true
logging:
to_syslog: false
level: warning
logging.metrics.enabled: false
Being started, it shows me log entries like this:
{"message":"Post \"https://api.sep.eu.securitycloud.symantec.com/v1/event-export/stream/1db8a6c9-7b60-4451-a698-f4d35a6c0271/0\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"}
However, while checking the resource.tracer file, I see these records:
- HTTP request with authentication info to token URL
- HTTP response with bearer token
- HTTP request to stream url with bearer token
- HTTP response with "http.response.status_code": 200, "http.response.body.content": "{"events": [ {"actor..., "http.response.body.truncated": false, "http.response.body.bytes": 359761, some other fields, and "error.message": "failed to read response body: net/http: request canceled".
This confuses me.
Could you please tell me what can be done to identify an issue?
Best,
Roman Levitsky,
Systems Administrator, Exadel.