I have a problem with accessing the ELK database and receiving a specific field with a token to form a further request to connect to the o365 api. I can't get data from this particular field.
scheme, I connect to the ELK database (into which the token is previously loaded) and try to get the access_token, but apparently I'm blowing something right, since I can't get the attached data.
Without providing more information you're unlikely to get much help.
What does your config look like?
What does the data you are trying to retrieve look like?
Thank you for your answer.
Thank you for your answer.
I managed to get the token by replacing the input from the http_poller with a direct connection to the Elasticsearch. But now I can’t parse the received data correctly using a split. My config
input {
elasticsearch {
hosts => ["elasticsearch:9200"]
index => "o365-token"
query => '{ "_source": ["access_token"], "query": { "match_all": {} }}'
user => "xxx"
password => "xxx"
schedule => "*/2 * * * *"
ssl => true
size => 200
docinfo => true
ca_file => "/etc/logstash/certs/trust/ca.pem"
}
}
filter {
http {
proxy => "http://proxy.com:8080/"
url => "https://manage.office.com/api/v1.0/<my-tenant>/ServiceComms/CurrentStatus"
verb => "GET"
headers => {
Accept => "application/json"
Authorization => "Bearer %{access_token}"
# body_format => "json"
}
}
split {
field => "[body]"
add_field => {
"number" => "%{[result][number]}"
}
remove_field => "result"
remove_field => "number"
}
split {
field => "[body][value][FeatureStatus]"
}
mutate {
remove_field => [ "access_token" ]
}
}
output {
stdout {
codec => rubydebug
}
}
Part of my debug
[DEBUG] 2021-12-22 12:54:01.682 [[main]>worker0] wire - http-outgoing-3 << "HTTP/1.1 200 OK[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.682 [[main]>worker0] wire - http-outgoing-3 << "Cache-Control: no-cache[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.682 [[main]>worker0] wire - http-outgoing-3 << "Pragma: no-cache[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.682 [[main]>worker0] wire - http-outgoing-3 << "Content-Length: 27581[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.682 [[main]>worker0] wire - http-outgoing-3 << "Content-Type: application/json; odata.metadata=minimal[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "Expires: -1[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "Server: Microsoft-IIS/10.0[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "OData-Version: 4.0[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "X-Instance-Name: Office365ServiceCommsWebService_Office365ServiceCommsWebService_IN_23[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "X-Activity-Id: 8ca33b52-c02f-4f8c-8e3e-831505bd4[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "X-Time-Taken: 1332[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "Server: Microsoft-IIS/10.0[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "X-AspNet-Version: 4.0.30319[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "X-Powered-By: ASP.NET[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "X-Powered-By: ASP.NET[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "Date: Wed, 22 Dec 2021 10:54:01 GMT[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "{[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << " "@odata.context":"https://office365servicecomms-prod.cloudapp.net/api/v1.0/<my-tenent>/$metadata#CurrentStatus","value":[[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << " {[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << " "FeatureStatus":[[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << " {[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << " "FeatureDisplayName":"Microsoft Bookings","FeatureName":"MicrosoftBookings","FeatureServiceStatus":"ServiceOperational","FeatureServiceStatusDisplayName":"Normal service"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << " }[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << " ],"Id":"Bookings","IncidentIds":[[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << " [\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << " ],"Status":"ServiceOperational","StatusDisplayName":"Normal service","StatusTime":"2021-12-21T10:54:00.3642702Z","Workload":"Bookings","WorkloadDisplayName":"Microsoft Bookings"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << " },{[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << " "FeatureStatus":[[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << " {[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << " "FeatureDisplayName":"Cloud App Security","FeatureName":"Cloud App Security","FeatureServiceStatus":"ServiceDegradation","FeatureServiceStatusDisplayName":"Service degradation"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << " },{[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << " "FeatureDisplayName":"Office 365 Advanced Security Management","FeatureName":"Office 365 Advanced Security Management","FeatureServiceStatus":"ServiceOperational","FeatureServiceStatusDisplayName":"Normal service"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " },{[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " "FeatureDisplayName":"Cloud App Security Discovery","FeatureName":"Cloud App Security Discovery","FeatureServiceStatus":"ServiceOperational","FeatureServiceStatusDisplayName":"Normal service"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " },{[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " "FeatureDisplayName":"Microsoft Cloud App Security","FeatureName":"Microsoft Cloud App Security","FeatureServiceStatus":"ServiceOperational","FeatureServiceStatusDisplayName":"Normal service"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " }[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " ],"Id":"cloudappsecurity","IncidentIds":[[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " "CS300127"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " ],"Status":"ServiceDegradation","StatusDisplayName":"Service degradation","StatusTime":"2021-12-19T15:18:17.777Z","Workload":"cloudappsecurity","WorkloadDisplayName":"Microsoft Defender for Cloud Apps"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " },{[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " "FeatureStatus":[[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " {[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " "FeatureDisplayName":"Other","FeatureName":"DynAXOther","FeatureServiceStatus":"ServiceOperational","FeatureServiceStatusDisplayName":"Normal service"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " },{[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " "FeatureDisplayName":"Sign in","FeatureName":"signin","FeatureServiceStatus":"ServiceOperational","FeatureServiceStatusDisplayName":"Normal service"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " },{[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " "FeatureDisplayName":"Sign up and Administration","FeatureName":"admin","FeatureServiceStatus":"ServiceOperational","FeatureServiceStatusDisplayName":"Normal service"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " },{[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " "FeatureDisplayName":"Environment Access","FeatureName":"orgaccess","FeatureServiceStatus":"ServiceOperational","FeatureServiceStatusDisplayName":"Normal service"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " },{[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " "FeatureDisplayName":"Environment Performance","FeatureName":"orgperf","FeatureServiceStatus":"ServiceOperational","FeatureServiceStatusDisplayName":"Normal service"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " },{[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " "FeatureDisplayName":"Components/Features","FeatureName":"comp","FeatureServiceStatus":"ServiceOperational","FeatureServiceStatusDisplayName":"Normal service"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " }[\r][\n]"
Part of my debug
[DEBUG] 2021-12-22 12:54:01.702 [[main]>worker0] http - success received {:code=>200, :body=>"{\r\n \"@odata.context\":\"https://office365servicecomms-prod.cloudapp.net/api/v1.0/4<my-tenant>/$metadata#CurrentStatus\",\"value\":[\r\n {\r\n \"FeatureStatus\":[\r\n {\r\n \"FeatureDisplayName\":\"Microsoft Bookings\",\"FeatureName\":\"MicrosoftBookings\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n }\r\n ],\"Id\":\"Bookings\",\"IncidentIds\":[\r\n \r\n ],\"Status\":\"ServiceOperational\",\"StatusDisplayName\":\"Normal service\",\"StatusTime\":\"2021-12-21T10:54:00.3642702Z\",\"Workload\":\"Bookings\",\"WorkloadDisplayName\":\"Microsoft Bookings\"\r\n },{\r\n \"FeatureStatus\":[\r\n {\r\n \"FeatureDisplayName\":\"Cloud App Security\",\"FeatureName\":\"Cloud App Security\",\"FeatureServiceStatus\":\"ServiceDegradation\",\"FeatureServiceStatusDisplayName\":\"Service degradation\"\r\n },{\r\n \"FeatureDisplayName\":\"Office 365 Advanced Security Management\",\"FeatureName\":\"Office 365 Advanced Security Management\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n },{\r\n \"FeatureDisplayName\":\"Cloud App Security Discovery\",\"FeatureName\":\"Cloud App Security Discovery\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n },{\r\n \"FeatureDisplayName\":\"Microsoft Cloud App Security\",\"FeatureName\":\"Microsoft Cloud App Security\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n }\r\n ],\"Id\":\"cloudappsecurity\",\"IncidentIds\":[\r\n \"CS300127\"\r\n ],\"Status\":\"ServiceDegradation\",\"StatusDisplayName\":\"Service degradation\",\"StatusTime\":\"2021-12-19T15:18:17.777Z\",\"Workload\":\"cloudappsecurity\",\"WorkloadDisplayName\":\"Microsoft Defender for Cloud Apps\"\r\n },{\r\n \"FeatureStatus\":[\r\n {\r\n \"FeatureDisplayName\":\"Other\",\"FeatureName\":\"DynAXOther\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n },{\r\n \"FeatureDisplayName\":\"Sign in\",\"FeatureName\":\"signin\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n },{\r\n \"FeatureDisplayName\":\"Sign up and Administration\",\"FeatureName\":\"admin\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n },{\r\n \"FeatureDisplayName\":\"Environment Access\",\"FeatureName\":\"orgaccess\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n },{\r\n \"FeatureDisplayName\":\"Environment Performance\",\"FeatureName\":\"orgperf\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n },{\r\n \"FeatureDisplayName\":\"Components/Features\",\"FeatureName\":\"comp\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n }\r\n ],\"Id\":\"DynamicsAX\",\"IncidentIds\":[\r\n \r\n ],\"Status\":\"ServiceOperational\",\"StatusDisplayName\":\"Normal service\",\"StatusTime\":\"2021-12-21T10:54:00.3642702Z\",\"Workload\":\"DynamicsAX\",\"WorkloadDisplayName\":\"Finance and Operations Apps\"\r\n },{\r\n \"FeatureStatus\":[\r\n {\r\n \"FeatureDisplayName\":\"Other\",\"FeatureName\":\"DynCRMOther\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n },{\r\n \"FeatureDisplayName\":\"Sign In\",\"FeatureName\":\"signin\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n },{\r\n \"FeatureDisplayName\":\"Sign up and administration\",\"FeatureName\":\"admin\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n },{\r\n \"FeatureDisplayName\":\"Organization access\",\"FeatureName\":\"orgaccess\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n },{\r\n \"FeatureDisplayName\":\"Organization performance\",\"FeatureName\":\"orgperf\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n },{\r\
and next
[WARN ] 2021-12-22 12:54:01.707 [[main]>worker0] split - Only String and Array types are splittable. field:[body] is of type = Hash
[WARN ] 2021-12-22 12:54:01.707 [[main]>worker0] split - Only String and Array types are splittable. field:[body][value][FeatureStatus] is of type = NilClass
[DEBUG] 2021-12-22 12:54:01.708 [[main]>worker0] mutate - filters/LogStash::Filters::Mutate: removing field {:field=>"access_token"}
[DEBUG] 2021-12-22 12:54:01.711 [[main]>worker0] elasticsearch - Sending final bulk request for batch. {:action_count=>1, :payload_size=>24493, :content_length=>24493, :batch_offset=>0}
after parsing json, I expected to see something like
breakdown by services, I managed to implement this when I used http_poller, but when I use http in the filter block, I can't parse the data correctly.
Thanks, I solved this problem.
It is necessary to use several splits in succession.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.