Receiving data to the ELK database using logstash

San9 · December 21, 2021, 9:53am

I have a problem with accessing the ELK database and receiving a specific field with a token to form a further request to connect to the o365 api. I can't get data from this particular field.
scheme, I connect to the ELK database (into which the token is previously loaded) and try to get the access_token, but apparently I'm blowing something right, since I can't get the attached data.

warkolm · December 22, 2021, 12:00am

Without providing more information you're unlikely to get much help.

What does your config look like?
What does the data you are trying to retrieve look like?

San9 · December 22, 2021, 12:31pm

Thank you for your answer.
Thank you for your answer.
I managed to get the token by replacing the input from the http_poller with a direct connection to the Elasticsearch. But now I can’t parse the received data correctly using a split. My config

input {
 elasticsearch {
        hosts => ["elasticsearch:9200"]
        index => "o365-token"
        query => '{    "_source": ["access_token"],    "query": {      "match_all": {}    }}'
        user => "xxx"
        password => "xxx"
        schedule => "*/2 * * * *"
        ssl => true
        size => 200
        docinfo => true
        ca_file => "/etc/logstash/certs/trust/ca.pem"
      }
    }
filter {
http {
        proxy => "http://proxy.com:8080/"
        url => "https://manage.office.com/api/v1.0/<my-tenant>/ServiceComms/CurrentStatus"
        verb => "GET"
        headers => {
          Accept => "application/json"
          Authorization => "Bearer %{access_token}"
#          body_format => "json"
        }
       }
split {
        field => "[body]"
        add_field => {
            "number" => "%{[result][number]}"
        }
        remove_field => "result"
        remove_field => "number"
}
split {
        field => "[body][value][FeatureStatus]"
  }
mutate {
        remove_field => [ "access_token" ]
      }
}
output {
     stdout {
        codec => rubydebug
    }
}

Part of my debug

[DEBUG] 2021-12-22 12:54:01.682 [[main]>worker0] wire - http-outgoing-3 << "HTTP/1.1 200 OK[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.682 [[main]>worker0] wire - http-outgoing-3 << "Cache-Control: no-cache[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.682 [[main]>worker0] wire - http-outgoing-3 << "Pragma: no-cache[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.682 [[main]>worker0] wire - http-outgoing-3 << "Content-Length: 27581[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.682 [[main]>worker0] wire - http-outgoing-3 << "Content-Type: application/json; odata.metadata=minimal[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "Expires: -1[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "Server: Microsoft-IIS/10.0[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "OData-Version: 4.0[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "X-Instance-Name: Office365ServiceCommsWebService_Office365ServiceCommsWebService_IN_23[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "X-Activity-Id: 8ca33b52-c02f-4f8c-8e3e-831505bd4[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "X-Time-Taken: 1332[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "Server: Microsoft-IIS/10.0[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "X-AspNet-Version: 4.0.30319[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "X-Powered-By: ASP.NET[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "X-Powered-By: ASP.NET[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "Date: Wed, 22 Dec 2021 10:54:01 GMT[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "{[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "  "@odata.context":"https://office365servicecomms-prod.cloudapp.net/api/v1.0/<my-tenent>/$metadata#CurrentStatus","value":[[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "    {[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "      "FeatureStatus":[[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "        {[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "          "FeatureDisplayName":"Microsoft Bookings","FeatureName":"MicrosoftBookings","FeatureServiceStatus":"ServiceOperational","FeatureServiceStatusDisplayName":"Normal service"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "        }[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "      ],"Id":"Bookings","IncidentIds":[[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "        [\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "      ],"Status":"ServiceOperational","StatusDisplayName":"Normal service","StatusTime":"2021-12-21T10:54:00.3642702Z","Workload":"Bookings","WorkloadDisplayName":"Microsoft Bookings"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "    },{[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "      "FeatureStatus":[[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "        {[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "          "FeatureDisplayName":"Cloud App Security","FeatureName":"Cloud App Security","FeatureServiceStatus":"ServiceDegradation","FeatureServiceStatusDisplayName":"Service degradation"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "        },{[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "          "FeatureDisplayName":"Office 365 Advanced Security Management","FeatureName":"Office 365 Advanced Security Management","FeatureServiceStatus":"ServiceOperational","FeatureServiceStatusDisplayName":"Normal service"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << "        },{[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << "          "FeatureDisplayName":"Cloud App Security Discovery","FeatureName":"Cloud App Security Discovery","FeatureServiceStatus":"ServiceOperational","FeatureServiceStatusDisplayName":"Normal service"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << "        },{[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << "          "FeatureDisplayName":"Microsoft Cloud App Security","FeatureName":"Microsoft Cloud App Security","FeatureServiceStatus":"ServiceOperational","FeatureServiceStatusDisplayName":"Normal service"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << "        }[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << "      ],"Id":"cloudappsecurity","IncidentIds":[[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << "        "CS300127"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << "      ],"Status":"ServiceDegradation","StatusDisplayName":"Service degradation","StatusTime":"2021-12-19T15:18:17.777Z","Workload":"cloudappsecurity","WorkloadDisplayName":"Microsoft Defender for Cloud Apps"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << "    },{[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << "      "FeatureStatus":[[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << "        {[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << "          "FeatureDisplayName":"Other","FeatureName":"DynAXOther","FeatureServiceStatus":"ServiceOperational","FeatureServiceStatusDisplayName":"Normal service"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << "        },{[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << "          "FeatureDisplayName":"Sign in","FeatureName":"signin","FeatureServiceStatus":"ServiceOperational","FeatureServiceStatusDisplayName":"Normal service"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << "        },{[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << "          "FeatureDisplayName":"Sign up and Administration","FeatureName":"admin","FeatureServiceStatus":"ServiceOperational","FeatureServiceStatusDisplayName":"Normal service"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << "        },{[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << "          "FeatureDisplayName":"Environment Access","FeatureName":"orgaccess","FeatureServiceStatus":"ServiceOperational","FeatureServiceStatusDisplayName":"Normal service"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << "        },{[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << "          "FeatureDisplayName":"Environment Performance","FeatureName":"orgperf","FeatureServiceStatus":"ServiceOperational","FeatureServiceStatusDisplayName":"Normal service"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << "        },{[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << "          "FeatureDisplayName":"Components/Features","FeatureName":"comp","FeatureServiceStatus":"ServiceOperational","FeatureServiceStatusDisplayName":"Normal service"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << "        }[\r][\n]"

San9 · December 24, 2021, 2:21pm

Part of my debug

[DEBUG] 2021-12-22 12:54:01.702 [[main]>worker0] http - success received {:code=>200, :body=>"{\r\n  \"@odata.context\":\"https://office365servicecomms-prod.cloudapp.net/api/v1.0/4<my-tenant>/$metadata#CurrentStatus\",\"value\":[\r\n    {\r\n      \"FeatureStatus\":[\r\n        {\r\n          \"FeatureDisplayName\":\"Microsoft Bookings\",\"FeatureName\":\"MicrosoftBookings\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n        }\r\n      ],\"Id\":\"Bookings\",\"IncidentIds\":[\r\n        \r\n      ],\"Status\":\"ServiceOperational\",\"StatusDisplayName\":\"Normal service\",\"StatusTime\":\"2021-12-21T10:54:00.3642702Z\",\"Workload\":\"Bookings\",\"WorkloadDisplayName\":\"Microsoft Bookings\"\r\n    },{\r\n      \"FeatureStatus\":[\r\n        {\r\n          \"FeatureDisplayName\":\"Cloud App Security\",\"FeatureName\":\"Cloud App Security\",\"FeatureServiceStatus\":\"ServiceDegradation\",\"FeatureServiceStatusDisplayName\":\"Service degradation\"\r\n        },{\r\n          \"FeatureDisplayName\":\"Office 365 Advanced Security Management\",\"FeatureName\":\"Office 365 Advanced Security Management\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n        },{\r\n          \"FeatureDisplayName\":\"Cloud App Security Discovery\",\"FeatureName\":\"Cloud App Security Discovery\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n        },{\r\n          \"FeatureDisplayName\":\"Microsoft Cloud App Security\",\"FeatureName\":\"Microsoft Cloud App Security\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n        }\r\n      ],\"Id\":\"cloudappsecurity\",\"IncidentIds\":[\r\n        \"CS300127\"\r\n      ],\"Status\":\"ServiceDegradation\",\"StatusDisplayName\":\"Service degradation\",\"StatusTime\":\"2021-12-19T15:18:17.777Z\",\"Workload\":\"cloudappsecurity\",\"WorkloadDisplayName\":\"Microsoft Defender for Cloud Apps\"\r\n    },{\r\n      \"FeatureStatus\":[\r\n        {\r\n          \"FeatureDisplayName\":\"Other\",\"FeatureName\":\"DynAXOther\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n        },{\r\n          \"FeatureDisplayName\":\"Sign in\",\"FeatureName\":\"signin\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n        },{\r\n          \"FeatureDisplayName\":\"Sign up and Administration\",\"FeatureName\":\"admin\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n        },{\r\n          \"FeatureDisplayName\":\"Environment Access\",\"FeatureName\":\"orgaccess\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n        },{\r\n          \"FeatureDisplayName\":\"Environment Performance\",\"FeatureName\":\"orgperf\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n        },{\r\n          \"FeatureDisplayName\":\"Components/Features\",\"FeatureName\":\"comp\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n        }\r\n      ],\"Id\":\"DynamicsAX\",\"IncidentIds\":[\r\n        \r\n      ],\"Status\":\"ServiceOperational\",\"StatusDisplayName\":\"Normal service\",\"StatusTime\":\"2021-12-21T10:54:00.3642702Z\",\"Workload\":\"DynamicsAX\",\"WorkloadDisplayName\":\"Finance and Operations Apps\"\r\n    },{\r\n      \"FeatureStatus\":[\r\n        {\r\n          \"FeatureDisplayName\":\"Other\",\"FeatureName\":\"DynCRMOther\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n        },{\r\n          \"FeatureDisplayName\":\"Sign In\",\"FeatureName\":\"signin\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n        },{\r\n          \"FeatureDisplayName\":\"Sign up and administration\",\"FeatureName\":\"admin\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n        },{\r\n          \"FeatureDisplayName\":\"Organization access\",\"FeatureName\":\"orgaccess\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n        },{\r\n          \"FeatureDisplayName\":\"Organization performance\",\"FeatureName\":\"orgperf\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n        },{\r\

and next

[WARN ] 2021-12-22 12:54:01.707 [[main]>worker0] split - Only String and Array types are splittable. field:[body] is of type = Hash
[WARN ] 2021-12-22 12:54:01.707 [[main]>worker0] split - Only String and Array types are splittable. field:[body][value][FeatureStatus] is of type = NilClass
[DEBUG] 2021-12-22 12:54:01.708 [[main]>worker0] mutate - filters/LogStash::Filters::Mutate: removing field {:field=>"access_token"}
[DEBUG] 2021-12-22 12:54:01.711 [[main]>worker0] elasticsearch - Sending final bulk request for batch. {:action_count=>1, :payload_size=>24493, :content_length=>24493, :batch_offset=>0}

San9 · December 24, 2021, 2:28pm

after parsing json, I expected to see something like

breakdown by services, I managed to implement this when I used http_poller, but when I use http in the filter block, I can't parse the data correctly.

San9 · January 11, 2022, 7:56am

Thanks, I solved this problem.
It is necessary to use several splits in succession.

system · February 8, 2022, 7:56am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Build filter chain Logstash	2	1160	June 3, 2018
Ingesting data from a database, via a API into ES Elasticsearch	3	491	July 16, 2018
Logstash elasticsearch input error Logstash	15	776	March 4, 2022
Query Data in ELK Elasticsearch	5	296	January 11, 2021
Logstash http poller question Logstash	2	287	January 15, 2019

Receiving data to the ELK database using logstash

Related topics