# Receiving data to the ELK database using logstash

**URL:** <https://discuss.elastic.co/t/receiving-data-to-the-elk-database-using-logstash/292547>\
**Category:** Logstash\
**Created:** [December 21, 2021, 9:53am UTC](https://discuss.elastic.co/t/receiving-data-to-the-elk-database-using-logstash/292547 "2021-12-21T09:53:06Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![San9](https://avatars.discourse-cdn.com/v4/letter/s/a9adbd/32.png) [@San9](https://discuss.elastic.co/u/San9)\
**Post date:** [December 21, 2021, 9:53am UTC](https://discuss.elastic.co/t/receiving-data-to-the-elk-database-using-logstash/292547/1 "2021-12-21T09:53:06Z")

</div>

I have a problem with accessing the ELK database and receiving a specific field with a token to form a further request to connect to the o365 api. I can't get data from this particular field.  
scheme, I connect to the ELK database (into which the token is previously loaded) and try to get the access\_token, but apparently I'm blowing something right, since I can't get the attached data.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [December 22, 2021, 12:00am UTC](https://discuss.elastic.co/t/receiving-data-to-the-elk-database-using-logstash/292547/2 "2021-12-22T00:00:53Z")

</div>

Without providing more information you're unlikely to get much help.

What does your config look like?  
What does the data you are trying to retrieve look like?

---

<div class="post-metadata">

**Author:** ![San9](https://avatars.discourse-cdn.com/v4/letter/s/a9adbd/32.png) [@San9](https://discuss.elastic.co/u/San9)\
**Post date:** [December 22, 2021, 12:31pm UTC](https://discuss.elastic.co/t/receiving-data-to-the-elk-database-using-logstash/292547/3 "2021-12-22T12:31:31Z")

</div>

Thank you for your answer.  
Thank you for your answer.  
I managed to get the token by replacing the input from the http\_poller with a direct connection to the Elasticsearch. But now I can’t parse the received data correctly using a split. My config

```auto
input {
 elasticsearch {
        hosts => ["elasticsearch:9200"]
        index => "o365-token"
        query => '{ "_source": ["access_token"], "query": { "match_all": {} }}'
        user => "xxx"
        password => "xxx"
        schedule => "*/2 * * * *"
        ssl => true
        size => 200
        docinfo => true
        ca_file => "/etc/logstash/certs/trust/ca.pem"
      }
    }
filter {
http {
        proxy => "http://proxy.com:8080/"
        url => "https://manage.office.com/api/v1.0/<my-tenant>/ServiceComms/CurrentStatus"
        verb => "GET"
        headers => {
          Accept => "application/json"
          Authorization => "Bearer %{access_token}"
# body_format => "json"
        }
       }
split {
        field => "[body]"
        add_field => {
            "number" => "%{[result][number]}"
        }
        remove_field => "result"
        remove_field => "number"
}
split {
        field => "[body][value][FeatureStatus]"
  }
mutate {
        remove_field => ["access_token"]
      }
}
output {
     stdout {
        codec => rubydebug
    }
}

```

Part of my debug

```auto
[DEBUG] 2021-12-22 12:54:01.682 [[main]>worker0] wire - http-outgoing-3 << "HTTP/1.1 200 OK[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.682 [[main]>worker0] wire - http-outgoing-3 << "Cache-Control: no-cache[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.682 [[main]>worker0] wire - http-outgoing-3 << "Pragma: no-cache[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.682 [[main]>worker0] wire - http-outgoing-3 << "Content-Length: 27581[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.682 [[main]>worker0] wire - http-outgoing-3 << "Content-Type: application/json; odata.metadata=minimal[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "Expires: -1[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "Server: Microsoft-IIS/10.0[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "OData-Version: 4.0[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "X-Instance-Name: Office365ServiceCommsWebService_Office365ServiceCommsWebService_IN_23[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "X-Activity-Id: 8ca33b52-c02f-4f8c-8e3e-831505bd4[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "X-Time-Taken: 1332[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "Server: Microsoft-IIS/10.0[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "X-AspNet-Version: 4.0.30319[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "X-Powered-By: ASP.NET[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "X-Powered-By: ASP.NET[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "Date: Wed, 22 Dec 2021 10:54:01 GMT[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << "{[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << " "@odata.context":"https://office365servicecomms-prod.cloudapp.net/api/v1.0/<my-tenent>/$metadata#CurrentStatus","value":[[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << " {[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << " "FeatureStatus":[[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << " {[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << " "FeatureDisplayName":"Microsoft Bookings","FeatureName":"MicrosoftBookings","FeatureServiceStatus":"ServiceOperational","FeatureServiceStatusDisplayName":"Normal service"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << " }[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << " ],"Id":"Bookings","IncidentIds":[[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << " [\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << " ],"Status":"ServiceOperational","StatusDisplayName":"Normal service","StatusTime":"2021-12-21T10:54:00.3642702Z","Workload":"Bookings","WorkloadDisplayName":"Microsoft Bookings"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << " },{[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << " "FeatureStatus":[[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << " {[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << " "FeatureDisplayName":"Cloud App Security","FeatureName":"Cloud App Security","FeatureServiceStatus":"ServiceDegradation","FeatureServiceStatusDisplayName":"Service degradation"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << " },{[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.683 [[main]>worker0] wire - http-outgoing-3 << " "FeatureDisplayName":"Office 365 Advanced Security Management","FeatureName":"Office 365 Advanced Security Management","FeatureServiceStatus":"ServiceOperational","FeatureServiceStatusDisplayName":"Normal service"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " },{[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " "FeatureDisplayName":"Cloud App Security Discovery","FeatureName":"Cloud App Security Discovery","FeatureServiceStatus":"ServiceOperational","FeatureServiceStatusDisplayName":"Normal service"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " },{[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " "FeatureDisplayName":"Microsoft Cloud App Security","FeatureName":"Microsoft Cloud App Security","FeatureServiceStatus":"ServiceOperational","FeatureServiceStatusDisplayName":"Normal service"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " }[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " ],"Id":"cloudappsecurity","IncidentIds":[[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " "CS300127"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " ],"Status":"ServiceDegradation","StatusDisplayName":"Service degradation","StatusTime":"2021-12-19T15:18:17.777Z","Workload":"cloudappsecurity","WorkloadDisplayName":"Microsoft Defender for Cloud Apps"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " },{[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " "FeatureStatus":[[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " {[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " "FeatureDisplayName":"Other","FeatureName":"DynAXOther","FeatureServiceStatus":"ServiceOperational","FeatureServiceStatusDisplayName":"Normal service"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " },{[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " "FeatureDisplayName":"Sign in","FeatureName":"signin","FeatureServiceStatus":"ServiceOperational","FeatureServiceStatusDisplayName":"Normal service"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " },{[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " "FeatureDisplayName":"Sign up and Administration","FeatureName":"admin","FeatureServiceStatus":"ServiceOperational","FeatureServiceStatusDisplayName":"Normal service"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " },{[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " "FeatureDisplayName":"Environment Access","FeatureName":"orgaccess","FeatureServiceStatus":"ServiceOperational","FeatureServiceStatusDisplayName":"Normal service"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " },{[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " "FeatureDisplayName":"Environment Performance","FeatureName":"orgperf","FeatureServiceStatus":"ServiceOperational","FeatureServiceStatusDisplayName":"Normal service"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " },{[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " "FeatureDisplayName":"Components/Features","FeatureName":"comp","FeatureServiceStatus":"ServiceOperational","FeatureServiceStatusDisplayName":"Normal service"[\r][\n]"
[DEBUG] 2021-12-22 12:54:01.684 [[main]>worker0] wire - http-outgoing-3 << " }[\r][\n]"

```

---

<div class="post-metadata">

**Author:** ![San9](https://avatars.discourse-cdn.com/v4/letter/s/a9adbd/32.png) [@San9](https://discuss.elastic.co/u/San9)\
**Post date:** [December 24, 2021, 2:21pm UTC](https://discuss.elastic.co/t/receiving-data-to-the-elk-database-using-logstash/292547/5 "2021-12-24T14:21:26Z")

</div>

> [@San9](#):
>
> Part of my debug

Part of my debug

```auto
[DEBUG] 2021-12-22 12:54:01.702 [[main]>worker0] http - success received {:code=>200, :body=>"{\r\n \"@odata.context\":\"https://office365servicecomms-prod.cloudapp.net/api/v1.0/4<my-tenant>/$metadata#CurrentStatus\",\"value\":[\r\n {\r\n \"FeatureStatus\":[\r\n {\r\n \"FeatureDisplayName\":\"Microsoft Bookings\",\"FeatureName\":\"MicrosoftBookings\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n }\r\n],\"Id\":\"Bookings\",\"IncidentIds\":[\r\n \r\n],\"Status\":\"ServiceOperational\",\"StatusDisplayName\":\"Normal service\",\"StatusTime\":\"2021-12-21T10:54:00.3642702Z\",\"Workload\":\"Bookings\",\"WorkloadDisplayName\":\"Microsoft Bookings\"\r\n },{\r\n \"FeatureStatus\":[\r\n {\r\n \"FeatureDisplayName\":\"Cloud App Security\",\"FeatureName\":\"Cloud App Security\",\"FeatureServiceStatus\":\"ServiceDegradation\",\"FeatureServiceStatusDisplayName\":\"Service degradation\"\r\n },{\r\n \"FeatureDisplayName\":\"Office 365 Advanced Security Management\",\"FeatureName\":\"Office 365 Advanced Security Management\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n },{\r\n \"FeatureDisplayName\":\"Cloud App Security Discovery\",\"FeatureName\":\"Cloud App Security Discovery\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n },{\r\n \"FeatureDisplayName\":\"Microsoft Cloud App Security\",\"FeatureName\":\"Microsoft Cloud App Security\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n }\r\n],\"Id\":\"cloudappsecurity\",\"IncidentIds\":[\r\n \"CS300127\"\r\n],\"Status\":\"ServiceDegradation\",\"StatusDisplayName\":\"Service degradation\",\"StatusTime\":\"2021-12-19T15:18:17.777Z\",\"Workload\":\"cloudappsecurity\",\"WorkloadDisplayName\":\"Microsoft Defender for Cloud Apps\"\r\n },{\r\n \"FeatureStatus\":[\r\n {\r\n \"FeatureDisplayName\":\"Other\",\"FeatureName\":\"DynAXOther\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n },{\r\n \"FeatureDisplayName\":\"Sign in\",\"FeatureName\":\"signin\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n },{\r\n \"FeatureDisplayName\":\"Sign up and Administration\",\"FeatureName\":\"admin\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n },{\r\n \"FeatureDisplayName\":\"Environment Access\",\"FeatureName\":\"orgaccess\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n },{\r\n \"FeatureDisplayName\":\"Environment Performance\",\"FeatureName\":\"orgperf\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n },{\r\n \"FeatureDisplayName\":\"Components/Features\",\"FeatureName\":\"comp\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n }\r\n],\"Id\":\"DynamicsAX\",\"IncidentIds\":[\r\n \r\n],\"Status\":\"ServiceOperational\",\"StatusDisplayName\":\"Normal service\",\"StatusTime\":\"2021-12-21T10:54:00.3642702Z\",\"Workload\":\"DynamicsAX\",\"WorkloadDisplayName\":\"Finance and Operations Apps\"\r\n },{\r\n \"FeatureStatus\":[\r\n {\r\n \"FeatureDisplayName\":\"Other\",\"FeatureName\":\"DynCRMOther\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n },{\r\n \"FeatureDisplayName\":\"Sign In\",\"FeatureName\":\"signin\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n },{\r\n \"FeatureDisplayName\":\"Sign up and administration\",\"FeatureName\":\"admin\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n },{\r\n \"FeatureDisplayName\":\"Organization access\",\"FeatureName\":\"orgaccess\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n },{\r\n \"FeatureDisplayName\":\"Organization performance\",\"FeatureName\":\"orgperf\",\"FeatureServiceStatus\":\"ServiceOperational\",\"FeatureServiceStatusDisplayName\":\"Normal service\"\r\n },{\r\

```

and next

```auto
[WARN] 2021-12-22 12:54:01.707 [[main]>worker0] split - Only String and Array types are splittable. field:[body] is of type = Hash
[WARN] 2021-12-22 12:54:01.707 [[main]>worker0] split - Only String and Array types are splittable. field:[body][value][FeatureStatus] is of type = NilClass
[DEBUG] 2021-12-22 12:54:01.708 [[main]>worker0] mutate - filters/LogStash::Filters::Mutate: removing field {:field=>"access_token"}
[DEBUG] 2021-12-22 12:54:01.711 [[main]>worker0] elasticsearch - Sending final bulk request for batch. {:action_count=>1, :payload_size=>24493, :content_length=>24493, :batch_offset=>0}

```

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/3/a349f9283c949cfee98bfb295ae571c4538a90e1.png)  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/a/6a6cb505f7bdfb432bc249eb6eae9667bb17a9c9.png)

---

<div class="post-metadata">

**Author:** ![San9](https://avatars.discourse-cdn.com/v4/letter/s/a9adbd/32.png) [@San9](https://discuss.elastic.co/u/San9)\
**Post date:** [December 24, 2021, 2:28pm UTC](https://discuss.elastic.co/t/receiving-data-to-the-elk-database-using-logstash/292547/6 "2021-12-24T14:28:14Z")

</div>

after parsing json, I expected to see something like

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/8/484f682f889d97009695b5a6d65080a105d71bb5.png)

breakdown by services, I managed to implement this when I used http\_poller, but when I use http in the filter block, I can't parse the data correctly.

---

<div class="post-metadata">

**Author:** ![San9](https://avatars.discourse-cdn.com/v4/letter/s/a9adbd/32.png) [@San9](https://discuss.elastic.co/u/San9)\
**Post date:** [January 11, 2022, 7:56am UTC](https://discuss.elastic.co/t/receiving-data-to-the-elk-database-using-logstash/292547/7 "2022-01-11T07:56:20Z")

</div>

Thanks, I solved this problem.  
It is necessary to use several splits in succession.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 8, 2022, 7:56am UTC](https://discuss.elastic.co/t/receiving-data-to-the-elk-database-using-logstash/292547/8 "2022-02-08T07:56:38Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.