# Elastic-agent with Misp integration policy no data received while no errors comes up

**URL:** <https://discuss.elastic.co/t/elastic-agent-with-misp-integration-policy-no-data-received-while-no-errors-comes-up/328183>\
**Category:** Elastic Agent\
**Created:** [March 21, 2023, 3:58pm UTC](https://discuss.elastic.co/t/elastic-agent-with-misp-integration-policy-no-data-received-while-no-errors-comes-up/328183 "2023-03-21T15:58:05Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nicolas\_Pellletier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nicolas_pellletier/32/118685_2.png) [@Nicolas\_Pellletier](https://discuss.elastic.co/u/Nicolas_Pellletier)\
**Post date:** [March 21, 2023, 3:58pm UTC](https://discuss.elastic.co/t/elastic-agent-with-misp-integration-policy-no-data-received-while-no-errors-comes-up/328183/1 "2023-03-21T15:58:05Z")

</div>

Hello,  
I've got a standalone elastic-agent deployed on localhost where my MISP instance is running.  
I'm trying to integrate MISP IOC's to Elastic in order to use the dashboard.  
I don't understand why i don't receive any logs in data stream panel of integration while  
when i look at the elastic-agent logs everything seems to going alright (small extract):

```auto
{
  "log.level": "info",
  "@timestamp": "2023-03-21T16:50:48.976+0100",
  "message": "Process another repeated request.",
  "component": {
    "binary": "filebeat",
    "dataset": "elastic_agent.filebeat",
    "id": "httpjson-default",
    "type": "httpjson"
  },
  "log": {
    "source": "httpjson-default"
  },
  "ecs.version": "1.6.0",
  "log.logger": "input.httpjson-cursor",
  "id": "httpjson-ti_misp.threat-e53c4180-4444-4379-9a7f-8d739dcc0980",
  "input_source": "https://localhost/events/restSearch",
  "input_url": "https://localhost/events/restSearch",
  "log.origin": {
    "file.line": 132,
    "file.name": "httpjson/input.go"
  },
  "service.name": "filebeat"
}
{
  "log.level": "info",
  "@timestamp": "2023-03-21T16:49:49.083+0100",
  "message": "request finished: 40 events published",
  "component": {
    "binary": "filebeat",
    "dataset": "elastic_agent.filebeat",
    "id": "httpjson-default",
    "type": "httpjson"
  },
  "log": {
    "source": "httpjson-default"
  },
  "log.logger": "input.httpjson-cursor",
  "service.name": "filebeat",
  "input_source": "https://localhost/events/restSearch",
  "input_url": "https://localhost/events/restSearch",
  "ecs.version": "1.6.0",
  "log.origin": {
    "file.line": 445,
    "file.name": "httpjson/request.go"
  },
  "id": "httpjson-ti_misp.threat-e53c4180-4444-4379-9a7f-8d739dcc0980"
}

```

Here is the configuration of my elastic-agent:

```auto
id: 21c71600-c7ef-11ed-8ee2-b3eb91e32567
revision: 2
outputs:
  default:
    type: elasticsearch
    hosts:
      - 'http://localhost:9200'
    username: 'elastic'
    password: 'changeme'
output_permissions:
  default:
    _elastic_agent_monitoring:
      indices: []
    _elastic_agent_checks:
      cluster:
        - monitor
    e53c4180-4444-4379-9a7f-8d739dcc0980:
      indices:
        - names:
            - logs-ti_misp.threat-default
          privileges:
            - auto_configure
            - create_doc
agent:
  download:
    sourceURI: 'https://artifacts.elastic.co/downloads/'
  monitoring:
    enabled: false
    logs: false
    metrics: false
inputs:
  - id: httpjson-ti_misp-e53c4180-4444-4379-9a7f-8d739dcc0980
    name: MISP Integration
    revision: 1
    type: httpjson
    use_output: default
    meta:
      package:
        name: ti_misp
        version: 1.10.1
    data_stream:
      namespace: default
    package_policy_id: e53c4180-4444-4379-9a7f-8d739dcc0980
    streams:
      - id: httpjson-ti_misp.threat-e53c4180-4444-4379-9a7f-8d739dcc0980
        data_stream:
          dataset: ti_misp.threat
          type: logs
        config_version: '2'
        interval: 2m
        request.method: POST
        request.url: 'https://localhost/events/restSearch'
        request.ssl:
          verification_mode: none
        request.timeout: 30s
        request.body: null
        request.transforms:
          - set:
              target: header.Authorization
              value: BEpdSXuPb2lRyhVjNy9nHiA7EApYdD9ajMRafBZQ
          - set:
              target: body.limit
              value: 1
          - set:
              target: body.returnFormat
              value: json
          - set:
              target: body.timestamp
              value: '[[.cursor.timestamp]]'
              default: '[[formatDate (now (parseDuration "-6000h")) "UnixDate"]]'
        response.split:
          target: body.response
          split:
            target: body.Event.Attribute
            ignore_empty_value: true
            keep_parent: true
            split:
              target: body.Event.Object
              keep_parent: true
              split:
                target: body.Event.Object.Attribute
                keep_parent: true
        cursor:
          timestamp:
            value: '[[.last_event.Event.timestamp]]'
        tags:
          - preserve_original_event
          - forwarded
          - misp-threat
        publisher_pipeline.disable_host: true

```

---

<div class="post-metadata">

**Author:** ![Marius\_Iversen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_iversen/32/68988_2.png) [@Marius\_Iversen](https://discuss.elastic.co/u/Marius_Iversen)\
**Post date:** [March 21, 2023, 7:31pm UTC](https://discuss.elastic.co/t/elastic-agent-with-misp-integration-policy-no-data-received-while-no-errors-comes-up/328183/2 "2023-03-21T19:31:43Z")

</div>

How are you checking that you don't have any data? From the logs it does indeed look successfull.

---

<div class="post-metadata">

**Author:** ![Nicolas\_Pellletier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nicolas_pellletier/32/118685_2.png) [@Nicolas\_Pellletier](https://discuss.elastic.co/u/Nicolas_Pellletier)\
**Post date:** [March 21, 2023, 7:53pm UTC](https://discuss.elastic.co/t/elastic-agent-with-misp-integration-policy-no-data-received-while-no-errors-comes-up/328183/3 "2023-03-21T19:53:53Z")

</div>

Hello Marius,

Thank you for your response,

It's when i look at the kibana data stream panel i've got nothing related to misp. Here is the link: [http://localhost:5601/app/fleet/data-streams](http://localhost:5601/app/fleet/data-streams) while i received all agent metrics/logs when i toggled on this field. But nothing for Misp.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/7/07a614038e04d7743a842a4e8b91569c24a123da.png)

---

<div class="post-metadata">

**Author:** ![Marius\_Iversen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_iversen/32/68988_2.png) [@Marius\_Iversen](https://discuss.elastic.co/u/Marius_Iversen)\
**Post date:** [March 21, 2023, 8:08pm UTC](https://discuss.elastic.co/t/elastic-agent-with-misp-integration-policy-no-data-received-while-no-errors-comes-up/328183/4 "2023-03-21T20:08:24Z")

</div>

You should be seeing a lot more datastreams there than only MISP, if you can't see any it's almost like nothing has started up yet.  
If you are collecting Agent Metrics and logs then those datastreams should also be there, yet you see nothing.

Are you sure you don't have any filters on? Like the Dataset, Type, Namespace or Integration filters in your picture.

Another better way to make sure, is to go to Discover, and filter on `data_stream.dataset: ti_misp.threat`

---

<div class="post-metadata">

**Author:** ![Nicolas\_Pellletier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nicolas_pellletier/32/118685_2.png) [@Nicolas\_Pellletier](https://discuss.elastic.co/u/Nicolas_Pellletier)\
**Post date:** [March 21, 2023, 8:18pm UTC](https://discuss.elastic.co/t/elastic-agent-with-misp-integration-policy-no-data-received-while-no-errors-comes-up/328183/5 "2023-03-21T20:18:35Z")

</div>

Actually if you look at my `elastic-agent.yml` file I did my best not to get any other logs than the ones I am interested in (those of MISP). Because I had memory problems in the past (heap size reach the limit).  
I went to Discover view in Kiban and filter on `data_stream.dataset: ti_misp.threat` but got nothing  
Anyway after i toggle on the monitoring of elastic agent, i receive some data about elastic agent in the kibana data stream view but still nothing about MISP.

I have documents stored related to MISP if you look at the index view in Elasticsearch:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/b/eb640f8c493889dcaebed9fd355bd9685e0f4b2a.png)

Furthermore when i check the document stored in elasticsearch threat\_intel index (MISP) with this API request: [http://localhost:9200/.ds-logs-ti\_misp.threat-default-2023.03.21-000002/\_search?size=10000](http://localhost:9200/.ds-logs-ti_misp.threat-default-2023.03.21-000002/_search?size=10000), i see all the MISP event information with attribute infos.

```auto
{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 10000,
      "relation": "gte"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": ".ds-logs-ti_misp.threat-default-2023.03.21-000002",
        "_id": "faOSTgqmuCG/ybKey/JN2Lt1ApU=",
        "_score": 1,
        "_source": {
          "input": {
            "type": "httpjson"
          },
          "agent": {
            "name": "nicop-IdeaPad-5-Pro-14ARH7",
            "id": "ab85b3d0-f7d9-4b59-b810-814d5695b83c",
            "ephemeral_id": "f402f0c8-a8a0-4d59-802d-abf23b0b79c7",
            "type": "filebeat",
            "version": "8.6.2"
          },
          "@timestamp": "2023-03-18T16:04:05.000Z",
          "ecs": {
            "version": "8.6.0"
          },
          "data_stream": {
            "namespace": "default",
            "type": "logs",
            "dataset": "ti_misp.threat"
          },
          "elastic_agent": {
            "id": "ab85b3d0-f7d9-4b59-b810-814d5695b83c",
            "version": "8.6.2",
            "snapshot": false
          },
          "misp": {
            "date": "2018-08-17",
            "threat_level_id": 1,
            "attribute_count": 57,
            "orgc": {
              "name": "ESET",
              "id": "9",
              "uuid": "55f6ea5e-51ac-4344-bc8c-4170950d210f",
              "local": false
            },
            "published": true,
            "distribution": "3",
            "uuid": "5b773e07-e694-458b-b99c-27f30a016219",
            "orgc_id": "9",
            "publish_timestamp": "1679155445",
            "disable_correlation": false,
            "extends_uuid": "",
            "proposal_email_lock": false,
            "sharing_group_id": "0",
            "org_id": "1",
            "context": {
              "attribute": {
                "distribution": 5,
                "type": "filename",
                "object_id": "0",
                "uuid": "5b773e89-9738-4bbb-90bc-2fb20a016219",
                "to_ids": false,
                "disable_correlation": false,
                "deleted": false,
                "event_id": "3996",
                "sharing_group_id": "0",
                "comment": "",
                "id": "632076",
                "category": "Artifacts dropped",
                "value": "%appdata%\\Microsoft\\Windows\\scawrdot.db",
                "timestamp": "1534805111"
              }
            },
            "id": "3996",
            "attribute": {
              "distribution": 5,
              "type": "link",
              "object_id": "63363",
              "uuid": "84e013cb-ecaf-4f21-9ee8-796886e3454a",
              "object_relation": "permalink",
              "to_ids": false,
              "disable_correlation": false,
              "deleted": false,
              "event_id": "3996",
              "sharing_group_id": "0",
              "comment": "",
              "id": "913508",
              "category": "External analysis",
              "timestamp": "1535632211"
            },
            "locked": false,
            "info": "Turla Outlook White Paper"
          },

```

Do you have any advice to give me on how to deepen the research, or to get anything related to MISP in Kibana ?

---

<div class="post-metadata">

**Author:** ![Marius\_Iversen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_iversen/32/68988_2.png) [@Marius\_Iversen](https://discuss.elastic.co/u/Marius_Iversen)\
**Post date:** [March 22, 2023, 1:57pm UTC](https://discuss.elastic.co/t/elastic-agent-with-misp-integration-policy-no-data-received-while-no-errors-comes-up/328183/7 "2023-03-22T13:57:00Z")

</div>

When you went to discover, what Data view do you use at the top left of the UI? It should say `logs-`\*

![image](https://us1.discourse-cdn.com/elastic/original/3X/7/0/7078e27d500b217fcc1bafc259986641eaa0da91.png)

As long as the data is already in the datastream, and you can query it, then this is either a permission issue for your user (unless you are superuser), or you have not choosen the correct data-view in discover as the screenshot above.

Seems like there are multiple discuss topics now? Let's keep it to one: [Bug: No Misp event data send to Kibana when Threat intel module used](https://discuss.elastic.co/t/bug-no-misp-event-data-send-to-kibana-when-threat-intel-module-used/327979)

---

<div class="post-metadata">

**Author:** ![Nicolas\_Pellletier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nicolas_pellletier/32/118685_2.png) [@Nicolas\_Pellletier](https://discuss.elastic.co/u/Nicolas_Pellletier)\
**Post date:** [March 22, 2023, 2:19pm UTC](https://discuss.elastic.co/t/elastic-agent-with-misp-integration-policy-no-data-received-while-no-errors-comes-up/328183/8 "2023-03-22T14:19:22Z")

</div>

Hello Marius,

Thank you for your response 🙂

Actually i received some MISP information data in kibana. But very lately i think if we compare to the time the elastic-agent was launched. As i was running threat\_intel module i didn't know if this data was comming from filebeat or elastic-agent.  
As the data wasn't parsed and all event information were contained in one field `event.message ` I think it's fileabeat data that needs to be parse with processors and other thing i think.

Unfortunately i decided to restart it from begining and erase all persistant data to restart from scratch but the data was in logs field of discover view and all data was comming from `ti_misp.threat-default-2023.03.21-000001` index

Now that i've restarted i'm stick with this error: [Elastic-agent "Process another repeated request" in loop indefinitely](https://discuss.elastic.co/t/elastic-agent-process-another-repeated-request-in-loop-indefinitely/328251)

---

<div class="post-metadata">

**Author:** ![Marius\_Iversen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_iversen/32/68988_2.png) [@Marius\_Iversen](https://discuss.elastic.co/u/Marius_Iversen)\
**Post date:** [March 22, 2023, 2:35pm UTC](https://discuss.elastic.co/t/elastic-agent-with-misp-integration-policy-no-data-received-while-no-errors-comes-up/328183/9 "2023-03-22T14:35:06Z")

</div>

Try to keep it all in one issue, so that its easier to follow up, I will take a look at the new one then, which seems healthier.

---

<div class="post-metadata">

**Author:** ![Nicolas\_Pellletier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nicolas_pellletier/32/118685_2.png) [@Nicolas\_Pellletier](https://discuss.elastic.co/u/Nicolas_Pellletier)\
**Post date:** [March 22, 2023, 2:37pm UTC](https://discuss.elastic.co/t/elastic-agent-with-misp-integration-policy-no-data-received-while-no-errors-comes-up/328183/10 "2023-03-22T14:37:58Z")

</div>

Ok sorry about that,

Thank you, i will reply you on this post [Bug: No Misp event data send to Kibana when Threat intel module used - #5 by Marius\_Iversen](https://discuss.elastic.co/t/bug-no-misp-event-data-send-to-kibana-when-threat-intel-module-used/327979/5).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 19, 2023, 2:38pm UTC](https://discuss.elastic.co/t/elastic-agent-with-misp-integration-policy-no-data-received-while-no-errors-comes-up/328183/11 "2023-04-19T14:38:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.