# Ingest Manager - Custom Files not being uploaded

**URL:** <https://discuss.elastic.co/t/ingest-manager-custom-files-not-being-uploaded/253397>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [October 27, 2020, 7:21am UTC](https://discuss.elastic.co/t/ingest-manager-custom-files-not-being-uploaded/253397 "2020-10-27T07:21:51Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![\_Meir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/_meir/32/77877_2.png) [@\_Meir](https://discuss.elastic.co/u/_Meir)\
**Post date:** [October 27, 2020, 7:21am UTC](https://discuss.elastic.co/t/ingest-manager-custom-files-not-being-uploaded/253397/1 "2020-10-27T07:21:51Z")

</div>

I'm trying to get custom logs uploaded from my server to my elastic cloud .  
I'm using the Ingest Manager via fleet deployment.  
I've added Windows, System and Custom Logs integrations.  
I'm only seeing logs from MetricBeat on the Discover page and nothing from the custom logs.  
I get this message in the elastic agent log: , error: failed to filter configuration: provided datastream dataset is invalid.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/0/70f345e304687cd50b6d6097ea29c7a91b34d6cb.png)  
In the Custom log integration I specified a new dataset name.

Any ideas how I can get this to work?

---

<div class="post-metadata">

**Author:** ![Alex\_Kristiansen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_kristiansen/32/46086_2.png) [@Alex\_Kristiansen](https://discuss.elastic.co/u/Alex_Kristiansen)\
**Post date:** [October 27, 2020, 3:53pm UTC](https://discuss.elastic.co/t/ingest-manager-custom-files-not-being-uploaded/253397/2 "2020-10-27T15:53:51Z")

</div>

@_Meir can you copy and paste the error? It looks like part of it is getting cut off.

---

<div class="post-metadata">

**Author:** ![\_Meir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/_meir/32/77877_2.png) [@\_Meir](https://discuss.elastic.co/u/_Meir)\
**Post date:** [October 27, 2020, 5:36pm UTC](https://discuss.elastic.co/t/ingest-manager-custom-files-not-being-uploaded/253397/3 "2020-10-27T17:36:40Z")

</div>

```auto
|2020-10-27T11:54:02.301+0200|DEBUG|application/fleet_gateway.go:142|FleetGateway calling Checkin API|
|---|---|---|---|
|2020-10-27T11:54:02.306+0200|DEBUG|kibana/client.go:170|Request method: POST, path: /api/ingest_manager/fleet/agents/a7fb57f9-b53d-4a98-8f6d-92b31c04e7c9/checkin|
|2020-10-27T11:54:02.911+0200|DEBUG|application/action_dispatcher.go:81|Dispatch 1 actions of types: *fleetapi.ActionConfigChange|
|2020-10-27T11:54:02.912+0200|DEBUG|application/handler_action_policy_change.go:23|handlerConfigChange: action 'action_id: b14afaa3-8e64-4397-a056-f730290339ef, type: CONFIG_CHANGE' received|
|2020-10-27T11:54:02.914+0200|DEBUG|application/handler_action_policy_change.go:34|handlerConfigChange: emit configuration for action action_id: b14afaa3-8e64-4397-a056-f730290339ef, type: CONFIG_CHANGE|
|2020-10-27T11:54:02.914+0200|DEBUG|application/emitter.go:39|Transforming configuration into a tree|
|2020-10-27T11:54:02.915+0200|DEBUG|application/action_dispatcher.go:93|Failed to dispatch action 'action_id: b14afaa3-8e64-4397-a056-f730290339ef, type: CONFIG_CHANGE', error: failed to filter configuration: provided datastream dataset is invalid: /go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/agent/application/filters/stream_checker.go[20]: unknown error|
|2020-10-27T11:54:02.915+0200|ERROR|application/fleet_gateway.go:159|failed to dispatch actions, error: failed to filter configuration: provided datastream dataset is invalid: /go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/agent/application/filters/stream_checker.go[20]: unknown error|
|2020-10-27T11:54:02.915+0200|DEBUG|application/fleet_gateway.go:162|FleetGateway is sleeping, next update in 1s|
    `indent preformatted text by 4 spaces`

```

---

<div class="post-metadata">

**Author:** ![Alex\_Kristiansen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_kristiansen/32/46086_2.png) [@Alex\_Kristiansen](https://discuss.elastic.co/u/Alex_Kristiansen)\
**Post date:** [October 30, 2020, 3:50pm UTC](https://discuss.elastic.co/t/ingest-manager-custom-files-not-being-uploaded/253397/4 "2020-10-30T15:50:03Z")

</div>

> [@\_Meir](#):
>
> ```auto
> 2020-10-27T11:54:02.915+0200|DEBUG|application/action_dispatcher.go:93|Failed to dispatch action 'action_id: b14afaa3-8e64-4397-a056-f730290339ef, type: CONFIG_CHANGE', error: failed to filter configuration: provided datastream dataset is invalid: /go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/agent/application/filters/stream_checker.go[20]: unknown error|
> |2020-10-27T11:54:02.915+0200|ERROR|application/fleet_gateway.go:159|failed to dispatch actions, error: failed to filter configuration: provided datastream dataset is invalid: /go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/agent/application/filters/stream_checker.go[20]: unknown error|
> 
> ```

At very least, that's an unhelpful error message. I'm gonna look more into it.

---

<div class="post-metadata">

**Author:** ![Alex\_Kristiansen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_kristiansen/32/46086_2.png) [@Alex\_Kristiansen](https://discuss.elastic.co/u/Alex_Kristiansen)\
**Post date:** [October 30, 2020, 3:51pm UTC](https://discuss.elastic.co/t/ingest-manager-custom-files-not-being-uploaded/253397/5 "2020-10-30T15:51:02Z")

</div>

@_Meir can you post your fleet config? Did you change any defaults? What are you doing with the custom log integration?

---

<div class="post-metadata">

**Author:** ![Alex\_Kristiansen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_kristiansen/32/46086_2.png) [@Alex\_Kristiansen](https://discuss.elastic.co/u/Alex_Kristiansen)\
**Post date:** [October 30, 2020, 3:53pm UTC](https://discuss.elastic.co/t/ingest-manager-custom-files-not-being-uploaded/253397/6 "2020-10-30T15:53:56Z")

</div>

@_Meir You can copy-and-paste the raw config by going to the config and selecting Actions-\>View Policy

 ![Screen Shot 2020-10-30 at 8.53.02 AM](https://us1.discourse-cdn.com/elastic/original/3X/8/9/89a3cc6cb5de83d4a5b4c4e58f25a31095f51bb1.png)

---

<div class="post-metadata">

**Author:** ![dao](https://avatars.discourse-cdn.com/v4/letter/d/a6a055/32.png) [@dao](https://discuss.elastic.co/u/dao)\
**Post date:** [October 30, 2020, 6:27pm UTC](https://discuss.elastic.co/t/ingest-manager-custom-files-not-being-uploaded/253397/7 "2020-10-30T18:27:23Z")

</div>

Same issue for me. I'd like to switch to this feature, but cannot do custom easily. Is there a doc on this topic?

I am in a middle of tries and fails, and always the same issue as you

---

<div class="post-metadata">

**Author:** ![\_Meir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/_meir/32/77877_2.png) [@\_Meir](https://discuss.elastic.co/u/_Meir)\
**Post date:** [November 1, 2020, 9:14am UTC](https://discuss.elastic.co/t/ingest-manager-custom-files-not-being-uploaded/253397/8 "2020-11-01T09:14:00Z")

</div>

Here's the config.  
I'm not aware of any significant changes to default configs.

```auto
id: 18ffc050-1787-11eb-ad66-a14f0fd0af4a
revision: 13
outputs:
  default:
    type: elasticsearch
    hosts:
      - 'https://e232f0e098bc40f5b28d3c6c4c908d89.us-east-1.aws.found.io:443'
agent:
  monitoring:
    enabled: true
    use_output: default
    logs: true
    metrics: true
inputs:
  - id: 1acc1be0-1787-11eb-ad66-a14f0fd0af4a
    name: system-1
    type: logfile
    use_output: default
    meta:
      package:
        name: system
        version: 0.5.3
    data_stream:
      namespace: default
    streams:
      - id: logfile-system.auth
        data_stream:
          dataset: system.auth
          type: logs
        paths:
          - /var/log/auth.log*
          - /var/log/secure*
        exclude_files:
          - .gz$
        multiline:
          pattern: ^\s
          match: after
        processors:
          - add_locale: null
          - add_fields:
              target: ''
              fields:
                ecs.version: 1.5.0
      - id: logfile-system.syslog
        data_stream:
          dataset: system.syslog
          type: logs
        paths:
          - /var/log/messages*
          - /var/log/syslog*
        exclude_files:
          - .gz$
        multiline:
          pattern: ^\s
          match: after
        processors:
          - add_locale: null
          - add_fields:
              target: ''
              fields:
                ecs.version: 1.5.0
  - id: 1acc1be0-1787-11eb-ad66-a14f0fd0af4a
    name: system-1
    type: system/metrics
    use_output: default
    meta:
      package:
        name: system
        version: 0.5.3
    data_stream:
      namespace: default
    streams:
      - id: system/metrics-system.cpu
        data_stream:
          dataset: system.cpu
          type: metrics
        metricsets:
          - cpu
        cpu.metrics:
          - percentages
          - normalized_percentages
        period: 10s
      - id: system/metrics-system.diskio
        data_stream:
          dataset: system.diskio
          type: metrics
        metricsets:
          - diskio
        diskio.include_devices: null
        period: 10s
      - id: system/metrics-system.load
        data_stream:
          dataset: system.load
          type: metrics
        metricsets:
          - load
        period: 10s
      - id: system/metrics-system.memory
        data_stream:
          dataset: system.memory
          type: metrics
        metricsets:
          - memory
        period: 10s
      - id: system/metrics-system.network
        data_stream:
          dataset: system.network
          type: metrics
        metricsets:
          - network
        period: 10s
        network.interfaces: null
      - id: system/metrics-system.process
        data_stream:
          dataset: system.process
          type: metrics
        metricsets:
          - process
        period: 10s
        process.include_top_n.by_cpu: 5
        process.include_top_n.by_memory: 5
        process.cmdline.cache.enabled: true
        process.cgroups.enabled: true
        processes:
          - .*
      - id: system/metrics-system.process_summary
        data_stream:
          dataset: system.process_summary
          type: metrics
        metricsets:
          - process_summary
        period: 10s
      - id: system/metrics-system.socket_summary
        data_stream:
          dataset: system.socket_summary
          type: metrics
        metricsets:
          - socket_summary
        period: 10s
      - id: system/metrics-system.uptime
        data_stream:
          dataset: system.uptime
          type: metrics
        metricsets:
          - uptime
        period: 10s
  - id: 82acd240-1787-11eb-ad66-a14f0fd0af4a
    name: aws-1
    type: aws/metrics
    use_output: default
    meta:
      package:
        name: aws
        version: 0.2.4
    data_stream:
      namespace: default
    streams:
      - id: aws/metrics-aws.billing
        data_stream:
          dataset: aws.billing
          type: metrics
        period: 12h
        metricsets:
          - billing
        credential_profile_name: AWS_Metirics1
      - id: aws/metrics-aws.cloudwatch_metrics
        data_stream:
          dataset: aws.cloudwatch_metrics
          type: metrics
        period: 300s
        metricsets:
          - cloudwatch
        metrics:
          - name:
              - CPUUtilization
              - DiskWriteOps
            statistic:
              - Average
              - Maximum
            namespace: AWS/EC2
            resource_type: 'ec2:instance'
        credential_profile_name: AWS_Metirics1
      - id: aws/metrics-aws.ec2_metrics
        data_stream:
          dataset: aws.ec2_metrics
          type: metrics
        period: 5m
        tags_filter: null
        metricsets:
          - ec2
        credential_profile_name: AWS_Metirics1
      - id: aws/metrics-aws.elb_metrics
        data_stream:
          dataset: aws.elb_metrics
          type: metrics
        period: 1m
        tags_filter: null
        metricsets:
          - elb
        credential_profile_name: AWS_Metirics1
      - id: aws/metrics-aws.lambda
        data_stream:
          dataset: aws.lambda
          type: metrics
        period: 5m
        tags_filter: null
        metricsets:
          - lambda
        credential_profile_name: AWS_Metirics1
      - id: aws/metrics-aws.rds
        data_stream:
          dataset: aws.rds
          type: metrics
        period: 1m
        tags_filter: null
        metricsets:
          - rds
        credential_profile_name: AWS_Metirics1
      - id: aws/metrics-aws.sns
        data_stream:
          dataset: aws.sns
          type: metrics
        period: 5m
        tags_filter: null
        metricsets:
          - sns
        credential_profile_name: AWS_Metirics1
      - id: aws/metrics-aws.sqs
        data_stream:
          dataset: aws.sqs
          type: metrics
        period: 5m
        metricsets:
          - sqs
        credential_profile_name: AWS_Metirics1
      - id: aws/metrics-aws.usage
        data_stream:
          dataset: aws.usage
          type: metrics
        period: 1m
        metricsets:
          - usage
        credential_profile_name: AWS_Metirics1

```

---

<div class="post-metadata">

**Author:** ![dao](https://avatars.discourse-cdn.com/v4/letter/d/a6a055/32.png) [@dao](https://discuss.elastic.co/u/dao)\
**Post date:** [November 5, 2020, 3:31pm UTC](https://discuss.elastic.co/t/ingest-manager-custom-files-not-being-uploaded/253397/9 "2020-11-05T15:31:00Z")

</div>

Nobody in elastic to understant what is going on?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 3, 2020, 5:31pm UTC](https://discuss.elastic.co/t/ingest-manager-custom-files-not-being-uploaded/253397/10 "2020-12-03T17:31:06Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.