Ingest Manager - Custom Files not being uploaded

_Meir · October 27, 2020, 7:21am

I'm trying to get custom logs uploaded from my server to my elastic cloud .
I'm using the Ingest Manager via fleet deployment.
I've added Windows, System and Custom Logs integrations.
I'm only seeing logs from MetricBeat on the Discover page and nothing from the custom logs.
I get this message in the elastic agent log: , error: failed to filter configuration: provided datastream dataset is invalid.

In the Custom log integration I specified a new dataset name.

Any ideas how I can get this to work?

Alex_Kristiansen · October 27, 2020, 3:53pm

@_Meir can you copy and paste the error? It looks like part of it is getting cut off.

_Meir · October 27, 2020, 5:36pm

|2020-10-27T11:54:02.301+0200|DEBUG|application/fleet_gateway.go:142|FleetGateway calling Checkin API|
|---|---|---|---|
|2020-10-27T11:54:02.306+0200|DEBUG|kibana/client.go:170|Request method: POST, path: /api/ingest_manager/fleet/agents/a7fb57f9-b53d-4a98-8f6d-92b31c04e7c9/checkin|
|2020-10-27T11:54:02.911+0200|DEBUG|application/action_dispatcher.go:81|Dispatch 1 actions of types: *fleetapi.ActionConfigChange|
|2020-10-27T11:54:02.912+0200|DEBUG|application/handler_action_policy_change.go:23|handlerConfigChange: action 'action_id: b14afaa3-8e64-4397-a056-f730290339ef, type: CONFIG_CHANGE' received|
|2020-10-27T11:54:02.914+0200|DEBUG|application/handler_action_policy_change.go:34|handlerConfigChange: emit configuration for action action_id: b14afaa3-8e64-4397-a056-f730290339ef, type: CONFIG_CHANGE|
|2020-10-27T11:54:02.914+0200|DEBUG|application/emitter.go:39|Transforming configuration into a tree|
|2020-10-27T11:54:02.915+0200|DEBUG|application/action_dispatcher.go:93|Failed to dispatch action 'action_id: b14afaa3-8e64-4397-a056-f730290339ef, type: CONFIG_CHANGE', error: failed to filter configuration: provided datastream dataset is invalid: /go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/agent/application/filters/stream_checker.go[20]: unknown error|
|2020-10-27T11:54:02.915+0200|ERROR|application/fleet_gateway.go:159|failed to dispatch actions, error: failed to filter configuration: provided datastream dataset is invalid: /go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/agent/application/filters/stream_checker.go[20]: unknown error|
|2020-10-27T11:54:02.915+0200|DEBUG|application/fleet_gateway.go:162|FleetGateway is sleeping, next update in 1s|
    `indent preformatted text by 4 spaces`

Alex_Kristiansen · October 30, 2020, 3:50pm

_Meir:

2020-10-27T11:54:02.915+0200|DEBUG|application/action_dispatcher.go:93|Failed to dispatch action 'action_id: b14afaa3-8e64-4397-a056-f730290339ef, type: CONFIG_CHANGE', error: failed to filter configuration: provided datastream dataset is invalid: /go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/agent/application/filters/stream_checker.go[20]: unknown error|
|2020-10-27T11:54:02.915+0200|ERROR|application/fleet_gateway.go:159|failed to dispatch actions, error: failed to filter configuration: provided datastream dataset is invalid: /go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/agent/application/filters/stream_checker.go[20]: unknown error|

At very least, that's an unhelpful error message. I'm gonna look more into it.

Alex_Kristiansen · October 30, 2020, 3:51pm

@_Meir can you post your fleet config? Did you change any defaults? What are you doing with the custom log integration?

Alex_Kristiansen · October 30, 2020, 3:53pm

@_Meir You can copy-and-paste the raw config by going to the config and selecting Actions->View Policy

dao · October 30, 2020, 6:27pm

Same issue for me. I'd like to switch to this feature, but cannot do custom easily. Is there a doc on this topic?

I am in a middle of tries and fails, and always the same issue as you

_Meir · November 1, 2020, 9:14am

Here's the config.
I'm not aware of any significant changes to default configs.

id: 18ffc050-1787-11eb-ad66-a14f0fd0af4a
revision: 13
outputs:
  default:
    type: elasticsearch
    hosts:
      - 'https://e232f0e098bc40f5b28d3c6c4c908d89.us-east-1.aws.found.io:443'
agent:
  monitoring:
    enabled: true
    use_output: default
    logs: true
    metrics: true
inputs:
  - id: 1acc1be0-1787-11eb-ad66-a14f0fd0af4a
    name: system-1
    type: logfile
    use_output: default
    meta:
      package:
        name: system
        version: 0.5.3
    data_stream:
      namespace: default
    streams:
      - id: logfile-system.auth
        data_stream:
          dataset: system.auth
          type: logs
        paths:
          - /var/log/auth.log*
          - /var/log/secure*
        exclude_files:
          - .gz$
        multiline:
          pattern: ^\s
          match: after
        processors:
          - add_locale: null
          - add_fields:
              target: ''
              fields:
                ecs.version: 1.5.0
      - id: logfile-system.syslog
        data_stream:
          dataset: system.syslog
          type: logs
        paths:
          - /var/log/messages*
          - /var/log/syslog*
        exclude_files:
          - .gz$
        multiline:
          pattern: ^\s
          match: after
        processors:
          - add_locale: null
          - add_fields:
              target: ''
              fields:
                ecs.version: 1.5.0
  - id: 1acc1be0-1787-11eb-ad66-a14f0fd0af4a
    name: system-1
    type: system/metrics
    use_output: default
    meta:
      package:
        name: system
        version: 0.5.3
    data_stream:
      namespace: default
    streams:
      - id: system/metrics-system.cpu
        data_stream:
          dataset: system.cpu
          type: metrics
        metricsets:
          - cpu
        cpu.metrics:
          - percentages
          - normalized_percentages
        period: 10s
      - id: system/metrics-system.diskio
        data_stream:
          dataset: system.diskio
          type: metrics
        metricsets:
          - diskio
        diskio.include_devices: null
        period: 10s
      - id: system/metrics-system.load
        data_stream:
          dataset: system.load
          type: metrics
        metricsets:
          - load
        period: 10s
      - id: system/metrics-system.memory
        data_stream:
          dataset: system.memory
          type: metrics
        metricsets:
          - memory
        period: 10s
      - id: system/metrics-system.network
        data_stream:
          dataset: system.network
          type: metrics
        metricsets:
          - network
        period: 10s
        network.interfaces: null
      - id: system/metrics-system.process
        data_stream:
          dataset: system.process
          type: metrics
        metricsets:
          - process
        period: 10s
        process.include_top_n.by_cpu: 5
        process.include_top_n.by_memory: 5
        process.cmdline.cache.enabled: true
        process.cgroups.enabled: true
        processes:
          - .*
      - id: system/metrics-system.process_summary
        data_stream:
          dataset: system.process_summary
          type: metrics
        metricsets:
          - process_summary
        period: 10s
      - id: system/metrics-system.socket_summary
        data_stream:
          dataset: system.socket_summary
          type: metrics
        metricsets:
          - socket_summary
        period: 10s
      - id: system/metrics-system.uptime
        data_stream:
          dataset: system.uptime
          type: metrics
        metricsets:
          - uptime
        period: 10s
  - id: 82acd240-1787-11eb-ad66-a14f0fd0af4a
    name: aws-1
    type: aws/metrics
    use_output: default
    meta:
      package:
        name: aws
        version: 0.2.4
    data_stream:
      namespace: default
    streams:
      - id: aws/metrics-aws.billing
        data_stream:
          dataset: aws.billing
          type: metrics
        period: 12h
        metricsets:
          - billing
        credential_profile_name: AWS_Metirics1
      - id: aws/metrics-aws.cloudwatch_metrics
        data_stream:
          dataset: aws.cloudwatch_metrics
          type: metrics
        period: 300s
        metricsets:
          - cloudwatch
        metrics:
          - name:
              - CPUUtilization
              - DiskWriteOps
            statistic:
              - Average
              - Maximum
            namespace: AWS/EC2
            resource_type: 'ec2:instance'
        credential_profile_name: AWS_Metirics1
      - id: aws/metrics-aws.ec2_metrics
        data_stream:
          dataset: aws.ec2_metrics
          type: metrics
        period: 5m
        tags_filter: null
        metricsets:
          - ec2
        credential_profile_name: AWS_Metirics1
      - id: aws/metrics-aws.elb_metrics
        data_stream:
          dataset: aws.elb_metrics
          type: metrics
        period: 1m
        tags_filter: null
        metricsets:
          - elb
        credential_profile_name: AWS_Metirics1
      - id: aws/metrics-aws.lambda
        data_stream:
          dataset: aws.lambda
          type: metrics
        period: 5m
        tags_filter: null
        metricsets:
          - lambda
        credential_profile_name: AWS_Metirics1
      - id: aws/metrics-aws.rds
        data_stream:
          dataset: aws.rds
          type: metrics
        period: 1m
        tags_filter: null
        metricsets:
          - rds
        credential_profile_name: AWS_Metirics1
      - id: aws/metrics-aws.sns
        data_stream:
          dataset: aws.sns
          type: metrics
        period: 5m
        tags_filter: null
        metricsets:
          - sns
        credential_profile_name: AWS_Metirics1
      - id: aws/metrics-aws.sqs
        data_stream:
          dataset: aws.sqs
          type: metrics
        period: 5m
        metricsets:
          - sqs
        credential_profile_name: AWS_Metirics1
      - id: aws/metrics-aws.usage
        data_stream:
          dataset: aws.usage
          type: metrics
        period: 1m
        metricsets:
          - usage
        credential_profile_name: AWS_Metirics1

dao · November 5, 2020, 3:31pm

Nobody in elastic to understant what is going on?

system · December 3, 2020, 5:31pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Problem configuring "Custom configurations" in the "Custom Logs" integration of Elastic Agent Elastic Agent filebeat	1	243	July 13, 2023
Fleet Windows agent: failed to filter configuration: provided datastream dataset is invalid Elastic Observability fleet	1	323	November 4, 2022
Custom Logs Ingest Pipeline Elasticsearch fleet	9	3810	February 21, 2022
Filebeats not sending log file to ingest pipeline for my custom module Beats filebeat	1	475	February 10, 2021
Fleet managed Elastic Agent - Wrong ES output configuration for Filebeat/Metricbeat Beats fleet	2	1320	March 29, 2022

Ingest Manager - Custom Files not being uploaded

Related topics