_Meir
(Meir)
October 27, 2020, 7:21am
1
I'm trying to get custom logs uploaded from my server to my elastic cloud .
I'm using the Ingest Manager via fleet deployment.
I've added Windows, System and Custom Logs integrations.
I'm only seeing logs from MetricBeat on the Discover page and nothing from the custom logs.
I get this message in the elastic agent log: , error: failed to filter configuration: provided datastream dataset is invalid.
In the Custom log integration I specified a new dataset name.
Any ideas how I can get this to work?
@_Meir can you copy and paste the error? It looks like part of it is getting cut off.
_Meir
(Meir)
October 27, 2020, 5:36pm
3
|2020-10-27T11:54:02.301+0200|DEBUG|application/fleet_gateway.go:142|FleetGateway calling Checkin API|
|---|---|---|---|
|2020-10-27T11:54:02.306+0200|DEBUG|kibana/client.go:170|Request method: POST, path: /api/ingest_manager/fleet/agents/a7fb57f9-b53d-4a98-8f6d-92b31c04e7c9/checkin|
|2020-10-27T11:54:02.911+0200|DEBUG|application/action_dispatcher.go:81|Dispatch 1 actions of types: *fleetapi.ActionConfigChange|
|2020-10-27T11:54:02.912+0200|DEBUG|application/handler_action_policy_change.go:23|handlerConfigChange: action 'action_id: b14afaa3-8e64-4397-a056-f730290339ef, type: CONFIG_CHANGE' received|
|2020-10-27T11:54:02.914+0200|DEBUG|application/handler_action_policy_change.go:34|handlerConfigChange: emit configuration for action action_id: b14afaa3-8e64-4397-a056-f730290339ef, type: CONFIG_CHANGE|
|2020-10-27T11:54:02.914+0200|DEBUG|application/emitter.go:39|Transforming configuration into a tree|
|2020-10-27T11:54:02.915+0200|DEBUG|application/action_dispatcher.go:93|Failed to dispatch action 'action_id: b14afaa3-8e64-4397-a056-f730290339ef, type: CONFIG_CHANGE', error: failed to filter configuration: provided datastream dataset is invalid: /go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/agent/application/filters/stream_checker.go[20]: unknown error|
|2020-10-27T11:54:02.915+0200|ERROR|application/fleet_gateway.go:159|failed to dispatch actions, error: failed to filter configuration: provided datastream dataset is invalid: /go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/agent/application/filters/stream_checker.go[20]: unknown error|
|2020-10-27T11:54:02.915+0200|DEBUG|application/fleet_gateway.go:162|FleetGateway is sleeping, next update in 1s|
`indent preformatted text by 4 spaces`
_Meir:
2020-10-27T11:54:02.915+0200|DEBUG|application/action_dispatcher.go:93|Failed to dispatch action 'action_id: b14afaa3-8e64-4397-a056-f730290339ef, type: CONFIG_CHANGE', error: failed to filter configuration: provided datastream dataset is invalid: /go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/agent/application/filters/stream_checker.go[20]: unknown error|
|2020-10-27T11:54:02.915+0200|ERROR|application/fleet_gateway.go:159|failed to dispatch actions, error: failed to filter configuration: provided datastream dataset is invalid: /go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/agent/application/filters/stream_checker.go[20]: unknown error|
At very least, that's an unhelpful error message. I'm gonna look more into it.
@_Meir can you post your fleet config? Did you change any defaults? What are you doing with the custom log integration?
@_Meir You can copy-and-paste the raw config by going to the config and selecting Actions->View Policy
dao
(olivier hodac)
October 30, 2020, 6:27pm
7
Same issue for me. I'd like to switch to this feature, but cannot do custom easily. Is there a doc on this topic?
I am in a middle of tries and fails, and always the same issue as you
_Meir
(Meir)
November 1, 2020, 9:14am
8
Here's the config.
I'm not aware of any significant changes to default configs.
id: 18ffc050-1787-11eb-ad66-a14f0fd0af4a
revision: 13
outputs:
default:
type: elasticsearch
hosts:
- 'https://e232f0e098bc40f5b28d3c6c4c908d89.us-east-1.aws.found.io:443'
agent:
monitoring:
enabled: true
use_output: default
logs: true
metrics: true
inputs:
- id: 1acc1be0-1787-11eb-ad66-a14f0fd0af4a
name: system-1
type: logfile
use_output: default
meta:
package:
name: system
version: 0.5.3
data_stream:
namespace: default
streams:
- id: logfile-system.auth
data_stream:
dataset: system.auth
type: logs
paths:
- /var/log/auth.log*
- /var/log/secure*
exclude_files:
- .gz$
multiline:
pattern: ^\s
match: after
processors:
- add_locale: null
- add_fields:
target: ''
fields:
ecs.version: 1.5.0
- id: logfile-system.syslog
data_stream:
dataset: system.syslog
type: logs
paths:
- /var/log/messages*
- /var/log/syslog*
exclude_files:
- .gz$
multiline:
pattern: ^\s
match: after
processors:
- add_locale: null
- add_fields:
target: ''
fields:
ecs.version: 1.5.0
- id: 1acc1be0-1787-11eb-ad66-a14f0fd0af4a
name: system-1
type: system/metrics
use_output: default
meta:
package:
name: system
version: 0.5.3
data_stream:
namespace: default
streams:
- id: system/metrics-system.cpu
data_stream:
dataset: system.cpu
type: metrics
metricsets:
- cpu
cpu.metrics:
- percentages
- normalized_percentages
period: 10s
- id: system/metrics-system.diskio
data_stream:
dataset: system.diskio
type: metrics
metricsets:
- diskio
diskio.include_devices: null
period: 10s
- id: system/metrics-system.load
data_stream:
dataset: system.load
type: metrics
metricsets:
- load
period: 10s
- id: system/metrics-system.memory
data_stream:
dataset: system.memory
type: metrics
metricsets:
- memory
period: 10s
- id: system/metrics-system.network
data_stream:
dataset: system.network
type: metrics
metricsets:
- network
period: 10s
network.interfaces: null
- id: system/metrics-system.process
data_stream:
dataset: system.process
type: metrics
metricsets:
- process
period: 10s
process.include_top_n.by_cpu: 5
process.include_top_n.by_memory: 5
process.cmdline.cache.enabled: true
process.cgroups.enabled: true
processes:
- .*
- id: system/metrics-system.process_summary
data_stream:
dataset: system.process_summary
type: metrics
metricsets:
- process_summary
period: 10s
- id: system/metrics-system.socket_summary
data_stream:
dataset: system.socket_summary
type: metrics
metricsets:
- socket_summary
period: 10s
- id: system/metrics-system.uptime
data_stream:
dataset: system.uptime
type: metrics
metricsets:
- uptime
period: 10s
- id: 82acd240-1787-11eb-ad66-a14f0fd0af4a
name: aws-1
type: aws/metrics
use_output: default
meta:
package:
name: aws
version: 0.2.4
data_stream:
namespace: default
streams:
- id: aws/metrics-aws.billing
data_stream:
dataset: aws.billing
type: metrics
period: 12h
metricsets:
- billing
credential_profile_name: AWS_Metirics1
- id: aws/metrics-aws.cloudwatch_metrics
data_stream:
dataset: aws.cloudwatch_metrics
type: metrics
period: 300s
metricsets:
- cloudwatch
metrics:
- name:
- CPUUtilization
- DiskWriteOps
statistic:
- Average
- Maximum
namespace: AWS/EC2
resource_type: 'ec2:instance'
credential_profile_name: AWS_Metirics1
- id: aws/metrics-aws.ec2_metrics
data_stream:
dataset: aws.ec2_metrics
type: metrics
period: 5m
tags_filter: null
metricsets:
- ec2
credential_profile_name: AWS_Metirics1
- id: aws/metrics-aws.elb_metrics
data_stream:
dataset: aws.elb_metrics
type: metrics
period: 1m
tags_filter: null
metricsets:
- elb
credential_profile_name: AWS_Metirics1
- id: aws/metrics-aws.lambda
data_stream:
dataset: aws.lambda
type: metrics
period: 5m
tags_filter: null
metricsets:
- lambda
credential_profile_name: AWS_Metirics1
- id: aws/metrics-aws.rds
data_stream:
dataset: aws.rds
type: metrics
period: 1m
tags_filter: null
metricsets:
- rds
credential_profile_name: AWS_Metirics1
- id: aws/metrics-aws.sns
data_stream:
dataset: aws.sns
type: metrics
period: 5m
tags_filter: null
metricsets:
- sns
credential_profile_name: AWS_Metirics1
- id: aws/metrics-aws.sqs
data_stream:
dataset: aws.sqs
type: metrics
period: 5m
metricsets:
- sqs
credential_profile_name: AWS_Metirics1
- id: aws/metrics-aws.usage
data_stream:
dataset: aws.usage
type: metrics
period: 1m
metricsets:
- usage
credential_profile_name: AWS_Metirics1
dao
(olivier hodac)
November 5, 2020, 3:31pm
9
Nobody in elastic to understant what is going on?
system
(system)
Closed
December 3, 2020, 5:31pm
10
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.