Synthetic monitor data to specific indices

Elastic Stack Elasticsearch
1

Hi,
Is it possible to redirect data of synthetic monitoring agent to a specific index?

elk 8

thanks

2

Hi,

Yes you can do this.
If you are using heartbeat then you can set index value in output filter.

But if you change index name then you need to configure the setup.template.name and setup.template.pattern options as well.

The index name to write events to when you’re using daily indices. The default is "heartbeat-%{[agent.version]}-%{+yyyy.MM.dd}" , for example, "heartbeat-8.0.1-2022-03-08" . If you change this setting, you also need to configure the setup.template.name and setup.template.pattern options (see Elasticsearch index template ).

Using Synthetic Integration from Fleet, it seems like not possible from Integration UI. You can provide your custom namespace name which default value is default.

3

I am able to redirect the data to a specific index. However, unable to view it in uptime monitor with custom indices.
As soon as I change the default indices from settings, I am getting the following error

Heartbeat mappings missing

Incorrect mappings detected! Perhaps you forgot to run the heartbeat setup command?

You can learn how to troubleshoot this issue in the docs(opens in a new tab or window).

Any idea how to solve it

4

Please check this documentation.

  1. To access this page, go to Observability > Uptime .
  2. At the top of the page, click Settings .
  3. provide your custom index name in uptime indices setting.
5

I have already changed and then I am getting that error.

6

As error said it is mapping issue, but you need to check logs then only it will be clear. can you please raise seperate quesion for same.

Did you try solution mentioned here.

7

I have tried that . Also tried to do it manually using fields.yml from heartbeat.
output.Elasticsearch:
hosts: 'http://elasticsearch:9200'
username: 'elastic'
password: 'elastic'
index: "test-heartbeat-%{[agent.version]}-%{+yyyy.MM.dd}"
setup.template.name: "test-heartbeat"
setup.template.pattern: "test-heartbeat"
setup.template.overwrite: true
setup.template.fields: /usr/share/heartbeat/fields.yml
setup.kibana:
host: 'http://elasticsearch:5601'

$ heartbeat test config
Config OK
$ heartbeat test output
Elasticsearch: http://elasticsearch:9200...
parse url... OK
connection...
parse host... OK
dns lookup... OK
addresses: 10.1.150.98
dial up... OK
TLS... WARN secure connection disabled
talk to server... OK
version: 8.0.0

$heartbeat setup -e
{"log.level":"info","@timestamp":"2022-03-10T11:25:12.866Z","log.origin":{"file.name":"instance/beat.go","file.line":679},"message":"Home path: [/usr/share/heartbeat] Config path: [/usr/share/heartbeat] Data path: [/usr/share/heartbeat/data] Logs path: [/usr/share/heartbeat/logs]","service.name":"heartbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-10T11:25:12.867Z","log.origin":{"file.name":"instance/beat.go","file.line":687},"message":"Beat ID: da9e5d30-9e42-4b38-884b-3f1077a78446","service.name":"heartbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-03-10T11:25:15.867Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":80},"message":"read token request for getting IMDSv2 token returns empty: Put "http://169.254.169.254/latest/api/token": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"heartbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-10T11:25:15.868Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1050},"message":"Beat info","service.name":"heartbeat","system_info":{"beat":{"path":{"config":"/usr/share/heartbeat","data":"/usr/share/heartbeat/data","home":"/usr/share/heartbeat","logs":"/usr/share/heartbeat/logs"},"type":"heartbeat","uuid":"da9e5d30-9e42-4b38-884b-3f1077a78446"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-03-10T11:25:15.868Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1059},"message":"Build info","service.name":"heartbeat","system_info":{"build":{"commit":"2ab3a7334016f570e0bfc7e9a577a35a22e02df5","libbeat":"8.0.0","time":"2022-02-03T18:08:05.000Z","version":"8.0.0"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-03-10T11:25:15.868Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1062},"message":"Go runtime info","service.name":"heartbeat","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":4,"version":"go1.17.6"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-03-10T11:25:15.869Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1066},"message":"Host info","service.name":"heartbeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-02-22T05:56:21Z","containerized":true,"name":"test-heartbeat","ip":["127.0.0.1/8","172.31.0.2/16"],"kernel_version":"3.10.0-862.el7.x86_64","mac":["02:42:ac:1f:00:02"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.3 LTS (Focal Fossa)","major":20,"minor":4,"patch":3,"codename":"focal"},"timezone":"UTC","timezone_offset_sec":0},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-03-10T11:25:15.870Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1095},"message":"Process info","service.name":"heartbeat","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":["net_raw"],"effective":["net_raw"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null},"cwd":"/usr/share/heartbeat","exe":"/usr/share/heartbeat/heartbeat","name":"heartbeat","pid":8184,"ppid":8176,"seccomp":{"mode":"filter"},"start_time":"2022-03-10T11:25:01.830Z"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-03-10T11:25:15.870Z","log.origin":{"file.name":"instance/beat.go","file.line":332},"message":"Setup Beat: heartbeat; Version: 8.0.0","service.name":"heartbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-10T11:25:15.871Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":105},"message":"Elasticsearch url: http://elasticsearch:9200","service.name":"heartbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-10T11:25:15.871Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: test-heartbeat","service.name":"heartbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-10T11:25:15.871Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":105},"message":"Elasticsearch url: http://elasticsearch:9200","service.name":"heartbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-10T11:25:15.884Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":284},"message":"Attempting to connect to Elasticsearch version 8.0.0","service.name":"heartbeat","ecs.version":"1.6.0"}
Overwriting ILM policy is disabled. Set setup.ilm.overwrite: true for enabling.

{"log.level":"info","@timestamp":"2022-03-10T11:25:15.884Z","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":231},"message":"Auto ILM enable success.","service.name":"heartbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-10T11:25:15.886Z","log.logger":"index-management.ilm","log.origin":{"file.name":"ilm/std.go","file.line":118},"message":"ILM policy heartbeat exists already.","service.name":"heartbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-10T11:25:15.886Z","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":366},"message":"Set settings.index.lifecycle.name in template to {heartbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.","service.name":"heartbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-10T11:25:15.890Z","log.logger":"template","log.origin":{"file.name":"template/load.go","file.line":198},"message":"Existing template will be overwritten, as overwrite is enabled.","service.name":"heartbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-10T11:25:15.998Z","log.logger":"template_loader","log.origin":{"file.name":"template/load.go","file.line":134},"message":"Try loading template test-heartbeat to Elasticsearch","service.name":"heartbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-10T11:25:16.017Z","log.logger":"template_loader","log.origin":{"file.name":"template/load.go","file.line":126},"message":"Template with name "test-heartbeat" loaded.","service.name":"heartbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-10T11:25:16.017Z","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":267},"message":"Loaded index template.","service.name":"heartbeat","ecs.version":"1.6.0"}
Index setup finished.
heartbeat@test-heartbeat:~$
Once this is done, I have created "data view/index pattern" and checked data.
Then modified uptime -> settings-> indices Uptime indices

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.