Solved: How to get JSON example from blog to work with new index (hint: format YML correctly!)

[devops_mike]

I am following along with this guide: https://www.elastic.co/blog/structured-logging-filebeat

It very closely matches my data. The difference is I need this to go into its own index. My data is JSON formatted and only includes a few simple fields.

Since my data is already json formatted, I was planning on skipping logstash. And that blog seemed like a great match.

I am having difficulties tracking the error. When I include the lines to create/use the new index in the filebeat.yml file, the service fails to start silently. If I remove those lines, the log file has the expected "Exiting: setup.template.name and setup.template.pattern have to be set if index name is modified." error.

I have also read the guide at https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-template.html . I was hoping to just use the filebeat template. According to that blog, I will see my custom fields appear. I have tried several variations of the settings below. This is the version which explains what I am trying to do. (Use the existing filebeat template, but apply to index pattern octobeat-*)

Am I overthinking this? Is there a simple way to get a json message into ElasticSearch under an index named octobeat-* ? Thank you very much!

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    #- /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*
    - e:\DevOpsLogs\octopus_deployLog.log*

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 3
  #index.codec: best_compression
  #_source.enabled: false
setup.template.name: "octobeat"
setup.template.pattern: "octobeat-*"
setup.template.path: "C:\Program Files\Filebeat_Octopus\fields.yml"

setup.kibana:
  host: "myelkserver:5601"

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["myelkserver:9200"]
  index: "octobeat-%{[beat.version]}-%{+yyyy.MM.dd}"

JSON message = {"Timestamp":"2018-11-09T13:17:44.8896393-05:00","OctopusProjectName":"The Octopus Project Name","OctopusReleaseNumber":"The Octopus Release Number As String","OctopusEnvironmentName":"The Octopus Environment Name"}

[devops_mike]

Not sure if this is helpful, but doing a similar task via powershell was trivial. I switched to filebeat because it's nice to have those log files in case something breaks and I need to replay everything to build up the indexes again. Not criticizing filebeat here, I understand there are a LOT more features included such as backpressure, just trying to give an example of what I am trying to replicate.

$post_body = @{
    @timestamp = $deploy_time
    deploy_version = $deploy_version
    deploy_targetserver = $deploy_targetserver
    deploy_project = $deploy_project 
}

$json = $post_body | ConvertTo-Json

$elastic_server = 'http://myelasticserver:9200/octopus/_doc'
$body = $json

Write-Host $body
Invoke-RestMethod -Uri $elastic_server -Method Post -Body $body -ContentType 'application/json'

[devops_mike]

I am still stuck on sending events to a new index using Filebeats.

Does anyone have an example of a filebeat.yml file which contains a new index?

From: https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#index-option-es

The index name to write events to. The default is "filebeat-%{[beat.version]}-%{+yyyy.MM.dd}" (for example, "filebeat-6.4.3-2018-11-06" ). If you change this setting, you also need to configure the setup.template.name and setup.template.pattern options (see Load the Elasticsearch index template ). If you are using the pre-built Kibana dashboards, you also need to set the setup.dashboards.index option (see Load the Kibana dashboards ).

You can set the index dynamically by using a format string to access any event field. For example, this configuration uses a custom field, fields.log_type , to set the index:

output.elasticsearch: hosts: ["http://localhost:9200"] index: "%{[fields.log_type]}-%{[beat.version]}-%{+yyyy.MM.dd}"

I then checked out Load the Elasticsearch index template which has the following information:

setup.template.pattern

The template pattern to apply to the default index settings. The default pattern is filebeat-* . The Filebeat version is always included in the pattern, so the final pattern is filebeat-%{[beat.version]}-* . The wildcard character -* is used to match all daily indices.

Example:
setup.template.name: "filebeat"
setup.template.pattern: "filebeat-*"

I think I may not be adding setup.template.name and pattern to the correct part of the YML. I haven't found anything from the filebeat.reference.yml which helps with this.

Thanks!

[devops_mike]

Solved:

It turns out I had misconfigured my YML. Sorry for the distraction.

This is how I got it working:

setup:
  settings.index.number_of_shards: 3
  #index.codec: best_compression
  #_source.enabled: false
  template.name: "octobeat"
  template.pattern: "octobeat-*"