# Solved: How to get JSON example from blog to work with new index (hint: format YML correctly!)

**URL:** <https://discuss.elastic.co/t/solved-how-to-get-json-example-from-blog-to-work-with-new-index-hint-format-yml-correctly/156103>\
**Category:** Beats\
**Created:** [November 9, 2018, 8:04pm UTC](https://discuss.elastic.co/t/solved-how-to-get-json-example-from-blog-to-work-with-new-index-hint-format-yml-correctly/156103 "2018-11-09T20:04:56Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![devops\_mike](https://avatars.discourse-cdn.com/v4/letter/d/50afbb/32.png) [@devops\_mike](https://discuss.elastic.co/u/devops_mike)\
**Post date:** [November 9, 2018, 8:04pm UTC](https://discuss.elastic.co/t/solved-how-to-get-json-example-from-blog-to-work-with-new-index-hint-format-yml-correctly/156103/1 "2018-11-09T20:04:56Z")

</div>

I am following along with this guide: [https://www.elastic.co/blog/structured-logging-filebeat](https://www.elastic.co/blog/structured-logging-filebeat)

It very closely matches my data. The difference is I need this to go into its own index. My data is JSON formatted and only includes a few simple fields.

Since my data is already json formatted, I was planning on skipping logstash. And that blog seemed like a great match.

I am having difficulties tracking the error. When I include the lines to create/use the new index in the filebeat.yml file, the service fails to start silently. If I remove those lines, the log file has the expected "Exiting: setup.template.name and setup.template.pattern have to be set if index name is modified." error.

I have also read the guide at [https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-template.html](https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-template.html) . I was hoping to just use the filebeat template. According to that blog, I will see my custom fields appear. I have tried several variations of the settings below. This is the version which explains what I am trying to do. (Use the existing filebeat template, but apply to index pattern octobeat-\*)

Am I overthinking this? Is there a simple way to get a json message into ElasticSearch under an index named octobeat-\* ? Thank you very much!

```auto
filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    #- /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*
    - e:\DevOpsLogs\octopus_deployLog.log*

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 3
  #index.codec: best_compression
  #_source.enabled: false
setup.template.name: "octobeat"
setup.template.pattern: "octobeat-*"
setup.template.path: "C:\Program Files\Filebeat_Octopus\fields.yml"

setup.kibana:
  host: "myelkserver:5601"

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["myelkserver:9200"]
  index: "octobeat-%{[beat.version]}-%{+yyyy.MM.dd}"

```

`JSON message = {"Timestamp":"2018-11-09T13:17:44.8896393-05:00","OctopusProjectName":"The Octopus Project Name","OctopusReleaseNumber":"The Octopus Release Number As String","OctopusEnvironmentName":"The Octopus Environment Name"}`

---

<div class="post-metadata">

**Author:** ![devops\_mike](https://avatars.discourse-cdn.com/v4/letter/d/50afbb/32.png) [@devops\_mike](https://discuss.elastic.co/u/devops_mike)\
**Post date:** [November 9, 2018, 8:59pm UTC](https://discuss.elastic.co/t/solved-how-to-get-json-example-from-blog-to-work-with-new-index-hint-format-yml-correctly/156103/2 "2018-11-09T20:59:20Z")

</div>

Not sure if this is helpful, but doing a similar task via powershell was trivial. I switched to filebeat because it's nice to have those log files in case something breaks and I need to replay everything to build up the indexes again. Not criticizing filebeat here, I understand there are a LOT more features included such as backpressure, just trying to give an example of what I am trying to replicate.

```
$post_body = @{
    @timestamp = $deploy_time
    deploy_version = $deploy_version
    deploy_targetserver = $deploy_targetserver
    deploy_project = $deploy_project 
}

$json = $post_body | ConvertTo-Json

$elastic_server = 'http://myelasticserver:9200/octopus/_doc'
$body = $json

Write-Host $body
Invoke-RestMethod -Uri $elastic_server -Method Post -Body $body -ContentType 'application/json'
```

---

<div class="post-metadata">

**Author:** ![devops\_mike](https://avatars.discourse-cdn.com/v4/letter/d/50afbb/32.png) [@devops\_mike](https://discuss.elastic.co/u/devops_mike)\
**Post date:** [November 12, 2018, 1:49pm UTC](https://discuss.elastic.co/t/solved-how-to-get-json-example-from-blog-to-work-with-new-index-hint-format-yml-correctly/156103/3 "2018-11-12T13:49:10Z")

</div>

I am still stuck on sending events to a new index using Filebeats.

Does anyone have an example of a filebeat.yml file which contains a new index?

From: [https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#index-option-es](https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#index-option-es)

The index name to write events to. The default is `"filebeat-%{[beat.version]}-%{+yyyy.MM.dd}"` (for example, `"filebeat-6.4.3-2018-11-06"` ). If you change this setting, you also need to configure the `setup.template.name` and `setup.template.pattern` options (see [_Load the Elasticsearch index template_](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-template.html)). If you are using the pre-built Kibana dashboards, you also need to set the `setup.dashboards.index` option (see [_Load the Kibana dashboards_](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-dashboards.html)).

You can set the index dynamically by using a format string to access any event field. For example, this configuration uses a custom field, `fields.log_type` , to set the index:

output.elasticsearch: hosts: ["[http://localhost:9200](http://localhost:9200)"] index: "%{[fields.log\_type]}-%{[beat.version]}-%{+yyyy.MM.dd}" ![](https://www.elastic.co/guide/en/beats/filebeat/current/images/icons/callouts/1.png)

I then checked out [_Load the Elasticsearch index template_](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-template.html) which has the following information:

**`setup.template.pattern`**

The template pattern to apply to the default index settings. The default pattern is `filebeat-*` . The Filebeat version is always included in the pattern, so the final pattern is `filebeat-%{[beat.version]}-*` . The wildcard character `-*` is used to match all daily indices.

## Example: setup.template.name: "filebeat" setup.template.pattern: "filebeat-\*"

I think I may not be adding setup.template.name and pattern to the correct part of the YML. I haven't found anything from the filebeat.reference.yml which helps with this.

Thanks!

---

<div class="post-metadata">

**Author:** ![devops\_mike](https://avatars.discourse-cdn.com/v4/letter/d/50afbb/32.png) [@devops\_mike](https://discuss.elastic.co/u/devops_mike)\
**Post date:** [November 12, 2018, 3:56pm UTC](https://discuss.elastic.co/t/solved-how-to-get-json-example-from-blog-to-work-with-new-index-hint-format-yml-correctly/156103/4 "2018-11-12T15:56:26Z")

</div>

Solved:

It turns out I had misconfigured my YML. Sorry for the distraction.

This is how I got it working:

```auto
setup:
  settings.index.number_of_shards: 3
  #index.codec: best_compression
  #_source.enabled: false
  template.name: "octobeat"
  template.pattern: "octobeat-*"

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 10, 2018, 5:56pm UTC](https://discuss.elastic.co/t/solved-how-to-get-json-example-from-blog-to-work-with-new-index-hint-format-yml-correctly/156103/5 "2018-12-10T17:56:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.