New index pattern to current index - topbeat-*?

[plonka2000 (Kareem)]

Hi all,

I'm trying to update an existing topbeat-[YYYY-MM-DD] index to accept an updated index pattern.

I am doing this because the topbeat sample dashboard has issues with beat.name and beat.hostname showing up as analysed fields. The dashboards do not appreciate hostnames that contain a hyphen "-" them, if it is an analysed field. The result is that the dashboard shows multiple results (The issue is also reported in this thread). The index pattern in the sample dashboards release seems to contain an updated index pattern.

By deleting, and running the load.ps1 import script I am able to import the recommended/updated index pattern, but I understand that I need to update the indices themselves.

I have also checked how to update the indices settings, and seen this elastic article on how to do so using the REST API.

I am just wondering at this point, what JSON command(s) do I need to accomplish this.

Does anyone know the correct JSON required to update an existing index to inherit the updated index pattern, in particular the beat.name and beat.hostname fields?

NOTE: I'm still somewhat new to this, so if I'm barking up the wrong tree here, I'd welcome any pointers.

[magnusbaeck (Magnus Bäck)]

You can't change the mapping of existing fields without reindexing, i.e. creating a new index to which the existing data is copied, followed by a deletion of the old index. How to reindex data is a fairly common topic here (it has been covered in the past couple of days).

[plonka2000 (Kareem)]

Thanks @magnusbaeck, much appreciated advice.

I also got some advice from @andrewkroh in the other topic here:

@andrewkroh

@plonka2000 You would need to re-index your existing data. It takes a bit of work and the way you accomplish the task varies by your technology. Basically you read the data out of ES then re-write it to a new index. See the links below.

Reindexing Your Data | Elasticsearch: The Definitive Guide [2.x] | Elastic
https://david.pilato.fr/blog/2015-05-20-reindex-elasticsearch-with-logstash/

In the future this will be easier to do: Reindex API · Issue #15201 · elastic/elasticsearch · GitHub

Thanks very much @andrewkroh, I'm going to investigate the possibility of reindexing, and if it proves too complicated/time-consuming/scary I may go with the original 'delete and create' solution (I'll chalk this one up to experience and better planning next time).

Thanks to you both for your help.

[plonka2000 (Kareem)]

In the end I deleted all the indices and started over, but the problem persists.

I double-checked to make sure the index pattern is as desired:

In the end I deleted all traces of the sample dashboards:
-Deleted all imported Searches
-Deleted all imported Dashboards
-Deleted all imported index patterns
-Deleted all imported Visualisations

Then:
-Imported latest beats-dashboards-1.1.1 sample dashboards.

The same results after all changes.

Screenshot of hostname listed in sample Kibana dashboard:
(This should be a single hostname win-lhc7bf0fdt1 but registers as two)

I'm at somewhat of a loss as to why this is happening now. Shouldn't this work?

I've even downloaded and used the latest topbeat-1.1.1 release on the client.

[magnusbaeck (Magnus Bäck)]

Are you sending directly from Topbeat to ES or do you have a Logstash instance inbetween? In the former case, what does your configuration look like?

[plonka2000 (Kareem)]

I'm sending directly to ES, not via Logstash in this instance.

Below is my topbeat config (I've removed all "#" lines):

input:
  period: 5
  procs: [".*"]
  stats:
    system: true
    proc: true
    filesystem: true
    cpu_per_core: false
output:
  elasticsearch:
    hosts: ["http://myesclusterurl:9200"]
    protocol: "http"
    username: "beatuser"
    password: "beatpassword"
    index: "topbeat"
shipper:
logging:
  to_files: true
  files:
    path: c:\beatlogs
    name: topbeat.log
    keepfiles: 20
  level: debug

I think its a pretty standard config.

Thanks for having a look.

[magnusbaeck (Magnus Bäck)]

When you recreated the index(es) after deleting them, did you make sure to mark the hostname field as not analyzed? The evidence suggests that it's still analyzed.

[plonka2000 (Kareem)]

I don't believe I did that, no.
I thought the index patterns would be applied to the newly created indices?

The screenshot I provided shows in Kibana that the beat.hostname and beat.name fields are not analyzed.
Again, I could be misunderstanding what I'm looking at here.

Is there a way to manually mark them?

Update: I assume this is possible via the ES REST API, and I'm trying to determine what instructions I need, but REST API functions are where I'm unsure with ES.

[plonka2000 (Kareem)]

It does seem that its not working.

I have just deleted all indices, added new data and refreshed the fields, and now my fields show up analyzed:

I'm unsure what to do to make the configuration stick.

[plonka2000 (Kareem)]

I think I got it to work!

Kibana now shows:

What I did:
I read up on the Index templates article here.

Ran a query against the cluster to check if there was a valid template for topbeat:
curl -XGET localhost:9200/_template/
Response:

[code]logstash

@{order=0; template=logstash-*; settings=; mappings=; aliases=}[/code]

So, I think there is no topbeat index template...

Next I used the content of topbeat.template.json file (included with topbeat-1.1.1), and used kopf to create a new index template for topbeat:

I saw that refreshing still did no good.

So once again, I deleted everything (Didn't delete index template in kopf) and re-imported the entire stack:

@plonka2000

In the end I deleted all traces of the sample dashboards:
-Deleted all imported Searches
-Deleted all imported Dashboards
-Deleted all imported index patterns
-Deleted all imported Visualisations

Then:
-Imported latest beats-dashboards-1.1.1 sample dashboards.

Now it looks like it works!

I'm quite sure there is some way to do this all via the ES REST API, but I have not figured out that part yet.

[DenysK]

Hello Eveybody,

All previous solutions did not help me with Elasticsearch analyzer.
I did the next steps:

1. Stop topbeat on source host
2. Remove "topbeat-*" index from Kibana
3. Remove topbeat template from Elasticsearch by curl -XDELETE 'http://IP:9200/topbeat-*'
4. Add new custom template by using "kopf" plugin:

Custom part in template default section:

"hostname": {
"type": "string",
"index": "not_analyzed"
},
"name": {
"type": "string",
"index": "not_analyzed"
},

topbeat.template.json:

{
"mappings": {
"default": {
"_all": {
"enabled": true,
"norms": {
"enabled": false
}
},
"dynamic_templates": [
{
"template1": {
"mapping": {
"doc_values": true,
"ignore_above": 1024,
"index": "not_analyzed",
"type": "{dynamic_type}"
},
"match": ""
}
}
],
"properties": {
"@timestamp": {
"type": "date"
},
"hostname": {
"type": "string",
"index": "not_analyzed"
},
"name": {
"type": "string",
"index": "not_analyzed"
},
"cpu": {
"properties": {
"system_p": {
"doc_values": "true",
"type": "float"
},
"user_p": {
"doc_values": "true",
"type": "float"
}
}
},
"fs": {
"properties": {
"used_p": {
"doc_values": "true",
"type": "float"
}
}
},
"load": {
"properties": {
"load1": {
"doc_values": "true",
"type": "float"
},
"load15": {
"doc_values": "true",
"type": "float"
},
"load5": {
"doc_values": "true",
"type": "float"
}
}
},
"mem": {
"properties": {
"actual_used_p": {
"doc_values": "true",
"type": "float"
},
"used_p": {
"doc_values": "true",
"type": "float"
}
}
},
"proc": {
"properties": {
"cpu": {
"properties": {
"user_p": {
"doc_values": "true",
"type": "float"
}
}
},
"mem": {
"properties": {
"rss_p": {
"doc_values": "true",
"type": "float"
}
}
}
}
},
"swap": {
"properties": {
"used_p": {
"doc_values": "true",
"type": "float"
}
}
}
}
}
},
"settings": {
"index.refresh_interval": "5s"
},
"template": "topbeat-"
}

5 Start topbeat on source host
6 Problem solved. Now beat.hostname and beat.name works correctly.