New index pattern to current index - topbeat-*?

plonka2000 · February 12, 2016, 3:06pm

Hi all,

I'm trying to update an existing topbeat-[YYYY-MM-DD] index to accept an updated index pattern.

I am doing this because the topbeat sample dashboard has issues with beat.name and beat.hostname showing up as analysed fields. The dashboards do not appreciate hostnames that contain a hyphen "-" them, if it is an analysed field. The result is that the dashboard shows multiple results (The issue is also reported in this thread). The index pattern in the sample dashboards release seems to contain an updated index pattern.

By deleting, and running the load.ps1 import script I am able to import the recommended/updated index pattern, but I understand that I need to update the indices themselves.

I have also checked how to update the indices settings, and seen this elastic article on how to do so using the REST API.

I am just wondering at this point, what JSON command(s) do I need to accomplish this.

Does anyone know the correct JSON required to update an existing index to inherit the updated index pattern, in particular the beat.name and beat.hostname fields?

NOTE: I'm still somewhat new to this, so if I'm barking up the wrong tree here, I'd welcome any pointers.

magnusbaeck · February 14, 2016, 4:38pm

You can't change the mapping of existing fields without reindexing, i.e. creating a new index to which the existing data is copied, followed by a deletion of the old index. How to reindex data is a fairly common topic here (it has been covered in the past couple of days).

plonka2000 · February 15, 2016, 11:01am

Thanks @magnusbaeck, much appreciated advice.

I also got some advice from @andrewkroh in the other topic here:

Thanks very much @andrewkroh, I'm going to investigate the possibility of reindexing, and if it proves too complicated/time-consuming/scary I may go with the original 'delete and create' solution (I'll chalk this one up to experience and better planning next time).

Thanks to you both for your help.

plonka2000 · February 15, 2016, 4:37pm

In the end I deleted all the indices and started over, but the problem persists.

I double-checked to make sure the index pattern is as desired:

In the end I deleted all traces of the sample dashboards:
-Deleted all imported Searches
-Deleted all imported Dashboards
-Deleted all imported index patterns
-Deleted all imported Visualisations

Then:
-Imported latest beats-dashboards-1.1.1 sample dashboards.

The same results after all changes.

Screenshot of hostname listed in sample Kibana dashboard:
(This should be a single hostname win-lhc7bf0fdt1 but registers as two)

I'm at somewhat of a loss as to why this is happening now. Shouldn't this work?

I've even downloaded and used the latest topbeat-1.1.1 release on the client.

magnusbaeck · February 15, 2016, 6:01pm

Are you sending directly from Topbeat to ES or do you have a Logstash instance inbetween? In the former case, what does your configuration look like?

plonka2000 · February 16, 2016, 9:44am

I'm sending directly to ES, not via Logstash in this instance.

Below is my topbeat config (I've removed all "#" lines):

input:
  period: 5
  procs: [".*"]
  stats:
    system: true
    proc: true
    filesystem: true
    cpu_per_core: false
output:
  elasticsearch:
    hosts: ["http://myesclusterurl:9200"]
    protocol: "http"
    username: "beatuser"
    password: "beatpassword"
    index: "topbeat"
shipper:
logging:
  to_files: true
  files:
    path: c:\beatlogs
    name: topbeat.log
    keepfiles: 20
  level: debug

I think its a pretty standard config.

Thanks for having a look.

magnusbaeck · February 16, 2016, 9:50am

When you recreated the index(es) after deleting them, did you make sure to mark the hostname field as not analyzed? The evidence suggests that it's still analyzed.

plonka2000 · February 16, 2016, 10:07am

I don't believe I did that, no.
I thought the index patterns would be applied to the newly created indices?

The screenshot I provided shows in Kibana that the beat.hostname and beat.name fields are not analyzed.
Again, I could be misunderstanding what I'm looking at here.

Is there a way to manually mark them?

Update: I assume this is possible via the ES REST API, and I'm trying to determine what instructions I need, but REST API functions are where I'm unsure with ES.

plonka2000 · February 16, 2016, 10:36am

It does seem that its not working.

I have just deleted all indices, added new data and refreshed the fields, and now my fields show up analyzed:

I'm unsure what to do to make the configuration stick.

plonka2000 · February 16, 2016, 11:16am

I think I got it to work!

Kibana now shows:

What I did:
I read up on the Index templates article here.

Ran a query against the cluster to check if there was a valid template for topbeat:
curl -XGET localhost:9200/_template/
Response:

[code]logstash

@{order=0; template=logstash-*; settings=; mappings=; aliases=}[/code]

So, I think there is no topbeat index template...

Next I used the content of topbeat.template.json file (included with topbeat-1.1.1), and used kopf to create a new index template for topbeat:

I saw that refreshing still did no good.

So once again, I deleted everything (Didn't delete index template in kopf) and re-imported the entire stack:

Now it looks like it works!

I'm quite sure there is some way to do this all via the ES REST API, but I have not figured out that part yet.

DenysK · April 11, 2016, 4:29pm

Hello Eveybody,

All previous solutions did not help me with Elasticsearch analyzer.
I did the next steps:

Stop topbeat on source host
Remove "topbeat-*" index from Kibana
Remove topbeat template from Elasticsearch by curl -XDELETE 'http://IP:9200/topbeat-*'
Add new custom template by using "kopf" plugin:

Custom part in template default section:

"hostname": {
"type": "string",
"index": "not_analyzed"
},
"name": {
"type": "string",
"index": "not_analyzed"
},

topbeat.template.json:

{
"mappings": {
"default": {
"_all": {
"enabled": true,
"norms": {
"enabled": false
}
},
"dynamic_templates": [
{
"template1": {
"mapping": {
"doc_values": true,
"ignore_above": 1024,
"index": "not_analyzed",
"type": "{dynamic_type}"
},
"match": ""
}
}
],
"properties": {
"@timestamp": {
"type": "date"
},
"hostname": {
"type": "string",
"index": "not_analyzed"
},
"name": {
"type": "string",
"index": "not_analyzed"
},
"cpu": {
"properties": {
"system_p": {
"doc_values": "true",
"type": "float"
},
"user_p": {
"doc_values": "true",
"type": "float"
}
}
},
"fs": {
"properties": {
"used_p": {
"doc_values": "true",
"type": "float"
}
}
},
"load": {
"properties": {
"load1": {
"doc_values": "true",
"type": "float"
},
"load15": {
"doc_values": "true",
"type": "float"
},
"load5": {
"doc_values": "true",
"type": "float"
}
}
},
"mem": {
"properties": {
"actual_used_p": {
"doc_values": "true",
"type": "float"
},
"used_p": {
"doc_values": "true",
"type": "float"
}
}
},
"proc": {
"properties": {
"cpu": {
"properties": {
"user_p": {
"doc_values": "true",
"type": "float"
}
}
},
"mem": {
"properties": {
"rss_p": {
"doc_values": "true",
"type": "float"
}
}
}
}
},
"swap": {
"properties": {
"used_p": {
"doc_values": "true",
"type": "float"
}
}
}
}
}
},
"settings": {
"index.refresh_interval": "5s"
},
"template": "topbeat-"
}

5 Start topbeat on source host
6 Problem solved. Now beat.hostname and beat.name works correctly.

Topic		Replies	Views
Topbeat beat.hostname analyzed Beats	11	4747	July 5, 2017
Topbeat index name appends YYYY.MM.DD Beats	6	1260	July 5, 2017
How can I overwrite default index pattern name while importing dashboards for beats Beats	10	2707	June 6, 2017
Mapping of index containing date in index(Ex. filebeat-2016.05.24) Elasticsearch	7	957	July 5, 2017
Index_Pattern set to metricbeat-* on sample dashboards Beats metricbeat	4	1188	March 9, 2018

New index pattern to current index - topbeat-*?

[code]logstash

Related topics