I collect logs in Elasticsearch and keep track of certain values in the logs with fields.
Since the values are numeric, I would like to calculate the sum over a certain time interval and see the transition.
However, when I select the field in the Vertical axis of the Lens, I cannot select Sum.
I can only select Differences, Moving average and Unique count.
Why is this?
Is there a problem with the field settings in Elasticsearch?
Any advice would be appreciated. Thanks in advance.
Can you send me an example of the field's mapping? It should allow a sum...what happens when you click "sum" and you select this field?
Thanks for answering my question.
The field is assigned an integer value. Specifically, it is the amount of access log transfer.
If I select Sum in Select a function, Select a field can only select log.offset.
Please show the mapping and tell us which field you are trying to sum, Lens is acting like is type keyword
The common reason for the behavior that you're seeing is an incorrect data mapping.
Please show the mapping and tell us which field you are trying to sum,
How can we get this across to you?
I will send you the screen I am looking at.
Hi @its-ogawa
So first yes this is your exact problem you bytes field is not a numeric type.
So first question how did you import the data? and it looks like perhaps it is apache or ngnix logs? What kind of data is it?
Short story you need to define the mapping / data types to properly use your data...
It looks like you important data without defining a mapping (think schema) so a default mapping was created automatically and the type is not correct.
Did you create a template when you imported the data (that will specify the mapping) for all matching indices
You can get the mappings by going to
Kibana -> Dev Tools
GET my-index-name
Answer a few questions and perhaps we can help.
I am getting the following data. You are right, it is an nginx log.
XXX.XXX.XXX.XXX.XXX - - [27/Jul/2022:23:23:06 +0900] "GET /favicon.ico HTTP/1.0" 200 417 "https://XXXX/sp/item.php?pn=20994&fbclid=PAAaYp1mNWdhHcfJxyDJn-lFuhNqPPBK0BEGXusOa3sGNH3p5vKH76nI-S9J0" "Mozilla/5.0 (Linux; Android 11; SO-41A Build/59.1.B.1.226; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/103.0.5060.129 Mobile Safari/537.36 Instagram 244.1.0.19.110 Android (30/11; 420dpi; 1080x2394; Sony/docomo; SO-41A; SO-41A; qcom; ja_JP; 384108453)"
XXX.XXX.XXX.XXX.XXX - - [27/Jul/2022:23:23:02 +0900] "GET /api/mailmag_open_notify.php?mgid=6210504 HTTP/1.0" 200 - "-" "Outlook-Express/7.0 (MSIE 7.0; Windows NT 6.2; WOW64; Trident/7.0; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR 2.0.50727; .NET CLR 3.0.30729; YTB730; McAfee; MALNJS; TmstmpExt)"
... snip ...
I imported this log into Elasticsearch via Filebeat and Logstash.
I have the following filter settings in the Logstash conf file.
grok {
match => {
#"message" => '%{IPV4:ip-address}\s-\s-\s\s[%{DATA:timestamp}\]\s"%{DATA}"\s%{DATA}\s%{DATA}\s"%{DATA:url}"\s"%{DATA}"'
"message" => '%{IP:ip-address}\s%{DATA:username}\s%{DATA:user}\s\s[%{DATA:timestamp}\]\s"%{DATA:action}\s%{DATA:path}\s%{DATA:protocol}"\s%{DATA:status}\s%{DATA:bytes}\s"%{DATA:url}"\s"%{DATA:user_agent}"'
}
}
Also, before indexing, the following index template is created.
$ curl -X PUT "localhost:9200/_index_template/its_index_template?pretty" -H 'Content-Type: application/json' -d'
{
"index_patterns" : [ ... snip ... "kin-*" ... snip ... ],
"priority" : 1,
"template": {
"settings" : {
"number_of_shards": 1,
"number_of_replicas": 0
}
}
}
'
I then ran the command in Dev Tools.
... snip ...
"bytes" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
... snip ...
I think I finally understand what mapping is here.
"type" : "text",!!!
I want this place to be numeric.
How can I get him to recognize it as a number?
In the template create / add a mapping section with all the correct mapping types.
Create the template with the correct mapping types already in..
"bytes" : {
"type" : "integer",
}
BTW there is an nginx filebeat module that does all this for you... you would not need to use logstash just Filebeat -> Elasticsearch and the data would all be nicely parse and typed and there are built in dashboards etc.
Quicks start just do the ngnix module which the quickstart show how to do
Ah, so you need to set the type in the index template.
That would mean I need to have a template for each type of log, right?
Since I already collect 10 different logs, does that mean I need to prepare 10 different index templates?
Nginx module | Filebeat Reference [8.11] | Elastic
So the information you gave me will be of great help to me.
One thing I am concerned about is what will happen to the index names.
I am using nginx with 10 different services.
Can I name them for each service?
Currently, I set index_name as the name in Filebeat and explicitly specify the name of the index in Logstash.
For example,
- type: log
enabled: true
paths:
- /var/log/www/httpd-access_log
fields: /var/log/www/httpd-access_log
index_name: kin-www-access
if([fields][index_name]) {
elasticsearch {
hosts => ["localhost"].
index => "%{[fields][index_name]}-%{+YYYYY.MM.dd}"
}
}
Can I do something similar to this?
"bytes" : { "type" : "integer", }
Following the above advice, I created the following index template.
curl -X PUT "localhost:9200/_template/httpd-access-template?pretty" -H 'Content-Type: application/json' -d'
{
"index_patterns": ["*-www-access-*", "*-httpd-access-*"],
"mappings": {
"_source": {
"enabled": false
},
"properties": {
"bytes": {
"type": "integer"
}
}
}
}
'
Then I get an error that timestamp is not found and it is not displayed by Kibana.
I just want to change the type of a given property.
You need to add all the fields not just that one...with the correct types etc
Eh. That's a hassle...
I tried to get all the types and set them again.
curl -X PUT "localhost:9200/_template/httpd-access-template?pretty" -H 'Content-Type: application/json' -d'
{
"index_patterns": ["*-www-access-*", "*-httpd-access-*"],
"mappings" : {
"properties" : {
"@timestamp" : {
"type" : "date"
},
"@version" : {
"type" : "text"
},
"action" : {
"type" : "text"
},
"agent" : {
"properties" : {
"ephemeral_id" : {
"type" : "text"
},
"hostname" : {
"type" : "text"
},
"id" : {
"type" : "text"
},
"name" : {
"type" : "text"
},
"type" : {
"type" : "text"
},
"version" : {
"type" : "text"
}
}
},
"bytes" : {
"type" : "integer"
},
"ecs" : {
"properties" : {
"version" : {
"type" : "text"
}
}
},
"fields" : {
"properties" : {
"index_name" : {
"type" : "text"
}
}
},
"host" : {
"properties" : {
"architecture" : {
"type" : "text"
},
"containerized" : {
"type" : "boolean"
},
"hostname" : {
"type" : "text"
},
"ip" : {
"type" : "text"
},
"mac" : {
"type" : "text"
},
"name" : {
"type" : "text"
},
"os" : {
"properties" : {
"codename" : {
"type" : "text"
},
"family" : {
"type" : "text"
},
"kernel" : {
"type" : "text"
},
"name" : {
"type" : "text"
},
"platform" : {
"type" : "text"
},
"type" : {
"type" : "text"
},
"version" : {
"type" : "text"
}
}
}
}
},
"input" : {
"properties" : {
"type" : {
"type" : "text"
}
}
},
"ip-address" : {
"type" : "text"
},
"log" : {
"properties" : {
"file" : {
"properties" : {
"path" : {
"type" : "text"
}
}
},
"offset" : {
"type" : "long"
}
}
},
"message" : {
"type" : "text"
},
"path" : {
"type" : "text"
},
"protocol" : {
"type" : "text"
},
"status" : {
"type" : "text"
},
"tags" : {
"type" : "text"
},
"timestamp" : {
"type" : "text"
},
"url" : {
"type" : "text"
},
"user" : {
"type" : "text"
},
"user_agent" : {
"type" : "text"
},
"username" : {
"type" : "text"
}
}
},
"settings" : {
"number_of_shards" : "1",
"number_of_replicas" : "0"
}
}'
However, the type is not changed. Why?
Not sure what that means... Have to tell us /show what you're seeing. Probably means the templates not being applied.
Hence why we make modules... That do a it all... Understand the desire to name all your indexes It's typically a bit overrated... You can put them all in the filebeat indices and then just put a tag at the source for the type and you'll always be able to filter with one simple filter You could argue. It's easier than changing the index pattern... Kind of the initial idea.
But I understand the desire but data into different index names... There are some good reasons to do that.
You could use the module of then copy the mappings into your own template..
BTW most of those text fields should be type keyword
Bottom line if you want to do it all into your own indexes then you need to create the templates which are the schema.
Could call the ngnix pipeline ...
Bottom line if you want to do your own indexes you need to create a template and a pipeline and an ILM policy if you want it to all be taken care of.
Sorry. I will explain exactly.
I created the index template in the above way.
And got the following result
curl -X GET "localhost:9200/_template/httpd-access-template?pretty"
{
"httpd-access-template" : {
"order" : 0,
"index_patterns" : [
"*-www-access-*",
"*-httpd-access-*"
],
"settings" : {
"index" : {
"number_of_shards" : "1",
"number_of_replicas" : "0"
}
},
"mappings" : {
"properties" : {
... snip ...
"bytes" : {
"type" : "integer"
},
... snip ...
}
}
}
}
However, I found that the type of index created has not been changed.
curl -X GET "localhost:9200/kin-www-access-2022.07.29?pretty"
{
"kin-www-access-2022.07.29" : {
"aliases" : { },
"mappings" : {
"properties" : {
... snip ...
"bytes" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
... snip ...
}
}
}
}
Does it not change the type of the index already created?
If by creating an index template, it will be applied to indexes that are created in the future (after tomorrow), then that is fine.
I would also like to change the type for indexes that have already been obtained.
For example, I executed the following command
curl -X PUT "localhost:9200/kin-www-access-2022.07.29/_mapping?pretty" -H 'Content-Type: application/json' -d'
{
"properties" : {
"@timestamp" : {
"type" : "date"
},
"@version" : {
"type" : "text"
},
"action" : {
"type" : "text"
},
"agent" : {
"properties" : {
"ephemeral_id" : {
"type" : "text"
},
"hostname" : {
"type" : "text"
},
"id" : {
"type" : "text"
},
"name" : {
"type" : "text"
},
"type" : {
"type" : "text"
},
"version" : {
"type" : "text"
}
}
},
"bytes" : {
"type" : "integer"
},
"ecs" : {
"properties" : {
"version" : {
"type" : "text"
}
}
},
"fields" : {
"properties" : {
"index_name" : {
"type" : "text"
}
}
},
"host" : {
"properties" : {
"architecture" : {
"type" : "text"
},
"containerized" : {
"type" : "boolean"
},
"hostname" : {
"type" : "text"
},
"ip" : {
"type" : "text"
},
"mac" : {
"type" : "text"
},
"name" : {
"type" : "text"
},
"os" : {
"properties" : {
"codename" : {
"type" : "text"
},
"family" : {
"type" : "text"
},
"kernel" : {
"type" : "text"
},
"name" : {
"type" : "text"
},
"platform" : {
"type" : "text"
},
"type" : {
"type" : "text"
},
"version" : {
"type" : "text"
}
}
}
}
},
"input" : {
"properties" : {
"type" : {
"type" : "text"
}
}
},
"ip-address" : {
"type" : "text"
},
"log" : {
"properties" : {
"file" : {
"properties" : {
"path" : {
"type" : "text"
}
}
},
"offset" : {
"type" : "long"
}
}
},
"message" : {
"type" : "text"
},
"path" : {
"type" : "text"
},
"protocol" : {
"type" : "text"
},
"status" : {
"type" : "text"
},
"tags" : {
"type" : "text"
},
"timestamp" : {
"type" : "text"
},
"url" : {
"type" : "text"
},
"user" : {
"type" : "text"
},
"user_agent" : {
"type" : "text"
},
"username" : {
"type" : "text"
}
}
}'
However, it does not work.
I get the following error
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "mapper [bytes] cannot be changed from type [text] to [integer]"
}
],
"type" : "illegal_argument_exception",
"reason" : "mapper [bytes] cannot be changed from type [text] to [integer]"
},
"status" : 400
}
Can I achieve what I want to do?
Filebeat's nginx module feature looks very attractive.
However, the index name seems to default to a form like filebeat-7.12.0-2022.07.29-000001.
On nginx, I have access.log and error.log, but I can't distinguish between them, so I want to change the index name.
How can I do this?
I found the following article but cannot set the index name properly
I have added the following statement
# vi /etc/filebeat/modules.d/nginx.yml
- module: nginx
access:
enabled: true
var.paths: ["/var/log/nginx/access.log"]
# vi /etc/filebeat/filebeat.yml
... snip ...
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]
indices:
- index: "else-httpd-access-%{+yyyy.MM.dd}"
when.equals:
event.module: "nginx"
- default: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"
... snip ...
However, I get the following error.
# tail /var/log/filebeat/filebeat
... snip ...
2022-07-29T19:27:17.106+0900 ERROR instance/beat.go:971 Exiting: error initializing publisher: missing output.elasticsearch.indices.1.index
... snip ...
I would like to have index names such as {my service name}-httpd-access-{date}, {my service name}-httpd-error-{date} for each access.log and error.log, and for each outgoing server.
For example, kin-httpd-access-2022.07.29, kin-httpd-error-2022.07.29, els-httpd-access-2022.07.29, els-httpd-error-2022.07.29, etc ...
Only go forward index
That can not be done.
This is actually how filebeat modules are designed to work.... by intention 
I often ask.... What are you actually trying to accomplish with separating the index name?... AGAIN that is fine but often it does not actually do much other allowing the users to see 2 index names... it is generally not more efficient etc, no storage space, more to manage etc and I would argue unless you do a good job of naming etc could make analysis more challenging etc...
With the module Yes both the data sets goes into the same index (in 8.x same Data Stream) ... but every doc is tagged with the access / error so any and all queries, visualizations, data etc...etc can always show one and the other or both etc. and since they are in the same index you can do aggregation
Example here are the dashboards etc using the modules... here
To do this : This is what folks do for custom data, it is a very common approach.
Create your own template (or copy / modify the filebeat template)
Create your own pipeline / parsing (or copy / modify the ngnix one provided)
Create or reuse and ILM policy
Then set / use your own names etc..
This is what happens with using the module
Hmmm. That's because you are monitoring the services of multiple companies.
For example, if you load-balance Company A's service with 5 servers and Company B's service with 20 servers, it would be possible to have a single index of the access logs from Company A's 5 servers (which is exactly what we are doing now).
However, it is not very desirable to have the access logs of Company A's server and Company B's server mixed together.
This is because although the purpose of monitoring access logs is the same, the timing we want to monitor is completely different between the two.
There are more or less the same number of servers to be load averaged, but the problem is that there are several access logs to be distinguished.
One way to do this is through what we call separate indexes.
Is there a way to solve the above problem if the indexes are grouped?
For example, putting a tag in the FIELD that distinguishes each service?
Ahhh Multi-Tenancy
Short answer yes, you can add as many fields or tags to the filebeat data at the source by just adding them into the filebeat.yml using add_fields or add_tags processor. The more advanced users do this via automation when deploying filebeat... Even more sophisticated Use cloud metadata to tag and then use.
So you have many options on the table.
One end of the spectrum is everything into a single index with tags or identifiers for company and services etc. Then all separation happens at query time.
The other end of the spectrum is an index for each company and service. Which is totally valid.
I know it seems a bit hard When you first get started but setting up a template, an ILM policy and reusing the pipeline is really only a couple hours work.
Either these are fine. It really depends on what your Multi-Tenancy requirements are Perhaps it's some combination of the two.
Example: I had a use case with 2,500 customers... It was a big use case and complex and my solution was moderately complex as well.
My top 10 customers all got their own indices because they created a lot of volume and they were high touch and I needed to do charge back and all those wonderful things.
My next 500 customers went into a common index.
And then the last 2000 small customers went into an index.
This afforded me a nice balance of control flexibility versus complexity, etc.
Everything was tagged for customer and service along the way.
You have lots of choices... your solution will depend on what your requirements are.
Let us know your thoughts.
I don't have that many clients, but the examples you show are very useful to me.
I would like to create two Dashboards for each company you demoed.
For this purpose, we think it is better to create an index for each company rather than defining fields and tags.
That is because I want to create one for each company in Discover as well as Dashboard.
I have two problems.
The first is that when using Filebeat and Logstash to define index names, I need to create a template with all property types mapped.
This requires knowing all properties in advance. Also, when a property changes, you have to enumerate all the properties again. This is a hassle, and at the same time, it is not possible to change the type for an already created index.
To top it off, in my environment, the type is still text in Kibana even though I changed the type of the template.
# curl -X GET "localhost:9200/_template/httpd-access-template?pretty"
{
"httpd-access-template" : {
"mappings" : {
"properties" : {
"bytes" : {
"type" : "integer"
},
Second, when using Filebeat's nginx module, the index name is filebeat-7.12.0. This is the default index name, but I can't figure out how to change it.
The method I have tried is the one shown in #16, but it doesn't seem to work.
It would be very nice to be able to pre-determine the index name in Filebeat's nginx module.
Is there a solution for these?
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.
