Hi All,
I am using elastic. co to check account and billing information.
Can I make a dashboard in kibana from elastic.co data present in account and billing info.
Please let me know if there is a way around to extract billing data from elastic.co to kibana and build elastic,co cost analysis dashboard in kibana.
AFAIk there's no way to do this.
I would suggest contacting the Support team to make sure, and they might be able to pass this onto the product team as a feature request.
@Divyank_Mahalle Thanks for using Elastic Cloud
Actually there is an Account Billing API Here
You will need to pull the data and the ingest it into your elasticsearch cluster ... yeah not so easy, I think at some point we will work on making this easier... but you can then do build a dashboard if you want.
Oh that's awesome!
Thanks for the Solution.
For this below GET request, I got deployment id but was unable to find organization_id in elastic.co id.
Is organisation_id and cluster_ID are same?
I generate API key using for Billing API in elastic.co feature tab.Can I used this API key to request billing data or there is different way?
GET /api/v1/billing/costs/{organization_id}/deployments/{deployment_id}/items
Once organiization ID is confirmed, here is Psuedo API request looks like-
Is it correct?
curl -XGET https://api.elastic-cloud.com/api/v1/billing/costs/{<
>}/deployments/{< >}/items \
-H "Authorization: ApiKey <AKJSBEHXk13123ksdkwbeiabds>"
Sorry for asking the novice questions, as I am using it for first time.
Thank you once again!
@Divyank_Mahalle No Problem
The organization_id should be your account id on The Account and Billing Page
With respect to the API Syntax Yes that looks correct (although there are several variations on the page I referenced) and you need to make sure you get the Deployment ID correct From the Deployments page you can either copy it from the URL or from the Copy Cluster ID link.
I've deleted that last post, you probably don't want to be publicly posting your auth key.
Thanks, Got it both organization and deployment id.
I tried running API requests on the browser to check if it's fetching data, but it showing an error message.
I guess I am missing something here.
#{"errors":[{"code":"root.unauthenticated","message":"The resource requires authentication, which was not supplied with the request"}]}
Psuedo API request format-
https://api.elastic-cloud.com/api/v1/billing/costs/<org id>/deployments/<deployement id>/items \
-H "Authorization: ApiKey <API Key>"
I ran mine directly as documented with curl and I got the results
Did you follow these instructions to get the proper API key?
You should try with curl not in a browser.
Yes, I follow the same procedure and generate API Key.
Still getting the same error while running the using curl command in the VM/linux.
Please let me know where i am doing wrong.
$ curl -XGET https://api.elastic-cloud.com/api/v1/billing/costs/<org id> \ -H "Authorization: ApiKey $EC_<Api key>"
{"errors":[{"code":"root.unauthenticated","message":"The resource requires authentication, which was not supplied with the request"}]}curl: (6) Could not resolve host: -H; Unknown error
curl: (6) Could not resolve host: Authorization; Unknown error
Don't Prefix the API KEY with the $EC_
Just paste in directly.
curl -XGET https://api.elastic-cloud.com/api/v1/billing/costs/<organization_id> \
-H "Authorization: ApiKey <yourapikey>"
OR
if you use the syntax in the docs set this before
That is an example when using an environment variable.
export EC_API_KEY=<yourapikey>
then
curl -XGET https://api.elastic-cloud.com/api/v1/billing/costs/<organization_id> \
-H "Authorization: ApiKey $EC_API_KEY"
Also if you put it on 1 line do not include \
Thank you so much API is working and I am able to get output results.
I checked with "Get costs overview for the organization" and getting correct values for current values which are billed till today for the current month.
curl -XGET https://api.elastic-cloud.com/api/v1/billing/costs/<organization_id> \
-H "Authorization: ApiKey <yourapikey>"
{
"costs" : {
"total" : 55.5166,
"resources" : 40.8226,
"data_transfer_and_storage" : 14.694
},
"trials" : 0.0,
"hourly_rate" : 0.1989,
"balance" : {
"available" : 0.0,
"remaining" : 0.0,
"line_items" : [
]
}
While I checked for "Get itemized costs for the organization" its showing different values from elastic.co billing and showing values for 734 hours instead it showing calculate hours from 1 st aug till today.
Is it forecasting the values of resources for the next 20 days?
curl -XGET https://api.elastic-cloud.com/api/v1/billing/costs/{organization_id}/items
-H "Authorization: ApiKey API_KEY"
{
"costs" : {
"total" : 160.76,
"resources" : 146.0128,
"data_transfer_and_storage" : 14.7472
},
"resources" : [
{
"hours" : 734,
"instance_count" : 1,
"period" : {
"start" : "2021-08-01T00:00:00.000Z",
"end" : "2021-08-31T14:06:06.783Z"
},
"kind" : "apm",
"price" : 0.0,
"price_per_hour" : 0.0,
"name" : " <Name>" ",
"sku" : "<Name>"
},
{
"hours" : 734,
"instance_count" : 1,
"period" : {
"start" : "2021-08-01T00:00:00.000Z",
"end" : "2021-08-31T14:06:06.783Z"
},
"kind" : "elasticsearch",
"price" : 132.1383,
"price_per_hour" : 0.18,
"name" : "<NAME>",
"sku" : "<NAME>"
},
{
"hours" : 734,
"instance_count" : 1,
"period" : {
"start" : "2021-08-01T00:00:00.000Z",
"end" : "2021-08-31T14:06:06.783Z"
},
"kind" : "elasticsearch",
"price" : 13.8745,
"price_per_hour" : 0.0189,
"name" : "<NAME>",
"sku" : "<NAME>"
},
{
"hours" : 734,
"instance_count" : 1,
"period" : {
"start" : "2021-08-01T00:00:00.000Z",
"end" : "2021-08-31T14:06:06.783Z"
},
"kind" : "kibana",
"price" : 0.0,
"price_per_hour" : 0.0,
"name" : "<NAME>",
"sku" : "<NAME>"
}
]
Yes if you look at the documentation specifically here
| Name | Type | Required | Description |
|---|---|---|---|
from |
string |
N | A datetime for the beginning of the desired range for which to fetch costs. Defaults to start of current month. Note: there is currently a three-month maximum date range. |
to |
string |
N | A datetime for the end of the desired range for which to fetch costs. Defaults to end of current month. Note: there is currently a three-month maximum date range. |
I am a little curious why the end date is not 23:59.59 on Aug 31 (I am asking internally)
Remember this is still experimental meaning this is new and can change.
I would also recommend reading the docs closely.
I got your point and referring Billing API documentation,
Now I am testing billing API for "Get costs to overview for the organization"
Config file and Output attached. Getting error of Invalid URL or root unauthenticated.
Although I search for HTTP poller authentication, got this link, but didn't get much idea how to process it in URL.
https://discuss.elastic.co/t/http-poller-authentication-encode-token/143532
This config setting is just for testing purpose-
input{
http_poller {
type => "billing"
urls => {
url => "https://api.elastic-cloud.com/api/v1/billing/costs/<orgid>-HAuthorization:ApiKey<my api key>"
}
request_timeout => 60
schedule => { every => "5s"}
codec => "json"
}
}
output {
stdout {
codec => rubydebug
}
#elasticsearch {
#host ip to be changed to new test system
# hosts => ["<hostaname>"]
#index => "<Billing>"
# user => "<username>"
# password => "<password>"
#}
}
Output
-bash-4.2$ sudo bin/logstash -f /etc/logstash/conf.d/billing1.conf
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in ve rsion 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (fil e:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.11.1.jar) to met hod sun.nio.ch.NativeThread.signal(long)
WARNING: Please consider reporting this to the maintainers of com.headius.backpo rt9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflect ive access operations
WARNING: All illegal access operations will be denied in a future release
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/conf ig or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.pr operties. Using default config which logs errors to the console
[WARN ] 2021-08-10 07:49:26.800 [LogStash::Runner] multilocal - Ignoring the 'pi pelines.yml' file because modules or command line options are specified
[INFO ] 2021-08-10 07:49:26.828 [LogStash::Runner] runner - Starting Logstash {" logstash.version"=>"7.8.0", "jruby.version"=>"jruby 9.2.11.1 (2.5.7) 2020-03-25 b1f55b1a40 OpenJDK 64-Bit Server VM 11.0.12+7-LTS on 11.0.12+7-LTS +indy +jit [l inux-x86_64]"}
[INFO ] 2021-08-10 07:49:35.000 [Converge PipelineAction::Create<main>] Reflections - Reflections took 328 ms to scan 1 urls, producing 21 keys and 41 values
[INFO ] 2021-08-10 07:49:37.791 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/billing1.conf"], :thread=>"#<Thread:0x54177e5c run>"}
[INFO ] 2021-08-10 07:49:41.357 [[main]-pipeline-manager] http_poller - Registering http_poller Input {:type=>"billing", :schedule=>{"every"=>"5s"}, :timeout=>nil}
[INFO ] 2021-08-10 07:49:41.459 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2021-08-10 07:49:41.769 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2021-08-10 07:49:43.077 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
"type" => "billing",
"@version" => "1",
"@timestamp" => 2021-08-10T07:49:51.423Z,
"errors" => [
[0] {
"message" => "The resource requires authentication, which was not supplied with the request",
"code" => "root.unauthenticated"
}
]
}
You should open a separate thread since you now have a Logstash question.
Per the docs here the first example shows how to use headers
test2 => {
# Supports all options supported by ruby's Manticore HTTP client
method => get
user => "AzureDiamond"
password => "hunter2"
url => "http://localhost:9200/_cluster/health"
headers => {
Accept => "application/json"
}
}
Sure @stephenb ,Thank you so much for addressing my queries.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.