Issue in graph

Elastic Stack Kibana
1

I need a simple graph.

my search query is returning two fields, filed-1 and field-2

I want to plot field-1 in X-Axis and field-2 in Y-Axis.

I have checked Visualization graphs i.e Line graph , Area graph , Vertical graph where I can not select field for Y-axis.....there is only metrics.

How do I plot of X-Y axis fields graph then ?

2

is there any connection between field 1 and field 2

3

field-1 is employee ID
field-2 is salary

I need a simple graph , field-1 in X-Axis and field-2 in Y-Axis.

How to plot this ?

4

in y axis it is already predefined like count,sum,average u can mention whatever u want for x axis click bucketfield->xaxis-> under aggregation select terms and after that select ur field name...suppose if u want to related ur empid and salary den click add subbucket and follow the same above procedure

5

When I click "subbucket" I get this two options .

Which type to select ?

6

anything u can

7

X-Axis --> Split Series > Terms Aggregation > Select your employee ID field
Y-Axis --> Sum on salary.

You gotta be sure the employee ID is unique in your data, cause if you have 2 docs with same employee ID, the graph will show the sum of they're salaries

8

Why Sum ?

my intention is to plot employee Id vs Salary . How Sum can help ?

9

Because on the X-axis you're technically making an aggregation. It means that each point of the X-axis is an aggregation of all those documents with the same employee ID.

Then, on the Y-axis you canno simply plot a static value but you have to perform a mathematical aggregation (like a sum, a average or whatever else), which is done on a set of documents (the ones aggregated in your X-axis point).

Though, IF your employee_id is always unique on your field, on the X-axis you'll have "aggregations" of single document. And making a sum (or an average or whatever else) on 1 value is like plotting that value. I mean, a sum over value 29 is again 29. Just like the average of 29 is 29.

If you want to plot it that way AND your employee_id is unique in your index, you can do that.

10

Okay. I tried this
Error incompatible field type.

I believe this, This is happening since I am using salary of String type (not number).
In this case, how do you make it?

11

You need to define a new mapping for your index and reindex your data.

Did you apply any custom mapping or template to your current index?

12

No custom mapping.

13

Why re-index ?

Why "Sum" does not work ?

Is it because all fields are of type String ?

14

Sum doesn't work because that "salary" field of yours is not numeric. You cannot make a sum (or an average, or a maximum ..) on a field of type string.
That's why you have to reindex your data applying a different mapping (unfortunately you cannot change the mapping of a field which is already defined on an index).

Thus, supposing your index is called "test-index", you can do something like

  • create a new index "test-index2" (with no documents for now) with a mapping which includes the property "salary" of type "float" (or double, depending on how much your employees earn).

  • reindex your data from "test-index" to "test-index2". You'll see that the docs in this new index now will have salary field mapped as a number.

  • delete the old "test-index" (make sure all the documents have been coorectly reindexed). Re-create a new "test-index", this time with a mapping like you did for "test-index2".

  • reindex your data from "test-index2" to "test-index" (making sure all the data have been transferred and the new docs in the new "test-index" have field "salary" mapped as a number).

  • delete "test-index2"

Obviously, if there is any other field that requires to be converted, do it now in order to avoid making this re-index process every time.

15

When I create new "index-pattern" we dont give mapping in Kibana.

16

I didn't write index pattern. I wrote index. You have to create another index.

How many docs are now in your index? What's the name of the index?

17

Where ? at Logstash or at Kibana ?

Currently I write index name in logstash.conf .... Do you mean I should be put a different index name here ? YES/NO ?

But note, this wont transfer data from "existing" index to new index. This will create new index . Since logs are rolled over , there will not be same number of docs.

18

Dude, I wanna help you, but you gotta help me do it in the first place and you can do it following a simple rule:

When you receive a reply, try following it, googling what you don't know and, if you find nothing, asking here that specific thing.

Now, you didn't speak of Logstash in the first place (plus we're in the Kibana section here), then why should I? If I meant something external I would name it. You can create an index by simply using the Dev Tools section of kibana (which is basically a fancy way to make cURL calls). Typing

PUT /test-index2
{
    "mappings" : {
        "properties" : {
            "salary" : { "type" : "float" }
        }
    }
}

You would have known it, if only you'd googled "create index elasticsearch".

Then, in order not to lose your current data, I specifically asked you to reindex you data. Which means to copy your data from one index (the old one) to another index (the one you've just created with the right mapping). Which you can do (again in the Dev Tools) by typing:

POST _reindex
{
  "source": {
    "index": "test-index"
  },
  "dest": {
    "index": "test-index2"
  }
}

Again, you would have known it if only you'd googled "reindex elasticsearch". And if you didn't know the reindex and didn't make the effort to google it, at least ask here what it is.

Now, once you've checked all the docs have been copied from one index to the other (Management>Index Management and see the number of docs of the 2 indices) and that the new index docs have a salary field of type "float", then you can delete the old index (in the Management>Index Management section flagging it and deleting it or in the Dev Tools typing DELETE test-index), create again the test-index index with the proper mapping as you did for test-index2, reindex data (this time from test-index2 to test-index) and delete the test-index2.

Also, you have to answers all the questions that are asked you, because they might be fundamental to establish the best approach to achieve what you want.
I specifically asked you (and you didn't reply)

How many docs are now in your index?

This is useful to know because the reindex takes a time proportional to the number of docs it has to copy from one index to the other.

Also, now you mention Logstash so, are the data still flowing? Because as you started your post I assumed you had a static situation.
If the data are flowing you need to redirect your flow to the new test-index2 while reindexing (obviously make the right mapping first or it'll be useless) and then switch the flow back to test-index before reindexing back.
Not to have any problem with chronological order of the docs, though, they must have a timestamp field, which I believe they have.

Another unreplied question of mine (made to simplify your work and write for you some ready-to-use calls) is:

What's the name of the index?

19

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.