How to use ingest pipelines and processors

[ailengcon]

So thankful if someone could help me out.

I have synced data from a mongodb to elastic with Monstache. Some of the data is mapped so when I make searches, the presentation of the data is rather messy and unstructured, not in the way it is suppose to be in database. Aggregations and such works fine, it's just that I want the user to be able to have the data presented so they can find the values they are looking for quicker.

I have attached images of how I want it to look like (1st image from from Robo3T), and how it looks in Kibana (Second image). It sort of groups the fields separately. I wonder if I can use an ingest pipeline to alter the mapping somehow? But not sure which processor to use? I can't find any good tutorials or instructions for this?

[warkolm (Mark Walkom)]

Welcome to our community!

Can you share a little more of what you see in Kibana, as it's lacking some context.

[ailengcon]

Sure, so Parameters is an array and should be grouped for each index, as image one above. But instead this is what I get.

So if I want to look at this document's parameter index 67 to see what the value of that index is, it would be really annoying because I have to count manually to find it in this mess. Sure I can search for it so that it is highlighted, but my employer (who I am evaluating elastic for) want to be able to examine the different parameters in a document to get a good overview, therefore it needs the original structure (image one in the original post).

Hope this helps, and that there is a way for me to alter it. Thank you.

[warkolm (Mark Walkom)]

What does that look like as a json doc rather than that view?

[ailengcon]

Like this. Does it help?

[warkolm (Mark Walkom)]

Yeah, it looks like you only have one single document, which seems odd.
Does mongo only have a single document in it with all these values?

[warkolm (Mark Walkom)]

Ah ok, thanks for clearing that up.

If you want to have something like the n elements or n fields values, you could probably add that n count with a script processor that will iterate through and create it.

[ailengcon]

I'm sorry, I'm not sure I understand. I can see that the instruction is a console example also, isn't there any tutorial that is on another level?

Also not sure if I understand "n elements"... Thank you for all you help, sorry if I'm not getting it.

I need to sync the data all over again, right? to use the pipeline?

This example is super clear, but I am not sure how to insert the fields I want to change so that I do it right.

[ailengcon]

Maybe index template is what I'm looking for to make it work?

[warkolm (Mark Walkom)]

You basically want to have a document that has a field showing the total count of value in another field. eg;

parameters: [ list of parameters here ]
parameters_count: [ a count of the above parameters ]

At least that is what I am understanding based on the above posts.

[ailengcon]

No not exaclty, or maybe - I started a new topic on it ( [Index templates] ) where maybe it's easier to understan. I at least want it structured the same way as in the database. I tried using index template but that resulted in elastic apparently making (millions) more documents than there is. (I read about it having something to do with the nested datatype)?

I started a new thread and got the advice to bring the topic to Elasticsearch instead of KIbana. I can do so, I also got a tip trying to use mongoconnector of some sort instead of monstache to synd the data. But I can't find any good instructions or tutorial on that. Can this help fix this issue?