I have a number of different load injector boxes and processes that
generate load through our system under test. The tool in use produces
masses of logs out but none of it is easily accessible or readable. I am
therefore using ELK to process the loads with success, however, presenting
some of the data is problematic.
The process names have a structure of:
__StressThread
but there are around 180 of them. Each process thread generates 1 of 11
different message types. The message types are only distinguishable using a
10 character substring within a field containing strings and semi_colon
delimited text and generally in the same location (between character 60 and
character 70.
In elasticsearch none of these fields are analysed as this makes the
queries and results even messier in Kibana and poses the same problem when
choosing analysed elements of a field (only picking element 1&2 or 10-12).
When aggregation is done on either of these fields, "message type" is
presented as the long string in the visualisation key with only the first
10-15 characters showing, and "process name" resutls in all 180 processes
rather than the 7 process types.
These processes are likely to change over time as we introduce new test
scenarios and message types so I do not want to hard code them in just in
case we miss something.
Is it possible to have elasticsearch return substrings, partials, lefts,
rights etc of a field and group them as such rather than the entire field
content so all _a are grouped together and <msg_type>A are
grouped together? Ideally without code edits to either elastic or kibana?
Something in JSON Input such as { "field_length": 10 } or {
"partial_start": 60, "partial_for": 15 } would suffice.
This is akin to renaming keys, columns and rows.
Thanks
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/52076863-dc85-43c4-be6c-fba643391583%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
{"script": "_value.substring(0,8)"} works for you, needing groovy sandbox
enabled.
On Wednesday, April 29, 2015 at 9:39:33 PM UTC+8, Stuart Kenworthy wrote:
I have a number of different load injector boxes and processes that
generate load through our system under test. The tool in use produces
masses of logs out but none of it is easily accessible or readable. I am
therefore using ELK to process the loads with success, however, presenting
some of the data is problematic.
The process names have a structure of:
__StressThread
but there are around 180 of them. Each process thread generates 1 of 11
different message types. The message types are only distinguishable using a
10 character substring within a field containing strings and semi_colon
delimited text and generally in the same location (between character 60 and
character 70.
In elasticsearch none of these fields are analysed as this makes the
queries and results even messier in Kibana and poses the same problem when
choosing analysed elements of a field (only picking element 1&2 or 10-12).
When aggregation is done on either of these fields, "message type" is
presented as the long string in the visualisation key with only the first
10-15 characters showing, and "process name" resutls in all 180 processes
rather than the 7 process types.
These processes are likely to change over time as we introduce new test
scenarios and message types so I do not want to hard code them in just in
case we miss something.
Is it possible to have elasticsearch return substrings, partials, lefts,
rights etc of a field and group them as such rather than the entire field
content so all _a are grouped together and <msg_type>A are
grouped together? Ideally without code edits to either elastic or kibana?
Something in JSON Input such as { "field_length": 10 } or {
"partial_start": 60, "partial_for": 15 } would suffice.
This is akin to renaming keys, columns and rows.
Thanks
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/266c13a5-6a95-4aea-bd3e-1e7edf6eb977%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
2 Likes