Unsure How to Handle via Logstash/Elasticsearch

Elastic Stack Elasticsearch
1

I was hoping for some advice on how to handle the following, I'm not sure how to do it via query or logstash filter.

We have many different logs in the format below coming into Elasticsearch. The ActionSequenceId field correlates activity across the many logs. We've been asked to pull that data in and present it back as per the requested output.

I tried collecting this data via the logstash aggregate filter but because it's coming from many different sources, the ID fields are not reliably clustered. And for the life of me I can't seem to figure out the kind of query I need to build to pull this data out. I have not looked at Elasticsearch ingestion nodes but I'm headed there next.

Input This is going into Elasticsearch via the CSV filter w/o issue.
| INFO | 2017-01-09 06:29:45,414 | ActionSequenceId | Message | UserId | DocBaseCode | IPAddress | URI | FunctionName | ApplicationId | Exception |
| DEBUG | 2017-01-09 06:29:45,618 | 6361954018443 | entering | BOWKERR | SomeDocbaseCode | SomeIP | http://someurl | SomeFunctionName | SomeGuid | |

Requested Output
ActionSequenceID
AllFunctionNamesAndTimestampsAssociatedWithActionSequenceId[{Func1,TS1}
{Func2, TS2}]
TimeElapsed
#OfCallsByLogType

Can someone recommend an approach?

2

Ingest node won't help as it doesn't have much of a concept of cross event aggregations like this.

You may need to do a dual step process, index each event, then group them into another index.

3

What would you use for the 2nd step? Should I just write a program to reindex the data into the format I want once it's done or is there some other process? Is there no query that can help me assemble the data in the format I need (I know I can't find one, but I'm a rookie)?

4

You could Logstash again, you can then run a query to grab the docs.

1 Like
5

Oh nice, I didn't realize that I could use Elasticsearch as an input but makes total sense. So would I just set up 2 logstash instances, one to ingest log data and another to reindex the ingested data or can you run them both from within a single instance?

6

You can run from a single instance.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.