Index sql statements into elasticsearch

[liorg2]

Hello
I would like to clean sql statements that im indexing into elastic

Any recommended method?

[Christian_Dahlqvist]

Have you looked at using Logstash with a JDBC input plugin?

[Christian_Dahlqvist]

As it is not clear what your requirements are it is hard to give advice. If you have complex requirements and Logstash does not fit you might be better off creating a script.

[Christian_Dahlqvist]

I am afraid I do not understand. If you require a lot of flexibility it might be easier to build a script or application using one of the language clients.

It may help if you give some examples of what you are looking to do and what you expect the result to be.

[liorg2]

I just assume that it's a common problem when ingesting logs as well, and I wonder how others deal with that using elastic (if any).. for example

if theoretically you have the following code in a C# app, which are being streamed into elastic.

System.Diagnostics.Trace.WriteLine("User" + UserId + " have done a deposit of " + amount)

now you want to group by log message, but you have lots of combinations of userId/ amount.

I think my example is simple:
imagine I got 3 messages inserted into a field named "sql"

1. "select * from profile where userid=123456"
2. "select * from profile where userid=456789"
3. select * from profile where userid=1255"

now I want to count messages and get the following (hopefully terms aggregation )

"select * from profile where userid=", doc_count=3

hope it's clear

[Christian_Dahlqvist]

OK. I misunderstood and thought you wanted to index data based on running SQL statements.

If the data comes from logs you should look at grok and dissect filters in Logstash or possibly to use these filters in an ingest pipeline in Elasticsearch.