Conditional search in Lucene or Query DSL

Elastic Stack Elasticsearch
1

Hi guys,

I'm trying to do a search via Lucene or Query DSL that does the following:

I have a log that contains several fields, among them the sourceIp and resourceKey fields. I need to create a survey that shows me all the logs where the same sourceIP brings more than 1 resourceKey value, the problem is that I don't know where to start.

Thanks

2

hey,

can you share a sample document, and mapping, that would make it easier to come up with a proper query.

Thank you!

--Alex

3

Thanks for answering @spinscale. Below is a sample of the log.
What I need is to bring the sourceIPs that have + than x resourceKey values.

So it would be, if 1 IP connects to 1 or more resourceKey brings the result.

@timestamp
Jun 18, 2021 @ 18:28:43.040

@version
1

_id
X5gGIXoBknzImTVfq8np

_index
xxxxxxxxxxxx

_score
** - **

_type
_doc

apiKey
xxxxxxxxxxxxx

application
xxxxxxxxxxxxx

brand
xxxxxxxxxxx

category
xxxxxxxxxx

class
xxxxxxxxxxx

cluster_name
team-checkout-pf

date
Jun 18, 2021 @ 18:28:42.000

environment
aws

file
BaseExceptionHandlerControllerAdvice.java

headers.content_length
114051

headers.content_type
application/json

headers.http_accept
** - **

headers.http_host
xxxxxxxxxxxxxxxxxx

headers.http_user_agent
Fluent-Bit

headers.http_version
HTTP/1.1

headers.request_method
POST

headers.request_path
/

headers.x_amzn_trace_id
xxxxxxxxxxxxx

headers.x_forwarded_for
xxxxxxxxxxxxx

headers.x_forwarded_port
443

headers.x_forwarded_proto
https

host
xxxxxxxxxxx

hostname
xxxxxxxxxxxxxxxx

httpCode
403

httpMethod
POST

kubernetes.annotations.cattle.io/timestamp
Aug 21, 2020 @ 01:35:58.000

kubernetes.annotations.checksum/config
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

kubernetes.annotations.checksum/secret
dd043c802fd10d3a603b89e83c2e8b6c8b55b5e464be1c7303d0ac30e3016da3

kubernetes.annotations.kubernetes.io/psp
eks.privileged

kubernetes.annotations.sidecar.istio.io/inject
false

kubernetes.container_hash
xxxxxxxxxxxx

kubernetes.container_image
xxxxxxxxxxxx

kubernetes.container_name
customer-token

kubernetes.docker_id
e60d2a0c6496b01eb848104ebf7ef2ad0461629f9966219ee46599c266af2715

kubernetes.host
xxxxxxxxxxx

kubernetes.labels.app
customer-token-v1

kubernetes.labels.brand
xxxxxxxxx

kubernetes.labels.chart
deployment-1.26

kubernetes.labels.elbv2.k8s.aws/pod-readiness-gate-inject
enabled

kubernetes.labels.flow
pf

kubernetes.labels.heritage
Tiller

kubernetes.labels.name
xxxxxxxxx

kubernetes.labels.pod-template-hash
5bdb6769b

kubernetes.labels.release
xxxxxxxxxxxx

kubernetes.namespace_name
team-customer

kubernetes.pod_id
xxxxxxxxxxx

kubernetes.pod_name
xxxxxxxxxxx

line_number
113

log_level
ERROR

log_message
Nao foi possivel autenticar o cliente com o nome de usuario e senha informados. Login: garrastazuxxxx@xxxxxx.com.br

method
handleServCorpException

referer
UNKNOWN_REFERER

resourceKey
garrastazuxxxx@xxxxxx.com.br

service.name
customer_token

sourceIp
222.XXX.92.XXX

thread_name
http-nio-8080-exec-4

throwable.exception_class
xxxxx

throwable.exception_message
Nao foi possivel autenticar o cliente com o nome de usuario e senha informados. Login: garrastazuxxx@xxxxx.com.br

4

Please take some more time to properly write up a problem statement and a minimal example. There is no index provided, no mapping, and no sample document in JSON format that others could use for testing. A multi page long sample document, where only 2 columns are needed out of 20 is also not helpful to understand the full context.

Also

What I need is to bring the sourceIPs that have + than x resourceKey values.

is something that does not make too much sense to me. Coming up with a sample query (even if verbalized) plus a document that matched and also explaining when a document does not match would help a lot.

I know this require some more work up front, but otherwise getting help is a lot harder and community members will much more likely ignore your problem.

Thank you!