Ingest pipeline execute external command or call rest?

Elastic Stack Elasticsearch
1

I'd like to know if it's possible to execute an external command from an ingest pipeline?

My usecase is to do a unix dig command on an IP and see who owns it. We have Akamai fronting a lot of our web traffic and I'd like to identify Akamai requests.

Manually I can take an IP and execute the following:

$ dig -x 88.221.222.138

; <<>> DiG 9.8.3-P1 <<>> -x 88.221.222.138
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12542
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;138.222.221.88.in-addr.arpa. IN PTR

;; ANSWER SECTION:
138.222.221.88.in-addr.arpa. 86400 IN PTR a88-221-222-138.deploy.akamaitechnologies.com.

;; Query time: 136 msec
;; SERVER: 2001:578:3f::30#53(2001:578:3f::30)
;; WHEN: Thu Aug 24 09:44:14 2017
;; MSG SIZE rcvd: 104

I'd like to grep for that akamaitechnologies.com like and flag the request as from Akamai and update a field.

2

Hey,

this is not possible. We dont want to introduce blocking calls into the ingest infrastructure. You could do that with writing your own processor, but this also implies that your ingestion pipeline and thus your capability to index documents depends on the availibility of external services. Maybe indexing first and then having a second process that uses a scroll search is a better idea, or use the update API later in time.

--Alex

3

Totally makes sense. Thanks for the response. I will investigate what you mentioned.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.