So the good news is we found your issue.
The not-so-good news is that this was hard to find because whoever set up your elasticsearch cluster did not set ingest node roles which is highly unusual (perhaps just a misunderstanding) NONE of our integration that use pipelines would work, that is ~95% of them. No Custom Ingest Pipeline would work either which removes a major capability from elasticsearch... not a best practice for sure.
With No Nodes with ingest role, ingest pipeline can not / will not work, ingest pipeline can not be executed and thus data will NOT be ingested.
You do NOT have to have separate ingest nodes, that role can run on the data nodes unless you have some extreme conditions which I suspect you do not.
Explanation:
node.role, r, role, nodeRole
(Default) Roles of the node. Returned values include c (cold node), d (data node), f (frozen node), h (hot node), i (ingest node), l (machine learning node), m (master-eligible node), r (remote cluster client node), s (content node), t (transform node), v (voting-only node), w (warm node), and - (coordinating node only).
So looking at your _cat/nodes
You have 4 data nodes for explanation see here
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
9.9.9.9 14 97 8 0.27 0.23 0.26 hs - ec1
10.10.10.10 70 97 1 0.07 0.04 0.05 hs - ec2
7.7.7.7 54 90 1 0.00 0.00 0.00 hs - ed5
6.6.6.6 47 98 7 0.15 0.11 0.12 hs - ed4
So when these nodes were set up they were set up with h=hot and s=content ONLY which is good BUT they did not include ingest (at least) and transform (which is good to have)
No Ingest Role, No Ingest Pipeline Will Work, Apache Module nor Any PIpeline will work
Quick look at my data node
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
10.44.3.133 58 99 36 1.83 1.86 1.71 hist - instance-0000000001
10.44.2.186 42 94 33 1.91 2.43 2.52 hist * instance-0000000000
Your nodes should probably have at least these roles
hist
h=hot
i=ingest
s=content
t=transform
Which translates in the elasticsearch.yml file see Here to read about node roles
Today on those data nodes it will look like
node.roles: [ data_hot, data_content ]
But it should look like the following (transform is a good role as you might want to use it in the future and it works with basic license.
node.roles: [ data_hot, data_content, ingest, transform]
THIS is your ISSUE.
You need to get this fixed... no other way around this!
Then once the elasticsearch is fixed, if you go all the way back to the simple setup with the apache.yml and the file input turned of etc.. etc. just follow the quickstart it should all work.
