Is Network Traffic (Fleet Integration) the new Packetbeat? - Missing Netfow?

zx8086 · September 30, 2021, 11:08pm

Hi Guys,

I noticed that the integration didn't create the indices for network flow. Is this a bug or missed configuration?

{"type":"security_exception","reason":"action [indices:admin/auto_create] is unauthorized for API key id [y_XFOHwBVDdS0yUQQK5L] of user [elastic/fleet-server] on indices [logs-network_traffic.flow-default-2021.09.30], this action is granted by the index privileges [auto_configure,create_index,manage,all]"}, dropping event

andrewkroh · October 1, 2021, 3:19pm

I saw a similar permissions issue in No Elastic Security Events but Agents status is "green" - #2 by andrewkroh. With that issue, recreating the policy for that agent cleared up the problem.

stephenb · October 1, 2021, 3:38pm

Hi @zx8086 First we are not all guys

I repeated this as well ...

Curious...Where did you pull that error log line from was?

Can you provide your elastic stack version / agent version method of install OS for elastic and where you deployed the agent to?

zx8086 · October 1, 2021, 3:44pm

Curious...Where did you pull that error log line from was?

Can you provide your elastic stack version / agent version method of install OS for elastic and where you deployed the agent to?

7.15
Network Traffic from Elastic-Agent Integration
From Observability Logs affecting all host Debian (RPi4) and Darwin (Mac OSX 10.13) deployed by Debian Buster 10 node

stephenb · October 1, 2021, 3:49pm

Thanks

Ok I guess I meant was that an Elastic Agent log, Kibana log or Elasticsearch log message, no worries we will take a look.

EDIT that is a log message from the agent... we are looking.

zx8086 · October 2, 2021, 12:11am

@stephenb @andrewkroh

From index .ds-logs-network_traffic.dns-default-2021.09.30-000001

Also with Network Traffic...

TLS (when Enabled)

{"type":"mapper_parsing_exception","reason":"failed to parse field [tls.detailed.server_certificate_chain] of type [keyword] ... "caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:3620"}}, dropping event!

Not sure how Fleet Managed integrations have mapping issues for default ingestion / pipeline setups.

andrewkroh · October 13, 2021, 3:29pm

I think I found the problem with logs-network_traffic.flow and have opened a fix.

github.com/elastic/beats

[Packetbeat] Fix data stream name for network flows under Agent

elastic:master ← andrewkroh:bugfix/pb/network-flow-data-stream-index

opened 03:23PM - 13 Oct 21 UTC

andrewkroh

+2 -1

## What does this PR do? This fixes and issue with network flows being writte…n to the wrong index when using the Network Packet Capture integration in Fleet. The error was: `{"type:"security_exception", "reason":"action [indices:admin/auto_create] is unauthorized for API key id [xxx] of user [elastic/fleet-server] on indices [logs-network_traffic.flow-default-2021.10.13] …"}` The cause is that flows were setting `index` rather than `raw_index`. With `index` Beats adds the date suffix, but since this is a data stream we want `raw_index` where the value passes through as-is. ## Why is it important? Fleet could not ingest network flows from Packetbeat. ## Checklist - [x] My code follows the style guidelines of this project - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] I have made corresponding change to the default configuration files - [ ] I have added tests that prove my fix is effective or that my feature works - [x] I have added an entry in `CHANGELOG.next.asciidoc` or `CHANGELOG-developer.next.asciidoc`.