Envoyproxy

[mcapua]

Hello,

I'm trying to send envoyproxy logs to SIEM but I'm receiving a WARN message.
ISTIO was configured with stdout access logs and running on Kubernetes

> Finished:false, Fileinfo:(*os.fileStat)(0xc0035c2ea0), Source:"/var/lib/docker/containers/4476e4a217a2b90ccceb9d79e4885a046e26b13a84ae0b9d58129fbdacb6ca1a/4476e4a217a2b90ccceb9d79e4885a046e26b13a84ae0b9d58129fbdacb6ca1a-json.log", Offset:1549114, Timestamp:time.Time{wall:0xbf4ec7b36643d493, ext:5433491068, loc:(*time.Location)(0x30d3480)}, TTL:-1, Type:"docker", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x1653503, Device:0xca01}}, TimeSeries:false}, Flags:0x1} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [http.response.body.bytes] of type [long] in document with id 'Bj0eqmwBx02RG54jRxkG'. Preview of field's value: '\"-\"'","caused_by":{"type":"illegal_argument_exception","reason":"For input string: \"\"-\"\""}}

Here's the configuration file:

data:
  filebeat.yml: |-
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          host: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log
          templates:
            - condition:
                equals:
                  kubernetes.container.name: "istio-proxy"
              config:
                - module: envoyproxy
                  log:
                    input:
                      type: docker
                      #containers.stream: stdout
                      containers.ids:
                        - "${data.kubernetes.container.id}"

Istio format log:

"format": "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% \"%DYNAMIC_METADATA(istio.mixer:status)%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME%\n"

Has anyone set up this module?

[alakahakai]

Can you post the relevant log file text here? It seems that bytes_received field is set to "-", which seems strange. thanks.

I think you will need to remove \"%DYNAMIC_METADATA(istio.mixer:status)%\" from the istio format output.

[johnmario (john)]

To build Envoy with Bazel in a production environment, where the [Envoy dependencies are typically independently sourced, the following steps should be followed:

1. Configure, build and/or install the Envoy dependencies.
2. bazel build -c opt //source/exe:envoy-static from the repository root.

Quick start Bazel build for developers

As a developer convenience, a WORKSPACE and [rules for building a recent version of the various Envoy dependencies are provided. These are provided as is, they are only suitable for development and testing purposes. The specific versions of the Envoy dependencies used in this build may not be up-to-date with the latest security patches. See this doc updating-an-external-dependency-version) for how to update or override dependencies.

1. Install external dependencies libtool, cmake, ninja, realpath and curl libraries separately. On Ubuntu, run the following command:

sudo apt-get install \
   libtool \
   cmake \
   clang-format-8 \
   automake \
   autoconf \
   make \
   ninja-build \
   curl \
   unzip \
   virtualenv

On Fedora (maybe also other red hat distros), run the following:

dnf install cmake libtool libstdc++ ninja-build lld patch aspell-en

On macOS, you'll need to install several dependencies. This can be accomplished via Homebrew

brew install coreutils wget cmake libtool go bazel automake ninja clang-format autoconf aspell

notes : coreutils is used for realpath , gmd5sum and gsha256sum XCode is also required to build Envoy on macOS. Envoy compiles and passes tests with the version of clang installed by XCode 9.3.0: Apple LLVM version 9.1.0 (clang-902.0.30).In order for bazel to be aware of the tools installed by brew, the PATH variable must be set for bazel builds. This can be accomplished by setting this in your $HOME/.bazelrc file:

build --action_env=PATH="/usr/local/bin:/opt/local/bin:/usr/bin:/bin"

Alternatively, you can pass --action_env on the command line when running bazel build / bazel test .Having the binutils keg installed in Brew is known to cause issues due to putting an incompatible version of ar on the PATH, so if you run into issues building third party code like luajit consider uninstalling binutils.
2. Install Golang on your machine. This is required as part of building BoringSSL and also for
Buildifer which is used for formatting bazel BUILD files.
3. go get -u github.com/bazelbuild/buildtools/buildifier to install buildifier. You may need to set BUILDIFIER_BIN to $GOPATH/bin/buildifier in your shell for buildifier to work.
4. bazel build //source/exe:envoy-static from the Envoy source directory.

For more assistance about code or need any support visit Data Recovery Dubai.