Hi All,
I ship windows events using ListenBeats in NiFi, however some messages arrive truncated. So far I can't identify a pattern why this happens.
what I get on NiFi is this mess:
I can see some non-printable characters in the content of arrived message:
the original event which I can see in Event Viewer has nothing suspicious though:xxx.corp.yyyyyyy.ru
Subject:
Logon Type: 3
New Logon:xxx.corp.yyyyyyy.RU
Event in sequence: 1 of 1
Group Membership:
The subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.
The logon type field indicates the kind of logon that occurred. The most common types are 2 (interactive) and 3 (network).
The New Logon fields indicate the account for whom the new logon was created, i.e. the account that was logged on.
This event is generated when the Audit Group Membership subcategory is configured. The Logon ID field can be used to correlate this event with the corresponding user logon event as well as to any other security audit events generated during this logon session.xxxxxxxxx.ru xxxxxxx.RU
winlogbeat config winlogbeat.yml :
winlogbeat.event_logs:
- name: Application
ignore_older: 168h
fields_under_root: true
fields:
client_name: "region"
tags: [ "application","windows" ]
- name: Security
ignore_older: 168h
fields_under_root: true
fields:
client_name: "region"
tags: [ "security","windows" ]
processors:
- drop_event.when.not.or:
- range.winlog.event_id: { gte: 4608, lte: 4609 }
- equals.winlog.event_id: 4616
- equals.winlog.event_id: 4621
- range.winlog.event_id: { gte: 4624, lte: 4625 }
- equals.winlog.event_id: 4627
- equals.winlog.event_id: 4634
- range.winlog.event_id: { gte: 4647, lte: 4649 }
- equals.winlog.event_id: 4656
- equals.winlog.event_id: 4658
- range.winlog.event_id: { gte: 4662, lte: 4663 }
- range.winlog.event_id: { gte: 4672, lte: 4675 }
- equals.winlog.event_id: 4688
- equals.winlog.event_id: 4696
- range.winlog.event_id: { gte: 4698, lte: 4702 }
- range.winlog.event_id: { gte: 4704, lte: 4707 }
- range.winlog.event_id: { gte: 4713, lte: 4720 }
- range.winlog.event_id: { gte: 4722, lte: 4735 }
# - equals.winlog.event_id: 4724
# - equals.winlog.event_id: 4728
- range.winlog.event_id: { gte: 4737, lte: 4764 }
# - equals.winlog.event_id: 4738
- equals.winlog.event_id: 4767
- range.winlog.event_id: { gte: 4774, lte: 4794 }
- range.winlog.event_id: { gte: 4800, lte: 4803 }
- range.winlog.event_id: { gte: 4864, lte: 4867 }
- equals.winlog.event_id: 4902
- range.winlog.event_id: { gte: 4904, lte: 4908 }
- equals.winlog.event_id: 4912
- equals.winlog.event_id: 4964
- range.winlog.event_id: { gte: 5024, lte: 5025 }
- range.winlog.event_id: { gte: 5027, lte: 5030 }
- range.winlog.event_id: { gte: 5032, lte: 5035 }
- equals.winlog.event_id: 5037
- range.winlog.event_id: { gte: 5058, lte: 5059 }
- range.winlog.event_id: { gte: 5136, lte: 5140 }
- range.winlog.event_id: { gte: 5142, lte: 5144 }
- range.winlog.event_id: { gte: 5148, lte: 5149 }
- equals.winlog.event_id: 5168
- equals.winlog.event_id: 5378
- range.winlog.event_id: { gte: 5632, lte: 5633 }
- equals.winlog.event_id: 6416
- range.winlog.event_id: { gte: 6419, lte: 4624 }
- name: System
ignore_older: 168h
fields_under_root: true
fields:
client_name: "region"
tags: [ "system","windows" ]
output:
logstash:
hosts: ["xxxxxxxxxxxxxxxxx:5044"]
worker: 1
compression_level: 0
bulk_max_size: 1
pipelining: 0
codec.json:
pretty: false
escape_html: true
winlogbeat.shutdown_timeout: 10s
nifi.properties config file:
nifi.flow.configuration.file=./conf/flow.xml.gz
nifi.flow.configuration.archive.enabled=true
nifi.flow.configuration.archive.dir=./conf/archive/
nifi.flow.configuration.archive.max.time=30 days
nifi.flow.configuration.archive.max.storage=500 MB
nifi.flow.configuration.archive.max.count=
nifi.flowcontroller.autoResumeState=true
nifi.flowcontroller.graceful.shutdown.period=10 sec
nifi.flowservice.writedelay.interval=500 ms
nifi.administrative.yield.duration=30 sec
nifi.bored.yield.duration=10 millis
nifi.queue.backpressure.count=10000
nifi.queue.backpressure.size=1 GB
nifi.authorizer.configuration.file=./conf/authorizers.xml
nifi.login.identity.provider.configuration.file=./conf/login-identity-providers.xml
nifi.templates.directory=./conf/templates
nifi.ui.banner.text=
nifi.ui.autorefresh.interval=30 sec
nifi.nar.library.directory=./lib
nifi.nar.library.autoload.directory=./extensions
nifi.nar.working.directory=./work/nar/
nifi.documentation.working.directory=./work/docs/components
nifi.state.management.configuration.file=./conf/state-management.xml
nifi.state.management.provider.local=local-provider
nifi.state.management.provider.cluster=zk-provider
nifi.state.management.embedded.zookeeper.start=false
nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties
nifi.database.directory=./database_repository
nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE
nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
nifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog
nifi.flowfile.repository.directory=./flowfile_repository
nifi.flowfile.repository.partitions=256
nifi.flowfile.repository.checkpoint.interval=2 mins
nifi.flowfile.repository.always.sync=false
nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
nifi.queue.swap.threshold=20000
nifi.swap.in.period=5 sec
nifi.swap.in.threads=1
nifi.swap.out.period=5 sec
nifi.swap.out.threads=4
nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
nifi.content.claim.max.appendable.size=1 MB
nifi.content.claim.max.flow.files=100
nifi.content.repository.directory.default=./content_repository
nifi.content.repository.archive.max.retention.period=12 hours
nifi.content.repository.archive.max.usage.percentage=50%
nifi.content.repository.archive.enabled=true
nifi.content.repository.always.sync=false
nifi.content.viewer.url=../nifi-content-viewer/
nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
nifi.provenance.repository.debug.frequency=1_000_000
nifi.provenance.repository.directory.default=./provenance_repository
nifi.provenance.repository.max.storage.time=24 hours
nifi.provenance.repository.max.storage.size=1 GB
nifi.provenance.repository.rollover.time=30 secs
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=2
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship
nifi.provenance.repository.index.shard.size=500 MB
nifi.provenance.repository.max.attribute.length=65536
nifi.provenance.repository.concurrent.merge.threads=2
nifi.provenance.repository.buffer.size=100000
nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository
nifi.components.status.repository.buffer.size=1440
nifi.components.status.snapshot.frequency=1 min
nifi.remote.input.secure=false
nifi.remote.input.socket.port=
nifi.remote.input.http.enabled=true
nifi.remote.input.http.transaction.ttl=30 sec
nifi.remote.contents.cache.expiration=30 secs
nifi.web.war.directory=./lib
nifi.web.http.host=
nifi.web.http.port=8080
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200
nifi.web.max.header.size=16 KB
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
nifi.sensitive.props.provider=BC
nifi.security.user.authorizer=managed-authorizer
nifi.security.user.oidc.discovery.url=
nifi.security.user.oidc.connect.timeout=5 secs
nifi.security.user.oidc.read.timeout=5 secs
nifi.security.user.knox.cookieName=hadoop-jwt
nifi.cluster.protocol.heartbeat.interval=5 sec
nifi.cluster.protocol.is.secure=false
nifi.cluster.is.node=false
nifi.cluster.node.protocol.threads=10
nifi.cluster.node.protocol.max.threads=50
nifi.cluster.node.event.history.size=25
nifi.cluster.node.connection.timeout=5 sec
nifi.cluster.node.read.timeout=5 sec
nifi.cluster.node.max.concurrent.requests=100
nifi.cluster.flow.election.max.wait.time=5 mins
nifi.cluster.load.balance.host=
nifi.cluster.load.balance.port=6342
nifi.cluster.load.balance.connections.per.node=4
nifi.cluster.load.balance.max.thread.count=8
nifi.cluster.load.balance.comms.timeout=30 sec
nifi.zookeeper.connect.string=
nifi.zookeeper.connect.timeout=3 secs
nifi.zookeeper.session.timeout=3 secs
nifi.zookeeper.root.node=/nifi
nifi.kerberos.spnego.authentication.expiration=12 hours
system
October 28, 2019, 4:07pm
4
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
