# Winlogbeat v7 to NiFi partial messages

**URL:** <https://discuss.elastic.co/t/winlogbeat-v7-to-nifi-partial-messages/201436>\
**Category:** Beats\
**Tags:** winlogbeat\
**Created:** [September 27, 2019, 4:21pm UTC](https://discuss.elastic.co/t/winlogbeat-v7-to-nifi-partial-messages/201436 "2019-09-27T16:21:28Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Pavel201909](https://avatars.discourse-cdn.com/v4/letter/p/ee7513/32.png) [@Pavel201909](https://discuss.elastic.co/u/Pavel201909)\
**Post date:** [September 27, 2019, 4:21pm UTC](https://discuss.elastic.co/t/winlogbeat-v7-to-nifi-partial-messages/201436/1 "2019-09-27T16:21:29Z")

</div>

Hi All,

I ship windows events using ListenBeats in NiFi, however some messages arrive truncated. So far I can't identify a pattern why this happens.

what I get on NiFi is this mess:

 ![err1](https://us1.discourse-cdn.com/elastic/original/3X/5/0/509658112594cfb247a41476da77ca8a80857df8.jpeg)

I can see some non-printable characters in the content of arrived message:

 ![err2](https://us1.discourse-cdn.com/elastic/original/3X/9/f/9fd25b03f089e2c19e29b34244c18519a12b8b1f.jpeg)  
 ![3](https://us1.discourse-cdn.com/elastic/original/3X/7/8/7873a26de4d4451b302986b2861dd1d23ccee75f.jpeg)

the original event which I can see in Event Viewer has nothing suspicious though:  
Log Name: Security  
Source: Microsoft-Windows-Security-Auditing  
Date: 9/26/2019 11:05:45 PM  
Event ID: 4627  
Task Category: Group Membership  
Level: Information  
Keywords: Audit Success  
User: N/A  
Computer: [xxx.corp.yyyyyyy.ru](http://xxx.corp.yyyyyyy.ru)  
Description:  
Group membership information.

Subject:  
Security ID: NULL SID  
Account Name: -  
Account Domain: -  
Logon ID: 0x0

Logon Type: 3

New Logon:  
Security ID: SYSTEM  
Account Name: zzzzzz01$  
Account Domain: [xxx.corp.yyyyyyy.RU](http://xxx.corp.yyyyyyy.RU)  
Logon ID: 0x14303B0E

Event in sequence: 1 of 1

Group Membership:   
BUILTIN\Administrators  
Everyone  
BUILTIN\Pre-Windows 2000 Compatible Access  
BUILTIN\Users  
BUILTIN\Windows Authorization Access Group  
NT AUTHORITY\NETWORK  
NT AUTHORITY\Authenticated Users  
NT AUTHORITY\This Organization  
CORP\zzzzz01$  
CORP\Domain Controllers  
NT AUTHORITY\ENTERPRISE DOMAIN CONTROLLERS  
Authentication authority asserted identity  
CORP\Denied RODC Password Replication Group  
CORP\RAS and IAS Servers  
Mandatory Label\System Mandatory Level

The subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.

The logon type field indicates the kind of logon that occurred. The most common types are 2 (interactive) and 3 (network).

The New Logon fields indicate the account for whom the new logon was created, i.e. the account that was logged on.

This event is generated when the Audit Group Membership subcategory is configured. The Logon ID field can be used to correlate this event with the corresponding user logon event as well as to any other security audit events generated during this logon session.  
Event Xml:  
  
  
  
4627  
0  
0  
12554  
0  
0x8020000000000000  
  
20279178  
  
  
Security  
[xxxxxxxxx.ru](http://xxxxxxxxx.ru)  
  
  
  
S-1-0-0  
-  
-  
0x0  
S-1-5-18  
DC01$  
[xxxxxxx.RU](http://xxxxxxx.RU)  
0x14303b0e  
3  
1  
1  
  
%{S-1-5-32-544}  
%{S-1-1-0}  
%{S-1-5-32-554}  
%{S-1-5-32-545}  
%{S-1-5-32-560}  
%{S-1-5-2}  
%{S-1-5-11}  
%{S-1-5-15}  
%{S-1-5-21-900100797-3563623125-4166005187-1000}  
%{S-1-5-21-900100797-3563623125-4166005187-516}  
%{S-1-5-9}  
%{S-1-18-1}  
%{S-1-5-21-900100797-3563623125-4166005187-572}  
%{S-1-5-21-900100797-3563623125-4166005187-553}  
%{S-1-16-16384}

---

<div class="post-metadata">

**Author:** ![Pavel201909](https://avatars.discourse-cdn.com/v4/letter/p/ee7513/32.png) [@Pavel201909](https://discuss.elastic.co/u/Pavel201909)\
**Post date:** [September 28, 2019, 7:19pm UTC](https://discuss.elastic.co/t/winlogbeat-v7-to-nifi-partial-messages/201436/2 "2019-09-28T19:19:34Z")

</div>

winlogbeat config **winlogbeat.yml** :

```
winlogbeat.event_logs:
  - name: Application
    ignore_older: 168h
    fields_under_root: true
    fields:
      client_name: "region"
    tags: ["application","windows"]
  - name: Security
    ignore_older: 168h
    fields_under_root: true
    fields:
      client_name: "region"
    tags: ["security","windows"]
    processors:
      - drop_event.when.not.or:
        - range.winlog.event_id: { gte: 4608, lte: 4609 }
        - equals.winlog.event_id: 4616
        - equals.winlog.event_id: 4621
        - range.winlog.event_id: { gte: 4624, lte: 4625 }
        - equals.winlog.event_id: 4627
        - equals.winlog.event_id: 4634
        - range.winlog.event_id: { gte: 4647, lte: 4649 }
        - equals.winlog.event_id: 4656
        - equals.winlog.event_id: 4658
        - range.winlog.event_id: { gte: 4662, lte: 4663 }
        - range.winlog.event_id: { gte: 4672, lte: 4675 }
        - equals.winlog.event_id: 4688
        - equals.winlog.event_id: 4696
        - range.winlog.event_id: { gte: 4698, lte: 4702 }
        - range.winlog.event_id: { gte: 4704, lte: 4707 }
        - range.winlog.event_id: { gte: 4713, lte: 4720 }
        - range.winlog.event_id: { gte: 4722, lte: 4735 }
# - equals.winlog.event_id: 4724
# - equals.winlog.event_id: 4728
        - range.winlog.event_id: { gte: 4737, lte: 4764 }
# - equals.winlog.event_id: 4738
        - equals.winlog.event_id: 4767
        - range.winlog.event_id: { gte: 4774, lte: 4794 }
        - range.winlog.event_id: { gte: 4800, lte: 4803 }
        - range.winlog.event_id: { gte: 4864, lte: 4867 }
        - equals.winlog.event_id: 4902
        - range.winlog.event_id: { gte: 4904, lte: 4908 }
        - equals.winlog.event_id: 4912
        - equals.winlog.event_id: 4964
        - range.winlog.event_id: { gte: 5024, lte: 5025 }
        - range.winlog.event_id: { gte: 5027, lte: 5030 }
        - range.winlog.event_id: { gte: 5032, lte: 5035 }
        - equals.winlog.event_id: 5037
        - range.winlog.event_id: { gte: 5058, lte: 5059 }
        - range.winlog.event_id: { gte: 5136, lte: 5140 }
        - range.winlog.event_id: { gte: 5142, lte: 5144 }
        - range.winlog.event_id: { gte: 5148, lte: 5149 }
        - equals.winlog.event_id: 5168
        - equals.winlog.event_id: 5378
        - range.winlog.event_id: { gte: 5632, lte: 5633 }
        - equals.winlog.event_id: 6416
        - range.winlog.event_id: { gte: 6419, lte: 4624 }
  - name: System
    ignore_older: 168h
    fields_under_root: true
    fields:
      client_name: "region"
    tags: ["system","windows"]

output:
  logstash:
    hosts: ["xxxxxxxxxxxxxxxxx:5044"]
    worker: 1
    compression_level: 0
    bulk_max_size: 1
    pipelining: 0
    codec.json:
      pretty: false
      escape_html: true

winlogbeat.shutdown_timeout: 10s
```

---

<div class="post-metadata">

**Author:** ![Pavel201909](https://avatars.discourse-cdn.com/v4/letter/p/ee7513/32.png) [@Pavel201909](https://discuss.elastic.co/u/Pavel201909)\
**Post date:** [September 30, 2019, 4:07pm UTC](https://discuss.elastic.co/t/winlogbeat-v7-to-nifi-partial-messages/201436/3 "2019-09-30T16:07:10Z")

</div>

nifi.properties config file:

```
nifi.flow.configuration.file=./conf/flow.xml.gz
nifi.flow.configuration.archive.enabled=true
nifi.flow.configuration.archive.dir=./conf/archive/
nifi.flow.configuration.archive.max.time=30 days
nifi.flow.configuration.archive.max.storage=500 MB
nifi.flow.configuration.archive.max.count=
nifi.flowcontroller.autoResumeState=true
nifi.flowcontroller.graceful.shutdown.period=10 sec
nifi.flowservice.writedelay.interval=500 ms
nifi.administrative.yield.duration=30 sec

nifi.bored.yield.duration=10 millis
nifi.queue.backpressure.count=10000
nifi.queue.backpressure.size=1 GB

nifi.authorizer.configuration.file=./conf/authorizers.xml
nifi.login.identity.provider.configuration.file=./conf/login-identity-providers.xml
nifi.templates.directory=./conf/templates
nifi.ui.banner.text=
nifi.ui.autorefresh.interval=30 sec
nifi.nar.library.directory=./lib
nifi.nar.library.autoload.directory=./extensions
nifi.nar.working.directory=./work/nar/
nifi.documentation.working.directory=./work/docs/components

nifi.state.management.configuration.file=./conf/state-management.xml

nifi.state.management.provider.local=local-provider

nifi.state.management.provider.cluster=zk-provider

nifi.state.management.embedded.zookeeper.start=false

nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties

nifi.database.directory=./database_repository
nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE

nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
nifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog
nifi.flowfile.repository.directory=./flowfile_repository
nifi.flowfile.repository.partitions=256
nifi.flowfile.repository.checkpoint.interval=2 mins
nifi.flowfile.repository.always.sync=false

nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
nifi.queue.swap.threshold=20000
nifi.swap.in.period=5 sec
nifi.swap.in.threads=1
nifi.swap.out.period=5 sec
nifi.swap.out.threads=4

nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
nifi.content.claim.max.appendable.size=1 MB
nifi.content.claim.max.flow.files=100
nifi.content.repository.directory.default=./content_repository
nifi.content.repository.archive.max.retention.period=12 hours
nifi.content.repository.archive.max.usage.percentage=50%
nifi.content.repository.archive.enabled=true
nifi.content.repository.always.sync=false
nifi.content.viewer.url=../nifi-content-viewer/

nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
nifi.provenance.repository.debug.frequency=1_000_000
nifi.provenance.repository.directory.default=./provenance_repository
nifi.provenance.repository.max.storage.time=24 hours
nifi.provenance.repository.max.storage.size=1 GB
nifi.provenance.repository.rollover.time=30 secs
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=2
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false

nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship

nifi.provenance.repository.index.shard.size=500 MB

nifi.provenance.repository.max.attribute.length=65536
nifi.provenance.repository.concurrent.merge.threads=2

nifi.provenance.repository.buffer.size=100000

nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository
nifi.components.status.repository.buffer.size=1440
nifi.components.status.snapshot.frequency=1 min

nifi.remote.input.secure=false
nifi.remote.input.socket.port=
nifi.remote.input.http.enabled=true
nifi.remote.input.http.transaction.ttl=30 sec
nifi.remote.contents.cache.expiration=30 secs

nifi.web.war.directory=./lib
nifi.web.http.host=
nifi.web.http.port=8080
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200
nifi.web.max.header.size=16 KB
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
nifi.sensitive.props.provider=BC
nifi.security.user.authorizer=managed-authorizer

nifi.security.user.oidc.discovery.url=
nifi.security.user.oidc.connect.timeout=5 secs
nifi.security.user.oidc.read.timeout=5 secs
nifi.security.user.knox.cookieName=hadoop-jwt

nifi.cluster.protocol.heartbeat.interval=5 sec
nifi.cluster.protocol.is.secure=false

nifi.cluster.is.node=false
nifi.cluster.node.protocol.threads=10
nifi.cluster.node.protocol.max.threads=50
nifi.cluster.node.event.history.size=25
nifi.cluster.node.connection.timeout=5 sec
nifi.cluster.node.read.timeout=5 sec
nifi.cluster.node.max.concurrent.requests=100
nifi.cluster.flow.election.max.wait.time=5 mins

nifi.cluster.load.balance.host=
nifi.cluster.load.balance.port=6342
nifi.cluster.load.balance.connections.per.node=4
nifi.cluster.load.balance.max.thread.count=8
nifi.cluster.load.balance.comms.timeout=30 sec

nifi.zookeeper.connect.string=
nifi.zookeeper.connect.timeout=3 secs
nifi.zookeeper.session.timeout=3 secs
nifi.zookeeper.root.node=/nifi

nifi.kerberos.spnego.authentication.expiration=12 hours
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 28, 2019, 4:07pm UTC](https://discuss.elastic.co/t/winlogbeat-v7-to-nifi-partial-messages/201436/4 "2019-10-28T16:07:24Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.