ELK - 8.6 (Filebeat to logstash) - only write ops with an op_type of create are allowed in data streams

Kvoyce2023 · March 23, 2023, 11:40pm

Hello everyone:

I got 2 physical server - one got elastic and 2nd one got Logstash & Kibana.
I got filebeat running on our customer server from where I am harvesting data from the application log.

Below is the error what I get on logstash-plain.log with "Could not index event to Elasticsearch" message.

[2023-03-24T07:04:25,618][WARN ][logstash.outputs.elasticsearch][main][e7986997653af8b09b27dcd506b8f086e304591f5cf2313247788405fd9a9ac4] Could not index event to Elasticsearch. status: 400, action: ["index", {:_id=>nil, :_index=>"filebeat-2023.03.22", :routing=>nil}, {"ecs"=>{"version"=>"8.0.0"}, "@version"=>"1", "event"=>{"original"=>"2023-03-22 04:37:13,429\t1563725138\tNOTICE\t3              \tSystem.MicroservicesUseHttps, securityType:None, tags:\t[(null)]\tConfig.Infrastructure"}, "message"=>"3              \tSystem.MicroservicesUseHttps, securityType:None, tags:\t[(null)]\tConfig.Infrastructure", "debug"=>"timestamp_matched", "@timestamp"=>2023-03-22T08:37:13.429Z, "curdate"=>"2023-03-22 04:37:13,429", "level"=>"NOTICE", "log"=>{"file"=>{"path"=>"C:\\ProgramData\\app1\\Logs\\app1.SupportLog.txt"}, "offset"=>790758}, "host"=>{"id"=>"3fd8fca3-45f8-42c1-a539-cebdd7816035", "mac"=>["00-50-56-B2-BE-21"], "ip"=>["10.111.32.4"], "os"=>{"family"=>"windows", "version"=>"10.0", "name"=>"Windows Server 2016 Standard", "kernel"=>"10.0.14393.5717 (rs1_release.230203-1742)", "build"=>"14393.5717", "platform"=>"windows", "type"=>"windows"}, "name"=>"app01", "architecture"=>"x86_64", "hostname"=>"app01"}, "agent"=>{"id"=>"bec165c5-d9b2-4a1c-91e5-399671c5b3d3", "name"=>"app01", "type"=>"filebeat", "version"=>"8.6.2", "ephemeral_id"=>"e99a8777-81a2-47d8-a834-2028f73ff43a"}, "fields"=>{"log_type"=>"app204", "customer_name"=>"conyers", "log_subtype"=>"Config"}, "tags"=>["filebeat", "beats_input_codec_plain_applied"], "input"=>{"type"=>"log"}}], response: {"index"=>{"_index"=>"filebeat-2023.03.22", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"only write ops with an op_type of create are allowed in data streams"}}}

Please advise how to rectify the error ? Thank you.

stephenb · March 24, 2023, 12:21am

Hi @Kvoyce2023 welcome to the community!

Share your filebeat and logstash pipeline config.

Short answer your logstash output section should probably look like what is on this page

Particularly note

action => "create"

Kvoyce2023 · March 24, 2023, 1:20am

Hi Stephen:

Here are the requested file content. But as soon as I add action => "create", I did not got any data on logstash.

filebeatpipeline.conf file

output{
  if "filebeat" in [tags] {
   elasticsearch  {
      hosts => ["http://10.111.206.4:9200"]
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      document_type => "%{[@metadata][type]}"
      user => "elastic"
      password => "hidden"      
      ssl_certificate_verification => false
    }
  }
}

filebeat.yml

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

# For more available modules and options, please see the filebeat.reference.yml sample
# configuration file.

# ============================== Filebeat inputs ===============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

# filestream is an input for collecting log messages from files.
- type: log

  # Unique ID among all inputs, an ID is required.
  id: my-filestream-id

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    #- /var/log/*.log
    - C:\ProgramData\app1\Logs\app1.SupportLog.txt
   
  fields:
    log_type: app01
    log_subtype: Config
    customer_name: ${customer:cust} 
    
    ### Multiline options
  multiline.pattern: \d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}
  multiline.negate: true
  multiline.match: after
  
  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  # Line filtering happens after the parsers pipeline. If you would like to filter lines
  # before parsers, use include_message parser.
  #exclude_lines: ['^DBG']

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  # Line filtering happens after the parsers pipeline. If you would like to filter lines
  # before parsers, use include_message parser.
  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #prospector.scanner.exclude_files: ['.gz$']

  # Optional additional fields. These fields can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  #  level: debug
  #  review: 1

# ============================== Filebeat modules ==============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false


# ================================== General ===================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  #host: "localhost:5601"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

# =============================== Elastic Cloud ================================

# These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  #username: "elastic"
  #password: "changeme"

# ------------------------------ Logstash Output -------------------------------
output.logstash:
  # The Logstash hosts
    hosts: ["10.111.206.5:5046"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

# ================================== Logging ===================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publisher", "service".
#logging.selectors: ["*"]

# ============================= X-Pack Monitoring ==============================
# Filebeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#monitoring.enabled: false

# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Filebeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
#monitoring.cluster_uuid:

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:

# ============================== Instrumentation ===============================

# Instrumentation support for the filebeat.
#instrumentation:
    # Set to true to enable instrumentation of filebeat.
    #enabled: false

    # Environment in which filebeat is running on (eg: staging, production, etc.)
    #environment: ""

    # APM Server hosts to report instrumentation results to.
    #hosts:
    #  - http://localhost:8200

    # API Key for the APM Server(s).
    # If api_key is set then secret_token will be ignored.
    #api_key:

    # Secret token for the APM Server(s).
    #secret_token:


# ================================= Migration ==================================

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true

stephenb · March 24, 2023, 1:28am

and Per the docs I provided

output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => "https://061ab24010a2482e9d64729fdb0fd93a.us-east-1.aws.found.io:9243"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}" 
      action => "create"  <!--- CRITICAL
      pipeline => "%{[@metadata][pipeline]}" 
      user => "elastic"
      password => "secret"
    }
  } else {
    elasticsearch {
      hosts => "https://061ab24010a2482e9d64729fdb0fd93a.us-east-1.aws.found.io:9243"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}" 
      action => "create" <!--- CRITICAL
      user => "elastic"
      password => "secret"
    }
  }
}

Kvoyce2023 · March 24, 2023, 1:42am

As I mentioned earlier, I did tried action => "create" - but then no data was pushed to logstash.

stephenb · March 24, 2023, 1:48am

Apologies but I don't know what you mean ....

That setting goes in your filebeatpipeline.conf
That is the exact reason for your error message that you posted at the top which comes from elasticsearch.

That setting has nothing to do with filebeat sending to Logstash.

If something is still not working, please post the complete logs from both logstash and filebeat. We can't help without more information.

Also, do you know that filebeat will not reread a file that is already read?

Do You know about deleting the filebeat registry so that you can reread the same file?

You need to do that if you want to debug and run the same file over and over.

Depending on how you installed file beat, look up the registry location in the directory layout and clean it up.

Kvoyce2023 · March 24, 2023, 3:04am

That setting goes in your filebeatpipeline.conf

yes, I have added "action=>"create" in output.logstash section in filebeatpipeline.conf only.

Do You know about deleting the filebeat registry so that you can reread the same file?

yes, I have stopped filebeat svc, removed the log.json file from the filebeat\data\registry folder and then started the filebeat svc to reread the file. Those portions were working, until I added "action=>"create" in output.logstash section in filebeatpipeline.conf

Below are the logstash and filebeat log file content :

logstash.log

[2023-03-24T13:48:14,153][INFO ][logstash.runner          ] Log4j configuration path used is: C:\elastic\logstash-8.6.2-windows-x86_64\logstash-8.6.2\config\log4j2.properties
[2023-03-24T13:48:14,168][WARN ][logstash.runner          ] The use of JAVA_HOME has been deprecated. Logstash 8.0 and later ignores JAVA_HOME and uses the bundled JDK. Running Logstash with the bundled JDK is recommended. The bundled JDK has been verified to work with each specific version of Logstash, and generally provides best performance and reliability. If you have compelling reasons for using your own JDK (organizational-specific compliance requirements, for example), you can configure LS_JAVA_HOME to use that version instead.
[2023-03-24T13:48:14,168][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.6.2", "jruby.version"=>"jruby 9.3.10.0 (2.6.8) 2023-02-01 107b2e6697 OpenJDK 64-Bit Server VM 17.0.6+10 on 17.0.6+10 +indy +jit [x86_64-mswin32]"}
[2023-03-24T13:48:14,168][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2023-03-24T13:48:14,200][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2023-03-24T13:48:15,726][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2023-03-24T13:48:16,510][INFO ][org.reflections.Reflections] Reflections took 164 ms to scan 1 urls, producing 127 keys and 444 values
[2023-03-24T13:48:17,678][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "document_type" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document types are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"document_type", :plugin=><LogStash::Outputs::ElasticSearch ssl_certificate_verification=>false, password=><password>, hosts=>[http://10.111.206.4:9200], index=>"%{[@metadata][beat]}-%{[@metadata][version]}", manage_template=>false, id=>"47bb8b719f702eabfa4d33d71f82031a0765b04d6a633b2d54f5908eb057cbd2", user=>"elastic", document_type=>"%{[@metadata][type]}", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_c72263c3-3356-406b-a92c-b9ff88751ae5", enable_metric=>true, charset=>"UTF-8">, workers=>1, sniffing=>false, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false, retry_initial_interval=>2, retry_max_interval=>64, data_stream_type=>"logs", data_stream_dataset=>"generic", data_stream_namespace=>"default", data_stream_sync_fields=>true, data_stream_auto_routing=>true, template_overwrite=>false, template_api=>"auto", doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_on_conflict=>1, ilm_enabled=>"auto", ilm_pattern=>"{now/d}-000001", ilm_policy=>"logstash-policy", dlq_on_failed_indexname_interpolation=>true>}
[2023-03-24T13:48:17,688][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2023-03-24T13:48:17,724][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://10.111.206.4:9200"]}
[2023-03-24T13:48:17,831][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:xxxxxx@10.111.206.4:9200/]}}
[2023-03-24T13:48:17,907][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://elastic:xxxxxx@10.111.206.4:9200/"}
[2023-03-24T13:48:17,911][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.6.2) {:es_version=>8}
[2023-03-24T13:48:17,918][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[2023-03-24T13:48:17,926][INFO ][logstash.outputs.elasticsearch][main] Not eligible for data streams because config contains one or more settings that are not compatible with data streams: {"index"=>"%{[@metadata][beat]}-%{[@metadata][version]}", "document_type"=>"%{[@metadata][type]}"}
[2023-03-24T13:48:17,927][INFO ][logstash.outputs.elasticsearch][main] Data streams auto configuration (`data_stream => auto` or unset) resolved to `false`
[2023-03-24T13:48:17,938][WARN ][logstash.outputs.elasticsearch][main] Elasticsearch Output configured with `ecs_compatibility => v8`, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning.
[2023-03-24T13:48:17,984][WARN ][logstash.filters.grok    ][main] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2023-03-24T13:48:18,046][WARN ][logstash.filters.grok    ][main] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2023-03-24T13:48:18,132][WARN ][logstash.filters.grok    ][main] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2023-03-24T13:48:18,167][WARN ][logstash.filters.grok    ][main] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2023-03-24T13:48:18,204][WARN ][logstash.filters.grok    ][main] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2023-03-24T13:48:18,236][WARN ][logstash.filters.grok    ][main] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2023-03-24T13:48:18,270][WARN ][logstash.filters.grok    ][main] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2023-03-24T13:48:18,361][WARN ][logstash.filters.grok    ][main] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2023-03-24T13:48:18,383][WARN ][logstash.filters.grok    ][main] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2023-03-24T13:48:18,405][WARN ][logstash.filters.grok    ][main] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2023-03-24T13:48:18,426][WARN ][logstash.filters.grok    ][main] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2023-03-24T13:48:18,447][WARN ][logstash.filters.grok    ][main] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2023-03-24T13:48:18,467][WARN ][logstash.filters.grok    ][main] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2023-03-24T13:48:18,487][WARN ][logstash.filters.grok    ][main] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2023-03-24T13:48:18,836][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["C:/elastic/logstash-8.6.2-windows-x86_64/logstash-8.6.2/config/filebeatlogstash.conf"], :thread=>"#<Thread:0x209f77f@C:/elastic/logstash-8.6.2-windows-x86_64/logstash-8.6.2/logstash-core/lib/logstash/java_pipeline.rb:131 run>"}
[2023-03-24T13:48:22,057][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>3.22}
[2023-03-24T13:48:22,059][INFO ][logstash.inputs.beats    ][main] Starting input listener {:address=>"10.111.206.5:5046"}
[2023-03-24T13:48:22,065][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2023-03-24T13:48:22,107][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2023-03-24T13:48:22,271][INFO ][org.logstash.beats.Server][main][a177e3d0612db9d19fc8e6f8acb6ab4f9428e7fcf3f2fddbbd7b4c807768716e] Starting server on port: 5046
[2023-03-24T13:48:48,419][WARN ][logstash.outputs.elasticsearch][main][47bb8b719f702eabfa4d33d71f82031a0765b04d6a633b2d54f5908eb057cbd2] Could not index event to Elasticsearch. status: 400, action: ["index", {:_id=>nil, :_index=>"filebeat-8.6.2", :routing=>nil}, {"level"=>"NOTICE", "log"=>{"offset"=>333, "file"=>{"path"=>"C:\\ProgramData\\Nexidia\\Analytics\\Logs\\Nexidia.Config.SupportLog.txt"}}, "input"=>{"type"=>"log"}, "message"=>"Main           \tIn RemotableConfigService ctor, initialLoad:True.\t[(null)]\tConfig.Infrastructure", "debug"=>"timestamp_matched", "@timestamp"=>2020-12-18T10:38:32.129Z, "event"=>{"original"=>"2020-12-18 05:38:32,129\t1352    \tNOTICE\tMain           \tIn RemotableConfigService ctor, initialLoad:True.\t[(null)]\tConfig.Infrastructure"}, "@version"=>"1", "host"=>{"mac"=>["00-50-56-B2-BE-21"], "architecture"=>"x86_64", "ip"=>["10.111.32.4"], "name"=>"app01", "os"=>{"kernel"=>"10.0.14393.5717 (rs1_release.230203-1742)", "family"=>"windows", "platform"=>"windows", "name"=>"Windows Server 2016 Standard", "version"=>"10.0", "type"=>"windows", "build"=>"14393.5717"}, "hostname"=>"app01", "id"=>"3fd8fca3-45f8-42c1-a539-cebdd7816035"}, "tags"=>["filebeat", "beats_input_codec_plain_applied"], "ecs"=>{"version"=>"8.0.0"}, "fields"=>{"customer_name"=>"conyer", "log_subtype"=>"Config", "log_type"=>"app204"}, "agent"=>{"id"=>"bec165c5-d9b2-4a1c-91e5-399671c5b3d3", "type"=>"filebeat", "name"=>"app01", "version"=>"8.6.2", "ephemeral_id"=>"94503edc-ba48-4c8c-ba4b-b1cad2c75753"}, "curdate"=>"2020-12-18 05:38:32,129"}], response: {"index"=>{"_index"=>"filebeat-8.6.2", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"only write ops with an op_type of create are allowed in data streams"}}}

Kvoyce2023 · March 24, 2023, 3:09am

filebeat.log

{"log.level":"info","@timestamp":"2023-03-24T13:48:40.640+1100","log.origin":{"file.name":"instance/beat.go","file.line":724},"message":"Home path: [C:\\beats\\filebeat] Config path: [C:\\beats\\filebeat] Data path: [C:\\ProgramData\\filebeat] Logs path: [C:\\ProgramData\\filebeat\\logs]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-24T13:48:40.641+1100","log.origin":{"file.name":"instance/beat.go","file.line":732},"message":"Beat ID: bec165c5-d9b2-4a1c-91e5-399671c5b3d3","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-03-24T13:48:43.655+1100","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":81},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-24T13:48:43.656+1100","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1096},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"C:\\beats\\filebeat","data":"C:\\ProgramData\\filebeat","home":"C:\\beats\\filebeat","logs":"C:\\ProgramData\\filebeat\\logs"},"type":"filebeat","uuid":"bec165c5-d9b2-4a1c-91e5-399671c5b3d3"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-24T13:48:43.657+1100","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1105},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"9b77c2c135c228c2eedc310f6e975bb1a76169b1","libbeat":"8.6.2","time":"2023-02-12T04:37:19.000Z","version":"8.6.2"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-24T13:48:43.657+1100","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1108},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"windows","arch":"amd64","max_procs":16,"version":"go1.18.10"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-24T13:48:43.665+1100","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1112},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2023-03-04T02:17:36+11:00","name":"app01","ip":["10.111.32.4/24","::1/128","127.0.0.1/8"],"kernel_version":"10.0.14393.5717 (rs1_release.230203-1742)","mac":["00:50:56:b2:be:21"],"os":{"type":"windows","family":"windows","platform":"windows","name":"Windows Server 2016 Standard","version":"10.0","major":10,"minor":0,"patch":0,"build":"14393.5717"},"timezone":"AEDT","timezone_offset_sec":39600,"id":"3fd8fca3-45f8-42c1-a539-cebdd7816035"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-24T13:48:43.665+1100","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1141},"message":"Process info","service.name":"filebeat","system_info":{"process":{"cwd":"C:\\Windows\\system32","exe":"C:\\beats\\filebeat\\filebeat.exe","name":"filebeat.exe","pid":17512,"ppid":868,"start_time":"2023-03-24T13:48:40.498+1100"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-24T13:48:43.666+1100","log.origin":{"file.name":"instance/beat.go","file.line":296},"message":"Setup Beat: filebeat; Version: 8.6.2","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-24T13:48:44.997+1100","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: app01","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-24T13:48:44.997+1100","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":120},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-03-24T13:48:45.000+1100","log.origin":{"file.name":"beater/filebeat.go","file.line":164},"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-24T13:48:45.000+1100","log.origin":{"file.name":"instance/beat.go","file.line":486},"message":"filebeat start running.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-24T13:48:45.000+1100","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":145},"message":"Starting metrics logging every 30s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-24T13:48:45.002+1100","log.origin":{"file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for 'C:\\ProgramData\\filebeat\\registry\\filebeat'. Active transaction id=0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-03-24T13:48:45.003+1100","log.origin":{"file.name":"beater/filebeat.go","file.line":290},"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-24T13:48:45.003+1100","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":109},"message":"States Loaded from registrar: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-24T13:48:45.003+1100","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":71},"message":"Loading Inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-24T13:48:45.003+1100","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":117},"message":"starting input, keys present on the config: [filebeat.inputs.0.enabled filebeat.inputs.0.fields.customer_name filebeat.inputs.0.fields.log_subtype filebeat.inputs.0.fields.log_type filebeat.inputs.0.id filebeat.inputs.0.multiline.match filebeat.inputs.0.multiline.negate filebeat.inputs.0.multiline.pattern filebeat.inputs.0.paths.0 filebeat.inputs.0.type]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-03-24T13:48:45.004+1100","log.logger":"cfgwarn","log.origin":{"file.name":"log/input.go","file.line":90},"message":"DEPRECATED: Log input. Use Filestream input instead.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-24T13:48:45.004+1100","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":172},"message":"Configured paths: [C:\\ProgramData\\app01\\logs\\SupportLog.txt]","service.name":"filebeat","input_id":"942082e1-8649-41bd-a2ce-d33037a7c09b","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-24T13:48:45.004+1100","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":148},"message":"Starting input (ID: 3899641480751316770)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-24T13:48:45.004+1100","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":106},"message":"Loading and starting Inputs completed. Enabled inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-24T13:48:45.004+1100","log.origin":{"file.name":"cfgfile/reload.go","file.line":164},"message":"Config reloader started","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-24T13:48:45.005+1100","log.logger":"input.harvester","log.origin":{"file.name":"log/harvester.go","file.line":310},"message":"Harvester started for paths: [C:\\ProgramData\\app01\\logs\\SupportLog.txt]","service.name":"filebeat","input_id":"942082e1-8649-41bd-a2ce-d33037a7c09b","source_file":"C:\\ProgramData\\app01\\logs\\SupportLog.txt","state_id":"native::786432-133237-304127642","finished":false,"os_id":"786432-133237-304127642","harvester_id":"fe495d11-1488-421b-b27a-3a37cd045849","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-24T13:48:45.005+1100","log.origin":{"file.name":"cfgfile/reload.go","file.line":224},"message":"Loading of config files completed.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-24T13:48:46.663+1100","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":102},"message":"add_cloud_metadata: hosting provider type not detected.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-24T13:48:47.678+1100","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":139},"message":"Connecting to backoff(async(tcp://10.111.206.5:5046))","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-24T13:48:47.679+1100","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":147},"message":"Connection to backoff(async(tcp://10.111.206.5:5046)) established","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-24T13:49:15.006+1100","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":140,"time":{"ms":140}},"total":{"ticks":374,"time":{"ms":374},"value":374},"user":{"ticks":234,"time":{"ms":234}}},"info":{"ephemeral_id":"94503edc-ba48-4c8c-ba4b-b1cad2c75753","name":"filebeat","uptime":{"ms":34457},"version":"8.6.2"},"memstats":{"gc_next":30876520,"memory_alloc":25304072,"memory_sys":43302520,"memory_total":80208968,"rss":65912832},"runtime":{"goroutines":39}},"filebeat":{"events":{"active":0,"added":1955,"done":1955},"harvester":{"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0},"reloads":1,"scans":1},"output":{"events":{"acked":1954,"active":0,"batches":2,"total":1954},"read":{"bytes":18},"type":"logstash","write":{"bytes":201616}},"pipeline":{"clients":1,"events":{"active":0,"filtered":1,"published":1954,"retry":1953,"total":1955},"queue":{"acked":1954,"max_events":4096}}},"registrar":{"states":{"current":1,"update":1955},"writes":{"success":2,"total":2}},"system":{"cpu":{"cores":16},"handles":{"open":249}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-24T13:49:45.007+1100","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":140},"total":{"ticks":374,"value":374},"user":{"ticks":234}},"info":{"ephemeral_id":"94503edc-ba48-4c8c-ba4b-b1cad2c75753","uptime":{"ms":64455},"version":"8.6.2"},"memstats":{"gc_next":30876520,"memory_alloc":25401544,"memory_sys":262144,"memory_total":80306440,"rss":65953792},"runtime":{"goroutines":39}},"filebeat":{"events":{"active":0},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"handles":{"open":2}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-24T13:50:15.010+1100","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":156,"time":{"ms":16}},"total":{"ticks":390,"time":{"ms":16},"value":390},"user":{"ticks":234}},"info":{"ephemeral_id":"94503edc-ba48-4c8c-ba4b-b1cad2c75753","uptime":{"ms":94455},"version":"8.6.2"},"memstats":{"gc_next":30876520,"memory_alloc":25487808,"memory_total":80392704,"rss":65998848},"runtime":{"goroutines":39}},"filebeat":{"events":{"active":0},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"handles":{"open":2}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-24T13:50:45.016+1100","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":156},"total":{"ticks":390,"value":390},"user":{"ticks":234}},"info":{"ephemeral_id":"94503edc-ba48-4c8c-ba4b-b1cad2c75753","uptime":{"ms":124457},"version":"8.6.2"},"memstats":{"gc_next":30876520,"memory_alloc":25568920,"memory_total":80473816,"rss":66023424},"runtime":{"goroutines":39}},"filebeat":{"events":{"active":0},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-24T13:51:15.031+1100","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":156},"total":{"ticks":390,"value":390},"user":{"ticks":234}},"info":{"ephemeral_id":"94503edc-ba48-4c8c-ba4b-b1cad2c75753","uptime":{"ms":154469},"version":"8.6.2"},"memstats":{"gc_next":35680840,"memory_alloc":17234384,"memory_total":80554160,"rss":66195456},"runtime":{"goroutines":39}},"filebeat":{"events":{"active":0},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"handles":{"open":2}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-24T13:51:45.033+1100","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":156},"total":{"ticks":390,"value":390},"user":{"ticks":234}},"info":{"ephemeral_id":"94503edc-ba48-4c8c-ba4b-b1cad2c75753","uptime":{"ms":184468},"version":"8.6.2"},"memstats":{"gc_next":35680840,"memory_alloc":17322616,"memory_total":80642392,"rss":66207744},"runtime":{"goroutines":39}},"filebeat":{"events":{"active":0},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"handles":{"open":-1}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-24T13:52:15.024+1100","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":156},"total":{"ticks":390,"value":390},"user":{"ticks":234}},"info":{"ephemeral_id":"94503edc-ba48-4c8c-ba4b-b1cad2c75753","uptime":{"ms":214456},"version":"8.6.2"},"memstats":{"gc_next":35680840,"memory_alloc":17400696,"memory_total":80720472,"rss":66211840},"runtime":{"goroutines":39}},"filebeat":{"events":{"active":0},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-24T13:52:45.036+1100","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":156},"total":{"ticks":390,"value":390},"user":{"ticks":234}},"info":{"ephemeral_id":"94503edc-ba48-4c8c-ba4b-b1cad2c75753","uptime":{"ms":244464},"version":"8.6.2"},"memstats":{"gc_next":35680840,"memory_alloc":17479256,"memory_total":80799032,"rss":66236416},"runtime":{"goroutines":39}},"filebeat":{"events":{"active":0},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-24T13:53:15.036+1100","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":156},"total":{"ticks":390,"value":390},"user":{"ticks":234}},"info":{"ephemeral_id":"94503edc-ba48-4c8c-ba4b-b1cad2c75753","uptime":{"ms":274462},"version":"8.6.2"},"memstats":{"gc_next":34946280,"memory_alloc":16869648,"memory_total":80875848,"rss":62791680},"runtime":{"goroutines":39}},"filebeat":{"events":{"active":0},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"handles":{"open":2}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-24T13:53:45.039+1100","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":156},"total":{"ticks":390,"value":390},"user":{"ticks":234}},"info":{"ephemeral_id":"94503edc-ba48-4c8c-ba4b-b1cad2c75753","uptime":{"ms":304462},"version":"8.6.2"},"memstats":{"gc_next":34946280,"memory_alloc":16962360,"memory_total":80968560,"rss":62795776},"runtime":{"goroutines":39}},"filebeat":{"events":{"active":0},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-24T13:53:52.017+1100","log.logger":"input.harvester","log.origin":{"file.name":"log/harvester.go","file.line":341},"message":"File is inactive. Closing because close_inactive of 5m0s reached.","service.name":"filebeat","input_id":"942082e1-8649-41bd-a2ce-d33037a7c09b","source_file":"C:\\ProgramData\\app01\\logs\\SupportLog.txt","state_id":"native::786432-133237-304127642","finished":false,"os_id":"786432-133237-304127642","harvester_id":"fe495d11-1488-421b-b27a-3a37cd045849","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-24T13:54:15.038+1100","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":156},"total":{"ticks":390,"value":390},"user":{"ticks":234}},"info":{"ephemeral_id":"94503edc-ba48-4c8c-ba4b-b1cad2c75753","uptime":{"ms":334457},"version":"8.6.2"},"memstats":{"gc_next":34946280,"memory_alloc":17057880,"memory_total":81064080,"rss":62865408},"runtime":{"goroutines":33}},"filebeat":{"events":{"active":0,"added":1,"done":1},"harvester":{"closed":1,"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0,"filtered":1,"total":1}}},"registrar":{"states":{"current":1,"update":1},"writes":{"success":1,"total":1}},"system":{"handles":{"open":1}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-24T13:54:45.040+1100","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":156},"total":{"ticks":390,"value":390},"user":{"ticks":234}},"info":{"ephemeral_id":"94503edc-ba48-4c8c-ba4b-b1cad2c75753","uptime":{"ms":364458},"version":"8.6.2"},"memstats":{"gc_next":34946280,"memory_alloc":17141752,"memory_total":81147952,"rss":62873600},"runtime":{"goroutines":33}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-24T13:55:15.045+1100","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":156},"total":{"ticks":406,"time":{"ms":16},"value":406},"user":{"ticks":250,"time":{"ms":16}}},"info":{"ephemeral_id":"94503edc-ba48-4c8c-ba4b-b1cad2c75753","uptime":{"ms":394458},"version":"8.6.2"},"memstats":{"gc_next":34854808,"memory_alloc":16859568,"memory_total":81229672,"rss":62873600},"runtime":{"goroutines":33}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-24T13:55:45.051+1100","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":171,"time":{"ms":15}},"total":{"ticks":421,"time":{"ms":15},"value":421},"user":{"ticks":250}},"info":{"ephemeral_id":"94503edc-ba48-4c8c-ba4b-b1cad2c75753","uptime":{"ms":424461},"version":"8.6.2"},"memstats":{"gc_next":34854808,"memory_alloc":16939632,"memory_total":81309736,"rss":62873600},"runtime":{"goroutines":33}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-24T13:56:15.056+1100","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":171},"total":{"ticks":421,"value":421},"user":{"ticks":250}},"info":{"ephemeral_id":"94503edc-ba48-4c8c-ba4b-b1cad2c75753","uptime":{"ms":454464},"version":"8.6.2"},"memstats":{"gc_next":34854808,"memory_alloc":17021368,"memory_total":81391472,"rss":62873600},"runtime":{"goroutines":33}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-24T13:56:45.057+1100","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":171},"total":{"ticks":421,"value":421},"user":{"ticks":250}},"info":{"ephemeral_id":"94503edc-ba48-4c8c-ba4b-b1cad2c75753","uptime":{"ms":484463},"version":"8.6.2"},"memstats":{"gc_next":34854808,"memory_alloc":17098528,"memory_total":81468632,"rss":62873600},"runtime":{"goroutines":33}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-24T13:57:15.066+1100","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":171},"total":{"ticks":421,"value":421},"user":{"ticks":250}},"info":{"ephemeral_id":"94503edc-ba48-4c8c-ba4b-b1cad2c75753","uptime":{"ms":514468},"version":"8.6.2"},"memstats":{"gc_next":34827576,"memory_alloc":16841840,"memory_total":81548736,"rss":62922752},"runtime":{"goroutines":33}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-24T13:57:45.066+1100","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":171},"total":{"ticks":421,"value":421},"user":{"ticks":250}},"info":{"ephemeral_id":"94503edc-ba48-4c8c-ba4b-b1cad2c75753","uptime":{"ms":544467},"version":"8.6.2"},"memstats":{"gc_next":34827576,"memory_alloc":16926672,"memory_total":81633568,"rss":62922752},"runtime":{"goroutines":33}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-24T13:58:15.068+1100","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":171},"total":{"ticks":421,"value":421},"user":{"ticks":250}},"info":{"ephemeral_id":"94503edc-ba48-4c8c-ba4b-b1cad2c75753","uptime":{"ms":574469},"version":"8.6.2"},"memstats":{"gc_next":34827576,"memory_alloc":17003760,"memory_total":81710656,"rss":62939136},"runtime":{"goroutines":33}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-24T13:58:45.064+1100","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":171},"total":{"ticks":421,"value":421},"user":{"ticks":250}},"info":{"ephemeral_id":"94503edc-ba48-4c8c-ba4b-b1cad2c75753","uptime":{"ms":604463},"version":"8.6.2"},"memstats":{"gc_next":34827576,"memory_alloc":17079912,"memory_total":81786808,"rss":62943232},"runtime":{"goroutines":33}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-24T13:59:15.059+1100","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":171},"total":{"ticks":421,"value":421},"user":{"ticks":250}},"info":{"ephemeral_id":"94503edc-ba48-4c8c-ba4b-b1cad2c75753","uptime":{"ms":634456},"version":"8.6.2"},"memstats":{"gc_next":34821880,"memory_alloc":16839408,"memory_total":81866888,"rss":62984192},"runtime":{"goroutines":33}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-24T13:59:45.068+1100","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":171},"total":{"ticks":421,"value":421},"user":{"ticks":250}},"info":{"ephemeral_id":"94503edc-ba48-4c8c-ba4b-b1cad2c75753","uptime":{"ms":664463},"version":"8.6.2"},"memstats":{"gc_next":34821880,"memory_alloc":16925360,"memory_total":81952840,"rss":62996480},"runtime":{"goroutines":33}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-24T14:00:15.073+1100","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":171},"total":{"ticks":421,"value":421},"user":{"ticks":250}},"info":{"ephemeral_id":"94503edc-ba48-4c8c-ba4b-b1cad2c75753","uptime":{"ms":694465},"version":"8.6.2"},"memstats":{"gc_next":34821880,"memory_alloc":17008456,"memory_total":82035936,"rss":63004672},"runtime":{"goroutines":33}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"handles":{"open":2}}},"ecs.version":"1.6.0"}}

stephenb · March 24, 2023, 3:46am

take out
document_type => "%{[@metadata][type]}"

Try my exact configs just change your path
Use default logstash.yml, Did you make changes in logstash.yml?
Make sure you only have 1 .conf file in the directory... they get concatenated. (yup they do)

Here is my full filebeat

filebeat.inputs:
- type: filestream
  id: my-filestream-id
  enabled: true
  paths:
    - /var/log/*.log
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
output.logstash:
  hosts: ["localhost:5044"]
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

Here is my full beats-logstash.conf

input {
  beats {
    port => 5044
  }
}

output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => "http://localhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}" 
      action => "create" 
      pipeline => "%{[@metadata][pipeline]}" 
      # user => "elastic"
      # password => "secret"
    }
  } else {
    elasticsearch {
      hosts => "http://localhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}" 
      action => "create"
      # user => "elastic"
      # password => "secret"
    }
  }
}

I have an empty cluster

Start Logstash (use the windows version)
./bin/logstash -r -f config/beats-logstash.conf

Clean your clean registry
Delte filebeat-8.6.2 datastream in kibana

Start filebeat (use the windows version)

./filebeat -e

No ECS or document type errors in logstash
Data Loads
Data in Discover

Kvoyce2023 · March 24, 2023, 10:31pm

Not sure where I am making mistake, I did what you have suggested.

but no luck . whenever I add action => "create" into beats-logstash.conf file - no data is flowing from filebeat server to logstash.

Below are my logstash.conf and filebeat.yml file. Please advise.

) filebeatlogstash.conf

input {
  beats {
    port => 5046
    host => "20.21.211.1"
    
  }
}

output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => ["http://20.21.211.4:9200"]
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"       
	  action => "create"
      pipeline => "%{[@metadata][pipeline]}" 
      user => "elastic"
      password => "hidden"   
    }
  } else {
    elasticsearch {
      hosts => ["http://20.21.211.4:9200"]
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"       
	  action => "create"
      user => "elastic"
      password => "hidden"   
    }
  }
}

filebeat.yml

filebeat.inputs:
- type: log
  id: my-filestream-id
  enabled: true
  paths:
    - C:\ProgramData\app01\Logs\app01.SupportLog.txt
  fields:
    log_type: app01
    log_subtype: Config
    customer_name: ${customer:customername} 
     ### Multiline options
  multiline.pattern: \d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}
  multiline.negate: true
  multiline.match: after
        
filebeat.config.modules:
  path: "C:\\beats\\filebeat\\modules.d\\*.yml"
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
output.logstash:
  hosts: ["20.21.211.1:5046"]
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

stephenb · March 24, 2023, 11:13pm

Can you try type filestream

Yours
- type: log

Mine
- type: filestream

Also what version of elasticsearch

Also did you make any changes to logstash.yml

Something else weird... Your error does not look right can you show the error from your lastest tries?

Did you clean up remove all the old indices and data stream?

Do you notice

The name of the data stream should be

filebeat-8.6.2

The indices should look like what you see in my image.

Do you have any other .yml like a copy in your filebeat directory..

Something basic not right

Kvoyce2023 · March 29, 2023, 12:48am

Hi Stephen -

I changed from "- type: log" to "- type: filestream" on filebeat.yml , but no luck.

I am using ELK version 8.6

I have deleted all the old indices and data stream including "filebeat-8.6.2". And then started filebeat svc. But still no data about applicationLog which used to appear as tail on logstash-plain.log.

filebeatlogstash.conf remains same as above.

logstash.yml got the below code:

node.name: node01
path.data: C:\elastic\logstash-8.6.2-windows-x86_64\logstash-8.6.2\data
pipeline.id: main

Please advise the next steps ?

stephenb · March 29, 2023, 1:54am

Not sure what's going on

There is something fundamental.

You can try to start Start filebeat and logstash in the foreground instead of as a service.... as I already suggested

Is there more than one .conf file in the configuration directories?

It seems like it's not using the the logstash conf file that you think it is like that's not using the actual output.

Do you have a pipelines.yml file?

As I showed above, if you reinstall logstash from scratch and then use my two files and only change the path to your data file, it should work.

Example, why did you change the beats port? Do you have other inputs going on?

Pretty sure you've changed something somewhere?

system · April 26, 2023, 1:55am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.