Problem logs redirection


(Guillaume) #1

Hello,

I'm meeting an issue with redirection of logs (winlogbeat,filebeat,...). Normally my logs is stocked in /var/lib/elasticsearch/nodes/0/indices. But now logs are redirected in /var/logs/logstash and now the logs are totaly readable.

logstash.yml :

# Settings file in YAML
#
# Settings can be specified either in hierarchical form, e.g.:
#
#   pipeline:
#     batch:
#       size: 125
#       delay: 5
#
# Or as flat keys:
#   pipeline.batch.size: 125
#   pipeline.batch.delay: 5
#
# ------------  Node identity ------------
#
# Use a descriptive name for the node:
#
# node.name: test
#
# If omitted the node name will default to the machine's host name
#
# ------------ Data path ------------------
#
# Which directory should be used by logstash and its plugins
# for any persistent needs. Defaults to LOGSTASH_HOME/data
#
path.data: /var/lib/logstash
#
# ------------ Pipeline Settings --------------
# This defaults to the number of the host's CPU cores.
#
# pipeline.workers: 2
#
# How many workers should be used per output plugin instance
#
# pipeline.output.workers: 1
#
# How many events to retrieve from inputs before sending to filters+workers
#
# pipeline.batch.size: 125
#
# How long to wait before dispatching an undersized batch to filters+workers
# Value is in milliseconds.
#
# pipeline.batch.delay: 5
#
# pipeline.unsafe_shutdown: false
#
# ------------ Pipeline Configuration Settings --------------
#
# Where to fetch the pipeline configuration for the main pipeline
#
path.config: /etc/logstash/conf.d
#
# Pipeline configuration string for the main pipeline
#
# config.string:
#
# At startup, test if the configuration is valid and exit (dry run)
#
# config.test_and_exit: false
#
# config.reload.automatic: false
#
# How often to check if the pipeline configuration has changed (in seconds)
#
# config.reload.interval: 3
#
# Show fully compiled configuration as debug log message
# NOTE: --log.level must be 'debug'
#
# config.debug: false
#
# ------------ Metrics Settings --------------
#
# Bind address for the metrics REST endpoint
#
# http.host: "127.0.0.1"
#
# Bind port for the metrics REST endpoint, this option also accept a range
# (9600-9700) and logstash will pick up the first available ports.
#
# http.port: 9600-9700
#
# ------------ Debugging Settings --------------
#
# Options for log.level:
#   * fatal
#   * error
#   * warn
#   * info (default)
#   * debug
#   * trace
#
# log.level: info
path.logs: /var/log/logstash
#
# ------------ Other Settings --------------
#
# Where to find custom plugins
# path.plugins: []

elasticsearch.yml :

# ======================== Elasticsearch Configuration =========================
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 127.0.0.1
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.zen.ping.unicast.hosts: ["host1", "host2"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes: 3
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

thread_pool.search.size: 7
thread_pool.search.queue_size: 5000
action.auto_create_index: .security,.monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*

30-elasticsearch-output.conf :

output {
  if [type] == "procurve" {
        elasticsearch {  
            hosts => ["127.0.0.1:9200"]    
            index => "procurve-%{+YYYY.MM.dd}"   
        }
  } else if [type] == "wifi" {
  	 elasticsearch {
            hosts => ["127.0.0.1:9200"]
            index => "wifi-%{+YYYY.MM.dd}"
        }  
  } else {
        elasticsearch {
            hosts => ["127.0.0.1:9200"]
            sniffing => true
            manage_template => false
            index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
            document_type => "%{[@metadata][type]}"
        }
    }
}

(Mark Walkom) #2

What do you mean by that?


(Guillaume) #3

For example i can see active directory logs directly in the files /var/logs/logstash.
2018-11-16T09:52:05.898L’ouverture de session d’un compte s’est correctement déroulée.

Sujet :
        ID de sécurité :                S-1-0-0
        Nom du compte :         -
        Domaine du compte :             -
        ID d’ouverture de session :             0x0

Type d’ouverture de session :                   3

Nouvelle ouverture de session :
        ID de sécurité :                S-x-x-x-xxxxxxxxx-xxxxxxxx-xxxxxxxxx-xxxxxxxxx
        Nom du compte :         xxxxxxxx
        Domaine du compte :             xxxxxxxxx
        ID d’ouverture de session :             xxxxxxxxxxx
        GUID d’ouverture de session :           {00000000-0000-0000-0000-000000000000}

Informations sur le processus :
        ID du processus :               0x0
        Nom du processus :              -

Informations sur le réseau :
        Nom de la station de travail :  xxxxx
        Adresse du réseau source :      xxx.xxx.xxx.xxx
        Port source :           xxxxx

Informations détaillées sur l’authentification :
        Processus d’ouverture de session :              NtLmSsp
        Package d’authentification :    NTLM
        Services en transit :   -
        Nom du package (NTLM uniquement) :      NTLM V2
        Longueur de la clé :           

Cet événement est généré lors de la création d’une ouverture de session. Il est généré sur l’ordinateur sur lequel l’ouverture de session a été effectuée.

Le champ Objet indique le compte sur le système local qui a demandé l’ouverture de session. Il s’agit le plus souvent d’un service, comme le service Serveur, ou un processus local tel que Winlogon.exe ou Services.exe.

Le champ Type d’ouverture de session indique le type d’ouverture de session qui s’est produit. Les types les plus courants sont 2 (interactif) et 3 (réseau).

Le champ Nouvelle ouverture de session indique le compte pour lequel la nouvelle ouverture de session a été créée, par exemple, le compte qui s’est connecté.

Les champs relatifs au réseau indiquent la provenance d’une demande d’ouverture de session à distance. Le nom de la station de travail n’étant pas toujours disponible, peut être laissé vide dans certains cas.

Les champs relatifs aux informations d’authentification fournissent des détails sur cette demande d’ouverture de session spécifique.
        - Le GUID d’ouverture de session est un identificateur unique pouvant servir à associer cet événement à un événement KDC .
        - Les services en transit indiquent les services intermédiaires qui ont participé à cette demande d’ouverture de session.
        - Nom du package indique quel est le sous-protocole qui a été utilisé parmi les protocoles NTLM.

Also i saw an error in /var/logs/logstash :

[WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>404, :action=>["index", {:_id=>nil, :_index=>"winlogbeat-2018.11.16", :_type=>"wineventlog", :_routing=>nil},


(Mark Walkom) #4

404 is a not found.
Is there anything in the Elasticsearch logs?


(Guillaume) #5
[2018-11-16T09:21:31,190][INFO ][o.e.e.NodeEnvironment    ] [jFgfZVV] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [79.4gb], net total_space [297.3gb], spins? [unknown], types [rootfs]
[2018-11-16T09:21:31,190][INFO ][o.e.e.NodeEnvironment    ] [jFgfZVV] heap size [3.9gb], compressed ordinary object pointers [true]
[2018-11-16T09:21:32,452][INFO ][o.e.n.Node               ] node name [jFgfZVV] derived from node ID [jFgfZVVlS_CHwYSTAMaYgw]; set [node.name] to override
[2018-11-16T09:21:32,453][INFO ][o.e.n.Node               ] version[5.4.1], pid[1950], build[2cfe0df/2017-05-29T16:05:51.443Z], OS[Linux/3.10.0-514.21.1.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_131/25.131-b11]
[2018-11-16T09:21:32,453][INFO ][o.e.n.Node               ] JVM arguments [-Xms4g, -Xmx4g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-11-16T09:21:33,891][INFO ][o.e.p.PluginsService     ] [jFgfZVV] loaded module [aggs-matrix-stats]
[2018-11-16T09:21:33,891][INFO ][o.e.p.PluginsService     ] [jFgfZVV] loaded module [ingest-common]
[2018-11-16T09:21:33,892][INFO ][o.e.p.PluginsService     ] [jFgfZVV] loaded module [lang-expression]
[2018-11-16T09:21:33,892][INFO ][o.e.p.PluginsService     ] [jFgfZVV] loaded module [lang-groovy]
[2018-11-16T09:21:33,892][INFO ][o.e.p.PluginsService     ] [jFgfZVV] loaded module [lang-mustache]
[2018-11-16T09:21:33,892][INFO ][o.e.p.PluginsService     ] [jFgfZVV] loaded module [lang-painless]
[2018-11-16T09:21:33,892][INFO ][o.e.p.PluginsService     ] [jFgfZVV] loaded module [percolator]
[2018-11-16T09:21:33,892][INFO ][o.e.p.PluginsService     ] [jFgfZVV] loaded module [reindex]
[2018-11-16T09:21:33,892][INFO ][o.e.p.PluginsService     ] [jFgfZVV] loaded module [transport-netty3]
[2018-11-16T09:21:33,892][INFO ][o.e.p.PluginsService     ] [jFgfZVV] loaded module [transport-netty4]
[2018-11-16T09:21:33,893][INFO ][o.e.p.PluginsService     ] [jFgfZVV] no plugins loaded
[2018-11-16T09:21:36,285][INFO ][o.e.d.DiscoveryModule    ] [jFgfZVV] using discovery type [zen]
[2018-11-16T09:21:37,030][WARN ][o.e.c.u.IndexFolderUpgrader] [/var/lib/elasticsearch/nodes/0/indices/Ra5xr0J5TQqgteQ0MeYG8w] no index state found - ignoring
[2018-11-16T09:21:37,031][WARN ][o.e.c.u.IndexFolderUpgrader] [/var/lib/elasticsearch/nodes/0/indices/4UI069uPQAGmz9lgE37y7A] no index state found - ignoring
[2018-11-16T09:21:37,411][WARN ][o.e.c.u.IndexFolderUpgrader] [/var/lib/elasticsearch/nodes/0/indices/PNV4imvDS2mA7TEuxjJxfA] no index state found - ignoring
[2018-11-16T09:21:37,413][WARN ][o.e.c.u.IndexFolderUpgrader] [/var/lib/elasticsearch/nodes/0/indices/f7llplivQ-uiTg3ImxaRPQ] no index state found - ignoring
[2018-11-16T09:21:38,339][INFO ][o.e.n.Node               ] initialized
[2018-11-16T09:21:38,339][INFO ][o.e.n.Node               ] [jFgfZVV] starting ...
[2018-11-16T09:21:38,586][INFO ][o.e.t.TransportService   ] [jFgfZVV] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2018-11-16T09:21:41,667][INFO ][o.e.c.s.ClusterService   ] [jFgfZVV] new_master {jFgfZVV}{jFgfZVVlS_CHwYSTAMaYgw}{WOlwA1zKRcm4oQ2NrwtzOQ}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-11-16T09:21:41,708][INFO ][o.e.h.n.Netty4HttpServerTransport] [jFgfZVV] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}
[2018-11-16T09:21:41,712][INFO ][o.e.n.Node               ] [jFgfZVV] started
[2018-11-16T09:21:45,213][INFO ][o.e.g.GatewayService     ] [jFgfZVV] recovered [137] indices into cluster_state
[2018-11-16T09:29:43,122][INFO ][o.e.c.r.a.AllocationService] [jFgfZVV] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.monitoring-data-2][0], [.kibana][0]] ...]).

No logs since 09:29:43 am, when i'm writing it's 11:11 am.


(Guillaume) #6

I found a solution, it's cause by the line #action.auto_create_index: .security,.monitoring*,.watches,.triggered_watches,.watcher-history*,.ml* in the elasticsearch.yml. You just have to comment.