Unable to send json to elastic

Magnuss · April 11, 2020, 10:04pm

Hi,

I have some issues sending a json file to elastic. Logstash can connect to elastic and also in the logging are no errors. I may think it has to do with the json file or maybe the format.(or do i need to use the filter?)

    input {
      file {
        path => "/var/log/logstash/data/data*.json"
      }
    }

    output {
      elasticsearch {
        index => "%{[ecs-file]}"
        hosts => ["192.168.23.135"]
      }
    }

Luca_Belluccini · April 11, 2020, 11:12pm

I would need some extra information to troubleshoot this.

I see you would like to create an index which has the content of the field ecs-file, but you never grab it from the input events.

I am also tempted to say you see no error because the data actually getting indexed, but into an index named exactly %{[ecs-file]}.

If it is the case (GET _cat/indices?v to list the indices), I think you should:

Specify the codec of the input lines

Delete the "wrongly" created index %{[ecs-file]} which can be deleted using DELETE %25%7B%5Becs-file%5D%7D

If it is not the case, then I would suggest setting Logstash in debug mode (log.level: debug in logstash.yml), search if there are any errors and show the version of Logstash & Elasticsearch.

Also, if Logstash already parsed the files, the sincedb (documentation) has already marked the input file(s) are read and you should remove it in order to parse again the file.

Magnuss · April 12, 2020, 7:36am

Hi Luca,

Thanks for the quick reply.

I created a index as follow:

PUT _template/ecs-file
{
    "order" : 0,
    "index_patterns" : [
      "ecs-file-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "delete_after_7days_3primaries",
          "rollover_alias" : "ecs-file-*"
        }
      }
    },
    "mappings": { },
    "aliases" : { }
}

PUT ecs-file-000001
{
  "aliases": {
    "ecs-file": {
      "is_write_index": true
    }
  }
}

cat indices

GET _cat/indices?

Output

green open ecs-file-000001

I have restarted the logstash in this is the logging i get:

[2020-04-12T03:28:01,100][INFO ][logstash.javapipeline    ][testfile] Pipeline started {"pipeline.id"=>"testfile}
[2020-04-12T03:28:01,281][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:alipress], :non_running_pipelines=>[]}
[2020-04-12T03:28:01,523][INFO ][filewatch.observingtail  ][testfile] START, creating Discoverer, Watch with filend sincedb collections
[2020-04-12T03:28:02,318][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600
[2020-04-12T03:30:01,069][WARN ][logstash.runner          ] SIGTERM received. Shutting down.
[2020-04-12T03:30:01,250][INFO ][filewatch.observingtail  ] QUIT - closing all files and shutting down.
[2020-04-12T03:30:01,667][INFO ][logstash.javapipeline    ] Pipeline terminated {"pipeline.id"=>"testfile"}
[2020-04-12T03:30:02,602][INFO ][logstash.runner          ] Logstash shut down.
[2020-04-12T03:30:42,664][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.4.2"}
[2020-04-12T03:30:46,651][INFO ][org.reflections.Reflections] Reflections took 149 ms to scan 1 urls, producing 20 ys and 40 values
[2020-04-12T03:30:49,188][INFO ][logstash.outputs.elasticsearch][testfile] Elasticsearch pool URLs updated {:chans=>{:removed=>[], :added=>[http://192.168.23.135:9200/]}}
[2020-04-12T03:30:49,802][WARN ][logstash.outputs.elasticsearch][testfile] Restored connection to ES instance {:u=>"http://192.168.23.135:9200/"}
[2020-04-12T03:30:49,930][INFO ][logstash.outputs.elasticsearch][testfile] ES Output version determined {:es_versn=>7}
[2020-04-12T03:30:49,948][WARN ][logstash.outputs.elasticsearch][testfile] Detected a 6.x and above cluster: the ype` event field won't be used to determine the document _type {:es_version=>7}
[2020-04-12T03:30:50,017][INFO ][logstash.outputs.elasticsearch][testfile] New Elasticsearch output {:class=>"Logash::Outputs::ElasticSearch", :hosts=>["//192.168.23.135"]}
[2020-04-12T03:30:50,263][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][testfile] A gauge metc of an unknown type (org.jruby.RubyArray) has been create for key: cluster_uuids. This may result in invalid seriazation.  It is recommended to log an issue to the responsible developer/development team.
[2020-04-12T03:30:50,319][INFO ][logstash.javapipeline    ][testfile] Starting pipeline {:pipeline_id=>"testfile, "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, :thrd=>"#<Thread:0x31c7a556 run>"}
[2020-04-12T03:30:50,324][INFO ][logstash.outputs.elasticsearch][testfile] Using default mapping template
[2020-04-12T03:30:50,538][INFO ][logstash.outputs.elasticsearch][testfile] Attempting to install template {:managtemplate=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_oshards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"ring", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"strin, "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], roperties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "propertie=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type>"half_float"}}}}}}}
[2020-04-12T03:30:51,417][INFO ][logstash.inputs.file     ][testfile] No sincedb_path set, generating one based othe "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_9ad389108014868079fae20d6a174d21 :path=>["/var/log/logstash/data/data*.json"]}
[2020-04-12T03:30:51,544][INFO ][logstash.javapipeline    ][testfile] Pipeline started {"pipeline.id"=>"testfile}
[2020-04-12T03:30:51,847][INFO ][filewatch.observingtail  ][testfile] START, creating Discoverer, Watch with filend sincedb collections
[2020-04-12T03:30:51,876][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:alipress], :non_running_pipelines=>[]}
[2020-04-12T03:30:52,954][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600

When i remove the index it automatically creating the logstash index. It look likes logstash is not sending the json file.

Magnuss · April 12, 2020, 11:20am

What i have done:

Checked if the paths are correct.
Tried json example data
changed .rb file to .conf
Checked if the permissions were set correct

Here the debug logging:

  [2020-04-12T07:14:08,583][DEBUG][logstash.javapipeline    ] Pipeline started successfully {:pipeline_id=>"file", :thread=>"#<Thread:0x5999d9de run>"}
  [2020-04-12T07:14:08,655][DEBUG][org.logstash.execution.PeriodicFlush][file] Pushing flush onto pipeline.
  [2020-04-12T07:14:08,788][INFO ][filewatch.observingtail  ][file] START, creating Discoverer, Watch with file and sincedb collections
  [2020-04-12T07:14:08,848][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:file], :non_running_pipelines=>[]}
  [2020-04-12T07:14:09,135][DEBUG][logstash.agent           ] Starting puma
  [2020-04-12T07:14:09,158][DEBUG][logstash.agent           ] Trying to start WebServer {:port=>9600}
  [2020-04-12T07:14:09,307][DEBUG][logstash.api.service     ] [api-service] start
  [2020-04-12T07:14:07,548][DEBUG][org.logstash.config.ir.CompiledPipeline][file] Compiled output
   P[output-elasticsearch{"hosts"=>["192.168.23.135"]}|[str]pipeline:9:3:```
  elasticsearch {
  #    index => "%{[ecs-file]}"
      hosts => ["192.168.23.135"]
    }
  ```]
   into
   org.logstash.config.ir.compiler.ComputeStepSyntaxElement@c96a6eaa
  [2020-04-12T07:14:09,910][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
  [2020-04-12T07:14:11,226][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
  [2020-04-12T07:14:11,227][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
  [2020-04-12T07:14:12,096][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"/etc/logstash/pipelines.yml"}
  [2020-04-12T07:14:12,176][DEBUG][logstash.config.source.local.configpathloader] Skipping the following files while reading config since they don't match the specified glob pattern {:files=>[]}
  [2020-04-12T07:14:12,179][DEBUG][logstash.config.source.local.configpathloader] Reading config file {:config_file=>"/etc/logstash/conf.d/file.rb"}
  [2020-04-12T07:14:12,222][DEBUG][logstash.agent           ] Converging pipelines state {:actions_count=>0}
  [2020-04-12T07:14:13,595][DEBUG][org.logstash.execution.PeriodicFlush][file] Pushing flush onto pipeline.
  [2020-04-12T07:14:15,085][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"/etc/logstash/pipelines.yml"}
  [2020-04-12T07:14:15,101][DEBUG][logstash.config.source.local.configpathloader] Skipping the following files while reading config since they don't match the specified glob pattern {:files=>[]}
  [2020-04-12T07:14:15,102][DEBUG][logstash.config.source.local.configpathloader] Reading config file {:config_file=>"/etc/logstash/conf.d/file.rb"}
  [2020-04-12T07:14:15,131][DEBUG][logstash.agent           ] Converging pipelines state {:actions_count=>0}
  [2020-04-12T07:14:16,255][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
  [2020-04-12T07:14:16,257][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
  [2020-04-12T07:14:18,086][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"/etc/logstash/pipelines.yml"}
  [2020-04-12T07:14:18,110][DEBUG][logstash.config.source.local.configpathloader] Skipping the following files while reading config since they don't match the specified glob pattern {:files=>[]}
  [2020-04-12T07:14:18,111][DEBUG][logstash.config.source.local.configpathloader] Reading config file {:config_file=>"/etc/logstash/conf.d/file.rb"}
  [2020-04-12T07:14:18,120][DEBUG][logstash.agent           ] Converging pipelines state {:actions_count=>0}
  [2020-04-12T07:14:18,595][DEBUG][org.logstash.execution.PeriodicFlush][file] Pushing flush onto pipeline.
  [2020-04-12T07:14:21,085][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"/etc/logstash/pipelines.yml"}
  [2020-04-12T07:14:21,113][DEBUG][logstash.config.source.local.configpathloader] Skipping the following files while reading config since they don't match the specified glob pattern {:files=>[]}
  [2020-04-12T07:14:21,143][DEBUG][logstash.config.source.local.configpathloader] Reading config file {:config_file=>"/etc/logstash/conf.d/file.rb"}
  [2020-04-12T07:14:21,147][DEBUG][logstash.agent           ] Converging pipelines state {:actions_count=>0}
  [2020-04-12T07:14:21,267][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
  [2020-04-12T07:14:21,268][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
  [2020-04-12T07:14:23,601][DEBUG][org.logstash.execution.PeriodicFlush][file] Pushing flush onto pipeline.
  [2020-04-12T07:14:24,089][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"/etc/logstash/pipelines.yml"}
  [2020-04-12T07:14:24,109][DEBUG][logstash.config.source.local.configpathloader] Skipping the following files while reading config since they don't match the specified glob pattern {:files=>[]}
  [2020-04-12T07:14:24,110][DEBUG][logstash.config.source.local.configpathloader] Reading config file {:config_file=>"/etc/logstash/conf.d/file.rb"}
  [2020-04-12T07:14:24,143][DEBUG][logstash.agent           ] Converging pipelines state {:actions_count=>0}
  [2020-04-12T07:14:26,296][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
  [2020-04-12T07:14:26,297][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
  [2020-04-12T07:14:27,084][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"/etc/logstash/pipelines.yml"}
  [2020-04-12T07:14:27,092][DEBUG][logstash.config.source.local.configpathloader] Skipping the following files while reading config since they don't match the specified glob pattern {:files=>[]}
  [2020-04-12T07:14:27,093][DEBUG][logstash.config.source.local.configpathloader] Reading config file {:config_file=>"/etc/logstash/conf.d/file.rb"}
  [2020-04-12T07:14:27,100][DEBUG][logstash.agent           ] Converging pipelines state {:actions_count=>0}
  [2020-04-12T07:14:28,595][DEBUG][org.logstash.execution.PeriodicFlush][file] Pushing flush onto pipeline.
  [2020-04-12T07:14:30,084][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"/etc/logstash/pipelines.yml"}
  [2020-04-12T07:14:30,100][DEBUG][logstash.config.source.local.configpathloader] Skipping the following files while reading config since they don't match the specified glob pattern {:files=>[]}
  [2020-04-12T07:14:30,101][DEBUG][logstash.config.source.local.configpathloader] Reading config file {:config_file=>"/etc/logstash/conf.d/file.rb"}
  [2020-04-12T07:14:30,129][DEBUG][logstash.agent           ] Converging pipelines state {:actions_count=>0}
  [2020-04-12T07:14:31,328][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
  [2020-04-12T07:14:31,329][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
  [2020-04-12T07:14:33,082][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"/etc/logstash/pipelines.yml"}
  [2020-04-12T07:14:33,092][DEBUG][logstash.config.source.local.configpathloader] Skipping the following files while reading config since they don't match the specified glob pattern {:files=>[]}
  [2020-04-12T07:14:33,093][DEBUG][logstash.config.source.local.configpathloader] Reading config file {:config_file=>"/etc/logstash/conf.d/file.rb"}
  [2020-04-12T07:14:33,117][DEBUG][logstash.agent           ] Converging pipelines state {:actions_count=>0}
  [2020-04-12T07:14:33,595][DEBUG][org.logstash.execution.PeriodicFlush][file] Pushing flush onto pipeline.
  [2020-04-12T07:14:36,084][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"/etc/logstash/pipelines.yml"}
  [2020-04-12T07:14:36,091][DEBUG][logstash.config.source.local.configpathloader] Skipping the following files while reading config since they don't match the specified glob pattern {:files=>[]}
  [2020-04-12T07:14:36,093][DEBUG][logstash.config.source.local.configpathloader] Reading config file {:config_file=>"/etc/logstash/conf.d/file.rb"}
  [2020-04-12T07:14:36,102][DEBUG][logstash.agent           ] Converging pipelines state {:actions_count=>0}
  [2020-04-12T07:14:36,354][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
  [2020-04-12T07:14:36,374][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
  [2020-04-12T07:14:38,596][DEBUG][org.logstash.execution.PeriodicFlush][file] Pushing flush onto pipeline.
  [2020-04-12T07:14:39,084][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"/etc/logstash/pipelines.yml"}
  [2020-04-12T07:14:39,088][DEBUG][logstash.config.source.local.configpathloader] Skipping the following files while reading config since they don't match the specified glob pattern {:files=>[]}
  [2020-04-12T07:14:39,090][DEBUG][logstash.config.source.local.configpathloader] Reading config file {:config_file=>"/etc/logstash/conf.d/file.rb"}
  [2020-04-12T07:14:39,124][DEBUG][logstash.agent           ] Converging pipelines state {:actions_count=>0}
  [2020-04-12T07:14:41,385][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
  [2020-04-12T07:14:41,386][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
  [2020-04-12T07:14:42,084][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"/etc/logstash/pipelines.yml"}
  [2020-04-12T07:14:42,095][DEBUG][logstash.config.source.local.configpathloader] Skipping the following files while reading config since they don't match the specified glob pattern {:files=>[]}
  [2020-04-12T07:14:42,097][DEBUG][logstash.config.source.local.configpathloader] Reading config file {:config_file=>"/etc/logstash/conf.d/file.rb"}
  [2020-04-12T07:14:42,102][DEBUG][logstash.agent           ] Converging pipelines state {:actions_count=>0}
  [2020-04-12T07:14:43,595][DEBUG][org.logstash.execution.PeriodicFlush][file] Pushing flush onto pipeline.
  [2020-04-12T07:14:45,087][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"/etc/logstash/pipelines.yml"}
  [2020-04-12T07:14:45,108][DEBUG][logstash.config.source.local.configpathloader] Skipping the following files while reading config since they don't match the specified glob pattern {:files=>[]}
  [2020-04-12T07:14:45,110][DEBUG][logstash.config.source.local.configpathloader] Reading config file {:config_file=>"/etc/logstash/conf.d/file.rb"}
  [2020-04-12T07:14:45,115][DEBUG][logstash.agent           ] Converging pipelines state {:actions_count=>0}

Luca_Belluccini · April 12, 2020, 3:02pm

Hello @Magnuss

Ok, got it. You want to use ILM and there are few things to take care of.

Unfortunately it is not possible to create rollover_aliase as you've done.
You really need to define a rollover_alias explicit name, now wildcards allowed.

E.g. if you will have 5 expected values (e.g. ecs-file-one, ecs-file-two, ecs-file-three...), you have to define 5 different index templates (which can use the same ILM policy).

If you can "accept" to write all the data into a single index, you have 2 choices:

Rely on Logstash ILM integration
Handle ILM manually (as you've done)

Let's review the steps you've done for (2).

Let's restart from scratch and delete the ecs-file-00001 index (the alias will go away with it).

The rollover_alias must be ecs-file.

PUT _template/ecs-file
{
    "order" : 0,
    "index_patterns" : [
      "ecs-file-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "delete_after_7days_3primaries",
          "rollover_alias" : "ecs-file"
        }
      }
    },
    "mappings": { },
    "aliases" : { }
}

Then we can bootstrap the index:

PUT ecs-file-000001
{
  "aliases": {
    "ecs-file": {
      "is_write_index": true
    }
  }
}

Logstash must be configured with ilm_enabled set to false (doc).

output {
  elasticsearch {
    ilm_enabled => false
    index => "ecs-file"
    hosts => ["192.168.23.135"]
  }
}

Unfortunately, due to the design of the ILM rollover alias creation API, Logstash cannot create the rollover_alias on the fly.
If you decide to use the ILM integration with Logstash, the rollover_alias gets created at Logstash startup and cannot be dynamic (based on a variable).

github.com/logstash-plugins/logstash-output-elasticsearch

Is it possible to support indexing of dynamic variables using rollover?

opened 07:24AM - 04 Apr 19 UTC

closed 02:33PM - 18 Aug 22 UTC

zcola

output { elasticsearch { hosts => [ 'ccc.om:9200' ] … ilm_enabled => true ilm_rollover_alias => "cbg_%{product}_%{log}_loghub" } } ``` def setup_ilm return unless ilm_enabled? if default_index?(@index) || !default_rollover_alias?(@ilm_rollover_alias) logger.warn("Overwriting supplied index #{@index} with rollover alias #{@ilm_rollover_alias}") unless default_index?(@index) @index = @ilm_rollover_alias maybe_create_rollover_alias maybe_create_ilm_policy end end ``` Now it is time to determine if rolloverneeds to create rollover when it starts. @robbavey

If you want to use the approach (1), it is only feasible if you know in advance all the possible values of ecs-file variable and you have to manually bootstrap all the indices.

In the test you've done with debug enabled, index => "%{[ecs-file]}" is commented out.

To improve this in the future, we're introducing the concept of data streams:

github.com/elastic/elasticsearch

Data Streams

opened 12:02PM - 04 Mar 20 UTC

closed 08:12PM - 14 Jul 20 UTC

martijnvg

release highlight Meta :Data Management/Indices APIs Top Ask Dependency:Endpoint Team:Data Management v7.9.0

**Update:** the description is out dated, for more information take a look at [t…he unreleased data streams docs](https://www.elastic.co/guide/en/elasticsearch/reference/7.x/data-streams.html). The meta issue will track the development of a new feature named data streams. # Background Data streams are targeted towards time-based data sources and enable us to solve bootstrap problems when indexing via a write alias (logstash-plugins/logstash-output-elasticsearch#858). Today aliases have some deficiencies in how they are implemented in Elasticsearch; namely they are not first-class concepts, making them confusing to use. Aliases have a broad type of usages, whereas data streams will be a solution focused for time-based data sources. Data streams should be a first-class concept, but should also be non intrusive. # Concept A data stream formalizes the notion of a stream of data for time based data sources. Data streams groups indices from the same time-based data source together as an opaque container. A data stream keeps track of a list of indices ordered by generation. A generation starts at 0 and each time the stream is rolled over the generation is incremented by one. Writes are forwarded to the index with the highest generation (last index). Searches and other multi-index APIs forward requests to all indices that are part of a data stream (this is similar to how aliases are resolved in these APIs). Because data streams are aimed at time-series data sources, a date field is required and must be identified as the "timestamp" for the documents in the data stream. This will enable Kibana to detect that is dealing with time-series data automatically, and we can internally apply some optimizations (e.g., automatically sorting on the timestamp field). Indices that are contained with a data stream are hidden. The idea is that users interact with the data stream as much as possible and not directly with the indices that are contained within a data stream. Data streams only accept append-only writes (index requests with op_type=create). Deletes and updates are rejected. If specific documents need to be updated or deleted then these operations should happen via the index these documents reside in. The reason that these write operations are rejected via a data stream is that these operations work as expected until a data stream is rolled over and then these operations result in 404 errors. Therefore it is better to reject these operations consistently. The rollover API needs to understand how to handle a data stream. The data stream’s generation needs to be incremented and a new hidden index needs to be created atomically. The name of the index is based on the name of the data stream and its current generation, which looks like this: [datastream-name]-[datastream-generation]. A data stream can be created with the create data stream API and remove via the delete data stream API. It should also be possible to reserve a namespace to be a data stream before actually creating the data stream. For this data streams will depend on index templates v2. Index templates will get an additional setting, named data_stream, which will create a data stream with the name of what should be the concrete index and a hidden index: * A user creates an index template (in v2 format) with a desired index pattern, mappings, index settings and sets the data_stream setting to true. * This user starts ingesting data and the auto create index functionality kicks in. The previously created template matches, but instead of creating an index the following is created: * A data stream with the targeted index name as name. * A hidden index with the following name: [data-stream-name]-0. * The hidden index is added to the list of indices of the data stream and the current generation is set to 0. Additionally, data streams will be integrated into Kibana. For example, Kibana can automatically generate index patterns based on data streams, and identity the timestamp field. # Data Streams and ILM The main change will be that it will no longer be required to configure index.lifecycle.rollover_alias setting for ilm. ILM can automatically figure out if an index is part of a datastream and act accordingly. An index can only be part of a single data stream, the user doesn’t create the index and the index is hidden, because of this clear structure ILM can make assumptions and doesn’t need additional configuration. Compared to using aliases (even with alias templates) this wouldn’t be true and this is a big upside of using datastreams. ILM should also be able to update data streams atomically. For example in the context of ILM’s shrink action, the data stream should be updated to not refer to the un-shrunken index and refer to the shrunken index. For the rest, ILM should be able work as it is today. # Integration with security TBD # APIs The APIs and they way data streams are used from other APIs is not final and may change in the future. ## Index expressions and data streams in APIs Data streams should be taken into account when resolving an index expression. Also the API that resolves an index expressions is important. If a multi-index API tries to resolve the name of a data stream then all the indices of a stream should be resolved and if a write API tries to resolve the name of a data stream then only the latest index should be resolved. The following write APIs will resolve a data stream to the latest index: bulk API and index API. The following multi index APIs should resolve a data stream to all indices it contains: search, msearch, field caps and eql search. Single document read APIs should fail when being used via a data stream. The following APIs fall into this category: explain, get, mget, termvector. There are many admin APIs that are multi index. These apis should be able to resolve data streams and resolve to the latest hidden index backed by a data stream. Examples of these APIs are: put mapping, get mapping, get settings or get index. The rollover API both accepts a write alias and a data stream. ## Get index API The get index api should if an index is part of a data stream include what data stream it is part of. ## Create data stream API Request: ``` PUT /_data_stream/[name] { "timestamp_field": ... } ``` The create data stream API allows to create new data stream and the first backing index. This API creates a new data stream with the provided name and the provided timestamp field. The generation is set to 0. The backing index is created with the following name: '[data-stream-name]-000000'. The settings & mappings originate from any index template that match. If an existing data stream, index or alias already exists with the same name as the provided data stream name then the create data stream will return with an error. Also if indices or aliases exist with the same prefix in the name as the provided data stream name then an error is returned as well. ## Get data streams API Request: ``` GET /_data_streams/[name] ``` Returns the list of data streams based on the specified name and for each data stream additional meta data is included. (for example list of backing indices and current generation). If no name is provided then all data streams are returned. Also wildcard expressions are supported. ## Delete data stream API Request: ``` DELETE /_data_stream/[name] ``` Deletes the specified data stream. Also the indices that are part of the data stream are removed. ## Updating a data stream TBD A data stream can not be updated to include allow system or indices that are already part of data stream.

Magnuss · April 12, 2020, 3:07pm

@Luca_Belluccini,

Thanks for helping out. What the real problem is that logstash is not sending the json data from the file into Elastic.

The ilm information i provided was for troubleshooting purposes.

For now i'm looking for a solution to get the json data into elasticsearch using json files.

Luca_Belluccini · April 12, 2020, 6:33pm

In my previous answers I provided some information on how to solve some errors done setting up the index template and bootstrapping the rollover index, including current limitations with ILM rollover aliases.

What is the current content of the pipeline?
Do you wish to send data to one or multiple indices? Do you want to use ILM rollover or not?
Can you share one or 2 lines of the json files you want to ingest?

Magnuss · April 12, 2020, 6:46pm

@Luca_Belluccini

Json file -> Logstash -> Elastic
I would like it in one index. ILM rollover is not important.
See sample below:

[{
  "id": 4000828499889,
  "link": "https://www.aliexpress.com/item/4000828499889.html?algo_pvid=d6d52e6d-b14b-4967-ade3-9f13a4f660cc&algo_expid=d6d52e6d-b14b-4967-ade3-9f13a4f660cc-17&btsid=0ab6f83115866312995255394e208c&ws_ab_test=searchweb0_0,searchweb201602_,searchweb201603_",
  "title": "Disney Pixar Cars 2  38 Style Metallic Finish Silver Chrome Lightning McQueen 1:55 Diecast Metal Toy Car Kids Gift",
  "tradeAmount": "2 orders",
  "averageStar": "0.0",
  "descriptionURL": "https://aeproductsourcesite.alicdn.com/product/description/pc/v2/en_US/desc.htm?productId=4000828499889&key=H35f9b08890e7490b80dd4898f6cb5aa2W.zip&token=f28c2bf43fe46bd68ddeb6cda974149a",
  "store": {
    "followingNumber": 65907,
    "establishedAt": "Apr 8, 2019",
    "positiveNum": 8021,
    "positiveRate": "97.5%",
    "name": "patrol dogs Store",
    "id": 4988381,
    "url": "https://www.aliexpress.com/store/4988381",
    "topRatedSeller": true
  },
  "specs": [
    {
      "Brand Name": "Disney"
    },
    {
      "Material": "Metal"
    },
    {
      "Age Range": "> 3 years old"
    },
    {
      "Barcode": "No"
    },
    {
      "Ship/Naval Vessel": "Other"
    },
    {
      "Features": "Diecast"
    },
    {
      "Certification": "3C"
    },
    {
      "3C": "Certificate"
    },
    {
      "Certificate Number": "3587945615"
    },
    {
      "Model Number": "3587944565"
    },
    {
      "Scale": "1:55"
    },
    {
      "Warning": "Cars 3"
    },
    {
      "Type": "Car"
    },
    {
      "Sizz": "As shown"
    },
    {
      "colour": "As shown"
    },
    {
      "Style": "As shown"
    },
    {
      "Features": "Entertainment Mini Education"
    },
    {
      "disney": "disney pixar cars"
    },
    {
      "cars disney": "cars 2 disney"
    },
    {
      "cars disney pixar": "cars 3"
    },
    {
      "disney cars": "mcqueen"
    }
  ],
  "categories": [
    "All Categories",
    "Toys & Hobbies",
    "Diecasts & Toy Vehicles"
  ],

input {
  file {
    path => "/var/log/logstash/*.json"
    codec => json
  }
}

output {
  elasticsearch {
    hosts => ["192.168.23.135"]
  }
}

Luca_Belluccini · April 12, 2020, 7:50pm

Hello @Magnuss

Given your snippet:

[{
  "id": 4000828499889,
  "link": "https://www.aliexpress.com/item/4000828499889.html?algo_pvid=d6d52e6d-b14b-4967-ade3-9f13a4f660cc&algo_expid=d6d52e6d-b14b-4967-ade3-9f13a4f660cc-17&btsid=0ab6f83115866312995255394e208c&ws_ab_test=searchweb0_0,searchweb201602_,searchweb201603_",
  "title": "Disney Pixar Cars 2  38 Style Metallic Finish Silver Chrome Lightning McQueen 1:55 Diecast Metal Toy Car Kids Gift",
  "tradeAmount": "2 orders",
  "averageStar": "0.0",

There are 2 problems:

the JSON file contains an ARRAY of JSON objects (it starts with a [)
Except if you've pretty printed the JSON content, the JSON objects span over multiple lines

If we can make the assumption that every file starts with [{ and ends with }] (with an ending newline), you can use the following pipeline.

input {
	file {
		path => "..."
		start_position => "beginning"
		codec => multiline {
		  pattern => "^}?\]"
		  negate => true
		  what => next
		}
	}
}
filter { 
    json {
		source => "message"
		target => "json"
		remove_field => "message"
	}
	split {
		field => "json"
	}
}
output {
	 stdout { codec => rubydebug }
}

In any case this is highly inefficient as Logstash will buffer all the log lines prior to sending them to the filter and split the events.
The best would be to have a file containing JSON lines, one per line, without leading [ and ].