# Logstash fails 409 conflict unknown column: 'type'\\"

**URL:** <https://discuss.elastic.co/t/logstash-fails-409-conflict-unknown-column-type/364886>\
**Category:** Logstash\
**Created:** [August 14, 2024, 6:44am UTC](https://discuss.elastic.co/t/logstash-fails-409-conflict-unknown-column-type/364886 "2024-08-14T06:44:56Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![ridish](https://avatars.discourse-cdn.com/v4/letter/r/839c29/32.png) [@ridish](https://discuss.elastic.co/u/ridish)\
**Post date:** [August 14, 2024, 6:44am UTC](https://discuss.elastic.co/t/logstash-fails-409-conflict-unknown-column-type/364886/1 "2024-08-14T06:44:57Z")

</div>

Hello Everyone,

I'm a newbie to logstash and i'm trying to configure logstash with manticore which is based on elasticsearch 6.3.2. And i have the following config

```auto
input {
  # Input for the first log file
  file {
    path => "${LOG_BUNDLE}/*/cvm_logs/stargate.*" # Path to the first log file
    start_position => "beginning"
    exit_after_read => true
    sincedb_clean_after => "1 day"
    sincedb_path => "/dev/null"
    mode => "read"
    file_completed_action => "log"
    file_completed_log_path => "/dev/null"
    type => "cpp_stargate"
  }
}

filter {
  # Filter for the first log type
  if [type] == "cpp_stargate" {
    if [message] =~ /^Log|^Running/ {
      drop { }
    }
    grok {
      match => {
        "message" => "^(?<log_level>(I|W|E|F)+)(?<time>\d{4}\d{2}\d{2} \d{2}:\d{2}:\d{2}.\d{3})\d{3}Z\s+(?<pid>[^]+)\s+(?<source_log_filename>[^\:]+):[^\]]+]*%{GREEDYDATA:message}"
      }
      overwrite => ["message"]
      break_on_match => true
    }
    date {
      match => ["time", "yyyyMMdd HH:mm:ss.SSS"]
      target => "@timestamp"
      timezone => "UTC"
    }
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9308"]
    index => "nu_logs1"
    data_stream => false
    document_id => "@timestamp"
    retry_on_conflict => 0
    ecs_compatibility => disabled
    manage_template => false
    ilm_enabled => false
    http_compression => false
    #action => "update"
    #doc_as_upsert => true
  }
  stdout {
    codec => rubydebug
  }
}

```

However, when i run this config on my log bundle i get the following error:

```auto
[ERROR] 2024-08-14 02:30:02.538 [[main]>worker10] elasticsearch - Encountered a retryable error (will retry with exponential backoff) {:code=>409, :url=>"http://localhost:9308/_bulk?filter_path=errors,items.*.error,items.*.status", :content_length=>11484, :body=>"{\"error\":\"unknown column: 'type'\"}"}

```

I'm not sure why this error? I don't see the usual tag errors like dateparserfailure or grokparserfailure. What could be going wrong here? Please help.

---

<div class="post-metadata">

**Author:** ![nfeekery](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nfeekery/32/117349_2.png) [@nfeekery](https://discuss.elastic.co/u/nfeekery)\
**Post date:** [August 14, 2024, 8:32am UTC](https://discuss.elastic.co/t/logstash-fails-409-conflict-unknown-column-type/364886/2 "2024-08-14T08:32:16Z")

</div>

Category updated on August 14, 2024, 8:32am UTC

From #Elastic Search to #Logstash

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [August 14, 2024, 10:34am UTC](https://discuss.elastic.co/t/logstash-fails-409-conflict-unknown-column-type/364886/3 "2024-08-14T10:34:32Z")

</div>

> [@ridish](#):
>
> `"{\"error\":\"unknown column: 'type'\"}`

Have you manually created the template? It seems "type" is missing.

---

<div class="post-metadata">

**Author:** ![ridish](https://avatars.discourse-cdn.com/v4/letter/r/839c29/32.png) [@ridish](https://discuss.elastic.co/u/ridish)\
**Post date:** [August 14, 2024, 3:10pm UTC](https://discuss.elastic.co/t/logstash-fails-409-conflict-unknown-column-type/364886/4 "2024-08-14T15:10:39Z")

</div>

Hi Rios,

Thanks for responding. Manticore doesn't support templates yet. So in their documentation, they have explicitly mentioned to disable manage\_template which i have done.

```auto
https://manticoresearch.com/blog/integration-of-manticore-with-logstash-filebeat/

```

Am i missing anything? Appreciate your help.

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [August 14, 2024, 3:44pm UTC](https://discuss.elastic.co/t/logstash-fails-409-conflict-unknown-column-type/364886/5 "2024-08-14T15:44:45Z")

</div>

> [@ridish](#):
>
> `if [message] =~ /^Log|^Running/ {`

Check does the log start with Log or Running.  
Go to Elasticsearch and also check is there the template dpkg\_log and does it have the `type` field.  
Also you can add below _date_ to remove type from JSON structure for ES:  
` mutate { remove_field => ["type"] }`

---

<div class="post-metadata">

**Author:** ![ridish](https://avatars.discourse-cdn.com/v4/letter/r/839c29/32.png) [@ridish](https://discuss.elastic.co/u/ridish)\
**Post date:** [August 15, 2024, 4:29am UTC](https://discuss.elastic.co/t/logstash-fails-409-conflict-unknown-column-type/364886/6 "2024-08-15T04:29:18Z")

</div>

Hi Rios,

Removing the type field still the same error so did some research on manticore and changed the config as follows:

```auto
input {
  # Input for the first log file
  file {
    path => "${LOG_BUNDLE}/*/cvm_logs/stargate.*" # Path to the first log file
    start_position => "beginning"
    exit_after_read => true
    sincedb_clean_after => "1 day"
    sincedb_path => "/dev/null"
    mode => "read"
    file_completed_action => "log"
    file_completed_log_path => "/dev/null"
    type => "cpp_stargate"
  }
}

filter {
  # Filter for the first log type
  if [type] == "cpp_stargate" {
    if [message] =~ /^Log|^Running/ {
      drop { }
    }
    grok {
      match => {
        "message" => "^(?<log_level>(I|W|E|F)+)(?<time>\d{4}\d{2}\d{2} \d{2}:\d{2}:\d{2}.\d{3})\d{3}Z\s+(?<pid>[^]+)\s+(?<source_log_filename>[^\:]+):[^\]]+]*%{GREEDYDATA:message}"
      }
      overwrite => ["message"]
      break_on_match => true
    }
    date {
      match => ["time", "yyyyMMdd HH:mm:ss.SSS"]
      target => "@timestamp"
      timezone => "UTC"
    }
  }

  # Extract the source file name from the path
  mutate {
    add_field => {
      "filepath" => "%{path}"
      "filename_without_ext" => "%{path}"
      "filename" => "%{path}"
      "log_bundle_path" => "${LOG_BUNDLE}"
    }
  }
  # Extract the base filename from the full path
  mutate {
    gsub => ["filepath", "^.*/", ""]
  }

  mutate {
    update => { "time" => "@timestamp" }
  }

  mutate { remove_field => ["type", "tags", "@timestamp", "@version", "log", "event"] }

  # Calculate a hash value from the source filename
  #ruby {
  # code => "
  # require 'digest'
  # filename = event.get('filepath')
  # hash = Digest::MD5.hexdigest(filename)
  # index_number = hash.to_i(16) % 32 + 1
  # event.set('index_number', index_number)
  # "
  #}
}

output {
  elasticsearch {
    hosts => ["http://localhost:9308"]
    #index => " nu_logs"
    index => " nu_logs1"
    ilm_enabled => false
    manage_template => false
    http_compression => false
    #document_id => "@timestamp"
  }
}

```

After that now, i see that there are no other unknown column issues as before and now i face this error where it says this nu\_logs1 index is absent in the elasticsearch as follows:

```auto
[WARN] 2024-08-15 00:21:01.145 [[main]>worker30] elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>" nu_logs1", :routing=>nil}, {"message"=>" ParseReturnCodes: Backend returns error kCASFailure for <vdisk, block>: <26472428930, 26624>", "filename"=>"%{path}", "log_bundle_path"=>"/home/ml/log_bundles/NCC-logs-06-14-2024-1718330601674411443-0005c928-4262-a90d-4a5b-d4f5ef3c9d60", "time"=>"@timestamp", "filepath"=>"%{path}", "pid"=>"5203", "log_level"=>"E", "filename_without_ext"=>"%{path}", "source_log_filename"=>"vdiskmap_ops.cc", "host"=>{"name"=>"ml-worker"}}], :response=>{"index"=>{"_index"=>" nu_logs1", "_type"=>"doc", "_id"=>"0", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"table ' nu_logs1' absent, or does not support INSERT"}}}}

```

What could be the reason?

This table is defined in the manticore DB:

```auto
mysql> DESCRIBE nu_logs1;
+----------------------+-----------+----------------+
| Field | Type | Properties |
+----------------------+-----------+----------------+
| id | bigint | |
| message | text | indexed stored |
| filename_without_ext | string | |
| filename | string | |
| filepath | string | |
| time | timestamp | |
| log_level | string | |
| source_log_filename | string | |
| pid | bigint | |
| cvm_ip | string | |
| log_bundle_path | string | |
| anomaly | bool | |
| anomaly_category | string | |
+----------------------+-----------+----------------+
13 rows in set (0.00 sec)

```

These are shards that belongs to the table nu\_logs

```auto
mysql> DESCRIBE nu_logs;
+-----------+-------+
| Agent | Type |
+-----------+-------+
| nu_logs1 | local |
| nu_logs2 | local |
| nu_logs3 | local |
| nu_logs4 | local |
| nu_logs5 | local |
| nu_logs6 | local |
| nu_logs7 | local |
| nu_logs8 | local |
| nu_logs9 | local |
| nu_logs10 | local |
| nu_logs11 | local |
| nu_logs12 | local |
| nu_logs13 | local |
| nu_logs14 | local |
| nu_logs15 | local |
| nu_logs16 | local |
| nu_logs17 | local |
| nu_logs18 | local |
| nu_logs19 | local |
| nu_logs20 | local |
| nu_logs21 | local |
| nu_logs22 | local |
| nu_logs23 | local |
| nu_logs24 | local |
| nu_logs25 | local |
| nu_logs26 | local |
| nu_logs27 | local |
| nu_logs28 | local |
| nu_logs29 | local |
| nu_logs30 | local |
| nu_logs31 | local |
| nu_logs32 | local |
+-----------+-------+
32 rows in set (0.00 sec)

```

Kindly help. Any ideas?

---

<div class="post-metadata">

**Author:** ![ridish](https://avatars.discourse-cdn.com/v4/letter/r/839c29/32.png) [@ridish](https://discuss.elastic.co/u/ridish)\
**Post date:** [August 16, 2024, 12:10am UTC](https://discuss.elastic.co/t/logstash-fails-409-conflict-unknown-column-type/364886/7 "2024-08-16T00:10:29Z")

</div>

Please close this ticket. I was able to resolve this using the following config:

```auto
input {
  # Input for the first log file
  file {
    path => "${LOG_BUNDLE}/*/cvm_logs/stargate.*" # Path to the first log file
    start_position => "beginning"
    exit_after_read => true
    sincedb_clean_after => "1 day"
    sincedb_path => "/dev/null"
    mode => "read"
    file_completed_action => "log"
    file_completed_log_path => "/dev/null"
    type => "cpp_stargate"
  }
}

filter {
  # Filter for the first log type
  if [type] == "cpp_stargate" {
    #if [message] =~ /^Log|^Running/ {
    # drop { }
    #}
    grok {
      match => {
        "message" => "^(?<log_level>(I|W|E|F)+)(?<log_time>\d{4}\d{2}\d{2} \d{2}:\d{2}:\d{2}.\d{3})\d{3}Z\s+(?<pid>[^]+)\s+(?<source_log_filename>[^\:]+):[^\]]+]*%{GREEDYDATA:message}"
      }
      overwrite => ["message"]
      break_on_match => true
    }
    date {
      match => ["log_time", "yyyyMMdd HH:mm:ss.SSS"]
      target => "@timestamp"
      timezone => "UTC"
    }
  }

  # Calculate a hash value from the source filename
  ruby {
    code => "
      require 'digest'
      timestamp = event.get('@timestamp')
      epoch_timestamp = timestamp.to_i
      event.set('time', epoch_timestamp)
      index_number = epoch_timestamp % 32 + 1
      event.set('index_number', index_number)
    "
  }
  # Extract the source file name from the path
  mutate {
    add_field => {
      "filepath" => "%{[@metadata][path]}"
      "filename_without_ext" => "%{[@metadata][path]}"
      "filename" => "%{[@metadata][path]}"
      "log_bundle_path" => "${LOG_BUNDLE}"
      "[@metadata][shard_id]" => "%{index_number}"
      "[@metadata][_id]" => "@timestamp"
    }
    remove_field => [
      "type", "tags", "@timestamp", "@version",
      "log", "event", "index_number", "host", "log_time"
    ]
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9308"]
    index => "nu_logs%{[@metadata][shard_id]}"
    ilm_enabled => false
    manage_template => false
    http_compression => false
    #document_id => "%{[@metadata][_id]}"
  }
  #stdout {
  # codec => rubydebug
  #}
}

```

Thanks for all the help!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 13, 2024, 12:10am UTC](https://discuss.elastic.co/t/logstash-fails-409-conflict-unknown-column-type/364886/8 "2024-09-13T00:10:43Z")

</div>

Closed on September 13, 2024, 12:10am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.