Add value to blank field

Elastic Stack Logstash
1

Hi,

I am running with an issue,
I have a shipment date column which is having a blank value until your order is not shipped, so if the order is not shipped (you can say it's pending now) i want to add a value to that field. so that i can filter the value when creating a visualization.
it's not adding the value to shipment field.

if [Shipment Date] != "" {
    date {
      match => ["Shipment Date", "dd-MMM-yy"]
      target => "Shipment Date"
    }
    }
    if [Shipment Date] = "" {
      mutate{
      add_field => { "Shipment Date" => "OrderisPending" }
    }
    }

Any help?

Thanks
Rupesh

2

Use == instead of = for Logstash conditionals.

if [Shipment Date] == ""
1 Like
3

If you are sending the data to elasticsearch then you may have a problem with that. You may end up getting the error

"status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [Shipment Date] of type [date] in document with id '...'. Preview of field's value: 'OrderisPending'"

If the first record in the index has a date in that field then elasticsearch will set the field type to date, and if it subsequently receives a string it will throw that exception.

You may be able to use a placeholder such as 1970/01/01 to indicate the shipment date is pending.

1 Like
4

Right i am getting date parse error,
Let me follow your advice.

Thanks Badger

5

A _dateparsefailure is a completely different issue. If you use

output { stdout { codec => "rubydebug" }}

then what does the [Shipment Date] field look like? Ideally with examples for events that do or do not get that tag.

6

Hi Badger thanks for helping.
I used this below code to segregate blank field having shipment date or not. but it works for to filter field have no shipment date.

translate {
    field => "Shipment Date"
    destination => "Order Status"
    dictionary => {
      "" => "OrderisPending"
        }
    fallback => "AlreadyInvoiced"
    refresh_interval => 60
    refresh_behaviour => replace
  }

With No Shipment Date

{
                     "Cust #" => 996879,
             "OrderYesterday" => 0,
          "Total Order Value" => 0.0,
                    "message" => "\"O      417182\",996879,\"O\",\"      417182\",\"      417182\",\"O\",\"True\",09-Nov-20,0,0,0,\"      462176\",,09-Nov-20,,0,0,0,\"    HP\",\"Harold Peet\",\" GS\"\r",
            "Order Complete?" => "True",
         "Total Dollar Value" => 0.0,
                        "Loc" => " TS",
                "Salesperson" => "    HP",
                 "Order Date" => 2020-11-09T05:00:00.000Z,
                       "host" => "PC",
                      "Doc #" => 417182,
                   "Doc Type" => "O",
               "TotalShipped" => 0,
                      "Inv #" => "      462176",
                 "Order Type" => "O",
                       "path" => "D:/ELK/logstash-7.9.3/config/sortbyweek.csv",
                    "Order #" => 417182,
                  "Order Key" => "O      417182",
    "TotalOrderReceivedToday" => 0,
                 "@timestamp" => 2020-12-03T16:11:13.270Z,
               "Order Status" => "OrderisPending",
           "Salesperson name" => "salesperson1",
                   "@version" => "1",
         "Expected Ship Date" => 2020-11-09T05:00:00.000Z,
                   "Inv Date" => nil,
              "Shipment Date" => nil,
           "ShippedYesterday" => 0
}

Have Shipment Date

{
                     "Cust #" => 825838,
             "OrderYesterday" => 0,
          "Total Order Value" => 59.98,
                    "message" => "\"I      462172\",825848,\"I\",\"      462172\",\"      417185\",\"O\",\"True\",09-Nov-20,59.98,67.78,0,\"      462172\",09-Nov-20,09-Nov-20,09-Nov-20,0,0,0,\"    GD\",\"Gary Dykstra\",\" GS\"\r",
            "Order Complete?" => "True",
         "Total Dollar Value" => 67.78,
                        "Loc" => " TS",
                "Salesperson" => "    GD",
                 "Order Date" => 2020-11-09T05:00:00.000Z,
                       "host" => "PC",
                      "Doc #" => 462172,
                   "Doc Type" => "I",
               "TotalShipped" => 0,
                      "Inv #" => "      462172",
                 "Order Type" => "O",
                       "path" => "D:/ELK/logstash-7.9.3/config/sortbyweek.csv",
                    "Order #" => 417185,
                  "Order Key" => "I      462172",
    "TotalOrderReceivedToday" => 0,
                 "@timestamp" => 2020-12-03T16:11:13.269Z,
               "Order Status" => "AlreadyInvoiced",
           "Salesperson name" => "salesperson2",
                   "@version" => "1",
         "Expected Ship Date" => 2020-11-09T05:00:00.000Z,
                   "Inv Date" => 2020-11-09T05:00:00.000Z,
              "Shipment Date" => 2020-11-09T05:00:00.000Z,
           "ShippedYesterday" => 0
}
7

OK, the [Shipment Date] field exists, but has the value nil. You can test for that using

    if ! [Shipment Date] {
        mutate { add_field => { "Order Status" => "Pending" } }
    }
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.