kumomta/docs/reference/kumo/make_queue_config.md

kumo.make_queue_config { PARAMS }

Constructs a configuration object that specifies how a queue will behave.

This function should be called from the get_queue_config event handler to provide the configuration for the requested queue.

The following keys are possible:

egress_pool

The name of the egress pool which should be used as the source of this traffic.

If you do not specify an egress pool, a default pool named unspecified will be used. That pool contains a single source named unspecified that has no specific source settings: it will just make a connection using whichever IP the kernel chooses.

See kumo.make_egress_pool().

max_age

Limits how long a message can remain in the queue. The default value is "7 days".

kumo.on('get_queue_config', function(domain, tenant, campaign, routing_domain)
  return kumo.make_queue_config {
    -- Age out messages after being in the queue for 20 minutes
    max_age = '20 minutes',
  }
end)

max_message_rate

{{since('2024.06.10-84e84b89')}}

Optional string.

Specifies the maximum permitted rate at which messages can move from this scheduled queue and into the ready queue for the appropriate egress source.

The value is of the form quantity/period where quantity is a number and period can be a measure of time.

Examples of throttles:

"10/s" -- 10 per second
"10/sec" -- 10 per second
"10/second" -- 10 per second

"50/m" -- 50 per minute
"50/min" -- 50 per minute
"50/minute" -- 50 per minute

"1,000/hr" -- 1000 per hour
"1_000/h" -- 1000 per hour
"1000/hour" -- 1000 per hour

"10_000/d" -- 10,000 per day
"10,000/day" -- 10,000 per day

Throttles are implemented using a Generic Cell Rate Algorithm.

If the throttle is exceeded the message will be re-inserted into the scheduled queue with a delay based on the acceptance rate of the throttle.

This option is distinct from the egress path max_message_rate option in that this one applies to a specific scheduled queue, whilst the egress path option applies to the ready queue for a specific egress path, through which multiple scheduled queues send out to the internet.

If you have configured max_message_rate both here and in an egress path, the effective maximum message rate will be the lesser of the two values; both constraints are applied independently from each other at different stages of processing.

max_retry_interval

Messages are retried using an exponential backoff as described under retry_interval below. max_retry_interval sets an upper bound on the amount of time between delivery attempts.

The default is that there is no upper limit.

The value is expressed in seconds.

kumo.on('get_queue_config', function(domain, tenant, campaign, routing_domain)
  return kumo.make_queue_config {
    -- Retry at most every hour
    max_retry_interval = '1 hour',
  }
end)

protocol

Configure the delivery protocol. The default is to use SMTP to the domain associated with the queue, but you can also configure delivering to a local maildir, or using custom lua code to process a message

Example of smart-hosting with the SMTP protocol

{{since('2023.08.22-4d895015')}}

Rather than relying on MX resolution, you can provide an explicit list of MX host names or IP addresses to which the queue should deliver. The addresses will be tried in the order specified.

kumo.on('get_queue_config', function(domain, tenant, campaign, routing_domain)
  if domain == 'smarthost.example.com' then
    -- Relay via some other internal infrastructure.
    -- Enclose IP (or IPv6) addresses in `[]`.
    -- Otherwise the name will be resolved for A and AAAA records
    return kumo.make_queue_config {
      protocol = {
        smtp = {
          mx_list = {
            'smart.host.local',
            { name = 'mx.example.com', addr = '10.0.0.1' },
          },
        },
      },
    }
  end
  -- Otherwise, just use the defaults
  return kumo.make_queue_config {}
end)

Example of using the Maildir protocol

kumo.on('get_queue_config', function(domain, tenant, campaign, routing_domain)
  if domain == 'maildir.example.com' then
    -- Store this domain into a maildir, rather than attempting
    -- to deliver via SMTP
    return kumo.make_queue_config {
      protocol = {
        maildir_path = '/var/tmp/kumo-maildir',
      },
    }
  end
  -- Otherwise, just use the defaults
  return kumo.make_queue_config {}
end)

!!! note Maildir support is present primarily for functional validation rather than being present as a first class delivery mechanism.

Failures to write to the maildir will cause the message to be delayed and retried approximately 1 minute later. The normal message retry schedule does not apply.

Using Lua as a delivery protocol

kumo.on('get_queue_config', function(domain, tenant, campaign, routing_domain)
  if domain == 'webhook' then
    -- Use the `make.webhook` event to handle delivery
    -- of webhook log records
    return kumo.make_queue_config {
      protocol = {
        custom_lua = {
          -- this will cause an event called `make.webhook` to trigger.
          -- You can pick any name for this event, so long as it doesn't
          -- collide with a pre-defined event, and so long as you bind
          -- to it with a kumo.on call
          constructor = 'make.webhook',
        },
      },
    }
  end
  return kumo.make_queue_config {}
end)

-- This event will be called each time we need to make a connection.
-- It needs to return a lua object with a `send` method
kumo.on('make.webhook', function(domain, tenant, campaign)
  -- Create the connection object
  local connection = {}

  -- define a send method on the connection object.
  -- The return value is the disposition string for a successful
  -- delivery; that string will get logged in the resulting log record.
  -- If the delivery failed, you can use `kumo.reject` to raise the
  -- error with an appropriate 400 or 500 code.
  -- 400 codes will be retried later. 500 codes will log a permanent
  -- failure and no further delivery attempts will be made for the message.
  function connection:send(message)
    print(message:get_data())
    if failed then
      kumo.reject(400, 'failed for some reason')
    end
    return 'OK'
  end

  return connection
end)

See should_enqueue_log_record for a more complete example.

retry_interval

Messages are retried using an exponential backoff. retry_interval sets the base interval; if a message cannot be immediately delivered and encounters a transient failure, then a (jittered) delay of retry_interval seconds will be applied before trying again. If it transiently fails a second time, retry_interval will be doubled and so on, doubling on each attempt.

The default is "20 minutes".

kumo.on('get_queue_config', function(domain, tenant, campaign, routing_domain)
  return kumo.make_queue_config {
    retry_interval = '20 minutes',
  }
end)

reap_interval

{{since('dev')}}

Optional duration string. The default is "10m". It controls how long the queue should remain empty and idle before we reap it from the queue management layer and free its associated resources and metrics.

refresh_interval

{{since('dev')}}

Optional duration string. The default is "1m". It controls how long the queue should wait before refreshing the configuration for that queue by triggering the get_queue_config event.

strategy

{{since('dev')}}

Optional string to specify the scheduled queue strategy. There are two possible values:

• "TimerWheel" - the default. The timer wheel has O(1) insertion and O(1) pop costs, making it good for very large scheduled queues, but that comes in exchange for a flat periodic tick overhead. As the number of scheduled queues increases and/or the retry_interval decreases, so does the aggregate overhead of maintaining timerwheel queues.
• "SkipList" - A skiplist has O(log n) insertion and O(1) pop costs, making it a good general purpose queue, but becomes more expensive to insert into as the size of the queue increases. That higher insertion cost is in exchange for the overall maintenance being cheaper, as the queue can go to sleep until the time of the next due item. The ongoing and aggregate maintenance is therefore cheaper than a TimerWheel but the worst-case scenario where the destination is not accepting mail and causing the scheduled queue to grow is logarithmically more expensive as the queue grows.

Which should you use? Whichever works best for your environment! Make sure that you test normal healthy operation with a lot of queues as well as the worst case scenario where those queues are full and egress is blocked.

If you have very short retry_interval set for the majority of your queues you may wish to adopt SkipList for its lower idle CPU cost, or alternatively use TimerWheel and find a timerwheel_tick_interval that works for your typical number of queues.

!!! note Changing the strategy for a given queue requires that the queue either be aged out, or for kumod to be restarted, before it will take effect. This restriction may be removed in a future release.

timerwheel_tick_interval

{{since('dev')}}

When using the default strategy = "TimerWheel", the timer wheel needs to be ticked regularly in order to promote messages into the ready queue. The default tick interval is computed as retry_interval / 20 and clamped to be within the range >= 1s && <= 1m.

If you have a short retry_interval and a lot of scheduled queues you may find that your system is spending more time ticking over than is desirable, so you can explicitly select the tick interval via this option.

The value is an optional string duration like 1m.

If you have to set this, our recommendation is generally for this to be as long as possible.

!!! note The maintainer will also tick over whenever the refresh_interval elapses, so there isn't a tangible benefit to setting timerwheel_tick_interval larger than refresh_interval.