Configuring Traffic Shaping

By default, the KumoMTA server will deliver messages as quickly as possible to each destination, with few restrictions regarding number of connections, number of messages per connection, or number of messages per second. Because unthrottled sending is unwelcome by most MailBox Providers (MBPs) it is highly recommended that KumoMTA users implement traffic shaping in order to limit sending speeds to something more aligned with the expectations of the individual MBPs.

Common throttles include concurrent connection limits, messages per connection, rate of opening connections, and rate of sending messages. In addition, users can set options for various timeouts, queue sizes, and what encryption rules to use when communicating with remote hosts.

Scoping Traffic Shaping Rules

When KumoMTA needs to connect to a remote host to deliver messages, the get_egress_path_config event is fired in order to determine what configuration to use for that specific pathway.

kumo.on('get_egress_path_config', function(domain, egress_source, site_name)
  return kumo.make_egress_path {
    enable_tls = 'OpportunisticInsecure',
  }
end)

The event passes three attributes that are keyed to determine the desired traffic shaping rules, namely the domain, egress_source, and site_name.

Where the domain in the event call is the actual domain from the recipient address (for example, corp.com for a message destined to user@corp.com), the site_name is defined in the server by building an expression that represents all MX servers returned in an MX lookup for the domain. For example, if user@corp.com is hosted by Google Workspace then the site_name might be something similar to (alt1|alt2|alt3|alt4)?.aspmx.l.google.com.

The site_name concept allows managing traffic more effectively for MBPs that host a large number of domains. Rather than treating each domain as a separate destination with its own queues and traffic shaping counters, the traffic can be grouped and shaped as a single MX group, which is what the MBPs expect when they receive incoming traffic.

For example: if a sender wanted to limit connections to 10 per domain, and Google Workspace hosted 1,000 domains in the sender's queues, a server without MX rollup or sitenames would open 10,000 connections (1000 domains * 10 connections each). With the use of site_name, KumoMTA merges the 1,000 domains under a single site_name and maintains the limit of 10 connections per egress_source to the Google Workspace servers.

Messages in the Ready Queue are grouped into separate queues based on the combination of egress_source and site_name. The domain is provided for convenience when working out what parameters to use.

The `shaping.lua` Helper

While users are free to implement their traffic shaping rules as they see fit, the KumoMTA team has provided the shaping.lua helper as a pre-built implementation based on static configuration files in either JSON or TOML format, with support for various configuration scopes and automated rules using Traffic Shaping Automation.

The remainder of this section of the documentation is focused on the use of the shaping.lua helper.

Traffic Shaping Automation

Many of the largest MailBox Providers (MBPs) operate platforms that provide feedback to senders through their response codes during the SMTP conversation. This feedback will include information related to the traffic shaping patterns in use by the sending including bounces for too many connections, too many messages per connection, sending rate, and sender reputation.

To ensure optimum throughput and deliverability, KumoMTA features Traffic Shaping Automation (TSA) that monitors responses from the MBPs and adjusts traffic shaping rules on a granular level to ensure compliance with the guidelines of the MBPs in realtime.

TSA Architecture

To better support clustered installations, as well as to limit load on the primary kumod service, Traffic Shaping Automation is implemented via a standalone daemon called the kumo-tsa-daemon.service that starts automatically when its configuration is present.

The daemon monitors the events sent to it by the KumoMTA kumod process and instructs the kumod process to make adjustments to its traffic shaping rules according to user-defined actions.

There are three configuration locations required to implement TSA:

init.lua - The server's init.lua must be modified in order to properly interact with the TSA daemon.
tsa_init.lua - A configuration file that controls the behavior of the TSA daemon.
shaping.toml - A special automation entry is added to domain rules to power TSA adjustments.

Using The Shaping.lua Helper

The following steps must be followed to configure traffic shaping in KumoMTA.

Set Up Traffic Shaping Rules Files

The shaping.lua policy reads one or more configuration files in either TOML or JSON formats.

The Default shaping.toml File

By default, shaping.lua reads a shaping.toml file maintained by the KumoMTA team and included in all repos, available at https://github.com/KumoCorp/kumomta/blob/main/assets/policy-extras/shaping.toml, and which is structured as follows:

TOMLJSON

["default"]
connection_limit = 10
max_connection_rate = "100/min"
max_deliveries_per_connection = 100
max_message_rate = "100/s"
idle_timeout = "60s"
enable_tls = "Opportunistic"
consecutive_connection_failures_before_delay = 100

[provider."yahoo"]
match=[{MXSuffix=".yahoodns.net"}]
max_deliveries_per_connection = 20

[[provider."yahoo".automation]]
regex = "\\[TS04\\]"
action = "Suspend"
duration = "2 hours"

["foo.com"]
mx_rollup = false
max_deliveries_per_connection = 100
connection_limit = 3

{
  "default": {
    "connection_limit": 10,
    "max_connection_rate": "100/min",
    "max_deliveries_per_connection": 100,
    "max_message_rate": "100/s",
    "idle_timeout": "60s",
    "enable_tls": "Opportunistic",
    "consecutive_connection_failures_before_delay": 100,
  },
  "provider": {
    "yahoo": {
      "match": [
        {
          "MXSuffix": ".yahoodns.net",
        },
      ],
      "max_deliveries_per_connection": 20,
      "automation": [
        {
          "regex": "\\[TS04\\]",
          "action": "Suspend",
          "duration": "2 hours",
        },
      ],
    },
  },
  "foo.com": {
    "mx_rollup": false,
    "max_deliveries_per_connection": 100,
    "connection_limit": 3,
  },
}

As a special case, the domain can be named default, in which case those settings will apply globally. The global settings are superseded by the domain settings, which are superseded by the source settings.

The full set of available options is listed in the kumo.make_egress_path page of the Reference Manual.

The Community shaping.toml File

In addition, the KumoMTA Github repo includes a traffic shaping rules file maintained by the community, available at https://github.com/KumoCorp/kumomta/blob/main/assets/community/shaping.toml that can also be loaded explicitly as an additional resource for users.

Custom Shaping Files

Finally, users can opt to create their own shaping rules file that can augment or replace the rules defined in the previous two files.

If you intend to manually maintain your own shaping rules, we recommend using TOML as your file format, whereas if you intend to automatically maintain your traffic shaping rules, we recommend using JSON as your file format. Both example formats are listed above.

While you can place a custom shaping file at any readable location, it is common to place the file at /opt/kumomta/etc/policy/shaping.[toml|json] for consistency with examples used elsewhere in the documentation.

Order of Definition and Configuration File Precedence

The order in which your traffic shaping configuration files are defined affects how they are loaded and in turn which options are preserved when there is a conflict between the files.

The shaping.lua helper reads files sequentially, and the most recently defined file will overwrite any option set by a previously defined file.

Because of this, we recommend setting up shaping following this pattern, which will be explained later in this page:

local shaping = require 'policy-extras.shaping'

local shaper = shaping:setup_with_automation {
  publish = { 'http://127.0.0.1:8008' },
  subscribe = { 'http://127.0.0.1:8008' },
  extra_files = {
    '/opt/kumomta/share/policy-extras/shaping.toml',
    '/opt/kumomta/share/community/shaping.toml',
    '/opt/kumomta/etc/policy/shaping_custom.toml',
  },
}

Note

When a given scope is defined in multiple files, the more recently read file does not completely replace the configuration defined in the previous file, the options within that scope are merged.

If you want to completely replace the information for a given block, you can indicate that by using replace_base = true:

TOMLJSON

["gmail.com"]
# Discard any other `gmail.com` rules provided by earlier files
replace_base = true
connection_limit = 10

{
  "gmail.com": {
    // Discard any other `gmail.com` rules provided by earlier files
    "replace_base": true,
    "connection_limit": 10,
  },
}

replace_base is only meaningful in the context of the current domain section in the current file; subsequent sections for that same domain will continue to merge in as normal, unless they also use replace_base.

MX Rollups and Option Inheritance

By default, shaping.lua treats each domain entry as applying to the site_name generated for that domain, and those settings apply to any destination domain that also maps to the site. If you need to explicitly override a setting for a destination domain without consideration for the site_name, you need to set the mx_rollup option to false when configuring the domain.

If you configure a domain that belongs to a configured site without setting the mx_rollup option to false, you will cause an error.

Consider the following example, with foo.com being a domain hosted by Yahoo!:

TOMLJSON

["default"]
connection_limit = 10
max_connection_rate = "100/min"
max_deliveries_per_connection = 100
max_message_rate = "100/s"
idle_timeout = "60s"
enable_tls = "Opportunistic"
consecutive_connection_failures_before_delay = 100

["yahoo.com"]
max_deliveries_per_connection = 20

["foo.com"]
mx_rollup = false
max_deliveries_per_connection = 50
connection_limit = 3

["foo.com".sources."IP-1"]
max_deliveries_per_connection = 5

{
  "default": {
    "connection_limit": 10,
    "max_connection_rate": "100/min",
    "max_deliveries_per_connection": 100,
    "max_message_rate": "100/s",
    "idle_timeout": "60s",
    "enable_tls": "Opportunistic",
    "consecutive_connection_failures_before_delay": 100,
  },
  "yahoo.com": {
    "max_deliveries_per_connection": 20,
  },
  "foo.com": {
    "mx_rollup": false,
    "max_deliveries_per_connection": 50,
    "connection_limit": 3,
    "sources": {
      "IP-1": {
        "max_deliveries_per_connection": 5,
      },
    },
  },
}

This example would result in the following active settings for mail being sent to foo.com on the IP-1 Egress Source:

TOMLJSON

connection_limit = 3
max_deliveries_per_connection = 5
max_connection_rate = "100/min"
max_message_rate = "100/s"

{
  "connection_limit": 3,
  "max_deliveries_per_connection": 5,
  "max_connection_rate": "100/min",
  "max_message_rate": "100/s",
}

The mx_rollup option indicates whether or not the settings should apply to the domain or the site_name. In the example above, even though foo.com is hosted by Yahoo! we want to override the message throttle for the foo.com domain. The mx_rollup option is true by default and only needs to be specified for domains that override the main site name entry.

While the default max_deliveries_per_connection is 100, it is overridden for yahoo.com (and all domains that share the same site name as the yahoo.com domain) to 20. The foo.com domain is part of the same site name as yahoo.com, but because mx_rollup is set to false the foo.com domain is treated separately and instead is set to 50. Because there is a sources entry for IP-1, the max_deliveries_per_connection is further overridden to 5 for that source's traffic in particular.

Pattern Matching Rollups

Since: Dev Builds Only

The functionality described in this section requires a dev build of KumoMTA. You can obtain a dev build by following the instructions in the Installation section.

There are a number of mailbox providers that host multiple domains, but which do not provide consistent MX results across all hosted domains.

For those situations is is desirable to adopt pattern matching against the hostnames used by the provider's MX records, and employ connection limits and message rate throttles for all destination hosts that match.

You can configure this using a provider block in your shaping file(s).

For an example, let's consider Microsoft. Microsoft hosts two different consumer email domains (Outlook and Hotmail) as well as Office 365. While the consumer domains are run on the same servers, they have two different MX patterns:

dig +short mx hotmail.com
2 hotmail-com.olc.protection.outlook.com.
dig +short mx outlook.com
5 outlook-com.olc.protection.outlook.com.

We can see that the individual MX hostnames have the same .olc.protection.outlook.com suffix, so we can use that to identify the consumer MXes.

In addition, Microsoft has recently announced a change to the MX hostnames used by Office 365, where existing MX records will end in .mail.protection.outlook.com but any user who wishes to active DANE to enhance security is to use an MX hostname that ends in .mx.microsoft.

To address these three scenarios, we can use the following provider blocks:

TOMLJSON

[provider."outlook"]
match=[{MXSuffix=".olc.protection.outlook.com"}]
max_deliveries_per_connection = 50
provider_connection_limit = 5

[[provider."outlook".automation]]
regex = "temporarily rate limited due to IP reputation"
action = "Suspend"
duration = "1 hour"

[provider."office365"]
match=[{MXSuffix=".mail.protection.outlook.com"}]
max_deliveries_per_connection = 50
provider_connection_limit = 5

[provider."office365-dane"]
match=[{MXSuffix=".mx.microsoft"}]
enable_dane = true
max_deliveries_per_connection = 50
provider_connection_limit = 5

{
  "provider": {
    "outlook": {
      "match": [
        {
          "MXSuffix": ".olc.protection.outlook.com",
        },
      ],
      "max_deliveries_per_connection": 50,
      "provider_connection_limit": 5,
      "automation": [
        {
          "regex": "temporarily rate limited due to IP reputation",
          "action": "Suspend",
          "duration": "1 hour",
        },
      ],
    },
    "office365": {
      "match": [
        {
          "MXSuffix": ".mail.protection.outlook.com",
        },
      ],
      "max_deliveries_per_connection": 50,
      "provider_connection_limit": 5,
    },
    "office365-dane": {
      "match": [
        {
          "MXSuffix": ".mx.microsoft",
        },
      ],
      "enable_dane": true,
      "max_deliveries_per_connection": 50,
      "provider_connection_limit": 5,
    },
  },
}

Now, messages destined for either hotmail.com or outlook.com, or any other domain whose MX host names all have the suffix .olc.protection.outlook.com, will match the outlook provider block and have the options defined there applied, including an automation rule, and any Office 365 hosted domain using the new .mx.microsoft pattern will have DANE enabled.

The match field is an array and can list multiple match candidates. A provider block matches if any of the match elements are matched, as seen in this example:

TOMLJSON

[provider."google"]
match=[
        {MXSuffix=".google.com"},
        {MXSuffix=".googlemail.com"}
]
max_deliveries_per_connection = 50
provider_connection_limit = 5
consecutive_connection_failures_before_delay = 5
provider_max_message_rate = "100/s"

{
  "provider": {
    "google": {
      "match": [
        {
          "MXSuffix": ".google.com",
        },
        {
          "MXSuffix": ".googlemail.com",
        },
      ],
      "max_deliveries_per_connection": 50,
      "provider_connection_limit": 5,
      "consecutive_connection_failures_before_delay": 5,
      "provider_max_message_rate": "100/s",
    },
  },
}

The match can be one of two possible options:

{MXSuffix="SUFFIX"} - matches if ALL of the individual MX hostname suffixes match the specified suffix string.
{DomainSuffix="SUFFIX"} - matches if the domain name suffix matches the specified suffix string.

The provider block introduces two new options: provider_connection_limit and provider_max_message_rate. When a provider is defined, it does not merge the various site_name queues covered by the provider together, which means that the connection_limit and max_message_rate options will not be enforced across all matching queues, but will be applied separately to each ready queue covered by the provider block.

When the provider_connection_limit and provider_max_message_rate options are set, the throttles defined will be enforced across all matching site_name ready queues for that provider. This is typically the desired behavior. One example of a scenario where the provider_ options would not be used is Mimecast: each regional MX pattern used by Mimecast is a separate set of servers in that region, but traffic shaping expectations are the same for all regions. To address this we use a provider block without the provider_ throttles:

TOMLJSON

[provider."mimecast"]
match=[{MXSuffix=".mimecast.com"},{MXSuffix=".mimecast.co.za"},{MXSuffix=".mimecast-offshore.com"}]
max_deliveries_per_connection = 100
connection_limit = 10

{
  "provider": {
    "mimecast": {
      "match": [
        {
          "MXSuffix": ".mimecast.com",
        },
        {
          "MXSuffix": ".mimecast.co.za",
        },
        {
          "MXSuffix": ".mimecast-offshore.com",
        },
      ],
      "max_deliveries_per_connection": 100,
      "connection_limit": 10,
    },
  },
}

In this case we can define traffic shaping rules that apply to Mimecast globally, but which are still enforced by each region's ready queue without limiting worldwide traffic.

Shaping Option Resolution Order and Precedence

When resolving the configuration for a site, the options are resolved in the following order:

The values for the default domain block are taken as the base
Any matching provider blocks are then merged in
Any matching provider + source blocks for the current source are merged in
Any matching site name blocks are merged in. These are domain blocks that have the default (implied) or explicitly configured mx_rollup = true option set in them.
Any matching domain blocks are merged in. These are domain blocks that have mx_rollup=false set in them.
Any matching site name + source blocks are merged.
Any matching domain + source blocks are merged.

Within any of these steps above, the options are merged in the order that they appear across your configuration files, so the most recently specified value will take precedence overall.

You can specify replace_base=true in a block to have that block override the current set of accumulated values.

Warning

There is currently no mechanism for unsetting an option previously merged in. If there is a throttle set earlier (for example in [default]) that you wish to unset rather than explicitly define a different throttle then you must use replace_base=true to replace all previously merged options.

Most options merge directly over the top of earlier options, but the additional_connection_limits and additional_message_rate_throttles options merge the maps together.

Writing Your Own Traffic Shaping Rules

The /opt/kumomta/share/policy-extras/shaping.toml file provides a collection of traffic shaping rules provided by the KumoMTA team that are useful for new servers. In addition, a community-maintained set of traffic shaping rules is available at /opt/kumomta/share/community/shaping.toml.

The files listed above are maintained within the KumoMTA GitHub repository and are updated with each release, meaning that any local edits to these files will be lost any time the KumoMTA install is updated.

In addition, neither of these files are all-encompassing, you will likely encounter scenarios that require you to implement your own logic, either to address your specific reputation or to reflect specialized knowledge you have gained.

To maintain your own traffic shaping rules, create a separate file with your own traffic shaping rules in either TOML or JSON formats, typically called /opt/kumomta/etc/policy/custom-shaping.[toml|json] and pass it as part of the call to set up traffic shaping.

Test Your Shaping Files

Included in the standard deployment is a validation tool for testing the syntax of your shaping.toml override file. The file located at /opt/kumomta/sbin/validate-shaping can be used to validate the syntax of your shaping file. If there are no errors, it will return an "OK".

$ /opt/kumomta/sbin/validate-shaping /opt/kumomta/etc/policy/custom-shaping.toml
OK

Configure Traffic Shaping In Your `init.lua` Server Policy

Note

It's easiest to reference the Example Config to see how the complete configuration looks.

The server's init.lua file will require modifications to enable it to be used with TSA.

First, the following should be added to the start of the init.lua file, just below the initial local kumo = require 'kumo' line:

local shaping = require 'policy-extras.shaping'

local shaper = shaping:setup_with_automation {
  publish = { 'http://127.0.0.1:8008' },
  subscribe = { 'http://127.0.0.1:8008' },
  extra_files = {
    '/opt/kumomta/share/policy-extras/shaping.toml',
    '/opt/kumomta/share/community/shaping.toml',
    '/opt/kumomta/etc/policy/shaping_custom.toml',
  },
}

This section enabled communication with the TSA daemon. The publish and subscribe URLs correspond to the TSA daemon's HTTP listener endpoint defined in its tsa_init.lua. For a single node deployment the values shown here are sufficient. You may list multiple publish and/or subscribe endpoints to publish to multiple hosts and read shaping configuration from multiple hosts, respectively. In addition, while the setup_with_automation call is aware of the community shaping rules file, any custom file must be identified in the extra_files directive as seen in the example above.

Warning

As mentioned previously, your rules merge with the other files listed unless a given block has replace_base=true. To fully remove the defaults provided by the KumoMTA team you need the following:

local shaping = require 'policy-extras.shaping'

local shaper = shaping:setup_with_automation {
  publish = { 'http://127.0.0.1:8008' },
  subscribe = { 'http://127.0.0.1:8008' },
  no_default_files=true,
  extra_files = { 
        '/opt/kumomta/share/community/shaping.toml`, 
        '/opt/kumomta/etc/policy/shaping_custom.toml',
        },
}

Next, the following should be added within the kumo.on('init', function() block:

-- Configure publishing of logs to automation daemon
shaper.setup_publish()

This enables the logging required by the TSA daemon.

Finally, the following must be added outside the init event to enable the TSA manipulations:

-- Attach various hooks to the shaper
kumo.on('get_egress_path_config', shaper.get_egress_path_config)

Configure The `tsa_init.lua` File

The tsa_init.lua file controls the behavior of the TSA daemon, and should be written to /opt/kumomta/etc/policy/tsa_init.lua.

An example of the tsa_init.lua file is below:

local tsa = require 'tsa'
local kumo = require 'kumo'

kumo.on('tsa_init', function()
  tsa.start_http_listener {
    listen = '0.0.0.0:8008',
    trusted_hosts = { '127.0.0.1', '::1' },
  }
end)

local cached_load_shaping_data = kumo.memoize(kumo.shaping.load, {
  name = 'tsa_load_shaping_data',
  ttl = '5 minutes',
  capacity = 4,
})

kumo.on('tsa_load_shaping_data', function()
  local shaping = cached_load_shaping_data {
    '/opt/kumomta/share/policy-extras/shaping.toml',
    '/opt/kumomta/share/community/shaping.toml',
    '/opt/kumomta/etc/policy/custom-shaping.toml',
  }
  return shaping
end)

Note

The tsa_init.lua has no implicit loading of the default shaping.toml file. To avoid loading the default file simply omit it.

Monitoring the TSA Daemon

Adjustments to the traffic shaping rules are achieved by creating a custom shaping.toml file that is maintained by the TSA daemon and loaded as an overlay on the existing `shaping.toml file created by the user.

The generated TOML can be monitored by making an HTTP request. One example using curl:

$ curl -s 'http://localhost:8008/get_config_v1/shaping.toml'
# Generated by tsa-daemon
# Number of entries: 0

This call returns the current set of shaping rules in the same format as shaping.toml, the example is of an empty set.

Debugging Tips

If the tsa-deamon does not appear to be working, you can check to see if it is running with 'sudo systemctl status kumo-tsa-daemon' which should return a message that includes "active (running)". If not you can stop and start it in a similar way.

sudo systemctl stop kumo-tsa-daemon
sudo systemctl start kumo-tsa-daemon

Data for the TSA daemon is just like any other message in KumoMTA and will follow the same retry rules. The default is to retry in 20 minutes with exponential fallback. If desired, this (or any other) scheduled queue can be customized with the get_queue_config hook.

Clustering

There are special considerations when implementing traffic shaping in a clustered environment, see the Clustering Chapter for more information.

Configuring Traffic Shaping

Scoping Traffic Shaping Rules

The shaping.lua Helper

Traffic Shaping Automation

TSA Architecture

Using The Shaping.lua Helper

Set Up Traffic Shaping Rules Files

The Default shaping.toml File

The Community shaping.toml File

Custom Shaping Files

Order of Definition and Configuration File Precedence

MX Rollups and Option Inheritance

Pattern Matching Rollups

Shaping Option Resolution Order and Precedence

Writing Your Own Traffic Shaping Rules

Test Your Shaping Files

Configure Traffic Shaping In Your init.lua Server Policy

Configure The tsa_init.lua File

Monitoring the TSA Daemon

Debugging Tips

Clustering

The `shaping.lua` Helper

Configure Traffic Shaping In Your `init.lua` Server Policy

Configure The `tsa_init.lua` File