Wez Furlong
10dbdf1cb7
When load testing the v2 timerwheel, I was able to produce a panic when rebinding; the panic was caused by linking an already linked list through the associated messages. Digging into the state management, I believe that the issue is due to a data race between the rebind (and other similar drain operations) and the timer wheel dispatch. The timer wheel first locks to pop the currently due items, unlocks, and then resolves and inspects the per-scheduled-queue hash sets to fully de-queue the items. The drain operation locks both the wheel and the hash set and atomically removes its elements. Critically, the drain removal doesn't know whether the messages that were in its hash set either were in the linkage for the timeq while it held the lock, or were recently removed by the timer wheel dispatch and are pending the resolve and insertion mentioned above. What this commit does is make it possible to reason about that linkage by introducing a `cancel` method on the timeq. It will return true/false to indicate whether the message was unlinked. To facilitate this, we need to add an index to map the message id to the entry within the timeq, and a way to invalidate that entry if a message is removed and re-scheduled. Since we want the cancellation to be ostensibly an O(1) operation, we implement it with another source of indirection; the entry slot id. Each time an entry is inserted into a bucket slot we increment its entry id. When we subsequently iterate over the slots we can resolve that id to its message id and from there resolve its entry. If that entry points back to the entry slot that we just used to discover it, then that entry is current. Otherwise, it is a stale entry and we can skip over (or remove) it. Cancellation is therefore a simple O(1) removal of the msg id -> entry mapping and doesn't need to directly intrude into the wheel buckets. With a cancellation function in place, it means that the per-scheduled-queue drain operation can now atomically (while both the hash set and the wheel locks are held) reconcile the hash set contents with the corresponding wheel contents while it cancels them: if the cancellation fails then it means that the wheel runner has de-queued that particular message and that it will shortly (after we release the lock on the hash set) then try to check it against the hashset. That means that we need to retain those particular messages in the hash set, but drain the rest.
KumoMTA
KumoMTA is an open-source Message Transfer Agent (MTA) designed for high-performance outbound email functionality, similar to commercial enterprise MTAs such as Momentum, PowerMTA, and Halon.
The KumoMTA project was founded by a group of email industry veterans with decades of experience building and managing high-performance On-Prem MTAs and is supported by a community of some of the largest senders in the world.
Because it is designed for high-performance sending environments, KumoMTA is for experienced email operations professionals who are accustomed to high-performance sending environments and familiar with DevOps practices.
Learn more in our FAQ and at https://kumomta.com/.
Documentation
You can learn more about KumoMTA from the Documentation.
Community
Real-time discussion is available on Our Discord.
Developers
If you are interested in contributing/extending KumoMTA, take a look at DEVELOPERS.md. The #devel channel on Our Discord is for contributors to discuss KumoMTA development.
Reporting Bugs
See How to Report Bugs.
Getting Help
See How to Get Help. For paid support see https://kumomta.com/support.
Getting Updates
You can subscribe to updates at https://kumomta.com/subscribe, we send updates periodically and will never sell nor share your information.
Talk to Us
We're available to talk about the project, book us at https://cal.com/team/kumomta/talk-with-kumomta.