Every widget tap runs as its own headless task in the same JS runtime,
and the tasks interleave at every await. Each one loaded the stored
snapshot, edited it and saved it back, then refreshed from the server.
That went wrong in three ways:
- two taps loaded the same snapshot and the second save dropped the
first change;
- the refresh of the first tap read the server before the second tap's
archive landed and was stored anyway, so the message came back (seen
on the emulator with a slow `Email/set`);
- a redraw of an older snapshot could land after a newer one, leaving
buttons that act on a message the widget no longer shows.
The snapshot now only holds what the server last said. A button's
change is kept as a pending op next to it and laid over it when a
widget draws; it stays until a refresh that started after the server
took it has been stored (calendar and task changes only count as caught
up by a refresh that loaded calendar and tasks). A call that fails
drops its op, which also undoes an archive made offline: before, the
failed refresh kept the edited snapshot and the message stayed hidden
although it was never archived. Reads, writes and redraws of the
widget data run one at a time, and every redraw draws the latest view.
react-native-android-widget registers every button's PendingIntent with
the current time in milliseconds as its request code and
FLAG_CANCEL_CURRENT. All buttons of a widget share the same action and
provider, so the request code is the only thing telling them apart: two
buttons registered within one millisecond got the same intent, and the
second registration cancelled the first. That button then did nothing
when tapped. The emulator takes 3-10 ms per button, a current phone well
under one, so which button died depended on timing.
The library is patched with patch-package (run on postinstall) to hand
out request codes that count up from the time, unique in the process
and clear of the codes an earlier process left on the launcher. The
patch keeps the library's CRLF line endings, hence `-text` for
`patches/*.patch`.
App store terms put limits on recipients, such as device caps and DRM,
that section 10 of the AGPL does not allow, so the app could not be
shipped through the Apple App Store or Google Play without breaking the
licence.
Add a "Bulwark Mobile" header and a section 7 additional permission to
LICENSE that allows app store distribution as long as the source of
every shipped version is public, free, linked from the listing or the
app, and not further restricted.
The permission is not final yet: not every earlier contributor has
answered the consent request. A reservation limits it to contributions
made on or after 30 September 2026 and to older parts whose authors
have agreed. The README licence line says the same.
Brings feat/device-sync into main: on Android, each account's address
books and calendars can sync both ways with the phone's Contacts and
Calendar apps (Settings -> Contacts / Calendar -> Sync to this
device), through a "Bulwark" Android account with a contacts and a
calendar sync adapter.
- native side: the account, both sync adapters, provider access scoped
to the account, push routing to the syncs, sync problem notices
- the sync engine in `src/device-sync`, run as a headless task
- the settings sections, sign-out and removal handling, translations
- design, limitations and verification in `docs/device-sync.md`
The fixes since the first device runs were checked on the emulators
too: moves that stay the same event, statuses, server deletions, items
moved away before a deletion, quiet runs, the permission flow,
uninstalling and reinstalling, and the settings' collection count. The
list of what only tests cover shrinks accordingly.
The device runs also showed that Stalwart stores a calendar's default
alerts on the series, so removing every reminder of an occurrence is
put back and reported for such a series as well, and that Google
Calendar's first edit of an occurrence brings its reminders back
without a report.
The planner tests that build meetings of 25 attendees on 52 overrides
take 0.3-1.8 s alone, but one of them passed the default 5 s limit in
a full run next to a release build. The two slowest tests of the file
already had 60 s; the other heavy ones get the same.
The account row in the sync settings counts the synced address books
or calendars from its own copy of the list, loaded when the row
appears. The chooser loads a fresh list when it opens, so a calendar
created meanwhile showed there but not in the count ("4 of 5 sync"
stayed after a sixth appeared, and turning it on or off changed
nothing) until the settings page was opened again.
Every fresh list now reaches every view of that account's collections,
so the count follows the chooser.
Brings the doc up to the code after the second round of fixes and the
code review: moves of meetings, deletions of items another client
moved away, big meetings written in several transactions, occurrence
colours and reminders, the stored status of runs that did nothing,
push types by server capability, reminders while the phone's calendar
can't sync, and sign-out waiting for a turn-on in progress.
The known limitations gain what the fixers and the device runs found
but left alone: pairs whose patch fails, a field the server changes
during a pending move, Google Calendar's copy-and-delete moves and its
copied reminders, group cards deleted after they moved away, and a
status line that shows item errors but not conflicts.
A contact or event deleted on the phone that another client meanwhile
filed in an address book or calendar the phone doesn't sync was
destroyed on the server by the phone's next sync, and the other
client's move with it. 0faee56 made the sync decide such a deletion on
the server's version, but the planners still destroyed it: the
contacts planner saw the card in no synced book and destroyed it, and
the calendar planner could not take the new version at all (it placed
the event before looking at the deletion), so the old one decided.
The deletion is now moot when the server's version holds none of the
collections the rows were written for and none the phone syncs: only
the rows go, nothing is uploaded, and the object stays where the other
client put it. A deletion still goes up when the book or calendar was
deselected on the phone before the upload, and when the object was
moved into another collection the phone syncs, since the phone would
otherwise miss an object it should show. The calendar planner now lets
a deleted row's shadow follow the server before it places the event.
A sync that had to start React Native left a listener on the React
host until a context came up. When the start failed (a debug build
without Metro) or never finished, every retried sync added another
one, each holding its run, account and extras, for the life of the
process, and they all fired at once when a context finally came up.
The wait now ends once, and removes its listener, for whatever
reason: a ready context, a start task that faulted, or 10 minutes
(no sync or push job waits longer; JobScheduler stops them). A
context that comes up later runs nothing of it. The adapter's own
check of its run in the callback still covers a run it gave up
sooner.
Turning device sync on retried with the app account's registry id as
the Android account name whenever adding the account failed. That
fallback is for one case only: the login name turned out to belong to
another app account's Android account, which the native module
reports with the `conflict` code. A passing AccountManager failure or
a permission error instead created a second account named after the
opaque id, which then showed in the Contacts and Calendar apps.
Only a `conflict` rejection takes the fallback now; any other failure
fails the turn-on as before.
Turning device sync on or off runs one step after the other per
account, but signing out did not wait its turn. A sign-out that
started while a turn-on was still setting up (or queued behind a slow
turn-off) found no Android account yet and went ahead; the turn-on
then created the Android account, stored it and requested a sync for
the signed-out account, and the next check posted "Sign in to keep
syncing". A turn-off still tearing down could likewise store its
preferences after the sign-out had forgotten the account.
The release before a sign-out is now a step in the queue of every
account it releases, taken in a fixed order: it waits for the steps
under way and releases what they set up. A turn-on or turn-off asked
for while the sign-out runs comes after it, and does nothing once the
account was released; if the user stays signed in it goes ahead.
With calendar sync on and the calendar app reminding, Bulwark skipped
its own reminders for the synced calendars whenever sync was on in the
app, even when the phone's calendar received nothing: its calendar sync
paused in Android's account settings, Android's auto-sync off, or the
calendar permission revoked. Events created on the server meanwhile
never reached the phone, and nobody reminded of them.
The reminder filter now keeps what Android last said about each
account (the permission, `getMasterSyncAutomatically` and the
calendar's `getSyncAutomatically`) and stays quiet only while the
phone's calendar receives events. Auto-sync off counts as not
receiving: Android then runs manual syncs only, and the periodic,
pushed and app-triggered syncs are not manual. Android's side is read
when the scheduler subscribes and again whenever the filter is asked
for; a changed answer makes the scheduler reschedule. Until it was
read, Bulwark reminds: a double reminder is better than a missed one.
ContactsProvider soft-deletes every raw contact and group of a sync
account, also one that never reached the server (DELETED=1, SOURCE_ID
NULL). When a contact or group was created and deleted on the phone
before a run claimed it, no upload phase took the row: without an
identity or a pending create it belongs to no JMAP account, and the
claim step skipped deleted rows. The row stayed for good, and the
upload-only check and the teardown counted it as a waiting change:
every upload sync the providers schedule became a full run, and
turning sync off or signing out always asked about a change that would
be lost.
The claim step now hands such a row to the planner, whose plan for it
is a purge, and `hasLocalWork` and `countPending` leave out deleted
rows that have neither an identity nor a pending create. Nothing is
uploaded for them. Calendar has no such gap: CalendarProvider
hard-deletes an app-deleted event without `_SYNC_ID`, and a claimed
event carries `~pending/<uid>`, so it is soft-deleted with its pending
create and destroyed by uid as before.
The native push router and the push task decided separately whether a
push may carry mail, and disagreed. For an account entry that is not
an object, or a map that names no type, the router started the mail
task as possible mail, but the task's guard read the same payload as
"no mail" and returned, so new mail it announced was never looked for.
A push without a kind was judged by its map in the router but taken
for mail by the task.
Both now apply the same rules: anything but a StateChange, a map that
is missing, unreadable or names no type, an unreadable account entry,
or a mail type means the mail task runs. Only a readable StateChange
naming contact, calendar or other non-mail types skips it.
Checked with one table of payloads run against the router's decision
and the task's guard alike.
On main every push ran the mail task, whose legacy path notifies the
newest unread inbox mail. Device sync narrowed that to pushes naming
EmailDelivery, so contact and calendar changes never notify. But a
subscription made by an older build listens to Email and Mailbox until
the app is opened again, and a server without EmailDelivery reports
new mail only as those changes. Their new mail no longer showed a
notification while the app was closed.
Email and Mailbox now count as possible mail again, in the native push
router and in the push task alike. A push that names only contact or
calendar types still never takes the mail path.
Checked with the push router and push task tests: an Email or Mailbox
change starts the task and the legacy path notifies the message,
next to the syncs a mixed map asks for.
The push subscription got the contact and calendar types whenever the
account had device sync on, without asking whether its server has
contacts or calendars. Sync can be on for such an account: the settings
list every signed-in account, and a server can drop a capability later.
A server that refuses unknown types then refused the update and the
replacement, and push setup failed: a new account got no mail push, an
existing one lost it when its subscription expired.
The types are now added only for a capability the account's session
advertises, and a write the server refuses while it carries device
sync's types is retried with EmailDelivery alone, so mail push never
depends on device sync.
Checked with the push setup tests against a fake server that refuses
unknown types: a new subscription is created mail-only, an existing
one is kept and refreshed, and another account's subscription is left
alone when its server lacks the capability.
The settings show the stored status of the last sync. A sync that
finds nothing to do still replaced it, so a change the server refused
and the phone put back, or a conflict, was shown for a second or two
and then gone. The previous fix covered only the upload sync Android
starts right after a manual one. When changes arrive during a sync,
the adapter runs a full sync right after it, which finds nothing new
on the server and replaced the status 1-2 s after the real one; a
manual sync right after another does the same.
Now what counts is what a run did, not how it started. A run that
wrote nothing either way and reported no conflict, item error, note or
failure keeps the details of the run before it and moves only the time
on. After a failed run it shows the sync as working again, without the
failure's message. Any other run replaces the status, so a problem
stays only until the next run that does something.
Etar moves an event by inserting a copy from its editor and deleting
the old row. Its editor keeps the user's own attendee row out of the
attendee list, so the copy of a meeting that lists the user has no row
for the user, and the copy of one that does not gets the user as its
organizer. The pairing compared the attendee addresses of both rows
exactly, so a moved meeting never paired: it was deleted on the server
and created again under a new uid, the other attendees' answers were
lost, and everyone got a cancellation and a new invitation.
The pairing now leaves the user's own addresses out of that
comparison. Anyone else still has to be on both rows, so a copy that
lost or gained another attendee is not taken for the same event.
Checked with planner tests for both kinds of meeting and a run through
the engine with a fake Etar move.
The fake providers and the fake server were more lenient than the
real ones, so three bugs showed only on a device: an event inserted
without a status, small contact photos, and meetings too big for one
provider transaction.
The fake CalendarProvider leaves STATUS NULL when an app writes none.
The fake ContactsProvider keeps a photo within the 96 px thumbnail as
the thumbnail alone, without a PHOTO_FILE_ID, and serves it to
readPhoto from there. A batch fails with tooLarge when it weighs more
than one Binder transaction, a photo counted by its bytes. The fake
server drops empty maps inside recurrence overrides, as Stalwart does.
Tests now read photos the way the engine does, and the helpers that
made up for the old fakes are gone.
Android starts an upload-only sync right after a manual sync, and 30 s
after any app write. When nothing waited on the device it ended at
once, but it still stored its status, so the settings showed a plain
"synced" and lost the conflicts and item errors of the run just
before it.
Such a run did nothing and now leaves the stored status alone.
Clearing the colour of one occurrence of a coloured series removed the
occurrence's own colour, so it showed the series' colour again after
the sync. The occurrence now gets an empty colour, which Stalwart keeps
and which shows as the calendar's colour, as on the phone.
Removing every reminder of one occurrence sent an empty set of alerts,
which Stalwart does not store: the occurrence then showed the series'
reminders again, without a word. When the series uses the calendar's
default reminders, the occurrence leaves the defaults and has none, as
before. When the series has reminders of its own, the occurrence would
show them whatever is sent, so the change is put back and the report
says why (`remindersNotRepresentable`).
Checked against calcard 0.3.14, with planner tests and runs through
the engine and the fake server.
A recurring meeting with many attendees on many occurrences (25 on 52,
100 on 12) is about 2 MB of provider operations, and one Android
provider transaction carries 1 MB. Every event was written as one
group that the engine never splits, so such a meeting was refused every
run and never reached a phone that synced it for the first time.
An item too big for one group now comes as a chain of groups that the
engine applies one by one, each in a transaction of its own. A new
event is written master first, with its identity, then its occurrences
in groups that fit, then its shadow; a write into an existing event
writes the occurrences first and the series with its new shadow last.
When a run stops between two groups, the next one finds the event by
its identity and writes only what is missing, planned from the old
shadow, so no row is written twice and no server change is lost.
Checked with planner tests for both meeting sizes and every cut point,
and runs through the engine with a provider that refuses transactions
over 1 MB, a crash after every batch included.
When another client moved an event into a calendar this device does
not sync while the device had an unsent edit of it, the calendar
planner refused to merge the server's version, since the event is in
no synced calendar any more. The edit then waited on every run as a
planner error, and the event never left the device.
The server's version now merges into the rows where they are, so the
edit uploads against it without undoing the other client's changes,
and the rows go once the server has taken the edit.
Checked with a planner test and a run through the engine and the fake
server: the series rescheduled on the device keeps the occurrence the
other client added, and no rows are left.
The device sync doc still described the feature as first built. The
review fixes since then changed pairs, merges of items outside the
synced collections, the order of uploads, group handling, photos, the
provider-op scope checks, the settings actions and what the report
says about changes put back on the phone.
Each fix's doc notes are applied against the code as it is now, the
known limitations gain what the device runs found (an identity as
organizer, Stalwart refusing answers in read-only calendars, Fossify
Calendar around DST changes, AOSP's one-line address edits), and a new
Verification section lists what was checked on emulators and the scale
run's numbers.
The strings of device sync were English only: the "Sync to this
device" sections in the Contacts and Calendar settings with their
dialogs, and the notifications about signing in again and about sync
problems. Each language keeps the form of address its file already
uses, the app's own terms, and Android's names for its settings,
auto-sync and the Contacts and Calendar apps.
Checked: every message parses as ICU with the same arguments as the
English one and the plural categories of its language,
`node scripts/rn-missing-translations.mjs <lang>` lists no device sync
key for any language, `npm run i18n:check` and the i18n tests pass.
A contact added on the phone to a group in a read-only address book
(one shared with the user) could never reach the server, yet it stayed
waiting on every run: the planner puts such a membership back, but the
engine never told it which groups are read-only, so only groups of
another account were recognised. Turning sync off or signing out then
always asked about a change that could never go up.
The engine now tells the planner which synced groups are in read-only
books only, from the group cards it holds and the books' rights. The
membership is put back and reported, and nothing is left waiting.
A change the server cannot take was put back on the phone without a
word: an edit in a read-only calendar or address book, an answer
where answers are not allowed, an event moved to another account's
calendar, an occurrence the server keeps without its series, a
contact added to a group this device can't change. The change just
vanished.
The planners now say why they put a change back, and the sync report
lists the item with that reason (`readOnly`, `rsvpRefused`,
`instanceOnly`, `crossAccountMove`, `groupNotWritable`). A contact
whose only change still uploads through a group card is not listed.
A refused edit was also reported as "invitations not sent", as if it
had been stored without them; the note now appears only when the
change was stored without its scheduling messages.
An event created on the phone in a calendar the user may only read was
meant to be removed, since the server takes no changes there, but the
claim step ignored that answer: the event stayed on the phone, was
never sent or reported, and turning sync off always warned that a
change had not reached the server.
The claim step now applies it: the row goes, the report lists it as
read-only, and nothing is left waiting.
Etar moves an event by deleting the row and inserting a copy. When the
event was deleted on the server meanwhile, the sync removed the deleted
row but kept the copy, which then uploaded as a new event: the event
the server had deleted came back in the target calendar, for everyone.
That happened whether the download found the event gone first, which
is the usual case, or the patch of the move did.
A move is an edit, and the server's deletion wins over an edit: the
copy that pairs with the deleted row now goes with it, and the report
counts the conflict. A copy edited beyond the move stays a new event.
Checked through the engine with the event destroyed before the run,
while the move was sent, and with the patch answered notFound.
Etar moves an event, or turns a series into a single event, by deleting
the row and inserting a copy. The pair uploaded the move as a patch of
the event, but then took the copy as it stood for the new baseline and
wrote the server's version over it, so everything else saved with the
move was dropped: a changed reminder came back, and an edit made before
the move never reached the server. The synced occurrences, which Etar
does not copy, were deleted with the old row, losing an occurrence edit
that was still waiting.
The copy now starts from the old row's baseline with only what the move
changed, so any other difference stays dirty and uploads right after
the move as an ordinary update. The old row's occurrences move to the
new row with what they wait to upload. Attendee rows of the copy are
the app's re-insert (Etar writes everyone as required, without a
status) and are not read as edits.
Checked with planner tests and runs through the engine and the fake
server: a reminder saved with the move and a pending occurrence edit
both reach the server, the event keeps its uid, and the echo writes
nothing.
When another client moved a card out of every address book the device
syncs while the device deleted the contact, the engine noted the card
for dropping but never showed its new version to the planner, so the
deletion was decided on the old copy and destroyed the card. A deleted
item is now merged like any changed one first; the contacts planner
then leaves a card that is in no synced book. When the planner cannot
place the object, the deletion stays as it was.
Turning sync off uploads what the device changed without downloading
first; every upload carries the state the device last synced, so the
server refuses it when something changed meanwhile and the engine
downloads first. Before the first sync of an account had ended there
was no such state, and edits and deletions went up unguarded, able to
overwrite what another client had changed. They now wait, and the app
asks before they are dropped; new items still go up.
Turning sync off first uploads what the device changed, then removes
the synced rows. Started from the settings screen, that work yielded
the JS thread with timers between its steps, and React Native pauses
timers while the app is in the background and no headless task runs:
leaving the app stopped it halfway, holding the account's sync lock,
until the app came back. The teardown now waits without timers.
The batch budget counted JSON characters, but a Parcel carries strings
as UTF-16 plus a few hundred bytes per operation, so a full batch was
about 700 KB of a 1 MB Binder buffer the whole process shares. Batches
are now weighed at two bytes per character plus 300 bytes per op and
kept under 500 KB of that. The state op no longer joins a batch whose
last item already fills a provider yield window of 499 operations,
which ContactsProvider refused.
When a group's members change on the server, the download fetches the
contacts whose membership flipped in a chunk after the group's. The
group's chunk already stored its new version, so when the sync stopped
before the members' chunk, the next sync found no flip any more and
the phone kept the old members. Those contacts are now listed with the
group's rows, as objects to fetch first, until their own chunk ran.
A full reconcile that ran out of time resumes after the last object it
processed, in the sorted order of every listed id. It listed the
collections selected at the time it resumed and marked them all loaded
at its end, so a book turned on in between lost the objects that sorted
before that point, and was never loaded again. The reconcile now keeps
the collections it started with; one selected since is loaded after it
like any newly selected collection.
A sync with nothing to download, and the upload before sync is turned
off, used to plan uploads with an empty group index, so the write after
the upload deleted the uploaded contact's membership rows on the phone.
The previous commit's refresh of the index before planning uploads
fixes both; these tests cover the two paths so it stays fixed.
Checked: both tests fail without that refresh.
Linus Rath