A new contact is claimed for an address book before its create is
sent, so a retry after a lost response finds it again. When that book
was deleted on the server before the create went up, every attempt
failed and was retried after up to a day, forever. A book that is gone
can hold nothing of an earlier attempt, so the contact is now claimed
again for the book the settings choose.
An event cannot move on its own to another calendar: a create into a
calendar that is gone is no longer sent, and the report says why, until
the user moves the event in the calendar app.
A group whose latest server version the device could not store waits
on the stale list, and nothing uploads for it until it has been fetched
again. Membership edits bypassed that check: adding a contact to such a
group sent the whole member list as the device last knew it, which
removed members another client had added meanwhile. Membership edits
for a stale group now wait for that fetch, and the report lists them.
Whether a membership edit still has to go up is decided against the
group cards on the server, which the engine indexes from their shadows.
The index was only read before planning a download that held contacts,
so in a run that downloaded no contact, such as the upload sync after
an edit, it was empty: removing a contact from a group looked already
done on the server, the contact was marked clean and the removal was
never sent. The engine now reads the index before planning uploads too.
A contact stays marked as changed until the group cards on the server
show its membership edits. The engine uploaded those edits as patches
of the group cards after the contacts, but never looked at the contact
again in that run, so it stayed marked until the next sync. Turning
contacts sync off or signing out right after adding a contact to a
group therefore asked whether to drop one waiting change, although
nothing was waiting.
Once the membership patches are accepted, the contacts that have
nothing else to upload are now cleared in the same run, behind the
same version check as any other clean write.
The contacts SyncState lists the group rows on the device, so a group
an app hard-deleted (Fossify does) is found by its absence and deleted
on the server. The list changed only with a later state op, not with
the batch that inserted or deleted the rows. When a run stopped in
between (cancelled, out of time, a crash, or turning sync off while a
deselected book was being dropped), a group the engine had removed was
still listed, taken for a hard delete by the next run or by the
teardown's upload, and its card destroyed on the server; a group it had
inserted was never listed, so a later hard delete never went up.
A batch that inserts or deletes a group row now ends with a state op
holding that change, retries included, so the list and the rows never
disagree. A group a download finds unchanged is listed too, for rows
whose batch lost its state op at a provider yield point. And an absent
group whose card is no longer a group or is in no synced book is only
forgotten, never destroyed.
Checked with a crash at every checkpoint of a deselection, a teardown
and a deadline stop during one, a server-side move in a page of several
chunks, and a group written before an interruption and then deleted in
Fossify.
When another client moved a contact or an event out of every address
book or calendar the device syncs while the device had an unsent edit
of it, the download only noted it for dropping after the upload. Its
shadow stayed at the old version while the stored state moved past the
change, so the upload passed `ifInState` and resent whole maps built
from the old shadow: a number or an occurrence added on the server
meanwhile was lost. A full reconcile that found such an object did the
same.
The download now merges the new version into the dirty rows like any
other change, so the upload builds on it, and the rows still go once
it is accepted. A contact's collections are read from its shadow:
SYNC1 lists only the selected books and is empty for a card that left
them all, so such a contact was never dropped after its upload. The
calendar planner cannot merge an event that is in no synced calendar
yet; until it can, such an event waits on the device and is reported
instead of overwriting the server.
Checked with the real planners through `/changes` and through a full
reconcile.
Etar moves an event to another calendar, and turns a series into a
single event, by deleting the row and inserting a copy. The engine was
meant to pair the two rows and upload a patch of the existing event,
but it claimed every new row (a uid and a pending create) before it
asked the planner for pairs, and the planner pairs only unclaimed rows.
No pair ever formed: the event was destroyed and created again under a
new uid, which lost what the device cannot show (keywords, virtual
locations, links, the other attendees' answers, alerts relative to the
end) and sent every attendee a cancellation and a new invitation.
Pairs now upload before new rows are claimed, and a paired deletion no
longer counts toward the deletion threshold, since it is a move. When
an app changes the rows while the patch is on its way, they are read
and paired again; rows that no longer pair are left without the
identity instead of having the server's version written into the new
row, which overwrote the edit and left two rows with one identity.
Checked with the real planners: a move and a series turned single keep
their uid, keywords and attendee answers and send nothing; an edit made
during the upload is kept; an unchanged rewrite still pairs.
"Android account settings" opened Android's page for the Bulwark
account through `ACCOUNT_SYNC_SETTINGS` with the account as the
`account` extra. The Settings app of Android 12L and later passes only
`:settings:show_fragment_args` on to that page, which then found no
account and closed at once ("Account provided does not exist: null").
Since starting it succeeded, the fallback to the list of synced
accounts never ran, and the button seemed to do nothing.
The account now also goes in the fragment arguments, where those
Settings apps read it; older ones still read the plain extra.
Checked on the API 32 and API 34 emulators: the button opens the
account's "Account sync" page with its Contacts and Calendar switches.
A contact created on the device and put into a group before its first
sync lost that group: the write after its create took the rows' own
memberships as what the server already had, found them missing there
and deleted the rows, so the group card was never patched.
The create's write now starts from no memberships, as its fallback
already did: the device's memberships count as additions, stay on the
device, keep the contact dirty until the group cards show them, and
upload as patches of the group cards.
Checked with a planner test and end to end through the engine.
ContactsProvider keeps a photo within its 96 px thumbnail as the
thumbnail alone, without a PHOTO_FILE_ID, and the thumbnail does not
read back. Such a Photo row looked empty to the planner. Every
download of the card inserted the photo again (two, then three
rows), and any edit of the contact on the device deleted the photo on
the server.
A Photo row now always counts as the contact's photo. Photo rows that
earlier downloads added again are deleted by the contact's next clean
write, one row is kept, and nothing reaches the server for them. A
photo set on the device is still uploaded.
Checked with planner tests and end to end through the engine, with a
provider that stores small photos as thumbnails as the device does.
A contact's rows are written in one provider batch, and one batch is
one Binder transaction of at most 1 MB. A contact with a larger photo
failed with `tooLarge` on every run, stayed on `stale`, never showed
on the device, and its device edits never uploaded.
Photos above about 512 KiB are now kept on the server only: the
contact is written without the photo. The shadow records that the
device holds no row for the card's photo, so the missing row is not
read as a deletion. The same applies to a photo whose bytes could not
be fetched, which an unrelated edit of the contact used to delete from
the server. A photo set on the device still replaces the server's, and
one deleted there is still deleted on the server.
Checked with planner tests for the insert, the size limit, a photo
that could not be fetched and a merge, and end to end through the
engine with the probe's 900 KB photo.
Types Android has but RFC 9553 lacks uploaded as a label naming them,
yet every label read back as a custom type, and MMS and company main
read back as Other and Main. A number typed Car showed up as
"Custom: car" right after the sync, an MMS number as Other, a mobile
email as "Custom: mobile" and a blog link as "Custom: blog".
Such a label now names its type again, for phones only next to the
feature the type uploads with, so a custom label stays custom. MMS
uploads its `mms` label next to `text`, which keeps a text-capable
number from reading as MMS, and `main-number` with `work` reads as
company main.
Checked with a round trip of every phone, email and website type,
both for a contact created on the device and for a type changed on
a synced entry.
Accepting a membership patch of a group card wrote the server's title
over the device's and cleared DIRTY. A rename that had not reached the
server in the same run, because the server refused it or the user made
it after the group's own upload was planned, was silently undone.
The accepted write now keeps a pending rename that the accepted card
does not show: it stores the new shadow but leaves the title and
DIRTY, so the rename uploads against it. A card that shows the
device's title, or a different one from a concurrent server rename,
is taken as before.
Checked with a planner test and end to end through the engine with a
refused rename and a membership change in the same run.
AOSP Contacts edits a postal address as one formatted line; the
provider then copies it into the street and empties the other parts.
The upload sent only the new `full`, so the components on the server
kept the old values: CardDAV clients and the app's editor, which show
the parts, still showed the old city.
A changed line now uploads with the parts it implies. They are read
back along the old line: each of its segments must be some of the old
parts, and a segment whose change is confined to one part rewrites
that part like an in-place edit of its field. When the layout changed
or a change spans two parts, the address keeps the whole line as its
street, which is what the device holds, rather than an address
without parts that the app's editor would show empty and drop on its
next save. An unchanged line still uploads nothing, and editors that
write the parts themselves are handled as before.
Checked with planner tests for the edit from the device test, a
street-only edit, the fallback, a re-inserted line and an editor
that keeps the parts, and with unit tests of the read-back.
A contact added to or removed from a group this device can't change
stayed dirty on every run: the membership upload skips such groups,
while the contact's clean write kept the change pending until the
server showed it. Turning sync off and signing out then always warned
about changes waiting for the server.
The clean write now puts such a membership back to what the server
has, as it does for any edit of a read-only contact, and clears DIRTY;
when that was the contact's only change the upload counts it as
skipped. Memberships of writable groups in the contact's own account
still wait until the group card shows them.
This covers groups of another JMAP account. For groups in read-only
address books the planner is ready, but the engine does not tell it a
group's rights yet; that follows separately.
Checked with planner tests for additions, removals, both kinds of
group and a change next to an edit that uploads, and end to end
through the engine for a group of a shared account, where turning
sync off afterwards finds nothing waiting.
A contact or group deleted on the device was uploaded against the
shadow of its last sync: a download into a deleted item wrote nothing.
When the server had meanwhile also filed the card in an address book
this device does not sync, the deletion still saw only the synced book
and destroyed the card, its copy in the other book included.
A download into a deleted contact or group now writes the new server
card into its shadow, and takes over an identity that was never
written, so deleting our own lost create goes by id rather than by a
uid lookup. The rows and the group title stay as they are: the device
delete still wins. The upload then removes only the synced memberships
of a card that is also in other books, as the calendar planner already
does for events.
Checked with planner tests for contacts and groups and end to end
through the engine with the real planner: the card stays in the
unsynced book, the device rows are purged, and an echo writes nothing.
After an edit on the phone, every write into a recurring event asserted
each attendee and reminder row of every occurrence. For a meeting with
25 attendees and 52 occurrences with details of their own that alone
made the write 1.6 MB, over Android's 1 MB transaction limit: the event
could no longer be written, went stale in every run and kept its edit.
Only rows that are dirty themselves now assert their attendees and
reminders. A clean row's DIRTY flag already catches an app edit of
them, since CalendarProvider sets it on every app write to an event,
its attendees or reminders, so such a write still fails and is planned
again. The same meeting's writes now take about 0.65 MB.
Checked with the meeting above through the planner, and with an app
edit of a clean occurrence's attendees between planning and writing.
An app that moves an event by changing its calendar in place can move
it into a calendar of another account on the device. A patch cannot
move an event between accounts, so the move was dropped; with nothing
else changed, the row was then marked clean where it stood. The phone
showed the event in the other account's calendar while the server kept
it in the old one, until the next server change moved it back.
The row now goes back to its calendar at once, rewritten from the last
server version. Changes made together with the move still upload, and
their accepted write puts the row back as before.
Checked with a run through the engine and the fake server with a shared
account: the event is back in its calendar and clean, and the server
and the other account are untouched.
In a calendar the user may only answer (mayRSVP without write rights),
an answer to the whole series was uploaded, but an answer to a single
occurrence goes into the occurrence's override and counted as an edit:
it was undone on the next sync without a word.
A change that holds nothing but the user's own answer, to the series
or to occurrences, now counts as an RSVP and is uploaded. Stalwart
0.16.23 refuses every update in a shared calendar without write rights,
answers included; the engine then puts the rows back from the server
and reports the refusal, as it already did for an answer to the series,
so the answer is no longer lost silently.
Checked with a planner test (a new and an existing override, and an
extra edit that is still reverted) and a run through the engine and
the fake server.
In a series that uses the calendar's default reminders, a reminder set
on one occurrence went into the occurrence's override, but the
occurrence still used the defaults it inherits from the series, which
ignore an event's own alerts. The phone showed the default reminder
again after the sync while the server held the new one.
The override now also leaves the defaults (`useDefaultAlerts: false`),
so only that occurrence uses its own alerts and the series keeps the
calendar's defaults. Stalwart stores the flag of an override with the
override (as a JSPROP) and serves it back.
Checked with a planner test (an existing override and a new "this
event" edit) and a run through the engine and the fake server.
Google Calendar leaves STATUS empty on the events and exceptions it
creates, and CalendarProvider has no default for it. Device sync read
the empty value as 0, STATUS_TENTATIVE, so every event and every "this
event" edit made there uploaded as "tentative" and came back to the
phone that way; a copy made with "Copy to" also never paired with the
deleted original.
An empty STATUS now sets no status: a create leaves `status` out, so
the server keeps its default of confirmed, an override gets none of its
own, an edited event never uploads one, a merge takes the server's
without counting a conflict, and a pair ignores it. After the upload
the rows hold the server's status.
A new series from Etar's "this and following" also no longer uploads
the old series' exclusions from before its start, which Etar copies
into the new row's EXDATE.
Checked with planner tests for each path and a run through the engine
and the fake server.
Clearing the description or location of one occurrence of a series in a
calendar app removed the value from the occurrence's override. An
override without a value of its own shows the series' one, so the write
after the upload put "Daily agenda" or "Room 1" back into the occurrence
and made it clean: the user's edit was gone without a trace.
An existing override now gets an explicit empty value while the series
has one to fall back to: the empty text for the description, and a
location without a name, the only empty forms Stalwart keeps (calcard
writes no LOCATION line for an empty map, which then reads back as the
series' locations). Without a series value the override's own is removed
as before. New exception rows keep treating an empty value as "inherit".
Checked with a planner test and a run through the engine and the fake
server: the row stays cleared and clean, and the echo writes nothing.
The native side ANDs the account's scope onto each selection the engine
sends, as `(where) AND (scope)`. A selection such as `1) OR (1` closed
the scope's parenthesis itself: on the calendar tables, where
CalendarProvider adds no account filter of its own, an update or delete
built that way would have reached the events, attendees and calendars
of other accounts, e.g. the user's Google calendar. The engine only
sends constant selections, so this guards against bugs rather than
anything an outside app could send, which is what the scope is for.
- Selections and sort orders must balance their parentheses outside
quoted literals, and may not end the statement, carry comments or
query other tables (`SELECT`, `UNION`, `ATTACH`, `PRAGMA`).
- An update of a data row by id that writes `data1` without restating
the mimetype has the row's mimetype looked up: on a membership row
`data1` names a group, which must be the account's, as on insert.
Such writes by selection are refused.
- `data14`, a photo row's file id, is never written: another account's
file id would have let `readPhoto` read that account's photo.
Checked: 58 JVM tests (`gradlew :app:testDebugUnitTest`, three new), the
debug build, and every selection the engine and planners send against
the new rules.
The design in f666699 was written before the code, and building it
changed a number of details, several of them after runs against the
real providers on the emulators. The doc now says what the code does:
- a sync run's steps in the order they run, the stale list, the
reconcile cursor, the owner check, the time margins, and accounts that
are left out when they refuse to list their collections;
- uploads in phases across accounts, the `ifInState` catch-up, account
refusals, series splits and copied events, creates adopting only in
their target collection, and poison markers;
- the contacts and calendar mappings as built (membership rows, text
limits, postal and relation rules, read-only rows and calendars,
exceptions, reminders under the old owner), and the columns the
providers refuse to return;
- the native module's scoping, the app's store, lifecycle and triggers,
and the failure matrix;
- decisions and limitations, with the questions still open.
Checked: every statement against the code of this branch.
The sync adapter logged only the SyncResult counters of a run, so a
failed sync showed up in logcat as `ioErrors=1` and nothing else. The
reason was in the run's report, and JS warnings from a headless run
don't reach logcat; finding out that a run failed on "Invalid column
raw_contact_is_read_only" took reading the app's storage. The log line
now carries the report's outcome, its message (up to 300 characters)
and the number of failed items.
Checked: 55 JVM tests (`gradlew :app:testDebugUnitTest`, one new) and
the debug build.
On Android, Settings > Contacts and Settings > Calendar get a "Sync to
this device" section with a row per signed-in account. This makes device
sync reachable:
- The switch asks for the contacts or calendar permission after a short
explanation (denied: sync stays off and says why, with a link to the
app's system settings once Android stops asking), creates the
account's Bulwark account in Android and syncs right away. Turning it
off uploads what the device changed first and asks before anything
that could not go up is lost. The switch shows Android's own state, so
sync turned off in Android Settings shows as paused.
- Which address books or calendars sync: the personal account's are on
by default, shared ones and the Trusted Senders book off; read-only
ones are marked, and a note says when another signed-in account
already syncs the same shared one.
- The address book new contacts from other apps go to, who reminds of
synced events (the calendar app or Bulwark, asked when calendar sync
is turned on while Bulwark's reminders are on), and how often Android
syncs on its own (every 15 minutes to every 6 hours, or not at all).
- "Sync now", the last sync with its result, or the error with what to
do about it ("Sign in again", "Grant access", "Review deletions"), and
a link to the account in Android Settings.
- Accounts the app signed out while they synced are listed with "Sign
in again" and "Remove from this device".
The settings search finds both sections. iOS, and Android builds without
the native module, show nothing new.
Checked: the collection listing and status tests, the settings search
and i18n tests, `npm run typecheck`, `npm test` and `npm run i18n:check`
on this commit, and on the API 32 emulator with usera on the local
server: the section, turning contacts sync on (Android account created,
152 contacts on the device, one per card) and the status after a run.
434c554 left an account that refuses to list its address books or
calendars out of the run, but reported it as an item error. The sync
settings then said "1 item could not be synced" after every run for an
account that merely shares a mailbox: nothing the user can fix, and no
item behind it. Such an account is now only logged.
Checked: the engine tests, `npm run typecheck` and `npm test` on this
commit, and the sync settings of usera on the API 32 emulator after a
new run.
The contacts planner read RAW_CONTACT_IS_READ_ONLY with every raw
contact, but ContactsProvider takes that column only on writes: every
projection and selection that names it fails with "Invalid column", on
API 32 and API 35 alike. Every contacts sync on a device therefore
failed before it wrote a row. The fake provider accepted the column, so
no test noticed.
The flag is now only written, along with any raw contact write that
happens anyway, so echoes stay write-free. Whether a contact's device
edits are put back is decided from the server's current rights on the
card, which is also right when only the rights changed. The fake now
refuses the column in queries and selections like the real providers,
and every other column the planners query was checked against the real
providers on both emulators.
Checked: the contacts, engine and fake provider tests, `npm run
typecheck` and `npm test` on this commit, and a contacts sync of usera
on the API 32 emulator, which now completes.
Stalwart lists an account shared with the user for mail only with every
capability, and answers `forbidden` when the user asks for its address
books or calendars. The engine treated that like any failed request, so
on the local test server every sync of usera ended with an I/O error and
never reached the device, only because userb shares a mailbox.
An account that refuses to list its collections (`forbidden`,
`accountNotFound`, `accountNotSupportedByMethod`) is now left out of the
run and reported as an item error; the other accounts sync as before.
The rows written for such an account stay on the device: a refusal can
pass, and dropping the account's contacts and events for one refused
request would throw away device edits that are still waiting.
Checked: three new engine tests (a shared account that never listed,
one that stops listing its address books, one that stops listing its
calendars), which fail without the fix, `npm run typecheck` and
`npm test` on this commit, and a contacts sync of usera on the API 32
emulator, which now ends `ok` with one item error.
When an account's calendars sync to the device and the user chose the
device's calendar app to remind them, that app gets the events'
reminders. Bulwark's own reminder scheduler now skips the events of
those calendars, which would otherwise remind the user twice. Tasks
never sync and keep Bulwark's reminders, and a change of who reminds, or
of which calendars sync, reschedules right away.
Checked: the reminder filter and scheduler tests, `npm run typecheck`
and `npm test` on this commit.
Signing out of an account (or of all of them, or removing one) now first
uploads what the device changed in its synced contacts and calendars,
then deletes them and removes its Android account, while the credentials
still work. When changes could not be uploaded (offline, server errors)
the app asks before signing out and losing them; staying signed in
keeps the account syncing. Several accounts are released in parallel,
and an internal error never keeps anyone from signing out: the next
check against Android's accounts stops what was left from syncing.
A successful sign-in clears the account's "Sign in again" notification
and resumes an account that stopped syncing when it was signed out.
Checked: `npm run typecheck` and `npm test` on this commit (the release
and resume paths are covered by the lifecycle tests of e67e240).
With device sync on, Android should hear about a change as soon as the
app does, not only at the next periodic sync. This adds what tells it:
- The app's own contact and calendar edits, StateChanges on the live
stream for `AddressBook`/`ContactCard` and `Calendar`/`CalendarEvent`,
and the app coming back to the foreground request a sync of that
account and authority. Requests within 5 s fold into one, and a
StateChange the engine caused itself (its own upload echoing back) is
skipped.
- While the app is closed, the account's push subscription carries
those types too. The FCM and UnifiedPush services hand every push to
the native router from f743ab0, which requests the syncs, and start
the mail push task only when a push has `EmailDelivery`; the task also
returns early for such pushes, so a contact change can never show the
"newest unread mail" notification.
- At launch, on foreground and when Android's accounts change, the app
checks Android's accounts against its own: an account removed in
Android Settings shows sync off and is not recreated, and one whose app
account is gone stops syncing and posts "Sign in again", keeping its
data.
The functions that turn sync on and off (`src/device-sync/app/lifecycle.ts`)
come with this; the settings that call them land in a later commit, so
nothing here runs before an account has device sync on.
Checked: the new store, lifecycle, trigger and push tests, `npm run
typecheck`, `npm test` and `npm run i18n:check` on this commit, and the
debug build.
This is the planner for contacts (`src/device-sync/contacts`), the last
piece the sync task was waiting for: it turns JSContact cards into raw
contacts and data rows, and device edits back into JMAP writes.
- Names, nicknames, organizations and titles, emails, phones, postal
addresses, websites, birthdays and anniversaries (also without a
year), relations, notes and photos map both ways, with their types and
custom labels. Group cards become groups, and their members become
group memberships.
- Device edits upload as patches of the fields whose rows changed; a
whole map goes up only where Stalwart needs one. Whatever a card holds
that Android can't show (pronouns, online services, keywords, a second
note, an address's coordinates) is never touched, so an edit on the
phone leaves it intact on the server.
- Rows are matched to their card's entries by the keys they carry and,
where an editor re-inserted them without (Fossify saves that way), by
their values. An entry the editor didn't show is never read as
deleted, and the keys Stalwart assigns by position to cards from other
clients are followed when they shift.
- Contacts in read-only address books are put back when the device
changes them. Text beyond what the provider stores is cut on the device
and never uploaded.
Checked: 151 contacts tests (mappings both ways, one test per edited
field, Fossify and AOSP editing, re-keyed cards, merges, groups,
read-only books, a full lifecycle), the engine's tests with both real
planners, and `npm run typecheck`, `npm test` and `npm run i18n:check`
on this commit.
The engine hands every event it syncs to a planner that turns it into
CalendarContract rows, and device edits back into JMAP writes. This is
that planner for calendars (`src/device-sync/calendar`), now used by the
sync task:
- Calendars become calendar rows with what the apps need (SYNC_EVENTS=1,
an owner, colours, the reminders allowed); calendars of shared accounts
carry the account's name.
- Events keep their time zone, floating and all-day times, recurrence
rules, excluded and changed instances, reminders, attendees and the
user's own reply. A rule Android can't hold (RSCALE, SKIP, leap months)
stays on the server untouched, and local times Stalwart can't resolve
(a start in a repeated hour, an UNTIL in a skipped one) are written in
a form it keeps.
- Device edits upload as patches of what differs from the last synced
state, never a whole event, and only what the device can show is ever
sent. A single-instance edit or deletion becomes an override of its
series; "this and following" caps the series and creates a new one.
How Etar and Fossify save (a cancelled instance instead of an
exclusion, attendee rows deleted and re-inserted) is read as the edit
it stands for, not as a deletion. Invitations go out on the changes a
calendar app would notify about.
- Events in read-only calendars and subscribed feeds are put back when
the device changes them; only the user's own reply goes up, where the
server allows it. Tasks, task-only calendars and the birthday calendar
never reach the device.
- Reminders are written as Reminders rows when the calendar app is to
remind the user, and left out when Bulwark does.
Checked: 179 calendar tests (download, upload, exceptions, the edits
Etar, Fossify and CONTENT_EXCEPTION_URI make, echoes that write nothing,
time zone and DST edges), and `npm run typecheck`, `npm test` and `npm
run i18n:check` on this commit.
The sync adapters from ed2525e start a headless task for every sync, but
the task only reported "disabled". This adds the engine it runs
(`src/device-sync/engine`, `src/device-sync/jmap`): for one Android
account and authority it downloads the server's changes, uploads what
the device changed, and hands a report back to SyncManager.
- Downloads follow `/changes` from the state kept in the provider's
SyncState. Rows and the state they reach are written in one batch, so
a killed run resumes where it stopped. Without a state, after
`cannotCalculateChanges` or after the user discarded deletions, a full
reconcile compares ids instead.
- Uploads go in three phases across accounts (creates, updates,
deletions), so moving an item to another account never destroys it
before the copy exists. A create carries a uid written to the device
first, and the server is asked for that uid before creating, so a lost
response or a killed run never makes a duplicate. Updates are patches
guarded by `ifInState`; nothing is replaced whole.
- More than 50 deletions that are also more than a fifth of the synced
items hold every deletion back and report `tooManyDeletions`, so
Android asks the user; "undo" restores the items from the server.
- An item the server refuses is marked and not sent again until it
changes or its back-off (1 hour, doubling to a day) ends. Such
refusals and planner errors stay with their item; the rest of the run
goes on.
- Runs of one account and authority take a lock, stop at checkpoints
when cancelled or close to the deadline, and never throw: every
failure becomes an outcome, and a rejected sign-in posts one "Sign in
again" notification.
- The JMAP side uses a client of its own per app account, never the
UI's, since a run can share the JS runtime with the app.
Turning sync off or signing out runs the same upload first
(`teardownAuthority`) and deletes the rows only once nothing is waiting,
or when the user agreed to lose it.
The planners that map items to rows land in the next commits; until
then a run fails before it touches a row.
Checked: 94 engine tests against the fake provider and JMAP server
(sync flows, failures, crashes and retries, teardown), and `npm run
typecheck`, `npm test` and `npm run i18n:check` on this commit.
The engine plans its provider reads and writes in JS, but only native
code can reach ContactsProvider and CalendarProvider. This adds that
bridge to the BulwarkDeviceSync module, with the rest of what the engine
and the app need from it:
- `query`, `applyBatch`, `readSyncState` and `readPhoto` take the
`ProviderQuery`/`ProviderOp[]` JSON of `src/device-sync/types.ts`,
check it (`sync/ProviderOps.kt`) and run it for one Android account of
our type (`sync/ProviderScope.kt`, `sync/ProviderIo.kt`). Every
selection gets the account ANDed in, every row an op names by id or
points to as a parent must be the account's, and every URI carries
`caller_is_syncadapter`, so nothing the sync writes marks rows dirty or
schedules an upload. The calls run one at a time on the module's own
thread through an unstable provider client, so a long batch never
stalls other module calls and a provider that dies fails the call
instead of the app.
- `showSyncProblem` and `clearSyncProblem` post and clear the one
notification per account that needs the user, e.g. "Sign in again".
- `setPushRoutes` stores which Android accounts each JMAP account feeds,
for the push router in `sync/PushRouting.kt`; the push services start
using it in a later commit.
- A run now ends exactly once and refuses a report that comes after
that. A sync requested while a run is open makes the run ask for
another sync when it ends, because SyncManager drops such requests as
duplicates. `onPerformSync` lets nothing escape any more: an exception
there killed the whole process, UI included.
- `ensureAccount` no longer hands an existing Android account to another
app account (`conflict`): its rows and SyncState belong to the first.
Payloads cross as JSON strings, which the bridge moves much faster than
nested maps. `src/device-sync/native.ts` is the typed JS side. The fake
provider follows the rules the emulator runs turned up: contacts
`data1`..`data14` read back as text, events and calendars the sync
adapter inserts have DIRTY NULL, attendee rows are addressed by id, an
op by id expects one row, and a vanished parent fails with `assert`
while another account's fails with `scope`.
Checked: 54 JVM tests (`gradlew :app:testDebugUnitTest`), debug and
release (R8) builds, `npm run typecheck` and `npm test` on this commit,
and on the API 32 emulator through the debug receiver: inserts with
back-references and group memberships, SyncState, photo writes and
scaled reads, asserts and expected counts, both scope refusals, the
499-op yield boundary, `tooLarge` for a 1.3 MB batch, and the
notification.
AOSP Contacts, and the apps built on its account-type parser, check the
EditSchema in `xml/contacts.xml` and drop the whole account type when it
breaks one of their rules: every data kind with a type column needs
`<Type>` children, the other kinds must have none, and name, photo,
nickname, organization, note and group membership need maxOccurs="1".
The schema from ed2525e listed phone, email, postal, event and
relationship without types, so the editors would not have offered the
Bulwark account at all.
The kinds now list the types device sync maps to JSContact, and the
rules are written down in the file so the next edit keeps them.
Checked on the API 32 emulator (AOSP Contacts): with a Bulwark account
present, "Create new contact" offers it and opens the editor saving to
it.
The device sync engine reaches Android's providers through a native
bridge and the server through a detached JMAP client, and neither runs
in vitest. This adds the stand-ins the engine and the mappers are tested
against, and the contracts they implement:
- `src/device-sync/types.ts`, `planner.ts`, `wire.ts`,
`android-columns.ts`: the bridge, the provider op model, the JMAP
port, and the interface between the engine and the contacts and
calendar planners
- a fake ContactsProvider and CalendarProvider with the behaviour the
design relies on: the DIRTY and VERSION rules, soft and hard deletes,
exception linking, normalisation, asserts, the yield-point limit, and
a "user" face that edits the way AOSP Contacts, Fossify and Etar do
- a fake Stalwart with `/changes` paging and compaction,
`cannotCalculateChanges`, `ifInState`, uid checks behind a lagging
search index, positional re-keying and fault injection (lost
responses, 429, auth, SetErrors)
- shared pure helpers (canonical JSON, PatchObjects with Stalwart's
missing-parent rule, key and uid minting) and a test that walks the
import graph, so no mapper or engine module pulls in React Native
- `src/stores/device-sync-store.ts`: the per-account preferences and
last run status the engine and the settings will share
Checked: `npx vitest run src/device-sync` (91 tests), the full suite
(2209 tests) and `npm run typecheck`.
Issue #34 asks for contacts and calendars that show up in Android's own
apps and whose edits there reach the server. docs/device-sync.md is the
design the implementation follows: Android's sync framework drives the
runs, a thin Kotlin shell starts a headless JS task, and the engine is
TypeScript on a detached JMAP client, so token refresh, mTLS and session
discovery keep a single owner. A pure-Kotlin engine was rejected: it
would have had to redeem rotating refresh tokens outside that owner and
re-implement the client.
It pins down what every provider column holds, the JSContact and
JSCalendar mapping tables, how device edits are told apart from provider
normalisation (a baseline per row), how rows are matched when an editor
re-inserts them, the merge and conflict rules, uploads as patches with
`ifInState`, the native API, the failure matrix and the limitations.
The platform and server facts it relies on come from AOSP's providers
and calendar/contacts apps, Fossify, Etar and Stalwart v0.16.23's
source. These were checked on the local test server:
- map keys written through JMAP survive patches; `uid` duplicates are
refused; unknown and plural recurrence properties are rejected
- a start or `until` in a DST gap or overlap is silently dropped (a
daily rule came back unbounded); `Etc/GMT-2` keeps both
- `ifInState` yields `stateMismatch`; `relatedTo/<key>: null` does not
delete; result references to `/set` are refused
An adversarial review of the first draft found 20 paths that could lose
data, duplicate items or loop (own creates duplicated after a lost
response, unsynced edits dropped on deselection or eviction, Fossify
rewrites uploaded as edits, Etar moves turned into destroy + create,
and more); the design was changed for each before any engine code.
Android syncs the contacts and calendars of other services through
accounts and sync adapters. This adds the Android side of device sync:
a "Bulwark" account type (`${applicationId}.account`, set in
build.gradle), an authenticator that sends Settings → Add account to
the app instead of asking for credentials, and one sync adapter each
for the contacts and the calendar provider.
An adapter does no syncing itself. It starts the BulwarkDeviceSync
headless JS task from the main thread, through the helper the push job
uses, waits for the task's report and hands it to SyncManager as the
SyncResult. The engine that will write the report runs in JS, so the
existing client stays the only owner of token refresh, mTLS and
session discovery; for now the task only reports that device sync is
off. Nothing in the app creates an account yet, so the feature stays
unreachable until its settings land. Debug builds get an adb hook for
the device tests (DeviceSyncDebugReceiver, shell only).
Hard errors are reported without progress: SyncManager retries a
failed sync at once when it made progress, which would loop an auth
failure.
Checked on the API 34 and API 32 emulators with the debug build, from
logcat and `dumpsys content`:
- app in the foreground: report after 319-334 ms
- app in the background: 320-327 ms
- process killed: SyncManager starts it for the sync service, React
Native starts headless, and the report arrives after 16.2 s (API 34)
and 8.5 s (API 32), most of it the debug bundle download from Metro
- an io report is retried with back-off and an auth report is not
retried; too many deletions shows the system's "Attempted to delete
too many Contacts." notification
- with Metro unreachable the adapter gives up after 2 s instead of
waiting for its deadline
- `gradlew :app:testDebugUnitTest`: 10 JVM tests for the report mapping
The code that starts React Native for a headless JS task, when the
process was woken without any UI, lived inside BulwarkPushJobService.
The device sync adapters need the same start, so it moves to a shared
HeadlessJs object, with a small startTask helper next to it.
Push behaviour does not change:
- onStartJob still calls the start code on the main thread, and the
listener-then-ready-check order is the same code, moved verbatim
- the task still starts from `UiThreadUtil.runOnUiThread`, its id is
still recorded in that same UI-thread turn, and a refusal because the
app came to the foreground still logs and finishes the job
- the start code now returns React Native's start task, so a caller can
notice a start that fails (a debug build with no Metro); the push job
ignores it
The helper documents that it must run on the main thread: the ReactHost
is created on first use without a lock, so a background caller racing
MainActivity could create a second one.
Checked with `gradlew assembleDebug -PreactNativeArchitectures=x86_64`.
Linus Rath