Turning sync off uploads what the device changed without downloading
first; every upload carries the state the device last synced, so the
server refuses it when something changed meanwhile and the engine
downloads first. Before the first sync of an account had ended there
was no such state, and edits and deletions went up unguarded, able to
overwrite what another client had changed. They now wait, and the app
asks before they are dropped; new items still go up.
Turning sync off first uploads what the device changed, then removes
the synced rows. Started from the settings screen, that work yielded
the JS thread with timers between its steps, and React Native pauses
timers while the app is in the background and no headless task runs:
leaving the app stopped it halfway, holding the account's sync lock,
until the app came back. The teardown now waits without timers.
The batch budget counted JSON characters, but a Parcel carries strings
as UTF-16 plus a few hundred bytes per operation, so a full batch was
about 700 KB of a 1 MB Binder buffer the whole process shares. Batches
are now weighed at two bytes per character plus 300 bytes per op and
kept under 500 KB of that. The state op no longer joins a batch whose
last item already fills a provider yield window of 499 operations,
which ContactsProvider refused.
When a group's members change on the server, the download fetches the
contacts whose membership flipped in a chunk after the group's. The
group's chunk already stored its new version, so when the sync stopped
before the members' chunk, the next sync found no flip any more and
the phone kept the old members. Those contacts are now listed with the
group's rows, as objects to fetch first, until their own chunk ran.
A full reconcile that ran out of time resumes after the last object it
processed, in the sorted order of every listed id. It listed the
collections selected at the time it resumed and marked them all loaded
at its end, so a book turned on in between lost the objects that sorted
before that point, and was never loaded again. The reconcile now keeps
the collections it started with; one selected since is loaded after it
like any newly selected collection.
A sync with nothing to download, and the upload before sync is turned
off, used to plan uploads with an empty group index, so the write after
the upload deleted the uploaded contact's membership rows on the phone.
The previous commit's refresh of the index before planning uploads
fixes both; these tests cover the two paths so it stays fixed.
Checked: both tests fail without that refresh.
The SyncState's `selected` said which collections were on the device,
and changed only when a load or a removal had finished. A book turned
off and on again while the device was still removing it looked loaded,
so the contacts removed so far never came back; a calendar came back
empty. A book turned on and off again while it was still loading never
looked loaded, so the contacts loaded so far were never removed.
`selected` now names only collections whose rows are all on the device:
a collection leaves it before its first row is removed, and the
contacts SyncState lists collections that are loaded or removed only in
part, so the next run loads them again when they are selected, or
removes the rest when they are not. A book that still holds a change
waiting for its upload stays in that list instead of `selected`.
The settings chooser and the reminder filter treat only the capability's
primary account as the user's own, while the engine also counted any
account the server marks as personal. On a server that lists a second
such account, its address books and calendars synced although the
chooser showed them off. The engine now uses the same rule as the app.
A provider that is busy may commit a batch up to a yield point and then
fail it. The engine retried every group of such a batch and relied on
the groups' asserts to refuse one that was already committed, but a
download into an unchanged event asserts only the event's own columns:
when it had only added a reminder or an attendee, the retry added it a
second time.
A download is now planned again from a new read after a failed batch
instead of being sent again, which turns a committed group into an echo.
The fake provider can commit a batch up to a yield point to test it.
When the server refused an edit made on the device (a book or calendar
that turned read-only, an event in a calendar where only one's own
events may be changed), the engine put the item back by deleting its
rows and inserting a fresh copy. A contact lost its star, ringtone and
the link to the contacts it was joined with; an event got a new row id.
The rows are now rewritten in place from the server's version, as after
an accepted upload. A refused deletion still brings the item back as a
new copy.
After an upload the stored state stays where the download left it, and
the engine relied on /changes to report its own creates back. /changes
leaves out an object that was created and destroyed after the state it
is asked from (Stalwart does, RFC 8620 allows it), so an event or
contact created on the device and deleted on the server before the next
download stayed on the device for good.
The SyncState now lists the objects our uploads created, written with
their identity, and the next run fetches them first: one that is gone
loses its rows, an unchanged one writes nothing. An upload-only sync
runs in full while the list is not empty.
A new contact is claimed for an address book before its create is
sent, so a retry after a lost response finds it again. When that book
was deleted on the server before the create went up, every attempt
failed and was retried after up to a day, forever. A book that is gone
can hold nothing of an earlier attempt, so the contact is now claimed
again for the book the settings choose.
An event cannot move on its own to another calendar: a create into a
calendar that is gone is no longer sent, and the report says why, until
the user moves the event in the calendar app.
A group whose latest server version the device could not store waits
on the stale list, and nothing uploads for it until it has been fetched
again. Membership edits bypassed that check: adding a contact to such a
group sent the whole member list as the device last knew it, which
removed members another client had added meanwhile. Membership edits
for a stale group now wait for that fetch, and the report lists them.
Whether a membership edit still has to go up is decided against the
group cards on the server, which the engine indexes from their shadows.
The index was only read before planning a download that held contacts,
so in a run that downloaded no contact, such as the upload sync after
an edit, it was empty: removing a contact from a group looked already
done on the server, the contact was marked clean and the removal was
never sent. The engine now reads the index before planning uploads too.
A contact stays marked as changed until the group cards on the server
show its membership edits. The engine uploaded those edits as patches
of the group cards after the contacts, but never looked at the contact
again in that run, so it stayed marked until the next sync. Turning
contacts sync off or signing out right after adding a contact to a
group therefore asked whether to drop one waiting change, although
nothing was waiting.
Once the membership patches are accepted, the contacts that have
nothing else to upload are now cleared in the same run, behind the
same version check as any other clean write.
The contacts SyncState lists the group rows on the device, so a group
an app hard-deleted (Fossify does) is found by its absence and deleted
on the server. The list changed only with a later state op, not with
the batch that inserted or deleted the rows. When a run stopped in
between (cancelled, out of time, a crash, or turning sync off while a
deselected book was being dropped), a group the engine had removed was
still listed, taken for a hard delete by the next run or by the
teardown's upload, and its card destroyed on the server; a group it had
inserted was never listed, so a later hard delete never went up.
A batch that inserts or deletes a group row now ends with a state op
holding that change, retries included, so the list and the rows never
disagree. A group a download finds unchanged is listed too, for rows
whose batch lost its state op at a provider yield point. And an absent
group whose card is no longer a group or is in no synced book is only
forgotten, never destroyed.
Checked with a crash at every checkpoint of a deselection, a teardown
and a deadline stop during one, a server-side move in a page of several
chunks, and a group written before an interruption and then deleted in
Fossify.
When another client moved a contact or an event out of every address
book or calendar the device syncs while the device had an unsent edit
of it, the download only noted it for dropping after the upload. Its
shadow stayed at the old version while the stored state moved past the
change, so the upload passed `ifInState` and resent whole maps built
from the old shadow: a number or an occurrence added on the server
meanwhile was lost. A full reconcile that found such an object did the
same.
The download now merges the new version into the dirty rows like any
other change, so the upload builds on it, and the rows still go once
it is accepted. A contact's collections are read from its shadow:
SYNC1 lists only the selected books and is empty for a card that left
them all, so such a contact was never dropped after its upload. The
calendar planner cannot merge an event that is in no synced calendar
yet; until it can, such an event waits on the device and is reported
instead of overwriting the server.
Checked with the real planners through `/changes` and through a full
reconcile.
Etar moves an event to another calendar, and turns a series into a
single event, by deleting the row and inserting a copy. The engine was
meant to pair the two rows and upload a patch of the existing event,
but it claimed every new row (a uid and a pending create) before it
asked the planner for pairs, and the planner pairs only unclaimed rows.
No pair ever formed: the event was destroyed and created again under a
new uid, which lost what the device cannot show (keywords, virtual
locations, links, the other attendees' answers, alerts relative to the
end) and sent every attendee a cancellation and a new invitation.
Pairs now upload before new rows are claimed, and a paired deletion no
longer counts toward the deletion threshold, since it is a move. When
an app changes the rows while the patch is on its way, they are read
and paired again; rows that no longer pair are left without the
identity instead of having the server's version written into the new
row, which overwrote the edit and left two rows with one identity.
Checked with the real planners: a move and a series turned single keep
their uid, keywords and attendee answers and send nothing; an edit made
during the upload is kept; an unchanged rewrite still pairs.
"Android account settings" opened Android's page for the Bulwark
account through `ACCOUNT_SYNC_SETTINGS` with the account as the
`account` extra. The Settings app of Android 12L and later passes only
`:settings:show_fragment_args` on to that page, which then found no
account and closed at once ("Account provided does not exist: null").
Since starting it succeeded, the fallback to the list of synced
accounts never ran, and the button seemed to do nothing.
The account now also goes in the fragment arguments, where those
Settings apps read it; older ones still read the plain extra.
Checked on the API 32 and API 34 emulators: the button opens the
account's "Account sync" page with its Contacts and Calendar switches.
A contact created on the device and put into a group before its first
sync lost that group: the write after its create took the rows' own
memberships as what the server already had, found them missing there
and deleted the rows, so the group card was never patched.
The create's write now starts from no memberships, as its fallback
already did: the device's memberships count as additions, stay on the
device, keep the contact dirty until the group cards show them, and
upload as patches of the group cards.
Checked with a planner test and end to end through the engine.
ContactsProvider keeps a photo within its 96 px thumbnail as the
thumbnail alone, without a PHOTO_FILE_ID, and the thumbnail does not
read back. Such a Photo row looked empty to the planner. Every
download of the card inserted the photo again (two, then three
rows), and any edit of the contact on the device deleted the photo on
the server.
A Photo row now always counts as the contact's photo. Photo rows that
earlier downloads added again are deleted by the contact's next clean
write, one row is kept, and nothing reaches the server for them. A
photo set on the device is still uploaded.
Checked with planner tests and end to end through the engine, with a
provider that stores small photos as thumbnails as the device does.
A contact's rows are written in one provider batch, and one batch is
one Binder transaction of at most 1 MB. A contact with a larger photo
failed with `tooLarge` on every run, stayed on `stale`, never showed
on the device, and its device edits never uploaded.
Photos above about 512 KiB are now kept on the server only: the
contact is written without the photo. The shadow records that the
device holds no row for the card's photo, so the missing row is not
read as a deletion. The same applies to a photo whose bytes could not
be fetched, which an unrelated edit of the contact used to delete from
the server. A photo set on the device still replaces the server's, and
one deleted there is still deleted on the server.
Checked with planner tests for the insert, the size limit, a photo
that could not be fetched and a merge, and end to end through the
engine with the probe's 900 KB photo.
Types Android has but RFC 9553 lacks uploaded as a label naming them,
yet every label read back as a custom type, and MMS and company main
read back as Other and Main. A number typed Car showed up as
"Custom: car" right after the sync, an MMS number as Other, a mobile
email as "Custom: mobile" and a blog link as "Custom: blog".
Such a label now names its type again, for phones only next to the
feature the type uploads with, so a custom label stays custom. MMS
uploads its `mms` label next to `text`, which keeps a text-capable
number from reading as MMS, and `main-number` with `work` reads as
company main.
Checked with a round trip of every phone, email and website type,
both for a contact created on the device and for a type changed on
a synced entry.
Accepting a membership patch of a group card wrote the server's title
over the device's and cleared DIRTY. A rename that had not reached the
server in the same run, because the server refused it or the user made
it after the group's own upload was planned, was silently undone.
The accepted write now keeps a pending rename that the accepted card
does not show: it stores the new shadow but leaves the title and
DIRTY, so the rename uploads against it. A card that shows the
device's title, or a different one from a concurrent server rename,
is taken as before.
Checked with a planner test and end to end through the engine with a
refused rename and a membership change in the same run.
AOSP Contacts edits a postal address as one formatted line; the
provider then copies it into the street and empties the other parts.
The upload sent only the new `full`, so the components on the server
kept the old values: CardDAV clients and the app's editor, which show
the parts, still showed the old city.
A changed line now uploads with the parts it implies. They are read
back along the old line: each of its segments must be some of the old
parts, and a segment whose change is confined to one part rewrites
that part like an in-place edit of its field. When the layout changed
or a change spans two parts, the address keeps the whole line as its
street, which is what the device holds, rather than an address
without parts that the app's editor would show empty and drop on its
next save. An unchanged line still uploads nothing, and editors that
write the parts themselves are handled as before.
Checked with planner tests for the edit from the device test, a
street-only edit, the fallback, a re-inserted line and an editor
that keeps the parts, and with unit tests of the read-back.
A contact added to or removed from a group this device can't change
stayed dirty on every run: the membership upload skips such groups,
while the contact's clean write kept the change pending until the
server showed it. Turning sync off and signing out then always warned
about changes waiting for the server.
The clean write now puts such a membership back to what the server
has, as it does for any edit of a read-only contact, and clears DIRTY;
when that was the contact's only change the upload counts it as
skipped. Memberships of writable groups in the contact's own account
still wait until the group card shows them.
This covers groups of another JMAP account. For groups in read-only
address books the planner is ready, but the engine does not tell it a
group's rights yet; that follows separately.
Checked with planner tests for additions, removals, both kinds of
group and a change next to an edit that uploads, and end to end
through the engine for a group of a shared account, where turning
sync off afterwards finds nothing waiting.
A contact or group deleted on the device was uploaded against the
shadow of its last sync: a download into a deleted item wrote nothing.
When the server had meanwhile also filed the card in an address book
this device does not sync, the deletion still saw only the synced book
and destroyed the card, its copy in the other book included.
A download into a deleted contact or group now writes the new server
card into its shadow, and takes over an identity that was never
written, so deleting our own lost create goes by id rather than by a
uid lookup. The rows and the group title stay as they are: the device
delete still wins. The upload then removes only the synced memberships
of a card that is also in other books, as the calendar planner already
does for events.
Checked with planner tests for contacts and groups and end to end
through the engine with the real planner: the card stays in the
unsynced book, the device rows are purged, and an echo writes nothing.
After an edit on the phone, every write into a recurring event asserted
each attendee and reminder row of every occurrence. For a meeting with
25 attendees and 52 occurrences with details of their own that alone
made the write 1.6 MB, over Android's 1 MB transaction limit: the event
could no longer be written, went stale in every run and kept its edit.
Only rows that are dirty themselves now assert their attendees and
reminders. A clean row's DIRTY flag already catches an app edit of
them, since CalendarProvider sets it on every app write to an event,
its attendees or reminders, so such a write still fails and is planned
again. The same meeting's writes now take about 0.65 MB.
Checked with the meeting above through the planner, and with an app
edit of a clean occurrence's attendees between planning and writing.
An app that moves an event by changing its calendar in place can move
it into a calendar of another account on the device. A patch cannot
move an event between accounts, so the move was dropped; with nothing
else changed, the row was then marked clean where it stood. The phone
showed the event in the other account's calendar while the server kept
it in the old one, until the next server change moved it back.
The row now goes back to its calendar at once, rewritten from the last
server version. Changes made together with the move still upload, and
their accepted write puts the row back as before.
Checked with a run through the engine and the fake server with a shared
account: the event is back in its calendar and clean, and the server
and the other account are untouched.
In a calendar the user may only answer (mayRSVP without write rights),
an answer to the whole series was uploaded, but an answer to a single
occurrence goes into the occurrence's override and counted as an edit:
it was undone on the next sync without a word.
A change that holds nothing but the user's own answer, to the series
or to occurrences, now counts as an RSVP and is uploaded. Stalwart
0.16.23 refuses every update in a shared calendar without write rights,
answers included; the engine then puts the rows back from the server
and reports the refusal, as it already did for an answer to the series,
so the answer is no longer lost silently.
Checked with a planner test (a new and an existing override, and an
extra edit that is still reverted) and a run through the engine and
the fake server.
In a series that uses the calendar's default reminders, a reminder set
on one occurrence went into the occurrence's override, but the
occurrence still used the defaults it inherits from the series, which
ignore an event's own alerts. The phone showed the default reminder
again after the sync while the server held the new one.
The override now also leaves the defaults (`useDefaultAlerts: false`),
so only that occurrence uses its own alerts and the series keeps the
calendar's defaults. Stalwart stores the flag of an override with the
override (as a JSPROP) and serves it back.
Checked with a planner test (an existing override and a new "this
event" edit) and a run through the engine and the fake server.
Google Calendar leaves STATUS empty on the events and exceptions it
creates, and CalendarProvider has no default for it. Device sync read
the empty value as 0, STATUS_TENTATIVE, so every event and every "this
event" edit made there uploaded as "tentative" and came back to the
phone that way; a copy made with "Copy to" also never paired with the
deleted original.
An empty STATUS now sets no status: a create leaves `status` out, so
the server keeps its default of confirmed, an override gets none of its
own, an edited event never uploads one, a merge takes the server's
without counting a conflict, and a pair ignores it. After the upload
the rows hold the server's status.
A new series from Etar's "this and following" also no longer uploads
the old series' exclusions from before its start, which Etar copies
into the new row's EXDATE.
Checked with planner tests for each path and a run through the engine
and the fake server.
Clearing the description or location of one occurrence of a series in a
calendar app removed the value from the occurrence's override. An
override without a value of its own shows the series' one, so the write
after the upload put "Daily agenda" or "Room 1" back into the occurrence
and made it clean: the user's edit was gone without a trace.
An existing override now gets an explicit empty value while the series
has one to fall back to: the empty text for the description, and a
location without a name, the only empty forms Stalwart keeps (calcard
writes no LOCATION line for an empty map, which then reads back as the
series' locations). Without a series value the override's own is removed
as before. New exception rows keep treating an empty value as "inherit".
Checked with a planner test and a run through the engine and the fake
server: the row stays cleared and clean, and the echo writes nothing.
The native side ANDs the account's scope onto each selection the engine
sends, as `(where) AND (scope)`. A selection such as `1) OR (1` closed
the scope's parenthesis itself: on the calendar tables, where
CalendarProvider adds no account filter of its own, an update or delete
built that way would have reached the events, attendees and calendars
of other accounts, e.g. the user's Google calendar. The engine only
sends constant selections, so this guards against bugs rather than
anything an outside app could send, which is what the scope is for.
- Selections and sort orders must balance their parentheses outside
quoted literals, and may not end the statement, carry comments or
query other tables (`SELECT`, `UNION`, `ATTACH`, `PRAGMA`).
- An update of a data row by id that writes `data1` without restating
the mimetype has the row's mimetype looked up: on a membership row
`data1` names a group, which must be the account's, as on insert.
Such writes by selection are refused.
- `data14`, a photo row's file id, is never written: another account's
file id would have let `readPhoto` read that account's photo.
Checked: 58 JVM tests (`gradlew :app:testDebugUnitTest`, three new), the
debug build, and every selection the engine and planners send against
the new rules.
The design in f666699 was written before the code, and building it
changed a number of details, several of them after runs against the
real providers on the emulators. The doc now says what the code does:
- a sync run's steps in the order they run, the stale list, the
reconcile cursor, the owner check, the time margins, and accounts that
are left out when they refuse to list their collections;
- uploads in phases across accounts, the `ifInState` catch-up, account
refusals, series splits and copied events, creates adopting only in
their target collection, and poison markers;
- the contacts and calendar mappings as built (membership rows, text
limits, postal and relation rules, read-only rows and calendars,
exceptions, reminders under the old owner), and the columns the
providers refuse to return;
- the native module's scoping, the app's store, lifecycle and triggers,
and the failure matrix;
- decisions and limitations, with the questions still open.
Checked: every statement against the code of this branch.
The sync adapter logged only the SyncResult counters of a run, so a
failed sync showed up in logcat as `ioErrors=1` and nothing else. The
reason was in the run's report, and JS warnings from a headless run
don't reach logcat; finding out that a run failed on "Invalid column
raw_contact_is_read_only" took reading the app's storage. The log line
now carries the report's outcome, its message (up to 300 characters)
and the number of failed items.
Checked: 55 JVM tests (`gradlew :app:testDebugUnitTest`, one new) and
the debug build.
On Android, Settings > Contacts and Settings > Calendar get a "Sync to
this device" section with a row per signed-in account. This makes device
sync reachable:
- The switch asks for the contacts or calendar permission after a short
explanation (denied: sync stays off and says why, with a link to the
app's system settings once Android stops asking), creates the
account's Bulwark account in Android and syncs right away. Turning it
off uploads what the device changed first and asks before anything
that could not go up is lost. The switch shows Android's own state, so
sync turned off in Android Settings shows as paused.
- Which address books or calendars sync: the personal account's are on
by default, shared ones and the Trusted Senders book off; read-only
ones are marked, and a note says when another signed-in account
already syncs the same shared one.
- The address book new contacts from other apps go to, who reminds of
synced events (the calendar app or Bulwark, asked when calendar sync
is turned on while Bulwark's reminders are on), and how often Android
syncs on its own (every 15 minutes to every 6 hours, or not at all).
- "Sync now", the last sync with its result, or the error with what to
do about it ("Sign in again", "Grant access", "Review deletions"), and
a link to the account in Android Settings.
- Accounts the app signed out while they synced are listed with "Sign
in again" and "Remove from this device".
The settings search finds both sections. iOS, and Android builds without
the native module, show nothing new.
Checked: the collection listing and status tests, the settings search
and i18n tests, `npm run typecheck`, `npm test` and `npm run i18n:check`
on this commit, and on the API 32 emulator with usera on the local
server: the section, turning contacts sync on (Android account created,
152 contacts on the device, one per card) and the status after a run.
434c554 left an account that refuses to list its address books or
calendars out of the run, but reported it as an item error. The sync
settings then said "1 item could not be synced" after every run for an
account that merely shares a mailbox: nothing the user can fix, and no
item behind it. Such an account is now only logged.
Checked: the engine tests, `npm run typecheck` and `npm test` on this
commit, and the sync settings of usera on the API 32 emulator after a
new run.
The contacts planner read RAW_CONTACT_IS_READ_ONLY with every raw
contact, but ContactsProvider takes that column only on writes: every
projection and selection that names it fails with "Invalid column", on
API 32 and API 35 alike. Every contacts sync on a device therefore
failed before it wrote a row. The fake provider accepted the column, so
no test noticed.
The flag is now only written, along with any raw contact write that
happens anyway, so echoes stay write-free. Whether a contact's device
edits are put back is decided from the server's current rights on the
card, which is also right when only the rights changed. The fake now
refuses the column in queries and selections like the real providers,
and every other column the planners query was checked against the real
providers on both emulators.
Checked: the contacts, engine and fake provider tests, `npm run
typecheck` and `npm test` on this commit, and a contacts sync of usera
on the API 32 emulator, which now completes.
Stalwart lists an account shared with the user for mail only with every
capability, and answers `forbidden` when the user asks for its address
books or calendars. The engine treated that like any failed request, so
on the local test server every sync of usera ended with an I/O error and
never reached the device, only because userb shares a mailbox.
An account that refuses to list its collections (`forbidden`,
`accountNotFound`, `accountNotSupportedByMethod`) is now left out of the
run and reported as an item error; the other accounts sync as before.
The rows written for such an account stay on the device: a refusal can
pass, and dropping the account's contacts and events for one refused
request would throw away device edits that are still waiting.
Checked: three new engine tests (a shared account that never listed,
one that stops listing its address books, one that stops listing its
calendars), which fail without the fix, `npm run typecheck` and
`npm test` on this commit, and a contacts sync of usera on the API 32
emulator, which now ends `ok` with one item error.
When an account's calendars sync to the device and the user chose the
device's calendar app to remind them, that app gets the events'
reminders. Bulwark's own reminder scheduler now skips the events of
those calendars, which would otherwise remind the user twice. Tasks
never sync and keep Bulwark's reminders, and a change of who reminds, or
of which calendars sync, reschedules right away.
Checked: the reminder filter and scheduler tests, `npm run typecheck`
and `npm test` on this commit.
Signing out of an account (or of all of them, or removing one) now first
uploads what the device changed in its synced contacts and calendars,
then deletes them and removes its Android account, while the credentials
still work. When changes could not be uploaded (offline, server errors)
the app asks before signing out and losing them; staying signed in
keeps the account syncing. Several accounts are released in parallel,
and an internal error never keeps anyone from signing out: the next
check against Android's accounts stops what was left from syncing.
A successful sign-in clears the account's "Sign in again" notification
and resumes an account that stopped syncing when it was signed out.
Checked: `npm run typecheck` and `npm test` on this commit (the release
and resume paths are covered by the lifecycle tests of e67e240).
With device sync on, Android should hear about a change as soon as the
app does, not only at the next periodic sync. This adds what tells it:
- The app's own contact and calendar edits, StateChanges on the live
stream for `AddressBook`/`ContactCard` and `Calendar`/`CalendarEvent`,
and the app coming back to the foreground request a sync of that
account and authority. Requests within 5 s fold into one, and a
StateChange the engine caused itself (its own upload echoing back) is
skipped.
- While the app is closed, the account's push subscription carries
those types too. The FCM and UnifiedPush services hand every push to
the native router from f743ab0, which requests the syncs, and start
the mail push task only when a push has `EmailDelivery`; the task also
returns early for such pushes, so a contact change can never show the
"newest unread mail" notification.
- At launch, on foreground and when Android's accounts change, the app
checks Android's accounts against its own: an account removed in
Android Settings shows sync off and is not recreated, and one whose app
account is gone stops syncing and posts "Sign in again", keeping its
data.
The functions that turn sync on and off (`src/device-sync/app/lifecycle.ts`)
come with this; the settings that call them land in a later commit, so
nothing here runs before an account has device sync on.
Checked: the new store, lifecycle, trigger and push tests, `npm run
typecheck`, `npm test` and `npm run i18n:check` on this commit, and the
debug build.
This is the planner for contacts (`src/device-sync/contacts`), the last
piece the sync task was waiting for: it turns JSContact cards into raw
contacts and data rows, and device edits back into JMAP writes.
- Names, nicknames, organizations and titles, emails, phones, postal
addresses, websites, birthdays and anniversaries (also without a
year), relations, notes and photos map both ways, with their types and
custom labels. Group cards become groups, and their members become
group memberships.
- Device edits upload as patches of the fields whose rows changed; a
whole map goes up only where Stalwart needs one. Whatever a card holds
that Android can't show (pronouns, online services, keywords, a second
note, an address's coordinates) is never touched, so an edit on the
phone leaves it intact on the server.
- Rows are matched to their card's entries by the keys they carry and,
where an editor re-inserted them without (Fossify saves that way), by
their values. An entry the editor didn't show is never read as
deleted, and the keys Stalwart assigns by position to cards from other
clients are followed when they shift.
- Contacts in read-only address books are put back when the device
changes them. Text beyond what the provider stores is cut on the device
and never uploaded.
Checked: 151 contacts tests (mappings both ways, one test per edited
field, Fossify and AOSP editing, re-keyed cards, merges, groups,
read-only books, a full lifecycle), the engine's tests with both real
planners, and `npm run typecheck`, `npm test` and `npm run i18n:check`
on this commit.
The engine hands every event it syncs to a planner that turns it into
CalendarContract rows, and device edits back into JMAP writes. This is
that planner for calendars (`src/device-sync/calendar`), now used by the
sync task:
- Calendars become calendar rows with what the apps need (SYNC_EVENTS=1,
an owner, colours, the reminders allowed); calendars of shared accounts
carry the account's name.
- Events keep their time zone, floating and all-day times, recurrence
rules, excluded and changed instances, reminders, attendees and the
user's own reply. A rule Android can't hold (RSCALE, SKIP, leap months)
stays on the server untouched, and local times Stalwart can't resolve
(a start in a repeated hour, an UNTIL in a skipped one) are written in
a form it keeps.
- Device edits upload as patches of what differs from the last synced
state, never a whole event, and only what the device can show is ever
sent. A single-instance edit or deletion becomes an override of its
series; "this and following" caps the series and creates a new one.
How Etar and Fossify save (a cancelled instance instead of an
exclusion, attendee rows deleted and re-inserted) is read as the edit
it stands for, not as a deletion. Invitations go out on the changes a
calendar app would notify about.
- Events in read-only calendars and subscribed feeds are put back when
the device changes them; only the user's own reply goes up, where the
server allows it. Tasks, task-only calendars and the birthday calendar
never reach the device.
- Reminders are written as Reminders rows when the calendar app is to
remind the user, and left out when Bulwark does.
Checked: 179 calendar tests (download, upload, exceptions, the edits
Etar, Fossify and CONTENT_EXCEPTION_URI make, echoes that write nothing,
time zone and DST edges), and `npm run typecheck`, `npm test` and `npm
run i18n:check` on this commit.
The sync adapters from ed2525e start a headless task for every sync, but
the task only reported "disabled". This adds the engine it runs
(`src/device-sync/engine`, `src/device-sync/jmap`): for one Android
account and authority it downloads the server's changes, uploads what
the device changed, and hands a report back to SyncManager.
- Downloads follow `/changes` from the state kept in the provider's
SyncState. Rows and the state they reach are written in one batch, so
a killed run resumes where it stopped. Without a state, after
`cannotCalculateChanges` or after the user discarded deletions, a full
reconcile compares ids instead.
- Uploads go in three phases across accounts (creates, updates,
deletions), so moving an item to another account never destroys it
before the copy exists. A create carries a uid written to the device
first, and the server is asked for that uid before creating, so a lost
response or a killed run never makes a duplicate. Updates are patches
guarded by `ifInState`; nothing is replaced whole.
- More than 50 deletions that are also more than a fifth of the synced
items hold every deletion back and report `tooManyDeletions`, so
Android asks the user; "undo" restores the items from the server.
- An item the server refuses is marked and not sent again until it
changes or its back-off (1 hour, doubling to a day) ends. Such
refusals and planner errors stay with their item; the rest of the run
goes on.
- Runs of one account and authority take a lock, stop at checkpoints
when cancelled or close to the deadline, and never throw: every
failure becomes an outcome, and a rejected sign-in posts one "Sign in
again" notification.
- The JMAP side uses a client of its own per app account, never the
UI's, since a run can share the JS runtime with the app.
Turning sync off or signing out runs the same upload first
(`teardownAuthority`) and deletes the rows only once nothing is waiting,
or when the user agreed to lose it.
The planners that map items to rows land in the next commits; until
then a run fails before it touches a row.
Checked: 94 engine tests against the fake provider and JMAP server
(sync flows, failures, crashes and retries, teardown), and `npm run
typecheck`, `npm test` and `npm run i18n:check` on this commit.
The engine plans its provider reads and writes in JS, but only native
code can reach ContactsProvider and CalendarProvider. This adds that
bridge to the BulwarkDeviceSync module, with the rest of what the engine
and the app need from it:
- `query`, `applyBatch`, `readSyncState` and `readPhoto` take the
`ProviderQuery`/`ProviderOp[]` JSON of `src/device-sync/types.ts`,
check it (`sync/ProviderOps.kt`) and run it for one Android account of
our type (`sync/ProviderScope.kt`, `sync/ProviderIo.kt`). Every
selection gets the account ANDed in, every row an op names by id or
points to as a parent must be the account's, and every URI carries
`caller_is_syncadapter`, so nothing the sync writes marks rows dirty or
schedules an upload. The calls run one at a time on the module's own
thread through an unstable provider client, so a long batch never
stalls other module calls and a provider that dies fails the call
instead of the app.
- `showSyncProblem` and `clearSyncProblem` post and clear the one
notification per account that needs the user, e.g. "Sign in again".
- `setPushRoutes` stores which Android accounts each JMAP account feeds,
for the push router in `sync/PushRouting.kt`; the push services start
using it in a later commit.
- A run now ends exactly once and refuses a report that comes after
that. A sync requested while a run is open makes the run ask for
another sync when it ends, because SyncManager drops such requests as
duplicates. `onPerformSync` lets nothing escape any more: an exception
there killed the whole process, UI included.
- `ensureAccount` no longer hands an existing Android account to another
app account (`conflict`): its rows and SyncState belong to the first.
Payloads cross as JSON strings, which the bridge moves much faster than
nested maps. `src/device-sync/native.ts` is the typed JS side. The fake
provider follows the rules the emulator runs turned up: contacts
`data1`..`data14` read back as text, events and calendars the sync
adapter inserts have DIRTY NULL, attendee rows are addressed by id, an
op by id expects one row, and a vanished parent fails with `assert`
while another account's fails with `scope`.
Checked: 54 JVM tests (`gradlew :app:testDebugUnitTest`), debug and
release (R8) builds, `npm run typecheck` and `npm test` on this commit,
and on the API 32 emulator through the debug receiver: inserts with
back-references and group memberships, SyncState, photo writes and
scaled reads, asserts and expected counts, both scope refusals, the
499-op yield boundary, `tooLarge` for a 1.3 MB batch, and the
notification.
Linus Rath