TODO

• Is the asynchronous API design useful enough to warrant more complex implementation?
  • DBus operations can be run asynchronously so UI responsiveness shouldn't be an issue
  • For save() calls activity needs to wait for result (containing new version_id) before it can invoke save() again for the same object which can take quite some time if save() is sync - especially if other activities are saving at the same time.
• examine / define API call interactions, esp. in case we switch to fully synchronous design
• do we want an optimized way to determine (only) the branch HEADs of a given tree_id?
• example activity <-> data store interaction
• Tomeu: What about having a separate call that returns synchronously a new tree_id and/or version_id?
  • need to handle missing intermediate versions in that case (parent_id set to a never-committed version)
• cross-tree_id duplicate detection/optimization
• literal copies (timestamp etc. not changed) for backup/restore

(A)synchronicity of API calls

• metadata/data write requests:
  • synchronous submit with ACID guarantee
  • actual commit async, with callback on completion
• data read requests:
  • synchronous check for existence
  • data provided async, with callback on completion
• metadata read requests:
  • fully synchronous
• VCS operations won't happen in parallel (single-threaded) so we don't need locking for them
  • they're usually IO-bound anyway

Notes

• using the term "ACID" for simplicity even when we need only parts of it (e.g. no isolation - there's only a single datastore)
• using symlink instead of hardlink for "incoming" queue since we want to support directory trees, not just files
• since we need to manage the version numbers ourselves (so we can do the actual commit in the background) the VCS doesn't strictly need to support branches
• since an index rebuild can take a lot of time we need to provide UI feedback while doing that
• detecting identical files across objects isn't as important since duplicates are mostly expected to occur as versions of the same object
• For technical reasons callbacks to the activity are implemented as DBus signals. The Python API needs to set up a DBus signal handler using string matching.

Requirements

• data should be provided on the same FS as the Journal (for hardlinking)
• activities must not modify the entry they submit to data store before it has been commited (and the activity notified about that)
  • Python API may copy file if activity indicates it doesn't fulfill this requirement
• activities must not modify the entry they received from datastore
  • Python API may copy file if activity indicates it doesn't fulfill this requirement
• activities must make the entry readable by the data store user
  • It might be possible to set up Rainbow in a way that datastore is running isolated as well and shared directories with appropriate permissions are created so activity and datastore can share directories without having to change ownership in any way.
• activities should not submit new entries while the previously submitted one hasn't been fully committed yet
• VCS backend must not modify files that have been provided as input (for commit) or checked out => link-breaking
  • if the VCS doesn't support this, the backend needs to implement it by copying
• VCS backend must be crash-proof, i.e. ACID
  • if the VCS doesn't support this, the backend needs to implement it, e.g. by copying/hardlinking and calling sync()

ID / metadata definitions

tree_id

Identifies the ancestry graph (i.e. all versions) of an entry. Should be treated as an opaque string by API consumers. Potentially unique across systems (UUID).

version_id

Identifies a specific version of an object. Should be treated as an opaque string by API consumers. Usually combined with tree_id but unique on its own, potentially across systems (UUID).

parent_id

The version_id of the parent of an object, set by datastore. Empty for the root of the ancestry graph.

ctime

Unix timestamp of creation time of an entry, set by datastore. Not updated on changes of the version-specific metadata.

bundle_id

bundle_id of the Activity template associated with the entry. Only set for session continuation objects. Will be used for resuming the session.

entry_type

Valid values are action and object. Intended for use by Journal for implementing action-oriented view.

creator

bundle_id of the Activity template the entry has been created by (e.g. org.laptop.WebActivity for downloaded files).

mime_type

MIME type of data, used for determining compatible Activity templates.

Workflows

data+metadata creates / updates

• activity saves data to a disk, ensuring it has been committed (sync) and proper access rights for data store
• activity calls datastore DBus API to submit new/updated entry
• data store allocates tree_id if not given
• data store allocates (new) version number
• data store submits metadata to database backend as "new" (with version number, ACID update)
• data store symlinks (=atomic) data to (on-disk) queue "incoming" with version number in filename
• data store adds metadata to queue in-memory queue "index-add"
• data store returns on API call with allocated version number
• VCS backend prepares commit (e.g. git checkout)
• VCS backend hardlinks data into working copy
• VCS backend commits data with allocated version number in commit message
• data store records "finished" state in database backend
• if activity requested it, data store deletes the submitted copy of the data
• data store removes symlink in "incoming" queue
• data store invokes callback to activity

metadata changes (to existing versions)

• activity calls datastore DBus API to submit updated metadata
• data store submits metadata to database backend (ACID update)
• data store returns on API call

crash recovery

• data store checks database for "new" entries that have no corresponding symlink in the "incoming" queue
  • for each such entry it asks the VCS backend for recovery and whether the commit has been finished
    • if commit exists, mark entry as "finished"
    • otherwise delete entry
• index backend checks for corruption
  • in case of corruption a full index rebuild takes place
• data store starts accepting regular API calls
• data store checks database for "deleted" entries. For each entry:
  • VCS backend deletes corresponding repository (if it still exists)
  • index backend removes corresponding entry (if it still exists)
  • database backend removes entry
• data store checks "incoming" queue for entries (none found => no recovery necessary)
• VCS backend performs global integrity check / recovery (none for git)
• for each entry in the "incoming" queue:
  • VCS backend performs integrity check / recovery for given entry
  • VCS backend checks whether entry has already been committed
    • if not: VCS backend performs commit like above
  • index backend adds entry (if it does not yet exist)
  • data store removes symlink in "incoming" queue

checkout

• activity / shell calls datastore DBus API to request entry
• data store checks "incoming" queue for entry
  • if it exists we return on API call but block waiting for the commit to finish before checkout it out
• database backend checks entry for validity ("deleted", existence)
• data store returns on API call
• VCS backend checks out entry
• data store hardlinks entry into target directory
  • files read-only and directories read-write for activity (=> ensure link-breaking)
• data store invokes callback to activity

search

• data store checks "index-add" queue for matching entries
• data store checks index backend for matching entries (eliminating dupes)
• database backend retrieves metadata for matching entries and checks for validity (not "deleted", does still exist)
• data store returns all matches

delete

• database backend marks entry as "deleted"
• data store returns on API call
• data store removes entry from "index-add" queue (if it exists)
• index backend removes entry
• VCS backend removes corresponding repository
• database backend removes entry
• data store invokes callback to activity

API

Current datastore DBus API

Functions and signals

create(properties, filename, transfer_ownership)

Create new journal object with metadata given in properties, returns uid / object_id. If transfer_ownership is False the data store will make a copy of the file. Metadata storage and index update is synchronous and while file storage and optimizer run in a separate thread, the API call actually only returns after everything is done (including emitting a Created(uid) signal) and thus is fully synchronous. Returns uid (object_id).

update(uid, properties, filename, transfer_ownership)

Submit new version (currently replacing the old version) and/or new metadata of the object identified by uid (object_id). Similar to create() Metadata storage and index update is synchronous, file storage and optimizer run in a separate thread, but the API call only returns after everything is done, including emitting an Updated(uid) signal. Doesn't return anything.

delete(uid)

Remove given object from data store. Fully synchronous. Doesn't return anything, emits signal Deleted(uid).

get_properties(uid)

Returns (full) metadata for given object. Fully synchronous.

get_filename(uid)

Hardlinks or symlinks the given object into a global directory and returns the filename. Fully synchronous.

find(query, properties)

Finds entries matching query in the index, returning the values of the requested properties. If the index is unavailable, all entries are returned (but taking limit and offset into account). query is a dictionary and may contain propertyname=value mappings (that are directly matched against metadata - only timestamp (with a range), uid, activity, activity_id, keep and mime_type are supported so far) in addition to values for limit (max. number of results), offset (skip given number of results) and query ("full-text" search in all properties). Returns estimated number of matching entries (not affected by limit and offset) and requested metadata for matching entries.

Journal passes sort=-mtime in query; datastore currently ignores it (but sorts by timestamp anyway).

get_uniquevaluesfor(propertyname, query)

Supposedly would returns a list of all (unique) values associated with given property (metadata) name for all datastore entries matching query. Only supports propertyname='activity' and query=None, so a list of all activities (activity / bundle_id) is returned.

mount(uri, options)

Compat only. Noop, returns ''.

unmount(mount_point_id)

Compat only. Noop, doesn't return anything.

mounts()

Compat only. Noop, returns [{'id': 1}].

Stopped()

Signal emitted after datastore has been shut down.

Proposed datastore DBus API

Functions and signals

save(tree_id, parent_id, metadata, path, delete_after)

Submit new version (data and/or metadata) of the given object, returns tree_id, child_id. parent_id and child_id are the version_id of the predecessor resp. the to-be-saved entry.

If tree_id is empty parent_id must be empty as well and both will be newly allocated (equivalent to create() in previous API). If only parent_id is empty there must not be any entry with the same tree_id already in datastore.

Actual file storage and index update happen asynchronously, emitting the signal Saved(tree_id, parent_id, child_id) on completion.

If path is non-empty it designates a file or directory containing the data and the parent must contain of the same type (file / directory; not metadata-only), if a parent exists at all. If path is empty and the parent contained data then a metadata-only update is performed, reusing the data of the parent. If both parent_id and path are empty a metadata-only entry is created.

If delete_after is True the contents of path are deleted after they have been committed, but before sending the signal.

change_metadata(tree_id, version_id, metadata)

Changes the (unversioned/version-specific) metadata of the given object to match metadata. Fully synchronous, no return value. Emits signal ChangedMetadata(tree_id, version_id, metadata).

delete(tree_id)

Remove (all versions of) given object from data store. Fully synchronous. Doesn't return anything, emits signal Deleted(tree_id).

get_data(tree_id, version_id)

Hardlinks the given object into a global directory, making it readable (directories also writable) to the caller. Returns the path after checking the entry for validity; actual data is written asynchronously. Emits signal GotData(sender, tree_id, version_id, path) where sender is the bus name of the sender.

find(query, options)

Finds entries matching query and returns output according to options, including the number of matches (unaffected by limit) and the tree_id / version_id of each entry.

query is a dictionary with metadata names as keys and either basic types (exact/wildcard match), 2-tuples (range) or a list (multiple exact/wildcard matches) as values. Prefixing a key name with '!' inverts the sense of matching, OPTIONAL: prefixing it with '*' enables regular expression search (slow). An empty dictionary matches everything. Arbitrary key names are allowed, but speed may vary (i.e. not everything is indexed).

options is a dictionary which may contain values for the following keys:

metadata

gives a list of metadata names that should be returned for each entry; default is everything

all_versions

if True returns all matching versions of an object instead of only the latest one

sort

sort by given metadata name (does not need to be listed in metadata); prefix '+' for ascending order, prefix '-' for descending order; default is undefined, potentially by relevance

limit

only return the given number of results (starting at first match in sort order, subject to offset

offset

skip the given number of results

textsearch(querystring, query, options)

Preliminary API call for IR search using Xapian. Likely to be replaced for next version. querystring is passed to Xapians query parser; query and options are interpreted like for find(). Returns the same format as find().

find_unique_values(query, metadata_name)

Similar find(...), but returns the set of unique values for the requested metadata name.

check_ready()

Returns True if datastore is ready for processing (regular) API calls. False during startup / recovery.

Ready()

Signal emitted after datastore is ready for processing API calls.

Stopped()

Signal emitted after datastore has been shut down.

Exceptions

sugar.datastore.NoSpaceLeftError

There's not enough space left on disk to safely store data and/or metadata. Operation aborted, everything still at the previous state.

sugar.datastore.InvalidArgumentError

One or several of the passed parameters (or the combination thereof) were invalid. See formatted string for details. This should only happen if the caller is buggy.

sugar.datastore.CorruptionError

The internal data structures of datastore or one of its backends are corrupted. Should only happen in case of hardware defects or OS bugs.

sugar.datastore.BusyError

Datastore is busy recovering from a crash. Please wait for Ready() before starting to issue any API call (except check_ready()).