| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
| |
For easier debugging, log any exception we pass through to the caller.
|
| |
|
|
|
|
|
|
|
|
| |
When re-indexing entries, take the git commit id into account so that
we notice entries we already know about but where the metadata has
changed.
The first re-index run after an update will need to process all
entries in order to add the commit_id value to the index. This may
take a long time. This change is backwards compatible.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
When only the metadata of an entry got changed, gdatastore only
updated the Xapian index but neglected git. This caused the change not
to persist across an index rebuild and not to propagate across
gdatastore instances.
We now store the updated metadata in a new commit with the same ref
name and having the previous ref as parent. This should be enough for
the index rebuild case. It will also propagate nicely through git,
though we don't have code in place yet to notice updated entries
(rather than completely new ones).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Some users may have direct access to additional data stores in
gdatastore format. Allow the user to "register" them by placing links
inside the (new) "extra-stores" directory and use them as additional
sources.
Modification is only supported for the primary store. The additional
ones are used for reading only. In particular, trying to remove an
entry that's contained in an extra store will yield an error (no
shadowing).
The extra stores will need to use exactly the same on-disk format as
the primary one. No automatic migration is done.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
With the Sugar 0.84+ API (without version support), there are three
ways to look up the metadata of a single entry:
1. get_properties(uid)
2. find({'uid': uid}, {})
3. find({'query': 'uid:' + uid}, {})
The former two were already supported by explicit conversion inside
the API adapter class.
The last one matches the uid inside a Xapian query string. This is
more tricky to support as the string gets parsed by Xapian, not
ourselves.
We're going the simple route for now and just add a 'uid'
compatibility prefix for our custom Xapian query parser. This will
leak some Sugar API details to other APIs. If any issue comes up
because of that, we'll need to pass in different sets of prefixes to
Index.find() for each query, based on which API was used.
|
| |
|
|
|
|
|
|
| |
For the native interface, we switch to using byte arrays, which is the
only sane way to handle paths on POSIX file systems.
For the Sugar v2 interface, we assume UTF-8 representation, like Sugar
does.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
If available, gdatastore will now use sugar.mime to guess an extension
to add to checked out files, based on their MIME type.
The outside world (including emacs) is still surprisingly focused on
file extensions rather than MIME types. By adding an extension to the
file names returned by gdatastore we help other software that has to
interface with legacy components. The alternative would be to add code
to every piece of glue code to guess the extension and rename the
file in a race-free way. Doing it in gdatastore reduces the overall
complexity of the software system.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Passing data out-of-band using data paths has a couple of issues (different
user, chroot, ownership / deletion, in-place modification by callers, etc.)
that passing data via file descriptors can avoid. Recent versions of D-Bus and
python-dbus have support for fd passing, so we can add alternative versions of
the functions that take a data path and have the alternatives take a file
descriptor instead.
Support for this is still suboptimal as we make a copy and pass a path
internally. We can easily improve on that later as it's just a matter of
performance, not functionality.
We may even consider dropping the non-fd counterparts. However, since D-Bus
still does not support a "maybe" type, we'll need alternative versions of these
calls that don't take a file descriptor, for use with metadata-only entries. We
could also pass a reference to /dev/null, but that's more a hack than a
solution.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
We're already storing full metadata in git, using the Xapian index only for
performance reasons. By adding support for rebuilding or updating the index
based on what's in git, we can recover from index corruptions and interrupted
execution (power loss, crash). As an additional benefit, adventurous users can
synchronise their data stores using git fetch, forcing gdatastore to update the
index by removing the last_object_id file.
gdatastore before b3a5dad stored various data types as D-Bus types in the git
commit messages. Compatibility with these versions is provided by removing the
D-Bus type information on the fly before parsing.
|
| |
|
|
|
|
|
|
| |
Pure Python data types can be safely parsed using ast.literal_eval(), removing
the need for unsafe evaluation.
Previously, numeric data was stored as dbus types (e.g. "dbus.Int32(0,
variant_level=1)").
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
For reasons explained in ad061c0, we can't rely on the ancestry information
(chain of parent commits) stored in git being complete - at least not without
doing extensive additional processing (recreating branch history using
rebase). Additionally, storing the parent commit caused deleted entries to
still be referenced if there were more recent commits based upon them that
hadn't also been deleted, causing storage space not to be reclaimed.
For now, we simply don't record ancestry directly in git. We still have all
the information in the metadata (so no information loss) and don't actually
use the ancestry information stored in git.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Restoring a version that has no parent with a tree_id that we already have
entries for can be indicative of a bug in the caller (restoring entries in
the wrong order) and causes inconsistencies in the git repository (as the
root commits don't match). For that reason, we refused to store the (new) root
version.
However under certain conditions the above can happen even without any
component being broken: Since we allow the user to delete individual versions,
the root version may have been deleted even if more recent entries were kept.
If the user took a backup before deleting the root version and tries to
restore this backup, restore() will encounter exactly the situation described
above.
Explicitly skipping this check for restore() suffices for now, but we should
reexamine our usage of git (branches) when adding support for recovering from
a broken Xapian index.
|
| |
|
|
|
|
|
|
| |
InternalApi.save() returns the identifiers for the new entry, so we need to
either wrap the callback to drop the identifiers (like we do in
DBusApiSugarV2.update()) or adjust the DBus signature for
DBusApiNativeV1.restore(). While the caller is supposed to pass in all
identifiers, it doesn't hurt to return what was actually used.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
In order to support removing intermediate versions (on explicit user action or
as part of automatic space reduction) we should not require the parent version
to exist when restoring versions from backup.
At least the Restore activity needs to be careful to never restore the child
before the parent, otherwise we'd loose information in git (the parent
commit id) that would have been recoverable. The previous check in restore()
would have caught that, but also prevents restoring from backups which are
missing intermediate versions (because they have been deleted).
Alternatives:
* Let gdatastore (and thus Backup) store "fake" intermediate
versions on delete.
|
| |
|
|
|
| |
path='' means the caller wants to update only the metadata and "inherit" the
data from the previous version.
|
| |
|
|
| |
This is useful for manual inspection of the git repository.
|
| |
|
|
| |
Xapian has a hardcoded limit of 245 characters for terms.
|
| |
|
|
| |
We accidentally used the value as prefix and vice-versa.
|
| | |
|
| |
|
|
| |
Previously it returned only a fixed set of values to satisfy the test suite.
|
| |
|
|
| |
delete_after should be ignored if path is empty.
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
Once we have multiple D-Bus interfaces, we need to send signals on all of
them.
Signed-off-by: Sascha Silbe <sascha-pgp@silbe.org>
|
| |
|
|
| |
Signed-off-by: Sascha Silbe <sascha-pgp@silbe.org>
|
| |
|
|
|
|
|
|
|
| |
Having several different data store processes running, potentially serving
disjunct API versions (either because of race conditions or because several
different implementations are installed) would be a major source of confusion
and a recipe for disaster.
Signed-off-by: Sascha Silbe <sascha-pgp@silbe.org>
|
| |
|
|
|
|
| |
This should help with situations where the new gdatastore process is already
starting up (on a new session bus) while the old one is still busy shutting
down (see also: SL#1257).
|
| |
|
