Code to back up a live Xanadu backend is almost trivial.

[Michael McClary <michael>]

This morning (while showering) I realized something:

 - Snarfs are self-identifying.
 - Therefore, a backup need not write them in any particular order.
 - Once a snarf is written to a backup medium, the backup process will
   not need to look at it again.  Therefore, the view need not keep it
   around.

Therefore, a backup process could be integrated (almost trivially) into URDI.

There are several ways to do this, each with different characteristcs and
suitable for different purposes:

First, consider backing up the state of a disk >NOW<.  Using the current
URDI interface, you'd create a View, which would give you a virtual frozen
state.  You'd use this to read the snarfs as you copy them to tape.
Meanwhile, the Write View would continue modifying snarfs, and the original
version of each snarf would be saved in a SnarfBuffer until you released
your view.  Since copying an entire disk takes a while, you'd eventually
exhaust the VM {or that portion earmarked for SnarfBuffers} and block further
updates.  (This is why Views, except for the Write View, should be very
short-lifetime.)

Once a snarf has been written to the backup medium, however, the backup
process will never need it again.  Therefore, it is no longer necessary
to maintain its old state.  Combine that with the fact that a backup
need not write the snarfs in any particular order (as long as it writes
all of them), and you can define a BackupSnapView with the following
characteristics:

 - It maintains a list of which snarfs have been backed up.

 - It does not impose the normal requirement that the view-current
   contents of any of these snarfs be maintained.

 - When asked for "the next snarf":
    - If one it hasn't yet backed up has been brought in for
      modification, it returns the pre-modification contents of that
      one (so it can be written right away, and the BackupSnapView's
      lock on these old contents can be dropped).
    - If there are none such in, it returns a not-yet-backed snarf
      that is only in (so far) for content retrieval, not modification.
    - If there are no unbacked snarfs in at all, a convenient snarf will
      be read and returned.

Not only does this process successfully back up a live backend, it
piggybacks on the disk I/O being done for the backend's own work, only
causing disk reads when no unbacked data is already in.

This gives you a backup that's already slightly out-of-date, of course.
(In a backend with continuous transaction-level logging, the start of
the backup would be marked in the transacion log, so changes after the
backup could be applied to a restore, bringing things up-to-date.)

A second case is interesting.  (Though there are more efficient ways to
do it, it gives insights that I'll be applying later, and which should
transfer to non-snarf-based backup and logging procedures.  Also, this
could be hacked together very quickly if a client needed its functionality
early, and had a usage pattern that would support it.)

Consider the stream of snarfs being written to the Staging Area.  This
stream, if it were also being written to, say, a tape, would constitute
a transaction log suitable for replacing a crashed disk or seeding a new
system.  It would be inefficient, because typical updates change only part
of each snarf, yet the whole snarf would be written.  (An early user, with
an application involving infrequent but important updates, might chose to
accept such a system while waiting for something more efficient in a later
release.)

Such a log, of course, wouldn't contain snarfs that hadn't been modified.
A list of snarfs that haven't been written yet (or hadn't been written
lately) could be used to force rarely-modified snarfs to be logged, much
like the third "next snarf" case of the first example.  As with the
"no undo-redo" algorithm, a restore could (logically) work backward from
the last consistent set of changed snarfs until a complete set of snarfs
had been accumumated, which would happen no farther back than the second
point where the table of unwritten snarfs had been reset.  (This trick
also works for forcing periodic writing of all snarfs on a disk, should
media technology require it, and to bring a replacement drive on a shadow-
drive system up-to-date.)

When the log medium runs out (especially if it isn't replaced for a while),
unlogged snarf changes will accumulate rapidly.  If the logging task
maintains a list of snarfs with unlogged changes, the snarf changes can
be written and the previous contents discarded.  When the log medium is
again ready, the changed snarfs can be retrieved from the disk and written.
This squeezes out some of the consistent states the backend passed through,
but rapidly reaches a valid state.

This same mechanism can be used as a fall-back if bursts of activity
overwhelm the backup medium's speed.  Expected use patterns would
result in repeated changes to the same snarfs, so squeezing out
intermediate states would normally result in a significant reduction
of the data to be logged.

If the changes come rapildy enough that the drive can't catch up, a second
fallback position is available:  The backup process could create a view with
characteristics similar to a BackupSnapView (covering only the snarfs with
unlogged changes) logically "stopping" the updates until it has a consistent
state on the tape again.  This has the disadvantage that the unmodified
snarfs must be cached until they can be written to the backup, so once
the cache fills up, URDI jackpots slow the backend's response to match the
backup device's maximum speed.

Barring drive failure (which can be dealt with by dropping the view until
the drive is back) the slowdown is smooth, not a sudden wedge.  This is
because the snarfs of interest are backed up first, freeing their buffers.

A third interesting case is a variant of the second which leaves the
recently changed snarfs for last and stops at the first consistent state
after it has accumulated a complete set of snarfs.  This is a backup much
like the first case, except that the consistent state it saves is the one
at the END of its run, and it wastes some tape by inclusion of snarf
states that are obsoleted before the end of the run.

	michael

PS:  In case you haven't noticed already, the backups described here are
distinct from archiving.  These backups are of an entire partition, suitable
only for being restored in their entirety to a disk of the same capacity as
the original.  Archiving also addresses backup, restore, and inter-backend
transfers of portions of a partition or set of partitions, and transfers of
data between partitions of varying sizes.

On the other hand, a nearly-trivial on-the-fly crash-restore backup will
be DARNED NICE for first product.

	-m