git-annex

git annex command [options] [files...]

git-annex lets git manage files whose contents are too large to live in git history. Instead of the file, git tracks a small pointer, while the actual bytes are kept in a separate object store under .git/annex/objects. History, branches and merges all work as usual, because from git's point of view the repository only contains small pointer files.The idea it adds beyond "big files elsewhere" is that a repository does not need to hold every file's content. Each clone can carry whatever subset fits, and git-annex maintains a distributed record, in the git-annex branch, of which repositories hold which content. git annex whereis answers "where are the actual bytes", and git annex get fetches them on demand. This is what makes it workable for archives far larger than any single disk.Content is addressed by checksum, so identical files are stored once and corruption is detectable. git annex fsck verifies checksums and checks the numcopies guarantee, which is the mechanism that stops drop from removing the last copy of anything.By default annexed files appear as symlinks into the object store, which keeps them read-only and prevents accidental modification. git annex unlock replaces the symlink with a writable copy, and lock stores the result as a new version. The newer v6/v7 repository modes and the adjusted unlocked branch make unlocked files the normal state, which is friendlier for tools that cannot follow symlinks.Special remotes are the other half of the design: an annex remote does not have to be a git repository at all. S3, Backblaze B2, rsync targets, WebDAV, removable drives and many more can hold content, letting git-annex use them as storage while git itself never talks to them.

init [description]

Initialize a repository for use with annex, giving it a human-readable description.

Add files to the annex, replacing them with pointers to their content.

Download content from a remote that has it.

Remove local content, after verifying enough other copies exist.

Copy content to or from a remote, leaving the source copy in place.

Like copy, but drop the source copy afterwards.

Synchronize the git metadata with remotes. --content also transfers file contents.

List the repositories known to hold each file's content.

Unlock a file to modify it in place, then lock it again to store the new version.

Verify that stored content matches its checksums and that enough copies exist.

Set how many copies of each file must exist before drop will remove one.

Show repository or remote statistics, including disk usage.

Set up a special remote such as S3, rsync or a directory.

Display help information.

.git/annex/objects/

Local content store, holding files keyed by checksum.

Repository and remote configuration, including annex settings.

An orphan branch tracking location and metadata for all content. Synced between repositories; do not edit it by hand.

Controls which files are annexed rather than committed to git directly, via the annex.largefiles attribute.

The learning curve is real. The concepts (keys, backends, special remotes, numcopies, trust levels, the git-annex branch, locked versus unlocked files) do not map onto anything else in git, and the repository version history (v5 through v10) means much of the documentation you find online describes behaviour that has since changed.Locked files are symlinks. Editors that replace files on save, build systems that copy trees, and filesystems without symlink support all interact badly with them. Windows support in particular has always lagged, since it lacks proper symlinks.git annex drop refuses to remove content when it cannot verify that numcopies copies remain elsewhere, and it trusts its own location log to decide. If that log is stale, because another repository was deleted without being told about it, the record can claim copies exist that do not. git annex fsck is what reconciles this, and it is worth running periodically rather than only after trouble.A plain git clone of an annex repository gets the pointers, not the content: the files look present but are broken symlinks until git annex get fetches them.git annex sync does not transfer file contents unless you pass --content, which surprises people who expect it to behave like git pull plus git push.git-annex and git-lfs solve overlapping problems differently. LFS is simpler, has direct support from GitHub, GitLab and others, and always fetches content on checkout. git-annex is far more flexible about where content lives and what each clone carries. Migrating between the two is awkward, so the choice is worth making deliberately.

git-annex was written by Joey Hess and first released in 2010. Development of its assistant and web application was funded by a successful Kickstarter campaign in 2012. It is written in Haskell, and remains widely used for research data management, notably as the foundation of the DataLad project, as well as for personal media archives.

git(1), git-lfs(1), git-clone(1), rsync(1)

Source code · Homepage · Documentation