apt-repo-tools

Last updated: 2016-02-19 14:42:11 +0000

Upstream URL: git clone http://chriswarbo.net/git/apt-repo-tools.git

Contents of README follows

INTRO

Repositories are often seen as a difficulty in Debian and its derivatives. Whilst adding and removing packages has become easier and easier, the same is not true for package repositories.

However, repositories are just text files in /etc/apt/sources.list.d, and possibly a security key, both of which can be packaged.

Thus these tools aim to remove the complexities from managing repositories by simply packaging up every repository. These can then be installed, removed, updated, etc. just like any other package, with absolutely no changes to the way the APT packaging system works. No need to define arbitrary new protocols, distribution systems, system tools, etc. We just use Debian’s already excellent package facilities.

Since these tools are boring in themselves, also included are some scripts to fetch data from apt-get.org and Launchpad’s Personal Package Archives (PPAs). This is stored in cache.txt and ppa.txt. Running make_sources.py on these, followed by make_packages.py will generate over 8000 packages in the temp folder, one for each repository. It’s a start ;)

apt-get-org.py - Screen scrapes www.apt-get.org, a vast collection of third-party Debian repositories. Saves the data to a given file.

launchpad_ppa_grabber.py - Screen scrapes launchpad.net, site which hosts a few Ubuntu repositories

make_sources.py - Creates files suitable to put in sources.list.d. Automagically groups repos (eg. for matching deb and deb-src lines)

make_packages.py - Creates packages based on the contents of temp/lists

make_repo.py - Non-functional at the moment. Should make a repository out of repository packages.

cache-check.py - Does basic checks on a given file of repository data.

cache.txt - cached repo data (output from apt-get-org.py)

temp/ - Stores temporary files

temp/lists/ - Stores apt’s sources.list files

temp/package/ - Stores temporary files for making packages

APT-GET-ORG.PY

This script takes an output filename “-o=something” (which will be overwritten!) and fills it with repository data (in the format described below) taken from www.apt-get.org.

It goes to the main page (http://www.apt-get.org/main/) and “screen scrapes” data out of the page. It also gets package contents for each repository by loading the info page for each one. Since this can generate a load on their server, there is a half-second delay between handling each repository.

MAKE_SOURCES.PY

This script is given an input file (in the format described below) with the option “-f=something”. It then parses this to find unique combinations for repository/suite. Identical ones are grouped (eg. “deb something.com stable” and “deb-src something.com stable”) then put into files suitable for including in /etc/apt/sources.list.d

MAKE_PACKAGES.PY

This script looks through the “temp/lists” folder for sources.list.d files and generates a package for each one in “temp”. The packages are currently set to architecture “all”, maintainer “unknown@unknown.com” and have no dependencies or significant description.

CACHE-CHECK.PY

This script checks a given file of repository data (given with the -f=filename option) using the format described below. Some basic heuristics are used to validate the contents. This is useful since screen scraping is prone to errors, which should preferably be spotted ASAP rather than generating bogus packages.

Only the first error encountered is reported, since the parsing system may fail afterwards. This means that a check-fix-check cycle must be used until no errors are reported (and even then some obscure ones might still be present).

DATA FORMAT

The format currently used to store repostory data is XML-esque with the following structure