How various package managers work

[s243a]

This thread isn't about how to use a specific package manager. Instead it is about the files which package managers manipulate. The idea here is to give the reader clues about how they might write a minimal version of a specific package manager or how they might sync two or more package mangers or how they might repair things when something breaks.

My primary focus for the start of this thread will be about petget and tazpkg. I'm not sure if well get much into the other ones in this thread or instead we will discuss them in another thread.

Like many package managers the official package manager for pet-get does many things besides just installing packages onto a system; such as:
1. resolving and installing dependencies
2. updating various databases such as icon caches and mime types
3. update desktop menus.
4. show compatible packages for download
5. download compatible packages
6. authentic downloads

Depending on the application we might not need any of the above extra functionality and in which case it is rather trivial to write a package manager.

I consider the most core part of the package manager to:
7. copy the files into the system and
8. update the databases which indicate what packages are installed on the system.

The core aspects (Items #7 and #8 ) are rather minimal and one would only need busybox to implement them. However, it is handy to have other tools (e.g. AWK and find) to make this process easier and also because the petspec database structure is well suited to the use of AWK. Find is helps to make listing the package contents easier but one could write their own script to achieve this functionality.

Regarding Item#7 this is fairly trival. For example one could do the following:

Code: Select all

  cp --remove-destination -arf \
        "$source_dir/"* \
        "$dest_root"/ 2>/dev/null

or they could copy the files one by by either:
1. reading a list of files
2. using the find command
3. walking the source directory structure
or use another utility to copy the files such as: CPIO, rsync or install

Needless to say there is no shortage of ways to copy files in linux.

In a addition to copying the files into the system a list of files is created which itemizes the package contents. This list is stored in one of two places:

Code: Select all

/root/.packages/pkg_name-version.files
/root/.packages/builtin_files/pkgname

Note that in the .packages folder the list of files includes the package version but it typically doesn't in the built-in folder. This raises the question about when we remove a package should we remove all versions of that package or should we only remove a specific version. Also should multiple version of a package be allowed to coexist or should there only ever be one version of a package.

TazPkg also uses one text file per package to keep track of the package contents. It is stored in

Code: Select all

/var/lib/tazpkg/installed/$pkg_name/files.list

We could easily sync the information between the two package managers by writing some code to copy (or symlink or hardlink) this list of files between the two directories so each package manager knows what is installed and can use this information to uninstall said packages if needed.

The next piece of information that is stored is some meta information (i.e. the pet.specs) about the package. In petget this is stored typically in two files which are:

Code: Select all

/root/.packages/user-installed-packages
/root/.packages/woof-installed-packages

We can generalize this a bit and say that we might have multiple files of the form

Code: Select all

/root/.packages/*-installed-packages

The remove_builtin function in puppylinux takes advantage of this wildcard approach:

Code: Select all

sed -i "\%|${PKG}|%d" /root/.packages/*-installed-packages

remove_builtin#L61
(Note that the enclosing percent sign means use regular expressions, the -i means edit in place the d after the percent sign means delete).

In the case of remove_builtin we are matching the second field of the pet spec. This field is the name of the package without the version. If the filename in the buitin_files folder has the package version then this function will fail to remove the petspec from the above files. Also the way in which remove_builtin is written, the pet specs for all versions of the package will be removed from these pet spec tables. However, only one version will be deleted for the list of files for the package and for the actual files on the system.

TazPkg also stores similar meta data. One place that this metadata is stored is in :

Code: Select all

/var/lib/tazpkg/installed/pkg_name/receipt

P.S.
One application of having multiple files that store the pet spec (i.e. /root/.packages/*-installed-packages) might be to have different sfs files use different prefixes so that the spec list in each sfs can coexist with those with the other loaded sfs files.

(To be contined.....) It is time for me to go to sleep.

[s243a]

One more thing to add tonight before I sleep.

I wrote this function (not yet tested to parse a petspec into human readable variables:

Code: Select all

process_pet_specs(){
	local specs_txt=`cat "$1" | head --lines=1`
	#https://stackoverflow.com/questions/918886/how-do-i-split-a-string-on-a-delimiter-in-bash
	#http://www.linuxquestions.org/questions/programming-9/bash-shell-script-split-array-383848/#post3270796
	specs=(${specs_txt//|/ })
	process_pet_specs_FULL_NAME=specs[0]	
	process_pet_specs_PKG_NAME=[1]
	process_pet_specs_VERSION=specs[2]
	process_pet_CATEGORY=specs[4]
	process_pet_SIZE=specs[5]	
	process_pet_BRANCH=specs[6] #Repository Folder
	process_pet_FILE_NAME=specs[7] #File Namer
	process_pet_FILE_DEPENDS=specs[8]
	process_pet_FILE_SHORT_DESC=specs[9]
	process_pet_FILE_COMPILED WITHIN=specs[10]
	process_pet_FILE_COMPAT_DISTRO_VER=specs[11]	
	process_pet_REPO_NAME=specs[12]
}

This information could used to write the recept (that tazpkg version of pet.specs)

Code: Select all

pet_specs_to_tazpkg_installed(){
  local pkg_path=$1
  local pkg_name=`basename $pkg_path` #TODO: add option to use petspec instead for this
  local web_site=${2:-"http://puppylinux.org/"} #TODO: add option to use petspec instead for this
  local maintainer=${3:-"nobody@slitaz.org"} #TODO: add option to use petspec instead for this
  
  
  process_pet_specs $pkg_path
  echo 'PACKAGE="$process_pet_specs_PKG_NAME"
VERSION="$process_pet_specs_VERSION"
CATEGORY="$process_pet_CATEGORY"
SHORT_DESC="$process_pet_SHORT_DESC"
WEB_SITE="$web_site"
MAINTAINER="$maintainer"
DEPENDS="$process_pet_FILE_DEPENDS"
' > $curdir/litaz-rootfs$prefix/var/lib/tazpkg/installed/linux-modules/receipt #TODO add che for pkgmanager type and suport for toher package managers	 
	 
      xcurdir="$(echo "$curdir" | sed "s#\/#\\\/#g")" 
	 
      find $pkg_name -type -name "*" > $curdir/slitaz-rootfs$prefix/var/lib/tazpkg/installed/$pkg_name/files.list-tmp
      cat "$curdir/slitaz-rootfs$prefix/var/lib/tazpkg/installed/$pkg_name/files.list" | 
   touch "$curdir/slitaz-rootfs$prefix/var/lib/tazpkg/installed/$pkg_name/files.list"
   while read  -r -d $'\n' $aFile_prefixed ; do
     if [ ! -d "$aFile" ]; then
       aFile=${aFile_prefixed#"$pkg_path"}
       echo $aFile > "$curdir/slitaz-rootfs$prefix/var/lib/tazpkg/installed/$pkg_name/files.list"
     fi
   done <"$curdir/slitaz-rootfs$prefix/var/lib/tazpkg/installed/$pkg_name/files.list-tmp" 
}

The above function isn't tested yet. It updates for tazpkg both the recepit and also the list of files for a package. This information is taken from an extracted pet. I probably still have some work to do to finish these function.

[musher0]

Thanks s243a.

[technosaurus]

I wrote a function for parsing files like the puppy package database.
Example is here:
https://github.com/technosaurus/PD-snip ... ll/sh#L207

[sc0ttman]

I wrote a function for parsing files like the puppy package database.
Example is here:
https://github.com/technosaurus/PD-snip ... ll/sh#L207

I keep meaning to re-write bits of Pkg to use the IFS approach .. don't think it's going un-noticed... just literally taking me years to get round to it....

EDIT: in fact, I logged an issue: https://gitlab.com/sc0ttj/Pkg/issues/65