Features of Portage

Patrick Lauer

patrick@gentoo.org

What is Gentoo?

a Linux distribution?

support for MacOS, BSD ...

a Metadistribution?

build your own

a collection of information about software and a management tool?

a breed of small and fast penguins?

What is portage?

The Gentoo package management system

written in Python, Bash and shelltools

~35k lines of Code

quite portable, but makes some assumptions about FS layout etc.

Minimal documentation

easy to hack, easy to break

Some Numbers

~8500 individual programs

~40GB of sourcecode

~110k files with metadata (ebuilds etc)

Bas/c Build time And Statistics Client

build time estimates

most installed packages, ...

www.gentoo-stats.org

Features of Portage

recursive dependency solver

automatic “best version” finder

constraints: version is > / = / ! ...

friendly interface: “emerge”

other GUI frontends exist

lots of customization possible

Customization

Build from source:

CFLAGS, LDFLAGS, ASFLAGS, ...

FEATURES: ccache, distcc, maketest, buildpkg, gpg, ...

Use binary packages

DIY package mirror with your customization

USE Flags

package.{use mask unmask}

Xtreme Opimaliztionz!!

CFLAGS="-g0 -DTT_CONFIG_OPTION_BYTECODE_INTERPRETER -pipe -O3 -march=pentium4 -fweb -funswitch-loops -funroll-all-loops -funit-at-a-time -fsched2-use-traces -fsched2-use-superblocks -fsched-stalled-insns=12 -frename-registers -fprefetch-loop-arrays -fpeel-loops -fomit-frame-pointer -fmerge-all-constants -finline-limit=32768 -finline-functions -ffunction-sections -ffast-math -fdata-sections -fbranch-target-load-optimize2"

LDFLAGS='-Wl,-O1,--enable-new-dtags,--sort-common,--allow-multiple-definition,--allow-multiple-definition,--relax -z combreloc -z muldefs -s'

Branches

ARCH (“stable”) should only be well-tested programs

~ARCH (“testing”) contains new and untested ebuilds

package.mask for bleeding-edge testing and not yet working stuff

developer's overlays for the freaky stuff

USE Flags

enable/disable functionality

system-wide control of features

USE=”-gnome -gtk”

USE=”mmx mmx2 sse sse2”

...

~360 global flags

~800 local (single-package) flags

one ebuild for all variations of a package

SLOTs

Allow multiple versions of packages to be installed

version strings can be used

helper programs:

gcc-config

binutils-config

Even multiple KDE versions are possible! (may violate FHS)

ebuilds

“Build recipes”

Contain all metadata needed for program installation

One file for all supported Architectures, no duplication!

simple Bash-based format

automatic expansion of version numbers etc. from ebuild name

ebuilds - example

# Copyright 1999-2005 Gentoo Foundation

# Distributed under the terms of the GNU General Public License v2

DESCRIPTION="Some random program"

HOMEPAGE="http://foo.bar.com/"

SRC_URI="ftp://foo.bar.com/${P}.tar.gz"

LICENSE="GPL-2"	# over 450 already ...

SLOT="0"

KEYWORDS="~x86 -ppc sparc"

DEPEND="dev-util/ccache-2.3”

RDEPEND=""

ebuilds – example part 2

src_compile() {

./configure \

--host=${CHOST} \

--prefix=/usr \

--infodir=/usr/share/info \

--mandir=/usr/share/man || die "./configure failed"

emake || die "emake failed" }

src_install() {

make \

prefix=${D}/usr \

libdir=${D}/usr/$(get_libdir) \

install || die }

eclasses

Access to common functions

Allow “object-oriented” ebuilds

Implementation causes lots of discussion

Examples:

toolchain.eclass

flag-o-matic

~130 others

ebuild example

inherit eutils java

DEPEND="virtual/libc

!nptl? ( >=sys-libs/glibc-2.3.2-r3 )

!amd64? ( hardened? ( >=sys-libs/glibc-2.3.3 ) )

if use X; then

# Commands specific to X...

fi

Features in 2.0.51

virtuals: many programs provide one “feature”, e.g. virtual/java

parallel fetching

(parallel compiles)

building as user and in a sandbox

lockdown of specific versions

per-package USEflags

Features in 2.0.51

overlays for “personalized” repositories

not affected by updates

if newer version in official repo, it is used

creating and maintaining is easy

sites like BreakMyGentoo offer many modified (broken?) ebuilds

Fetching

mostly protocol-agnostic

files are in /usr/portage/distfiles ... you decide how they get there

Work in progress: binary diff server

only download diff between old and new

Work in progress: better mirror system

40G and growing ... 5M/s and growing ...

transparent caching proxies?

Compiling

ooffice: ~16h on a 2Ghz x86 ...

ccache, distcc

binary packages

problem: no official binpkg-server

which use-flags?

which ARCH, processor class, ...

x86 is pentium, pentium4, K7, C3, ...

we also have PPC, SPARC, MIPS, S390, ...

-> 1TB of binpkgs? :-)

Deploying

it's easy to keep many machines in sync – even multiple platforms

you can use your own metadata repository!

you can add your own (improved ;-)) ebuilds

you can even use your own distfiles mirror!

Quality Assurance

developer make mistakes ...

repoman:

syntax checking

dependency checking

are all deps satisfiable with different settings on all ARCHes?

other sanity checks

needs some code massaging ...

Commiting the ebuilds

At the moment CVS is used

~800M repo

automatic checkout, then pushed to rsync mirrors

update is available within ~30 minutes

“emerge sync” gets you the newest snapshot

“emerge –-update world” installs updates

Bug managment

bugs.gentoo.org

over 80k bugs have been opened

used for ebuild requests, ...

if you have a problem, look there

if problem not documented, please open a bug!

Security features

all downloaded source gets checksummed

all ebuilds should be gpg signed

GLSA for security-related updates

updates within the hour

source-based --> patching is easy

programs are as unpatched as possible

Tinderbox

one of the oldest still open bugs

build all packages in a known-good environment

automatic bugreports, binpkg ...

fully automated build cycle of all packages!

Work in progress ...

The Future

the codebase is ... hmmm ... organic?

user base is large and slow to update

next release might be 2.1

modularization

sanitized

“exotic” features like USE-based dependencies

more general structure

Portage 2.1

Full multilib support (32/64/128bit arches)

Parallel fetching

better Cross-compile support

True multi-repo support

IDS support, system integrity checks

Faster – better caching, algorithm cleanups, ...

Portage 2.1

multi-format checksums? MD5, SHA1 ...

confcache (speedup in ./configure)

transports (modular fetcher)

fakeroot (install to somewhere else)

full gpg signing of all files (support exists, but not mandatory)

better non-Linux support (BSD, Cygwin, MacOS, Plan9, ...)

Portage 3.0

vaporware ;-)

fully modular / OO / ...

buzzword compliant

separation of frontend, backend, ...

network-aware? (Corba / SOAP)

faster!!! (caching, “smart” code, ...)

multiple backends: flat files, DB, ...

support of .deb, .rpm, ... ?

TODO

USE-sets, e.g. “Multimedia”

recompile all packages depending on package after update (MySQL 3->4)

Speedup! “emerge info” can take 20secs ...

UTF-8 support in ebuilds?

if you have an idea, bugs.gentoo.org ... we want input from users!

Comparison

rpm: no good central sources, lots of compatibility problems, no recursive dependency management in rpm itself (yum, up2date, apt-rpm can do it)

deb: good format, but fragmented (knoppix, adamantix, ...)

slackware: ?

BSD ports: similar, but different :-)

portage could even run on cygwin ...

Questions?

Feature requests?

Ideas?

In-depth explanations?

Thanks

to the audience for being here

to all people that contributed to Gentoo

to all Open Source Evangelists and Hackers

to all people that supported my insanity

to Lars Weiler for the Ooffice template

Lars Weiler

2005-02-19T21:08:22