@@ -1,13 +1,18 @@

 Making a Kwant release

 ======================

+

 This document guides a contributor through creating a release of Kwant.

+

 Preflight checks

 ################

+

 The following checks should be made *before* tagging the release.

+

 Check that all issues are resolved

 ----------------------------------

+

 Check that all the issues and merge requests for the appropriate

 `milestone <https://gitlab.kwant-project.org/kwant/kwant/milestones>`

 have been resolved. Any unresolved issues should have their milestone

@@ -16,6 +21,7 @@ bumped.

 Ensure that all tests pass

 --------------------------

+

 This should be as simple as verifying that the latest CI pipeline succeeded.

 For major and minor releases we will be tagging the ``master`` branch.

 For patch releases, the ``stable`` branch.

@@ -23,6 +29,7 @@ For patch releases, the ``stable`` branch.

 Inspect the documentation

 -------------------------

+

 If the CI pipeline succeeded, then the latest docs should be available at:

     https://test.kwant-project.org/doc/<branch name>

@@ -32,6 +39,7 @@ Check that there are no glaring deficiencies.

 Update the ``whatsnew`` file

 ----------------------------

+

 Check that there is an appropriate ``whatsnew`` file in ``doc/source/pre/whatsnew``.

 This should be named as::

@@ -45,28 +53,313 @@ assigned to the release's milestons and get an idea of what was introduced

 from there.

+Make a release, but do not publish it yet

+#########################################

+

+Various problems can surface only during the process of preparing a release and

+make it necessary to fix the codebase.  It would be a pity to have to succeed

+the freshly released version by a minor release just to correct a glitch that

+was detected too late.  Therefore it is a good idea to pursue the release as

+far as possible without announcing it, such that it can be undone and corrected

+if necessary.  In the past tests that failed on the x86-32 architecture and

+wrongly declared dependencies have been detected in this way.

+

+

 Tag the release

-###############

-Make an *annotated*, *signed* tag for the release. The tag must have the semantic

-versioning format::

+---------------

-    v<major>.<minor>.<patch>

+A particularly good way to expose hidden problems is building Debian packages

+using an isolated minimal build environment (cowbuilder).  This approach is

+described here.

-Once the tag has been created, push it to Kwant Gitlab.

+Make an *annotated*, *signed* tag for the release. The tag must have the name::

+    git tag -s v1.2.3 -m "version 1.2.3"

-Post-tagging

-############

-Pushing a new tag to Kwant Gitlab will kick off a few CI jobs, but there is

-still some manual work to do.

+Be sure to respect the format of the tag name (leading "v").  The tag message

+format is the one that has been used so far.

+Do *not* yet push the tag anywhere, it might have to be undone!

-Prepare the Debian package

+

+Build a source taball and inspect it

+------------------------------------

+

+    ./setup.py sdist

+

+This creates the file dist/kwant-1.2.3.tar.gz.  It is a good idea to unpack it

+in /tmp and inspect that builds in isolation and that the tests run::

+

+    cd /tmp

+    tar xzf ~/src/kwant/dist/kwant-1.2.3.tar.gz

+    cd kwant-1.2.3

+    ./setup.py test

+

+

+Clone the repository of the Kwant Debian package

+------------------------------------------------

+

+This step needs to be performed only once.  The cloned repository can be reused

+for subsequent releases.

+

+Clone the "kwant-debian" repository and go into its root directory.  If you

+keep the Kwant source in "src/kwant", a good location for the Debian package

+repository is "src/debian/kwant".  The packaging process creates many files

+that are placed into the parent directory of the packaging repository, hence

+having an additional directory level ("src/debian") is a good way to keep these

+files separate::

+

+    mkdir debian

+    cd debian

+    git clone ssh://git@gitlab.kwant-project.org:443/kwant/debian-kwant.git kwant

+    cd kwant

+

+Create a local upstream branch::

+

+    git branch upstream origin/upstream

+

+Add a remote for the repository that contains the previously created tag::

+

+    git remote add upstream_repo ~/src/kwant

+

+Make sure that::

+

+    git config --get user.name

+    git config --get user.email

+

+show correct information.

+

+

+Release a new version of the Kwant Debian package

+-------------------------------------------------

+

+Fetch the git tag created above into the packaging repo::

+

+    git fetch --all

+

+Now there are two options.  If, as recommended above, the tarball of the new

+version has not been made public yet, it must be imported as follows::

+

+    gbp import-orig ~/src/kwant/dist/kwant-1.2.3.tar.gz

+

+Alternatively, the following commands will import the newest version from PyPI::

+

+    uscan --report      # This will report if a newer version exists on PyPI

+    gbp import-orig --uscan

+

+Update the debian changelog.  Add a point "New upstream release" if there was

+one, and describe any other changes to the Debian *packaging*::

+

+    DEBEMAIL=your.email@somewhere.org gbp dch -R --commit --distribution testing

+

+Now the package can be built with::

+

+    git clean -i

+    gbp buildpackage

+

+But this is *not* how the package should be built for distribution.  For that,

+see the following two sections.

+

+

+Setup git-pbuilder to build Debian packages

+-------------------------------------------

+

+Pbuilder is a tool to build Debian packages in an isolated chroot.  This allows

+to verify that the package indeed only has the declared dependencies.  It also

+allows to cross-build packages for i386 on amd64.

+

+The following describes how to setup git-pbuilder, see also

+https://wiki.debian.org/git-pbuilder.  This procedure needs to be executed only

+once for a Debian system.

+

+Install the Debian package git-buildpackage.

+

+As root, add the following lines to /etc/sudoers or /etc/sudoers.d/local

+

+    Cmnd_Alias BUILD = /usr/sbin/cowbuilder

+

+and

+

+    user     ALL = SETENV: BUILD

+

+Now create pbuilder images.  In the following, replace "buster" by the current

+Debian testing codename::

+

+    ARCH=i386 DIST=buster git-pbuilder create

+    ARCH=amd64 DIST=buster git-pbuilder create

+

+If the packages to be built have special dependencies, use the trick described in https://wiki.debian.org/git-pbuilder#Using_Local_Packages

+

+

+Build Kwant packages using git-pbuilder

+---------------------------------------

+

+Update the builder environment (again, replace "buster" with the name of the

+current Debian testing)::

+

+    ARCH=i386 DIST=buster git-pbuilder update

+    ARCH=amd64 DIST=buster git-pbuilder update

+

+Now build the packages.  First the i386 package.  The option "--git-tag" tags

+and signs the tag if the build is successful.  In a second step, the package is

+built for amd64, but only the architecture-dependent files (not the

+documentation package)::

+

+    gbp buildpackage --git-pbuilder --git-arch=i386 --git-dist=buster --git-tag

+    gbp buildpackage --git-pbuilder --git-arch=amd64 --git-dist=buster --git-pbuilder-options='--binary-arch'

+

+Another example: build source package only::

+

+    gbp buildpackage --git-export-dir=/tmp -S

+

+Build packports for the current Debian stable

+---------------------------------------------

+

+Create a changelog entry for the backport::

+

+    DEBEMAIL=your.email@somewhere.org dch --bpo

+

+Build backported packages::

+

+    gbp buildpackage --git-pbuilder --git-ignore-new --git-arch=i386 --git-dist=stretch

+    gbp buildpackage --git-pbuilder --git-ignore-new --git-arch=amd64 --git-dist=stretch --git-pbuilder-options='--binary-arch'

+

+Do not commit anything.

+

+Publish the release

+###################

+

+If the Debian packages build correctly that means that all tests pass both on

+i386 and amd64, and that no undeclared dependencies are needed.  We can be

+reasonable sure that the release is ready to be published.

+

+git

+---

+

+Push the tag to the official Kwant repository::

+

+    git push origin v1.2.3

+

+PyPI

+----

+

+PyPI (this requires a file ~/.pypirc with a vaild username and password)::

+

+    twine upload -s dist/kwant-1.2.3.tar.gz

+

+It is very important that the tarball uploaded here is the same (bit-by-bit,

+not only the contents) as the one used for the Debian packaging.  Otherwise it

+will not be possible to build the Debian package based on the tarball from

+PyPI.

+

+Kwant website

+-------------

+

+The tarball and its signature (generated by the twine command above) should be

+also made available on the website::

+

+    scp dist/kwant-1.2.3.tar.gz* kwant-website-downloads:downloads/kwant

+

+Debian packages

+---------------

+

+Go to the Debian packaging repository and push out everything::

+

+    git push --all --tags origin

+

+Now the Debian packages that we built previously need to be added to the

+repository of Debian packages on the Kwant website.  So far this the full

+version of this repository is kept on Christoph Groth's machine, so this

+instructions are for reference only.

+

+Go to the reprepro repository directory and verify that the configuration file

+"conf/distributions" looks up-to-date.  It should look something like this::

+

+    Origin: Kwant project

+    Suite: stretch-backports

+    Codename: stretch-backports

+    Version: 9.0

+    Architectures: i386 amd64 source

+    Components: main

+    Description: Unofficial Debian package repository of http://kwant-project.org/

+    SignWith: C3F147F5980F3535

+    

+    Origin: Kwant project

+    Suite: testing

+    Codename: buster

+    Version: 10.0

+    Architectures: i386 amd64 source

+    Components: main

+    Description: Unofficial Debian package repository of http://kwant-project.org/

+    SignWith: C3F147F5980F3535

+

+If the config had to be updated execute::

+

+    reprepro export

+    reprepro --delete createsymlinks

+

+Now the source and binary Debian packages can be added.  The last line has to

+be executed for all the .deb files and may be automated with a shell loop. (Be

+sure to use the appropriate <dist>: either testing or stretch-backports.)::

+

+    reprepro includedsc <dist> ../../src/kwant_1.2.3-1.dsc

+    reprepro includedeb <dist> python3-kwant_1.2.3-1_amd64.deb

+

+Once all the packages have been added, upload the repository::

+

+    rsync -avz --delete dists pool wfw:webapps/downloads/debian

+

+Ubuntu packages

+---------------

+

+Make sure ~/.dput.cf has something like this::

+    [ubuntu-ppa-kwant]

+    fqdn = ppa.launchpad.net

+    method = ftp

+    incoming = ~kwant-project/ppa/ubuntu/

+    login = anonymous

+    allow_unsigned_uploads = 0

+

+We will also use the following script (prepare_ppa_upload)::

+    #!/bin/sh

+    

+    if [ $# -eq 0 ]; then

+        echo -e "\nUsage: $(basename $0) lousy mourning2 nasty\n"

+        exit

+    fi

+    

+    version=`dpkg-parsechangelog --show-field Version`

+    mv debian/changelog /tmp/changelog.$$

+    

+    for release in $@; do

+        cp /tmp/changelog.$$ debian/changelog

+        DEBEMAIL=christoph.groth@cea.fr dch -b -v "$version~$release" -u low 'Ubuntu PPA upload'

+        sed -i -e "1,1 s/UNRELEASED/$release/" debian/changelog

+        debuild -S -sa

+    done

+    

+    mv /tmp/changelog.$$ debian/changelog

+

+Make sure that the Debian package builds correctly and go to its directory.

+

+Check https://wiki.ubuntu.com/Releases for the current releases and execute::

+    prepare_ppa_upload trusty vivid wily

+

+(if a second upload of the same Debian version is needed, something like vivid2 can be used)

+

+Now the changes files are "put" to start the build process on the PPA servers::

+

+    cd ..

+    dput ubuntu-ppa-kwant *~*.changes

 Gather the autobuilt packages from CI

 -------------------------------------

+

+(This section needs to be updated.  Using the sdist package as generated by

+PyPI requires pushing the tag to gitlab and as such is incompatible with doing

+the Debian packaging with an unpublished tag.)

+

 CI automatically generates:

 + HTML documentation

@@ -80,6 +373,10 @@ These can be found in the artifacts of the last CI job in the pipeline,

 Publish to the Kwant website

 ----------------------------

+

+(This section needs to be updated.  The twine tool creates the signature file

+during the upload.)

+

 To do the following you will need access to the webroots of ``downloads.kwant-project.org``

 and ``kwant-project.org``. Ask Christoph Groth if you need to be granted access.

@@ -104,6 +401,9 @@ on ``kwant-project.org``. Point the symbolic link ``<major>`` to this directory.

 Publish to PyPI

 ---------------

+

+(This also needs to be updated.)

+

 Install `twine <https://pypi.python.org/pypi/twine>` and use it to upload

 the tar archive in the ``sdist`` directory of the Ci artifacts downloaded

 in the previous step::

@@ -121,6 +421,7 @@ Publish to Launchpad

 Publish to Conda forge

 ----------------------

+

 Conda forge automates build/deploy by using CI on Github repositoried containing

 recipes for making packages from their source distributions.

@@ -139,6 +440,7 @@ to review and accept the pull request, so that Kwant will be rebuilt.

 Announce the release

 --------------------

+

 Write a short post summarizing the highlights of the release on the

 `Kwant website <https://gitlab.kwant-project.org/kwant/website>`, then

 post this to the mailing list kwant-discuss@kwant-project.org.

@@ -146,6 +448,7 @@ post this to the mailing list kwant-discuss@kwant-project.org.

 Working towards the next release

 ################################

+

 After finalizing a release, a new ``whatsnew`` file should be created for

 the *next* release, and this addition should be committed and tagged as::