=============================
Workflow for use by Debian CI
=============================

We want to allow ci.debian.net to use Debusine as a backend.  Our existing
:task:`Autopkgtest` task and :workflow:`autopkgtest` workflow can do most of
the hard work, but will need some changes.

Overall model
=============

ci.debian.net should have its own workspace and its own
:collection:`debian:qa-results` collections.  The ci.debian.net operators
should have the ``OWNER`` role on this workspace, while bot accounts should
have the ``CONTRIBUTOR`` role.  This workspace can have suitable
:ref:`expiration policies <expiration-of-data>`, retaining logs for a few
months and basic result history indefinitely (although ci.debian.net will
keep its own copies of the history as well, at least to begin with).

``debci`` will start workflows as needed, wait for them to complete, and
collect the results.  Until we have outbound webhooks to notify ``debci`` of
workflow completion, it should simply poll occasionally for the status of
workflows whose results it has not yet collected.

In future, it might be possible to have ``britney`` dispatch requests to
Debusine directly, and nothing in this design should make that more
difficult.  However, ``debci`` has quite a bit of intelligence of its own as
well as the ci.debian.net frontend, so we recommend that it be kept in the
loop at least for the first iteration of this work.

Looking up packages
===================

We can assume that Debusine's mirrored suites are reasonably up to date, and
use lookups into them.  This won't be quite perfect, especially for packages
that have been uploaded very recently; but the integration between
``britney`` and ci.debian.net already knows which version of a package was
tested and will repeat tests until it gets a result from the desired
version, so we can tolerate some slack here.  ``debci`` can also choose to
use lookups with versions so that workflows will fail immediately if the
desired version is unavailable.

Being able to test lookups from the client side (:issue:`691`) could help
with this.

Pinning packages
================

ci.debian.net currently uses the ``autopkgtest --pin-packages`` option.  For
example, a test run triggered by changes to source packages ``foo`` and
``bar`` from unstable would use ``--pin-packages=unstable=src:foo,src:bar``.
However, as noted in the :task:`Autopkgtest` task, we have deliberately not
added support for that until now due to concerns about it not being explicit
enough.

We could add support for this anyway and tolerate the unexpected behaviour.
However, with a few small changes, it should be possible to get
nearly-equivalent results using something similar to this:

.. code-block:: yaml

    context_artifacts:
    	- collection: sid@debian:suite
          category: debian:binary-package
          error_on_empty: true
          data__srcpkg_name: "foo"
          data__srcpkg_version: "1.0-1"
          data__architecture__in: [amd64, all]
        - collection: sid@debian:suite
          category: debian:binary-package
          error_on_empty: true
          data__srcpkg_name: "bar"
          data__srcpkg_version: "2.0-2"
          data__architecture__in: [amd64, all]

This requires adding ``error_on_empty: true`` to :ref:`dictionary lookups
<lookup-dict>` to make it be an error if the lookup returns no results, and
accepting the ``__in`` suffix in lookup matchers.

When used, this would be more precise: Debusine would know exactly which
binary artifacts were requested, automatically recording them in the dynamic
task data.  The required extensions seem worthwhile in general.

``britney`` does not currently pass the source version of pinned packages in
its request to ``debci`` (unless it were to parse the ``trigger`` field,
which seems undesirable).  We could change that protocol, or we could omit
``data__srcpkg_version`` from the example lookups above, which would at
least still record the binary artifacts in dynamic task data.

Unfortunately, this is still not quite enough.  `autopkgtest(1)
<https://manpages.debian.org/stable/autopkgtest/autopkgtest.1.en.html>`__
has this note about ``--pin-packages``:

    **Attention**: This does not currently resolve some situations where
    dependencies of the given packages can only be resolved in ``RELEASE``.
    In this case the apt pinning will be removed and package installation
    will be retried with the entirety of ``RELEASE``, unless
    ``--no-apt-fallback`` is specified.

The Debusine task cannot handle this itself, unless we were to add something
complicated like preemptive dependency resolution or parsing the output of
``autopkgtest`` for failures to resolve dependencies.  As a result, while we
can still use ``context_artifacts`` to increase precision, there seems no
alternative but to add ``pin_packages`` parameters to the
:task:`Autopkgtest` task and the :workflow:`autopkgtest` workflow, with
notes about their limitations.  It does appear to be possible to pass
``--pin-packages=unstable=`` (without a package list) to ``autopkgtest``
along with additional binaries, which should have the same fallback
behaviour.

Extra APT sources and signing keys
==================================

The ``extra_repositories`` option (defined in the same way as in the generic
:task:`PackageBuild` task) should already be good enough for this.

APT retries
===========

``debci`` uses ``autopkgtest --setup-commands`` to set ``Acquire::Retries
"10";`` in APT's configuration.  This seems like a good idea for robustness,
but there's no need for this to be done at the task level; we can simply use
``customization_script`` to adjust the environments we use.  This could be
done specifically for ``autopkgtest``, but we might as well do it across the
board.

Backend selection
=================

``incus-lxc`` should be suitable for most packages.
:ref:`task-configuration`, maintained by the owners of the relevant
workspace, can be used to configure some packages to use ``incus-vm``
instead.

Regression tracking
===================

Debusine supports tracking and analyzing regressions, controlled by various
options to QA-related workflows, particularly ``update_qa_results`` and
``enable_regression_tracking``.  ``britney`` already does similar work, and
the Debian release team doesn't want to outsource that.  However, ``debci``
needs to be able to schedule reference test runs, and it makes sense to use
parts of the same mechanism to do so even without the regression analysis
step.  This should be done using the following parameters:

.. code-block:: yaml

    qa_suite: forky@debian:suite
    reference_qa_results: forky@debian:qa-results
    update_qa_results: true

This will skip running the task if the reference
:collection:`debian:qa-results` collection already contains an analysis from
the last 30 days with a version matching the one published in ``qa_suite``.
However, reference runs sometimes break for other reasons such as external
data or the passage of time (e.g. expired certificates), so ``debci`` needs
to be able to force an update.  This will be done by changing the definition
of the ``update_qa_results`` parameter to :workflow:`autopkgtest` as
follows:

* ``update_qa_results`` (string, defaults to ``no``): whether to update
  reference QA results.  Allowed values are ``no``, ``yes``, and ``force``.
  When set to ``yes``, the workflow runs QA tasks and updates the collection
  passed in ``reference_qa_results`` with the results, unless that
  collection already contains a current matching result.  ``force`` is like
  ``yes``, but does not check whether the collection already contains a
  current matching result.

  For compatibility, boolean False is equivalent to ``no``, and boolean True
  is equivalent to ``yes``.

Between them, ``britney/scripts/debci-put.py`` and ``debci`` need to arrange
for the above parameters to be passed to Debusine when running
``migration-reference/0`` tests.  ``debci`` will then collect the results
and store them in the same kind of way that it collects all other results
from Debusine.
