More flexible workflow templates
================================

Possible future work
--------------------

Nested structures
~~~~~~~~~~~~~~~~~

Counterintuitively, only top-level keys are considered while merging
workflow template and user data: lists and dictionaries are not merged, and
if a list or a dictionary exists in both the workflow template and the user
data, the version in the user data is ignored.  Most of our current
workflows have been designed to take this into account, but that requires a
flat structure for workflow parameters that can be a little overwhelming.

However, changing this would have other implications.  ``dput-ng`` profiles
have similar nesting limitations, which caused us to unnest
Debusine-specific entries there in :mr:`1769`, and adding nesting within
``debusine_workflow_data`` would reintroduce that difficulty.

Also, the current flat structure suggests the possibility of a web UI to
start a workflow that offers each of the available parameters as options and
introspects the corresponding Pydantic models to pick the appropriate
widgets; it becomes less obvious how to do this if we increase nesting.

Additional variable templating
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Some workflow task data entries are naturally constructed from other
entries: for example, collection names can be constructed based on
vendor/codename.  A possible direction would be to add a template language
that can build template data based on some kind of variable expansion pass,
in order that only a minimal set of variables needs to be presented as
options to the user.

However, adding templates to generate templates carries intrinsic risk of
confusion, and many of the same benefits can be gained by more thoughtful
workflow design: workflow orchestrators are free to do their own
computations when building values for child work requests, and they can do
so in ways that would be difficult to express in a template language, such
as looking up state in the database.  For now, it seems best to defer this
until (if ever) we can design a domain-specific language powerful enough to
express everything we do in workflow orchestrators, at which point something
like this might be a good fit for it.

autopkgtest parameter choice restrictions
-----------------------------------------

A workspace administrator might want to allow setting
``autopkgtest_backend`` to ``unshare`` or ``incus-lxc`` but not to
``incus-vm`` or ``qemu``, for resource consumption reasons.

However, this particular case cannot be handled solely at the workflow
template level.  ``autopkgtest_backend: auto`` is currently synonymous with
``autopkgtest_backend: unshare``, but for :issue:`803` it would be useful to
have it be sensitive to whether the autopkgtest in question declares the
``isolation-container`` or ``isolation-machine`` restriction and use the
simplest possible backend that will work.

To make that work, the task itself would need to be responsible for deciding
what ``auto`` means.  Merging template and user task data happens before the
task runs, so neither new ``WorkflowTemplate`` fields nor the current
:ref:`task configuration <task-configuration>` mechanism can easily express
this restriction.

A new ``allow_backends`` parameter seems tempting, but there are some
problems.  The configuration context for the :task:`Autopkgtest` task is
currently just the codename, so we can't use that for architecture-specific
configuration and would need to complicate the task with explicit
by-architecture parameters.  This may be a reason to add multiple
configuration contexts.

We also need to handle this sort of backend selection at the workflow level
rather than the task level, since the decision of which worker to dispatch
the task to may depend on which backend is selected.  That introduces the
difficulty that the workflow doesn't currently have access to the
``Restrictions`` declared by a given source package's tests, and `that
information isn't in the .dsc <https://bugs.debian.org/847926>`__; it would
have to unpack the source package to find those, which would have to be done
on a worker since running ``dpkg-source -x`` on untrusted source packages
isn't safe, and then cache the result somewhere.
