Copr in the Modularity World

6 minute read

Copr is a community build service in Fedora that enables anyone with a FAS account to build their own RPM repositories. This post explains how we can use Copr in modularity, what is already supported, and what needs to be done.

Almost everything in this post starts with a view from the UX perspective. However, technical details are also provided as they are the crucial part of this post.

I have also created a YouTube video showing building modules in Copr + installing.

What I want to achieve

Build systems in Fedora

There are two build systems in Fedora currently:

  1. Koji is the official Fedora build service, that builds everything officially released in Fedora.
  • Copr is a community build service, available to anyone to build their own RPM repositories. Packages here are not reviewed. Copr in Fedora is something like AUR in Arch Linux.

    • The purpose of these two build systems should remain unchanged in the modular world. Fedora modules would be built in Koji, and community modules would be built in Copr.

    Producing modules

    A simplified examples of workflows to build modules in both build systems could be as follows:

    • **Koji:**
      1. By submitting a modulemd yaml. The whole module including its packages will be built automatically in the pipeline.
    • **Copr:**
      1. By submitting a modulemd yaml. The whole module including its packages will be built automatically in the pipeline.
      2. By “handcrafting” the module manually. User would submit the builds one by one – as they would in Copr today – and then build a module out of the project. This workflow is described in a more detail below.

    Copr could be used to build the initial versions of modules that would eventually end up in Fedora. The “Handcrafting” feature will allow packagers to build and develop their module gradually, and submit it for review in Fedora later when it is ready.

    Consuming modules

    Official/production modules in Fedora would be identified by ‘name-branch-version’. To install a LAMP stack from Fedora, user would do something like:

    $ module-thing install LAMP-something-42

    Community modules from Copr would be identified by ‘username/name-branch-version’. To install a LAMP stack from Copr created by user asamalik, user would do something like:

    $ module-thing install asamalik/LAMP-something-42

    Important detail in both scenarios is using of the same tools to consume both official and community versions.

    Technical – modules in Koji vs. modules in Copr

    This table is a summary of the main differences. It is also described in a bit more detail below.

    Koji Copr
    Modules are represented by Tags Projects
    Build roots are defined by RPMs with a specific tag External RPM repositories
    Finished module builds are RPMs with a specific tag RPM repository
    Multiple packages with the same NVR + dist-tag in the build system No Yes – every project has its own isolated namespace for packages.
    Multiple modules with the same name in the build system No Yes – every user has its own isolated namespace for modules.

    Koji

    Koji uses tags to work with modules, where tag is basically a group of RPM packages. Each module has a ‘build tag’ with all RPMs defining the build root, and a ‘module tag’ with all the built RPMs that are included in the module as components.

    A single RPM can be tagged with multiple tags. The problem is, that there can not be multiple RPMs in Koji with the same NVR and dist-tag. To workaround this problem, we modify the dist-tag of all RPMs in a module to include the name of the module.

    Koji can/will be able to create an RPM repository out of every ‘module tag’ using a ‘signed repo’ function.

    Copr

    Copr would use projects to work with modules, where project is something like a directory. One project can define a build root by selecting a predefined mock chroot. This build root can be modified by linking external RPM repositories. To build modules, Copr would introduce an empty mock chroot, and the build root could be defined only by the RPM repository.

    A single RPM can be included in only one project. To include one RPM in more projects, it needs to be rebuilt in them. However, there can be multiple packages with the same NVR and dist-tag in Copr. One in every project.

    Copr also supports namespaces which is done by usernames. For example, two users can create modules with the same name, stream, release, metadata, … and include an RPM package with the same NVR + dist-tag. These will be completely separated.

    Copr automatically creates a signed RPM repository out of every project. “Modularization” of the repository with a yaml spec file is already supported.

    Building modules in Copr – conceptual view

    This section describes steps that needs to be performed in Copr to build a module. The following steps can be done by a user or an automated tool like the orchestrator.

    Copr currently supports everything except empty chroots and releasing a module.

    Building a module

  • Create a project with empty chroot
  • Define build root by linking external repository
  • Build all your RPM packages
  • Release a module, two ways doing that:
    1. Define the module API, install profiles, and package filters by selecting binary packages from a list. [supported soon]
    2. Add the modulemd yaml to the repository. [already supported]

    • Building a stack

  • Create a project with empty chroot – let’s call it the main project
  • Create projects for all the “submodules” in your stack – let’s call them submodule projects
  • Define build root by linking external repository to all projects you have created
  • Build all your RPM packages of your “submodules” in their projects
  • Use the “fork” function in Copr to copy the built RPMs from all your submodule project to the main project
  • Delete all the submodule projects
  • Release a module, two ways doing that:
    1. Define the module API, install profiles, and package filters by selecting binary packages from a list. [supported soon]
    2. Add the modulemd yaml to the repository. [already supported]

    • Handcrafting projects in Copr

    This method of building modules is meant for the community members that use Copr today. The main benefits are:

  • Users will not need to write the yaml spec – Copr will automatically generate it for them.
  • User’s workflow of creating a module will be very similar to how they create RPM repositories in Copr today.

    • However, Copr will also support rebuilding modules from Fedora or building custom modules by submitting the yaml spec.

    Workflow

  • User creates a project
  • User defines build root by selecting a “base runtime module” from Fedora – roughly the same way as they would select chroots today
  • User builds all their RPM packages
  • User defines the module API, install profiles, and package filters by clicking checkboxes (see mockup below)
  • User then click on “release module”. Copr will generate the yaml spec for them automatically and create a modular RPM repository.

    • The picture below shows the UI in Copr to create a module out of a project.

    Producing traditional repo and module at the same time

    Copr supports multiple chroots in one project. That means I can build the same thing for example for Fedora 23, Fedora 24, Epel 6, and Epel 7. We could add “module” as another chroot, so user could build the same thing for old and new systems. Traditional repositories could be consumed anytime – including installation and updates. Modules would be released every time project enters a release milestone. Modules would remain unchanged forever – the same way as in Fedora.

    Updated: