Copr in the Modularity World

Copr is a community build service in Fedora that enables anyone with a FAS account to build their own RPM repositories. This post explains how we can use Copr in modularity, what is already supported, and what needs to be done.

Almost everything in this post starts with a view from the UX perspective. However, technical details are also provided as they are the crucial part of this post.

I have also created a YouTube video showing building modules in Copr + installing.

What I want to achieve

Build systems in Fedora

There are two build systems in Fedora currently:

  1. Koji is the official Fedora build service, that builds everything officially released in Fedora.
  2. Copr is a community build service, available to anyone to build their own RPM repositories. Packages here are not reviewed. Copr in Fedora is something like AUR in Arch Linux.

The purpose of these two build systems should remain unchanged in the modular world. Fedora modules would be built in Koji, and community modules would be built in Copr.

Producing modules

A simplified examples of workflows to build modules in both build systems could be as follows:

  • Koji:
    1. By submitting a modulemd yaml. The whole module including its packages will be built automatically in the pipeline.
  • Copr:
    1. By submitting a modulemd yaml. The whole module including its packages will be built automatically in the pipeline.
    2. By “handcrafting” the module manually. User would submit the builds one by one – as they would in Copr today – and then build a module out of the project. This workflow is described in a more detail below.

Copr could be used to build the initial versions of modules that would eventually end up in Fedora. The “Handcrafting” feature will allow packagers to build and develop their module gradually, and submit it for review in Fedora later when it is ready.

Consuming modules

Official/production modules in Fedora would be identified by ‘name-branch-version’. To install a LAMP stack from Fedora, user would do something like:

$ module-thing install LAMP-something-42

Community modules from Copr would be identified by ‘username/name-branch-version’. To install a LAMP stack from Copr created by user asamalik, user would do something like:

$ module-thing install asamalik/LAMP-something-42

Important detail in both scenarios is using of the same tools to consume both official and community versions.

Technical – modules in Koji vs. modules in Copr

This table is a summary of the main differences. It is also described in a bit more detail below.

Koji Copr
Modules are represented by Tags Projects
Build roots are defined by RPMs with a specific tag External RPM repositories
Finished module builds are RPMs with a specific tag RPM repository
Multiple packages with the same NVR + dist-tag in the build system No Yes – every project has its own isolated namespace for packages.
Multiple modules with the same name in the build system No Yes – every user has its own isolated namespace for modules.


Koji uses tags to work with modules, where tag is basically a group of RPM packages. Each module has a ‘build tag’ with all RPMs defining the build root, and a ‘module tag’ with all the built RPMs that are included in the module as components.

A single RPM can be tagged with multiple tags. The problem is, that there can not be multiple RPMs in Koji with the same NVR and dist-tag. To workaround this problem, we modify the dist-tag of all RPMs in a module to include the name of the module.

Koji can/will be able to create an RPM repository out of every ‘module tag’ using a ‘signed repo’ function.


Copr would use projects to work with modules, where project is something like a directory. One project can define a build root by selecting a predefined mock chroot. This build root can be modified by linking external RPM repositories. To build modules, Copr would introduce an empty mock chroot, and the build root could be defined only by the RPM repository.

A single RPM can be included in only one project. To include one RPM in more projects, it needs to be rebuilt in them. However, there can be multiple packages with the same NVR and dist-tag in Copr. One in every project.

Copr also supports namespaces which is done by usernames. For example, two users can create modules with the same name, stream, release, metadata, … and include an RPM package with the same NVR + dist-tag. These will be completely separated.

Copr automatically creates a signed RPM repository out of every project. “Modularization” of the repository with a yaml spec file is already supported.

Building modules in Copr – conceptual view

This section describes steps that needs to be performed in Copr to build a module. The following steps can be done by a user or an automated tool like the orchestrator.

Copr currently supports everything except empty chroots and releasing a module.

Building a module

  1. Create a project with empty chroot
  2. Define build root by linking external repository
  3. Build all your RPM packages
  4. Release a module, two ways doing that:
    1. Define the module API, install profiles, and package filters by selecting binary packages from a list. [supported soon]
    2. Add the modulemd yaml to the repository. [already supported]

Building a stack

  1. Create a project with empty chroot – let’s call it the main project
  2. Create projects for all the “submodules” in your stack – let’s call them submodule projects
  3. Define build root by linking external repository to all projects you have created
  4. Build all your RPM packages of your “submodules” in their projects
  5. Use the “fork” function in Copr to copy the built RPMs from all your submodule project to the main project
  6. Delete all the submodule projects
  7. Release a module, two ways doing that:
    1. Define the module API, install profiles, and package filters by selecting binary packages from a list. [supported soon]
    2. Add the modulemd yaml to the repository. [already supported]

Handcrafting projects in Copr

This method of building modules is meant for the community members that use Copr today. The main benefits are:

  • Users will not need to write the yaml spec – Copr will automatically generate it for them.
  • User’s workflow of creating a module will be very similar to how they create RPM repositories in Copr today.

However, Copr will also support rebuilding modules from Fedora or building custom modules by submitting the yaml spec.


  1. User creates a project
  2. User defines build root by selecting a “base runtime module” from Fedora – roughly the same way as they would select chroots today
  3. User builds all their RPM packages
  4. User defines the module API, install profiles, and package filters by clicking checkboxes (see mockup below)
  5. User then click on “release module”. Copr will generate the yaml spec for them automatically and create a modular RPM repository.

The picture below shows the UI in Copr to create a module out of a project.

Producing traditional repo and module at the same time

Copr supports multiple chroots in one project. That means I can build the same thing for example for Fedora 23, Fedora 24, Epel 6, and Epel 7. We could add “module” as another chroot, so user could build the same thing for old and new systems. Traditional repositories could be consumed anytime – including installation and updates. Modules would be released every time project enters a release milestone. Modules would remain unchanged forever – the same way as in Fedora.

Copr is a great place for contributors

We have recently moved our code to GitHub, and we support a local development environment using Vagrant – which means, that collaboration have never been easier. Even if you’re very new to the world of open source, this might be a good place for you to start. For example, an easy fix of a typo might be a great opportunity for you to make your first open source contribution! 🙂

What is Copr?

Copr is a community build service in Fedora that builds your code and provides you with your own RPM repository.

Source code is builded in copr and an RPM repository is created.

So, if you are a developer, you can build your application in Copr and make it available for everyone. Or you might be packager – a person, who creates packages of open source software, which are easy to install in Fedora. Then you have two options:

  1. Make the package a part of the Fedora distribution using Fedora Packaging Guidelines. If your package passes a review and all the formal requirements, the package will be added to Fedora and you will become an owner of the package. This means, that you will be responsible for making updates and reporting issues to the upstream, or fixing them. You will become an appreciated member of the Fedora community! 🙂
  2. Build the package in Copr. This option requires less commitment, and it is also suitable for testing or quick and dirty solutions. In fact, it’s totally up to you to decide the quality of your packages and how often you will update them. Copr is a build service for everyone. Learn more about using Copr.

Local development environment with Vagrant

Vagrant can create a local development environment on your workstation. It is very easy to use and will not break your environment, as it installs everything in virtual machines.

Starting the environment

$ git clone
$ cd git
$ vagrant up

This spawns two virtual machines on your workstation:

  1. Copr Frontend – including the web interface and database – http://localhost:5000
  2. Copr Dist Git – storage for the source code – http://localhost:5001

(optional) If you want to start only one specific virtual machine, run one of these commands:

$ vagrant up frontend
$ vagrant up distgit

Testing your changes

To test your changes, commit them to your local repository and reload the virtual machines.


$ git add .
$ git commit -m "changed something really important"
$ vagrant reload frontend

This will restart the Copr Frontend virtual machine, rebuild your package and install it.

Accessing the virtual machines

If you need to access the virtual machines, you can use either:

$ vagrant ssh frontend
$ vagrant ssh distgit

Rebuilding broken virtual machines

Have you destroyed one of the machines completely? That’s fine! To delete it and install it again from the scratch, use:

$ vagrant destroy frontend
$ vagrant up frontend

You can do the same for distgit as well.

Stopping the virtual machines

$ vagrant halt


Now, when you know what Copr is, how to make changes, and you maybe did some experiments, you are ready to change the world! Well, make a pull-request to the Copr project.

To find out what specifically needs to be done, visit our bugzilla. Or ask on our mailing list.

You should also familiarize yourself with pull-requests. You will need to:

  1. fork our project
  2. create a pull-request

And that’s it! If you have any questions about Copr, contributing, or anything related to this article, please leave a comment in the comment section below. I’m looking forward to hear from you!

Useful resources

Fedora 24 chroots in Copr + bonus

Yesterday, Miroslav added fedora-24-i386 and fedora-24-x86_64 chroots to Copr. Chroot for powerPC will be added soon and will be announced on the copr-devel mailing list.

And what is the bonus? We have added the fedora-24-* chroots to every project which had fedora-rawhide-* enabled, and we have copied the repositories as well. In another words, all projects with fedora rawhide should support fedora 24 automatically. You can thank to Jakub, who wrote the script, for that.

Copr, Dist Git and Patternfly

As in the last year, July, for some reason, happened to be a great time to post some news about our Copr Build Service. At this time, it’s about integrating Copr with:

  • Patternfly – an open interface project
  • Dist Git – a remote Git repository designed to hold RPM package sources

If you can’t wait to see it, you can check the development server that is hopefully running on But please, remember, it’s a development server – so all the projects built here are a subject of deletion, destruction and all kinds of randomization without notice.

Dist Git for Copr

It all started with a need of uploading sources to the Copr itself – as the only way of building your package was to provide a URL pointing to it. That, however, required all users to have their own public file storage.

We decided to go the Fedora way and use Dist Git – a combination of git repository to store spec files, lookaside cache to store sources, and Gitolite to manage access permissions. Each package will be stored in a repo named as ‘username/project/package’. Each repo will contain branches that represent a target platform. For example: ‘f22’ for Fedora 22, ‘epel7’ for Epel for Centos 7, etc.

It will be gradually deployed in production. The first step is to enable users to upload their .src.rpm files into Copr. At this step, Dist Git will be used as a storage only. Direct access to the repositories will come afterwards.

New User Interface  Patternfly

Enabling Dist Git required some changes to the user interface as well. We didn’t use any framework, that would help us to easily create new elements in the UI. Instead, we had a custom CSS that received new lines of code with each change. As you can imagine, each change took a bit longer than desired, the CSS became messy, and yes, the interface itself became messy as well. At this point, I decided that we need to do a step forward and we finally agreed to rewrite the user interface to Patternfly! Yay!

The Old Copr UI

The old Copr interface

The New Copr UI

The new Copr interface

We need your feedback!

I would like to make Copr as friendly as possible. If you want to help me with that, please provide a feedback as a comment. Do you like it? Do you hate it? Is there anything you miss in the interface? Your feedback is much appreciated!

Copr update – new features in July 2014

Copr is an easy build service for Fedora which can make you your own repo very easily. I have been contributing to this project since March 2014 and this month has been the best in terms of new features: you can track your builds much better, it is bit easier to use and also got much faster in some cases.

Build detail – packages and versions

This is a quite new view showing all the information about your build progress and results. This month’s new feature is the Build packages section – which shows you names and versions of the result packages which has been built.


API extension – build details

The same information as above is also available via API. Have a look at some more details about Copr API.

Overview page – modified buildroots and generated dnf command

Two new things has been added to the project’s overview page – one of which could help you when building Software collections.

Have you found a project with packages added to the minimal buildroot and want to see a list of them? There is a new button for it: [modified]

And if you decide that you want this repo enabled in your system, just copy the dnf command to your terminal.


Bit of speeding up

There are some mass rebuilds of loads of packages in which just some have changed since last time. Copr is now able to skip the unchanged packages almost immediately as I moved this process out of Mockremote – the build script- to the beginning of the worker process. This means massive speed-up in some cases.

New build states – skipped and starting

If you build a package successfully and some time later you submit the very same one again, it gets skipped. This saves your time and Copr’s resources, because there is no need of doing the same thing again. And from now on it would be marked as Skipped instead of Succeeded to make it more transparent.

The other state is an intermediate step while spawning a builder. It happens between picking up from the queue and the building process itself.