Copr in the Modularity World

Copr is a community build service in Fedora that enables anyone with a FAS account to build their own RPM repositories. This post explains how we can use Copr in modularity, what is already supported, and what needs to be done.

Almost everything in this post starts with a view from the UX perspective. However, technical details are also provided as they are the crucial part of this post.

I have also created a YouTube video showing building modules in Copr + installing.

What I want to achieve

Build systems in Fedora

There are two build systems in Fedora currently:

  1. Koji is the official Fedora build service, that builds everything officially released in Fedora.
  2. Copr is a community build service, available to anyone to build their own RPM repositories. Packages here are not reviewed. Copr in Fedora is something like AUR in Arch Linux.

The purpose of these two build systems should remain unchanged in the modular world. Fedora modules would be built in Koji, and community modules would be built in Copr.

Producing modules

A simplified examples of workflows to build modules in both build systems could be as follows:

  • Koji:
    1. By submitting a modulemd yaml. The whole module including its packages will be built automatically in the pipeline.
  • Copr:
    1. By submitting a modulemd yaml. The whole module including its packages will be built automatically in the pipeline.
    2. By “handcrafting” the module manually. User would submit the builds one by one – as they would in Copr today – and then build a module out of the project. This workflow is described in a more detail below.

Copr could be used to build the initial versions of modules that would eventually end up in Fedora. The “Handcrafting” feature will allow packagers to build and develop their module gradually, and submit it for review in Fedora later when it is ready.

Consuming modules

Official/production modules in Fedora would be identified by ‘name-branch-version’. To install a LAMP stack from Fedora, user would do something like:

$ module-thing install LAMP-something-42

Community modules from Copr would be identified by ‘username/name-branch-version’. To install a LAMP stack from Copr created by user asamalik, user would do something like:

$ module-thing install asamalik/LAMP-something-42

Important detail in both scenarios is using of the same tools to consume both official and community versions.

Technical – modules in Koji vs. modules in Copr

This table is a summary of the main differences. It is also described in a bit more detail below.

Koji Copr
Modules are represented by Tags Projects
Build roots are defined by RPMs with a specific tag External RPM repositories
Finished module builds are RPMs with a specific tag RPM repository
Multiple packages with the same NVR + dist-tag in the build system No Yes – every project has its own isolated namespace for packages.
Multiple modules with the same name in the build system No Yes – every user has its own isolated namespace for modules.


Koji uses tags to work with modules, where tag is basically a group of RPM packages. Each module has a ‘build tag’ with all RPMs defining the build root, and a ‘module tag’ with all the built RPMs that are included in the module as components.

A single RPM can be tagged with multiple tags. The problem is, that there can not be multiple RPMs in Koji with the same NVR and dist-tag. To workaround this problem, we modify the dist-tag of all RPMs in a module to include the name of the module.

Koji can/will be able to create an RPM repository out of every ‘module tag’ using a ‘signed repo’ function.


Copr would use projects to work with modules, where project is something like a directory. One project can define a build root by selecting a predefined mock chroot. This build root can be modified by linking external RPM repositories. To build modules, Copr would introduce an empty mock chroot, and the build root could be defined only by the RPM repository.

A single RPM can be included in only one project. To include one RPM in more projects, it needs to be rebuilt in them. However, there can be multiple packages with the same NVR and dist-tag in Copr. One in every project.

Copr also supports namespaces which is done by usernames. For example, two users can create modules with the same name, stream, release, metadata, … and include an RPM package with the same NVR + dist-tag. These will be completely separated.

Copr automatically creates a signed RPM repository out of every project. “Modularization” of the repository with a yaml spec file is already supported.

Building modules in Copr – conceptual view

This section describes steps that needs to be performed in Copr to build a module. The following steps can be done by a user or an automated tool like the orchestrator.

Copr currently supports everything except empty chroots and releasing a module.

Building a module

  1. Create a project with empty chroot
  2. Define build root by linking external repository
  3. Build all your RPM packages
  4. Release a module, two ways doing that:
    1. Define the module API, install profiles, and package filters by selecting binary packages from a list. [supported soon]
    2. Add the modulemd yaml to the repository. [already supported]

Building a stack

  1. Create a project with empty chroot – let’s call it the main project
  2. Create projects for all the “submodules” in your stack – let’s call them submodule projects
  3. Define build root by linking external repository to all projects you have created
  4. Build all your RPM packages of your “submodules” in their projects
  5. Use the “fork” function in Copr to copy the built RPMs from all your submodule project to the main project
  6. Delete all the submodule projects
  7. Release a module, two ways doing that:
    1. Define the module API, install profiles, and package filters by selecting binary packages from a list. [supported soon]
    2. Add the modulemd yaml to the repository. [already supported]

Handcrafting projects in Copr

This method of building modules is meant for the community members that use Copr today. The main benefits are:

  • Users will not need to write the yaml spec – Copr will automatically generate it for them.
  • User’s workflow of creating a module will be very similar to how they create RPM repositories in Copr today.

However, Copr will also support rebuilding modules from Fedora or building custom modules by submitting the yaml spec.


  1. User creates a project
  2. User defines build root by selecting a “base runtime module” from Fedora – roughly the same way as they would select chroots today
  3. User builds all their RPM packages
  4. User defines the module API, install profiles, and package filters by clicking checkboxes (see mockup below)
  5. User then click on “release module”. Copr will generate the yaml spec for them automatically and create a modular RPM repository.

The picture below shows the UI in Copr to create a module out of a project.

Producing traditional repo and module at the same time

Copr supports multiple chroots in one project. That means I can build the same thing for example for Fedora 23, Fedora 24, Epel 6, and Epel 7. We could add “module” as another chroot, so user could build the same thing for old and new systems. Traditional repositories could be consumed anytime – including installation and updates. Modules would be released every time project enters a release milestone. Modules would remain unchanged forever – the same way as in Fedora.

Fedora 24 on MacBook Pro 11,4 and 11,5 – suspend and brightness fix

I have a great news for those of you with a MacBook Pro 15″ 2015 (MacBook 11,4 and 11,5)!

As you might have noticed, when running Linux on this machine, it can’t be suspended, shut down, and you can’t even control brightness of the display. These issues have been reported a while ago, and yes, there are some patches, but they didn’t make it to the production code.

Suspend and shutdown:

So I took those patches, applied them to the kernel in Fedora and built it in my Copr project.

How to make it running on your machine? First, download and install Fedora 24. Then you just need to enable my Copr project and install the patched kernel:

$ sudo dnf copr enable asamalik/MacBook-kernel
$ sudo dnf install kernel-4.6.6-300.AdamsMacBookSleepBrightness.fc24

Reboot your machine with the patched kernel, and that’s it!

I will try to keep the repo updated, so you will be able to update kernel as usual.

Roles of PDC and BPO in Modularity – please give me feedback

There has been some confusion about the roles of PDC and BPO in Modularity as the roles might seem overlapping. This document will briefly explain the roles of both services and highlight the main differences.

Long story short:

  • PDC is a database that stores module metadata and dependency graphs
  • BPO is a web UI to display module build state and probably to browse some metadata

Main overlaps:

  • BPO has its own database. However, the database acts as a cache only to make the UI fast. All the data can be lost and recovered from other services such as Orchestrator and PDC.
  • PDC has its own web UI. However, browsing and displaying modules has not been implemented. And even if it was, PDC does not/should not store data such as build state.

From my understanding, this design has been already valid on the Modularity Infra wiki page:

PDC – Product Definition Centre

PDC is the primary database for module metadata such as:

  • Name, stream, release – or however are these going to be called
  • Dependencies and components
  • Koji tags
  • Maybe information such as SLA?

It provides a REST API to access and manipulate this data.

A part of PDC is also a web UI that will not be used in Modularity.

BPO – Build Pipeline Overview

BPO is a single web UI that will watch over the whole pipeline. Its primary function is to show the build state of each module.

Users will be also able to browse module metadata such as:

  • Dependencies
  • Components
  • Install profiles

BPO uses its own database as a cache only to make the UI faster. The database:

  • Will be updated by fedmsg and also queries of other services as reactions to some messages.
  • Can be deleted and recovered from other services.

Flock 2016

Last week, I visited the Flock 2016 conference in the beautiful city of Krakow. And it was brilliant!

If you asked me about three most important things I took from the conference, I would probably tell you the following:

First: I met a lot of interesting people. And I think this is the most important thing about any conference. I finally had a chance to meet people I knew before over email and irc (or at least knew they exist) and had a chance to talk to them personally. From my experience, it was always much easier to work remotely with someone I know or at least met before for a while. It makes conversations much more comfortable, especially over IRC where you can’t express or see emotions of the other side. I would love to do this more often! :-)

Second: I talked to many people and a lot of them were not from Red Hat! Also, during the keynote on Tuesday morning we saw some statistics including a chart showing that less than 1/3 of contributors to Fedora are Red Hatters. Fedora is a true open source project!

Third: The Modularity project I currently work on seems accepted by people. I didn’t hear anyone to say “no” and people seemed very interested in the topic. Both talk and workshop had full rooms, and there were a lot of questions asked. BTW, there is a demo on YouTube showing how Modularity looks from the user perspective.

Finally, big thanks to the the people who organized the event! You did a really great job. Everything just worked and it was a very nice experience!

Make your Gnome title bar smaller – Fedora 24 update

I wrote similar post before about making the massive title bar in Gnome smaller. But it doesn’t work anymore in Fedora 24!

Don’t worry, I have created an update for you – so you can enjoy more space on your desktop again!

gnome-window-title-barAll you need to do is to put the following css code into ~/.config/gtk-3.0/gtk.css

window.ssd headerbar.titlebar {
    padding-top: 4px;
    padding-bottom: 4px;
    min-height: 0;

window.ssd headerbar.titlebar button.titlebutton {
    padding: 0px;
    min-height: 0;
    min-width: 0;

Refreshed Look of Fedora Developer Portal

I have just deployed a new version of Fedora Developer Portal. The most visible part is refreshed look with more uniform layout. I have also compressed all the images in titles (from ~1.2MB to ~50kB in average) – so the loading should be much faster.

There is also new section called Start a project. This section contains/will contain content from a problem-to-be-solved perspective like ‘Create a Website’ or ‘Work with Arduino’. This new section is the opposite of Languages and Databases, which is a ‘language-specific’ section.

As I have already mentioned, the new section is partly empty. This is another new feature – we have also published a selection of unfinished/empty pages, that we would like to include. And this is your turn! You can help us by contributing content. All information is about contributing is on our new site for contributors – also linked from the portal.

If you would like to get involved, get in touch with us on our mailing list or ping us on irc #developer-portal at freenode.

Copr is a great place for contributors

We have recently moved our code to GitHub, and we support a local development environment using Vagrant – which means, that collaboration have never been easier. Even if you’re very new to the world of open source, this might be a good place for you to start. For example, an easy fix of a typo might be a great opportunity for you to make your first open source contribution! :-)

What is Copr?

Copr is a community build service in Fedora that builds your code and provides you with your own RPM repository.

Source code is builded in copr and an RPM repository is created.

So, if you are a developer, you can build your application in Copr and make it available for everyone. Or you might be packager – a person, who creates packages of open source software, which are easy to install in Fedora. Then you have two options:

  1. Make the package a part of the Fedora distribution using Fedora Packaging Guidelines. If your package passes a review and all the formal requirements, the package will be added to Fedora and you will become an owner of the package. This means, that you will be responsible for making updates and reporting issues to the upstream, or fixing them. You will become an appreciated member of the Fedora community! :-)
  2. Build the package in Copr. This option requires less commitment, and it is also suitable for testing or quick and dirty solutions. In fact, it’s totally up to you to decide the quality of your packages and how often you will update them. Copr is a build service for everyone. Learn more about using Copr.

Local development environment with Vagrant

Vagrant can create a local development environment on your workstation. It is very easy to use and will not break your environment, as it installs everything in virtual machines.

Starting the environment

$ git clone
$ cd git
$ vagrant up

This spawns two virtual machines on your workstation:

  1. Copr Frontend – including the web interface and database – http://localhost:5000
  2. Copr Dist Git – storage for the source code – http://localhost:5001

(optional) If you want to start only one specific virtual machine, run one of these commands:

$ vagrant up frontend
$ vagrant up distgit

Testing your changes

To test your changes, commit them to your local repository and reload the virtual machines.


$ git add .
$ git commit -m "changed something really important"
$ vagrant reload frontend

This will restart the Copr Frontend virtual machine, rebuild your package and install it.

Accessing the virtual machines

If you need to access the virtual machines, you can use either:

$ vagrant ssh frontend
$ vagrant ssh distgit

Rebuilding broken virtual machines

Have you destroyed one of the machines completely? That’s fine! To delete it and install it again from the scratch, use:

$ vagrant destroy frontend
$ vagrant up frontend

You can do the same for distgit as well.

Stopping the virtual machines

$ vagrant halt


Now, when you know what Copr is, how to make changes, and you maybe did some experiments, you are ready to change the world! Well, make a pull-request to the Copr project.

To find out what specifically needs to be done, visit our bugzilla. Or ask on our mailing list.

You should also familiarize yourself with pull-requests. You will need to:

  1. fork our project
  2. create a pull-request

And that’s it! If you have any questions about Copr, contributing, or anything related to this article, please leave a comment in the comment section below. I’m looking forward to hear from you!

Useful resources

Fedora 24 chroots in Copr + bonus

Yesterday, Miroslav added fedora-24-i386 and fedora-24-x86_64 chroots to Copr. Chroot for powerPC will be added soon and will be announced on the copr-devel mailing list.

And what is the bonus? We have added the fedora-24-* chroots to every project which had fedora-rawhide-* enabled, and we have copied the repositories as well. In another words, all projects with fedora rawhide should support fedora 24 automatically. You can thank to Jakub, who wrote the script, for that.

Make your Gnome title bars smaller

Update: See the updated version for Gnome 3.20 and Fedora 24.

I don’t like the size of title bars in the stock Gnome 3. They are big and take to much space on my tiny 12″ screen! But I’ve found an easy solution to this.

gnome-window-title-barAll you need to do is to put the following css code into ~/.config/gtk-3.0/gtk.css

.header-bar.default-decoration {
 padding-top: 3px;
 padding-bottom: 3px;
 font-size: 0.8em;

.header-bar.default-decoration .button.titlebutton {
 padding: 0px;