Package rebuilds in Modularity

People say that Modularity means rebuilding stuff over and over. Is that true? It might be for now. Important thing is that it’s not the goal, it’s just a first step. So why do we do that? It’s all about Application Programming Interface (API), Application Binary Interface (ABI), and automation.

Before we start, let’s refresh one of our goals. The goal is to make the whole process of building and releasing a distribution more efficient, and with the ability to ship more versions and features. That sounds like something we can’t do all at once.

The priority for us at the moment are humans. We want to automate as much of the packagers’ workflow as we can. Even for the cost of computers doing significantly more work. We’ll make it easy for the computers later.

The story of API and ABI compatibility

API compatibility is a compatibility on the source level. An example of this would be a package that builds for all releases (epel6, epel7, f26, f27, etc.) from the same source. While the source is the same, the binary for each release is different. The f27 binary might not work on epel6 etc.

ABI compatibility is a compatibility on the binary level. In this scenario, the same binary would run on all releases (epel6, epel7, f26, f27, etc.).

In the traditional Fedora release, we technically don’t expect any compatibility between releases. For every release of Fedora, we store the sources separately and we build them separately. That means the sources of all packages in the Fedora 26 release are stored in dist-git in an f26 branch, sources of Fedora 27 packages are in an f27 branch, etc.

So technically, there doesn’t need to be an API nor ABI compatibility between releases, because they are completely separated. However, in reality, many things are stable across multiple releases. If you are a packager, you might have a package that builds for all the releases from a single source just fine. That’s where Modularity helps.

Modularity introduces the concept of API compatibility between releases. Thanks to Arbitrary Branching, packagers can have a single branch of an application that is API compatible with multiple releases, and the build system will automatically build different binaries.

An important thing is that the build system will build the binaries automatically. We want to make the packagers’ life as easy as possible, and that’s why there is a new service called Freshmaker that will automate two things: building binaries when the source changes, and rebuilding existing binaries in case of an ABI change. The most important word from this paragraph is “will”—I’m mostly talking about the future.

The ultimate goal

The ideal state is that the build system knows exactly what needs to get built, and it reuses the binaries whenever possible.

In this ideal state, when a packager pushes an update to a certain package, the build system automatically rebuild the package to produce a binary for every release. If the package is ABI compatible with all the releases, it would build it once. If not, it would build it multiple times, but not more than needed.

The next step would be dependencies. If the package introduced an ABI change, it would automatically rebuild all its dependencies that wouldn’t work with this new ABI. In case the ABI is the same as before, no additional rebuilds would occur.

Implementing an automatic mechanism that would decide about when exactly to reuse a binary, and when to rebuild dependencies based on ABI changes is a rather complex task. While this is the ultimate goal, it’s not how it works these days.

The current state

At the time of writing this post, the build system doesn’t know anything about ABI compatibility. But we have the same requirements—builds need to be automatic, and things need to work.

In theory, without any information about the ABI, the only way of being 100 % sure things will work is to rebuild everything. When a new version of a package is built, all its dependencies need to be rebuilt as well. Also all the dependencies of the dependencies needs to be rebuilt, etc. This extreme approach is not maintainable in the real world. We don’t want every update of glibc to trigger a rebuild of the whole distribution. So there needs to be a reasonable compromise.

The current implementation is a combination of limiting the rebuilds, trusting the ABI of module streams, and testing.

Modules are rebuilt automatically when one of their components change. Package rebuilds are done automatically only within the module, with a simple optimisation using build groups which I have described in my previous Module builds – how do they work? post.

However, rebuilding a module will not automatically trigger a rebuild of other modules having a dependency on it. This is where we need to trust the ABI stability of a module stream, together with testing the result after every build.


So yes, there are some extra rebuilds, for now. Important is that this is not the goal. It is a first iteration of providing an automated build pipeline while making the the packagers’ lives easier.

Module builds – how do they work?

Have you ever wondered how modular builds work? In my previous post called Building Modules for the F27 Server, I described the steps a packager needs to go through in order to produce a successful build in the Fedora infrastructure. But what exactly is going on at the backstage?


The build pipeline sees modules as build recipes for RPM packages. Each module has a list of SRPM (source RPM) packages that get build within the module. These packages are divided into build groups which are processed in batches one by one. Build groups are useful in situations when certain packages are needed in the buildroot to build some other ones within the same module. How these build groups work and what services are they part of?

All packages are build in Koji—the same service that builds packages for the traditional Fedora release. Interesting is that Koji doesn’t know anything about modules or build groups. And it uses the same mechanics in both modular and traditional releases. How it knows what to do? That’s where the Module Build Service (MBS) comes in.

MBS orchestrates package builds in Koji. This service makes sure that packages are built in a defined order by leveraging the concept of build groups, and reports the overall state of module builds to the user and other services. While MBS manages most of the steps of module builds, it relies on Koji to do the actual building, and to store the output such as binary packages and logs.

When a module build is finished, it is processed by other services to produce the final artifacts such as RPM repositories, container base images, cloud images, or bootable iso images. Artifacts are typically composed out of multiple modules.

Building a module

Everything starts when a module build gets submitted to MBS. At the time of writing this post, this action is done manually by the packager. They need to run the `mbs-build submit`command in their dist-git repository to get the build submitted. However, a new service called Freshmaker is being deployed. This service will automatically submit builds for every commit to dist-git.

Initial check

When a build is submitted to MBS, it enters the “init” state. During this state, MBS checks if the modulemd file is valid, and that all components of the modules are available. If everything is alright, the state switches to “wait”.

Modules in the “wait” state are waiting for a capacity in the build system. When a capacity is available, MBS switches the state to “build”. And that’s where the real building starts.

Preparing a buildroot

Before diving into describing the “build” state, we need to understand two concepts Koji uses during module builds: tags and tag inheritance. Tags are basically groups of Koji builds. A single build can be tagged into multiple tags. Tags can represent buildroots, releases, or any other groups of builds—like modules. MBS creates two tags for every module build: the buildroot, and the result. Buildroots are typically composed of multiple modules. Koji uses the tag inheritance mechanism to combine multiple tags into one. How is a module buildroot created?

Module buildroot is a tag in Koji representing the environment—a set of packages—where a module gets build. Constructing a buildroot is a first step of the “build” state. The buildroot is an empty tag that inherits all the modules specified as build dependencies in the modulemd that is being built. When this is done, MBS can submit package builds into Koji.

The first package that is built as a part of every module build is the `module-build-macros`. It contains a set of macros that are used when building other packages. When this build is finished, the result is tagged back to the buildroot tag—so it is available for all other components.

Building packages in build groups

Packages are built in batches called build groups. These build groups are defined in modulemd using a buildorder value. The buildorder value is an integer, every component (package) has it, and the default value is zero. Components with the same buildorder value are part of the same build group.

MBS schedules build groups one-by-one, starting with the lowest buildorder value. All packages within a build group are submitted at once and built in parallel. When all builds within the build group finish, the results are tagged back to the buildroot tag—so they are available for the next ones. This is repeated until all build groups are finished.

When all build groups are finished, MBS changes the module build state. If all component builds succeeded, the build state is set to “done”. In case there is one or more failures, the state is set to “failed”. Both of these states mean that the build phase has ended.

After the build

There is another state in MBS indicating that modules are ready to be part of a larger compose and consumed by the end-user. The state is called “ready”. Right now, MBS switches all states “done” to “ready” automatically. However, in the future, this will be probably switched by an external service that sees the bigger picture. Part of the bigger picture can be rebuilding all modular dependencies, additional integration testing, etc.

Next step – release

Modules in the “ready” state are not automatically available to the end user. To have a certain module available in a release, it needs to be added to a compose which is then used to produce the final artifacts such as RPM repositories, container base images, cloud images, or bootable iso images. I might write additional blog post describing this in the future.

This post has been mostly focused on the build phase. There are many other interesting services in the Factory 2.0 that make Modularity possible. Have a look at the Factory 2.0 Focus Documents for more information. You can also learn how to build modules for the F27 Server, see the F27 content plan and help us with building more modules, or browse the Modularity documentation to get more information about the effort.

Building Modules for Fedora 27

Let me start with a wrong presumption that you have everything set up – you are a packager who knows what they want to achieve, you have a dist-git repository created, you have all the tooling installed. And of course, you know what Modularity is, and how and why do we use modulemd to define modular content. You know what Host, Platform, and Bootstrap modules are and how to use them.

Why would I make wrong presumptions like that? First of all, it might not be wrong at all. Especially, if you are a returning visitor, you don’t want to learn about the one-off things every time. I want to start with the stuff you will be doing on regular basis. And instead of explaining the basics over and over again, I will just point you to the right places that will help you solve a specific problem, like not having a dist-git repository.

Updating a module

Let’s go through the whole process of updating a module. I will introduce some build failures on purpose, so you know how to deal with them should they happen to you.

Start with cloning the module dist-git repository to get the modulemd file defining the installer module.

$ fedpkg clone modules/installer
$ cd installer
$ ls
Makefile installer.yaml sources tests

I want to build an updated version of installer. I have generated a new, updated version of modulemd (I will be talking about generating modulemd files later in this post), and I am ready to build it.

$ git add installer.yaml
$ git commit -m "building a new version"
$ git push
$ mbs-build submit
asamalik's build #942 of installer-master has been submitted

Now I want to watch the build and see how it goes.

$ mbs-build watch 942

 40 components in the COMPLETE state
 3 components in the FAILED state
asamalik's build #942 of installer-master is in the "failed" state

Oh no, it failed! I reviewed the Koji logs using the links above, and I see that:

  • NetworkManager and python-urllib3 failed randomly on tests. That happens sometimes and just resubmitting them would fix it.
  • realmd however needs to be fixed before I can proceed.

After fixing realmd and updating the modulemd to reference the fix, I can submit a new build.

$ git add installer.yaml
$ git commit -m "use the right version of automake with realmd"
$ git push
$ mbs-build submit
asamalik's build #942 of installer-master has been submitted

To watch the build, I use the following command.

$ mbs-build watch 943

 42 components in the COMPLETE state
 1 components in the FAILED state
asamalik's build #943 of installer-master is in the "failed" state

Good news is that realmd worked this time. However, sssd failed. I know it built before, and by investigating the logs I found out it’s one of the random test failures again. Resubmitting the build will fix it. In this case, I don’t need to create a new version, just resubmit the current one.

$ mbs-build submit
asamalik's build #943 of installer-master has been resubmitted

Watch the build:

$ mbs-build watch 943
 43 components in the COMPLETE state
asamalik's build #943 of installer-master is in the "ready" state

Rebuilding against new base

The Platform module has been updated and there is a soname bump in rpmlib. I need to rebuild the module against the new platform. However, I’m not changing anything in the modulemd. I know that builds map 1:1 to git commits, so I need to create an empty commit and submit the build.

$ git commit --allow-empty -m "rebuild against new platform"
$ git push
$ mbs-build submit
asamalik's build #952 of installer-master has been submitted

Making sure you have everything

Now it’s the time to make sure you are not missing a step in your module building journey! Did I miss something? Ask for it in the comments or tweet at me and I will try to update the post.

How do I get a dist-git repository?

To get a dist-git repository, you need to have your modulemd to go through a Fedora review process for modules. Please make sure your modulemd comply with the Fedora packaging guidelines for modules. Completing the review process wil result in having a dist-git repository.

What packages go into a module?

Your module need to run on top of the Platform module which together with the Host and Shim modules form the base operating system. You can see the definition of Platform, Host, and Shim modules on github.

You should include all the dependencies needed for your module to run on the Platform module, with few exceptions:

  • If your module needs a common runtime (like Perl or Python) that are already modularized, you shoud use these as a dependencies rather than bundling them to your module.
  • If you see that your module needs a common package (like net-tools), you shouldn’t bundle them either. They should be split into individual modules.
  • To make sure your module doesn’t conflict with other modules, it can’t contain the same packages as other modules. Your module can either depend on these other modules, or such packages can be split into new modules.

To make it easier during the transition from a traditional release to a modular one, there is a set of useful scripts in the dependency-report-scripts repository, and the results are being pushed to the dependency-report repository. You are welcome to participate.

Adding your module to the dependency-report

The module definitions are in the modularity-modules organizations, in a format simlar to the Platform and Host definition. The scripts will take these repositories as an input together with the Fedora Everything repository to produce dependency reports and basic modulemd files.

How can I generate a modulemd file?

The dependency-report-scripts can produce a basic modulemd, stored in the dependency-report repository. The modulemd files stored in the dependency-report repository reference their components using the f27 branch for now. To produce a more reproducible and predictable result, I recommend you to use the script inside the dependency-report-scripts repository:


The result will be stored at the same place as the previous modulemd, that was using branches as references.

$ ls output/modules/MODULE_NAME/MODULE_NAME.yaml

Where do I get the mbs-build tool?

Install the module-build-service package from Fedora repositories:

$ sudo dnf install module-build-service

What can I build to help?

See the F27 Content Tracking repository and help with the modules that are listed, or propose new ones. You can also ask on #fedora-modularity IRC channel.

Where can I learn more about Modularity?

Please see the Fedora Modularity documentation website.



Flock 2017 was awesome!

Flock is a Fedora contributors’ conference happening every year, alternating between the US and Europe. And this year it was a more action-oriented than in previous years which means there were more workshops and ‘do-sessions’, and less talks.

I must admit I liked that change because I think that presenting a project online is easier than quick discussions and getting things done online. So using this even with many people together to do stuff as opposed to presentations seems pretty effective.

I was there to represent the Fedora Modularity effort – giving two and a half talks, collecting feedback and ideas, and learning from awesome people.


Day one


Matt Miller talked about the current state of Fedora. There are many changes going on, and it’s a good thing. We don’t want to be completely on fire, but also not super stable because we want to innovate. So the answer is: Just a little bit of fire. Matt had the following chart on his slides showing what our audience

And this chart is what I took from the keynote. Doing innovation means braking stuff. And that’s fine as long as we keep it under control.

Advertise your session

Everyone got a chance to sell their presentations or workshops to others before the conference began. There was a long line of people and everyone was introducing themselves and their presentation or workshop to others. Interesting idea, I liked the concept, there were just so many people I was loosing track a little bit. My proposal for next time: could we group them by topic? What do you think?

User Feedback on Modularity

This was a bit special event, as it was running six times during the conference! Two slots every day, an hour before lunch, and an hour after lunch. This fact alone felt like Modularity is pretty important to Fedora.

And it was also important to the Modularity people. We got a lot of feedback – both nice and useful. Every single one liked the overall concept of Modularity, majority also liked the UX.


One of the things I liked was that every day was a little bit different. Why? Because we had Martin Hatina – a DNF developer – with us. And he was fixing the code every day as people were discovering little bugs!

Factory 2.0, Fedora, and the Future

Mike Bonnet gave an intro into the Factory 2.0 project, giving an overview of all the new and old services that are part of that project and that are being worked on. It was a verygood presentation, also serving as an introduction to other, more detailed talks the team was giving, as he provided a high-level overview of everything that’s going on in Factory, and pointed us to the other talks for more details.

He gave an introduction into the following projects:

  • Module Build Service (MBS) – A service for building modules in Fedora.
  • Arbitrary Branching – Naming branches to follow upstream version of the component instead of a Fedora release.
  • ResultsDB – Test results from the infrastructure on a single place.
  • WaiverDB – Allowing tests to be waived, turning a fail into a pass.
  • Greenwave – Gating for content to move to a new state – like pushing updates automatically.
  • Bodhi – Package (and now also module) updates in Fedora. Gating on Greenwave decisions now deployed!
  • Freshmaker – Automatically triggers builds of content when its source gets updated.
  • ODCS – On-demand Compose Service, generates repos of signed pre-release content to build other content like containers, ostrees, and to run tests.

State of Fedora Server

Stephen Gallagher talked about the history, current state, and the future of Fedora Server Edition.

People could learn that previously, there was just a single Fedora ‘edition’ designed to run on a laptop and on a server. And that was fine then, but later on, both groups started to have different needs – like different defaults on a fresh system. On a laptop, you probably want to have a window manager – like Gnome – running. On a server, you might want a different GUI to manage it remotely, like Cockpit. And that’s why they split the Server Edition and the Workstation Edition apart.

And that is very nice! When you install a fresh Fedora Server, it has Cockpit running by default. That means you can just open your web browser on your laptop, log in to your server, and you’ll be able to see all the logs, CPU and memory consumption, and to perform basic configuration like managing your LVM volumes. Thanks to Cockpit apps you are (or will be in some cases) also able to manage other services like your containers or the FreeIPA server.

While the Server and Workstation split enabled this, there are still needs like having multiple versions available or independent lifecycles of some components. And that’s where Modularity comes in. Modularity enables parallel availability of content – so people will be able to choose different versions of software to install, or even the Server WG will be able to choose different default (even with a different lifecycle) for the Server as opposed to the Workstation. During the discussion, someone mentioned that Modularity is like customizing a laptop. You can choose the color, the CPU – but you can only fit one, etc. And I liked that. It was also nice to see people who are not ‘us’ to understand Modularity well enough to explain it to others.

Proposal: Splitting docs from the software

This was my first talk. I gave a short introduction into the problem of having the software and the documentation in a single source. This also applies to tests. Right now, to build a package, you need to have all the build requires for your software, for the documentation, and for the tests. That grows the build dependencies a lot which is a problem with Modularity where we want to be as flexible as we can – so having many build dependencies lowers the flexibility. That’s a problem especially for the base which hould have as little build dependencies as possible. Texlive or python-sphinx might be a nice example.

After the short introduction, we had a discussion about how we could solve that problem. I proposed two solutions: having a standard macro in SPEC that could switch the documentation off, or separating docs from the software on a source level. There are advantages to both: the first one is easier to maintain, as you have just a single thing. Also, the version of the docs will always match the version of the software. The drawbacks are that you need to rebuild the SRPM with the correct macros to get the right build requires. The second option is easier for build, because you have two separate things.

I was lucky enough to have Jason Tibbitts from the Fedora Packagibg Comittee in the discussion. And we decided to go with the second option – separating docs from the software on a source level for the critical components. However, we won’t force anyone to do that. Instead, we will explicitly mention in the packaging guidelines that this is absolutely fine, and recommended for some components, and then we’ll ask maintainers of the core components to do the change.

Day two

How to write dist-git tests in Fedora with Ansible

Stef Walter believes that lot of work in the future will be done by robots! And that includes testing our packages. Tests for the CI.. I mean.. robots can be described using Ansible.

I learned that you can work in two scenarios:

  • setting up the environment like VMs or containers for the tests to run in
  • setting up the environment like VMs or containers as part of the tests

Learn more:


Jan Kaluza makes automation happen! He presented the project he works on called Freshmaker, that automatically makes fresh things. It is a service that triggers rebuilds of content on a source change. It will make packagers’ life easier by removing many manual steps from their workflow.

The short-term goal for Freshmaker is to:

  • trigger container rebuilds when an RPM source changes
  • trigger module rebuilds when a modulemd changes

Gating on automated tests in Fedora – Greenwave

Matt Jia talked about Greenwave – a service that makes clever decisions about pushing content to another stage based on test results (consumed from ResultsDB and WaiverDB) and policy. This, for example, will enable pushing updates in Bodhi automatically, based on transparent criteria.

Atomic Host 101

OK, I’m lying. I wasn’t really here. But Dusty made an awesome job with the worshop and made it available online with all the instructions you need. And it’s completely offline! Once you download all the bits and pieces, you can play with it anywhere. So I tried it on the way back home. You should try it, too!

The most impressive thing was changing the OS from Fedora to CentOS with a single command, and everything worked fine after a reboot. Go for it:

The Future of fedmsg?

Jeremy Cline explained the current state of fedmsg, the problems they’re having with it, and a possible solution.

Right now, when using fedmsg, you basically need to subscribe everything to everything. And the messages are not guaranteed to be delivered.

The most interesting for me was a discussion about guaranteed vs. not guaranteed messaging. Basically, when you have a guaranteed message bus, it makes things very complicated. The sender needs to know about all the receivers. So you can no longer just consume the messages as you can now with fedmsg. Also, you can’t entirely trust the message broker, so you need to have a code in your app to handle this anyway. It seemed to me that building a reliable messaging bus might not be worth it.. but is it the right answer? I dont’ know.. but it’s pretty interesting problem and got me thinking about that for sure.

Evening Activity @ Wackenhammer’s Clockwork Arcade and Carousel

It was very relaxing, good fun, here’s a picture of me winning a 1000 tickets, and let’s move on!


Day three

The Modularity day! We had four hours in the afternoon. So let’s start.

Modularity – the future, building, and packaging

Join session of Langdon White, Ralph Bean, and me. We got a full room!

Langdon talked about the vision, why are we doing it, and some of the benefits it brings. He talked about the F26 Boltron Server we have built, the F27 Server MVP we are currently working on, about building Atomic as rolling updates separated from the other editions which is now work in progress, and about Workstation using Flatpaks in the future.

One of my favourites was a demonstration how to build container images with multiple versions of things without changing the dockerfile. See Langdon’s slides for the example. He also emphasized how we want to make packagers’ life as easy as possible, mostly by generating many of the things and running tests in the infrastructure.

Ralph gave us update on the Module Build Service (MBS) – a service that builds modules in Fedora. It is for developement and production. MBS runs in the Fedora infrastructure and orchestrates builds in Koji, but the same thing can also run on your laptop and use Mock to do your local builds. That means they share the code-base between the infra and the client tool which is a good thing.

We could also learn how MBS reuses components. There is a concept of build groups people can define in their modulemd. And basically all the builds get reused if their buildroot (which is the previous build group) haven’t changed. It’s very nicely explained in Ralph’s slides – so have a look!

And finally, in my part I talked about packaging. I explained the difference between traditional distribution, where packages have a branch for every release, and releases are built out of these branches; and the modular distro, where we have arbitrary branching, multiple version of content, and we use modulemd to describe the builds and the modules produced out of it.

I also talked about an idea of defining spins and editions using modulemd files. See my slides for more details and a bunch of graphics!

When to go fully modular?

This was a discussion about the future of Modularity. During the session I realized I should have advertised it as a Fedora Modularity WG meeting live. That would have been awesome. So maybe next time.

How many modules will be there?

A module could be described as a group of packages with additional metadata. But how bit is the group? Do we create a ‘text-editors’ module, do we create ‘vim’, ‘nano’, ’emacs’, and others? And should they include all the dependencies, or shoud the dependencies be shared between modules? The result is that we want to do the later, a module per application or service. And that they should share the dependencies.

But how to share the dependnecies? In a huge ‘shared-userspace’ module? Or using many small modules? And how to coordinate all of this so the modules won’t overlap? The outcome wasn’t that clear this time. The consensus was that creating smaller sensible modules as dependencies instead of ‘shared-userspace’ is probably better. But how small do we want to go? Do we want to push it to the extreme and have modules with a single package? And what about the overlaps and coordination?

Flatpaks and containers make this much easier, as they can overlap with each other by design. But installing RPMs directly on a system makes it difficult. We will need a service that will help packagers to coordinate. Something similar to the dependency-report I have prototoyped, but as a service, and focusing on the long-term maintenance over one off migration.

Is there an everything else module?

There are basically two main approaches how to modularize a distro:

  1. Build a huge ‘everything else’ module and separate individual modules out of it as we go. This way, we would have all the content.
  2. Modularize everything – including dependencies – from the start. We wouldn’t have all the content from the beginning, but it would be much cleaner. And we could get rid of some packages that are no longer maintained, but are being automatically rebuilt over and over again.

Someone also had an idea to include the traditional Fedora Everything repo in the modular distro to give users the content. However, it would conflict with the modularized packages, and could also have binary incompatibilities. That was clearly a bad idea. But what about the everything _else_ module?

I think most of the people would go with the option 2, modularize everything. And that’s what we do with the F27 Server.

How could we do Rawhide in Modularity?

That was an interesting question. Also, how do we do releases in the pure modularity world? Will we have a releases as we know them now? Or will individual modules (or module stacks) have their own releases? Would we have a rawhide as we know it today, for the whole distro, or will modules have their own ‘rawhide’ streams? We haven’t agreed on this one.

How do we deal with module EOL?

When every module and every RPM can have independent EOL (end of life), it can turn into a pretty messy situation when something can expire any time. And that’s not good. Matt Miller proposed we have a standardized EOL dates, for example every 6 months. That way, it would be basically the same as we have now, except some RPMs and modules could live longer. And that was the consensus in the end.

Let’s create a module

Tomas Tomecek did a workshop for packagers where they could build their module and see the whole process from the beginning to the end. It’s availabe on github: Modularity Workshop.

Let’s create tests for modules/containers

Petr Hracek and Jan Scotka explained people how to use the Meta Test Family to write tests for modules. It supports writing tests in any framework and running them in any environment. More information here: Meta Test Family.

Day four

What did we do / Demo day

People had a chance to present what they have achieved during the conference. It was interesting to see what other people learned and got done. As a side-effect, it helped me remember some of the things I have been part of myself. Even though there was a bit pressure in the end asking people to come and speak, I think it was just because people were not used to this sort of thin on previous Flocks. I believe that next year the pressure will disappear as more people will be getting ready for this during the week. I still think it was useful and I liked it.


It was awesome. I believe that just meeting people face to face is so important, as it makes the collaboration for the upcoming year so much easier when people know each other. Besides the sessions, I had many hallway (and pub) discussions. I liked seing other people to understand Modularity well enough they could explain it to others. I liked getting all the feedback and great ideas – especially the little ones, that were part of the hallway discussions. I liked the positive perception of Modularity on the UX demos. I learned a bunch of new stuff, met some new people I haven’t known in person before.

Thank you for having me there, it was very useful and pleasant experience!


Modular F27 Server Edition – initial design

Recently, I have started a discussion on the Server mailing list about building the Fedora 27 Server Edition using Modularity. Langdon White is already working on a change request. If that happens, there will be a lot of work in front of us. So let’s start with writing blog posts!

To build the Fedora 27 Server Edition using Modularity, Adam thinks we need to focus on two things:

First is the initial design of the basic set of modules for the F27 Server Edition – including the Host and Platform modules, as well as other ‘application/content’ modules. To make this easier, I’m proposing a tool temporarily called The Graph Thing.

Second is a great packager UX for the build pipeline. This will lead to more content built by the community. It will include The Graph Thing and BPO, and I will be talking about it in a different post.

This post covers the first part – the initial design.

Modularizing the F27 Server Edition – introduction

To build Fedora 27 Server Edition using Modularity, we need to split the monolithic distro into smaller modules. Modules can come in multiple streams and on independent lifecycles. However, in Fedora 27, all modules will be on the same lifecycle of 13 months as the rest of Fedora 27.

Splitting Fedora 27 into modules

The Platform team defines the Host and Platform modules by deciding which packages are needed in which modules and including their dependencies into these modules. They already started defining the Host and Platform modules in the Host and Platform GitHub repository.

Other modules will be defined based on the Fedora 27 Server Edition use-cases. We need to create modules for all server roles and other components that will be part of the server. Defining these modules will also influence the Platform module as it will include some of the packages shared between other modules.

Splitting the distribution into individual modules is not easy. We need to work with hundreds of packages with complex dependencies and carefully decide what package goes into which module. To do the initial distribution design, I am proposing a tool that will help us. I temporarily call it The Graph Thing and I hope the name will change soon.

After the initial split, we expect other people adding modules to the distribution as well. Making the packager UX great is crucial for us, if we want to get a lot of content from the community. Packagers will be also able to use The Graph Thing for the initial design of their module, and Build Pipeline Overview (BPO) to monitor their builds.

The Graph Thing – a tool for the initial design

The Graph Thing is a tool that will help people with the initial design of individual modules when modularizing a distribution. It will also help packagers with adding other modules later on.

It will work with resources from the Fedora infrastructure, inputs from the user, and will produce a dependency graph as an output. An example with pictures is better than a thousand words:

Example – defining a nodejs module

The Dependency Thing - first run

In this example, there are three things available as resources:

  • Host module definition
  • Platform module definition
  • Fedora 27 repository – including all the Fedora 27 packages

A packager wants to add a new module: nodejs. The packager have specified in the input that they want to visualize the host and platform modules, as well as the new nodejs module which doesn’t exist in the infrastructure. The packager added a nodejs-6.1 package into that module and wants to see if that’s enough for the module, or if they need to add something else.

The output shows that the nodejs module will require two other packages directly, library-foo-3.6 and nodejs-doc-6.1, and that the nodejs-doc-6.1 also needs a crazy-thing-1.2.

Seeing this, the packager made the following decision:

  1. The library-foo-3.6 is a package specific for nodejs, so it should be part of the nodejs module.
  2. The nodejs module should not include documentation, as it will be shipped separately. They will modify the nodejs-6.1 package not to include the nodejs-doc-6.1.

With the decision done, the packager wants to see how it’s going to look like if they apply the changes. So they modify the input of The Graph Thing and run it again:

Great! It looks like that the module now includes everything that is needed. The packager can make the change in the nodejs-6.1 package and submit this module for a build in the Fedora infrastructure.

Value of The Graph Thing tool

As we saw in the example above, the tool can help with designing new modules by looking at the Fedora 27 packages and giving an instant view of how a certain change could look like, without rebuilding stuff.

It will also help us identify which packages need to be shared between individual modules, so we can decide if we include them in the Platform, or it we make a shared module. An obvious example of a shared module can be a language runtime, like Python or Perl. Other, not that obvious, can be identified with this tool.

Apart from the initial design of the Fedora 27 Server Edition, this tool could be also a part of the packager UX which I will be talking about soon.

Early implementation

I have already created an early partial implementation of this tool. You can find it in my asamalik/modularity-dep-viz GitHub repo.

Next steps

The tool needs to get rewritten to use libsolv instead of multiple repoquery queries – so it produces the correct output, and takes seconds rather than minutes to produce an output. It could also be merged with another similar project called depchase which has been used by the Platform team to define the Base Runtime and Bootstrap modules for the F26 Boltron release.

DORS/CLUC 2017 event report

DORS/CLUC is a regional open source software conference in Zagreb, Croatia. It happens every year at the Faculty of Electrical Engineering and Computing, targeting Linux users and professionals. I have attended the conference for the first time about two weeks ago, and I would like to share my notes from the conference with you.

Fedora got two talks this year:

  1. Fedora and VR glass by Gergely Rákosi
  2. Fedora Modularity by me, Adam Samalik

There is also a Fedora wiki page for the event.

How Adam did?

Let me give you some metrics of me being at the conference… I was using Twitter as my communication channel and to get some data about my presence.

The official DORS/CLUC Twitter account did about 60 tweets the first day, 10 were retweets, and I got 4 of them.

My most popular tweet by views got about 5000 views during the first 5 hours – before my talk, about 7600 for the whole duration of the conference, and 10133 at the time of writing this post. Thanks @fedora and others for retweeting!

The most retweeted one was about Gergley’s VR talk and got 12 retweets, 15 likes, and about 2500 views.

My talk was attended by roughly 50 – 80 people (I’m still not able to count people quickly enough) which was about the average. The conference had only one track, so there was not much choice anyway. I got a few questions right after the talks and some interesting discussions afterwards.

At the end of my talk, they tweeted about the Modularity YouTube channel and invited people to contribute.

I have published my slides at SlideShare: Fedora Modularity at DORS/CLUC 2017

Apart from the the Modularity talk, I also talked to people about Fedora and did four demos of the system on my laptop. One included installing and demonstrating KDE to a Windows user who really loved the customization options and installed Fedora KDE spin on his laptop later that evening.


Modularity update – sprint 31

Hello people of the world! Let me give you another update of the Fedora Modularity project.

What we did

We are moving the modules to the new Fedora Modularity: modules space on GitHub. All work regarding modules will be done primarily here and synced into fedora dist-git for builds. Each module has its own repository with an issue tracker.

The Modularity documentation has been updated and slightly restructured.

There is also a new demo about the dnf prototype supporting modules on our YouTube channel.

Try the DNF prototype

You can try everything you have seen in the video yourself! We have a container image with everything you need to get started. See the DNF Modularity Prototype repository for the source, or get the image directly from Docker hub:

$ docker pull jamesantill/flat-modules-dnf

Then start a container using this image with an interactive shell:

$ docker run --rm -it jamesantill/flat-modules-dnf /bin/bash

Now, when you are in the container, you can try all the commands. And please give us feedback either in the comment section below, in the issue tracker, or get in touch on the #fedora-modularity freenode channel.

What’s next?

  • Complete all modules, images, and tests.
  • Prepare an example and a demo of multiple streams of NodeJS.
  • Review and prioritize open issues.

Modularity update – sprint 30

The Fedora Modularity team already publishes sprint reports on the Modularity YouTube channel every two weeks. But this format might not be always suitable – like watching on a phone with limited data. So I would like to start writing short reports every two weeks about Modularity, so people have more choice of how to get updated.

What we did

  • We have the final list of modules we are shipping in F26 Boltron. The list shows Python 2 and Python 3 as not-included which is not entirely true. Even thought we won’t be shipping them as separate modules due to various packaging resons, they will be included in Boltron as part of the Base Runtime and shared-userspace.
  • One of them is shared-userspace which is a huge module that contains common runtime and build dependencies with proven ABI stability over time. Lesson learned: building huge modules is hard. We might want to create smaller ones join them together as a module stack.
  • To demonstrate multiple streams we will include NodeJS 6 as part of Boltron, and NodeJS 8 in Copr – built by its maintainer.
  • The DNF team has implemented a fully functional DNF version that supports modules.
  • We have changed the way we do demos on YouTube. Instead of posting a demo every two weeks of work per person, we will do a sprint review + user-focused demos as we go. I will also do my best with writing these posts. 🙂

What’s next?


  • clean up and make sure they deliver the use cases we promised
  • the same for containers if time allows

Documentation and community:

  • issue tracker for each module
  • revisiting documentation
  • revisiting how-to guides


  • we would love to make a demo based on a working compose (if we get a qcow)

Also, I’m going to Zagreb to give a Modularity talk at DORS/CLUC next week. Come if you’re near by! 😉

Fedora Modularity Documentation

Wiki pages are great for collaboration. But they are not that great in getting people’s attention. They can also become pretty messy and hard to navigate trough when using multiple pages that are related to each other – like documentation – which was what we had there. We needed something better. Something that would make it easy to go trough multiple pages of documentation. Something that would have a simple landing page explaining what we do. And having a simple way to review the changes people make before publishing them would be also great.

I knew we wanted something better, but I didn’t know what exactly. I also didn’t want to invent yet another way to build docs. So I looked around, and found the Fedora Release Engineering documentation. It’s hosted in Pagure Docs, it’s built with Python Sphinx, and it also used to be a wiki. And I got inspired!

So I created some drafts, made a proposal, and convinced people from the Modularity Working Group that we need a cool website. And then I just built it. Using the same tech as the Fedora Release Engineering team – Python Sphinx to build, Pagure Docs to host.

But because I’m a wanna-be designer, I also created a logo, and wrote a custom Sphinx template including a simple landing page that helps people quickly understand what Modularity is about.

And I’m happy to announce that the new Fedora Modularity documentation website has been published today!

To edit the documentation, please send pull-requests against our source repository. And maybe get in touch if you like the project!

Early module development with Fedora Modularity

So you like the Fedora Modularity project – where we separate the lifecycle of software from the distribution – and you want to start with module development early? Maybe to have it ready for the modular Fedora 26 Server preview? Start developing your modulemd on your local system now, and have it ready for later when the Module Build Service is in production!

Defining your module

To have your module build, you need to start with writing a modulemd file which is a definition of your module including the components, API, and all the information necessary to build your module like specifying the build root and a build order for the packages. Let’s have a look at an example Vim module:

document: modulemd
version: 1
    summary: A ubiquitous text editor
    description: >
        Vim is a highly configurable text editor built to make creating and
        changing any kind of text very efficient.
            - MIT
        content: []
    xmd: ~
            generational-core: master
            generational-core: master
                - vim-enhanced
                - vim-minimal
                - vim-X11
            - vim-minimal
            - vim-enhanced
            - vim-X11
        rpms: ~
                rationale: The minimal variant of VIM, /usr/bin/vi.
                rationale: The enhanced variant of VIM.
                rationale: The GUI variant of VIM.
                rationale: Common files needed by all VIM variants.
                rationale: The directory structure used by VIM packages.

Notice that there is no information about the name or version of the module. That’s because the build system takes this information from the git repository, from which the module is build:

  • Git repository name == module name
  • Git repository branch == module stream
  • Commit timestamp == module version

You can also see my own FTP module for reference.

To build your own module, you need to create a Git repository with the modulemd file. The name of your repo and the file must match the name of your module:

$ mkdir my-module
$ touch my-module/my-module.yml

The core idea about modules is that they include all the dependencies in themselves. Well, except the base packages found in the Base Runtime API – which haven’t been defined yet. But don’t worry, you can use this list of binary packages in the meantime.

So the components list in your modulemd need to include all the dependencies except the ones mentioned above. You can get a list of recursive dependencies for any package by using repoquery:

$ repoquery --requires --recursive --resolve PACKAGE_NAME

When you have this ready, you can start with building your module.

Building your module

To build a modulemd, you need to have the Module Build Service installed on your system. There are two ways of achieving that:

  1. Installing the module-build-service package with all its dependencies.
  2. Using a pre-built docker image and a helper script.

Both options provide the same result, so choose whichever you like better.

Option 1: module-build-service package

On Fedora rawhide, just install it by:

$ sudo dnf install module-build-service

I have also created a Module Build Service copr repo for Fedora 24 and Fedora 25:

$ sudo dnf copr enable asamalik/mbs
$ sudo dnf install module-build-service

To build your modulemd, run a command similar to the following:

$ mbs-manager build_module_locally file:////path/to/my-module?#master

An output will be a yum/dnf repository in the /tmp directory.

Option 2: docker image and a helper script

With this option you don’t need to install all the dependencies on your system, but it requires you to setenforce 0 before the build. 🙁

You only need to clone the asamalik/build-module repository on GitHub and use the helper script as follows:

$ build_module ./my-module ./results

An output will be a yum/dnf repository in the patch you have specified.

What’s next?

The next step would be installing your module on the Base Runtime and testing if it works. But as we are doing this pretty early, there is no Base Runtime at the moment I’m writing this. However, you can try your module in a container using a pre-build fake Base Runtime image.

To handcraft your modular container, please follow the Developing and Building Modules guide on our wiki which provides you all the necessary steps while showing you a way how modular containers might be built in the future infrastructure! 2017

Are you visiting in Brno? There is a talk about Modularity and a workshop where you can try building your own module as well. Both can be found in the 2017 Schedule.

  • Day 1 (Friday) at 12:30 – Fedora Modularity – How does it work?
  • Day 2 (Saturday) at 16:00 – Fedora Modularity – Build your own module!

See you there!