Modular F27 Server Edition – initial design

Recently, I have started a discussion on the Server mailing list about building the Fedora 27 Server Edition using Modularity. Langdon White is already working on a change request. If that happens, there will be a lot of work in front of us. So let’s start with writing blog posts!

To build the Fedora 27 Server Edition using Modularity, Adam thinks we need to focus on two things:

First is the initial design of the basic set of modules for the F27 Server Edition – including the Host and Platform modules, as well as other ‘application/content’ modules. To make this easier, I’m proposing a tool temporarily called The Graph Thing.

Second is a great packager UX for the build pipeline. This will lead to more content built by the community. It will include The Graph Thing and BPO, and I will be talking about it in a different post.

This post covers the first part – the initial design.

Modularizing the F27 Server Edition – introduction

To build Fedora 27 Server Edition using Modularity, we need to split the monolithic distro into smaller modules. Modules can come in multiple streams and on independent lifecycles. However, in Fedora 27, all modules will be on the same lifecycle of 13 months as the rest of Fedora 27.

Splitting Fedora 27 into modules

The Platform team defines the Host and Platform modules by deciding which packages are needed in which modules and including their dependencies into these modules. They already started defining the Host and Platform modules in the Host and Platform GitHub repository.

Other modules will be defined based on the Fedora 27 Server Edition use-cases. We need to create modules for all server roles and other components that will be part of the server. Defining these modules will also influence the Platform module as it will include some of the packages shared between other modules.

Splitting the distribution into individual modules is not easy. We need to work with hundreds of packages with complex dependencies and carefully decide what package goes into which module. To do the initial distribution design, I am proposing a tool that will help us. I temporarily call it The Graph Thing and I hope the name will change soon.

After the initial split, we expect other people adding modules to the distribution as well. Making the packager UX great is crucial for us, if we want to get a lot of content from the community. Packagers will be also able to use The Graph Thing for the initial design of their module, and Build Pipeline Overview (BPO) to monitor their builds.

The Graph Thing – a tool for the initial design

The Graph Thing is a tool that will help people with the initial design of individual modules when modularizing a distribution. It will also help packagers with adding other modules later on.

It will work with resources from the Fedora infrastructure, inputs from the user, and will produce a dependency graph as an output. An example with pictures is better than a thousand words:

Example – defining a nodejs module

The Dependency Thing - first run

In this example, there are three things available as resources:

  • Host module definition
  • Platform module definition
  • Fedora 27 repository – including all the Fedora 27 packages

A packager wants to add a new module: nodejs. The packager have specified in the input that they want to visualize the host and platform modules, as well as the new nodejs module which doesn’t exist in the infrastructure. The packager added a nodejs-6.1 package into that module and wants to see if that’s enough for the module, or if they need to add something else.

The output shows that the nodejs module will require two other packages directly, library-foo-3.6 and nodejs-doc-6.1, and that the nodejs-doc-6.1 also needs a crazy-thing-1.2.

Seeing this, the packager made the following decision:

  1. The library-foo-3.6 is a package specific for nodejs, so it should be part of the nodejs module.
  2. The nodejs module should not include documentation, as it will be shipped separately. They will modify the nodejs-6.1 package not to include the nodejs-doc-6.1.

With the decision done, the packager wants to see how it’s going to look like if they apply the changes. So they modify the input of The Graph Thing and run it again:

Great! It looks like that the module now includes everything that is needed. The packager can make the change in the nodejs-6.1 package and submit this module for a build in the Fedora infrastructure.

Value of The Graph Thing tool

As we saw in the example above, the tool can help with designing new modules by looking at the Fedora 27 packages and giving an instant view of how a certain change could look like, without rebuilding stuff.

It will also help us identify which packages need to be shared between individual modules, so we can decide if we include them in the Platform, or it we make a shared module. An obvious example of a shared module can be a language runtime, like Python or Perl. Other, not that obvious, can be identified with this tool.

Apart from the initial design of the Fedora 27 Server Edition, this tool could be also a part of the packager UX which I will be talking about soon.

Early implementation

I have already created an early partial implementation of this tool. You can find it in my asamalik/modularity-dep-viz GitHub repo.

Next steps

The tool needs to get rewritten to use libsolv instead of multiple repoquery queries – so it produces the correct output, and takes seconds rather than minutes to produce an output. It could also be merged with another similar project called depchase which has been used by the Platform team to define the Base Runtime and Bootstrap modules for the F26 Boltron release.

DORS/CLUC 2017 event report

DORS/CLUC is a regional open source software conference in Zagreb, Croatia. It happens every year at the Faculty of Electrical Engineering and Computing, targeting Linux users and professionals. I have attended the conference for the first time about two weeks ago, and I would like to share my notes from the conference with you.

Fedora got two talks this year:

  1. Fedora and VR glass by Gergely Rákosi
  2. Fedora Modularity by me, Adam Samalik

There is also a Fedora wiki page for the event.

How Adam did?

Let me give you some metrics of me being at the conference… I was using Twitter as my communication channel and to get some data about my presence.

The official DORS/CLUC Twitter account did about 60 tweets the first day, 10 were retweets, and I got 4 of them.

My most popular tweet by views got about 5000 views during the first 5 hours – before my talk, about 7600 for the whole duration of the conference, and 10133 at the time of writing this post. Thanks @fedora and others for retweeting!

The most retweeted one was about Gergley’s VR talk and got 12 retweets, 15 likes, and about 2500 views.

My talk was attended by roughly 50 – 80 people (I’m still not able to count people quickly enough) which was about the average. The conference had only one track, so there was not much choice anyway. I got a few questions right after the talks and some interesting discussions afterwards.

At the end of my talk, they tweeted about the Modularity YouTube channel and invited people to contribute.

I have published my slides at SlideShare: Fedora Modularity at DORS/CLUC 2017

Apart from the the Modularity talk, I also talked to people about Fedora and did four demos of the system on my laptop. One included installing and demonstrating KDE to a Windows user who really loved the customization options and installed Fedora KDE spin on his laptop later that evening.


Modularity update – sprint 31

Hello people of the world! Let me give you another update of the Fedora Modularity project.

What we did

We are moving the modules to the new Fedora Modularity: modules space on GitHub. All work regarding modules will be done primarily here and synced into fedora dist-git for builds. Each module has its own repository with an issue tracker.

The Modularity documentation has been updated and slightly restructured.

There is also a new demo about the dnf prototype supporting modules on our YouTube channel.

Try the DNF prototype

You can try everything you have seen in the video yourself! We have a container image with everything you need to get started. See the DNF Modularity Prototype repository for the source, or get the image directly from Docker hub:

$ docker pull jamesantill/flat-modules-dnf

Then start a container using this image with an interactive shell:

$ docker run --rm -it jamesantill/flat-modules-dnf /bin/bash

Now, when you are in the container, you can try all the commands. And please give us feedback either in the comment section below, in the issue tracker, or get in touch on the #fedora-modularity freenode channel.

What’s next?

  • Complete all modules, images, and tests.
  • Prepare an example and a demo of multiple streams of NodeJS.
  • Review and prioritize open issues.