Recently, I have started a discussion on the Server mailing list about building the Fedora 27 Server Edition using Modularity. Langdon White is already working on a change request. If that happens, there will be a lot of work in front of us. So let’s start with writing blog posts!
To build the Fedora 27 Server Edition using Modularity, Adam thinks we need to focus on two things:
First is the initial design of the basic set of modules for the F27 Server Edition – including the Host and Platform modules, as well as other ‘application/content’ modules. To make this easier, I’m proposing a tool temporarily called The Graph Thing.
Second is a great packager UX for the build pipeline. This will lead to more content built by the community. It will include The Graph Thing and BPO, and I will be talking about it in a different post.
This post covers the first part – the initial design.
Modularizing the F27 Server Edition – introduction
To build Fedora 27 Server Edition using Modularity, we need to split the monolithic distro into smaller modules. Modules can come in multiple streams and on independent lifecycles. However, in Fedora 27, all modules will be on the same lifecycle of 13 months as the rest of Fedora 27.
The Platform team defines the Host and Platform modules by deciding which packages are needed in which modules and including their dependencies into these modules. They already started defining the Host and Platform modules in the Host and Platform GitHub repository.
Other modules will be defined based on the Fedora 27 Server Edition use-cases. We need to create modules for all server roles and other components that will be part of the server. Defining these modules will also influence the Platform module as it will include some of the packages shared between other modules.
Splitting the distribution into individual modules is not easy. We need to work with hundreds of packages with complex dependencies and carefully decide what package goes into which module. To do the initial distribution design, I am proposing a tool that will help us. I temporarily call it The Graph Thing and I hope the name will change soon.
After the initial split, we expect other people adding modules to the distribution as well. Making the packager UX great is crucial for us, if we want to get a lot of content from the community. Packagers will be also able to use The Graph Thing for the initial design of their module, and Build Pipeline Overview (BPO) to monitor their builds.
The Graph Thing – a tool for the initial design
The Graph Thing is a tool that will help people with the initial design of individual modules when modularizing a distribution. It will also help packagers with adding other modules later on.
It will work with resources from the Fedora infrastructure, inputs from the user, and will produce a dependency graph as an output. An example with pictures is better than a thousand words:
Example – defining a nodejs module
In this example, there are three things available as resources:
- Host module definition
- Platform module definition
- Fedora 27 repository – including all the Fedora 27 packages
A packager wants to add a new module: nodejs. The packager have specified in the input that they want to visualize the host and platform modules, as well as the new nodejs module which doesn’t exist in the infrastructure. The packager added a nodejs-6.1 package into that module and wants to see if that’s enough for the module, or if they need to add something else.
The output shows that the nodejs module will require two other packages directly, library-foo-3.6 and nodejs-doc-6.1, and that the nodejs-doc-6.1 also needs a crazy-thing-1.2.
Seeing this, the packager made the following decision:
- The library-foo-3.6 is a package specific for nodejs, so it should be part of the nodejs module.
- The nodejs module should not include documentation, as it will be shipped separately. They will modify the nodejs-6.1 package not to include the nodejs-doc-6.1.
With the decision done, the packager wants to see how it’s going to look like if they apply the changes. So they modify the input of The Graph Thing and run it again:
Great! It looks like that the module now includes everything that is needed. The packager can make the change in the nodejs-6.1 package and submit this module for a build in the Fedora infrastructure.
Value of The Graph Thing tool
As we saw in the example above, the tool can help with designing new modules by looking at the Fedora 27 packages and giving an instant view of how a certain change could look like, without rebuilding stuff.
It will also help us identify which packages need to be shared between individual modules, so we can decide if we include them in the Platform, or it we make a shared module. An obvious example of a shared module can be a language runtime, like Python or Perl. Other, not that obvious, can be identified with this tool.
Apart from the initial design of the Fedora 27 Server Edition, this tool could be also a part of the packager UX which I will be talking about soon.
I have already created an early partial implementation of this tool. You can find it in my asamalik/modularity-dep-viz GitHub repo.
The tool needs to get rewritten to use libsolv instead of multiple repoquery queries – so it produces the correct output, and takes seconds rather than minutes to produce an output. It could also be merged with another similar project called depchase which has been used by the Platform team to define the Base Runtime and Bootstrap modules for the F26 Boltron release.