The result that I was able to achieve was to confirm that all of the instructions on the Fedora Wiki now work correctly and if you follow them through from start to finish, you should get a result of a working Koji system. I believe I was also able to clarify a lot of information that was previously unclear. and give users a good understanding of how everything works together.

You can view the work I have been doing over the semester by visiting the Fedora Koji Wiki and also by viewing the blog postings which I have listed below:

I hope that my work on the wiki will help out some people in the future as I have really enjoyed learning about the Koji system throughout this project. I would like to give a big thank you to Chris Tyler for a great semester which included an awesome trip to Arizona to participate in Fudcon 2011; as well as a semester full of opportunities which involved working with, and learning about a new technology in the ARM project (new in the sense of using them as a low power server). I have enjoyed every moment of working with the ARM project and I hope to continue contributing in anyway I can in the future.

For the most part the information on the wiki is accurate and the instructions will work if you follow them through step by step. The issue that I found both through my own experiences and also in speaking with my colleague Anthony Boccia who has been working on moving the CDOT Koji system over to ARM, is that the there is not enough information which explains how the different components work together and what each step of the instructions is actually doing. The problem with this is that when one of steps does not work as planned, it is difficult to trouble shoot without fully understanding what that step does and how the component in question fits into the Koji system as a whole.

Over the past week or two I have been concentrating on going through the notes I had taken while walking through the process as well as some notes passed onto me by Anthony regarding issues he encountered, and translating them into meaningful additions to the existing documentation. My goal here was to provide some clarity as to how each component of the Koji Build System interacts with each other as well as to explain what each set in the setup process is actually accomplishing and why it is important. At this point I would like to share that information through my previous posts which I have linked below:

I would also like to invite anyone who is interested to review the edited documentation on the Fedora Wiki, specifically the How To section which walks through setting up your own system:

Koji/ServerHowTo – Fedora Wiki

Over the next week or two I will be taking a look at the other sections of the Koji documentation to see what information could be clarified as well as making an additional changes which might come to my attention with the Koji setup process. Please feel free to contact me with any information you might like to share or suggestions you might like to make. Your feedback would be very valuable to me and greatly appreciated. You may also view the notes I have taken along the way at http://matrix.senecac.on.ca/~tewestac/koji/Koji%20Documentation%20Notes.pdf

[root@ireland ~]$ yum install koji-builder

Once the Koji Builder Daemon is installed we need to configure it to point to our server. We do that by editing the /etc/kojid/kojid.conf file and editing the following lines to reflect your own server address and username for the build machine:

; The URL for the xmlrpc server
server=http://ireland.proximity.on.ca/kojihub

; The username has to be the same as what you used with add-host
;in this example follow as below
user = kojibuilder1

Within the same configuration file we also need to turn on SSL Authentication and point it to the location of our certificates on the builder machine

;configuration for SSL authentication  

;client certificate
cert = /etc/kojid/ireland.pem

;certificate of the CA that issued the client certificate
ca = /etc/kojid/koji_ca_cert.crt

;certificate of the CA that issued the HTTP server certificate
serverca = /etc/kojid/koji_ca_cert.crt

It is important to note that for our test the builder is the same machine as the other Koji components so it already had the certificates which the configuration file points to. If your builder is on a separate machine which is normally the case, you would need to SCP the certificates from /etc/kojid/ on the server to /etc/koji/ on the builder.

Now that we have our Koji Builder configured we need to add it to the Koji Database so that it can be used to build packages. We do this by issuing the following command replacing kojibuilder with the username of your builder and arm with the architecture which the builder will use:

[kojiadmin@ireland ~]$ koji add-host kojibuilder1 arm

Because this is the only build machine we will be configuring for this test, we also need to add it to the createrepo channel so that it can handle the task of creating need repositories. Typically if you were installing a bunch of Koji Builders you would add the first few to this channel so that they could also handle creating new repositories.

[kojiadmin@ireland ~]$ koji add-host-to-channel kojibuilder1 createrepo

The last thing that we need to do is to start the Kojira service:

[root@ireland ~]$ /sbin/service kojid start

If you now go to the web interface and click on the hosts tab, you should see your builder listed and ready. At this point your Koji Build System should be fully operational and ready for packages!!!

Next up… I will be working for the remainder of this week on editing the Fedora Wiki to clarify so information which I have gathered through out this test. Stay tuned for updates on the editing process and feel free provide any feedback you might have on the changes I will be making at tewestac@learn.senecac.on.ca

The first thing we need to do is to install the Koji Web Interface and mod_ssl for SSL authentication:

[root@ireland ~]$ yum install koji-web mod_ssl

With the web interface and SSL installed we need to configure it by editing the file /etc/httpd/conf.d/kojiweb.conf and editing a few lines to point Koji Web to the correct URLs for our system and to tell it where to find the SSL certificates for authentication. First we will point Koji Web to the correct URLs with the following lines:

PythonOption KojiHubURL http://ireland.proximity.on.ca/kojihub

PythonOption KojiWebURL http://ireland.proximity.on.ca/koji

PythonOption KojiPackagesURL http://ireland.proximity.on.ca/mnt/koji/packages

Next we tell Koji Web where it can find the SSL certificates it needs by editing the following lines:

PythonOption WebCert /etc/pki/koji/kojiweb.pem
PythonOption ClientCA /etc/pki/koji/koji_ca_cert.crt
PythonOption KojiHubCA /etc/pki/koji/koji_ca_cert.crt

Finally we need to add two more lines to the configuration file which set a Login Timeout as well as a password:

PythonOption LoginTimeout 72
PythonOption Secret CHANGE_ME

Before we are finished with the /etc/httpd/conf.d/kojiweb.conf file, we need to uncomment the lines for SSL authentication:

<Location /koji/login>
SSLOptions +StdEnvVars
</Location>

Now that Koji Web is configured we just need to ensure that SSL is setup on the server correctly. We can do this by opening the  /etc/httpd/conf.d/ssl.conf file and adding the following lines:

SSLVerifyClient require
SSLVerifyDepth  10

If everything went well you should now be able to point your web browser to the URL of Koji Web and be able to pull up the interface. For the most part you will only be able to view information such as the users and hosts configured on the system. You should however be able to login as one of the users with the web certificate.

Next up….. Koji Builders

The first thing we need to do is to actually install the hub software on our server along with apache and a few other components related to SSL authentication:

root@chile$ yum install koji-hub httpd mod_ssl mod_python

Once we have Apache installed we need to configure it to work with the Koji Hub by editing the Apache configuration file:

[root@chile ~]# vi /etc/httpd/conf/httpd.conf

In order to prevent the server from becoming overloaded and crashing we will set the maximum number of requests to 100 before Apache spawns a new server:

<IfModule prefork.c>

StartServers 8

MinSpareServers 5

MaxSpareServers 20

ServerLimit 256

MaxClients 256

MaxRequestsPerChild 100

</IfModule>

<IfModule worker.c>

StartServers 2

MaxClients 150

MinSpareThreads 25

MaxSpareThreads 75

ThreadsPerChild 25

MaxRequestsPerChild 100

</IfModule>

Next we need to provide the Koji Hub with the database information:

[root@chile ~]# vi /etc/koji-hub/hub.conf

We will now add some lines which tell the Koji Hub about the database. The important part here is that the lines are edited to represent the configuration on your own system. Pay special attention to the KojiDir setting and the KojiWebURL as these will be different on your own system

# Basic options ##

DBName = koji

DBUser = koji

DBHost = localhost

KojiDir = /mnt/koji

LoginCreatesUser = On

KojiWebURL = http://chile.proximity.on.ca/koji

We also need to add a few lines in the SSL Authentication section, editing the DN name to represent your own server:

DNUsernameComponent = CN

ProxyDNs = /C=CA/ST=Ontario/O=Seneca CDOT/OU=/CN=chile.proximity.on.ca/emailAddress=tewestac@learn.senecac.on.ca

We also need to edit the vi /etc/httpd/conf.d/kojihub.conf file and uncomment the following lines to allow SSL”

Location /kojihub>
SSLOptions +StdEnvVars
/Location>

To complete the configuration part of the Koji Hub we need to edit the SSL configuration file to add the appropriate certificates:

[root@chile ~]# /etc/httpd/conf.d/ssl.conf

We will now take the certificates we generated earlier and add them to the VirtualHost_default_:443 section of the ssl.conf:

SSLCertificateFile /etc/pki/koji/certs/kojihub.crt
SSLCertificateKeyFile /etc/pki/koji/certs/kojihub.key
SSLCertificateChainFile /etc/pki/koji/koji_ca_cert.crt
SSLCACertificateFile /etc/pki/koji/koji_ca_cert.crt
SSLVerifyClient require
SSLVerifyDepth 10

We now need to allow Apache to connect with the Database through SELinux:

[root@chile ~]# setsebool -P httpd_can_network_connect_db 1

We should now have the Apache and Koji Hub configuration setup but we still need to setup the Koji Filesystem Skeleton by issuing the following commands:

[root@chile ~]#cd /mnt
[root@chile ~]#mkdir koji
[root@chile ~]#cd koji
[root@chile ~]#mkdir {packages,repos,work,scratch}
[root@chile ~]#chown apache.apache *

We also need to edit the /etc/koji.conf file adding the URLs and certificates of the other Koji Components so that the Hub can access them:

[koji]

;url of XMLRPC server
server = http://hongkong.proximity.on.ca/kojihub
;url of web interface
weburl = http://hongkong.proximity.on.ca/koji
;url of package download site
pkgurl = http://hongkong.proximity.on.ca/packages
;path to the koji top directory
topdir = /mnt/koji
;configuration for SSL athentication
;client certificate
cert = ~/.koji/client.crt
;certificate of the CA that issued the client certificate
ca = ~/.koji/clientca.crt
;certificate of the CA that issued the HTTP server certificate
serverca = ~/.koji/serverca.crt

The last thing we need to do is add the user and builder accounts. We can add the user account with the following commands:

[root@chile ~]# su – kojiadmin
[kojiadmin@chile ~]$ koji add-user kojira
[kojiadmin@chile ~]$ koji grant-permission repo kojira

The Builder accounts are added as follows:

[kojiadmin@chile ~]$ koji add-host arm1 arm
[kojiadmin@chile ~]$ koji add-host arm2 arm
[kojiadmin@chile ~]$ koji add-host arm3 arm

[kojiadmin@chile ~]$ koji add-host arm4 arm

[kojiadmin@chile ~]$ koji add-host arm5 arm

With the addition of all the user accounts the Koji Hub should now be setup successfully. I have updated my running notes which be accessed here.

The next step is to setup KojiWeb which I hope to have instructions on shortly

root@chile$ yum install postgresql-server

With Postgresql installed we need to initialize the database:

With the PostgreSQL installed and the database initialized we know need to create the “koji” user account which it will use:

After setting up the “koji” user on the machine we need to create the same user within our database and populate the database schema:

Now that we have our database populated we need to authorize the koji-hub and koji-web servers within the database. In order to do this we need to edit the /var/lib/pgsql/data/pg_hba.conf file and add the following lines:

After authorizing koji-hub and koji-web we need to make the changes live:

The last thing we need to do is to create an initial Admin account and give it the appropriate privileges to add new user accounts later on:

If all has gone well then we will now have a fully functional Koji Database.

Next up….. Setting up the Koji Hub

Throughout the process we have run into a few issues where following the existing documentation on the Fedora and Seneca Wikis have resulted in errors instead of the expected result. Unfortunately due to this we have fallen a little short of that goal and do not quite have the test system fully functional.

Currently we have been able to successfully generate the SSL certificates for each of the koji components and have the Koji Database fully operational. We are also quite close to having the Koji Hub fully functional as well but have run into an issue with adding the kojira and builder accounts.

I have decided to release the rough notes anyways in PDF format and will update them as I move forward. The current rough notes can be found at the following link:

Koji Documentation Notes

I have also posted previous blog entries for my notes on generating the SSL certificates and setting up the Koji Database. These postings can be found at the following links:

Koji Setup Documentation – SSL Authentication

Koji Setup Documentation – PostgreSQL

By the next scheduled release I hope to have completed the test build system and begun editing the Fedora Wiki to establish a complete step by step guide in one location.

The first thing we need to do is to create a directory to hold the certificates as well as the ssl.conf file which will  be used to generate the certificates:

sudo mkdir /etc/pki/koji

I then created the ssl.conf file using the code from the Fedora Wiki example and made the edited the following settings to customize the file for the koji components or our test system:

Now that we have our SSL configuration file we need to create a Certificate Authority by issuing the following commands one line at a time:

Once these commands are run you are asked to answer a few questions about your location. For the most part the defaults which we entered in the ssl.conf can be used until you are asked for the common name. This should be set to the domain name of the server which is being used for the CA.

Once we have a Certificate Authority we need to generate certificates for each of the koji components. We can do this by creating a loop and inserting the script to generate the certificates:

Once the loop begins to run we are asked the same questions as we were with the CA. Again we can accept the default settings we have setup in the ssl.conf until we are asked for the common name. For each certificate the common name should be set to the domain name for the corresponding server which will host the component.

p { margin-bottom: 0.08in; }

We now have SSL certificates for all of our Koji components which we can use to authenticate with the Koji hub.

Next up……Setting up the PostgreSQL Server

Currently I have been successful in setting up the certificates for all of the components in the Koji Test Build System and have made notes on areas which seemed unclear in the current documentation on the Fedora wiki. I used a combination of the instruction on the Fedora wiki along with a modified version of two scripts which Chris Tyler wrote for the Koji ARM Build System which creates a loop for generating multiple certificates. It did take a bit of playing with in order to fully understand the lines which I needed to edit in the SSL config file to achieve the results which I desired.

I am also close to having the Koji Database setup completely on my test system. Currently I am working on authentication for the database and have run into a snag when I try access the database as I always get the error “Ident authentication failed for user”. I will need to look into this further as there is most likely an issue with the pg_hba.conf file which I need to work out.

By my next post I hope to have this issue worked out and also have the Koji Hub completely installed and configured. This will only leave setting up the build machines on the test system. Once I have everything up and running correctly, I will release my notes on the issues that I ran into and will begin editing the wiki pages.

If you have any questions or have something which you might like to contribute then I can be reached as always at tewestac@learn.senecac.on.ca or on IRC as tewestacott in #koji or #seneca

The primary contacts that we have been in touch with for this project are Chris Tyler and Paul Whalen from the Seneca College CDot department as well as Dennis Gilmore through the IRC #Koji channel and the buildsys@lists.fedoraproject.org mailing list. We have had some preliminary talks with both Dennis Gilmore and Chris Tyler and have determined that there may be some information missing from the instructions which is common knowledge to the people who are quite familiar with setting up Koji systems, and quite foreign to those of us who sit at the beginner and novice levels. Our hope is that by walking through this process we will be able to pinpoint any areas where the instructions may need improvement for less experienced users.

We have also been in contact with Anthony Boccia who will be working on the “Koji Hub On ARM” project. We are looking at collaborating on these two projects in order to share our findings with Anthony as well as gain information on any problems or concerns that he encounters in the process of preparing a Koji hub on the ARM servers.

For more information on this project or if you have any knowledge that you would like to contribute please visit the project webpage.