The Fedora Legacy QA (Quality Assurance) process

Goals

The Fedora Legacy Project QA process is aimed at producing update packages which meet, to the best of our ability, the following goals.

The QA Process

Package updates are created by the community and then posted for QA testing. Package submissions in the QA process are stored in the Bugzilla system. A package will only be published after a miniumum number of other developers have checked the package against the QA Checklist below, posted their results to the Bugzilla entry, any found problems are fixed and tested again, and the package and reviews satisfy the PUBLISH criteria below.

How to help

First, please introduce yourself!

Being connected only through an anonymous network, it is important for us to get known to you. If you haven't done so yet, please post a self-introduction to the fedora-legacy-list mailing list. The SelfIntroduction page gives an idea of what kind of information we want to know. You may of course provide additional information about yourself if you desire. You are required to include a GnuPG key to enable us to check your signatures on packages that should be published.

Submitting Packages

If you decide to create update packages, the following is the procedure to follow in order to submit them for inclusion in the Fedora Legacy Project.

  1. Identify the vulnerability and the solution for it. You may want to get some consensus on your approach with the Fedora Legacy members before you proceed if you are uncertain about anything.
  2. Create the patch(es) needed to fix the vulnerability and update the package .spec file for the patch(es).
  3. Create your updated SRPM package.
  4. Check that your package conforms to the QA Checklist below.
  5. Sign the package with your GnuPG key. Do not use rpmbuild --sign but instead use rpm --addsign to sign the package.
  6. (Optional, but recommended:) Run rpmlint on the SRPM to catch most common errors, and fix any reported errors. The rpmlint package is available from the Fedora.us stable branch.
  7. Upload the package to a public server, preferably a web accessible server.
  8. Submit a new Bugzilla entry (select Fedora Meta for the package, which includes at least the following).

    • A direct URL to the package.
    • Short description of the package changes.
    • List upstream sources and/or advisories.
    • Add the keywords LEGACY and QA, as well as the OS version tag (e.g. rh72, rh73, rh80, etc).

    The Bugzilla entry should be GnuPG cleartext signed.

Testing packages

  1. Download the SRPM package from the location specified in Bugzilla.
  2. Inspect the code and .spec file looking for any problems. See the QA Checklist below for things to check.
  3. Rebuild the SRPM and note any rebuild problems.
  4. Install the package, and note any installation problems.
  5. Use the package (as appropriate for the package), and note any problems found.

After testing a package

Report your test results in the Bugzilla system. QA comments in Bugzilla should be GnuPG clearsigned. In cases where a QA message would lead to package approval or publication, the message MUST be GnuPG clearsigned, and contain sha1sums of the reviewed source RPM (and preferably all upstream sources also). The source RPM sha1sum is mandatory in order to be sure that the package to be built is the same as the reviewed one. GnuPG clearsigned messages that give good advice can all be traced back to your GnuPG key and accumulate over time. This process allows new developers to gain trust through technical correctness and hard work over a period of time, eventually being able to prove to the community that the developer can be trusted.

If QA comments are submitted listing things within the package that should be fixed, the original packager should update their SRPM or respond why they feel suggestions are not valid. Be sure to increment the release number before building the new SRPMS. Upload the new SRPMS along with updated clearsigned sha1sums to your web server.

QA Checklist

The following is a QA checklist (adapted from the fedora.us QAChecklist. This is simply a checklist that should be done for each package during the QA process, but it is not the entire QA process. Each package may have steps unique to that package which are not listed here.

  1. Are the sources from upstream pristine and free from trojans?

    • Whenever possible verify GnuPG signatures, or even ask the upstream developers themselves to confirm the integrity of sources.
    • Another good source is to look within SRPMS of other distributions.
  2. Are the pre- and post- install and uninstall scripts correct?

    • If it installs files named .so. into %{_libdir}, is there a %post -p /sbin/ldconfig and a %postun -p /sbin/ldconfig?
    • If it has info files, is there a %post script which installs them, and a %postun script which removes them (and only on removal, not on an upgrade)?
    • Do the scripts do anything which might be a security risk?

      • Changing or adding root GnuPG keys.
      • Changing existing firewall or tcp_wrapper settings.
      • Creating accounts without proper security or with default passwords.
      • Installing daemons which run as root, or as another standard user such as nobody, instead of as a dedicated user created for that deamon?

      If so, at a minimum the installer must be warned about the issue during the install process. Preferably such issues should be removed if at all possible.

    • If a daemon's configuration is changed, is the deamon restarted or forced to reload the configuration? If not, is a suitable message displayed to alert the installer about the changes?
  3. Check for missing BuildRequires. For more information, see this posting.
  4. Is the Group tag an official one? See /usr/share/doc/rpm-*/GROUPS.
  5. Are all paths within the .spec replaced with macros?
  6. If the package has a DesktopEntry, are the VFolder categories ok?

    • See FedoraDesktopEntryGuidelines
    • When the .spec uses desktop-file-install for to install menu entries, do not forget "BuildRequires: desktop-file-utils"
  7. Does the package have any %{buildroot} macros? If so, they should be replaced with $RPM_BUILD_ROOT.
  8. Does the package have any %{optflags} macros? If so, they should be replaced with $RPM_OPT_FLAGS.
  9. Does the package install any unowned directories? Read the thread at http://www.fedora.us/pipermail/fedora-devel/2003-June/001458.html for details.
  10. Parallel makes using -jX generally speed up builds, especially on SMP machines. This is correctly accomplished by using make %{?_smp_mflags} in .spec files. However, not all packages are parallel build clean. To catch some of these errors, packagers and QA testers may want to add %_smp_mflags -j3 to /.rpmmacros, even if they use UP machines.
  11. Does the package naming scheme follow the Fedora Legacy RpmVersioning conventions?

Publish Criteria

A package may be published once it has at least two clearsigned QA approvals from any Fedora Legacy member. However, a trusted Fedora Legacy member may veto the approval if they have a good reason to do so.

If a similar update is available for multiple similar OS versions (e.g. for both RHL 7.2 and RHL 7.3), and has passed the publish criteria for one OS version but not for the other, the second OS version may be released after a timeout period if no one has tested it. This prevents updates from being published for one OS version due to lack of testers when it has passed on other similar OS versions and is believed to therefore be safe. Such releases are at the descretion of the Fedora Legacy package publishers.