Duncan Mac-Vicar P.


yum and ZYpp speed / memory usage

with 18 comments

Michael Zucchi complains about yum memory usage, and points python as guilty.

Yum isn’t so yummy after-all. Re-enabled python so i could run yum. Wow 120mb of vm to install a couple of packages. Not bad considering the box only has 128mb. This is crap.

Hmm, should I try xubuntu – or will it be just as crappy and bloated and blighted by python poo?

Since our efforts to make the ZYpp really fast, by incorporating and integrating Michael Schroeder’s sat solver together with Michael Matz’s great work on the solv files and data storage, I never took the time to make a “quick comparison” on speed or memory usage. So lets have a quick look.

These are my repositories:

Software configuration management (openSUSE_10.3)
10.3 - Main Repository (NON-OSS) 
10.3 - Packman
openSUSE-10.3-Updates
Virtualization:VirtualBox
home:dgollub
KDE:KDE3
Mozilla based projects (openSUSE_10.3)
ZYPP SVN Builds (openSUSE_10.3) 
ZYPP SVN Builds (openSUSE_10.3)
home:prusnak
10.3 - VideoLan
openSUSE.org tools (openSUSE_10.3)
SUSE Feature Tracking Tool (openSUSE_10.3)
psmt's Home Project (openSUSE_10.3)
openSUSE:10.3
Duncan Mac-Vicar SUSE rpms (openSUSE_10.3)
Latest YaST svn snapshots (openSUSE_10.3)
building/openSUSE_10.3         

All these repos together are about 41.000 packages.

What I did was to symlink ZYpp repositories to the yum repo path so they use the same repos.

# rm -rf /etc/yum.repos.d/
# ln -s /etc/zypp/repos.d /etc/yum.repos.d
*NOTE:* I tested with yum 3.2.4. I know 3.2.14 is available, but that is what I had installed when doing the test. After doing this tests I upgraded to 3.2.14 but it did not accept my .repo file because the character “:” in repo names. However the changelog of yum since 3.2.4 shows: If using latest yum would invalidate this numbers (not as in 1 second, but as in an order of magnitude), let me know and I will repeat them when I make them work with my repo files.

Update 14.05.2008 : I did add yum 3.2.14. However it performed even worse, except for memory usage.

Update 15.05.2008 : added smart 0.52 numbers

libzypp is the one you see in factory since some days: 4.21.1.

yum and ZYpp behave differently, as yum downloads and parses filelists.xml and other.xml which we ignore. There fore I skipped the download metadata part and just timed the cache building process.

# yum clean dbcache
...
19 sqlite files removed

# time yum makecache
...
Metadata Cache Created

real    9m41.036s
user    2m34.766s
sys     0m11.545s

Almost 10 minutes. As this time includes parsing the two big files we ignore. I did it again, pressing Ctrl-C After yum finished with the primary data, which is what ZYpp uses:

# time yum makecache
...
Exiting on user cancel

real    4m6.730s
user    0m34.058s
sys     0m3.080s

Now, zypper’s turn:

# time zypper ref -B
...
All repositories have been refreshed.

real    0m18.472s
user    0m16.029s
sys     0m2.024s

So yum takes 13 times the ZYpp needs technically (primary 1:1 comparison), but 30 times the time the end user sees.

Now, installing a package. Times were measured till the “continue? yes/no” prompt, or till the first interactive question.

# time yum install fate
...
Is this ok [y/N]: n
Exiting on user Command
Complete!

real    0m19.143s
user    0m14.057s
sys     0m1.920s

zypper’s turn:

# time zypper in fate
...
Continue? [YES/no]: n

real    0m9.796s
user    0m8.509s
sys     0m0.624s

This time, ZYpp is only twice as fast as yum. Only ;-)

What happens when you want to upgrade your packages?

# time yum upgrade
...
real    0m45.152s
user    0m36.894s
sys     0m7.476s

(Note: yum did not even found a solution here).

# time zypper update
...
Continue? [YES/no]: n

real    0m8.988s
user    0m7.820s
sys     0m0.596s

yum needs 4 times the time ZYpp needs to calculate the upgrade.

Update 14.05.2008 : I was comparing update to upgrade, I fixed those numbers in the chart. However, I don’t have the update value for the old yum.

Summary:

Now, how much memory does each one need? For this, I just tested the install command with one package using valgrind massif, a heap profiler.

yum memory usage:

yum memory usage

ZYpp memory usage:

zypp memory usage

Update 14.05.2008 : memory usage chart for yum 3.2.14

yum 3.2.14 memory usage

Update 15.05.2008 : memory usage chart for smart 0.52

smart 0.52 memory usage

Here you can appreciate ZYpp goes a little bit over 20M, while yum goes over 180M, so yum uses about 9 times more memory. Update 14.05.2008 : yum 3.2.14 uses around 160 in the worst point of time.

I would be interested in tracking cpu usage too, but that will come later. What do you think about it?

Written by duncan

May 16th, 2008 at 7:30 pm

Posted in Software

Tagged with , , ,

18 Responses to 'yum and ZYpp speed / memory usage'

Subscribe to comments with RSS or TrackBack to 'yum and ZYpp speed / memory usage'.

  1. Very interesting numbers, thanks for the work!

    A collegue of mine just asked if there are some numbers comparing zypp with smart… Have you done some testing there, too?

    smart seems to be very popular among opensuse users…

    Robin

    14 May 08 at 7:24 am

  2. It would be also interesting to know how the new libzypp manages the “Case Studies” from http://svn.labix.org/smart/trunk/README

    I don’t know, but could be that yum fixed “Case 2″ and libzypp “fails”? So the extra time and memory over libzypp taken by yum to make the dep resolution would be justified.

    Me

    14 May 08 at 8:15 am

  3. [...] que la distro tiene muy, muy buena pinta. La gestión de paquetes va fantástica (sobre todo usando zypper frente a yum, por ejemplo) y la verdad es que aunque KDE 4 va un poquito tirado en algunas operaciones con [...]

  4. I have added the smart numbers.

    As for the solver, those cases are no real testcases, as they don’t describe what was in the system (like our libzypp-testsuite)

    duncan

    16 May 08 at 4:02 pm

  5. Could you do another test for me please: how does this new Zypp compare to Synaptic.

    A friend of mine has an old laptop and will convert to openSUSE if it’s faster than Ubuntu.

    Thanks mate!

    Radon

    16 May 08 at 4:11 pm

  6. [...] Duncan Mac-Vicar P. homepage « yum and ZYpp speed / memory usage [...]

  7. We are talking about very simple cases. I don’t think what was in the system is relevant in a test where you create five RPMs with the above spec files, create a repository with them and install ‘A’. There are any real packages that require/provide thing like “C” or “libA”???

    License: License Group: Group Summary: Summary Name: A Version: 1.0 Release: 1 Requires: BCD libB libC libD

    %description Description

    %files

    License: License Group: Group Summary: Summary Name: BCD Version: 1.0 Release: 1 Provides: libB libC libD

    %description Description

    %files

    License: License Group: Group Summary: Summary Name: B Version: 1.0 Release: 1 Provides: libB

    %description Description

    %files

    License: License Group: Group Summary: Summary Name: C Version: 1.0 Release: 1 Provides: libC

    %description Description

    %files

    License: License Group: Group Summary: Summary Name: D Version: 1.0 Release: 1 Provides: libD

    %description Description

    %files

    Me

    17 May 08 at 1:50 am

  8. I had problems with the last libzypp on 10.3, so I’m not going to test it at least until 11.0 final. But just tested yum 3.2.14 with these five RPMs… and yum still tries to install all five files when only BCD is needed to install A!!!

    Me

    17 May 08 at 2:08 am

  9. duncan

    17 May 08 at 2:18 am

  10. ( and you will see that you could test those cases yourself :-) we provide the complete simulation environment )

    duncan

    17 May 08 at 2:18 am

  11. Great work Duncan!

    F2

    17 May 08 at 6:24 pm

  12. cool – you have Gravatar support I see :-)

    F2

    17 May 08 at 6:24 pm

  13. I was very surprised that my home project (home:prusnak) appeared between the test repositories. What was so interesting about it that it made to the list? :)

    stick

    18 May 08 at 2:53 pm

  14. If the repo is on your local disk make sure you echo 1 > /proc/sys/vm/drop_caches as root to drop the kernel’s file cache before the start of a run (otherwise later runs may have the advantage of not doing disk I/O)

    Anon

    18 May 08 at 11:46 pm

  15. [...] improvements like the fast installation, how YaST became sexy, how YaST/ZYpp/zypper became fast, how YaST/ZYpp/zypper performs better than others and even that our solver is also really [...]

  16. [...] Zypper is wicked fast, and is now my favorite way to manage packages. All it lacks is tab-completion of package names, but running zypper se somepackage is so fast that I don’t mind too much. [...]

  17. [...] után a SOLV fájlokban történt változások, valamint az új solver szignifikáns teljesítménynövekedést eredményezettés kevesebb memóriát igényel összehasonlítva a Smarttal és a [...]

  18. I was asking myself the other day, “Why doesn’t openSUSE just use yum for package management.”

    I have the answers now.

    zypper still needs some more work, however. I can’t seem to find a way to show the installed packages that require PACKAGE (so that I can remove PACKAGE without upsetting the system), or show what packages PACKAGE requires (to measure how much I will need to download and install before installing PACKAGE).

    All-in-all I like the effort openSUSE puts in to create a decent distro.

    Felipe

    felipe alvarez

    27 Jan 09 at 12:04 pm

Leave a Reply