Blargon7 – An abundant source of alternative fuel

A Summer to Mentor

This summer I am mentoring Justin Potts – a university intern working on improving Mozilla’s add-ons related test automation, and Ana Ribeiro – an Outreachy participant working on enhancing the pytest-html plugin.

It’s not my first time working with Justin, who has been a regular team contributor for a few years now, and last summer helped me to get the pytest-selenium plugin released. It certainly helped to have previous experience working with Justin when deciding to take on the official role as mentor for his first internship. Unfortunately, his project is rather difficult to define, as he’s been working on a number of things, though mostly they are related to Firefox add-ons and running automated tests. There’s no shortage of challenging tasks for Justin to work on, and he’s taking them on with the enthusiasm that I expected he would. You can read more about Justin’s internship on his blog.

Ana’s project grew out of a security exploit discovered in Jenkins, which led to the introduction of the Content-Security-Policy header for static files being served. This meant that the fancy HTML reports generated by pytest-html were broken due to the use of JavaScript, inline CSS, and inline images. Along with a few other enhancements, providing a CSP friendly from the plugin became a perfect candidate project for Outreachy. As part of her application, Ana contributed a patch for pytest-variables, and I was impressed with her level of communication over the patch. To get Ana familiar with the plugin, her initial contributions were not related to the CSP issue, but she’s now making good progress on this. You can read more about Ana’s Outreachy project on her dedicated blog.

So far I have mostly enjoyed the experience of being a mentor – it especially feels great to see the results that Justin and Ana are producing. Probably the most challenging aspect for me is being remote – Justin is based in Mountain View, California, and Ana is based in Brazil. It’s hard to feel a connection when you’re dependent on instant messages and video conferencing, though I suspect it’s probably harder for them than it is for me. Fortunately, I did get to work with them a little in London during the all hands, and then some more with Ana in Freiburg during the pytest sprint.

There are still a few weeks left for their projects, and I’m hoping they’ll both be able to conclude them to their satisfaction!

Sprinting with pytest in Freiburg

Last week was the pytest development sprint located in the beautiful town of Freiburg, Germany. I had been really looking forward to the sprint, and being immediately after the Mozilla all-hands in London I was still buzzing with excitement when I started my journey to Freiburg.

On the first morning I really wasn’t sure about how to get to our sprint venue via public transport, and it didn’t seem to be far to walk from my hotel. It was a lovely sunny morning, and I arrived just in time for the introductions. Having been a pytest user for over five years I was already familiar with Holger, Ronny, and a few others, but this was the first time meeting them. We then spent some time planning out our first day, and coming up with things to work on throughout the week. My first activity was pairing up to work on pytest issues.

For my first task I paired with Daniel to work on an issue he had recently encountered, which I had also needed to workaround in latest versions of pytest. It turned out to be quite a complex issue related to determination of the root directory, which is used for relative reference to test files as well as a number of other things. The fix seemed simple at first – we just needed to exclude arguments that are not paths from consideration for determining the root directory, however there were a number of edge cases that needed resolving. The patch to fix this has not yet landed, but I’m feeling confident that it will be merged soon. When it does, I think we’ll be able to close at least three related issues!

Next, I worked with Bruno on moving a bunch of my plugins to the pytest-dev GitHub organisation. This allows any of the pytest core team to merge fixes to my plugins and means I’m not a blocker for any important bug fixes. I’m still intending on supporting the plugins, but it feels good to have a larger team looking after them if needed. The plugins I moved are pytest-selenium, pytest-html, pytest-variables, and pytest-base-url. Later in the week we also moved pytest-repeat with the approval of the author, who is happy for someone else to maintain the project.

If you’ve never used pytest, then you might expect to be able to simply run pytest on the command line to run your tests. Unfortunately, this isn’t the case, and the reason is that the tool used to be part of a collection of other tools, all with a common prefix of py. so you’d run your tests using the py.test command. I’m pleased to say that I worked with Oliver during the sprint to introduce pytest as the recommended command line entry point for pytest. Don’t worry – the old entry point will still work when 3.0 is released, but we should be able to skip a bunch of confusion for new users!

On Thursday we took a break and took a cable-car up a nearby peak and hiked around for a few hours. I also finally got an opportunity to order a slice of Schwarzwälder Kirschtorte (Black Forest gateau), which is named after the area and was a favourite of mine growing up. The break was needed after my brain had been working overtime processing the various talks, demonstrations, and discussions. We still talked a lot about the project, but to be out in the beautiful scenery watching para-gliders gracefully circling made for a great day.

When we returned to our sprint venue on Friday I headed straight into a bug triage with Tom, which ended up mostly focusing on one particular issue. The issue relates to hiding what is at first glance redundant information in the case of a failure, but on closer inspection there are actually many examples where this extra line in the explanation can be very useful.

Unfortunately I had to leave on Saturday morning, which meant I missed out on the final day of the sprint. I have to say that I can’t wait to attend the next one as I had so much fun getting to know everyone, learning some handy phrases in a number of different languages, and commiserating/laughing together in the wake of Brexit! I’m already missing my fellow sprinters!

Python testing sprint 2016

In June, the pytest developer community are gathering in Freiburg, Germany for a development sprint. This is being funded via an Indiegogo campaign, which needs your help to reach the goal! I am excited to say that I will be attending, which means that after over 5 years of using pytest, I’ll finally get to meet some of the core contributors.

I first learned about pytest when I joined Mozilla in late 2010. Much of the browser based automation at that time was either using Selenium IDE or Python’s unittest. There was a need to simplify much of the Python code, and to standardise across the various suites. One important requirement was the generation of JUnit XML reports (considered essential for reporting results in Jenkins) without compromising the ability to run tests in parallel. Initially we looked into nose, but there was an issue with this exact requirement. Fortunately, pytest didn’t have a problem with this – JUnit XML was supported in core and was compatible with the pytest-xdist plugin for running tests in parallel.

Ever since the decision to use pytest was made, I have not seen a compelling reason to switch away. I’ve worked on various projects, some with overly complex suites based on unittest, and I’ve always been grateful when I’ve been able to return to pytest. The active development of pytest has meant we’ve never had to worry about the project becoming unsupported. I’ve also always found the core contributors to be extremely friendly and helpful on IRC (#pylib on irc.freenode.net) whenever I need help. I’ve also more recently been following the pytest-dev mailing list.

I’ve recently written about the various plugins that we’ve released, which have allowed us to considerably reduce the amount of duplication between our various automation suites. This is even more critical as the Web QA team shifts some of the responsibility and ownership of some of their suites to the developers. This means we can continue to enhance the plugins and benefit all of the users at once, and our users are not limited to teams at Mozilla. The pytest user base is large, and that means our plugins are discovered and used by many. I always love hearing from users, especially when they submit their own enhancements to our plugins!

There are a few features I particularly like in pytest. Highest on the list is probably fixtures, which can really simplify setup and teardown, whilst keeping the codebase very clean. I also like being able to mark tests and use this to influence the collection of tests. One I find myself using a lot is a ‘smoke’ or ‘sanity’ marker, which collects a subset of the tests for when you can’t afford to run the entire suite.

During the sprint in June, I’d like to spend some time improving our plugins. In particular I hope to learn better ways to write tests for plugins. I’m not sure how much I’ll be able to help with the core pytest development, but I do have my own wishlist for improvements. This includes the following:

treating XPassing tests as failures
encoding parameterised ID lists
showing non-pytest warnings in the console
allow tests to be logged multiple times with pytest-xdist
test item scheduling does not schedule N items to N workers
when multiple configuration files are found, merge the contents

Maybe I’ll even be able to work on one of these, or any of the open issues on pytest with guidance from the experts in the room.

Selenium tests with pytest

When you think of Mozilla you most likely first associate it with Firefox or our mission to build a better internet. You may not think we have many websites of our own, beyond perhaps the one where you can download our products. It’s only when you start listing them that you realise how many we actually have; addons repository, product support, app marketplace, build results, crash statistics, community directory, contributor tasks, technical documentation, and that’s just a few! Each of these have a suite of automated functional tests that simulate a user interacting with their browser. For most of these we’re using Python and the pytest harness. Our framework has evolved over time, and this year there have been a few exciting changes.

Over four years ago we developed and released a plugin for pytest that removed a lot of duplicate code from across our suites. This plugin did several things; it handled starting a Selenium browser, passing credentials for tests to use, and generating a HTML report. As it didn’t just do one job, it was rather difficult to name. In the end we picked pytest-mozwebqa because it was only specific in addressing the needs of the Web QA team at Mozilla. It really took us to a new level of consistency and quality across all our our web automation projects.

This year, when I officially joined the Web QA team, I started working on breaking the plugin up into smaller plugins, each with a single purpose. The first to be released was the HTML report generation (pytest-html), which generates a single file report as an alternative to the existing JUnit report or console output. The plugin was written such that the report can be enhanced by other plugins, which ultimately allows us to include screenshots and other useful things in the report.

Next up was the variables injection (pytest-variables). This was needed primarily because we have tests that require an existing user account in the application under test. We couldn’t simply hard-code these credentials into our tests, because our tests are open source, and if we exposed these credentials someone may be able to use them and adversely affect our test results. With this plugin we are able to store our credentials in a private JSON file that can be simply referenced from the command line.

The final plugin was for browser provisioning (pytest-selenium). This started as a fork of the original plugin because much of the code already existed. There were a number of improvements, such as providing direct access to the Selenium object in tests, and avoiding setting a default implicit wait. In addition to supporting Sauce Labs, we also added support for BrowserStack and TestingBot.

Now that pytest-selenium has been released, we have started to migrate our own projects away from pytest-mozwebqa. The migration is relatively painless, but does involve changes to tests. If you’re a user of pytest-mozwebqa you can check out a few examples of the migration. There will no longer be any releases of pytest-mozwebqa and I will soon be marking this project as deprecated.

The most rewarding consequence of breaking up the plugins is that we’ve already seen individual contributors adopting and submitting patches. If you’re using any of these plugins let us know – I always love hearing how and where our tools are used!

Firefox Cufflinks

The custom made Firefox cufflinks have arrived! I’ll be working out shipping costs, and posting them out this week.

Custom Firefox Cufflinks

If you’re interested in a pair of custom Firefox logo cufflinks for $25 plus postage then please get in touch. I’ve been in contact with CuffLinks.com, and if I can place an initial order of at least 25 pairs then the mold and tooling fee will be waived. Check out their website for examples of their work. The Firefox cufflinks would be contoured to the outline of the logo, and the logo itself would be screen printed in full colour. If these go well, I’d also love a pair of dino head cufflinks to complement them!

Unless I can organise with CuffLinks.com to ship to multiple recipients and take payments separately, I will accept payments via Paypal and take care of sending the cufflinks out. As an example of postage costs, sending to USA would cost me approximately $12 per parcel. Let me know if you’re interested via a comment on this post, e-mail (find me in the phonebook), or IRC (:davehunt).

Whistler Work Week 2015

Last week was Mozilla’s first work week of 2015 in Whistler, BC. It was my first visit to Whistler having joined Mozilla shortly after their last summit there in 2010, and it was everything I needed it to be. Despite currently feeling jetlagged, I have been recharged and I have renewed enthusiasm for the mission and I’m even more than a little excited about Firefox OS again! I’d like to share a few of my highlights from last week…

S’mores – The dinner on Wednesday evening was followed by a street party, where I had my first S’more. It was awesome!
Firefox OS – Refreshing honesty over past mistakes and a coherant vision for the future has actually made me enthusiastic about this project again. I’m no longer working directly on Firefox OS, but I’ve signed up for the ~~dog~~foxfooding program and I’m excited about making a difference again.
LEGO – I got to build a LEGO duck, and we heard from David Robertson about the lessons LEGO learned from near bankruptcy.
Milkshake – A Firefox Q&A was made infinitely better by taking a spontaneous walk to Cow’s for milkshakes and ice cream with my new team!
Running – I got to run with #running friends old and new on Tuesday morning around Lost Lake. Then on Thursday morning I headed back and took on the trails with Matt. These were my first runs since my marathon, and running through the beautiful scenary was exactly what I needed to get me back into it.
Istanbul – After dinner on Tuesday night, Stephen and I sat down with Bob to play the board game Istanbul.
Hacking – It’s always hard to get actual code written during these team events, but I’m pleased to say we thought through some challenging problems, and actually even managed to land some code.
Hike – On Friday morning I joined Justin and Matt on a short hike up Whistler mountain. We didn’t have long before breakfast, but it was great to spend more time with these guys.
Whistler Mountain – The final party was at the top of Whistler Mountain, which was just breathtaking. I can’t possibly do the experience justice – so I’m not even going to try.

Thank you Whistler for putting up with a thousand Mozillians, and thank you Mozilla for organising such a perfect week. We’re going to keep rocking the free web!

Rock ‘n’ Roll Liverpool Marathon 2015

On Sunday I completed my first ever marathon, and I did it in under 4 hours! It’s only just sinking in what an achievement that is! When I signed up for it last year I predicted a finish time of under 5 hours, but after following a sub 4 training program I started feeling that I could actually do it.

I don’t have any other marathon experience to compare it to, but it was an incredible event. I’d like to thank everybody that helped to organise or came out to support the runners – even those saying we were nearly there before we hit the halfway point, or saying we were halfway at the 18 mile marker! There were times the support really helped – In particular those last 4 windy miles along the river. A special thanks go to my family and friends that cheered me at the start, at the 10 mile marker, and at the end. Also, thanks to my family for putting up with all those long runs during my training.

Inevitably I’ve already been asked about my next marathon, but I’m going to allow myself to recover before I give it too much thought. I have already entered the ballot for next year’s London marathon though, so who knows…

Joining Web QA

I’m excited to announce that as of last week I am officially on Mozilla’s Web QA team! Despite working closely with the team since I started at Mozilla over four years ago, I’ve always reported to another team. Originally I had a hybrid role, where I reported to the Director of QA and assisted with automation of both Web and Desktop products. Later, we formed the QA Automation Services team, which existed to provide automation services to any QA team. This was mostly absorbed into the A-Team, which shares a lot of the same goals but is not just focused on QA. During my time with the A-Team a lot of my work started to shift towards Firefox OS, so it made sense during a organisational shift towards vertical stacks for me to officially join the Firefox OS automation team.

Many times since I started at Mozilla I’ve felt that I had more to offer the Web QA team, and I’ve always been a keen contributor to the team. I can’t say it was an easy decision to move away from Firefox OS, as it’s a terrifically exciting project, but the thought of joining Web QA just had me bursting with enthusiasm! In a time where there’s been a number of departures, I’m pleased to say that I feel like I’m coming home. Look out web – you’re about to get even more automated!

Last week Stephen Donner interviewed me on being a Mozilla Web QA contributor. You can read the blog post over on the Web QA blog.

I’d like to take this opportunity to thank Stephen Donner, Matt Evans, Clint Talbert, Jonathan Griffin, and James Lal for their support and leadership. I’ve made so many great friendships at Mozilla, and with our Whistler work week just around the corner I’m so looking forward to catching up with many of them in person!

Performance testing Firefox OS on reference devices

A while back I wrote about the LEGO harness I created for Eideticker to hold both the device and camera in place. Since then there has been a couple of iterations of the harness. When we started testing against our low-cost prototype device, the harness needed modifying due to the size difference and position of the USB socket. At this point I tried to create a harness that would fit all of our current devices, with the hope of avoiding another redesign.

If you’re interested in creating one of these yourself, here’s the LEGO Digital Designer file and building guide.

Unfortunately, when I first got my hands on our reference device (codenamed ‘Flame’) it didn’t fit into the harness. I had to go back to the drawing board, and needed to be a little more creative due to the width not matching up too well with the dimensions of LEGO bricks. In the end I used some slope bricks (often used for roof tiles) to hold the device securely. A timelapse video of constructing the latest harness follows.

We now are 100% focused on testing against our reference device, so in London we have two dedicated to running our Eideticker tests, as shown in the photo below.

Again, if you want to build one of these for yourself, download the LEGO Digital Designer file and building guide. If you want to learn more about the Eideticker project check out the project page, or if you want to see the dashboard with the latest results, you can find it here.