Purism: Behind the (coreboot) scenes

This post does not reflect the opinion of the coreboot project, but reflects the personal opinion of its author, Alex Gagniuc.

Since the last time I talked about Purism’s stab at a coreboot laptop, a lot happened, including the launch of the non-libre Librem 15, alongside with plans for a 13″ version. My last post seems to have sparked quite some controversy on the subject, placing me on the CC field of many emails between not-so-happy supporters and Todd Weaver. I’ve avoided writing on this subject again, because I didn’t have anything good to say about Purism. However, considering the amount of new emails being generated lately, I think I should follow up on why the Librem 15 failed on the freedom front, and why another Purism laptop is just as likely to fail.

The road so far

A lot has happened since my last post, including a lot of media coverage on the issue. Now I can use acronyms like FSP, ME, and EC, without having to worry about losing 90% of my audience. It’s great that there is now a lot of non-technical coverage on the issue, something that non-corebooters can easily digest. However, some things also happened behind the scenes, which I’d now like to talk about.

Trying to meet Todd in person

After my initial post, a large number of emails flew back and forth between angry backers, Crowdsupply, and Purism, with me on the CC field. From this, I’ve had the chance to communicate with Todd Weaver directly, and express to him my concerns on why the Librem was about to fail. I also offered to set up a meeting with Stefan and Ron. I’ve talked to Stefan, and he seemed excited to speak with Todd and help put coreboot on the Librem. Ron shared the following over the email thread:

Todd, I think the overall concern with librem and your statements is the seeming lack of realization that you're walking over very well trod ground, and there are lessons learned, and we might as well pass them on.
I for one seeing you making the same mistakes that have been frequently made over 15 years, and there's benefit to learning what we've learned.

That was back in March. The meeting hasn’t yet happened. The discussion died down when I asked Todd to produce the current source code for the upcoming librem. After almost a month from my initial inquiry, on May 11, Todd wrote:

We will be releasing the source code once we get coreboot working on our rev1 which will be shipping within the next few weeks.

The Librem 15 launched

Fast forward a few months later, after the email exchanges died down, I get a tip that the Librem 15 has shipped with AMI UEFI firmware. While I do not have a Librem 15 in my possesion, this has been confirmed by PC World. Although I hate having been right about this, I love saying this: I told you so.

Todd claims he has coreboot developers working on Librem

A new wave of angry emails ensued just recently. I once again had the chance to communicate with Todd directly. He claimed to have three coreboot developers working for Purism, but they wanted to remain anonymous. He did, however, provide the name on one of the developers, whom I shall not mention for privacy reasons. Of course, I was curious to see what that person worked on:

[coreboot]$ git log |grep Author |grep -i <first name> -c
0
[coreboot]$ git log |grep Author |grep -i <last name> -c
0
[coreboot]$

I’ve informed Todd that this developer is not a coreboot contributor, and for the purpose of our discussion, does not count as a coreboot “developer”. I’ve asked Todd to produce git hashes of patches contributed by one or both of the other two developers. He has not done so.

Purism attacks Minifree (formerly Gluglug)

purism_attacks_minifree

Just this morning, a tweet was brought to my attention, which, to me, seems like a direct attack on Minifree from Purism. The tweet compared an “old heavy IBM Thinkpad”, with the Librem 15, showing the picture of a T60 Thinkpad running libreboot. This is also one of the pictures Gluglug (now Minifree) used on its product page when they were selling the T60 model.

For those of you unaware, Minifree (formerly Gluglug) sells laptop systems which are completely free, from the OS down to the firmware, and which are endorsed by the FSF through their Respects Your Freedom certification. The Minifree laptops are what we, as the community have been working to achieve since the inception of LinuxBIOS more than fifteen years ago, and the reason I have stuck with the project for over half a decade despite all the difficulties and roadblocks. To attack Minifree is to insult all of our hard work over the years, and to me, it indicates that Purism really doesn’t give a damn about your freedom, but they really like your support and money.

I know I promised I would also talk about why another Purism laptop is just as likely to fail as the Librem 15, but I’ve ranted enough for one post. I’ll describe that into more detail next time.


UPDATE: I just received the following private email from Todd Weaver:

Ouch, that is not an approved tweet. I asked to have it removed, since I am a big fan of what Gluglug did/does. And we provide it as an alternative from our own website.

Purism: Why Librem is not the same as libre

This post does not reflect the opinion of the coreboot project, but reflects the personal opinion of its author, Alex Gagniuc.

Purism Librem is an interesting project which promises to produce a completely libre laptop, designed to respect the user’s privacy and the user’s rights. The indivduals behind the project seem so confident as to “promise that a Purism system and all its components will be free according to the strictest of guidelines set forth by the Free Software Foundation’s Free Software Definition.” Can they deliver on this bold claim?

I first heard about Purism Librem sometime last October, when someone on #coreboot posted a link to Purism’s ideas/ideology page. It promised a fully libre system, from the ground up, with a high emphasis on libre firmware.

I only glanced over that page quickly before dismissing it as another over-ambitious and mis-informed project. There was no way that the Intel CPU and chipset they wanted could run libre, given Intel’s tight grip on the low level boot process. Coupled with their desire to include an Nvidia GPU — another high-profile offender in the libre software world — their ideas looked doomed.

The first red flag was that we, the coreboot hackers, were never contacted by Purism about what it would take to get such a design up and running on the firmware side. We could have immediately told them that there are major pieces of the initialization path for their CPU which were missing source. That is, they were only available as blobs.

The possibility of reverse engineering those blobs existed at the time. Although that takes a lot of effort, we’ve done it numerous times before. But they never asked. Had they done so, we would have also told them about another major offender. That’s the microcontroller in the chipset, which needs a signed firmware binary. By “signed” I mean a state-of-the art cryptographic verification mechanism. The chipset will refuse to run any firmware unless it was signed by a secret key held deep within Intel’s most secure dungeons. In short, this blob isn’t going away.

It was obvious to me from those first versions of Librem’s web page that the designers behind it had no idea about the binary situation. My impression was that they never contacted knowledgeable parties about it. They were reinventing the wheel and they were doomed to fail.

But that wasn’t to be. A month or so later, the subject popped up again on #coreboot. This time, they were considering crowd-funding. The information they provided via their website has changed as well. They were now claiming to be working with Intel about releasing source code for the early initialization blobs, and talking about disabling the chipset microcontroller entirely.

That sounded like a reasonable plan, with only one deadly fault: Intel doesn’t release this type of source code, and the Intel decision-makers do not talk with low-volume customers directly. Not even Google could get the source code out of Intel after shipping millions of Intel Chromebooks. The few hundred units that Purism was planning to build was definitely not going to cut it.

By that point, I made up my mind that the people behind Purism were either naive, or full of it. Deep in my heart, I wanted them to succeed, and I wanted to personally congratulate them for said success. I’m a coreboot developer; I know how this business rolls. I can make your firmware email me a daily digest of your passwords and Facebook activity, and you wouldn’t even know about it. I know what I’m talking about.

Fast-forward to the middle of January. Librem’s crowd-funding campaign had been extended from December to the end of January. A lot of new information also appeared on their web pages. Besides a quote from Richard Stallman, suggesting the FSF endorses their project, their campaign page was filled with buzz-words that seemed to have come straight out of Stallman’s most iconic speeches. The claim was that Librem is “designed to respect your privacy”.

There was also a graphic representation of the software stack. A green square meant libre, while a white square meant closed. There was so much green on the reverse pyramid of the graph, that it was too easy to miss the two or three white squares at the base. They were the platform firmware. That’s the part that emails me your passwords without any of the green dots above it being aware.

Finally, by January 22, a ZDNet article appeared which confirmed what I had predicted all along: that Librem will ship with proprietary firmware. That’s announcement came just a week shy of the crowd-funding campaign drawing to a close. With proprietary firmware, Librem is just as free as any other laptop on the market with GNU/Linux. Or in other words, it’s not any more free. It is certainly not libre.

So at the end of the day, Librem is bringing nothing new to the market. Laptops with libre operating systems have existed for decades. The only real innovators in this area have been Google and GluGlug. Google ships partially free firmware, although insufficiently libre to be able to provide the “respect your privacy” guarantee. GluGlug can make this claim, and it ships laptops with fully libre firmware. The downside of GluGlug is that it’s an aftermarket add-on. GluGlug and Google have been in business far longer than Purism. So, what has Purism brought in that’s new and exciting and libre? Nothing.

Reverse engineering blobs: adding diff to the toolkit

Last time I talked about the benefits of using sed to transform repetitive low-level patterns into meaninful function calls.  And still, doing all that regex magic did not get us a fully working replay. A great portion of the hardware initialization flow is based on situational awareness. What hardware is connected? What are our capabilities? What if …?

That means a simple sequence of writes is, in most cases, not sufficient. We may need to modify registers, wait on other hardware, or respond differently to hardware states. While that seems daunting and tedious, it gives us an unexpected advantage: that every execution of the blob produces a different trace.

This is where diff comes in. By getting a bunch of traces and diffing them, we can see the points where the firmware takes different decisions, and the states which determine those decisions.  It won’t tell us what condition triggers path A or path B, but it allows us to infer that by comparing the hardware states. Let’s have a look:

@@ -305,28 +305,28 @@ void run_replay(void)
 radeon_read_sync(0x6430); /* 04040101 */
 radeon_write_sync(0x6430, 0x04000101);
 radeon_write_sync(0x3f50, 0x00000000);
- radeon_read(0x3f54); /* 000dda12 */
- inl(0x2004); /* 000dda8a */
- inl(0x2004); /* 000ddb1a */
- inl(0x2004); /* 000ddb9c */
- ...
- inl(0x2004); /* 000de3a0 */
- inl(0x2004); /* 000de41a */
+ radeon_read(0x3f54); /* 000d9efb */
+ inl(0x2004); /* 000d9f73 */
+ inl(0x2004); /* 000da003 */
+ inl(0x2004); /* 000da081 */
+ ...
+ inl(0x2004); /* 000da883 */
+ inl(0x2004); /* 000da8fd */
 radeon_read_sync(0x611c); /* 00000000 */
 radeon_write_sync(0x611c, 0x00000002);
 radeon_write_sync(0x6ccc, 0x00007fff);

I’ve chosen an example that shows similarities rather than differences, as I find this to be a more interesting case. Since we’ve already established that 0x2004 is our data port, as long as we don’t touch the index port, we’ll be reading from the same register, in this case 0x3f54.

Now the values returned by this register are completely different in every trace, yet the behavior of the blob is strikingly similar every time. The first key observation is that this register increase monotonically in every trace. It also increases by roughly the same amount on successive reads. The differences between the last read and first read of the register are also strikingly similar in both traces: 0xa08 and 0xa02 respectively.

This register looks to be a monotonic timer, and the loop has all the elements of a delay loop. To determine the actual delay, we could try to extract absolute timing information when collecting the trace; however, in this specific case, I had the AtomBIOS tables handy. By comparing register accesses around this loop, I was able to figure out where in the tables this delay is occuring:

 0200: 0d250c1901 OR reg[190c] [...X] <- 01
 0205: 54300c19 CLEAR reg[190c] [.X..]
 0209: 5132 DELAY_MicroSec 32

The ’32’ in the delay is a hex number. Doing a bit of hex math we see we’re waiting about 51 ticks per microsecond. Comparing more loops, we get between 50 to 52 ticks per microsecond. Since a delay loop normally waits until the minimum time has elapsed, we now have a very convincing case that register 0x3f54 implements a 50MHz monotonic timer. Every time before accessing this register, we also poke register 0x3f50. That looks very much like the timer control register.

We now extend our sed script with:

timerctl=0x3f50
timer=0x3f54
...
sed "s/radeon_read($timer);[^$hex\r]*\([$hex]*\)[^\r]*\(\r\tinl($dport);[^$hex\r]*\([$hex]*\)[^\r]*\)*/radeon_delay(0x\3 - 0x\1);/g" |
sed "s/radeon_write_sync($timerctl, 0x0\{1,8\});[^\r]*\r\tradeon_delay(/radeon_delay(/g"

Now when we rerun our logs through the script, the results decrease in size from 20K lines to 13K lines. The diffs between processed logs also decrease in size significantly. All the more proof we were right!

There’s another way in which diff is excellent for our purpose. We can implement our helpers to generate the same output as the processed logs. That allows us to poke the replay from userspace, yet get the same output format. Now we can diff the replay and original log, and observe how the hardware state changes. We can even go as far as implementing our delay with usleep() instead of the timer at 0x3f54. When the diff is independent of the delay method we use, we have another strong proof our assumptions are true. This is the case here.

‘diff’ is an extremely powerful tool. Despite its name, it can show similarities just as well as differences. While regular expressions exaust their usability with simple patterns, diff can take us a lot further. Now that we’ve cracked the delay implementation of the blob, we can more easily see delay and wait loops — again, using diff. Complex, multivariable patterns are too awkward to handle with sed. I’ll go over those some other time. However, once such patterns are simplified to a function call, diff can once again show the story. Different GPU model? diff. New display? diff. HDMI connected? diff. It’s almost as versatile as det cord .

Reverse engineering blobs with replay and sed

I recently started re-visiting the HP Pavilion m6 1035dx with a recent coreboot master. As usual the benign VGA BIOS got in the way again. This time I decided to use coreboot’s YABEL realmode emulator to tell the story. Let’s dive in:

Executing Initialization Vector...
[0000f2cb]c000:3b69 inb(0x03c3) = 0x20
[0000f2cc]c000:3b6f inl(0x204c) = 0x00000000
[0000f2ce]c000:3e06 inl(0x2000) = 0x9ffffffc
[0000f2cf]c000:3b69 inb(0x03c3) = 0x20
[0000f2d0]c000:3b6f inl(0x204c) = 0x00000000
[0000f2d3]c000:3b69 inb(0x03c3) = 0x20
[0000f2d4]c000:3b6f inl(0x204c) = 0x00000000
[0000f2d5]c000:3e61 outl(0x00001728, 0x2000)
[0000f2d5]c000:3e67 outl(0x0008c000, 0x2004)
[0000f2d8]c000:3b69 inb(0x03c3) = 0x20
[0000f2d9]c000:3b6f inl(0x204c) = 0x00000000
[0000f2da]c000:3e49 outl(0x00003f54, 0x2000)
...

That’s how the coreboot log looks when we enable YABEL traces. Enabling direct hardware access produces a cleaner log to start with. We want to clean that up a little bit. A bit of simple regex substitution gets us there. Excuse the wrap-around:

egrep "c000:[0-9,a-f]{4}|x86emuOp_halt|runInt[0-9,a-f]{2}.*starting" $1 |
grep -v "Running option rom at c000:0003" |
sed "s/c000\:[0-9,a-f]\{4\}\ /\t/g" |

tr '\n' '\r' |

sed "s/\[[0-9,a-f]\{8\}\]//g" |
sed "s/)/);/g" |
sed "s/=\ 0x\([0-9,a-f]*\)/\/\*\ \1\ \*\//g" |

tr '\r' '\n'

That’s enough to get our trace to something that looks more like C code, which could be used as a replay function (HINT!):

 inb(0x03c3); /* 20 */
 inl(0x204c); /* 00000000 */
 inl(0x2000); /* 9ffffffc */
 inb(0x03c3); /* 20 */
 inl(0x204c); /* 00000000 */
 inb(0x03c3); /* 20 */
 inl(0x204c); /* 00000000 */
 outl(0x00001728, 0x2000);
 outl(0x0008c000, 0x2004);
 inb(0x03c3); /* 20 */
 inl(0x204c); /* 00000000 */
 outl(0x00003f54, 0x2000);
...

We’ve neatly kept the return values of IO input operations for future reference, but the current form doesn’t tell us much. We do, however see patterns emerging. IO to 0x2000 followed by 0x2004 is fairly common. That looks like an index/data pair. Also, access to 0x03c3 and 0x204c before poking the above pair is all too common. Let’s extend our script with:

iport=0x2000
dport=0x2004
stsport=0x204c

..

sed "s/outl(0x0\{0,4\}\([0-9,a-f]*\), $iport);[^\r]*\r\toutl(0x\([0-9,a-f]*\), $dport);/radeon_write(0x\1, 0x\2);/g" |
sed "s/outl(0x0\{0,4\}\([0-9,a-f]*\), $iport);[^\r]*\r\tinl($dport);/radeon_read(0x\1);/g" |

sed "s/inb(0x03c3);[^\r]*\r\tinl($stsport);[^\r]*/sync_read();/g" |

Since we’ve converted all our newlines to carriage returns, we can match patterns from multiple lines. It doesn’t matter what we think these patterns do, or how we call the new functions. We’re just interested in grouping them to see the bigger picture. I could as well have called these hamburger() and french_fries().

 sync_read();
 inl(0x2000); /* 9ffffffc */
 sync_read();
 sync_read();
 radeon_write(0x1728, 0x0008c000);
 sync_read();
 radeon_read(0x3f54); /* 002badc3 */
...
 sync_read();
 radeon_read(0x0670); /* 0000fe04 */
 sync_read();
 radeon_write(0x0670, 0x0000fe04);

Much better! We’ve replaced the low-level patterns with more meaningful descriptions. We could grep out the sync_read(), or, if we were using this for replay code, incorporate the _sync() into another function with radeon_(). Even in this form, we can start looking for higher-level patterns. If we look into the disassembly of the video BIOS provided by AtomDis, we see:

 0009: 07a59c01fc AND reg[019c] [.X..] <- fc
 000e: 0d659c0180 OR reg[019c] [..X.] <- 80

Since the registers are 32-bit, then their address would be reg << 2. Thus [019c] becomes 0x0760. These read-modify-write sequences appear in the replay trace.

Now we have an idea where we are with respect to the AtomBIOS tables, we see, at a low level, how those tables translate to hardware accesses. As more patterns are identified, they can be transformed into more meaningful function calls

Now we have the opportunity to look for more advanced patterns, and even identify any code that is not in the AtomBIOS tables. This, in turn can allow us to figure out what actually turns on the display. There is still a huge gap before having anything close to native init. What we’ve done here is develop a coreboot-level understanding of the init process, and made the first tiny step towards native VGA initialization.

While there is only so much regex substitution can do for us, it is a necessarry first step towards a larger understanding of the problem. Transforming 0x0760 to 0x019c is ill suited for regex.  Identifying more complex patterns such as waiting for a condition, or delaying a set amount of time is also a more demanding task; however, the power lies in the ability to script and automate the conversion from a nonsensical log into something more human-friendly. It then becomes trivial to diff several traces and see higher-level patterns. I’ll discuss those some other time.

Pencils down

This is it. The beginning for QiProg has ended. It has been a long and tedious journey to equip the Stellaris Launchpad for Low Pin Count mastering. The hardware is built, and it works. The software is there, although its contribution to the ecosystem is somewhat minimal — it is more of a bridge than a road in itself. The real value lies in the firmware. To the best of my knowledge, LPC bus mastering has not been implemented in a microcontroller without using ASICs or FPGAs. The vessels are here, and now it is time for the explorers to start their journeys.

Why do I talk like this is project is not over? It is not over. Although I have attained the minimal goals, there is a whole new world to explore. Integration in flashrom was one of the targets I had really hoped for, but was unable to complete. Is writing of the chip working? Yes. Is it reliable? Writing and verifying predetermined patterns works. Disturbingly, when I plugged in output from /dev/urand, the write did not verify. Some bits were simply not programmed, although the majority of the chip matched. Is it the LPC bus mastering that is at fault, is it the command sequence, or are we not giving the chip enough time to complete the operation? Will chips other than the SST 49LF080A work? Can we write a complete ROM image and boot a machine? This is the exploration phase: we have built it, now let’s make the most of it.

Time has been tight for the last month. I had forgotten to plan for the start of the fall semester, and my time since this event has been severely crippled. I wish I could have achieved more. I took the last two days off from the University to organize the last few weeks’ salad of stashed and uncommitted changes into readable patches.

I don’t feel sad, just tired. Exhausted, I have nothing left to keep me entertained but an Alec Bradley Presando cigar, which I have been saving for a special occasion. This is that occasion: QiProg has stopped being a commitment, and became a hobby, a child, something to care for. Now is the time to build your VultureProg, get the software, and start reporting those issues you know you will encounter. I’m in Houston, and I don’t have a problem.

Retrofitting QiProg for the space age

In the previous two posts, I was developing a bigger issue that has became more and more apparent. The USB QiProg protocol is not completely specified. I didn’t think much of the “TODO” items in Peter’s original protocol. I figured I’ll figure them out as I go along, and I did figure out a lot of little details as I went along. Of course, none of those details related to implementing the big TODOs. So, here I am in week 11(?) with an incomplete protocol.

Last week did not see much coding. I spent some quality time with premium cigars, a pen, and engineering paper. I wrote the name of each function I needed to implement, left a blank space, wrote the name of the next function, and so on. Then, I drew the packets, laid out their structure, went through the process of thought experimentation, crumpled the piece of paper, threw it again, and started from scratch. I eventually settled on a protocol “completion” which I though was acceptable. I wrote it up in digital form and submitted it to Peter for review.

Communication with Peter is asynchronous. Very rarely do we both meet up on IRC to discuss the details in real time. Therefore, I decided to implement the “completion” protocol as-is, and modify the implementation later should the protocol change. This should keep me occupied with coding. I am very happy I can get back to coding. Working out protocol details is exhausting and boring.

It is a bit frustrating. We have firmware code to read, erase, and byte-program, yet the puzzle is not complete. I was expecting the LPC bus mastering to be the most demanding, and USB communication to be an almost transparent add-on. That would have been fun! The reality is the opposite, but, besides the aforementioned frustration, it’s still fun — albeit a different type of “fun”. It’s time to scrap weekend plans for the next couple of weeks, and kick in the overtime. Until next time, get your Stellaris Launchpads, order your VultureProg PCBs, and stay tuned. This baby is going to run smokin’ hot.

A brief progress sheet

Last week, I laid out a list of things to do in order to get more of the protocol finalized. Most of the items are crossed off, however, one little item remains standing. This item, while innocent and seemingly harmless is more painful than falling on your buttocks from a 10 story building on solid granite. Let’s have a look at why this small item is of such significance.

  • new API call set_chip_size()

When people think of QiProg, they think of one gadget with one flash chip connected. This is the common case, and, for the foreseeable future, will be the de-facto way of using QIProg. However, the original USB specification was intended for a broader use case: a programmer with several, individually addressable chips connected. One who observes the qiprog_read_chip_id() call will notice that it translates to a READ_CHIP_ID request over USB. This request will return identification data for up to nine chips. Aye, there’s the rub.

How does this play into set_chip_size()? Simple, set_chip_size for which chip? Do we send a flat list of nine uint32_t sizes, thus only needing one round-trip (control request) for all chips? Do we use the wIndex field of the round-trip, at the cost of needing one such trip for each chip? Once this question is answered, it will determine the answer for set_[erase/write]_[size/command] call and their respective USB round-trips, thus completing the USB protocol, and bringing QiProg to a usable state.

It’s easy to see why this one little detail is a blocker for all other remaining issues. I am leaning towards the use of wIndex (not the glass cleaner). Implementing a new control request in software and firmware is a matter of minutes. Testing it, and making sure it works properly is, at most, a two hour endeavor. Getting the design right: priceless.

Experiments of mind

The time for writing code is over. The time to design hardware is over. After seven weeks, the vultureprog_action_shotbeginning has come to an abrupt end. I am severely behind schedule. In week seven I was supposed to implement erase functionality — tell the programmer how to erase the chip. This is not done. On the other hand, I have had code for weeks 8 and 9 almost ready, and just merged most of it last week. So, where am I? Am I ahead or behind schedule?

The fallacy of preemption

One of the requirements for applying as a GSoC coreboot student was to have a fully established, vultureprog_probingschedule from day[-1]. Establishing this schedule was a great experience, and it allowed me to think in depth about the problem and possible solutions — to a certain degree. I picked the steps I considered logical, in the order which I saw logical. Development is never about writing code in the order in which it will be executed. In this particular case, it was much easier to implement bulk writing without a predefined erase/write strategy, opting instead for a default just-do-it approach.

Why is this approach better than following the schedule, from a development point of view? We have had bulk read partially working for a while now. From the host point of view, reading and writing are symmetrical operations. The bulk of the code (pun definitely intended) is shared between the read and write operation. They both juggle data on the same endpoint. The only difference is the endpoint direction bit. It therefore made sense, once bulk reading was fixed for corner cases, to uses the same code to send data to the programmer. Making the programmer write that data was a matter of a couple of hours. There was no sensible reason to wait an additional two weeks before implementing this last bit.

Software development work is as much about making things work, as it is about the application of programming principles with unquestionable moral authority and correctness. In this case, implementing a trivial extension reusing code fresh in my mind was the preferred approach. Not only did it save me time by not having to re-examine the situation a few weeks from now, it also allows me to have a working program/verify scenario when implementing the erase strategies. As one might imagine, this makes the problem a lot easier. Attempting to preempt and enforce a schedule before the problem is thoroughly explored, occasionally conflicts with best practices of development. With this in mind, I am neither behind, nor ahead of schedule. I am precisely where I need to be.

A matter of experimentation

Most of the infrastructure and code is already in place. Bringing QiProg to completion is no longer an issue of adding functionality through code, but rather completing functionality by connecting the existing code. One issue I discovered after testing the bulk program code was a terrible race condition between read prefetching and the write loop. The prefetch logic incremented the internal address before data arrived. As a result, the new data would get written at the wrong address. Choosing the best solution to the problem is a matter of experimentation.

The “this won’t work because of that” and “what if this” turned into a series of exhausting thought experiments. I have been bugging Peter a lot in the past few days about a series of potential issues. Through tiring thought experimentation, we eventually agreed that the best way to proceed was to abstract a lot more through the API. This is a non-exhaustive list of the decisions we’ve made in the past week:

  • set_address() is hidden from the API
  • the internal address range is not exhausted once read or written
  • read and write operations must not be interdependent, the internal read and write pointers will be distinct (as a side effect, this change also eliminates the race condition depicted above)
  • set_address() + readn() turns into read(dev, where, n)
  • All API addresses begin at 0. The programmer translates that into an absolute address
  • new API call set_chip_size()
  • new API call to explicitly erase blocks or sectors (to be defined)
  • implicit erase on write can be enabled or disabled (to be defined)
  • implicit erase will erase the sector/block right before the first byte of the sector is written
  • exposing any USB specific dependencies in the API is strictly forbidden

My focus for the remainder of this week will be to shorten this list as much as possible. Once the dependency between read and write is unshackled, I will be able to erase/program/verify my faithful SST 49LF080A. From here, it will be a matter of finalizing and implementing the last obscure bits of the specification.

The state of QiProg for flashrom

As QiProg is still being finalized, implementing it as a flashrom programmer is still a long ways ahead. I do estimate that weeks 11 and 12 will provide ample time to integrate everything into flashrom, hopefully, in time for the 0.9.8 release.

VultureProg command center

command_center
Upgraded VultureProg command center

It was time to upgrade. I got a new desk last week. For me, a piece of furniture is as boring as watching stainless steel rust (it does rust eventually). A desk, on the other hand, is anathema to replace. I have 20+ wires connected to my workstation going in all sorts of places. I like to keep these wires carefully routed and out of sight, a task made all the more difficult by the lack of any wire management gizmos on new desks. Consequently, after I get a new desk, you are well within your rights to imagine me grabbing my drilling machine, a set of saw cutter bits and an assortment of hole covers — and you need not wait long to see it. Needless to say, getting back up and running has taken a few days, but we are back with a shiny new VultureProg command center (pictured above for your viewing convenience).

The 80-20 rule

In 2011 I used to work developing Android apps. My boss told me “you will spend 20% of the time writing 80% of the code, and you will spend the remaining 80% of the time writing the remaining 20% of the code”. Sadly, QiProg has proven to not be an exception to this rule. Although I have consistently fixed issues and improved the layout of the code, the number of lines of code has been more or less stagnating after the first three weeks of exponential growth. I know how to erase the chip, I know how to program the chip, I know how to read the chip, and I have code to do all that. I just have not yet had the chance to hook it up to the ecosystem.

I am currently working on improving the bulk reading code. There are a few corner cases which are not well handled. As expected, it is taking a lot of time, and I even had to write a special testcase before I started.

Did you want fries with that?

Although we know how to handle every single aspect of the identify/read/erase/write/verify cycle, finding a way of connecting everything together in a simple, elegant, and efficient manner is a different story altogether. We have three API calls for handling erase and write, namely qiprog_set_erase_sizeqiprog_set_erase_command, and qiprog_set_write_command. Peter wants everything to work without requiring an explicit erase. In his view, the VultureProg firmware should automatically erase a sector that is being written. Although this would simplify dumb cases where the whole chip or large sections of a chip are written at once, it spells disaster for flashrom’s aggressive optimization. What happens if the first part of a block matches contents, and flashrom decides to write the other part without needing an erase? VultureProg would erase the entire block, then write the second part to the block we never meant to erase. Questions such as this one need to be carefully thought over and elegantly answered. If I blindly start putting all the code together right now, I’ll most certainly have to fix it later.

Postal patronage

I received a very interesting package last week. I had a deal with Idwer to get him rid of a few LPC and FWH flash chips. I was able to take them off his hands for just a few euros, a deal made sweeter by the fact that those chips are no longer being sold anywhere. I also sent Peter a VultureProg board. Since he already has a Stellaris or two, I have just recruited an eager tester.

Can you send me a board?

Short answer, no. Just grab the gerbers and take them to SeeedStudio. They will be able to get boards in your hands for much less than what I pay to ship them. The two SMD capacitors are not hard to assemble. If you still want me to send you an assembled PCB, I will charge $50 (international) or $30 (US). I don’t have the physical time to sit and assemble PCBs by hand, plus, it would be very un-geekish of you to not assemble your own.

VultureProg: Equipped for galactic travel

vultureprog_ready_for_launchIt’s here! And it’s ready for takeoff. The VultureProg PCBs have finally arrived, and it is time to turn VultureProg from a proof-of-concept toy to a serious galactic tool. My major concern was that I could have misrouted one or two connections. The LAD pins are particularly sensitive, as they need to be mapped to sequential GPIO pins, and start from GPIO0, otherwise we need to do bitshitfs in every LPC cycle, killing any hope of decent performance. I was also worried that the particularly tight tolerances could be problematic during manufacture. Everything works as expected. Enough words, let’s see the porn.

vultureprog_hull
The bare PCBs
vultureprog_shuttle
Brings back memories, doesn’t it?
vultureprog_shuttle_angle
The world’s first fully assembled VultureProg board

From bitstream to reality

I found it interesting to look at the transformation from a spaghetti on the screen to something real, something tangible.

vultureprog_shuttle_sbs
From conception to birth: the complete transformation

No thorough and thoughtful post today. I have a new toy.