If you find yourself optimizing your code, then it means that the author of your compiler has failed.

This is just very, very untrue. Even if you strike algorithmic optimization from the picture, code optimization is still a very important and useful skill to have no matter what sort of programming you’re doing. Knowing your target platform, knowing the behavior of your compiler or interpreter, and knowing your data (hat tip again to Mike Acton for driving this point home to me) can provide you with ideas on how to transform your code and realize massive performance gains.

If you’re writing performance-critical code, no compiler is going to do everything for you – the best optimizer is still between your ears. The idea that poorly-performing code can be blamed on the compiler is simply naïve and defeatist.

There are only two debugging techniques in the universe:

1. printf.
2. /* */

Since I recently posted my own little list of some debugging techniques, I can’t resist weighing in on this assertion. While their list is a bit flip, changing behavior and running experiments to see the changes is, of course, a core debugging technique. However, the claim that debuggers are just extensions of this is a bit too reductionist for my tastes – it’s like claiming that a car is just a horse that takes longer to get tired, or that the printing press is just a scribe that works faster.

I also feel that “debugging techniques” shouldn’t be limited to “things you do to the program,” either – thought experiments, code change analyses, and the like are all valid in my book as well. My definition includes any form of investigation that gives you more insight into the problem.

Completely separate from the article linked above, I had a glance at Wikipedia’s article on debugging, and…well, there’s a lot of stuff in there that gets my back up:

• Citing language choice as having an impact on the debugging process is silly – your choice of language may make it more difficult to write buggy code, but once a bug is in there, I’m hard pressed to think of a reason why language choice would have a substantive impact on the actual debugging process. (Saying that C++ makes debugging easier than C because it has single-line comments is not funny.)
• “Generally, high-level programming languages, such as Java, make debugging easier, because they have features such as exception handling that make real sources of erratic behaviour easier to spot. In lower-level programming languages such as C or assembly, bugs may cause silent problems such as memory corruption, and it is often difficult to see where the initial problem happened.”

  1) Exception handling is only as useful as the exceptions that are thrown in the code.
  2) The claim that “high-level programming languages” like Java don’t suffer from “memory corruption” is misleading. Any imperative language with side effects is going to be capable of bugs that look like “memory corruption,” and can effectively be treated the same way with regards to the debugging process. (As an aside, it’s interesting to see how the definition of what qualifies as a “low-level language” have shifted over the years…yikes.)

• Static code analysis tools are meant to be used to fix code problems before you run into the bugs they cause. Using them as an example of a debugging tool is kind of missing the point of using them entirely. lint isn’t going to help you debug why something is busted – it’s only going to tell you that you’re using an uninitialized variable, and that’s something that shouldn’t have been in your compiled code in the first place.

Addendum to the earlier debugging post: One important quality of the printf and commenting techniques that I didn’t mention in my earlier post, though, is that they are techniques that will work even if you can’t actually attach a debugger to the process (or if there is no debugger for the environment in which you’re working). Sometimes this characteristic can really save your bacon – I remember having to change the screen background color register (the poor man’s printf, which doesn’t even need the C standard library) on a certain console in order to debug the DVD boot loading code of a game. Each line of code was prefaced with a call to set the background color to a distinct value, and the color remaining on the screen when the console froze allowed me to determine the location of the crash (and, eventually, the solution).

Whenever I run into a bug whose cause is not immediately obvious, I have a standard bag of tricks that I fall back upon. Every programmer has a toolbox like this – I figured I would write about some of the techniques that I use, and why I use them. Some of them are not applicable to every situation, but there are still many that can be applied to any given bug. These are presented in no particular order.

1. Change the inputs.

   Many times, by changing the inputs to a function, you can cause a recognizable change to occur in the output. This helps you to envision what’s actually happening in the function, and where things might be going wrong. This can include changing parameter values, input files, textures, etc.

2. Do things in a different sequence, or with different timing.

   Pretty self-explanatory, and related to the first item. The idea is to observe differences in the behavior of the program in similar circumstances. This is mostly useful for interactive programs.

3. Run all of your automated test code, even the slow tests, and examine any issues that are reported.

   This is helpful if for no other reason than as a sanity check.

4. Check the logs.

   This one is pretty standard. Even though most debug logs tend to be overflowing with spam messages, you might still find a smoking gun in there.

5. Ensure that you’re validating all of the return values of function calls.

   This is also known as the “be paranoid” rule, or perhaps the “re-check all of your assumptions” rule. It’s easy to forget to check return values, but it’s crucial to do so. Code that silently ignores failure can cause problems or symptoms unrelated to the function call that actually failed.

   A related problem is returning pointers to objects on the stack – this will result in havoc since they will be freed as the function exits.

6. Ensure that pointer values get cleared out when the struct or object to which they point is freed.

   Using stale pointer values is a surefire way to get in trouble. Clearing them out when the associated object is freed (except in very, very special circumstances) will help keep you sane.

7. Check for any masked exceptions in managed code.

   I wrote about this the other day. Ensure that no unexpected exceptions are being silently masked in your code.

8. Check the data.

   Make sure that the data you’re trying to use is actually valid! The GIGO principle is as true as ever. Check for data out of the expected range, QNANs being generated (which are infamous for screwing up subsequent floating-point operations), the ordering of data, legacy data, and correct offsets/sizes of data.

9. Put in additional logging or debug visualization code.

   This can help provide additional information, but this strategy can also backfire, as it can significantly change the timing of your code. (Disk, socket, and/or pipe I/O are relatively expensive operations.) Use with caution. If you have general-purpose code for validating the state of the application, sprinkle calls to that code throughout the application – this can be useful in determining when things go off the rails.

10. Check the crash dump, if you have one.

    Post-mortem analysis of core dumps is often extremely useful in tracking down bugs that you didn’t personally witness, or for which the steps to reproduce are lengthy or time-consuming.

11. Step through the code in the debugger.

    It can sometimes be quite slow if you’re processing large data sets, but it’s often the easiest way to monitor the control flow of a function.

12. Inspect the disassembly.

    This sounds hardcore, but being able to do this is invaluable in some cases. It’s useful not only for checking the compiler’s output, but also for cases where you’re examining a minidump of managed code. Unless you’re working with CLR4 and Visual Studio 2010, opening these minidumps in Windbg results in callstacks that don’t include line numbers. You do have the instruction pointer value, though, so you can actually print out the disassembly with the !u SOS command, and compare the disassembly with the original source code of the function to figure out the exact point at which the crash occurred.

13. Inspect memory.

    If you have pointer problems, look at the contents of memory in the debugger to try and figure out what’s going on. A frequent problem is an invalid offset, which results in struct member values “shifting” forwards or backwards. It helps to be familiar with memory representations of things like floating-point numbers – knowing a couple of common values (0x3f800000 == 1.0f, etc.) can be very handy.

14. Use conditional or memory breakpoints to isolate the bug.

    If you know that a particular object or memory address is related to the bug, you can set up breakpoints to pick out a particular loop iteration or write to a memory location. In cases where you’re interacting with a large body of unknown code, memory breakpoints can be particularly useful for tracking state changes.

15. Try a different build configuration (or turn on asserts).

    This is intended to provoke behavior changes, add validation, and otherwise provide additional data points for determining exactly what’s going on. Turning on validation such as array bounds checks and heap checking can help find some tough bugs (albeit at a tremendous cost in execution speed).

16. Run on a different platform, or build with a different compiler.

    There are often significant differences in timing and other behavior when you run software on a different platform. Endianness and word size also often differ between platforms, which can expose problems or bad assumptions about data in code. Like changing the inputs, careful observation of these differences can help you get an understanding of what’s actually happening. Additionally, if you are using a different compiler, you may see different warnings or code behavior due to optimization.

17. Check the version control history for anything suspicious.

    Examining changes to the source code can give you a good idea of how the behavior of the program has changed (even if you don’t know much about the changed code to begin with), and can shed some light on a bug. Checking all of the cases where a function is called to ensure that they take into account any changed behavior is essential.

18. If your codebase easily allows you to do so, try running the buggy code synchronously instead of asynchronously, for testing purposes.

    This can help determine if a race condition is what’s causing the bug to occur. (It should be noted that I’m not crazy about just adding random sleeps into an asynchronous function to determine this – it’s too unreliable for my tastes.)

19. Check (and re-check) your data dependencies in asynchronous code.

    Don’t fall into the trap of trying to envision multithreaded code by imagining each possible combination of instruction pointer values. Instead, when trying to prove correctness, focus entirely on data dependencies and ensuring that locks are used correctly and respected. (For deadlock bugs, inspect the order in which locks are taken, and check for proper use of back-off and other algorithms for avoiding deadlock.)

    Note that applying these techniques won’t help you write optimal multithreaded code – this requires much broader insight into the particular algorithm in question, and the overall architecture of the code. However, they will help in tracking down correctness issues.

20. Turn off chunks of the code, or switch to a different implementation of an interface.

    If you have multiple providers of an interface available, try using a different one, and see how the behavior of the program changes. (Using null/echo interfaces is a common debugging technique.) Additionally, you can try disabling features of your application to see if they are somehow related to the bug.

21. Use third-party validation tools and/or debugging information.

    This includes things like the Direct3D debug runtime, FxCop, lint, valgrind, the Visual C++ runtime debug heap functions, the Application Verifier, and the checked build of Windows. The more debugging aids you have active, the more likely that you’ll get a clue upon which you can act.

22. If the code is unfamiliar, find out who wrote it, and start asking them basic questions about it.

    This is similar to the “be paranoid” rule, except that by asking all of these basic questions of the author, you’re forcing them to re-check all of their assumptions. It’s not uncommon to have a eureka moment while explaining a bit of code to someone else.

23. Try turning off optimizations for a chunk of code.

    My experience has been that people tend to fall back on the “it must be a compiler bug” excuse explanation way earlier than they really should. Nevertheless, turning off optimizations for a section of code might help you debug a problem that occurs in optimized builds. (Whether it’s a genuine optimizer bug, or, say, a misuse of the C99 restrict type qualifier, is for you to find out. Anyone interested in using the latter, incidentally, should really read this excellent article by Mike Acton on the topic.) Performing a quasi-“binary search” when turning off optimizations can help minimize the time spent searching for the problem code snippet for a genuine optimizer bug.

24. Try running on a different machine, or piece of hardware.

    Hardware failure is another bug explanation of which programmers tend to be a little too fond. However, it does happen occasionally, so it’s definitely something worth testing if you run out of other ideas.

That’s pretty much all I can think of at this point. There are a few more tips that spring to my mind, but they are pretty specific to Windows or Visual Studio development, so I won’t recount them here.

Imagine that you have some code that runs in an interactive session, but which has a high-level catch block to catch and report errors. This catch block may attempt to continue execution when an error occurs – for example, writing one file in a batch might fail, but the code should continue writing other files. Unfortunately, the exception log produced by this might not provide sufficient information to debug the issue. For example, many of the I/O exception types fail to include information about which file or directory was being modified when the exception was thrown!

In tracking down cases like these, it’s often easier to set the debugger to catch that particular exception at the first chance it gets, and then examine the state of the calling function where the exception was thrown. Having both a call stack and the specific line of code where the exception was thrown often allows you to see the problem immediately, without further investigation.

Another illuminating debugging exercise is to turn on first chance exception handling for all exceptions, and then see where they are thrown. Code that masks exceptions (by using an untyped, empty catch block) is particularly nasty, as it can leave the program in an ill-defined state, but without any feedback to indicate that something failed. A different problem results from the frequent use of exceptions: they have a significant negative performance impact. Turning on first chance exception handling makes it trivial to find these cases, if you hadn’t already noticed the debug window spam that frequent exceptions tend to create.

* There’s a lot of stuff that gets posted there that gets my (figurative) blood pressure up, so we’ll see how long I stick with it. But I figured it would be a good exercise to read it regularly and write about things I see there.

As it turns out, there is a bit of chicanery going on inside Visual Studio that results in this behavior. Visual Studio actually runs the C# compiler in-process, as an optimization to avoid process start overhead. This in-process compiler gets fouled up, however, when C# source files are generated as part of a build step – it seems to load all of the source files when the build starts, so it winds up using the old version instead of the freshly generated one.

The solution is to add the UseHostCompilerIfAvailable property to the .csproj project file, like this:

<PropertyGroup>
	<UseHostCompilerIfAvailable>False</UseHostCompilerIfAvailable>
</PropertyGroup>

This will force Visual Studio to use the out-of-process compiler, which will cause the correct version of the grammar to be built. Building the project will be a little slower, but it’s better than having to build twice!

• Changing a structure definition such that a bunch of Win32 BOOLs use one bit each saves memory in the struct, because BOOL is typedef’d as an INT.
• Accessing the bitfield data members requires additional work (a couple of shifts, in the example). This chews up a little time, and some additional memory for the expanded code. The break-even point for overall reduced memory usage depends on the number of struct instances and the number of locations in the code that access the data.
• Packing data members in this fashion also prevents you from setting a data breakpoint on a particular packed flag or value.
• The atomic read/write operations work on words, making it more cumbersome to manipulate the packed values safely in a multithreaded environment.

Changing data members to bitfields has other effects, as well:

• When making this change, you might do a quick grep on your code (as part of your cost-benefit analysis) to see where the data member is touched. “Ah ha,” you say, “it’s only touched in one or two places! This change is a no-brainer!” Of course, if those one or two places are part of inlined functions, the impact of the higher instruction count might be much larger than you think. It’s important to dig a little deeper than a quick grep when making these sorts of changes, to make sure you aren’t psyching yourself out. (And, of course, being able to provide concrete before-and-after measurements is also of paramount importance.)
• Depending on your struct packing size, you might not see any benefit to bitfield optimization unless you reduce the struct’s size below the next lower multiple of the packing size.
• One potential positive effect is that reducing the size of your structs may help with cache utilization – if you can cram more data into the cache, the added cost of accessing the packed data may be offset.

The take-away from this exercise is that it’s important to consider many different factors and effects when optimizing. Most of the real-world cases where I made these optimizations were situations where the struct in question had hundreds, or thousands, of instances, and the tradeoff was definitely a win, but it’s still worth taking a moment to think about all of the consequences of such a change.

The comments on the post are amusing, running the gamut from “completely missing the point” (those who say you should always/never use bitfields – there are many of these comments), to those bringing up even more valid considerations (such as how bitfield layout is implementation-dependent behavior, and how trying to map bitfields to hardware structures is a potential endianness minefield). There’s also one rather insightful observation that made me laugh:

I must say: assembly language has no code of honor. Anything goes so long as it works.

So true.

I saw a post by the author commenting on the fact that getting the game to run on the PS3 would be a "gargantuan effort," and that got me thinking about the reasons why, along with how the game engine works to begin with.

First of all, the time-rewinding mechanism appears to be using input recording rather than state recording. That is, the state of the user’s controller is stored for each frame of the game, and is used to control the player’s character during time rewinds/fast forwards. I base this idea off of the fact that, in the world that uses your "time shadow," the shadow’s actions are based on the player’s controller input, and the replayed actions can be affected by the current state of the world.

Naturally, there is additional information stored to assist with the time mechanics of the game — input alone is not enough to reconstruct the state of the world. Jumping, for example, plays a significant role in the game, and being able to "rewind" a jump requires knowledge about when and where the jump started. (If it were possible to efficiently reconstruct the state of the game up to a certain time, you could avoid storing the "where" part, but I don’t think this is the case.) Additionally, the behavior of AI actors like the bunny and cannons would require some state to be saved in order to properly rewind or replay their actions.

The game runs at a steady 60 FPS. If we assume that replay data is stored at the same frequency, and we assume 128 bytes of replay data per frame (a number that I simply pulled out of a hat that said "reasonable assumptions"), then 15 minutes of gameplay results in about 7 MB of uncompressed replay data. This back-of-the-envelope calculation leads me to believe that the reason that Braid would be difficult to port to the PS3 is not related to utilizing more than 256 MB of RAM. (The Xbox 360′s 512 MB of RAM is unified, whereas the PS3 has a hard separation of 256 MB of main memory and 256 MB of GPU memory.) There should be plenty of memory for the replay data on both platforms, given the relative sparseness of the levels.

With that possibility ruled out, what does that leave? I think it’s related to the game engine (and replay system) architecture, not the amount of data being created. My guess is that the updates performed by the game engine are, by design, fully reversible, in order to minimize the amount of replay data that needs to be stored. Furthermore, in order to allow for really fast rewinding/replaying (up to 8x, while remaining at 60 FPS), the game update tasks are parallelized.

The game state that is touched by these parallel steps, however, is large enough such that it will not fit into the 256K memory on each SPE of the Cell processor. Furthermore, the game update phase is not organized in any way that would permit "windows" of the game data to be sent to (and modified on) the SPEs. This is pretty much the worst-case performance scenario for the PS3 and the Cell — the need to parallelize algorithms that touch a large data set in a random-access manner.

I don’t think this is an insoluble problem, but a solution that would work well on the Cell would likely look drastically different from what you would write if you only had to consider the PC and Xbox 360. And when you consider the size of the programming team that created Braid, it’s easy to make the choice between rearchitecting the code and moving on to the next game by scrapping the notion of a PS3 port.

Please note that all of the above is just speculation — I don’t have any idea if this is actually the case, but I like to speculate about game engine design and implementation, since I’ve had quite a bit of experience with it…

As part of my recent hard drive purge, I had to use a partition manager program to reset a drive to NTFS (from extfs) so I could overwrite it with the tools I had at hand. I downloaded and used the demo of  7Tools Partition Manager, which did the trick. (Strangely, “7Tools” seems to be a brand/front for Paragon Software Group.) However, when I uninstalled the program, it left behind a little surprise which was not to be revealed until a few days later.

I recently started a project to rip my entire audio CD collection (and, at the same time, to disassemble their jewel cases and switch to storing the CDs in binders). I’ll have more to write about this project later, but for the purposes of this article, the germane bit is that the CD ripping program interfaces with the CD drive through the ASPI layer (wnaspi32.dll).

On Friday, I went to rip the next batch of my audio CDs, and Exact Audio Copy (EAC) began behaving oddly. It would correctly read the track information for CDs (and get the track information from FreeDB), but when I went to actually rip it, it would suddenly claim that there was no CD in the drive. No amount of coaxing or tomfoolery would convince it otherwise. Attempts to use the feature autodetect, read mode autodetect, and C2 check would fail. After going through this cycle a couple of times, and with different CDs, I was convinced that there wasn’t anything wrong with the hardware itself, nor EAC.

A random Google search for the symptoms I was seeing produced some hits. These were forum posts, usually for people with dodgy or missing copies of the ASPI driver, so I checked my copy of wnaspi32.dll, discovering, to my surprise, that it was left over from 7Tools Partition Manager. (The “last modified” date, as well as the presence of a digital signature from Paragon Software Group, were what tipped me off. The digital signatures tab can be viewed in the Properties dialog for the file.) I had to reboot in safe mode to delete it, although I’m not sure that was strictly necessary — I may have inadvertently left a program open that was using it.

Naturally, after deleting it, everything was fine. My speculation is that, behind the scenes, one of the two following scenarios took place:

• The demo version of 7Tools Partition Manager had a “nag timer” that would pop up after using the program for a certain period of time, asking you to buy it. The expectation of the programmer was that this screen was to be launched from Partition Manager itself, not from some other program via the ASPI driver, and some sort of problem occurred inside the driver as a result of this false assumption.
• Alternately, the demo would simply stop working after a few days, and included code in the drivers to prevent any attempt to circumvent the time restriction. These restrictions were not visible from other programs using the ASPI layer, except as mysterious errors.

I haven’t kept up with InstallShield lately, but I think it would be really helpful if there was some kind of testing feature that allowed a developer to check what’s left behind after installing a program, running it, and then uninstalling it. (This would also be useful for verifying that user preferences and data which is meant to persist across installations is stored in the correct location.) Making these scenarios easier to test would make it easier for developers to do the right thing, and be conscientious about their “residue.” The closer installers get to the ideal (a cross between the Hippocratic Oath’s “do no harm” and the outdoors philosophy of “leave no trace”), the better the user experience will be.

(By the way, a quick look at their home page is all you need to reinforce the idea that InstallShield is all about squeezing every last dollar out of you. "Entitlement Relationship Management"? "Enterprise Software Compliance"? A picture of some smiling d-bag in a suit behind two rows of computer monitors? Bleagh. Oh, and their corporate rebranding has resulted in a web site with broken download links — try downloading updates for InstallShield 5, and have fun!)

As it so happens, yesterday I was trying to install a value edition copy of Heroes of Might and Magic IV Complete (in German, no less). The autorun program on the CD allows you to launch the installers for all three products on the disc. However, when I clicked to launch an installer, nothing would happen. The Task Manager revealed that setup.exe was running, but didn’t appear to be doing anything — it was hung up. Attaching a debugger to the process revealed that it was deadlocked — not good. (I couldn’t tell what function it was deadlocked in, unfortunately…I didn’t have the debugger set up to use the Windows symbol server.) Checking the properties for setup.exe revealed that it was an installer produced by version 5 of InstallShield.

I spent some time searching online for InstallShield problems, and discovered that older InstallShield versions have problems (lots and lots of problems) when running under Windows XP Service Pack 2. Note that this was divined from user postings, not any sort of official acknowledgment from InstallShieldAcresso. Also, one of those posts states that at least one of those problems is caused by an uninitialized variable in InstallShield — awesome. Very confidence-inspiring. InstallShield’s own updater page even skips over the version that built the installer I was having problems with (InstallShield 5), for some reason — it’s like they don’t even want to acknowledge that it ever existed.

Unfortunately, most of the user threads I linked either didn’t have any resolution, or suggested things that didn’t work for me (turning off the NMIndexingService, for example — it wasn’t even running on my machine). However, one suggestion that did work for me was to install the game by booting into safe mode. Yuck.

Just to make sure that all of the relevant details get picked up by search engines, if you are running an InstallShield installer (setup.exe) under Windows XP service pack 2 or higher, and it hangs or gets stuck, run the installer after booting into safe mode and it should work. This definitely applies to InstallShield 5 installers, and possibly later versions. I haven’t tried uninstalling the game yet, but I imagine that the same remedy will be required for uninstalling it.

Now, after spending two hours figuring this out, I might actually be able to play the game! Yeehaw.

• The installer will allow you to install modules for which you don’t have the prerequisites installed. This wouldn’t be an issue if the PHP interpreter handled this gracefully, but it doesn’t — it simply spits out a bunch of dynamic link errors and self-combusts. I agree with the last comment in that bug report — it may be a case of “pilot error,” but the expectation on the Windows platform is that, barring any warnings or errors that are clearly explained to the user, a complete install should just work.
• The current released version of PHP was completely busted on Windows Server 2003 — it doesn’t read its configuration file correctly. (For the benefit of those who may be Googling for a similar problem, this would result in Drupal, a CMS I was installing, reporting that “Your web server does not appear to support any common database types.”)
• Separately from that, the released version of PHP also didn’t set itself up correctly in IIS as a document handler. For some unfathomable reason, it used 8.3 file names, which have not been necessary in the last 13 years. This would cause the document type handler to fail, and 404s to be returned when accessing a PHP file through IIS.
• The documentation for the mysqli module (as well as the similar-but-different mysql module) recommends adding the PHP directory to the Windows PATH variable, for the express purpose of getting the libmysql DLL bundled with the installer to load correctly. This is terrible advice, in my opinion, because, depending on where the user adds the path, the bundled version may take priority over a more recent libmysql installed by MySQL itself. There are additional DLLs in the PHP directory, too, for SSL support, and the same situation could arise there.

  While the library vendors may not have a side-by-side versioning scheme for their DLLs, it’s still no excuse for advocating a dangerous practice.

For a product that is so widely used, these kinds of problems are really inexcusable, and only contribute to the flaky reputation PHP has.