First off, Happy St. Patrick’s Day.  Go forth and kill a snake.  Or drink a green beer.  Or … something!

Second off, sorry for the lack of updates lately.  I haven’t been feeling well.  I took my wife to the hospital for tests a while back and, of course, got sick.  (Because hospitals aren’t exactly places of congregation of the healthy.)  I got “well” in that I kicked the disease quickly enough.  But because I had to get some time-critical work done, I hadn’t been taking days off to rest, so it was just a struggle day-in, day-out, with weekends barely just giving me enough time to keep going.  I literally could pull myself together just enough for work.  I didn’t have enough left in me for blogging.  So anywhen, sometime in the future I’ll go through my notes and backdate posts, as I tend to do.  Now that I can take time to recover.

Third, the main point of this blog.  So I’ve been playing S.T.A.L.K.E.R.: Clear Sky lately.  I recently-ish bought it on Steam for dirt cheap.  Even though I hate, detest, abhor Steam, and almost always make sure to buy the disk instead of the music, game, or movie because of rights issues, this one time I caved and “downloaded” the game.  From Steam no less.  I’m a hypocrite of convenience.  I hate myself.  But I’ll get over it.

Anyway, so I’ve been playing S.T.A.L.K.E.R.: Clear Sky and enjoying it.  I think it might actually even rate as one of my favorite video games.  I wish I’d bought it sooner.  It’s a lot better than the first.  Not just in being more of a challenge, or in being able to do simple basic things like repair an item, but it just all plays / feels / works so much better.  I especially like the upgrades and the item maintenance.  (Even if I still don’t understand why you can’t do stupid things like permanently tack weld a scope to a Viper 5.)  Except for, you know, the bugs.  Those darn little things that forced me to restart the game because the main plot device broke and I was eternally stuck.  Nerf!  Oh well.

But so the thing is, just this weekend the darn game kept crashing.  Not even crashing out of the game, but in a weird in-game-ish stuck thing where it was like the underlying engine was still running, but the 3D graphics portion had crashed.  So there was no longer a user interface.  It just became a black screen with a mouse cursor.  That’s it.  Sometimes it’d be in the middle of playing.  Sometimes just in the menu trying to load a game or change 3D settings.  (Trying to debug the crashing.)  And sometimes even while the advertisement logo movies played while starting up the game!  It was absurd!

After much gnashing of teeth, trying offline mode in Steam, even trying to set the CD key in multi-player even though the single-player game doesn’t really use the CD key, I was going crazy.

Until I thought about it.  I’d just seen Windows do an update.  Could that somehow be it?

Yep!

I know, I know, I really should turn off the automatic updates because Microsoft does throw  out some real turds.  And this was one of them.

Update KB2670838 happens to update Windows 7 with some changes to Direct3D, DirectDraw, etc.  Yup.  Damn.  And sure enough, Microsoft “fixed” Direct 3D just enough to make S.T.A.L.K.E.R.: Clear Sky incredibly completely totally bonkers unstable.

Well nerds to that!

So I uninstalled Windows Update KB 2670838 and sure enough, S.T.A.L.K.E.R.: Clear Sky is nice and stable again.  Imagine that.

So if you still play S.T.A.L.K.E.R.: Clear Sky, I hope you find this blog because chances are, right now you’re not playing it because of this bug in Microsoft’s “fix” to whatever problems they thought they had.

Or, honestly, goodness how many other video games are crashing right now because of KB2670838.  Heck, programs in general.  Movie players.  Who knows what else uses those engines?  I don’t know what Microshaft is thinking about releasing that PoS, but that update was most definitely neither thoroughly tested nor kosher. 

So be warned!  If you play video games, pay close attention to how often things crash after you install KB 2670838.  You just may find yourself uninstalling that particular Microsoft update!

We already kicked off one heck of a security-minded year with a miasma of bad security in January. Now Microsoft is keeping that tradition alive just in time for Valentine’s Day with a Patch Tuesday chock full o’ love.

That’s right, this Patch Tuesday covers a lucky 13 security bulletins that close up a whopping 57 security holes. If you weren’t feeling insecure before, just think of how all of those vulnerabilities compromised your PCs. On the one hand it’s good that Microsoft is fixing these things. On the other hand, WTF?! 57?

From Internet Explorer (of course) to ActiveX DLLs (again, rather expected) to Windows kernel win32k.sys (doh!) this Patch Tuesday is a whopper that you really shouldn’t miss. Of course if you have automatic updates turned on, then you’re not going to miss it. Then again, if you find that sometimes the patch is worse than the bug, this is one you’ll want to keep apprised of.

Meanwhile Adobe Flash also recently released a patch and Java, well, those Java patches have been pretty half-baked and that mess is still trying to be sorted out while Oracle tries to convince the world to not just drop Java entirely from the web browser.

Security, gotta love it!

What does it mean when the president of the Windows division leaves just at what should be the height of his career, releasing Windows 8? Instead of celebrating, departing seems like a rather … unusual … move to make. Though everyone seems to be rather cordial about it. Ballmer didn’t throw a chair at Sinofsky. (That we know of.) No one seems to have heard him threatening to kill anyone because Sinofsky is leaving … Sinofsky included. So … what gives?

Well … there’s not a lot of official, and what it is, is pretty … fishy.  Strangely, the rumors seem more inclined to be the real story.  Scuttlebutt seems to indicate that while Sinofsky did the herculean task of wrestling the Windows OS back on track, he may have done so by burning a few bridges, making some enemies, and engendering an allegedly hostile work environment. To put it nicely, it sounds like he really didn’t play well with others. And specifically of importance to Microsoft, those “others” were Windows Phone and Xbox development teams. At a time when everyone – Microsoft included – is trying to create a more homogenous feel between their disparate products (for better or worse), it’s a bad time to be throwing your Windows OS weight around, even if it is to keep your product on schedule like your job description entails. Perhaps this is part of the reason why Windows 8 feels so … schizophrenic. (Though certainly rushing it to market so soon on the heels of Windows 7 has something to do with that too.) And why Windows RT is so … useless.

To keep a time table, hard decisions have to be made, for sure. And having made them, and kept the schedule, perhaps Sinofsky has found himself a rebel without a cause. OS development timetables are back on track. Job well done. But now Windows needs a whole new direction, one that perhaps Sinofsky just either isn’t suited for, or has no interest in.

Regardless of why he took his leave, certainly it’s an indication that Microsoft is ready to get back to usual, letting timetables slip as feature lists grow … whether they’re actually needed yet or not, or even useful or not. In other words, back to bloatware.  Standard Operating Procedure at Microsoft is back in full swing, by the look of things.

Which for Windows is, well, if not “good”, at least what the world expects.

But for Windows 8 RT … well, too bad, so sad. Your successor is probably going to be the heaviest “lightweight” OS ever.

And I can’t wait to see what Windows 9 will be!

Honestly, I can’t say whether or not Sinofsky leaving Microsoft makes me happy or sad. The one thing that I can say however is that his leaving doesn’t leave me indifferent. With a company like Microsoft, and a product like Windows, sometimes a little “no” goes a long way. When you can’t have everything and eat it too, you need someone to make hard decisions, to be disliked. Fluffy huggy love-ins are great fun, but don’t get products out the door. It should be an interesting future for Microsoft. And as for Sinofsky, can one remain content on just a hill after having climbed to the top of a mountain?

One thing is for certain though, with the Microsoft Surface cover woes, the Microsoft Skype security gaff of the century, and Sinofsky saying, “TTFN!” … it’s definitely not a good Christmas runup in Redmond.

Chances are that if this matters to you, it’s something that you’ve already gone through. After all, anyone still using Numeric in Python in this day and age is working with an incredibly outdated environment. Still, sometimes it happens. Sometimes in business settings validating a new environment is not such an easy thing to do as it is in academic or hobby worlds. So just in case, here are my experiences of upgrading from Numeric to NumPy:

Tip 1 ) Replace Numeric with NumPy’s Old Numeric

The first trick is that NumPy contains most of what you need already in the numpy.oldnumeric module. This saves an awful lot of effort as you don’t have to rewrite random portions of some million lines of code. The incredible vast majority of the work involved is one simple Pythonic twist:

    import numpy.oldnumeric as Numeric

And if you’re concerned about remaining backward compatible with your old environment then, you can even add an exception handler to choose which is the right one to use:

    try:
        import Numeric
    except ImportError:
        import numpy.oldnumeric as Numeric

Now, similarly, if you used some of the modules in Numeric, such as LinearAlgebra, it’s still almost as simple. You can do:

    from numpy.oldnumeric import linear_algebra as LinearAlgebra

Or, again, if you need backward compatibility:

    try:
        import Numeric
        import LinearAlgebra
    except ImportError:
        import numpy.oldnumeric as Numeric
        from numpy.oldnumeric import linear_algebra as LinearAlgebra

Tip 2 ) Fix minor type inconsistencies

There are some differences between Numeric and NumPy’s Old Numeric, and those are primarily in how Old Numeric doesn’t handle types in quite the same way. The biggest is character versus string arrays and floating point arrays. Now, say you used a string as data for your array. In Numeric this results in an array of a character type, AKA Numeric.Character or ‘c’ with a size of the number of characters in your string. But in NumPy this results in an array of a string type with a size of 1! That’s not very compatible. The solution? Just specify the type when constructing your array. So instead of Numeric.array(“datadatadata”) use Numeric.array(“datadatadata”, Numeric.Character). Yep, that one is really that simple.

Slightly less simple, though you may not even realize it is happening, is similarly related. Say you had a Numeric array of a float type using “f” to define it. Something like Numeric.array([1., 2., 3.], “f”). In Numeric this “f” type specifier results in a match to Float64. Something you may or may not have expected. This is because Numeric has an interesting string matching algorithm. In Numeric you can have a Float0, a Float8, a Float16, a Float32, a Float64, or even a Float128. Each could be specified in a string if you desired instead of using Numeric’s constants. Which means that if you specify a string of “float”, it leaves Numeric to try and decide which length float you want. And Numeric, thinking smart, matches the default float type used by Python, so it’s a nice match to your data. Which, by the way, is a Float64, otherwise known as a double. And so in the above example, if you specify type, “f” to numeric, yep, you guessed it, you end up with an array of Float64.

Where things get tricky is that NumPy doesn’t have the same float types like Numeric does. It just has float32 and float64, AKA, “f” and “d” respectively. So the same “f” that gave you a Float64 in Numeric will give you a Float32 in NumPy!

Now this might not be much of a problem to you if you’re sticking with Python-only code. Then again, with less accuracy it might. But where it can really kick your asterisk is if you did something silly like wrote a compiled module (Who would do a silly thing like that?) and pass it your array as one of the arguments. If you added type checking into your compiled code, if for no other reason than just good coding practices, this can end up throwing you for a loop when suddenly your arrays are no longer matching a Float64 type! The fix, of course, is simply to use the more descriptive Numeric.Float64 type instead of “f”. Or if you’re lazy and that’s too much typing, at least switch to “d” which both Numeric and numpy.oldnumeric will interpret as Float64.

Assuredly, if you’re really into variable typing, chances are there are other places where NumPy’s Old Numeric did not match Numeric as closely as perhaps it should have. For example Numpy.Int16 is “s”, where as numpy.oldnumeric.Int16 is “h”. I’m not sure what affect that has on anything, having not used it. But I noticed it. Goodness knows what else there may be too.  Your best bet is to first not use strings to define your types, but to go to the defined constants, and second to test test test.

Tip 3 ) Fix Numeric / NumPy inconsistencies

Besides just minor type inconsistencies when creating arrays are the bigger inconsistencies, namely in the method of types. In Numeric type constants are characters. Your Numeric.Int32 is literally the character ‘i’. Whereas in NumPy a dtype class was created for handling types. You can construct a numpy.dtype(‘i’), but it’s nothing as simple as just a character. But NumPy is even worse than that in practice, because there’s also a ‘type’ type used in NumPy, and that’s what the numpy.int32 constant is a type of, for example. It’s not the same as a dtype. And as you can see, it’s already getting messy when it comes to choosing your type constants.

Oh, wait, it gets worse.

Because in Numeric an array has a .typecode() that defines what type the array is of. And because Numeric just uses characters as type definitions, it simply returns a character of the type. It’s easy and straight forward.

NumPy ndarrays have no such method. Oh, they do have a type specifier built in. But it’s not a method named .typecode(). It’s a variable named .dtype. And it is a numpy.dtype class instance. But the NumPy type constants are ‘type’, not dtype. Yes, NumPy just by itself is messy. But now add in that all of your Numeric code is looking to compare “if array.typecode() == ‘i’:” and you just walked into a whole mess of incompatibility.

If you don’t care at all about backward compatibility, then you’re fairly well set. You can just replace array.typecode() with array.dtype.char. Yes, that’s right, the dtype class has a char member variable that (mostly, except for differences outlined in tip 2) you can compare against. If you’re more brave you can try even replacing that character with a constant so “if array.dtype == numpy.int32:” is a little more descriptive and cleaner. If you’re just moving forward.

However, for those who need to continue to support both environments with Numeric and environments where NumPy has replaced Numeric, using numpy.oldnumeric or not, you’ve walked into a world of hurt where you’re best making your own module of helper functions, because this gross incompatibility will make doing a simple type comparison in a backward-compatible way very messy, especially if your old environment doesn’t have any version of NumPy to help bridge the gap.

And this is especially true because NumPy is compiled code, so you can’t just sneakily add a .typecode() method to the ndarray class like you normally could in Python. I suppose you could try to do so to the base code and recompile all of NumPy for it. But the bigger question remains why in the world it wasn’t put there in the first place just to remain backward compatible? Welcome to a minor headache. But still, all considered, a pretty small problem compared to what you could be going through right now.

Tip 4 ) Want C++ NumPy? How about a Boost?

If you’ve got that nasty compiled code mixed in with your Python, you just may be using C++, and that means you might even be using Boost. (That’s what I’ve been using anyway.) If so, you might have been disappointed to find that Boost has no native NumPy support built in. And that the authors of NumPy seem to have no interest whatsoever in Boost, preferring Fortran to C++. I guess for a numerical package, I can’t really blame them for that, as that is Fortran’s forte. But it can be awfully inconvenient to the other 99.99% of the world who have forgotten that Fortran is even a programming language. (If they ever even knew it.) Well, cry not, for an unofficial Boost layer for NumPy exists. Enter a GitHub project for ndarray.

Mostly, it’s pretty straight forward and you’ll barely have to change your C++ code at all to use it. One important difference however is in your module export you’ll have to add a boost::numpy::initialize() call immediately at the top so that Boost knows how to template NumPy in order to match your Python to your C++.

And if you’re maintaining backward compatibility, well, it’s not the cleanest thing to do with C++ being not quite as flexible about that as Python. I think the best way to go about it is to turn a function into a template function on the cpp side, double up on an overloaded declaration on the h side, and then in the cpp side again have the overloaded implementations just call the template function with the array type. But wait, the pain isn’t over yet, because then you have to change your export definitions to specify which of the overloaded methods to use, which gets a wee bit messy. Or if you don’t want that mess, simply append a _NUMERIC and _NUMPY to the ends of your declarations to keep them separate instead of overloading. That makes things a lot easier in the export, but doesn’t look quite as smooth.

But then there’s one more monkey wrench in the works if you use Microsoft Visual C++, in that the Boost.NumPy layer needs some tweaking to work in Microsoft Land because M$ has yet to implement templates correctly. So you’ll have to add a cxxflags=/DBOOST_ALL_NO_LIB to your Boost build or else you’ll get multiply defined functions in your libraries because Microsoft still isn’t smart enough to weed out duplicate definitions when using templates, so Boost.NumPy ends up fighting with Boost.Python on MSVC. Doh!

Oh, wait, it actually gets worse because I almost forgot that you won’t even get that far in the first place. Because Microsoft also hasn’t implemented variable length arrays yet. So you’ll have to fix a couple of places in the Boost.NumPy code where they do Py_intptr_t dims[nd]; to become Py_intptr_t* dims = new Py_intptr_t[nd]; And, of course, not immediately return because if you don’t sneak that delete[] dims; line in there you’re going to have memory leaks. Yay!

The problem being that, well, sadly, not many Python users are on Windows, so the Boost.NumPy authors of that GitHub project just haven’t tested it on MSVC, apparently. But it all can be made to work. Honest.

You can even go the extra step and add the Boost.NumPy code straight into the Boost codebase locally before you build it. I mean you’re going to build it anyway, right? Might as well. To get it to work with the Boost build system I had to rename the src directory to build so that bjam could find the Jamfile in there. No biggie.

Of course if you do try to use the Boost.NumPy on Windows, don’t even bother trying to use SCons.  It’s not that SCons won’t work on Windows, because it will.  That’s the point.  It’s that Boost.NumPy’s SCons script won’t.  Why doesn’t it worn on Windows?  Well, you can kind of put that on Python, and kind of on the Boost.NumPy authors.  They chose to use the one part of distutils that doesn’t have full functionality on Windows: distutils.sysconfig.  Now, looking at the latest documentation on Python, you wouldn’t even know that there’s a problem with distutils.sysconfig on Windows.  But if you look at the base sysconfig Python module that distutils (strangely) gets distutils.sysconfig from (why the same Python module is basically duplicated is beyond me) you find this almost non-existent warning in the sysconfig documentation about configuration variables, “Notice that on Windows, it’s a much smaller set.“  What that almost impossible to find warning means is basically that on Windows pretty much None of the configuration variables exist, so sysutils.get_config_var and sysutils.get_config_vars are pretty much useless on Windows.  Thereby causing the Boost.NumPy SCons script to fail horribly.  So just don’t even try.  Use Boost’s bjam instead on Windows.

Conclusion

So, it’s “just” that easy. Uh-huh. In that there’s pretty much nada in the way of documentation, you can imagine it took me a while to sort some of this out. Hence why I’m putting it on Ye Olde Interwebs now, so that hopefully if you’re stuck doing the same thing you won’t waste nearly so much time coming up with answers. Numeric may be dead, but with NumPy’s Old Numeric your Python code doesn’t need to go through a massive rewrite. A bit more work though if you had C++ code too, but it can be done, and almost as cleanly.

Okay, okay, so maybe the title isn’t exactly a fair statement.  Maybe Nokia isn’t really “washing their hands” of Qt.  I mean it’s not like Nokia has been incompetently juggling an OS to replace Symbian for years now…

Wait.  Maemo, Meego, Qt, Microsoft Windows Phone … okay.  So yeah, they kind of have.

Huh.  But so it’s not like Nokia got themselves into a politically compromising situation by wholeheartedly committing to a draconian platform like Microsoft Windows after having spent years on Linux-based …

Honestly, no, I couldn’t keep a straight face there either.

So here’s the poop:  Nokia is desperate to salvage the mess that they’ve made of their once-top position.  They kept investing in brave new worlds in an effort to remain bleeding edge, but got caught up in the “shiny shiny” of Information Technology, and let their ADD/ADHD get the better of them.  They had a perfectly good Linux-based Symbian replacement in Maemo, but they let Intel sweet talk them into a joint venture to merge Inte’s Moblin with Nokia’s Maemo to create Meego.  But before the fruits of Maemo or Meego could even be realized, Nokia already saw a new shiny in the theoretically more robust Qt.  Which left Intel with a dead fish and Nokia once again with a lot more work to become relevant again in the smartphone industry.

So Nokia bought Qt from Trolltech, who admittedly seem to have kind of sold themselves out and let their work on Qt be a tad … “rushed” to get themselves bought out.  It was a desktop PC environment that they started wedging mobile phone interfaces into.  Enter Nokia, the new manhandlers of Qt, who concentrated that work even more heavily on mobile phones.  And who really seemed to have no interest whatsoever in supporting their own product.  (As a Qt user for many years now, from long before Nokia bought Qt from Trolltech, I saw firsthand how much of a decline Qt was under Nokia’s hand.)  To the point where the SDKs no longer even included the Qt source code and could barely even be comprehended, and often had glaring flaws like being unable to build for static library use.  Atrocities against desktop Qt were committed under Nokia’s ownership, in the name of making Qt more mobile-relevant.

But, just like Nokia has done before, instead of sticking to a platform long enough to make it work, they saw an easy out in the form of Microsoft Windows Phone, and they took it.  (And presumably a lot of cash to make that commitment.)

Now, having taken said cash, and becoming most thoroughly tied to Microsoft, whatever was Nokia to do with the heart of many Linux GUIs, Qt?  Talk about sleeping with the enemy!

So enter Digia, Qt’s new saviors.  Now Digia had already been taking over parts of Qt from Nokia already anyway, because frankly, Nokia was only interested in using Qt for phones.  Which is fair to Nokia, but completely and totally unfair to Qt, it being first and foremost a desktop multiplatform GUI and more set of libraries.  Digia had been selling their own commercial Qt licenses and support and, sadly, doing a better job of showing interest in the heart of Qt than Nokia ever was.  So when Nokia put their future into Microsoft’s hands, it’s no surprise at all that behind-closed-doors talks of selling Qt to Digia took place.  What is surprising is that it actually took this long after Nokia’s commitment to all things Microsoft to actually finalize their sale of Qt to anyone else.

So what now?

Well, nothing bad, as far as I can figure.  At least not for Qt.  (Nokia has already fallen way behind in the smartphone market, and being tied to Microsoft Windows Phone in the Apple/Google iPhone/Android game seems fairly unlikely to dig themselves out of their hole no matter how much initial financial improvement from MS it may have made Nokia to commit to that.)

Digia seems to honestly and legitimately be interested in all of Qt.  And as a business, to make actual money on the product of, not just as the middleware to a line of gadgets.  Digia knows Qt already, so they’re not blindly investing here, and there hopefully won’t be much or any stumbling with a new learning curve.  And even better are the canned quotes from Digia saying positive things like, “Digia’s targeted R&D investments will bring back focus on Qt’s desktop and embedded platform support, while widening the support for mobile operating systems.“  (Bold was my emphasis.)  There’s Windows 8 for desktops that Qt really needs to work on incorporating.  Then there’s all of those mobile platforms that Nokia had no interest in supporting when they bought Qt, like iOS, Android, and yes, even Windows Phone 7 and 8.

There’s a lot for Qt to catch up on.  And a lot of direction that was lost under Nokia’s ownership.  Digia seems honestly interested in fixing that.  So “Huzzah!“, I say.  It’ll be nice to see Qt returned to it’s former glory as the unquestionable leader of truly multiplatform desktop GUI technologies.

Now if only someone could convince Digia to take Qt Python support seriously, I’d be grinning ear to ear instead of just broadly smiling.