First off, Happy St. Patrick’s Day.  Go forth and kill a snake.  Or drink a green beer.  Or … something!

Second off, sorry for the lack of updates lately.  I haven’t been feeling well.  I took my wife to the hospital for tests a while back and, of course, got sick.  (Because hospitals aren’t exactly places of congregation of the healthy.)  I got “well” in that I kicked the disease quickly enough.  But because I had to get some time-critical work done, I hadn’t been taking days off to rest, so it was just a struggle day-in, day-out, with weekends barely just giving me enough time to keep going.  I literally could pull myself together just enough for work.  I didn’t have enough left in me for blogging.  So anywhen, sometime in the future I’ll go through my notes and backdate posts, as I tend to do.  Now that I can take time to recover.

Third, the main point of this blog.  So I’ve been playing S.T.A.L.K.E.R.: Clear Sky lately.  I recently-ish bought it on Steam for dirt cheap.  Even though I hate, detest, abhor Steam, and almost always make sure to buy the disk instead of the music, game, or movie because of rights issues, this one time I caved and “downloaded” the game.  From Steam no less.  I’m a hypocrite of convenience.  I hate myself.  But I’ll get over it.

Anyway, so I’ve been playing S.T.A.L.K.E.R.: Clear Sky and enjoying it.  I think it might actually even rate as one of my favorite video games.  I wish I’d bought it sooner.  It’s a lot better than the first.  Not just in being more of a challenge, or in being able to do simple basic things like repair an item, but it just all plays / feels / works so much better.  I especially like the upgrades and the item maintenance.  (Even if I still don’t understand why you can’t do stupid things like permanently tack weld a scope to a Viper 5.)  Except for, you know, the bugs.  Those darn little things that forced me to restart the game because the main plot device broke and I was eternally stuck.  Nerf!  Oh well.

But so the thing is, just this weekend the darn game kept crashing.  Not even crashing out of the game, but in a weird in-game-ish stuck thing where it was like the underlying engine was still running, but the 3D graphics portion had crashed.  So there was no longer a user interface.  It just became a black screen with a mouse cursor.  That’s it.  Sometimes it’d be in the middle of playing.  Sometimes just in the menu trying to load a game or change 3D settings.  (Trying to debug the crashing.)  And sometimes even while the advertisement logo movies played while starting up the game!  It was absurd!

After much gnashing of teeth, trying offline mode in Steam, even trying to set the CD key in multi-player even though the single-player game doesn’t really use the CD key, I was going crazy.

Until I thought about it.  I’d just seen Windows do an update.  Could that somehow be it?

Yep!

I know, I know, I really should turn off the automatic updates because Microsoft does throw  out some real turds.  And this was one of them.

Update KB2670838 happens to update Windows 7 with some changes to Direct3D, DirectDraw, etc.  Yup.  Damn.  And sure enough, Microsoft “fixed” Direct 3D just enough to make S.T.A.L.K.E.R.: Clear Sky incredibly completely totally bonkers unstable.

Well nerds to that!

So I uninstalled Windows Update KB 2670838 and sure enough, S.T.A.L.K.E.R.: Clear Sky is nice and stable again.  Imagine that.

So if you still play S.T.A.L.K.E.R.: Clear Sky, I hope you find this blog because chances are, right now you’re not playing it because of this bug in Microsoft’s “fix” to whatever problems they thought they had.

Or, honestly, goodness how many other video games are crashing right now because of KB2670838.  Heck, programs in general.  Movie players.  Who knows what else uses those engines?  I don’t know what Microshaft is thinking about releasing that PoS, but that update was most definitely neither thoroughly tested nor kosher. 

So be warned!  If you play video games, pay close attention to how often things crash after you install KB 2670838.  You just may find yourself uninstalling that particular Microsoft update!

We already kicked off one heck of a security-minded year with a miasma of bad security in January. Now Microsoft is keeping that tradition alive just in time for Valentine’s Day with a Patch Tuesday chock full o’ love.

That’s right, this Patch Tuesday covers a lucky 13 security bulletins that close up a whopping 57 security holes. If you weren’t feeling insecure before, just think of how all of those vulnerabilities compromised your PCs. On the one hand it’s good that Microsoft is fixing these things. On the other hand, WTF?! 57?

From Internet Explorer (of course) to ActiveX DLLs (again, rather expected) to Windows kernel win32k.sys (doh!) this Patch Tuesday is a whopper that you really shouldn’t miss. Of course if you have automatic updates turned on, then you’re not going to miss it. Then again, if you find that sometimes the patch is worse than the bug, this is one you’ll want to keep apprised of.

Meanwhile Adobe Flash also recently released a patch and Java, well, those Java patches have been pretty half-baked and that mess is still trying to be sorted out while Oracle tries to convince the world to not just drop Java entirely from the web browser.

Security, gotta love it!

For those of you who actually care, and few are those indeed, Intel just announced yet another blow to Itanium customers.

Of course, if you are an Itanium customer, then you probably aren’t in the least bit surprised.

Itanium has pretty much always been a rocky road for Intel. The whole shebang of Itanium started out with some lackluster performance and some highly questionable future purpose.  Itanium was either to be the golden child of computing, or a whole lot of smoke blown up your skirt.  And especially once AMD reminded people that they do servers too, the road of Itanium looked awfully rocky. Since then updates to the Itanium line have often been delayed (sometimes horrendously so) and have rarely met Intel’s own original expectations, let alone any customers’.

Still, for Really Big Iron … what else can you do? Itanium customers have little choice but to hold on for the ride and hope for the best, if they can’t be bothered to think a little smaller.

Well, turns out that Intel’s recent Kittson update is no different.

First on the chopping block? Gone is the long-rumored miraculous Xeon-Itanium uber-socket. Yep. That rumor/feature has been churning practically since Itanium first reared its ugly head into the beyond-x86 world. But people really thought it would actually finally happen with Tukwila. It didn’t. Then Poulson. It didn’t. Then Kittson. Now Intel says that’s a big NEGATORY as well.

Sorry!

(For those wanting to sound like IT experts, guess what you have a very high chance of predicting not happening for the next iteration of Itanium also…  And the one after that…  And the one after that…)

But it’s not all bad, right? Okay, so Kittson loses the socket panacea, but we all expected that anyway because honestly, what numbskull would want the world’s most outrageous socket for a cute “little” (by Itanium standards) Xeon anyway?  At least Kittson will still be fabbed with Intel’s shiny energy-saving 22nm Tri-Gate process, right?

Umm … no.

That’s gone too.

And, as a result, it remains to even be postulated what extra speed, cache, or even core changes whatsoever can be expected to be found in Kittson now.

For all practical purposes, Kittson is looking to just be … err … well … just another Poulson.

For those of you wondering just why Intel has let so much slip from Itanium’s grasp this time around … well … it’s pretty obvious, really.  If you’re wondering, you must have had your head in the sand or something.

Intel has seen the future (or the present, really) and the future is ARM eating up Intel’s server dominance. The future isn’t BIG Iron. It’s small iron. Cars are doing it. Now computer processors are too. Efficiency is the new black.  Smaller is, well, not necessarily better, but an emerging market at the very least, if not the buzzword of the day.  Green is where the green is. There’s money to be made in them there ARM chips. The only reason Intel hasn’t keeled over dead from a mass exodus to ARM in the server market is because ARM just hasn’t offered 64-bits … yet. 32-bits isn’t enough anymore, and quasi-there forty-somethings are half-asterisked solutions at best. But that’s all a-changin’. The answer my friend is blowing in the wind. On the horizon is true 64-bitness. It’s no longer a question of if, but when.

To fight that, Intel is pushing hard to reduce the energy consumption of their x86 chips. Not just because then they’ll (they hope) gain back all of that phone and tablet processor market that they already lost, but also because the other barn, the one that hasn’t had all the cows wander out after the door was left open, is at the eleventh hour of opening as well: the microserver.

And to be fair, there is something to be said for the increased agility of a lot of small chips … for certain uses.

Some not exactly stunning attempts have been made to push ARM into servers. (Personally, I rather liked the Lego-block cased Raspberry Pi “supercomputer”.) Intel still has a chance to not lose the microserver battle by bringing both Atom and their usual x86 offerings into ARM-level power consumption.

And that, of course, means that their shiny new 22nm Tri-Gate fab and related resources have to be applied to the battles that they can still win, not to Itanium, which, let’s face it, has pretty much been a lost cause since its very inception.

Intel announced this intention already. And to their credit, Atom just may even bite back a chunk out of ARM in phones and tablets yet if Intel really does push their SoC Atoms onto their most energy-efficient process to give the world a true ARM alternative.  Especially when it can also run Windows.  The real Windows.  Not that RT malarkey.

So it should come as no surprise then.

But it’s not always so easy to put two and two together. Or perhaps denial isn’t just a river in Egypt. Yes folks, because Intel is concentrating on low-power Atoms, Cores, and Xeons to save their server bacon from an ARM farm, it will leave Itanium’s process upgrade to be performed in another cycle.

But then, as Itanium customers, you really should be used to disappointment by now.

So what to do while you wait? Why not try to visualize whirled peas?

And if you’re HP? Heh heh. Sorry, Charlie! Only the best-tasting server iron gets to be Intel-Kist. Didn’t you hear? Efficient and agile is in. Big and bulky is out. Green is the new black. Sorry if that’s going to make you see red. Such is the way Ye Olde Cookie crumbleth after all. That writing has been on the wall for so long that even vagrants are strumming tunes to it on the street corners.

But then, as an Itanium customer, you really should already know that too.

Honestly, I’m kind of surprised that Itanium has even lasted this long.  It’s a product in search of a market.  It’s an ideal without a need.

But still, you’d have to be pretty daft to have continued to expect it to have any kind of decent future at t his point. It’s been living on borrowed time for longer than time has been able to be borrowed. You just might want to kind of sort of possibly think of potentially transitioning over to a different platform sometime. Maybe.

Eventually.

Just a thought.

So you’re hip. You’ve got your very own Raspberry Pi, right? And you’ve probably spent at least twice what you paid for your Raspberry Pi just to get it running.  Because as cheap as it is, keyboards, mice, etc. all add up.

Well, you’re already behind the fad, because those of you on the B-List just got gamed by the As.  You’ve overpaid for your Pi.  You’ve got too much hardware.

That’s right. When Raspberry Pi was first launched, it was launched as the much more usable B model, with that lovely ethernet port, 512MB of RAM, and two USB ports.

Which we all know, you don’t really need, right?

Well, the trimmer, slimmer, more fit (it’s the same size actually) Raspberry Pi A model has finally been released. It has only 256MB of RAM, no ethernet, and just one measly USB port. But it’s cheaper. And without all of that superfluous (allegedly) hardware, Model A consumes around just one-third of the power of Model B.

Which makes your Raspberry Pi A-game oriented at low-power uses such as, powered by the sun perhaps. Or maybe a hydrogen fuel cell? Or … who knows? The Pi’s the limit. Chances are though that your first USB peripheral is going to be … a network adapter. Probably some compound Wi-Fi / Bluetooth gizmo so that you can hook up a wireless keyboard and mouse as well as access data from World + Dog. (And then yank it out once whatever you’re developing is done?)

But the point is, we finally have the much-promised Raspberry Pi Lite.

Microsoft Surface – Finally Going Pro

For those of you who fancied the concept of a laptop that converts into a tablet … by means of ripping the keyboard off … but were too afraid of Windows RT on ARM to touch the Microsoft Surface, well you’re about to finally get your hands on the fatter glutton x86 flavor with the rotund resource-hungry full version of Windows 8. Yes, that’s right, in just a couple of days Microsoft is going to unleash the Microsoft Surface Pro onto the world.

Umm …

Yay?

Seriously folks … I don’t get it. Congrats, it’s finally a Microsoft Surface that you can actually (almost) use? Well, I guess there is that. But why anyone ever thought that Windows RT was in any way useful to begin with is beyond me. Congrats, it’s Windows … only without the ability to run your programs.  Because even if everyone were to suddenly recompile all of their applications to run on ARM, which they won’t, Windows RT still doesn’t have that lovely part of the Windows kernel that lets applications run.  It’s just Windows Phone … on a tablet.  So they called it Windows RT instead of Windows Phone because if they told you that their tablet ran Windows Phone you’d probably expect that it could actually be a phone?  I guess?  I don’t know. The whole concept of Windows RT is just plain dumb. It’s Windows CE. It’s Windows Phone. It’s Windows RT. It’s not Windows.

Well, that horsehockey is finally done and gone as the Microsoft Surface Pro is everything it should have been with a full x86 CPU running a full version of Windows 8 that runs real applications and not just “apps”. Huzzah!  All your software needs are met on the Pro version of the tablet.

(As if Microsoft couldn’t just put an Intel Atom – or AMD Fusion – into a Surface tablet to have run a full Windows 8 on, just with lower hardware specs than the Pro.)

But at the end of the day, the Microsoft Surface Pro is still just a convertible laptop where the keyboard is removable instead of swiveling into a concealed position under the monitor. And it’s still running Windows 8, not Windows 7. So you still probably don’t actually want the thing for actual everyday use, let alone to do real work on.

But at least soon it’ll be an option. Finally, a Surface tablet that you can almost use.

And if Microsoft hasn’t sold you on the benefits of Windows 8, well, you wouldn’t be the only one.  When was the last time that you saw a Windows commercial?  No, not a Surface tablet commercial, I mean an actual, “This is Windows 8.  It’s better than Windows 7 because…” commercial.  Advert  in a magazine?  Ad on the internet?  Any advertisement?  Anywhere?

For that matter, why is Windows 8 better than Windows 7?  Oh, right, because it’s harder to use, especially if you use a mouse and keyboard like the incredible vast majority of the world does.  Good.  Got it.  Sorted.  Well gee, I’m sold!  How about you?

One of the lovely things about Python is that there are so many free libraries to choose from. But sometimes that’s a bad thing, because people love to reinvent the wheel, thinking that they can make it somehow rounder and more efficient. This of course results in a lot of dead code and modules that haven’t been updated since the Stone Age.

Recently at work I found myself looking at a new package to replace one really old dead one: PyXML. We’ve got a good chunk of code that reads and writes data to an XML file basically as a flat-file database for when we (or our customers) are not using PostgreSQL, Oracle, etc.

Normally that wouldn’t be such a big deal, except that we have one requirement: XPath.

There are an awful lot of good XML parsers out there in the world. CPython now even comes with one build in: ElementTree. …And cElementTree (the compiled version of ElementTree) which is the same API, just a whole lot faster – unless you’re using PyPy, which we’re not – but I digress.

The problem with ElementTree however is that it doesn’t fully support XPath. In fact, it barely does at all. It’d be nice if it did, but it doesn’t much, so there you are.

The following is a run-down of my research into a few other Python libraries that do fully support XPath XML coding standards. I wish I could release the benchmark code that my performance evaluations are based on, but the benchmark code is based on real use cases of proprietary code. Meaning that they also don’t necessarily represent the on-paper perfect-world performance of these libraries, but more useful real-world use-cases. These were run under CPython 2.7.3 on a Windows 7 64-bit Intel Xeon workstation.

PyXML: The Baseline

You’d think that PyXML, having last been updated in 2004, would be long dead. And it is. But if you don’t mind fixing some minor things in this all-Python XML library, it actually does still work. You just have to find-and-replace the two places where “as” is used as a variable name, since Python now protects keywords with a vengeance. Simple changing the name to “_as” is enough to fix the problem, and then you can continue to use the long-dead PyXML on Python 2.7.3. Since this is the library that our code used in the past for Python XPath XML support, it’s what I used as a baseline to compare other libraries to. We also used Python’s minidom with PyXML, which is not exactly known for speed…

FourThought 4Suite: 2X

This is another dead project, having last been updated in 2006 as far as I can tell. It’s written by the same company that wrote most of what is in PyXML. (According to Wikipedia, it’s also the same company that brought you PowerPoint?) Unfortunately their corporate website seems to be down, meaning that they’re likely just as dead as their 4 Suite package.  Fortunately the great thing about places like SourceForge, besides the whole open-source thing, is that they’re also a great repository for dead packages and code.

The advantage of 4Suite is that because it was written by the same people as PyXML, 4Suite can be used with minimal code changes. It contains the PyXML API with very few differences. It just adds a whole lot more. But the one big differences is that you don’t use Python’s minidom, you use 4Suite’s cDomlette. Cute. And yes, it’s compiled code. And at least on CPython, it runs faster. It’s a little more than twice as fast as PyXML.

libxml2: 2.5X

Finally, a living breathing project! Based on a Python wrapping of the Gnome XML parser written in C, libxml2 is a breath of fresh air. Err … sort of. There’s a nice object-oriented wrapper written in Python. Which would be good … if it were documented. But darned if I can find any API documentation. And since the libxml2 wrapper changes the API dramatically from the original C code that it came from, it takes a bit of figuring out to use. It’s also slow. Oh, sure, the two-and-a-half times the performance of PyXML seems great. It’s even better than 4Suite. Barely. But if you’ll read through the rest, you’ll see it’s not so impressive after all, and there’s actually a very useful alternative right under its nose.

libxml2.libxml2mod: 6X

Yes, that’s right, it’s still technically the same libxml2 module as above. But if you load the libxml2mod.pyd file directly and skip the object-oriented Pythonic wrapper, going straight to the literal Gnome libxml2 APIs, you’ll have a lot more programming work (as the API is a lot more effort to code to) with a much better performance of six times the speed of PyXML. And it fully supports XPath. Who could ask for more?

Well, I could, actually. I don’t know if it’s the distribution that I got, or if it’s just not fully wrapped, or what, but there were some pieces of the Gnome C-code’s API missing from the Python libxml2mod.pyd file. The largest omission to me was XPath’s compile operation was completely missing. Since this can be vital to improving performance of executing an evaluate query across multiple nodes, it makes the 6X speed improvement even more impressive, as I was forced to do things the slow way, without compile. Which can of course be done. But it just makes you wonder, because the Gnome library definitely has this API, so it’s a mystery why the libxml2.pyd file didn’t.

PyQt4: -2.5X

If you’re using Qt4 as your Python GUI, you might as well use the Qt4 XML parser … right?

Well, maybe not.

Now don’t get me wrong. I love Qt.

Or at least, I loved the Qt that Trolltech put out.

But ever since Nokia bought Qt, it’s gone downhill. Fast. And this is a perfect example, right here.

The Qt4 XML parser is the darndest most complicated pile of API I’ve ever run across. Oh, it’s highly flexible. In theory. And it fully supports XPath. … In theory.  (I certainly haven’t tested every last feature.) But darned if I didn’t run into all sorts of mess just trying to convert the benchmark code to using Qt4’s XML parser. It was even worse when a bug (I don’t know if it’s in Qt4 or PyQt4) prevented me from evaluating to a QString, like you’re supposed to be able to do. So simple property lookups required the full QXmlResulItems overkill where I resolve the first result item from my results class instance, get the model index from that item, then use the model pointer from the index with the index to resolve it into a string.  Instead of just getting the first string, like I’d wanted and like it should have been able to do.  And not only is the API a mess (a highly flexible mess, but still a mess all the same), but it’s also two and a half times slower than PyXML. I honestly didn’t even think that it would be possible to write an XML parser slower for CPython than a pure-Python implementation that uses a DOM no less. Surely the C++ compiled-code PyQt4 would have a much faster XML parser than PyXML, right?

Well, apparently not!  As my benchmarks showed.

It was slow. Really slow.

Three-legged horse at the racetrack slow!

So I would highly suggest, to anyone using Qt4, DO NOT USE QT4’s XML PARSER! It’s that bad. To code for, and in performance. Find yourself another library for your XML needs. Trust me, you’ll be much happier that way.

I can only hope now that Digia owns Qt that some of these horrendous trainwrecks that have plagued Qt4 can finally be sorted out over time. Not likely to be seen in Qt5 though, as that’s still Nokia’s aborted afterbirth. Digia probably won’t get things straightened out until Qt6. And goodness knows how many years away that could end up being!

It’s hard to believe that with these lovely landmines in Qt that I still love it.  But the thing is, as bad as some parts of Qt are, no one has ever come close to doing anything better as an all-around solution to platform independent computer programming.  I just wish the original integrity of Trolltech had even remotely carried on to Nokia.  I just hope that Digia can give back some of the polish that Qt once had.

ElementTree: 3X

So I know, I already said that ElementTree doesn’t really support XPath properly yet. I really wish that it did. It’d be nice if I could just use the libraries built into Python for everything, and a good XML parser seems like a no-brainer. But for whatever reason, XPath is not really a part of ElementTree. They have kind of added beginning support to XPath type evaluate strings into the ElementTree find/findall queries, but a full implementation of XPath it is not. It doesn’t even support the full XPath string standard there.

Still, at least for enough of our use case, I was able to code for ElementTree. Converting the code from a full on XPath PyXML implementation to ElementTree and its lame partial implementation of XPath-based queries wasn’t as much work as it could have been. It’s nowhere near as much work as, say, converting the code to PyQt4, or even to libxml2. Which was pleasantly surprising. It’s a nice simple API, so I can see why people love it.

And the performance? It’s about three times faster than PyXML, making it a fair improvement. For a pure-Python implementation it’s actually quite amazing to squeeze that much out. But then, there’s a reason people don’t use DOM anymore. But the real treat comes next.

cElementTree: 18X

And here we have a real winner! Also included in Python, it’s the same API as ElementTree, just a very well written compiled-code implementation wrapped for Python. The same code that ran my ElementTree port also ran cElementTree with only the library name changing. Exactly like it should.

And the results were astounding. The real-use-case benchmark of XML parsing was a whopping eighteen times faster than our old PyXML code. Ding-dong, the DOM is dead!

Of course the problem is, all of our existing code is written for DOM using PyXML, so it’ll take a while to convert all of that to cElementTree.

As a side note, if any PyPy enthusiasts want to know why CPython programmers can’t convert to PyPy just yet (maybe not ever) here’s the reason why. A well-wrapped compiled code library runs like a champ in CPython. As a result, a lot of us big data/number crunchers have lots of compiled code in our Python projects. And since PyPy only just barely even runs compiled code, slowing things down far worse in that than native Python code, this leaves a lot of us CPython folks out in the cold. If you want the serious data crunchers to switch to PyPy then you have to start taking that compiled code lag more seriously or else we’re never going to be able to join you in your fancy little JIT Python interpreter’s dance.

Conclusion

So if you’re a Python 2.7 programmer looking for the best XPath XML parser ever, well, if you’re staying true to XPath at least, I’d say go with libxml2.

However, if you can swing it (and you’ll need to really evaluate your code to determine this) you might be able to get away with the rather unfinished XPath implementation in cElementTree. In which case you won’t need to install any third-party package for XML parsing and you’ll get blinding performance out the asterisk. (And obviously, if you’re coming at a whole new XML parser, and you don’t need XPath at all, then go with cElementTree since it’s what everyone in Python land is using and it’s got great performance.)

Hopefully the all-Python ElementTree runs just as great on PyPy, giving the world a pretty well rounded solution.

If any ElementTree authors catch this, hey, could you please work on supporting XPath a little more seriously?

And finally, dear god of all things software, whatever you do, avoid Qt4’s XML parser like the plague! Unfortunately I can’t speak to Qt5 yet as there’s still a lot of untested theory there that, professionally, we just don’t want to even approach mucking about with yet. Let things get a few more minor version numbers under the hood and then we can re-evaluate a PyQt5 upgrade path. (Or maybe even PySide.) But even if Qt’s XML parsing gets a major performance improvement, the API is still just as likely to suck wet donkey fur for being so “flexible”. Seriously, what committee designed that API? It’s everything that you could ever need … without being anything that you’d ever want! Yeesh!