I think I'm safe to go ahead and write this one, what w/ only one "working day" left in 2007. We'll see...

So, a user (we'll call her "Shelly") calls me indicating that her vertical-market time and billing accounting application is not spell checking data she enters. Further, Shelly states, she spent over an hour on the phone with the application manufacturer's technical support, and they've decided that the registry on her Windows XP Professional-based PC is "corrupt". Other users in the office don't seem to have any problem, but she can enter violently misspelled words and see no squiggly red lines warning her of misspellings.

I choked back my initial urge to comment on the use of the word "corrupt" ("What, has it been taking bribes or something?!?"), since I firmly believe that the words "corrupt" and "registry" coming out of a technical support representative's mouth together are really code for "I don't know what's wrong but I don't want this to be my problem anymore." I also choked back my urge to comment on the obscene yearly "maintenance" fee required by the software manufacturer that, apparently, entitles their Customers to receive slipshod, finger-pointing "support".

I agreed that I'd fire up a VPN connection, hop onto her PC with a remote desktop control application, and "ride along" to see what was happening.

I got connected to Shelly's PC and she demonstrated, in her characteristic ALL CAPS TYPING STYLE, the lack of spell checking. I've always cringed at the ALL CAPS-ness of her typing, but I know that her ways cannot be changed. She's been doing this since her office got time and billing accounting software back in the late 1980s.

"Did tech support have you logon to another PC where the spell checker works fine to see if the problem 'follows' your user settings," I asked. I was betting the spell-check component has both machine and user registry settings, and given that Shelly's office uses roaming profiles, a mal-adjusted user setting should 'follow' her, whereas a machine setting should stay in place.

"Uhh, no", she said. That's a strike against software manufacturer technical support, to me. I suppose they couldn't know that we were or weren't using roaming profiles, but even w/o roaming user profiles, different users on a Windows NT-derived operating system have different user registries.

I think it's reasonable to see if you can isolate an issue to being a "per-machine" or "per-user" issue at level 1. Of course, since they wrote the software to begin with, you'd think they'd already know where their settings were stored. To be honest, though, I'd guess that even their development staff don't have any idea where these supposedly "corrupt" settings are stored. My bet would be that they don't even have a way to talk to the people who wrote the spell checker, seeing as how it's a licensed third-party library.

I convinced Shelly that we should make sure the problem wasn't with her user settings. "But they said my registry is corrupt! My registry! Corrupt!" Oh, the horror. Luckily I've got a pretty good track record of solving problems for Shelly, and she let me proceed as I wanted.

I logged-on with my test user account and we attempted to reproduce the issue. My account received proper spell checking, unlike Shelly's. At least I knew that the underlying code was working fine, and likely this was a user registry issue.

I decided to head into the menus and have a look at the spell check options. Ultimately, I wanted to see where the settings are being saved in the registry. I figured I'd do my usual trick of taking a snapshot of the registry, changing a setting and closing the app, then taking a snapshot again and comparing the snapshots. (Sure, sure-- I could just run RegMon while I use the app, but I can do this faster and w/o downloading software.)

I never even made it to taking my "before" snapshot, though. When I looked at the options dialog, one (unchecked for my account) item jumped right out at me: "Ignore words in ALL CAPS".

"Umm, Shelly-- let's logon as you again and check one setting."

Sure enough, the setting "Ignore words in ALL CAPS" was checked in her settings. I read the setting's name out loud, and she replied: "Well, I set that because I type everything in all capital letters and I wanted it to..." She trailed off, then paused. "Oh," she said, "I understand. Sorry to have bothered you."

"No problem," I said, and disconnected from her PC. Nothing more could be said.

I suppose the only thing that could've made this better would've been if the spell checker had a setting to "Ignore misspelled words" and she'd enabled that.

I did a job for some friends this evening at a small dental practice replacing a server computer. I removed the "old server" and its "UPS" and had a big laugh. The alleged UPS, one of those APC "power strip with a battery" models (a Back-UPS ES 650), was providing standby power for the old server computer, or rather, was supposed to be.

When I pulled the UPS out from under the desk, I found the factory-applied bright yellow sticker advising the user to connect the battery still intact and firmly covering all the battery-backed outlets. The server computer had been plugged into the outlets labeled "Surge Protection Only" for the 5+ years that it was in production use.

I've just written a Group Policy Administrative Template for the "fix" outlined in Microsoft KB article 946627. If you're experiencing the crashes described in that KB article or the Microsoft Security Response Center blog entry, you may want to consider deploying this registry change. With this ADM, you can centrally apply this change using Group Policy.

Grab the file here: http://peeved.org/misc/946627.adm

Beware that this is not a true "policy" and that (a) you'll need to turn off "View / Filtering / Only show policy setting sthat can be fully managed", and (b) it will "tattoo" the registries of machines such that if you ever need to disable the setting you will need to create an evil "anti-policy" to disable it. Neither thing concerns me, but I thought I'd be up-front about it.

Good luck.

My business partners and I are deploying Office 2007 at a few Customer sites now, and we've quickly learned that Outlook 2007 is greatly "enhanced" with the addition of Windows Desktop Search (WDS). Without WDS, the "Find" functionality in Outlook is diminished.

With this in mind, I embarked on installing WDS as part of an Office 2007 deployment. This particular Customer was of the mind that client computers should not be indexing resources on server computers, being that they have an enterprise search product already implemented to do that. I started looking for a way to administer WDS via Group Policy and, specifically, how to disable indexing of resources via UNC or "mapped" "drive".

In version of WDS prior to 3, from what I could tell, there was a registry setting to disable indexing of any resources accessible via UNC. Further, from what I can tell by examining the EXEs and DLLs that come with WDS 3.01, the name of the registry value that controls this behavior, PreventIndexingNetworkShares, is no longer a string that occurs in the SearchIndexer.EXE executable. (As an aside, I did find the string in another file, Deskbar.DLL, but I don't expect that it would control indexing behavior.)

The Windows Desktop Search Administration Guide details extracting a group policy template, DesktopSearch30.ADM, out of the installation package for WDS, and I saw no specific setting to prevent indexing of all UNC resources. The next best setting, appeared to be PreventIndexingCertainPaths, known by the en-us localized string "Prevent Indexing Certain Paths". The syntax to specify UNC paths is odd to me, being of the form otfs://{*}/server/share/path. I wondered if I could exclude all UNC paths by specifying otfs://{*}/*/*, so I tried.

After specifying this setting in a GPO and rebooting my deployment test machine, I found that I was waiting approximately two (2) minutes at "Applying settings..." before the boot would proceed. A little searching turned up a conversation on the MSDN forums that described the issue I was having. It seemed that the Windows Search service (SearchIndexer.EXE, running as the WSearch service) was hanging up the boot process until the service control manager got tired of waiting on it. Helpfully, an Ari Polsky from Microsoft indicated that it was a "known issue", and offered two (2) mutually unacceptable "fixes": (1) To make the WSearch service dependent on the NLA service which several users indicated did not help, and (2) mark the WSearch service for manual startup, which sorta works.

I decided to go the route of the manual service start, and modified my GPO containing my WDS management policy to include a setting to make the WSearch service start manually. The next boot went very quickly, but I discovered upon logon that the service didn't start "as advertised" by Mr. Polsky's message, and that attempting to start the service gave me a "The service terminated with service-specific error 2147500053". This, too, was reported in the MSDN forum thread, and several readers figured out that marking the WSearch service for manual startup by altering only the Start parameter in the registry, rather than using the a service policy in a GPO, seemed to alleviate the service-specific error.

I knew that service policies in GPOs also alter the security descriptor assigned to the service, so using the sc sdshow command, I found that the security descriptor for a stock WSearch service was:

After modifying the service to start manually, using group policy, but not modifying the security settings, the security descriptor becomes:

These descriptors are expressed in the Security Descriptor Definition Language, or SDDL. It's painfully difficult to read, but, fortunately, some people at the University of Washington assembled a SDDL syntax guide that can help you decode these nasty strings. One thing that their guide doesn't contain, though, is a description of how the types of "rights" granted by the descriptor, which are phrased in their documentation in the manner of filesystem permissions such as "Create All Child Objects" or "List Contents", are applied to services.

Microsoft wrote a knowledge-base article, KB 914392, which describes the Best practices and guidance for writers of service discretionary access control lists. This article is dense and, unfortunately, uses abbreviations that are somewhat ambiguous to describe the meaning of the "rights" granted. Alun Jones, a Microsoft "Most Valued Professional" for security, wrote a nice article describing how SDDL rights relate to services that is understandable and clear.

So, armed with these articles, I performed an exercise to figure out what security descriptor was required in order for the WSearch service to start and run properly. Fortunately, I hit it right off the bat. The "stock" descriptor names "Built-in Administrators", "SYSTEM", "Authenticated Users", and "Power Users". The descriptor that is applied by group policy, by default, names "Built-in Administrators", "SYSTEM", and "INTERACTIVE". Just for kicks, I added "Authenticated Users" to my service policy GPO, and granted them "Read" and "Start, stop and pause" permission. This gave me a resulting security descriptor of:

That last bit-- the part that ends in ;;AU) now matches the "stock" descriptor for "Power Users" (the one ending in ;;;PU) in the first example).

One reboot later (which proceeded at normal speed), and I found that the WSearch service had started itself after the user logged-on (as indicated by time-stamps in the System event log), and did not terminate with any errors. Success! I'm not really concerned that plain ol' users can stop and start the WSearch service, because it seems to want to start itself (by way of the WDS user-land UI).

After doing all of this, I finally got a change to see if the otfs://{*}/*/* setting for the "Prevent Indexing Certain Paths" policy worked, and I'm happy to report that it does. The users will not be indexing the server computers via "mapped" "drive" or UNC.

In summary, if you're having trouble with long startup / bootup times on computers running Windows Desktop Search 3.01 with settings being applied via Group Policy, I'd recommend creating a service policy to set the Windows Search service to manual startup, and modifying the security settings in the policy to include "Authenticated Users" with "Read" and "Start, stop and pause" rights.

As an aside to this whole thing, it appears that Microsoft has implemented a Group Policy Client Extension (GUID {7933F41E-56F8-41d6-A31C-4148A711EE93}) in srchadmin.dll for WDS to process Administrative Template settings from. Searching Goooogle for keywords like CLASS MACHINE and CLIENTEXT, I'm not coming up with a lot of Administrative Template files, other than this one for WDS, that make use of the client extension capability. I've thought about doing it for some work that we've done, and after seeing how easy it is to integrate it into an Administrative Template, perhaps I should think more strongly about it.

Another in the series of "hope some other poor sod else finds this helpful" posts:

I was putting together Adobe Reader 8.1.1 for installation via Active Directory Group Policy Software Installation, and ran into a minor fun issue. After a diagnosis and resolution, I thought it'd be helpful to document it here.

I'd already obtained the full Adobe Reader 8.1 download from Adobe's site some time ago. Rather than download a full version of 8.1.1 (heck-- I don't even know if there is a full version of 8.1.1), I downloaded the 8.1.1 patch. I'd already created a transform with the Adobe Customization Wizard for Adobe Reader 8.1, and I was going to gamble that this transform would still work with 8.1.1.

I extracted the MSI and CAB from the Adobe Reader 8.1 setup program, and from there, performed an administrative install of the MSI using MSIEXEC /a AcroRead.msi. I stuck the administrative install point onto a server computer and used MSIEXEC /p ReaderUpd811_all_incr.msp /a \\path\to\MSI\AcroRead.msi /qb SHORTFILENAMES=TRUE to apply the patch to the administrative install point.

I setup an installation in an existing GPO to target the upgrade to my test deployment machine only and fired up the test machine. To my displeasure, however, a quick inspection of the event log after logon on the test machine showed the following:

An examination of the program files\Adobe\Reader 8.0 directory confirmed that there was no Update subdirectory. An examination of the program Files\Adobe directory, however, revealed a READER~1 directory. Ah ha!

I moved my faulty administrative install point aside, re-extracted an administrative install point from AcroRead.MSI, and re-applied the patch without using the SHORTFILENAMES=TRUE argument to MSIEXEC. I bounced my test machine and watched it properly install Adobe Reader 8.1.1 over Adobe Reader 8.0. Success!

My "conventional wisdom" in applying patches has been to blindly provide the SHORTFILENAMES=1 argument to patch installations. Obviously, this isn't always the correct thing to do.

Jeff and I were working tonight on some code that needed to search an Active Directory domain at its root, using the python-ldap library. We had a problem wherein our search would work successfully if performed at a base DN below the root of the domain, but not at the top of the domain. The error returned in the python-ldap exception was: LdapErr: DSID-0C090627, comment: In order to perform this operation a successful bind must be completed on the connection., data 0, vece

Normally, this error indicates that you're attempting to bind anonymously, which Active Directory (sensibly) doesn't allow by default. We were supplying credentials to bind, though, and changing the base DN on the search to a sub-OU was all that was necessary to get the search to work. It turns out that python-ldap was binding anonymously, so the error was only sort of a red herring.

Using a sniffer, we determined that the python-ldap library was chasing referrals being returned by Active Directory to our Configuration, ForestDNSZones, and DomainDNSZones NCs. I think that setting a LDAP_SERVER_DOMAIN_SCOPE_OID server control would have probably stopped Active Directory from returning the referrals in the first place, but it appears that python-ldap doesn't support server controls "yet". As such, we opted for adding the following to our code (right after importing 'ldap') to stop python-ldap from chasing the referrals:

We still get the referrals in our results, but we can wade thru those ourselves.

Thanks to Derek Anderson for a quasi-related posting that finally got us in gear...

This post is another in my ongoing series of postings about things that might help other "IT people".

I just built a "BartPE" bootable Windows CD to recover a Microsoft BKF-format backup image for a PC with a failed hard disk drive. I was surprised to get NTBACKUP running under the "BartPE" environment fairly easily.

My CD was built using the Windows XPE 1.0.7 plugin on top of a Windows Server 2003 SP1 slipstreamed CD. I copied NTBACKUP.EXE and NTMSAPI.DLL from a Windows XP SP2 machine into my "custom" folder and burned that onto the CD.

NTBACKUP comes up with an error writing its log file, and complains about the Removable Storage Service being inaccessible, but other than that it works fine. I'm quite pleased.

Off to restore a failed PC now...

I'm rolling out JET SP8 onto PCs via a startup script, and I wanted to force Sandbox Mode via Group Policy. Searching Gooogle for sandboxmode end.policy to see what somebody else might've already written, I came up with only one (1) result, from the MSAPPS-L@LIST.EMWAC.CZ mailing list.

I slashed up the template that Miroslav Pragl posted, and came up with Access-SandBox_SystemDB.adm. Hopefully this will be helpful for somebody-- it's doing that I need. Be sure to uncheck the "Only show policy settings that can be fully managed" checkbox when using this administrative template, since it's not a "fully managed" setting.

I've griped before about how frustrated I am about the "state of the art" in software issue resolution in the IT industry. This typically involves unskilled "technicians" grubbing around in software "manufacturer" "knowledge bases", blogs, and message boards, reading wild speculation written by people who may not actually understand how the subject software works, and who certainly don't have access to the source code (and probably couldn't interpret it if they did). I'm not proud to work in an industry of cargo cultists, alchemists, and idiots.

Eric Fitzgerald has a posting that really raises my ire. I've spent some time writing software to aggregate and generate real time notifications from Windows event logs, as of late. I've been frustrated at the lack of comprehensive documentation for the different events that can be generated by Windows, and astounded that Microsoft can't put together documentation on all the events that Windows itself can generate. I mean, it's not like they don't have the source code.

So, with that in mind, Eric says:

This just makes my blood boil. I'm sure that Joe Anybody inside Microsoft can't just go hop into the source code to Windows / Office / etc anytime they want to. I'm also sure, however, that the ability of a Microsoft employee to actually talk to a developer with source access (and, possibly even a developer working on the part of the product they're interest in) is a lot greater than my ability to do the same.

Accurate, thoughtful, and well written documentation about the behaviour of a software product is the best Customer service tool that I know of, next to access to the source code itself. My supposition has always been that Microsoft employees don't understand what it's like to work in a world where source code access isn't available for key infrastructure components, like TCP/IP stacks, filesystems, and hardware drivers. They live in an insular world where they have quick access to these things, and where knowledgeable personnel are an email away. Eric's posting serves to confirm this supposition nicely.

Because Microsoft employees work in this environment, they appear blinded to what it's like to work out here, in the real world, where we have little or no visibility into the inner workings of closed-source software, and no ability to affect change when that software doesn't do what it was promised to do. Microsoft may eat its own dog food, but its employees certanly don't have to clean up the dog's mess.

When I run into one of these areas where "the spec" and "the code" don't agree, I get to play "knowledge base roulette" (or, at the best, "debugger and symbol file roulette"), while a Microsoft employee can actually do some meaningful research. Because Microsoft corporate culture doesn't value writing detailed, thoughtful, and comprehensive documentation (and because they produce closed-source software), I get to look like an idiot in front of my Customer as I bungle around searching for answers, instead of just getting out the docs (or the source code) and solving the problem.

Sure, I could pay Microsoft to provide "support" for software that I already paid them for to begin with. For some reason, though, I feel pretty strongly that I should be able to pull out the docs and figure out what's going on. (It makes me thankful I didn't give Microsoft any more money to upgrade my MCSE certification to Windows 2003. You don't ever look much dumber than when a Customer says "But you said you were a Microsoft Engineer something-or-other. Why can't you make it work right?")

And why should Microsoft bother to write better documentation, anyway? As long as Customers are willing to accept that they won't ever receive quality documentation, and are willing to pay for support incidents associated with bugs and deviations from written specs, the vacuum created by this lack of good documentation actually creates revenue for Microsoft! It's a win/win situation for... well... erm. It's a win situation for Microsoft shareholders!

While I'm on the subject, why would "the spec" and "the code" not agree? Seems like bush-league software engineering to me. Maybe I'm just out of touch, because the only software projects that I've worked on have been for medical devices. When requirements changed, we updated our specs before we went off hacking on code. Our specs were vastly more important than our code. I can't imagine writing software (other than one-off scripts) any other way. (I am regularly amazed at the amateurish mistakes that an organization as big as Microsoft makes...)

The answer to all this, I think, is for Customers of Microsoft, and all the other software companies who have similar culture, to stand up and let the companies know how they feel about this. Using open-source software that directly competes with closed-source alternatives is one great way to do this. Complaining vociferously in public forums, blog postings, and articles is also great. As I see it, those of us who work as consultants, especially, need to let our Customers know that they don't have to accept things as being "just the way they are". If a couple of my Customers spent 25% of their annual anti-spam licensing fees, for example, on paying a portion of a developer's salary to hack on open-source anti-spam software, they'd see improved support response (Gee-- it took Symantec over a month to resolve a critical buffer overflow issue in an "Enterprise" product...) and more customized functionality. It'd be wild to see some Fortune 500 companies siphoning a small percentage of their yearly Microsoft Office licensing budget into OpenOffice.

For my part, I'll keep being angry, and I'll keep telling everybody I can, anytime I can, about why I'm angry and why they should be angry, too. I'll keep using open-source software everywhere that I can, and I'll try and spread the word about these alternatives to closed-source software companies with flawed cultures of Customer service.

Susan Bradley linked to a Microsoft Small Business Community blog posting that is just amusing the hell out of me. Essentially, the posting describes the "downgrade rights" afforded when purchasing a Microsoft Action Pack subscription. This Action Pack is described by Microsoft as "a full suite of Microsoft Not-for-Distribution (NFD) software to help you run your business, train your staff and demonstrate to your business customers."

Of course, the software in the Action Pack is updated and revised by Microsoft, and the contents change as new products are introduced. The Action Pack no longer comes with Windows XP Professional upgrade licenses, but instead comes with Windows Vista upgrade licenses. As such, the question was brought up on the Small Business Community blog as to whether or not subscribers have "downgrade rights" (rights to use a prior version of a product) when their Action Pack renews. Quoth the Small Business Community blog, who was, in turn, quoting the "Action Pack Team":

The complaints in the comments of this posting are so amusing to me. Have a look at a couple of them.

Welcome to subscription-based software licensing! If you thought of "purchasing" an Action Pack as anything other than "paying the yearly software bill", then you thought about it incorrectly. (Giggles abound...)

I worked for a small firm, years ago, that was a Microsoft Certified Partner, and the owner refused to accept that the entitlement licenses were time-limited to the subscription term. He just decided that they were perpetual grants of license, and that they were "additive" from year-to-year ("Now we've got twenty (20) seats of Windows 2000 Professional!"). I always secretly hoped that Microsoft would come audit him and set him straight...

It strikes me as intelligent, though I would guess serendipitous, on the part of Microsoft to get "partners" used to subscription-based software, since that's the direction they're going to need to move if they hope to continue growing their revenue stream (and to insure that no one can buy a computer and use it in perpetuity w/o paying them again and again and again). Get the people who sell your products into the habit of paying for software this way, and then they'll help indoctrinate their Customers!

To those of you who are asking "Who thought of this?", I'd think the answer would be obvious. I'm quite convinced that Microsoft sees their "partner" and certification programs as revenue opportunities, just as much as they do their regular sales channels. I decided a long time ago that my Microsoft Certified Systems Engineer credential was nigh useless, and never bothered to update it beyond where I did. In fact, touting loudly that I was "Microsoft Certified" cast aspersions on my ability to evaluate software and solutions for my Customers in a neutral manner. When we started our new company, my business partners and I decided that we would not pursue any manufacturer affiliations like many of our "competitors", so the company could maintain a shred of credibility as a neutral third party. Of course, it also helps that we are strictly service revenue based, too, and haven't fallen for the seductive but dangerous route of selling physical goods.

I'll have to rant another time about why commercial software is a service, and one that probably should be subscription-based... but that is for another time.