Don't google "dead horse" for images.

My response to Pie’s Quality of Documentum Over the Years bears repeating, even if I am beating a dead dmHorse:

I started with version 2, back when I was just a newly-minted UNIX geek. One thing you missed with the transition to 4i was the introduction of the DFC. DMCL had a very UNIX feel; a simple, open API designed to be glued into any programming language. DFC was just Java then, with a COM layer growing over it later. That was also the point where EMC became more marketing-driven and started chasing the Internet bubble at the expense of their existing clients.

Both were attempts to capitalize on hot topics of the time, Java and the Web. I never bought that the DFC would make a whole pool of talent available; Documentum’s about the model, not the means. However, the marketroids successfully reframed it. Hiring managers now believe they can take Java people and mold them into Documentum people, and I hear gasps of disbelief when I say Java or Visual Studio aren’t requirements to do Documentum–a good Java programmer is not necessarily a good Documentum developer.

This Java mentality did increase the number of people with Documentum on their resumes, but the talent didn’t increase as much as the volume. It just diluted (maybe also tainted) the pool. It became harder to find good people in the now-mirky waters.

The lack of focus then is what brings us to the lack of quality now. Innovation at the model and server level is rare, and frankly I don’t give Documentum much geek cred anymore because of it. Great ideas like BOF and Aspects are stapled into an API rather than made an inherent part of the product. Too much work up the stack (and on vertical solutions) has made the product top-heavy and tottery. EMC continues to chase markets (i.e., case management) rather than concentrate on making a solid core product.

Noted rumor-monger  Brilliant Leap speculates about Documentum in a world without EMC. Tongues were already wagging at EMC World about the SAP-EMC partnership leading to something a little more intimate. It has enough of the smell of truth to make an irresistible rumor.

As rumors go, I still prefer the Microsoft angle because of the obscene anatomy kissing that IIG is still doing.  Truth is it’s just to easy to dispel: Why buy the cow when you get the milk for free? I doubt Microsoft will be stamping shrink-wrapped boxes of SharePoint with Documentum Inside! anytime soon, but I’d bet they have an infinite number of code monkeys banging away to make their own document management Hamlet. Once they do, it’s bye-bye Documentum! Then all those monkeys will get down to business and start flinging feces IIG’s way.

Part of Documentum’s doom was being a software company bought by a hardware company; however, it won’t be saved if another software company buys it next. Such companies (SAP included) would buy Documentum to augment their flagship product, not eclipse it.  With no Fairy Godmother rescue from being passed around from one wicked step-mother to the next, this story’s ending will be more Le Boheme than Cinderella. Or worse, a more-jackal-than-wolf company that hasn’t innovated for decades might gobble it up to suck the last trickle of marrow from its cracked bones.  *cough* computer associates *cough*

I am no Wall Street cheerleader, especially after my time in Big Finance, but the closest thing to a Fairy Godmother out there is a technology-oriented private equity fund. Such a fund buys troubled companies to turn them around and sell them for a profit. Unlike most of Wall Street, they take the long view of years rather than a quarter or the milliseconds around a stock’s uptick.

Their methods can be harsh, but their goal unlike any step-mother’s would be to make Documentum the best product and most profitable (and saleable) brand it can be.  There may still be an ounce of brand left to save. By going private, the recuperating Documentum wouldn’t be burdened with public company regulation or the tyranny of speculative stockholders. It’s an imperfect cure for the age of gratuitous IPOs and acquisitions fueled more by irrational exuberance than smart business.

We have a test case with AOL selling Bebo to Criterion Capital Partners, LLC instead of just shutting it down. Taking Valdes’s animal shelter metaphor a little further, I’m sure Criterion will euthanize Bebo and reap their own “meaningful tax deduction” if the old dog can’t learn new tricks. Sometimes I think I’d rather see that happen to Documentum than sit through the EMC’s little opera until the consumption takes it.

I can’t find a video of the Mark Lewis keynote from EMC World 2010. Instead I’m depending on reporting from the event like Ron Miller’s article [Documentum group gets new name and new direction] and Pie’s tweets and blog posts. It’s probably for the better; I never had a taste for gruesome videos since Faces of Death, and this may be EMC finally decapitating the Documentum brand.  Rather than plunging into a pages-long diatribe about EMC’s unconditional surrender to the commoditizing of content management or the latest dish of scorn Lewis served up to Documentum veterans, let’s talk names.

Information Intelligence Group is EMC’s new moniker for the product that shall not be named. Lewis breaks down the name for us on his blog [Episode 91: EMC World 2010 - The Birth of the Information Intelligence Group]. It’s hard to read–let alone say–the name with a straight face, and this breakdown doesn’t help. At least the cumbersome and uninspired Content Management and Archiving accurately conveyed something about the product.  This new name is too broad and inherently meaningless; it will continue to erode mindshare for a product that was the de facto definition of document management. Let’s hope this new name doesn’t prove itself a compound oxymoron to boot.

The “Intelligent” product silos aren’t much better. Granted, this is a product that has to publish a separate guide with each new release to map old product names to new. Not a sheet or a few pages, a document. However, these new silos are so vertically restrictive that EMC had to toss the content server into case management.  Having done case management and having paid my dues in lines of server code, I’m perplexed. It’s like they had a very limited box of magnetic poetry to play with.

The continuing erosion of a strong brand means less mindshare among potential customers. Everybody knows SharePoint even though most don’t know what it really is. I’ve seen first-hand how good marketing trumps good product. Documentum had that name recognition–still does in many parts–and EMC seems determined to stamp it out without something sticky to replace it.

Unmentionable!

When Twitter changed their reply functionality, now called mentions, my initial reaction was unmentionable.  After a few weeks to ponder and play with it, I still think they made a big mistake.  A reply was originally a message that began with a twitter username, like this:

@zorak No, really?

Replies were public, but Twitter added a link so you could see just your replies and options to filter other people’s replies out of your friends stream.  According to the Twitter blog, the community came up with the convention that Twitter later embraced and enhanced.  Then Twitter added a separate API call and a “swoosh” button to their web site:

Just what I wanted! Twitter added metadata underneath so that a reply remembered which tweet it replies to.  Pretty soon every Twitter client included swoosh buttons and “in reply to” links.  This was a philosophical break for Twitter–whether they know it or not–because there was no way to distinguish swoosh and “@user …” via SMS.  Supporting SMS creates a larger potential user base, but it drastically limits functionality.  Until everybody has an iPhone, fledgling social networks like Brightkite must consider this trade-off.

The original reply syntax is still supported and continued to create confusion as Tweetie developer AteBits explained on his blog.  People put multiple names in the message or put the @ in the body of the message, assuming the right people would see the replies:

@me @myself @I Remember the milk.
Give @me some sugar, @baby!

Only @me sees the first tweet as a reply; nobody sees the second.  Search was already catching on thanks to other community-grown initiatives like hash tags; users and client developers began using search on @user instead of the reply API to catch such grammatically incorrect tweets.  Apparently this is a bad thing, or at least something Twitter discouraged, perhaps because of its impact on Twitter’s call throttling.  That and other scaling problems should make for a few good dissertations; I just hope Twitter is keeping the historical record and will be willing to share it.

This brings us to mentions which are basically just searches on @user.  Although it’s a good thing that Twitter learns from their community, the big mistake here was changing the functionality under the existing API calls.  I agree that instantly supporting new functionality in all Twitter clients is attractive to a provider, but it can–and did–create unintended consequences.  All those clients blessed with catching those malformed reply tweets were also cursed by all those side-bar mentions crowding the replies page.  Twitterati like @wilw get many more mentions than direct replies, and now there’s no easy way to sort out the two.

The lesson here is that it’s safer to create new API and UI elements for new-ish functionality and let the community migrate over than to replace the guts and hope nothing breaks.  As any API designer knows, developers will do all kinds of unexpected things once your API is released into the wild.  The Twitter community’s active, inventive role in shaping Twitter also provides for some real “They did what?” moments.  Tweaking reply functionality to support only swooshes and adding new methods for mentions would have made everybody happy.

There’s one thing I want from Twitter that they promise in the API FAQ; I want to see all replies for a given tweet.  I disagree the assertion in the Twitter blog post above that people don’t want to wander into the middle of an ongoing conversation.  Sometimes that’s the best way to discover new topics and interesting people.  When that happens, there is a need to go back and discover the source and all its tributaries.  Twitter is aware of the need, as this quote from the Twitter API Wiki FAQ shows:

How do I get all replies to a particular status?
For now, there’s not a great way to do this. We’ve heard the requests, though, and we’ll be providing a solution for it before too long.

Conversation functionality is cropping up in Twitter clients like Nambu and the soon-to-be-released Tweetie for Mac (20 April 2009).  Looks like Nambu constructs conversations based on cached tweets, building little trees as it discovers reply pointers to other already-fetched tweets.  This single-linked list structure makes it easy to find your immediate predecessor but difficult to walk up, across, and back down the tree.

Hmm, where have I seen this problem before?  Oh yeah, version trees in Documentum.  Every document remembers its immediate predecessor (i_antecedent_id) and the root of its version tree family (i_chronicle_id).  A single query on i_chronicle_id returns every version of that document.  That’s just what I want Twitter to do!

Twitter already has its own i_antecedent_id–with a better name I hope.  So add an equivalent to i_chronicle_id and a new getAllReplies API call.  I suggest topic_id since that’s what the root tweet of a tree of replies becomes.  It would be nice to go back and stitch up all the previous replies-of-replies, but I would understand if the hit on the database would be too big.  How many tweets are in there anyway?

Let’s deal with the underlying issue here:  The ESF article is talking about the other 99% of EMC that bought Documentum to sell disk.  EMC friends, I’m just kidding!  I can joke, right?  Since storage is not my expertise beyond some hacking around with Centera as primary content stores, I suppose EMC might really knock their customers’ socks off in the storage arena, but I doubt it.

Network disk isn’t something you see until it isn’t there, just like any good support technology.  It’s not sexy.  Apple’s products are shameless show-boaters meant to hog the spotlight.  They are meant to be seen, to be touched, and–dare I admit–even licked. Maybe that’s not a recommended way to unlock an iPhone, but what else can you do about winter, thick gloves, and a touch screen?

Would the average EMC storage user know the EMC logo on sight?  Would the average Apple user?

OK, so the EMC logo actually says “EMC” in it.  You get the point though, right?

Enderle’s article talks about quality, metrics, and customer loyalty.  All those things are important to Apple, although the often-excellent quality of Apple products is marred on a regular basis with things like incendiary power supplies and the worst product launch ever: iPhone 3G + MobileMe.  WORST!  LAUNCH!  EVER!

Only Starbucks matches Apple’s skill at selling Lifestyle. The synergy of Apple’s well-designed, well-integrated components had this caged bird singing gaily a few posts ago despite a healthy fear of monoculture coming from a science background.  That all misses the point, the one reason the whole discussion is apples and oranges:  Innovation.

Apple sets the bar for technology after technology:  operating systems, mp3 players, online music retailing, and of course smart phones.  The integration, the cool, the marketing are all icing on the cake because Apple does something better than anybody else:  They innovate, and they do it where they can define the market rather than chase it.

I don’t think Big Disk lives or dies on such radical innovation. In fact, their customers probably fear change more than most.  Change is not good for 24/7 availability.  I can hear the compliance officers, archivists, and system admins shrieking in terror at the thought of something that might as likely store their entire repositories on a postage stamp or burst into flames if looked at the wrong way.

There is a relentless integrity of concept, simplicity, and message spanning all Apple products that likely has a single source, Steve Jobs.  Such single-mindedness is what makes big, ambitious, risky, not-for-the-faint-of-heart products succeed or fail spectacularly.  Apple’s done both regularly.  It allows org-wide turn-on-a-dime changes, something that another industry titan *cough*Bill Gates*cough* executed brilliantly after completely missing the Internet as the Next Big Thing.

That conceptual integrity, that vision from the top is also why Apple clung to its single-button mouse a decade too long and why the iPod Touch and iThingThatWillNotBeNamed are missing the aesthetically unpleasing extra two or three buttons needed for touch-without-sight operation.  Because that’s how Steve Jobs sees it, end of story.

I’m sure Joe could give Steve a few helpful hints on running disk farms for MobileMe or handling eDiscovery for the next options scandal, but that’s not the point.  It’s what Jobs teaches his successor and if that successor has the Right Stuff to wield Apple as a single instrument of innovation, lest Apple repeat the recent catastrophes of their rivals to the North.

I have some long-time draft Documentum posts that should see daylight soon as well, and the new smart container feature in D6.5 has me pondering the fate of virtual documents and EMC’s folder fetish. Thanks to Brilliant Leap for sending me the link and distracting me from iPhone adoration for a few minutes.

Finally, kudos to the makers of WordPress 2.7.  Constant interface changes make me a little crazy (glares at Lotus Notes), but this truly was a big step forward for the project.  I’m looking forward to what they do in 2009.

Have you ever retired a Mac by mistake? No.

I bought a unibody MacBook Pro. Aside from being a thing of beauty, the improved GPU means all my games run well under Boot Camp; the Dell XPS 600 may get an early retirement (in the Blade Runner sense).  I love the trackpad and found myself forgoing a mouse while traveling–except under Windows where it’s squirrelly and inconsistent.  I hope Apple starts squeezing out Boot Camp updates as fast as iPhone updates to fix the track pad problem and take full advantage of the dual GPU.  Please cross all appropriate appendages.

I ended my contract in Rockville, MD.  The commute got to me and I ended my contract with F. after a one month extension to see one of my projects through its first deployment to prod. I’m making a pattern of very long separations to help with transition, and that’s both good and bad: Good this time because I think it really made a difference in the deployment; bad because I missed out on the before-end-of-year hiring cycle.  I hope it reboots as usual in the beginning of the new year despite the grim economic situation.

I got the flu.  My last day in Rockville was when the fever kicked in.  The following three weeks were pretty miserable, and I still have a bit of a cough after another two.  Next year I’m definitely getting a flu shot, and I’m wondering if I should still get one this year since they typically treat a handful of the most common strains.  Any medical professionals care to comment?

I drank the Kool Aid and bought an iPhone.  Expect a subsequent post with some of my favorite apps.  I’m testing how long it can last in stand-by mode with power consumption optimization (WiFi and 3G only when needed, no push, hourly pull) and it looks like the answer’s 72 hours.  I may have to bump my SMS messages up to 1500 from 200 which irks me to no end, but that’s one of a handful of gripes that are more about AT&T than the device itself.  Otherwise, it’s a Good Thing ™.

I attended MJD‘s talk at Philadelphia Linux Users Group (PLUG) about strong typing.  The slides for Atypical Types are on MJD’s site.  We agree that Java 1.5 is the first usable version of the 1970s-style programming language but for different reasons:  He asserted at the talk that Java typing got better because a Haskell guy rewrote the 1.5 compiler and force fed Java some good medicine like generics. Is Haskell the new programming language incubator?  Given things like type inference, how a smart compiler removes all strong-type clutter that stupid compilers require, could be.

I corrected a long-standing mistake about Documentum Composer in my I’d Rather Be in Philadelphia blog post.  Despite being a newer Documentum customer, F. had enough invested in its 5.3 architecture and how it integrated into their corp-wide build process to make Composer a non-starter.  I’m looking forward to a contract in the future where I get to use all the new 6.5 stuff without the drag of a 5.x (or even 6.0) install base.

Resume of John Kominetz (20080903)

1. Both folders did indeed have identical object names without leading or trailing whitespace.
2. They did NOT have any values in common between their i_folder_ids, so they weren’t really in the same place. But Desktop Client though otherwise.
3. One had r_folder_path values with trailing whitespace in one of its containing folders.
   e.g. “/cabinet/folder1 /folder2/folder3″ — Hard to eyeball the space after “folder1″ unless you’re looking for it!

After that, I couldn’t reproduce the problem. This docbase was well known for having “issues”, so nobody (including Documentum) was surprised that something impossible and inexplicable was happening. I mentally filed it under “Documentum moving in mysterious ways”, gave the business administrators a tedious but effective work-around, and got back to more urgent concerns.

The next year I ran into this problem again from the engineering side instead of the business unit. As a particular project wrapped up or budget ran out, the company passed me around to different groups. I’d worked in engineering back in 2001, and my knowledge of the company’s custom environment made letting me go difficult.

I ended up helping out with the 4.2.8/5.2.5 upgrades by developing a triage procedure and fixing some of the more dire problems. Ever see those pictures of tumorous, sooty lungs the Cancer Society uses to scare people away from cigarettes? That’s what some of these docbases looked like on the inside. Not fun. Not pretty.

After triaging a hundred docbases, I discovered some of them also had corrupt r_folder_paths, although in smaller numbers than the original-problem docbase. Thankfully it didn’t interfere with the upgrade procedure. I made a few more notes before continuing onto my next engineering task, the arduous quest to get Documentum to fix the horribly-broken-under-Sybase content replication process.

Another year passed, I was passed along to another business unit, and a new folder-related problem emerged. Business users (in webtop) reported empty folders and missing documents. The document management group (in desktop client) reported the documents were right where they filed them. If you guessed those empty folders had corrupt r_folder_paths, you were right. Webtop uses a slightly different query to determine folder contents than desktop client and as a result couldn’t “see” the contents of such a corrupt folder.

There Will Be Pain

You may want to take some ibuprofen before continuing …

This particular client as a rule doesn’t allow anybody to talk directly to the Sybase DBAs. Don’t ask why. Healthy docbases require attentive, knowledgeable DBAs working with developers and docbase administrators. I would probably have figured this problem out by year one if I’d had that kind of healthy relationship, but I didn’t.

Some suggested I master Sybase. Frankly, that’s wasted effort since it’s such a small part of Documentum’s market. If this were Oracle or SQL Server, sure. Thing is, the client desperately needs DBAs cross-trained in Documentum if they were going to tame the growing performance and stability problems. Despite getting a few people in management to shake their heads yes when I said this, nothing ever came of it. Sigh.

So I found the next-best thing to a current Sybase DBA, an ex-DBA in my part of the organization. I started asking questions, and he casually mentioned the “obvious” fact that the Sybase API strips all leading and trailing whitespace from VARCHARs passed to it. I was dumbfounded.

Call me a purist, but this is data corruption by design. If I programmatically pass a string to a database, I expect the exact same string back when I ask for it. That’s just common sense–unless you live on Planet Sybase–but it didn’t really explain the problems.

Most of the affected docbases either had custom scripts or Input Accel adding content, and they all had several layers of folders that related to content metadata. This was the second big clue; both methods of adding content had to sometimes create several layers of new folders to file the first document for a particular client, counterparty, or deal. That’s still didn’t explain it all.

The DMCL caches metadata for newly-created, fetched, and modified objects. That includes folders. It turns out that the cache doesn’t automatically refresh from the database after a save. Now we have the three components needed for the impossible to happen. Let’s walk through the process.

1. Open iapi and connect to a docbase.
2. Create a new folder “B ” whose name has a space at the end (OCR, bad typing, etc.), link it into cabinet “A”, and save it. The cache has object_name as “B ” and r_folder_path as “/A/B “. Sybase stores them as “A” and “/A/B”.
3. Create another new folder “C” in the same session, link it to “B”, and save it. It fetches “B”‘s r_folder_path from the cache and generates its own r_folder_path as “/A/B /C”. Note the space after “B”, and also note that sybase doesn’t have anything to trim because the space is within the string.
4. Put some stuff in “C” and close this session.
5. Navigate down through the new folders in desktop client and “C” has content.
6. Navigate down through the new folders in webtop and “C” appears empty.

Both client applications use “r_folder_path” in the query to find a folder’s content. One remembers and builds the path with some whitespace trimming as it goes along, the other uses literal values from r_folder_path. Can you figure out which is which? That difference is the root of the disappearing content mystery.

The duplicate folders mystery arises from the fact that the DMCL won’t remove corrupt r_folder_path entries when relinking a parent folder triggers a cascade update on its children. That junk stayed in the r_folder_path of the moved “Reports” folder but didn’t block scripts from creating another “Reports” folder there since the validation is on i_folder_id. Both end up with the same corrupt path, as would a third after moving the second, and so on.

There you go. A combination of the DMCL cache, Sybase trimming whitespace, a DMCL application building several levels of folders, one of the non-leaf folders having trailing whitespace, and relinking predecessor folders causes data corruption that manifests itself in several different ways. Fighting other fires, Sybase strangeness, and the improbability of it all kept everybody guessing for years.

By the way, I couldn’t retrieve the API code from the client that demonstrates the problem, but savvy readers should be able to reproduce it.

What should developers do about this?

In custom DMCL code, either trim the whitespace yourself before setting the folder’s name or do a revert/refresh on each folder object right after saving it. I seem to recall that was advice often ignored in the early days of DMCL programming.

In Input Accel, if you can’t hack it with VB or something to do the same, it’s time for a dm_job running dm_fix_folders as often as necessary. Running it regularly would be a good thing for everybody to do actually.

I think the DFC didn’t manifest the problem, and that may explain why I couldn’t reproduce the problem with webtop or Desktop Client–that or those apps trim whitespace along with other client-side validations.

What should Documentum and Sybase do about this?

It’s probably unreasonable to expect Sybase to simply stop trimming whitespace since it may break many systems depending on the behavior. They could however add configuration options to turn off trimming at the database and session levels. Then Documentum could provide options for both on their end, maybe in server.ini. Some installations may depend on standard Sybase behavior when sharing or exposing registered tables to other Sybase instances, so a session option would be polite.

Until and unless Sybase gives the docbase server or DBA some way to deactivate the offending behavior, Documentum could do the same things a DMCL programmer would do explicitly in the DMCL. I still can’t stand the idea of an API trimming whitespace, so I’d prefer they force a cache update on a folder object after a save. The representation in the persistent store should take precedence over a cache to assure consistency with other processes that may have pulled a fresh copy of their own.

Does SQL Server have the same problem?

I never got around to testing SQL Server. Being a Sybase derivative, I wonder. Oracle definitely doesn’t have the problem, but I’d appreciate if somebody with a SQL Server docbase would test this out and post a comment.

Composer looks competently-enough done to be a welcome replacement to the persnickety Documentum Application Builder (DAB). The look and feel (and clutter) is pure Eclipse–no sexy Apple minimalist influence here. The win here is one-stop shopping: Developers won’t split their attention (nor workstations their resources) across two programs. That’s definitely a Good Thing.

Aspects in theory are great. They allow per-instance extension of both state and behavior, so objects get what they need when they need it. The reduction in type bloat alone is worth it. There’s no need for all those “optional attributes” just in case a particular instance becomes a document of record or gets included in a submission. Fabulous. In happy wonderful fantasy land, I’d scrap dm_document completely for a true lightweight object, then staple on versioning, lifecycles, web content, retention, and whatever as needed.

Aspects bring some real architectural benefits, too. Adding an aspect at runtime doesn’t require ALTER TYPE activity which can crap out the database on some really big (or really underpowered) docbases. Older docbases will also benefit from this easy way to add and remove new functionality without the potential dangers of hacking around the existing type hierarchy. Documentum architects now have a built-in way to do composition, a technique that is largely replacing inheritance in OOP design theory because it keeps the design open and adaptable.

Aspects in practice aren’t so great. My fear that aspects would use the DBOF model of requiring Java code client-side has been confirmed, although they made an effort to minimally support DQL since attributes are involved. (No word on DQL limitations imposed by aspects yet.) Turning the docbase into a JAR pusher has improved the deployment situation, but it’s still pushing functionality too far up the stack for my liking. More moving parts mean more things can fail, get out of sync, or confound.

This design flaw will persist for as long as EMC is swinging their Java hammer, even on screws like scripting. Perhaps EMC catching onto jython a few months back means they’re reaching back into the toolbox. Funny how they had to make a simple script look so big with all the extraneous print statements–very Java of them. If only there were another way to pass messages from a client to Documentum without the DFC. Hmmm.

Here’s what I’m thinking: All custom client applications and integrations should use DFS going forwrd–assuming it doesn’t have its own issues. DFC programming should be restricted to the uber-server, that big box drawn around the real docbase servers, webtop servers, DFC servers, full-text indexing servers, site caching servers, etcetera. Only push DARs–yes, they had to go there–to Documentum application servers. What goes on in the uber-server stays in the uber-server–including every single line of DFC code.

Aspects are too useful to discount for a few architectural missteps–as long as you’re careful where you walk.

Tech Deep Dive: Exploring Documentum Architecture — Leveraging Composer and Aspects
Available soon on EMC Events On Demand (developer access may be required)