Reporting from The Data and Demos Meetup …

Stormy weather again over The Inquirier Building

Members of the Inquirer staff gave presentations during the first half of the meetup about how collecting, analyzing, and visualizing data works in the context of a large newspaper publisher without a large IT budget.  Some of the larger stories can take 6-12 months and a sizeable team to complete, like a study of violence in schools or the first fact-based comprehensive analysis of the impact of Philadelphia’s real estate tax reassessment. The data end of that often involves long fights for access to information; getting the goods from a “Freedom of Information” request is rarely as simple as asking. Even  agencies dabbling in transparency muddle analysis by changing data formats with the latest fad, and some withdraw from the web altogether after feeling the sting of disinfecting sunlight.  That may be the case with detailed public data on fracking operations in Pennsylvania gone missing without warning or explanation earlier this year.  The data journalism on the Marcellus controversy to that point was shining an unfavorable light on something our new administration in Harrisburg certainly favors without the burdens of regulation, taxation, and transparency.

Having heard from the hacks, the hackers talked about their projects in the second half.  The theme was social activism through social media: Cost of Freedom is a fledgling website to help voters get Photo IDs in states where laws have changed to disenfranchise young, old, and poor voters; Lobbying.ph provides a human-readable experience for exploring the Philadelphia lobbyist data recently made available by the city; WhoPaid is a prototype mobile app using the Shazam approach to identify political ads and who paid for them by capturing audio snippets on mobile phones.  Several of these applications came out of Random Hacks of Kindness, a movement around “technology for social good” that sponsors contests and hackathons.  Seeing my neighbors doing good works like this rekindles the pride I felt when I first called the Birthplace of American Liberty and City of Brotherly Love my home.

Another Case of Cautious Optimism

I’m often critical of Philadelphia’s low-tech standing despite being one of the country’s ten largest cities.  Finding a job in the city for a person like me is remarkably hard, partly due to a bad corporate tax structure and partly from a long history of first-but-no-longer claims to fame.  The meetup itself was amazing in organization, content, and participation.  The Inquirer hosts are facing another potential change of ownership, one in a series of such events that has underfunded and understaffed the newsroom across the board.  Several other attendees commented on how Philly’s tech scene is growing so slowly.  We have interested, capable people with good ideas; what’s the missing ingredient?  The bright side here is these are just the people with the journalistic and technical skills to figure that out.

Previously on DCTM: The Next Generation

Yesterday’s Docbase

There has been a growing disconnect between how we design for the back end and the front end. The front end developer can apply agile methodologies, use new patterns like composition over inheritance, and organize data based on tags and facets instead of deep hierarchies.  Back-end developers are getting these benefits from new systems like NoSQL databases, but Documentum architects are stuck in the 90s with the current content server–in no small part because it runs on relational databases like Oracle.  Documentum did an excellent job marrying objects and tables before ORM (object-relational mapping) was a catchy initialism; the inherent disparity between the models that was inconvenient before, but now it actively hinders new ideas on the back end.  Basing NGIS on EMC’s XML database xDB changes all that, and traits are a windfall from that decision.

The latest details on traits comes from a video by Jeroen van Rotterdam, IIG’s Chief Architect.  There are two basic constructs for designing with traits:  Trait definitions group data definitions, services, and event models into packages that are attached to objects at runtime.  Type definitions control which traits can or must be attached to objects and how those traits resolve conflicts; e.g., order of precedence or mutual exclusion.  Objects end up being truly lightweight–only an ID attribute for sure and very probably a type attribute as well.  This will be a very different world than the present-day Documentum schema of data-only inheritance, tacked-on type-aware behavior, and a bloated base object type with every possible attribute stuffed into it.

For Documentum architects and server-side programmers, this is the first exciting thing to happen to Documentum since, well, forever.  The model is one of the things that made Documentum stand out; it painted a rich, complex picture of what a document could be. Instead of document as file, a document included versions, renditions, and complex structures with flexible binding (i.e., virtual documents).  A document was a different thing (or collection of things) based on context and function.  It also stepped beyond the straight jacket of the relational model to include multi-value (repeating) attributes and SQL extensions that recognized the world is more like objects than tables.  Although this perspective on document management is still valid today, the methodologies to realize it feel outdated. Traits represent almost two decades of lessons learned since the first docbase schemas struggled out of the primordial scanned-document/shared-folder ooze.

Changes of this magnitude don’t happen often in software systems.  A mature market doesn’t take well to fundamental shifts that aren’t backward compatible, so there’s significant risk here assuming EMC wants to keep its current customers happy. Discussion about migration paths from the current content server to NGIS are still little more then speculation. Given how many customers are turning to other options, being bold may be the solution if EMC can make a better product with a less onerous migration path.  Let’s put aside those messy details for now and consider what traits may mean for the Documentum architect.

Will this become my new Linkedin profile picture?

A  Few of My Favorite Things

It’s hard to say if the implementation of traits will live up to the potential from what we know so far.  I had similar high hopes for DBOF, light-weight objects, Aspects, and DFS when talked about in theory, but they all fell short in implementation either in outright design or by being born prematurely.  Letting my inner optimist out of his cage for a moment, these are some things I hope to see the implementation of traits bring about:

Attributes become rich data rather than columns in a database. Defining and constraining attributes using XSD is an easy win since NGIS sits atop EMC’s XML-based xDB instead of an RDBMS, and it gets rid of the stapled-on data dictionary in the current Documentum toolbox.  I hope this means that attributes can contain complex data–perhaps small XML documents and certainly associative arrays–but removing size constraints and character set problems (odd escape characters, special characters, unicode, etc.) is a huge win regardless. Maybe this will help people realize that attributes aren’t all metadata.  Sometimes, they are the data; e.g., non-content objects.

The broken promises of DBOF and Aspects can finally be fully realized.  A weakness in Documentum’s original model was the separation of data and behavior.  Being object-based, there were familiar ways to organize data (e.g., inheritance) that fell short of real OOP because they didn’t equally apply to behavior.  DBOF was the first attempt to relate code to the type hierarchy; it was hobbled by being duct-taped onto the side of the client libraries (DFC) instead of integrated into the server directly.  Then Aspects repeated DBOF’s same fundamental mistakes.  Traits appear to marry data and behavior nicely, and I can’t imagine them being handled anywhere but together on the server.

Traits eliminate the need for inheritance and fat objects.  Inheritance as a design principle has been under fire for years:  Single inheritance is restricting; multiple inheritance introduces pitfalls along with greater flexibility; both are difficult to refactor.  The single inheritance nature of the Documentum object model wasn’t as restrictive then because it only included data.  With behavior coming along for the ride in NGIS, it becomes a bigger issue.  Just like in programming, it looks like EMC is favoring composition over inheritance.  I haven’t seen anything showing inheritance for types or traits in NGIS yet, and I won’t be surprised if it doesn’t.  Another consequence of this composition-based approach is the object becomes really, really lightweight–little more than identity on one end and trait container on the other.  That looks surprisingly like the really, really lightweight document in MongoDB with it’s single _ID default attribute.

The Documentum architect gets a whole new (NoSQL) toolbox to build and extend systems. Although Documentum grew out of the relational database culture of the time, it broke major relational conventions because the document management problem space felt more object-oriented than table-oriented; i.e., repeating attributes, virtual documents, object-ish extensions to SQL.  Now that NoSQL is here, it’s obviously a better fit:  Documentum architects taking a look at 10Gen’s webinar on schema design in MongoDB should quickly recognize the familiar ground.  What’s new is how dynamic, real-time the data model becomes with traits because they are only attached as needed at runtime, and objects can have different versions of a trait at the same time allowing for upgrades in a conditional or rolling manner.  The docbase will no longer be where content goes to die; it becomes a Livin’ Thing. Hopefully sizing, deploying, and scaling NGIS systems will also look more NoSQL than relational, but traits don’t give us any hints there.

Me if dm_history repeats itself

Cautious Optimism

I’ll admit to some cautious optimism here; traits may be a sign that NGIS will make Documentum architecture interesting again.  Beyond just updating the toolbox, the new capabilities may inspire organizations to start solving new, interesting problems again. Documentum did a good job of getting people to think about document management the first time around, but now everybody “knows” what document management is and what Documentum is “good for”.  Traits hint that NGIS could be a game changer on the much-less-talked-about back end; it won’t directly delight the New User, but a pretty front end on a shaky server foundation won’t be particularly useful to the New User. It’s up to EMC to get the implementation right and get the word out.

Based on the blog post Next Generation Information Server: Traits Explained | Jeroen’s Crazy Content and video:

Reposted from Google+.

I’ve been satisfied with the free account so far but haven’t taken Evernote to the next level because tools like OmniFocus, VoodooPad, and Google Docs are more integrated into my daily routine. That might change a little thanks to The Evernote Podcast. A podcast about a single app? I couldn’t imagine having enough to say about Evernote in a long-form podcast; something short form like Quick and Dirty Tips might make more sense. Then again, a show about nothing could be entertaining from time to time. I gave it a try, and it paid off.

Podcast #10 mentioned how to send blog posts from Google Reader to Evernote, a way to archive entire articles without leaving Reader. I live in Reader, so the odds of me using Evernote more–and listening to more Evernote podcasts–just shot up dramatically, especially because I added The Evernote Blogcast to Reader. The feed includes links to the podcasts, so I’ll probably delete the iTunes subscription and listen from the web when a summary catches my attention.

The podcast also mentioned using Evernote data as a screen saver since each note has a thumbnail image rendition. It’s not terribly practical since I can’t imagine anybody (sober) watching their screen savers anymore; flying toasters are so 20th century. Still, it’s a neat hack, and chance picked a relevant reminder about last week’s gay rights victory in New York and tomorrow’s distinctly Philadelphian holiday:

The only freedom which deserves the name is that of pursuing our own good, in our own way, so long as we do not attempt to deprive others of theirs, or impede their efforts to obtain it.

– John Stuart Mill

How to use Evernote as a Screen Saver in Mac OS X:

1. Open “System Preferences”, choose the “Desktop & Screensaver” icon, and choose the “Screen Saver” option in the tabby thingie.
2. Click on the “+” under Screen Savers pane and choose “Add folder of pictures”.
3. Navigate to “{HOME}/Library/Application Support/Evernote”, choose the data folder, and leave that folder selected in the Screen Savers pane.
4. Choose the middle option from the Display Style option bar.

GooglePlus is dominating my RSS feeds today. My excitement about Circles has faded a little because of articles like this:

Google Plus’ Circles System May Not be Sustainable — ReadWriteWeb

I posted an abridged version of the following in the article’s comments:

Good categorization is an expertise most people lack: “Work” isn’t particularly meaningful, but “Coworkers”, “Headhunters”, and “Professional Acquaintances” are. It’s more work to apply and maintain a richer taxonomy, and I’d imagine even fewer people will get equivalently greater value from such effort in social networking. We’ve been trained away from finding exactly what we want by the search-and-browse approach of unstructured searches like Google, so wading through irrelevancy is a more common skill.

Grouping mechanisms in other social networking systems also have upkeep problems; I suspect most people just don’t bother doing it, and the same will probably be true with GooglePlus. Maybe adding a feature to display circles as Venn diagrams would help data geeks like me.  For most people, public versus private might be just enough categorization to avoid social networking faux pas without making the posting process feel like taking the SAT.

Existing social networks don’t have the concept of categorization on both ends, posting and reading, and I wonder if GooglePlus Circles has the same deficiency because it hides the names of circles I’ve been added to. I’d like to subscribe to (and filter out) people’s circles instead of people themselves to control the noise of their posts about unshared interests; I’d probably disagree with how most of my non-professional acquaintances would categorize me. It doesn’t sound like GooglePlus makes the distinction of subscribing to people’s interests or topics instead of people themselves. Relevancy has to be a two-way street.

Or, perhaps, a four-way intersection.

GooglePlus Venn: Relevancy is an Intersection

With all those brainy data geeks at Google, I’m optimistic that they could create a Venn diagram of Circles, Sparks, +1′s, Reader Likes/Shares that could define social graph relevancy in that oh-so-Google statistical way. How many licks would it take to get to the center of that tasty relevancy tootsie-pop?  Lots, probably, but it creates a new kind of payoff that Twitter and Facebook are incapable of delivering.

Hey, Google, are you listening?

On a related note, I tried posting my comment using my Twitter account. Twitter on the web is totally brain-dead about multiple identities, so of course I posted my comment using the wrong account. It’s a perfect example of the mis-post problem I referenced in the preceding post. Then I reposted my comment using Google, and Disqus allowed me to choose from the three Google accounts I’m logged into right now. Google gets the issues of identity and context more than anybody else in the field, so I’m (somewhat) optimistic about GooglePlus.

Official Google Blog: Introducing the Google+ project: Real-life sharing, rethought for the web.

This isn’t just about hiding potentially embarrassing facts from prospective employers; it’s about targeting content to the audiences that are most likely to find it interesting. Current social media systems like Twitter and Facebook don’t get this. With Twitter in particular, I try to work around it by having a half dozen Twitter accounts. I restrict who I follow and what I post by account based on theme–personal, professional, gaming, etc. Hacks like this make for more work and are prone to mis-posts; it’s as discouraging to posting as wading through live-tweeted baseball games or diaper anecdotes are to reading.

Identity is also an issue as services like Facebook and LinkedIn expose our real-life names to the virtual world, something I experience more acutely because of this eponymously-named blog and professionally-oriented Twitter account.  I can’t prevent noise in my professional channel no matter how clever and diligent I am when less savvy friends and relatives can’t remember to use my personal non-eponymous identity for personal messages. Social search, realtime results, and consolidated logins will make this everybody’s problem in a few years.

Google’s social networking track record isn’t great; i.e., Orkut, Buzz, and Wave. It looks like Google+ doesn’t suffer from the lack of look-and-feel sophistication that may have hampered earlier efforts, and features like Circles address some of the fundamental design flaws in established products. However, the better product doesn’t always win, and Google will have to convince people to leave existing services. That’s a Catch 22 because the value of a network depends on its size, and it’s compounded because members of those networks don’t understand issues of identity, privacy, and relevance.

Call me cynical, but I think the odds are stacked against Google+. How many people realize the value of regular backups before losing everything to crashed disks or lost laptops? Those same people won’t realize why leaving Facebook for Google+ makes sense until they lose jobs or spouses for lack of caring. Please, Google, prove me wrong.

(Full article on microsoft.com)

The is a logical evolution of the one thing I like about Windows 7 and use regularly, window snapping. It’s also one step closer to a desktop that works like the Eclipse IDE with various resizable panels for different elements like object browser, code editor, and compilation messages. Apple is also taking a tablet-inspired approach in Mac OS X Lion. It’s about time operating system vendors step up and do something about the sad state of application window management, but I don’t expect Windows 8 or Mac OS X Lion to bring perfect solutions because of their inspiration: Tablets make great content consumption devices, but I need something inspired by a content creation tool as my desktop.

In the meantime, I’m using a little open source app called Shift It to get something like window snapping on the Mac. It uses keyboard shortcuts rather than the Windows-style border collision which the Mac already uses for virtual desktop management, Spaces. Here’s my somewhat-Eclipse-inspired desktop with the Shift It menu exposed:

Dock and Twitter on one side, desktop files on another, and a perfect half-width browser window (with too many tabs as usual) in the middle. Sized and positioned Chrome by shifting it left then center with Shift It.

Shift It isn’t a perfect solution. It has some problems with window size or position being a little off sometimes, and the top/bottom options are practically useless on a widescreen display. Most people won’t mind, but my particular pathological need to organize (also expressed by my obsession with The Container Store) requires precision and symmetry and flexibility.  Steve Jobs has a similar affliction, so I’m hoping Lion will be like digital Paxil for my application window management OCD.

Don't google "dead horse" for images.

My response to Pie’s Quality of Documentum Over the Years bears repeating, even if I am beating a dead dmHorse:

I started with version 2, back when I was just a newly-minted UNIX geek. One thing you missed with the transition to 4i was the introduction of the DFC. DMCL had a very UNIX feel; a simple, open API designed to be glued into any programming language. DFC was just Java then, with a COM layer growing over it later. That was also the point where EMC became more marketing-driven and started chasing the Internet bubble at the expense of their existing clients.

Both were attempts to capitalize on hot topics of the time, Java and the Web. I never bought that the DFC would make a whole pool of talent available; Documentum’s about the model, not the means. However, the marketroids successfully reframed it. Hiring managers now believe they can take Java people and mold them into Documentum people, and I hear gasps of disbelief when I say Java or Visual Studio aren’t requirements to do Documentum–a good Java programmer is not necessarily a good Documentum developer.

This Java mentality did increase the number of people with Documentum on their resumes, but the talent didn’t increase as much as the volume. It just diluted (maybe also tainted) the pool. It became harder to find good people in the now-mirky waters.

The lack of focus then is what brings us to the lack of quality now. Innovation at the model and server level is rare, and frankly I don’t give Documentum much geek cred anymore because of it. Great ideas like BOF and Aspects are stapled into an API rather than made an inherent part of the product. Too much work up the stack (and on vertical solutions) has made the product top-heavy and tottery. EMC continues to chase markets (i.e., case management) rather than concentrate on making a solid core product.

Noted rumor-monger  Brilliant Leap speculates about Documentum in a world without EMC. Tongues were already wagging at EMC World about the SAP-EMC partnership leading to something a little more intimate. It has enough of the smell of truth to make an irresistible rumor.

As rumors go, I still prefer the Microsoft angle because of the obscene anatomy kissing that IIG is still doing.  Truth is it’s just to easy to dispel: Why buy the cow when you get the milk for free? I doubt Microsoft will be stamping shrink-wrapped boxes of SharePoint with Documentum Inside! anytime soon, but I’d bet they have an infinite number of code monkeys banging away to make their own document management Hamlet. Once they do, it’s bye-bye Documentum! Then all those monkeys will get down to business and start flinging feces IIG’s way.

Part of Documentum’s doom was being a software company bought by a hardware company; however, it won’t be saved if another software company buys it next. Such companies (SAP included) would buy Documentum to augment their flagship product, not eclipse it.  With no Fairy Godmother rescue from being passed around from one wicked step-mother to the next, this story’s ending will be more Le Boheme than Cinderella. Or worse, a more-jackal-than-wolf company that hasn’t innovated for decades might gobble it up to suck the last trickle of marrow from its cracked bones.  *cough* computer associates *cough*

I am no Wall Street cheerleader, especially after my time in Big Finance, but the closest thing to a Fairy Godmother out there is a technology-oriented private equity fund. Such a fund buys troubled companies to turn them around and sell them for a profit. Unlike most of Wall Street, they take the long view of years rather than a quarter or the milliseconds around a stock’s uptick.

Their methods can be harsh, but their goal unlike any step-mother’s would be to make Documentum the best product and most profitable (and saleable) brand it can be.  There may still be an ounce of brand left to save. By going private, the recuperating Documentum wouldn’t be burdened with public company regulation or the tyranny of speculative stockholders. It’s an imperfect cure for the age of gratuitous IPOs and acquisitions fueled more by irrational exuberance than smart business.

We have a test case with AOL selling Bebo to Criterion Capital Partners, LLC instead of just shutting it down. Taking Valdes’s animal shelter metaphor a little further, I’m sure Criterion will euthanize Bebo and reap their own “meaningful tax deduction” if the old dog can’t learn new tricks. Sometimes I think I’d rather see that happen to Documentum than sit through the EMC’s little opera until the consumption takes it.

I can’t find a video of the Mark Lewis keynote from EMC World 2010. Instead I’m depending on reporting from the event like Ron Miller’s article [Documentum group gets new name and new direction] and Pie’s tweets and blog posts. It’s probably for the better; I never had a taste for gruesome videos since Faces of Death, and this may be EMC finally decapitating the Documentum brand.  Rather than plunging into a pages-long diatribe about EMC’s unconditional surrender to the commoditizing of content management or the latest dish of scorn Lewis served up to Documentum veterans, let’s talk names.

Information Intelligence Group is EMC’s new moniker for the product that shall not be named. Lewis breaks down the name for us on his blog [Episode 91: EMC World 2010 - The Birth of the Information Intelligence Group]. It’s hard to read–let alone say–the name with a straight face, and this breakdown doesn’t help. At least the cumbersome and uninspired Content Management and Archiving accurately conveyed something about the product.  This new name is too broad and inherently meaningless; it will continue to erode mindshare for a product that was the de facto definition of document management. Let’s hope this new name doesn’t prove itself a compound oxymoron to boot.

The “Intelligent” product silos aren’t much better. Granted, this is a product that has to publish a separate guide with each new release to map old product names to new. Not a sheet or a few pages, a document. However, these new silos are so vertically restrictive that EMC had to toss the content server into case management.  Having done case management and having paid my dues in lines of server code, I’m perplexed. It’s like they had a very limited box of magnetic poetry to play with.

The continuing erosion of a strong brand means less mindshare among potential customers. Everybody knows SharePoint even though most don’t know what it really is. I’ve seen first-hand how good marketing trumps good product. Documentum had that name recognition–still does in many parts–and EMC seems determined to stamp it out without something sticky to replace it.