The True Nature Of Beautiful Perfection

I’m always pleased when I can build a nice clean system for a customer.  I like to be able to look back and say, “That is beautiful.  I’m proud of that.”  Antoine de Saint-Exupery said, “A designer knows he has achieved perfection not when there is nothing left to add, but when there is nothing left to take away.”  I like that quote and try to build systems with that in mind.  I like to be able to simplify a customer’s business process so it makes more sense to them that it did before.

But there is a danger here that needs to be avoided.  We are not dealing with art or literature.  We are dealing with business systems, systems that receive input from untrusted sources.  This data needs to be checked.  Joel Splosky puts it best when describing things you should never do.  He talks about old code that “has grown little hairs and stuff on it and nobody knows why.”  He’s not describing something beautiful; he’s describing something that works.  Something that’s gone through the pain of having exceptions found and dealt with.

Exception Processing

We help customers automate their business.  The software products we sell all have a type of workflow build it.  Oracle IPM and Liquid Office have a true workflow component where you build your processes graphically.  ILINX Capture and Kofax Capture have the idea of queues and the ordering of queues.  Systems with combined software are generally designed to be used in stages such as scan, store and retrieve.  In each stage the data is moved from one queue to another or one piece of software to another and the data needs to arrive correctly.  These are the types of system interactions I’m focusing on.  Unfortunately, these systems aren’t always configured perfectly and something will happen.

Your scanned document will be unreadable.  Your form won’t be filled out completely.  You’ll have a power outage.  The database you rely on will have bad or missing data.  You network connection will drop.  You’ll need a strategy for handling these.

If you don’t handle exceptions well, your system will be consistently inconsistent.  Approvals will not be done on time.  Documents may be hard to find.  Users will become frustrated with extra unneeded steps to complete work.  Or worse, no one will notice until a year later and document XYZ is not in the system and the auditors that are in the room with you breathing down your neck lose their patience and your about to be sued.  You need to be able to handle these exceptions.

What An Exception Is Not

Just to be clear about the types of things I want to focus on, I’d like to list a few things I’m not talking about first.  When I say exception, I do not mean anything that can be considered a part of normal business processes.  For instance, in an accounting workflow, it might be part of the process to send invoices over $1,000,000 to a manager for special approval.  This is not an exception.  While the amount may be exceptional, the approval is a defined business process.  Conditional queues in your business process are also not exceptions.  If it is normal for a form to be missing specific data and there is a special queue for dealing with that, it is known and a normal part of the business processes.

What An Exception Is

Exceptions are when we are dealing with the unknown.  Exceptions are typically something that requires human intervention, special research or a new decision to be made that hasn’t been made before.  Exceptions are also when automated parts of the system suddenly stop doing the things they are suppose to.  (Actually, those are sometimes just programming bugs, but you need to handle those too.)

How To Handle Exceptions During Development & Testing

• Build maps and scripts that contain catch all routines. If you have a workflow script that needs to interface with an external system, make sure you can accurately report that the system is operational, that you can connect, that you can do the tasks you need to and that it will return successfully.  All of these steps may need their own error handling code so you can accurately report the problem.  You may also want to catch other exception outside the known ones so that you can log and report on them.  Workflow maps contain exception processing within them.  Most queues can return with an error, which can then be routed to a special queue.  Be willing to add those exception queues.  Allow users to be able to route their data to these exception queues with explanations of the issues.  They may not be used much, but you’ll be thankful there were there when something happens.

• Have good metrics and logging. The software we sell and implement already has logging features.  Those features need to be turned on.  When designing systems that have custom components, make sure they include a logging feature.  You will also need to make sure the data that is collected can be easily read.  This means investing some time into reviewing available reports or creating new reports.

• Include automatic reporting. Consider an automated way to flag problems or issues that can be detected.  Emailing a business administrator or system administrator is a good way to alert people of potential issues.  Be careful with this, however, as you can have too much.  You want your noise to signal ration to be low.  The last thing someone wants to do is to read through hundreds of automated emails because perhaps one indicates an issue.  (There is a reason most people turn off the Vista UAC.  Allow? Deny?)

• Test with real data. I’ve seen engineers send the same 5 documents through the test system thousands of times and declare the system is able to handle thousands of documents.  No, the system is able to handle those 5 documents.  That’s slightly different.  You need to test with thousands of different documents.  That is where your exceptions will be found.  You also need to test with real documents.  When some fills out a form for testing, of course they are going to fill it out correctly.  The programmer wants to make sure his code can read each and every field.  But what happens when they are not all filled in?  What happens when the data being used as a unique index is not there or not unique?  You need to test to find out.  Fake data will only test the common paths.  Real data will test for exceptions.

• Test with a lot of real data. I feel the need to state this twice.  Exceptions happen when users are using the system.  Use the system as much as possible with real data before you go live.

How To Handle Exceptions During Production

• Expect to find exceptions. Make it part of the production process to spend time looking at the log files and dealing with issues.  Every project our company manages includes what we call “rollout support.”  We know that exceptions are going to happen and we plan for the time to deal with and fix them.  Have someone whose job is to handle the exceptions from a business point of view, not just a IT administrator.  Give them the time required on a weekly basis to make sure the system is running smoothly, and that the users are happy with the way the system is running.

• Be willing to change your process. After an issue occurs a few times, it may make sense to consider it something that should be handled as normal processing.  Be willing to review the workflow map or order of queues.  Those conditional queues I mentioned earlier, the ones I said weren’t exceptions?  Those are really exceptions that everyone already knew about.  Now that you’ve found and know about one more, add it to the normal business process.

• Focus on the common problems. It’s okay to think, “This is the first and probably last time I will have to deal with this.”  As soon as the second time occurs, you need to plan to deal with more exceptions.

What To Take With You

By now you can see that your system is going to encounter exceptions too.  Your not alone.  We all have to deal with them.  You can see the importance of thinking about them, looking for them and dealing with them responsibly.  If you system is still being developed, you have the knowledge to be able to plan for them.  If you system is implemented, you now have good reasons to review and change it for the better.

I hope you can look at the design of your system and say, “The workflow map, scripts, queues and configuration may not be beautiful, but the system handles everything beautifully.”

Scott Hamilton
Senior Developer
ImageSource, Inc.

[photo by wikipedia user Patriarca12]

ECM Best Practices: Document Capture & Metadata Collection

The term “Data Strategy” and can be used to understand strategic requirements for enterprise metadata concepts and best practices.  Why is this important for an ECM solution?  The simple answer is why contribute to the disarray of an organizations data management when implementing an ECM solution.

The key to a successful ECM system is the ability for users to easily and quickly find information within the repository.  If specific information cannot be easily and quickly found, users may not be enthusiastic about using the system.

The biggest stumbling block in the way of finding specific or relevant information is when the search returns too many documents that the user will have to physically look through before finding the specific document or set of documents that exactly meet their requirements.

In order to prevent this impedance to effective document retrieval, organizations have to be able to refine the population of documents that are searched so that irrelevant information is not selected and returned in the query result set.

In a well designed ECM system, this is accomplished by using metadata (or taxonomy’s) to limit the search result set to items which meet  the user’s selected parameters (i.e. date range, case number, document type, etc.).  Metadata is used to describe content that the ECM system intakes.

Metadata not only provides a way to index (and therefore retrieve) content but also it provides the means of managing content throughout its lifecycle.  For a document or item of content, this means data about it such as its author, its title, the issue date, and other information which can usefully be associated with it.

Whether you are using ILINX Capture, Kofax Ascent Capture, or Captovation users will have to enter some metadata, though as much as possible should be created and collected automatically if possible given the document sources, types, and available databases linked to documents.  It is important to get the balance right, too few mandatory elements may result in little metadata being entered, too many mandatory elements may be seen as a tedious chore.

Consistency and accuracy of metadata values is crucial to the value of metadata.  Controlled vocabularies, pick lists, default values and inheritance are all important tools and techniques in this context.

When designing a system the recommendation is to apply metadata to all documents that are expected to be found easily in a query and all documents that are stored as “final” archived records in the ECM repository.  This process will be essential in the user adoption and success of the system.  For more information please visit the North West’s premier conference at http://nexusecm.com and register for one of the ECM break out sessions.

Jon Sutherland
Sr. Systems Engineer

ImageSource, Inc.

 

 

 

  

Distributed Capture for the Enterprise

Distributed Capture (for Scanning and Indexing) has been gaining ground in the last several years.  Used to be that documents were sent to the basement where a dedicated scan operator “fed the dragon” by scanning hundreds if not thousands of documents a day.  This was known as “Centralized Capture”.  Problem was, how to relay the vital index information necessary for search and retrieval to the scan operator?

The last few years have seen technology leap forward with the advent of inexpensive “desktop scanners”, browser based software such as ImageSource’s ILINX Capture http://www.ilinxcapture.com/index.htm and integration tools that “image enable” or “document enable” the existing Line of Business Systems used by most all departments such as ImageSource’s ILINX AIK Toolkit.  Line of Business Systems are those account driven applications dedicated to the function of a department.  SAP, Oracle Financials and JD Edwards for Finance.  Salesforce.com, Great Plains and Siebel for Sales.  Peoplesoft for Human Resources… just to name a few.  Since these systems manage the events that create documents, why not enable them to manage the documents directly.  The productivity gain may not be so obvious.

Most of us associate document enablement with the immediate benefit of displaying a scanned image on the screen with the click of a button in their line of business application. Productivity gains result from not having to perform a secondary search in the document management system for the documents that match the record in their Line of Business application.  This in itself is a powerful capability.  But what gains can be realized on the capture side by reversing the process in a Distributed Capture solution that uses desktop scanners and Line of Business Applications to index the documents into the system?

Here’s some real benefits to distributed capture:

1.) Distributed Capture eliminates the need to transport documents from the cubicle farm to the basement.

2.) Distributed Capture eliminates barcodes/patchcodes necessary for the proper functioning of a “batch oriented” centralized scanning system.

3.) Distributed Capture eliminates the need for a dedicated scanner operator and/or scanning department.

4.) Distributed Capture can capitalize on SDK toolkits that integrate line of business applications for more accurate indexing.

5.) Scanner costs can be reduced by relying on inexpensive desktop scanners rather than “the big iron” in the basement.

Distributed Capture used to be an unrealized capability in the enterprise.  Today, with more advanced scanner hardware, more technologically “in-tune” workers and more advanced toolkits for document management……….. it can be a reality.

  

SharePoint for the Enterprise

Microsoft has created a major presence in the enterprise content management arena with Microsoft Office SharePoint Server 2007.  Is SharePoint best suited to serve as a single ECM solution for the Enterprise?  The real question is what does it take to have a successful SharePoint implementation.  From my perspective, here are the things one should consider:

• Implementations must be carefully planned. Incorrect installations lead to a lot more time and money spent later on down the road to fix the problem. Where are the cost savings then?
• Tame the data chaos by defining document libraries based on content types. Be sure to not use a one solution fits all approach when defining metadata and requirements.
• Administration is critical from an IT perspective in terms of its support, maintenance, and updates.
• Have a roadmap, design documents, and a detailed project plan that defines roles and responsibilities, along with a risk assessment.  Don’t try to do it all at once and make sure everyone knows the plan and the timelines.

Organizations such as ImageSource are using SharePoint for managing active electronic content and supporting collaboration.  SharePoint can be easily used as a bolt on to existing ECM systems like Oracle IPM (Imaging and Business Process Management).

The value of implementing a duo like this increases exponentially with the ability to store fixed paper based content with in the Oracle IPM content repository and to leverage the SharePoint repository for your active electronic content and team collaboration portals.

Jon Sutherland
Sr. Systems EngineerImageSource, Inc.

 

  

Managing Multiple ECM Systems at an Organization

A trend that I have been noticing more and more is the presence of multiple Enterprise Content Management (ECM) Systems at organizations.  There are many reasons why the scenario of multiple ECM Systems can occur; 1) mergers & acquisitions, 2) strengths of the ECM products, 3) lack of internal communication and/or understanding of existing systems in the organization, 4) division of dollars at departmental levels, and so forth and so on…

One issue that can arise from having multiple ECM solutions within an organization is the difficulty of accessing information stored in the different systems.  Let’s look at an example of an organization that has two ECM systems used for content archiving and retrieval.  The first ECM system is used by the Human Resources department for storing documents and data related to employee on-boarding and personnel management.  The second ECM system was installed at a later date by the Accounts Payable department.  This system is used for managing documents related to payroll, as well as, acting as portal for employees to find information related to their pay.  The presence of multiple systems within the organization creates a number of headaches (i.e. managing of storage, security/permissions, what information is where, etc…), but for now let’s focus on the fact that there are users that need access to the documents and metadata stored in both systems.  The user’s have the ability to log into the first system and run a search to find HR documents, and then to find Payroll documents they have to switch over to the second system, log in, and run another search.  This requires the user’s to have knowledge and an understanding of both systems to perform their job functions, and the laborious nature of this task creates inefficiencies within the departments.

There are a number of ways to get around the issue of having documents and data, that user’s need access to, in multiple locations:

• Consolidation of the systems.  This can be an arduous task, but in the end there will be only one system to manage.  Consolidation is most commonly the option when multiple ECM solutions are the result of mergers & acquisitions.  This may not always the best solution because of business requirements and product strengths.
• Link the data and documents between the systems.  This functionality varies between products, but a good example of how this can be done is through the utilization of the ILINX SharePoint Connector software.  This software gives users that ability to search multiple content management systems through SharePoint.  The users have a single access point to all ECM related content using SharePoint, which alleviates the required system knowledge and consumption of time associated with searching through multiple systems.
• Leveraging the investment in one of the ECM systems.  Almost any top tier ECM system can be used to retrieve content from an external system.  For example, using Oracle Imaging and Process Management linked servers can be created to access content from external data sources.  Keep in mind that most native functionality for linking content within ECM systems has limitations.  When this is the case the use of middleware products, like the ILINX AIK, can alleviate these limitations.

 It is rare that companies have the horsepower to take on these tasks themselves, so there are solution integrators out there, like ImageSource, which can assist with this work.  Advantages of going with an experienced ECM integrator are that they can evaluate the current business requirements, assist in streamlining the current process, provide recommendations (if needed) on optimizing the solution, and provide risk mitigation throughout the configuration/redesign process.

The utilization of multiple ECM solutions within an organization can be a cumbersome endeavor.  By evaluating the overall business requirements and optimizing the organization’s ECM architecture these pains can be easily overcome and will make Enterprise Content Management a more valuable asset to the organization.

 

Ryan Keller
Project Manager / Sr. Systems Engineer
ImageSource, Inc.

 To find out about the latest trends in the industry and interact with industry experts, businss leaders, and the user community –> Check Out NEXUS 2009!