Testing, Testing, and More Testing….

You have spent a lot of time and effort to Design and Implement a system for the capture and processing of paper and electronic documents.  You have performed testing to insure the implementation works as designed.  Now comes the intricate part of BETA testing by selected End Users to verify the system works as designed and more importantly, performs the tasks the End Users need.

BETA testing is often a long and arduous process especially with a large complex system.  There can be thousands of test points that need to be checked and approved.  Items like document types that can be used for routing and/or different index structures all need to be selected and tested individually to make sure they are routed or indexed properly.  Other items can be things like updates to the Host ERP system where multiple status/process steps must be again tested and approved individually.  Preparation, performance, and review of the results can take a significant amount of time for each section that is tested.

Many companies underestimate the BETA acceptance testing process, especially if they have had little experience in the acceptance of large complex systems.  It is extremely important to emphasize to project teams the importance of the BETA acceptance testing and the time it will take to verify all the little features/functions that need to be tested.

The company should also recognize that the BETA testers must be given the time to perform the testing.  Some companies only allow an hour a day for a person to perform BETA testing.  This means that the BETA testing period will be extended in terms of when the BETA testing will be completed.  Also, BETA testing provides end users the opportunity to learn the new system interface and what it can and cannot do.  This is a very valuable part of the process as it builds knowledge for the company on the new system and is transferred directly to production.  These are just a few of the reasons not to skimp on the time given to BETA testers. 

In the future, when deciding on how much time to give, error on the conservative side and give them all that is necessary. Three to four hours a day is not too much to ask when it comes to an assignment. This will improve not only the time it takes to complete the project but provide a very large hand up on how the system works and what it can do.

 Chris Hillenburg

Senior Systems Engineer

ImageSource

Liquid Office to Autonomy Process Automation (APA)

Liquid Office has changed its name to Autonomy Process Automation (APA).  The version number is at 7.1 and 7.2 is due out soon.  It has a new feature than makes hooking a workflow process to an existing HTML form very easy to do.

Assume you had a form created with the following HTML.

 

 

This form has no links or hooks into APA.  It was created outside of the form designer.  Using the new eclipse plugin, I can create a *.dfd file and hook the form into APA.  Here are some of the screen shots of the tool and the file it creates.

JavaScript that hooks into existing form methods is stored as CDATA inside the dfd file.

Metadata can be defined and used later in the process scripting.

The eclipse plugin will communicate back to the main APA server and make sure the server side fields match the new HTML form fields.

Fields are mapped from the HTML to the APA fields defined on the server.

The following is the dfd file that is produced.

Using this new feature of APA, existing forms can now have workflow added to them.  Branding an online form is no longer a task of wresting with the HTML produced by the form designer.  This is a step in the right direction for any organization wanting to embrace eForms.

Scott Hamilton
ImageSource Inc.

 

 

 

 

 

 

IBM FileNet P8 Links

Looking for some useful links to information in regards to the installation and configuration of IBM FileNet? We have been deploying successful IBM FileNet implementations and will be providing useful information, as well as, tips and tricks on this blog.

Here are the main links that have been leveraging for some valuable information:

IBM FileNet Description	URL
P8 v5.1 Information Center	http://publib.boulder.ibm.com/infocenter/p8docs/v5r1m0/index.jsp?topic=%2Fcom.ibm.p8.relnotes.doc%2Fwn_cm.htm
Documentation	http://www-01.ibm.com/support/docview.wss?rs=3273&uid=swg27010422
IBM Downloads	http://www-947.ibm.com/support/entry/portal/Downloads

 

Ryan Keller

ImageSource, Inc.

 

IPM 10g is Going Away, Now What?

Earlier this month ImageSource hosted our annual ECM conference Nexus.  I had the chance to meet with many of our customers and have some really great conversations.  Many of these folks are running IPM 10g as a core component of their enterprise and since that product is being end of life’d they are taking a long hard look at their installations.  Everyone wanted to know what options they had and to talk about the best way for them to move forward.  The 11g version of IPM was frequently a core topic of conversation and everybody wanted to hear how to get from here to there.

This turns out to be a hard question to answer! There is no single bullet-proof solution that you can just pull the trigger on and have everything completed.  (Sorry, we’re still working on the ILINX® Magic Wand)   Depending on your environment and your goals there are different considerations.  Let’s take a look at a few common scenarios.

IPM 10g, Imaging Only

So in your environment you’re using IPM to store and retrieve images.  You’re not using workflow, you don’t have many complex integrations, IPM is basically acting solely as a repository.  You are in a great position for an upgrade to IPM 11g.  Unlike folks with workflows and complex integrations there is a clear and defined upgrade path for an Imaging-only IPM 10g system.  Oracle provides a migration tool that performs the actual transfer of documents (with annotations!) from 10g to 11g.  The upgrade is mostly a matter of setting the applications and searches up in 11g and letting this tool move the content over.

IPM 10g, Process

Your environment utilizes workflows to streamline your business processes.  All your content can be moved over no problem as described previously in the Imaging-only section but what about all these workflows?  Sadly there is no way to directly move these from 10g to 11g.  11g workflows utilizes technologies like BPM and BPEL which are architecturally far-removed from the Process engine in 10g.

Not all is lost however.  While not being built on the same technology, there is a strong mapping between the older 10g workflow and 11g workflow (especially when using BPM).  The workflows won’t have to be redesigned just reimplemented.  Much of the work in creating workflow is capturing the business process in the first place, implementing the design is in many ways easier.  Plus, this is a great time to take a look at your workflows and see which ones you are still using and which ones could do with some updates.  IPM 11g offers much more advanced workflow functionality and this is also the perfect time to add modern workflow features into your own workflows.

I’ve talked a great deal at Nexus and with customers about the benefits of 11g workflow.  If you’d like to learn more leave a comment or email me and we can talk about some of the features that can really add alot of value to your workflow business process.

Migration to Another Platform

Lastly since the switch to 11g is more of a migration rather than an upgrade now is a time to consider implementing other systems.  Many of our current IPM 10g customers are looking at a product of ours called ILINX Content Store as a replacement option.  IPM 11g is really geared towards the enterprise and everything from server hardware to licensing is based on that assumption.  But if you are using IPM 10g at a departmental level, or just as an imaging repository 11g is in many ways overkill for your needs.

ILINX Content Store is focused on easy and effective content management  with strong Microsoft Windows integration.  It’s built on modern technologies and  encapsulates our years of experience in the ECM space to provide an intuitive, easy to use and administer solution.  It is a natural replacement for an imaging only IPM 10g system and most end-users find Content Store’s user experience easier and just as powerful.

If you’re using a lot of workflow though your options drift back towards the enterprise.  Many of our customers are making Sharepoint a core component of their workflow strategy.  We are also exploring IBM’s P8 family of products (which live in their FileNet umbrella).

Wrapping Up

These three scenarios are the ones that I see come up again and again.  But every environment is different.  Your needs and requirements are going to be different.  If you have further questions I encourage you to leave a comment or email me and we can have a great discussion about where you are today and where you want to be.

Les Harris
Systems Engineer
ImageSource, Inc.

KTM TDS Model Building

Are you tired of separator sheets?  Tired of wasted paper and countless hours of flipping through pages and inserting a barcode sheet at the start of a new document just to take it out after the batch is scanned or leave it in the batch and have more paper to store?  Why not have the computer do the work for you?  That’s the idea behind the Project Planner module in KTM.  There is a standard separation functionality built into KTM that works very well on structured and semi-structured documents but when you have more complex separation rules the Project Planner component of KTM is what you need.  This program is designed to create a template or “model” for the automatic separation which is then used by the KTM Server during the normal batch workflow.  This is why you might also hear this process referred to as “model building.” I want to give you a brief look at the setup of a TDS (Trainable Document Separation) model and how to integrate it with a KTM project.

The first thing that you need to do is collect lots and lots of samples.  The program requires that each class or document type have at least 50 samples.  Each document needs to be in tiff format and have its own folder.  Furthermore, documents that are multiple pages should be separated into single page tiff images and placed into their own folder.  The next step is to take your collection of document folders and group them into folder of different document types (these will become your classes in your KTM project).  This is a very time consuming process but it will help when you go to import in documents and you will see why.

Another thing that you should be aware of is that Project Planner requires an additional license and does not get installed with the normal KTM install.  Following the standard KTM install, you can find the Project Planner setup.exe located in the install media under the “Kofax Transformation Modules” and “Project Planner.”

After you have project planner installed and you have created a new project, you need to import those documents that were just sorted.  Once imported, there is a handy tool that allows you to select where the separation for each class, document, and page are.  This will allow the system to automatically create the classes and import the appropriate documents into each class

After the files are imported you will see the classes automatically created for this model.  The next step is to run all of the documents through the OCR engine in order for the system to be able to read the documents.  This process can take hours for larger sample sets so there have been times that I just let it run overnight.

When all the documents have completed running through the OCR engine, you can begin the cleanup process.  This is simply a matter of confirming documents are part of a class or not by checking the checkbox on each document.  You only need confirm enough documents so that the system is confident on the classification based on the samples provided.  As you can see from the screen shot, the bar across the bottom is color coordinated to show the confidence of a particular class. 

Blue are the documents that you have confirmed, green means confident and red means unconfident.  As documents are confirmed the red bar will get smaller and eventually go away.  This cleanup process is complete when enough documents have been confirmed for each document class so that all of the red is gone.

The next step is to compile the information into a TDS model which the KTM project can use for separation.  This is done by creating two files in Project Planner.  The first is a classification file, or the mod file, that the system will use to distinguish what class each page belongs to.  You can either use a text classification or image classification.  The second file that needs to be compiled is the document separation file, or the ads file.  This allows the KTM Server to use the training provided in the cleanup step to know where to separate each document.

The final step is to link the model to your KTM project.   Open up a project in Project Builder and go to the project setting within Project Builder.  On the Document Separation tab, one of the options is to use the “Trainable Document Separation (TDS)”.  Select this and browse to the folder containing the mod and ads files.

When you click OK you should get a message that tells you that “The TDS project was successfully imported.  New classes were created according to the definition of the document separation model.”  If not already there, classes will be automatically created in your project.  You’re now ready to synchronize the project within Kofax Administration and publish the batch class.

In summary, by using KTM and combining it with the TDS model it will you save time and money by reducing the amount of document preparation required when scanning.  For example, in a recent install I worked with a company that had a whole room of employees (about 20) doing manual separation.  We installed KTM and used the TDS model for separation and now they only have 4-5 people doing the same volume of documents in less time.  This a very powerful tool that I would suggest to anyone who has a need for automatic separation of semi-structured and unstructured documents.

 

Brandon Konen
Systems Engineer
ImageSource Inc.

Uploading Large Files with ILINX Capture and IIS

From time to time I receive questions about large file uploads with ILINX Capture. ILINX Capture can upload files of any size. The limitation is within Internet Information Services(IIS) and or the amount of memory installed in the web server. This is not only true for ILINX Capture, but and ASP or ASP.Net application.

Depending on the architecture of the ASP or ASP.Net application files being uploaded to the web server are typically streamed into the web server’s memory during the upload process before being written to disk. Depending on the number of user concurrently uploading files and the size of the files being uploaded will determine how much physical memory should be installed in the server. By default IIS has a 200KB size limit for uploading a single file. This can be increased, but not any higher than necessary or you may risk overconsumption of the web server’s memory.

Configuring File Upload Size in IIS 6

1. Open Internet Information Services Manager by clicking the Windows Start Menu and Run. Type inetmgr and click OK.

2. Once IIS Manger opens navigate the tree and right click the server name and click properties.

3. From the server properties window check the Enable Direct Metabase Edit checkbox and click OK.

4. Browse to the C:\windows\system32\inetsrv directory and edit the Metabase.xml file with a text editor such as Notepad.

5. Search for the attribute AspMaxRequestEntityAllowed and edit the value to the size in bytes that you want to allow for a maximum upload size. Save and close the Metabase.xml file.

AspMaxRequestEntityAllowed=”204800″

6. Open the Registry editor and navigate to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSSOAP\30\SOAPISAP.

7. Modify the MaxPostSize key. Set the decimal value to the maximum upload size in bytes and click OK.

8. Reboot the web server to ensure the changes have taken effect.

Configuring File Upload Size in IIS 7

1. Open Internet Information Services Manager by clicking the Windows Start Menu and Run. Type inetmgr and click OK.

2. Navigate the tree to the Virtual Directory that you would like to enable large file uploads.

3. In the Features View pane double click ASP.

4. In the ASP setting pane edit the Maximum Requesting Entity and Response Buffering Limit columns. Set this to the maximum file upload size in bytes and click Apply.

 

5. Open the Windows Command Prompt and enter the following command. Change the maxAllowedContentLength to your maximum file upload size in bytes and hit enter to execute the command.

C:\Windows\System32\inetsrv\appcmd set config “Default Web Site” -section:requestFiltering -requestLimits.maxAllowedContentLength:104857600

9. Open the Registry editor and navigate to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSSOAP\30\SOAPISAP.

10. Modify the MaxPostSize key. Set the decimal value to the maximum upload size in bytes and click OK.

11. Reboot the web server to ensure the changes have taken effect.

Bryan Wilhelm
Senior Systems Engineer
ImageSource, Inc.

Oracle BPM 11g Install for a Development Environment

Who is finally ready to get off their laurels and start looking at Oracle BPM 11g? I knew I was, the question I had was: where do I start? I figured the best place to start would be to actually install the software. A special thanks to one of our Systems Engineer, Les Harris who helped in getting me going on installing the software. I installed the entire Oracle BPM 11g stack on my laptop and documented the procedure. The following outline is not for a production install, but rather for getting a development environment up and running. A production install would be different and require different install packages as well as a few more steps. This install is strictly for setting up a development environment which is why I used the versions of the software listed below. Sorry I didn’t take screenshots, but I documented everything I did and using these steps I have been able to install multiple environments without issue.

The first thing I had to do was go and download all of the different install packages that comprise the Oracle BPM stack, those are:

• JRockit (JDK R28.1.4)
• Oracle Database (10G Express Edition – Universal)
• Oracle Repository Creation Utility (version 11.1.1.5.0)
• Oracle Web Logic Server (version 10.3.5)
• Oracle SOA Suite (version 11.1.1.5.0)
• Oracle JDeveloper Studio Edition (version 11.1.1.5)

Next I had to unzip all the packages that came zipped up. I recommend 7-zip or pretty much anything else that isn’t the built in Windows utility, that thing is just too slow for these large files.

Now it’s time to start installing.

1. Install JRockit. Since I’ll need to type the path a lot I installed it to C:\Java.
2. Install the database server. It’s a very typical install, I used all the defaults. Be sure to write down the sys account password.

a. After installed launch SQL plus and run the following commands to up the process count:

i.      Connect
ii.      Enter the credentials (username: sys as sydba, password: whatever you specified during your database install)
iii.      alter system reset sessions scope=spfile sid=’*';
iv.      alter system set processes=400 scope=spfile;
v.      shutdown immediate
vi.      startup
vii.      exit

3. Install the RCU utility. This creates all the necessary components in the database.

a. This must be done from the command line. Install as follows:

i.      Cd:\<path to RCU install>rcuhome\bin
ii.      Set RCU_JDBC_TRIM_BLOCKS=TRUE
iii.      Rcu.bat

b. Use all the defaults, pointing it to the database server that was just installed (hostname = localhost, port = 1521, servicename = xe,)

i.      You will see an error message about the database not being supported. Ignore it, this isn’t a production install.When prompted for which components to install select SOA at the top level in order to get all the other necessary requirements.

4. Install Web Logic Server. This is the underlying foundation upon which the application runs.

a. This must be done from the command line. Install as follows:

i.      Cd:\<path to weblogic install>\
ii.      C:\Java\Bin\Java.exe – jar wls1035_generic.jar

b. Select a custom install but choose all the defaults when prompted.

5. Install SOA. This is the meatiest part of the Fusion Middleware stack.

a. This must be done from the command line. Install as follows:

i.      Cd:\ path to RCU install>\Disk1\
ii.      Setup.exe –jreLoc c:\Java

b. Select all the defaults.

6. Create the SOA domain.

a. This must be done from the command line. Install as follows:

i.      Cd:\oracle\middleware\oracle_SOA1\common\bin
ii.      Config.cmd

b. When prompted select the ‘Create domain’ option

i.      Select the BPM Suite for Development and Enterprise Manager option
ii.      Select the Admin Server, Managed Servers, and Deployments options

7. Now it’s time to test the install out (this may take a while, be patient).

a. This must be done from the command line.

i.      Cd:\ oracle\middleware\user_projects\domains\base_domain\
ii.      Startweblogic.cmd

b. When the server is all done with its first start-up the command window should say something like: ‘SOA platform is now running and accepting requests’)
c. You can now close out the WebLogic server (CTRL+C)

8. Install JDeveloper

a. Use all the defaults
b. Once installed, launch JDeveloper Studio from the start menu (it will be under Oracle WebLogic)
c. Go to Help|Check for Updates

i.      Select Next on the Source
ii.      In the search box type in SOA and select the SOA update, then type in BPM and select the BPM update.
iii.      Select Next, then Finish. When it is done, close out of JDeveloper.

9. Now it is time to create your very first BPM 11g project.

a. Launch JDeveloper Studio.

i.      Select File|New
ii.      In the prompt select Applications in the left pane and BPM Application in the right pane.
iii.      Select OK
iv.      Now you’ll be prompted to give your application a name, name it whatever you want or leave it default and select Next.
v.      Now you’ll be prompted to give your Project a name, name it whatever you want or leave it default. Make sure BPM and SOA are selected form the list of available technologies. Select Next.
vi.      Select the Composite with BPMN Process option and select Finish.
vii.      You should now be prompted to create a BPMN process, just select Finish and you can start from there.

You should now be looking at an empty project with a Start and an End event. Here is where the fun begins…

John Linehan

Sr. Systems Engineer
ImageSource Inc.

Installation Hand Off to the Client

As an integrator there are many things we can do as far as making the install a success, but only a very limited amount we can do to guarantee its continuing success. One of the most often seen issues is the handoff of an installation to the client, in particular the maintenance of the database.

I will talk in particular about installations using Microsoft SQL Server. When the database is created it is more often as not left to us as the integrators to create the database (sometimes even the installation of the MSQL Server instance as well). We will choose the path for the database file groups, and allow the defaults for everything else. For operational purposes of our software this will work, but there should be changes made and additional procedures put in place by the clients DBA to guarantee continued optimal performance of the newly implemented system.

When visiting these systems at a later date we find that log files have not been truncated, leaving to question whether any backups have been done at all. Log files are the record of changes which have taken place since the last log truncation; this allows that the database can be restored to a point in time since the last full backup to reverse an adverse action or system failure. These log files are often much larger than the data files. In a maintained system a DBA would have established a normal backup strategy which would have backed up the log file at different intervals through the day and truncated the log file after a full database backup.

The other is often finding the file growth setting for the database to be at a preset auto growth amount. These by default are either 10 mb or 10%. When the database was first established an estimate was made to come up with a number which would be used for the initial file size for the database and log files. For databases which have a need for growth these presets can be very inefficient, as among other things they are likely to do their growth during business hours. Take a 200gb database which started at 150 gb, if it grows by only 10 mb every time it needs to grow there will be a large number of inefficient file space allocations to have made up the 50gb of space. If told to grow by 10 percent, then we will have seen 15 gb file allocations taking place but during business hours which can have an impact upon users. One which the maintenance has been taken up by the DBA will have been monitored and auto growth will have been turned off. The DBA would have scheduled the data file growth based upon measurements taken of the databases past growth rate, and most likely the new space allocation will have taken place in off business hours.

There may be a myriad of reasons why this handoff has failed including the lack of a DBA in the organization. I do not purport to know what the answer is, only that when a follow on visit to a client is made, that inspection of the database for these settings would be worthwhile so that they can be brought the clients attention and hopefully then engagement by the IT department for the maintenance needing to take place.

 

Jeff Doyle
Senior System Architect
ImageSource, Inc.

Tuning Abbyy FlexiCapture Layouts and Document Definitions

So you have spent many hours analyzing and creating the layouts and definitions for the documents you need to be processed through Abbyy.  Now you should be almost ready for production, except you need to tune.  Many samples of the documents in question need to be run through and the results checked over very carefully to find and fix all the little issues that will be present.

Tuning involves not finding the bugs in your definitions but finding the little differences in the printed documents that are processed.  These differences may be due to printing offsets on the printed form that is then run through the printer where the actual data to extract is found.  In addition, there can be other cases where the Header or Footer elements are not extracted correctly.  All these differences can add up to Abbyy not detecting the correct document definition to apply to the scanned images.

In order to correct these issues a very careful analysis of results need to be viewed through the Design Studio.  Import the document in question into the Studio and then process it.  Look carefully at what was missed.  Many times it is due to the Search Area not being large enough to cover all the letters/numbers to be extracted.  Also, within a group the required and option flags have a lot to do with if the group is found or not.  All it takes is one search element within the group that is not found and the entire group may be marked as not found, so be sure to check them over the flags carefully.

There are going to be times with multiple Document Definitions that a specific document does not match the definition it should have, but some other definition.  This can be caused by the error percentage on the wrong document definition to be set too high a value when both document definitions share a similar field to extract.  To fix this just take the error percentage down a few points and try the recognition again.

It takes a lot more effort to tune a document definition especially when dealing with multiple document definitions and paper documents that are difficult to scan in with enough clarity for the OCR engine to work properly.  This is very true for Transcript type documents where each transcript has its own copy protection mechanism that the scan software must try and compensate.  However it works out, so be prepared to spend the time and effort to get the document definitions to the point where they work most of the time.

Christopher J. Hillenburg
Senior System Engineer
ImageSource, Inc.

Features in ILINX Capture (Part I)

The feature set in ILINX Capture is vast and it can be a drag reviewing and interpreting feature lists in software documentation.  Those of you not familiar with ILINX Capture can visit the following website www.ilinxcapture.com, or feel free to leave a comment and we can provide additional information and/or a hands-on demonstration.  In short, ILINX Capture is a web based capture platform that excels in distributed capture and custom capture workflow environments.  It is scalable to work on a single workstation or it can be extended to an enterprise wide global standard for capture in your organization.

I wanted to use this post to touch on a couple of the features that I see being used more and more in ILINX Capture.  These features became part of the product based on customer feedback, industry direction, and internal vision for the product.  All of the following features can be added to any point in your process flow map, so it provides not only the functionality but also the flexibility to adapt to the business needs of current processes in place today.

1. 2D Barcode Support   – This feature adds the ability to read metadata, classify and separate documents, and provide quality control checks through the recognition of 2D barcodes.  Through a GUI the user has the ability to parse the barcode data and map it to fields, separate and identify the type of document, and validate that the number of pages in the document match what was captured through the scanning or electronic import process. 
2. Web Service Integration  – This feature provides ILINX Capture with the ability to integrate with any existing web service.  Most commonly, we see this used to perform database lookups or validations against existing line of business systems.  Another way this is being utilized is to interact with different organization processes, for example, you can create a support ticket in an organization’s support system every time a process exception occurs in their fully automated capture workflow.
3. Queue Thresholds & Triggers – Work queues in ILINX Capture are areas where human interaction is required to process data or documents through the workflow.  The thresholds and triggers provide the ability to monitor the batches or documents in a queue and execute a function when a threshold or trigger is met.  This is useful to monitor escalations or the processing of high priority documents.  For example, if a fax comes in to the system for an auto loan or stock trade, in most cases, this is a time sensitive process that needs to move rapidly through the workflow.  Between the notification features and the thresholds/triggers, ILINX Capture can ensure that 1) a user is notified that there is high priority work to process, 2) the documents are processed within a defined time frame, and 3) if the documents are not processed the system can notify a manager or route the documents to another user group.

These are just a few of the features that have been added to extend the functionality of this product.  Stay tuned to this blog for additional information on other features that help shape this product to provide value to its customer community. 

Ryan Keller

ImageSource, Inc.