No Rules: Problems With Rules-Based Contract Provision Extraction

7 minute read

This Gordon Gecko-esque cell phone is a good metaphor for manual rule-based contract metadata extraction systems: Keyword search systems may be sophisticated compared to the status quo of non-technology-aided humans reviewing contracts from scratch. But they are pretty clunky next to a modern system.

The simplest way to find provisions in contracts is to write manual rules describing the provisions. These keyword searches can be executed in forms including regular expressions and Boolean search strings. For example, an “assignment” clause rule might be to return all sentences where the words “assign” and “agreement” co-occur (or perhaps something more complicated). The principal advantage to this approach is its simplicity—a large number of rules can be written and tweaked quickly. And these systems are quite effective on extracting provisions from known documents, like form agreements that you have an example of in advance of running the contract provision extraction software. They should also work well on provisions that are relatively consistent across agreements (e.g., something like governing law or items from within certain financial agreements that are drafted similarly). One other plus: it is easy to understand the rules themselves, and to see where they went wrong when mistakes occur.

Limitations of Keyword Searching Are Known in Other Domains

If you have experience with Boolean searching in other contexts (e.g., case law research databases), you know that there’s a catch to keyword searches. While potentially helpful, results can be heavily under- and over-inclusive. Expert systems based on manual rules were big in the 1980s. Since then, however, they have generally been replaced by machine learning approaches in areas ranging from translation* to eDiscovery. There have been a heavy stream of works critiquing keyword searches in text retrieval, including as compared with machine learning approaches (see, e.g., this paper). In a 2011 article on eDiscovery search technology, Southern District of New York Magistrate Judge Andrew Peck explains why keyword search approaches have become unpopular:

How effective is keyword searching? In 1985, scholars David Blair and M.E. Maron collected 40,000 documents from a Bay Area Rapid Transit accident, and instructed experienced attorney and paralegal searchers to use keywords and other review techniques to retrieve at least 75% of the documents relevant to 51 document requests. Searchers believed they met the goals, but their average recall was just 20%. This result has been replicated in the TREC Legal Track studies over the past few years.

Judicial decisions have critiqued keyword searches.

Blair and Maron thought so little of keyword searches (which they described as “full-text” searching) that they wrote:

A full-text retrieval system does not give us something for nothing. Full-text searching is one of those things, as Samuel Johnson put it so succinctly, that “. . . is never done well, and one is surprised to see it done at all.”

From what we understand, at least two well-funded automated contract provision extraction systems are based primarily on keyword/manual-rules searches.** A manual rules based system should work well on agreements you already have in advance or simpler provisions. If, for example, you are reviewing agreements on your own paper a keyword search system should be as good as any other. Ditto if you only need to extract simpler provisions like governing law. Based on (i) findings in other areas, (ii) our own research, and (iii) the experience of a well-funded but now defunct manual rules-based contract extraction company (which we’ll cover in more detail in the next post), keyword-based automatic contract provision extraction systems should have underwhelming accuracy on unknown documents. That said, we have not tried our competitor’s systems, and perhaps they have solved this long-documented unknown agreement accuracy problem.

Keywords + Poor Scans = Misses

Compounding the problems with manual rules-based systems, many documents reviewed in due diligence and contract management work are in the form of image files. As previously discussed, automatic contract provision extraction systems review image files by first converting documents into machine readable text using OCR software, then running their provision models on this text. The problem is that scans can be poor quality and OCR software is imperfect. What happens to a keyword search when critical words are mis-transcribed (e.g. “a3Signed” in place of “assigned”)?

Unknown Accuracy on Unfamiliar Agreements

Manual rules-based contract metadata extraction systems are susceptible to an additional problem: it is hard for their vendors to know their accuracy. As we wrote (when describing how we test the accuracy of our machine learning-based contract provision models):

One way to measure accuracy would be to test on the same documents used to build provision models. However, this is analogous to giving students the answers to a test beforehand. You can know that high scoring students are good at memorizing, but you cannot know if they’ve really learned the material. Computers are particularly good at memorizing, and thus you should never test on the training data to determine the accuracy of a learned model (unless the problem you are trying to evaluate is if a system can find already seen instances (which might be the case for an automated contract review system only intended to work on known documents like company forms)).

This requirement to test on “unseen” data is particularly difficult to meet for systems that use manual rules (i.e., human created rules, such as those built using Boolean search strings). If using a manual rules based system, the only way to obtain truly unbiased accuracy results is to keep testing documents secret from the people building the rules. This requires a great deal of discipline; it is very tempting to look at where rules are going wrong.

We can understand why other vendors chose to build their contract provision models primarily using manual rules. And perhaps they have succeeded in this where others have failed. But buyers who need good accuracy on unfamiliar documents should ask serious questions.

Summary of Manual Rule-Based Contract Provision Extraction Pros & Cons

Pros: easy to set up, works well on known documents, works well on similarly-written contract provisions, can build models with limited technical knowledge, can easily understand rules and see what went wrong when they make mistakes, can be quite impressive compared to traditional non-technology enhanced contract review
Cons: does not work well on unfamiliar documents, may have trouble on poorly scanned documents where keywords are obscured, hard for developers to know accuracy

This post is part of the Contract Review Software Buyer’s Guide. Previous posts introduced the guide and contract review software, and gave a high level view of how automated contract provision extraction systems work. The next post will be a case study into a well-funded company that built an automated contract abstraction system using manual rules technology, and had trouble. The few posts after that will cover comparison- and machine learning-based provision extraction technology.

* This is an interesting recent post on machine translation and litigation document review work, if interested in the subject.

** This is based on our reading of their publicly available materials as well as conversations with people who have spoken with these providers. Note: even companies that base the core of their system on manual-rules may use some machine learning based technology, or perhaps use underlying technology that has machine learning capabilities. For example, machine learning based entity extraction (e.g., title/parties/date) technology is readily available. And we would be surprised if our manual-rules-using competitors didn’t integrate this.

Contract Review Buyers Guide Series:

Share this article: