New York recently joined the growing number of jurisdictions that have adopted the Uniform Bar Exam (UBE).1 The UBE consists of 200 multiple-choice questions; six 30-minute essay questions covering a wide range of doctrinal areas,2 and two "performance" test questions in which examinees have 90 minutes to read a case file and write a client letter, memorandum, brief or other document. Applicants must also successfully complete an online course on "important and unique principles of New York law" in 12 subject areas and pass the New York Law Exam (NYLE), an online 50-question exam. Except for the materials provided for the performance test, the UBE exam is entirely closed book. The NYLE is open book, but electronic searching of the course materials is forbidden.

As a licensing exam, the purpose of the bar exam (both the UBE and the NYLE) is consumer protection - ensuring that new lawyers have the competencies required to practice law effectively. But critics in New York and elsewhere have long argued that the bar exam is not a valid measure of new lawyers' competence because it fails to test the wide array of skills lawyers need; the multiple-choice questions assess legal knowledge and analysis in an artificial and unrealistic context, and the closed-book format rewards the ability to memorize thousands of legal rules, a skill unrelated to law practice.3

A frequent response to these critiques is that, while the existing exam may not be perfect, it is the best we can do because we need an exam that is both psychometrically reliable and relatively inexpensive to administer to thousands of examinees.4 But we can do better. We describe two licensing exams that demonstrate how bar examiners could utilize an open-book format (already being used in the NYLE) and develop multiple-choice questions that assess a candidate's ability to engage in legal reasoning and analysis without demanding unproductive memorization of so many detailed rules of law.

The first example, the case file approach, is taken from a 1983 California "Performance Test" in which test-takers received a case file and a series of multiple-choice questions testing the candidates' ability to read, understand, and use cases to support their legal positions. The second example discusses the current law licensing exam administered by the Law Society of Upper Canada (LSUC), an open-book multiple-choice exam that tests the use of doctrinal knowledge in the context of law practice.

These two licensing approaches demonstrate how multiple-choice questions can be used in a licensing exam to measure legal analysis and reasoning skills as lawyers use those skills, and they demonstrate that we can do a better job of testing minimum competence, even with a multiple-choice exam.

A Case File Approach to Multiple-Choice Testing

What Is a Case File Approach?

Long before the National Conference of Bar Examiners (NCBE) began administering a performance test, California developed its own performance exam, consisting of factual material and appellate opinions that provided the basis for both multiple-choice and essay questions. For example, a 1983 exam packet5 focused on landowners' liability for injuries occurring on their property. The case file contained a memo from a supervising partner laying out some basic facts, notes from interviews with two witnesses, a memo from defendant company headquarters, a "slapped-together" complaint, a brief motion to dismiss for failure to "state a claim on which relief can be granted," and a case library of six short appellate opinions.

In other words, examinees had information lawyers might see when working on a client's case.

The case file served as the basis for both multiple-choice questions and a memo-writing assessment. We examine only the multiple-choice portion of the case file (though it is worth noting that the essay question asked not only for analysis of legal theories but also identification of "factual or proof problems" that the plaintiff might face and further facts that might be sought, skills that are also connected to the work that lawyers do for clients).

Sample Questions From the 1983 Case File Exam

The notable aspect of the multiple-choice questions is that they focus directly on an important analytical skill we expect lawyers to have: the ability to read and understand appellate cases and use those cases to support a party's legal contentions and theories. For example, several questions asked test-takers to choose which one or more of the opinions best supported particular stated rules of law (e.g., "if one enters uninvited onto the land of another, he cannot recover in negligence for injuries caused by the failure of the landowner to maintain the land in a safe condition"). The answer choices listed different combinations of cases from the case library.

Other multiple-choice questions tested the examinee's ability to understand a court's reasoning. For example, test-takers were asked to select which of the following five statements of law best describes the basis for the court's decision in one of the opinions in the case library:

(A) Plaintiff had no reason to foresee the risk that occurred when plaintiff entered upon defendant's property.

(B) Defendant impliedly consented to plaintiff's use of defendant's property.

(C) Defendant had no reason to foresee the kind of injury that plaintiff suffered while on defendant's property.

(D) A possessor of land must use reasonable care to avoid injury to others as a result of conditions on the land.

(E) A possessor of land must use reasonable care to avoid injury to others who are using an adjacent public way.

Examinees also answered questions about which cases were most supportive of plaintiff's position and which of three identified facts in each of the cases should be emphasized in preparing plaintiff's case. Finally, one question asked test-takers to identify the rule emerging from a synthesis of all the cases in the packet:

The cases establish that a landowner maintains a nuisance when conditions on his contiguous land are dangerous to users of a public way,

(A) so that the injured user is required, in seeking relief, to assert a wrong to the public generally.

(B) so that the injured user, despite the presence or absence of a wrong to the public generally, is required to show special damage to himself.

(C) so that it is possible to reason from them to absolute liability when a condition dangerous to users of the public way is present; that is, it is not necessary in such a case to prove negligence or the absence of contributory negligence.

(D) only as a way of expressing a duty to use due care with respect to users of the public way.

Answering these questions was not a simple matter of identifying case holdings but, instead, demanded close reading of the cases and understanding subtle differences in holdings and rationales.

How Does the Case File Approach Differ From the Status Quo?

According to the National Conference of Bar Examiners, the Multistate Bar Exam (MBE) multiple-choice questions test examinees' ability to "apply fundamental legal principles and legal reasoning to analyze given fact patterns."6 Examinees, relying on their memory of thousands of legal rules, answer 200 questions based upon a large number of discrete fact patterns.7

The case file method shifts the testing lens from the ability to recall  rules to the ability to discern and comprehend legal rules in typical legal materials. It tests legal analysis and reasoning the way lawyers do such analysis, understanding and using appellate opinions to support claims and defenses. Under the case file approach, examinees still must apply fundamental legal principles and legal reasoning to analyze fact patterns, but they do so utilizing analytic skills the way that lawyers actually function when assessing client problems.

Expanding Upon the California Case File Approach

The multiple-choice section of the 1983 California test consisted of 15 questions primarily focused on the plaintiff's case. It asked only one question about the defendant's case ("On which of the following cases would you expect the defendant . . . to place the most reliance?"). It easily could have asked additional questions about the defendant's legal theories and defenses, with or without supplementing the library of cases provided. The case file could have included additional facts that would allow questions asking examinees to identify additional legal theories that might be applicable upon further research, thus providing a way to test issue-spotting and foundational knowledge of a wide range of doctrinal areas. Statutes and regulations could be added to test examinees' ability to read and understand those materials. Questions could assess whether examinees are able to identify key missing factual information and the best way to develop those facts. In other words, the multiple-choice component of a case file exam could be developed to test a broader array of analytic and problem-solving competencies.

As was true in the 1983 California test, the case file could also serve as the basis for essay questions requiring the applicant to draft memoranda further analyzing the possible legal theories, identifying procedural and evidentiary or proof issues in presenting the case, and advocating on behalf of a client, thus combining essay and performance test questions with the multiple-choice portion to create an entire licensing exam that replicates how lawyers use and analyze facts and cases to solve a client's problem.

Can the Case File Approach Be Psychometrically Validated?

While we do not know if California engaged in a psychometric validation of its 1983 test, a study sponsored by the Law School Admission Council (LSAC) developed and validated a similar assessment method in a 2008 study, Developing an Assessment of First-Year Law Students' Critical Case Reading and Reasoning Ability.8 The study provided first-, second-, and third-year law students with multiple appellate opinions, then tested their ability to read the cases and identify accurately issues, holdings, reasoning, rules, and policy. They also assessed students' ability to identify and work with indeterminacies in legal doctrine ("all the rhetorical ways that statements offered by courts may be open to interpretation, such that there may be no way to tell precisely what a court means or precisely how it is reaching its conclusion").9 In other words, it tested the ability of the students to engage in higher-level legal analysis, not just to identify holdings and rationales in single cases.

The LSAC study demonstrates it is possible to develop a valid examination of legal reasoning ability using precisely the type of questions included on the 1983 California exam. It is worth noting that the LSAC study showed that many of the students lacked fundamental case analytical skills and, more disturbing, that their ability to accurately analyze appellate opinions and apply them to client problems did not improve between the first and third years of law school. If the bar exam tested appellate opinion analysis and application, it seems likely there would be increased efforts to ensure students, and therefore law graduates, would learn this fundamental legal skill more effectively.

Law Society of Upper Canada Exam10

Another example of an alternative multiple-choice methodology worth exploring is the approach taken by the Law Society of Upper Canada (LSUC). The LSUC governs the law licensing process for the province of Ontario, administering a psychometrically validated licensing exam to thousands of applicants each year. It uses a seven-hour multiple-choice test consisting of 220 to 240 multiple-choice questions to test a wide range of lawyering competencies including ethical and professional understanding, knowledge of the law, establishing and maintaining client relationships, practice management issues, and (for barristers) problem/issue identification, analysis, and assessment.11

An Open Book Exam Approach to Law Licensing

A notable feature of the LSUC exam is that it is open book. The LSUC provides examinees with online access to the necessary materials to study before the exam. To help them navigate the materials, the LSUC encourages test-takers to organize the material via color coding, short summaries, and index cards, or to use the resources to create study aids that best suit the examinee's learning style. Candidates may print and bring the materials to the exam. Examinees' ability to answer a question does not rest on whether they remember (or forget) a key element. Rather, examinees have access to the relevant legal rules, and just like practicing attorneys, when they cannot recall a rule or key element, they have the ability to look it up. Unlike closed-book exams like the UBE, an open-book exam tests a key lawyering competency - the ability to find appropriate and relevant legal information.12 The NYLE operates in a similar fashion, providing course materials that an applicant may access during the exam. In contrast, preparation for the UBE involves a few months of memorizing thousands of rules, quickly forgotten after the exam is over. The focus is on short-term memory rather than long-term understanding and the crucial ability to find or identify the relevant legal rule and then apply it. Neuroscience research attests to the ineffectiveness of that kind of preparation for knowledge retention and long-term learning,13 confirmed by lawyers' memories of their own experience taking the bar.14

Testing Knowledge in the Context of Law Practice

The LSUC exam, like U.S. bar exams, tests legal knowledge and analytical skills, but it often does so in a practice-oriented context, focusing on how knowledge of the law informs the proper representation of clients. For example, LSUC exam questions ask what information a client needs to make an informed decision, how a lawyer would respond to particular questions from a tribunal, and what research should be done on the law or facts to inform the lawyer's next steps. This approach to multiple-choice questions tests legal knowledge and analysis, but it does so with a focus on how lawyers use legal doctrine in practice.

Sample Questions From LSUC Exam

Below are samples of LSUC questions provided by the Law Society of Upper Canada. They test the applicant's understanding of the information a lawyer needs from the client or other sources, strategic and effective use of trial process, ethical responsibilities, and knowledge of the real property registration system, all in the service of proper representation of a client. These and other questions (and the answers) are available on the LSUC website.15

1. Gertrude has come to Roberta, a lawyer, to draw up a power of attorney for personal care. Gertrude will be undergoing major surgery and wants to ensure that her wishes are fulfilled should anything go wrong. Gertrude's husband is quite elderly and not in good health, so she may want her two adult daughters to be the attorneys. The religion of one of her daughters requires adherents to protect human life at all costs. Gertrude's other daughter is struggling financially. What further information should Roberta obtain from Gertrude?

(a) The state of her daughters' marriages.

(b) The state of Gertrude's marriage.

(c) Gertrude's personal care wishes.

(d) Gertrude's health status.

2. Tracy was charged with Assault Causing Bodily Harm. She has instructed her lawyer, Kurt, to get her the fastest jury trial date possible. The Crown has not requested a preliminary inquiry. Kurt does not believe that a preliminary inquiry is necessary because of the quality of the disclosure. How can Kurt get Tracy the fastest trial date?

(a) Waive Tracy's right to a preliminary inquiry and set the trial date.

(b) Bring an 11(b) Application to force a quick jury trial date.

(c) Conduct the preliminary inquiry quickly and set down the jury trial.

(d) Elect on Tracy's behalf trial by a Provincial Court Judge.

3. Peyton, a real estate lawyer, is acting for a married couple, Lara and Chris, on the purchase of their first home. Lara's mother will be lending the couple some money and would like to register a mortgage on title. Lara and Chris have asked Peyton to prepare and register the mortgage documentation. They are agreeable to Peyton acting for the three of them. Chris' brother is also lending them money but Lara and Chris have asked Peyton not to tell Lara's mother this fact. Should Peyton act?

(a) Yes, because the parties consented.

(b) No, because there is a conflict of interest.

(c) Yes, because the parties are related.

(d) No, because she should not act on both the purchase and the mortgage.

4. Prior to the real estate closing, in which jurisdiction should the purchaser's lawyer search executions?

(a) Where the seller previously resided.

(b) Where the seller's real property is located.

(c) Where the seller's personal property is located.

(d) Where the seller is moving.

Conclusion

As illustrated above, both the case file approach and the LSUC exam demonstrate that it is possible to design multiple-choice questions that test a wider range of lawyering competencies, including the all-important ability to read, analyze and effectively utilize cases. Both alternative approaches are open book, meaning they are much less dependent on memorization and more dependent on assessing knowledge in the context of how lawyers actually practice.

Neither exam addresses all of the flaws of existing U.S. bar exams. For example, both methods appear to require examinees to read rapidly, with little time for reflection. While lawyers must do their work efficiently and sometimes under time pressure, "speededness" in test-taking is a different skill. In updating or redesigning the bar exam, examiners should consider how to avoid making the wrong kind of speededness a significant variable in applicant performance.16

Additionally, none of the multiple-choice exams adequately address experiential skills such as client interviewing and negotiation, and alternative models of testing should be explored to assess those experiential learning skills. Recognizing this gap, the LSUC also requires applicants to "article" (a kind of apprenticeship with a law firm) or participate in the Law Practice Program (a four-month training course and a four-month work placement). That form of assessment has its own set of issues17 and would be challenging to implement in the larger U.S. market. One alternative model that has been proposed is the "standardized client," modeled after the standardized patient exam given to medical school graduates.18 Other options could include using a closely supervised law school clinical or externship experience to ensure appropriate skills development.

While the historical California performance test and the LSUC exam do not address all problems identified with the current U.S. bar exam, these licensing exams demonstrate that it is possible to develop psychometrically reliable multiple-choice questions that better test actual lawyering skills. Given the rising chorus of voices agreeing that our current bar exam fails to measure the wide array of lawyering skills required in the practice of law, there is every reason to explore these alternate approaches, at least as a starting point for a discussion about how to improve the existing bar exam to better protect the public.


  • At the time this article was written, 31 states, the District of Columbia, and the Virgin Islands had adopted the UBE. See Uniform Bar Examination: Jurisdictions That Have Adopted the UBE, Nat’l Conf. B. Examiners, http://www.ncbex.org/exams/ube/.
  • For a list of the subjects tested, and the topics within each doctrinal area, see Nat’l Conference of Bar Exam’rs, 2018 MEE Subject Matter Outline (2017), http://www.ncbex.org/pdfviewer/?file=%2Fdmsdocument%2F183.
  • See, e.g., Don J. DeBenedictis, Bar Winners and Losers: N.Y. Committee Suggests Lawyer Exams Test Wrong Skills, Unfair to Minorities, ABA J., May 1992, at 26 (discussing draft report from the New York City Bar); Kristin Booth Glen, Thinking Out of the Bar Exam Box: A Proposal to “MacCrate” Entry to the Profession, 23 Pace L. Rev. 343, 375-81 (2003); Society of American Law Teachers Statement on the Bar Exam, 52 J. Legal Educ. 446, 446-49 (2002).
  • See, e.g., Suzanne Darrow-Kleinhaus, A Response to the Society of American Law Teachers Statement on the Bar Exam, 54 J. Legal Educ. 442 (2004).
  • The 1983 California Performance Test is on file with the authors.
  • Multistate Bar Examination: Jurisdictions Administering the MBE, Nat’l Conf. B. Examiners, http://www.ncbex.org/exams/mbe/.
  • Examples of MBE questions (and answers) are available on the NCBE website. See Nat’l Conference of Bar Exam’rs, MBE Sample Test Questions (2016), http://www.ncbex.org/pdfviewer/?file=http%3A%2F%2Fwww.ncbex.org%2Fdmsdocument%2F17.
  • See Dorothy H. Evensen et al., Developing an Assessment of First-Year Law Students’ Critical Case Reading and Reasoning Ability: Phase 2 (LSAC Grants Report 08-02, Mar. 2008), https://www.lsac.org/docs/default-source/research-(lsac-resources)/gr-08-02.pdf.
  • Id. at 3.
  • For further discussion of the LSUC exam, see Eileen Kaufman, Andi Curcio & Carol Chomsky, A Better Bar Exam—Look to Upper Canada?, Law Sch. Café, July 25, 2017, https://www.lawschoolcafe.org/2017/07/25/a-better-bar-exam-look-to-upper-canada/.
  • For more information about the LSUC licensing process, including the list of competencies assessed, see Guide to the Barrister and Solicitor Licensing Examinations, Law Soc’y Ont., https://www.lsuc.on.ca/LawyerExaminationGuide/.
  • See Andrea A. Curcio, Carol L. Chomsky & Eileen Kaufman, Testing, Diversity, and Merit: A Reply to Dan Subotnik and Others, 9 UMass L. Rev. 206, 233-34 (2014).
  • See, e.g., Hillary Burgess, Deepening the Discourse Using the Legal Mind’s Eye: Lessons from Neuroscience and Psychology That Optimize Law School Learning, 29 Quinnipiac L. Rev. 1, 40-41 (2011) (discussing that bombarding a law student’s working memory with too much information may result in cognitive overload and forgetting information crucial to understanding and overall learning); Larry O. Natt Gantt, II, The Pedagogy of Problem Solving: Applying Cognitive Science to Teaching Legal Problem Solving, 45 Creighton L. Rev. 699, 730-35 (2012) (discussing the importance of connecting domain knowledge and problem-solving).
  • A New York City Bar Task Force noted, presumably based upon the task force members’ own experience, that the bar exam tests memory of “information that will be quickly forgotten after the exam.” N.Y.C. Bar Ass’n Task Force on New Lawyers in a Changing Profession, N.Y.C. Bar Ass’n, Developing Legal Careers And Delivering Justice in the 21st Century 78 (2013), https://www2.nycbar.org/pdf/developing-legal-careers-and-delivering-justice-in-the-21st-century.pdf.
  • See Sample Licensing Examination Items, Law Soc’y Ont., http://www.lsuc.on.ca/LawyerExaminationGuide/#Sample_Licensing_Examination_Items.
  • See William D. Henderson, The LSAT, Law School Exams, and Meritocracy: The Surprising and Undertheorized Role of Test-Taking Speed, 82 Tex. L. Rev. 975, 1034-38 (2004) (arguing that “time-pressured” tests, such as the LSAT or traditional law school exams, do not measure the kind of efficiency or “quickness” that is valuable in legal practice). See also Polina Harik et al., A Comparison of Experimental and Observational Approaches to Assessing the Effects of Time Constraints in a Medical Licensing Examination, 55 J. of Educ. Measurement 308 (demonstrating that speededness significantly affects examinee test-taking strategy and suggesting that providing more time would especially impact lower-performing test-takers).
  • See, e.g., Malcolm Mercer, The Never-Ending Debate: What Should Be Required in Order to Become a Lawyer?, Slaw, May 7, 2018, http://www.slaw.ca/2018/05/07/the-never-ending-debate-what-should-be-required-in-order-to-become-a-lawyer/ (discussing proposals to change the articling and Lawyer Practice Program portions of the LSUC exam).
  • Lawrence M. Grosberg, Medical Education Again Provides a Model for Law Schools: The Standardized Patient Becomes the Standardized Client, 51 J. Legal Educ. 212 (2001).