MANSFIELD, Circuit Judge.[fn*] [fn*] Heard the case as a District Judge and, after appointment to the Circuit Court of Appeals, was designated to sit on the District Court for the purpose of completing this phase of the case.

The fairness and validity of competitive examinations, once described by Gilbert and Sullivan as the means of attaining "a Duke's exalted station," have frequently been challenged in courts and elsewhere. E.g., Griggs v. Duke Power Company, 401 U.S. 424, 91 S.Ct. 849, 28 L.Ed.2d 158 (1971); Louisiana v. United States, 380 U.S. 145, 85 S.Ct. 817, 13 L.Ed.2d 709 (1965); Willner v. Committee on Character and Fitness, 373 U.S. 96, 83 S.Ct. 1175, 10 L.Ed.2d 224 (1963); Schware v. Board of Bar Examiners of New Mexico, 353 U.S. 232, 238-239, 77 S.Ct. 752, 1 L.Ed.2d 796 (1957); Armstead v. Starkville Municipal Separate School District, 325 F. Supp. 560 (N.D.Miss. April 7, 1971). We are here called upon to decide whether those examinations which have been prescribed and administered by the Board of Examiners of the City of New York (the "Board" herein) to candidates seeking licenses for permanent appointment to supervisory positions in the City's school system (principals, assistant principals, administrative assistants, etc.) are unconstitutional. We conclude that a sufficient showing has been made of violation of the Equal Protection Clause of the Fourteenth Amendment to warrant the issuance of preliminary injunctive relief.

The two named plaintiffs, who are respectively Black and Puerto Rican, have brought this purported class action on behalf of themselves and all other persons similarly situated pursuant to federal civil rights laws, 42 U.S.C. § 1981 and 1983. They allege that the competitive examinations, which must be passed by a candidate before he or she can qualify for licensing and appointment, discriminate against persons of Black and Puerto Rican race, and have not been validated or shown fairly to measure the skill, ability and fitness of applicants to perform the duties of the positions for which the examinations are given, with the result that success on the examination does not indicate in any way that the candidate will succeed as a supervisor. This racial discriminatory effect, coupled with lack of justification or predictive value as measurements of abilities required to perform the jobs involved, is alleged to violate not only plaintiffs' federal constitutional rights but also (based on pendent jurisdiction) Art. 5, § 6 of the New York State Constitution, and §§ 2590-j(3)(a)(1), 2569(1), and 2573(10) of the New York Education Law, McKinney's Consol. Laws, c. 16.

Plaintiffs seek a preliminary injunction under Rule 65, F.R.Civ.P., prohibiting the alleged violations of these laws. They also seek declaratory relief pursuant to 28 U.S.C. § 2201. We have jurisdiction under 28 U.S.C. § 1331 and 1343(3).

The Board of Education has not actively opposed the motion for preliminary injunction, and it agrees that plaintiffs have presented triable issues of fact. The Board of Examiners ("Board" herein), however, has vigorously opposed the motion.

In reaching our decision we have had the benefit of a plethora of lengthy affidavits and exhibits, a hearing at which oral testimony was taken, a series of arguments, and extensive briefing of the law and facts by the parties. In addition the following organizations have appeared as amici and filed briefs supporting plaintiffs: New Association of Black School Supervisors and Administrators, ASPIRA of America, Inc., and the Public Education Association.

An applicant for permanent appointment to a supervisory position in the New York City School System must, in addition to meeting state requirements for the position, obtain a New York City license. First, each such candidate must have met minimum education and experience requirements established by the City's Board of Education and the Chancellor, Harvey B. Scribner, who is the Chief Administrator of the School District of the City of New York. For instance, a candidate for principal of a day elementary school must, among other things, have had (1) four years' experience teaching in day schools under regular license and appointment as a teacher, and (2) two years' experience of supervision in day schools under license and appointment, or meet various alternative experience requirements.

Next the candidate must pass an examination procedure prepared and administered by the Board for the particular type or classification of supervisory post desired, which may take as long as two years to complete. If the candidate successfully completes the testing procedure, he or she is granted a license and placed on a list of those eligible for assignment to the type of supervisory position involved. The appropriate school governing authority — either a central board of education or a community school board under New York City's present decentralized system — then selects the person it wishes from the eligible list to fill an open position. Since appointments of permanent supervisory personnel in the New York City School System must be made from lists of eligibles who have passed examinations, the Board from time to time announces and conducts examinations for particular supervisory posts (of which there are more than 50 different types) following which the number of persons eligible for appointment are supplemented by promulgation of lists of those who passed the latest examination. If a successful candidate, after being listed as eligible for appointment, is not appointed within four years, he or she is dropped from the list and must again pass the qualifying examinations to be listed as eligible.

Only in the cities of Buffalo and New York does state law provide for examinations in addition to state certification, N YEducation Law § 2573(10-a), and only the New York City School District maintains a Board of Examiners and the specific examination and licensing procedure here under attack. The Board has described itself as "a highly select group with broad professional background in education and related fields chosen through the most objective and impartially searching examination given under civil service." (Ex. 10, Item 10, attached to 5/28/71 aff. of Richard S. Barrett)

Were it not for New York City's special examination and licensing procedure, plaintiffs Chance and Mercado would have been appointed permanent elementary school principals. Both have been certified by the state for that position, and both are specially trained to be principals, having graduated from a year-long Fordham University Instructional Administrators and Principals Internship Program in Urban Education.

Plaintiff Boston M. Chance has been employed in the New York City public school system for the last 15 years and is acting principal of P.S. 104, an elementary school in the Bronx. Chance, who is of the Black race, possesses all of the basic qualifications of education and experience established by law and by the Board of Education and the Chancellor of the New York City School District for the position of principal of an elementary school. However, he lacks a city license as elementary school principal and therefore is barred at present from securing a permanent position as principal. In September, 1968, Chance took the examination given by the Board for the position of Assistant Principal, Junior High School, but he failed it and thus was not placed on the eligibility list and was not issued a license entitling him to permanent appointment.

Plaintiff Louis Mercado, a Puerto Rican who also holds a New York State license as a principal, has been serving the New York City school system for the last 12 years. He is presently acting principal of P.S. 75, an elementary school in Manhattan, but he is barred from permanent appointment because he does not have a New York City license as an elementary school principal. Mercado is in a somewhat different position from Chance in that he does not allege that he has ever taken the relevant Board of Examiners' Supervisory Examination. While the present motion was pending — and while the parties were collecting statistical information pursuant to our order — the Board conducted their November, 1970 series of examinations for elementary school principal. Mercado withdrew from this examination and refused to take it on the grounds that the "Board of Examiners is not the appropriate agency for qualifying school personnel" and "the examination is not relevant * *."

Both Chance and Mercado were selected for their present acting principalships by their respective community school boards, in accordance with New York City's decentralized system. See generally, Council of Supervisory Associations, etc. v. Board of Education, 23 N.Y.2d 458, 297 N.Y.S.2d 547, 245 N.E.2d 204 (1969). In some instances such local school boards found, after interviewing licensed principals listed as eligible by the Board, that persons not so licensed were more qualified to serve as principals than those interviewed and that they performed their duties in a superior manner. (See Aff. of Peter J. Straus, 9/22/70).

There are approximately 1,000 licensed Principals of New York public schools of varying levels (e.g., elementary day, junior high school, high school, etc.), of whom some act as the heads of schools and others function in administrative positions. Of the 1,000 only 11 (or approximately 1%) are Black and only 1 is Puerto Rican. Furthermore, of the 750 licensed Principals of New York elementary schools only 5 (or less than 1%) are Black, and none is Puerto Rican. Of the 180 high school administrative assistants, none is Black or Puerto Rican.

Of the 1,610 licensed Assistant Principals of New York City junior high and elementary schools, only 7% are Black and only .2% are Puerto Rican. When the list for the position of Principal, elementary school, was originally promulgated, only 6 out of the 340 persons (or about 1.8%) were Black and none was Puerto Rican; and when the list for Principal, high school, was promulgated, none of the 22 licensed people was Black or Puerto Rican. The promulgated list of licensed Assistant Principals for junior high schools reveals that only 55 out of 690 persons (or 8%) were Black and none was Puerto Rican.

Plaintiffs contend that the written and oral examinations of the Board are the major factor accounting for this extremely low percentage of Black and Puerto Rican supervisors in a school system where 55% of the students are Black or Puerto Rican. Plaintiffs summarize their basic argument as follows:

"[T]hese tests place a premium on familiarity with organizational peculiarities of the New York City school system which, while having little to do with educational needs, are largely gained through coaching and assistance from present, predominately white, supervisory personnel.

* * * * * *

"The testing procedures do not indicate a candidate's ability to do the job being tested for. There is no evidence that they measure merit or fitness, they have never been validated, and they are unreliable psychological instruments." Amended Complaint ¶¶ 22, 23.

Rather than risk the endless delay that would be encountered while the parties obtained this essential evidence through pretrial discovery procedures, we directed the parties, in view of the importance of the issue, to use their best efforts to agree on a procedure whereby the Board of Examiners and the Board of Education would compile the necessary racial statistics. All parties cooperated fully and at considerable effort in working out almost all of the details of the procedure to be followed. Such differences as existed were resolved by court order. The result has been that after months of research we have been presented with the pass-fail statistics for the relevant racial and ethnic groupings of candidates for 50 supervisory examinations given over the past few years. In view of plaintiffs' claims that the examinations had a "chilling effect" inhibiting Black and Puerto Ricans from becoming candidates, this statistical survey ("the Survey") also includes figures as to those candidates who "Did Not Appear" to take the written test, which commenced the examination process, or who "Withdrew," i.e., took the written test but did not appear for subsequent parts of the examination.

All parties and amici have submitted briefs as to the relevance of the statistics thus adduced and the inferences that may be drawn from them. The parties also submitted affidavits by statistical experts. A hearing was held, at which each side's expert testified and was subject to cross-examination; and we heard more oral argument on the statistical data. After declining the opportunity to examine and cross-examine any other witnesses, including those presented by affidavit, both sides rested on the record thus adduced.

Upon the evidence thus presented we find that the examinations and testing procedures prepared and administered by the Board for the purpose of determining which candidates will be licensed as supervisory personnel have the effect of discriminating against Black and Puerto Rican candidates.

The Survey reveals that out of 6,201 candidates taking most of the supervisory examinations given in the last seven (7) years, including all such examinations within the last three (3) years, 5,910 were identified by race. Of the 5,910 thus identified, 818 were Black or Puerto Rican and 5,092 were Caucasian. Analysis of the aggregate pass-fail statistics for the entire group reveals that only 31.4% of the 818 Black and Puerto Rican candidates passed as compared with 44.3% of the 5,092 white candidates. Thus on an overall basis, white candidates passed at almost 1 1/2 times the rate of Black and Puerto Rican candidates. These overall figures, however, tell only part of the story. Of greater significance are the results of two examinations which had by far the largest number of candidates, those for Assistant Principal of Day Elementary School and Assistant Principal of Junior High School, which revealed the following:

Total % Pass Total % Pass Total % Pass Total % Pass THAN — 61.3% 45.26% 48.82% 25.60%

Thus white candidates passed the examination for Assistant Principal of Junior High School at almost double the rate of Black and Puerto Rican candidates, and passed the examination for Assistant Principal of Day Elementary School at a rate one-third greater than Black and Puerto Rican candidates. The gross disparity in passing rates on these two examinations is of particular significance not only because they were taken by far more candidates than those taking any other examinations conducted in at least the last seven years, resulting in licensing of the largest number of supervisors, but also because the assistant principalship has traditionally been the route to and prerequisite for the most important supervisory position, Principal. To the extent that Black and Puerto Ricans are screened out by the examination for Assistant Principal they are not only prevented from becoming Assistant Principals but are kept out of the pool of eligibles for future examinations for the position of Principal as well. The fact that the process involves a series of examinations and that to reach the top one must pass several examinations at different times in his or her career serves to magnify the statistical differences between the white and non-white pass-fail rates. For instance, if we take a group of 100 Black and Puerto Rican candidates, on the one hand, and 1,000 white candidates, on the other, and assume a passing rate of 25% for the former and of 50% for the latter on a given assistant principal's examination (as was approximately the case in the examination for Assistant Principal of Junior High School), the results would be as follows:

Black Puerto Rican 25% x 100 = 25 } } Licensed Assistant } Principals White 50% x 1000 = 500 }

The group of 525 licensed assistant principals would then form the pool of eligibles for the related principal's examination. Assuming the same relative pass rates, we have the following results:

Black Puerto Rican 25% x 25 = 6.25 } } Principals White 50% x 500 = 250 }

Thus the true resulting difference between the Black and Puerto Rican versus the white pass rates would be even more substantial: only 6.25% of the Blacks and Puerto Ricans would pass the two successive examinations as against 25% of the whites.

When we look at all 50 examinations which were the subject of the Survey, we find that only 34 were taken by at least one member of both the white and Black-Puerto Rican racial groups. One of these examinations (Assistant Administrative Director, given Dec. 1967, PF-17) was passed by everyone taking it. Another (Director, Bureau for Children with Retarded Mental Development, given Jan. 1968, PF-18) was not completed successfully by anyone taking it. There remain 32 examinations where one or the other of the two main racial groups — Black and Puerto Rican in one group and white in the other — had a larger percentage passing than did the other group. Of these 32 examinations the white group had a larger percentage passing in 25 examinations and the non-white group had a larger percentage passing in only 7 examinations. Thus the whites passed at a proportionately higher rate in three times as many examinations as the non-whites. The probability of these results occurring by chance is less than 1.05 in 1,000. Plaintiffs' Exhibit 2 to oral hearing of 5/21/71. See also transcript of oral hearing of 5/21/71 at 14-19.

Plaintiffs offered the testimony of Dr. Jacob Cohen, an expert in the field of statistics, in support of the validity and significance of the Survey results. The Board, in turn, adduced the testimony of Dr. Nathan Jaspen, an expert in the same field, in opposition. Both witnesses possess outstanding qualifications. After reviewing their testimony and appraising them as witnesses, we are more persuaded by the testimony of Dr. Cohen with respect to certain crucial matters affecting the significance of the figures for present purposes.

Turning first to the gross aggregate pass-fail statistics, which reveal that the overall pass rate of white candidates (44.3%) was almost half again as high as the non-white rate (31.4%), Dr. Cohen testified that on the basis of such a large sample (5,910 out of 6,201 candidates), the test results were especially valuable and formed a sound basis for drawing valid statistical conclusions as to the difference in passing rates between the ethnic groups involved. In analyzing the statistics he used the Chi-Square Test (Yates-corrected), which is a method using formulas generally accepted by statistical experts to determine whether an observed difference in any given sample is greater than that which would be expected on the basis of mere chance or probability. He found with respect to the aggregate test that by "the Chi-Square (Yates-corrected) statistical test, the probability of the difference being a chance result not related to the factor of race is determined as less than one in one billion." (Emphasis added)

In an effort to rebut Dr. Cohen's analysis the Board, after first dismissing the aggregate figures as insignificant on the ground that the "examinations are discrete competitions related to widely-varied, particular supervisory examinations" (Second Supp. Mem. 5/7/71, p. 7), offered Dr. Jaspen's testimony that he did not compute probabilities on the basis of the gross aggregate figures because of the possibility of "overlap," i.e., the taking of more than one test by the same persons. We reject this excuse for several reasons. In the first place, the Board, although it had the data as to any overlap within its control and was afforded the opportunity to adduce any relevant evidence, chose not to do so. Under such circumstances we cannot assume any significant overlap. Secondly, if a "random" overlap existed (i.e., one where the number of persons repeating an examination because of a previous failure approximates those taking a second, more advanced examination after passing the first), it would not substantially affect the probabilities. Dr. Jaspen conceded, for instance, that a random overlap of as much as 50% would not substantially affect the significance of the observed difference between the white and non-white pass rates. Lastly, Dr. Cohen (plaintiffs' expert) testified that because of the large Chi-Square value an overlap would not significantly affect the probability data. In other words, the observed difference between the aggregate pass-fail rates of whites and non-whites was too great to be a mere matter of chance, unrelated to race (the chance of such an occurrence being less than one in a billion) so that even if the probability were reduced somewhat because of overlap, the figure would still be significant, e.g., one in one million, or one in 500,000, and if the overlap were no greater than 15%, the statistical results would not be affected at all.

Turning to the 50 examinations forming the raw material of the Survey, some were taken by very few people. For instance, 41 of the 50 examinations were taken by only 83 (or 10.1%) of the total number of Black and Puerto Rican candidates. Because of the smallness of the sample in each instance the resulting figures of each such examination, when analyzed individually, cannot be accorded much weight or signficance for our purposes. Although statisticians can analyze very small samples through use of a method called the Fisher Exact Probability Test in conjunction with the Chi-Square (Yates-corrected) Test, in our opinion such a sample is less reliable than analyses based on larger samples, even after making allowances for greater margins of error in use of the small sample. Use of the small sample involves more extrapolation and theory superimposed on less fact. We prefer the greater fact content found in the larger sample. For the reasons explained by Dr. Cohen — principally the risk of spurious differences based on insufficient evidence — we do not believe that meaningful conclusions as to differences can be drawn from the meager data derived from any one of these 41 examinations.

Turning to the 9 examinations taken by 10 or more Black and Puerto Rican candidates," we find that Blacks and Puerto Ricans passed at a lower rate than whites in all of these examinations and at a rate traditionally accepted by statisticians as statistically significant in 5 out of the 9 examinations. If we limit our analysis even further to those 6 examinations which were taken by a minimum of 18 candidates from each of the two groups (white, on the one hand, and Black and Puerto Rican, on the other), the results are significant. On all 6 of these examinations the percentage of whites passing is higher than the percentage of Blacks and Puerto Ricans passing, with the differences ranging from 9% to 28%.

Finally we are impressed with the revealing statistics comparing the percentage of Black and Puerto Rican Principals to white Principals in the five largest school systems in the country:

Total No. of % Black and City Principals % Black % Puerto Rican Puerto Rican

Detroit 281 16.7% — 16.7% Philadelphia 267 16.7% — 16.7% Los Angeles 1,012 8.0% 1.7% 9.7% Chicago 479 6.9% — 6.9% NEW YORK 862 1.3% 0.1% 1.4%

Thus New York City has by far the lowest percentage of minority representation. The next lowest city, Chicago, has almost 5 times the percentage of minority principals found in New York City, and as the following table shows there is a similar imbalance of minority Assistant Principals:

Total No. of % Black and City Asst. Principals % Black % Puerto Rican Puerto Rican

Detroit 360 24.7% 0.2% 24.9% Philadelphia 225 37.0% — 37.0% Los Angles — — — — Chicago 714 32.5% — 32.5% NEW YORK 1,610 7.0% 0.2% 7.2%

Plaintiffs also argue that discrimination may be inferred from the fact that the percentage of Black and Puerto Rican Principals and Assistant Principals in New York City schools (1.4% and 7.2%, respectively) is far below the percentage of the total student body who are Black and Puerto Rican (55.8%) and when compared with similar figures for the five largest school systems in the country (New York, Los Angeles, Chicago, Detroit, and Philadelphia) constitutes not only the lowest minority representation in the supervisory ranks, but also the lowest ratio of such minority group supervisors to minority group students. We reject this contention. Supervisors are drawn from the pool of qualified teachers, most of whom attended elementary and high school long ago, and not from present-day students. Undoubtedly the low number of minority teachers eligible to take the supervisory examinations prescribed by the Board has been due in part to the fact that the percentage of minority students who 10 or 15 years ago went on to college and qualified for a teaching career, and thus provided the source of today's minority teachers, was much smaller than the number of white students following such a course, with the result that a larger pool of qualified white graduates entered the teaching profession. In addition the minority student population in New York City has increased during the same period, with the effect of increasing the racial imbalance between teachers and students. Current efforts to promote higher educational opportunities for minority groups will not produce qualified teachers for some years. But statistics as to the current dearth of qualified minority teachers do not have probative value with respect to the question before us, which is whether New York City's examination system discriminates against minority candidates who have already qualified as licensed teachers.

For the same reasons we are unimpressed with plaintiffs' comparisons between (1) the percentage of Black and Puerto Rican members of the general population in New York City, and (2) the percentage of Black and Puerto Rican Principals and Assistant Principals found in the City's total school supervisory personnel. Statistical comparisons to the general racial population of the community may be relevant in determining whether there is discrimination in job opportunities that are supposed to be open to the general public, see, e.g., Arrington v. Mass. Bay Transp. Auth., 306 F. Supp. 1355 (D.Mass. 1969) (bus drivers); Penn v. Stumpf, 308 F. Supp. 1238 (N.D.Cal. 1970) (policemen), in the selection of teachers from a pool of those already qualified and eligible for appointment, e.g., Porcelli v. Titus, 302 F. Supp. 726 (D.N.J. 1969), affd., 431 F.2d 1254 (3d Cir. 1970) (per curiam), or in qualification of voters or jurors, Smith v. Texas, 311 U.S. 128, 61 S.Ct. 164, 85 L.Ed. 84 (1940); Jones v. Georgia, 389 U.S. 24, 88 S.Ct. 4, 19 L.Ed.2d 25 (1967); Coleman v. Alabama, 389 U.S. 22, 88 S.Ct. 2, 19 L.Ed.2d 22 (1967); Swain v. Alabama, 380 U.S. 202, 208-209, 85 S.Ct. 824, 13 L.Ed.2d 759 (1967). But we are here dealing with candidates who must meet preliminary eligibility requirements as to education and experience that are not possessed by most of the general population. Where the education of our children is at stake, such insistence upon the highest possible quality in our teachers is a salutary and lawful objective, provided it does not result in racial discrimination between candidates who are otherwise eligible, which is the case here.

Notwithstanding the introduction of some evidence thus found irrelevant, the evidence establishes to our satisfaction that the examinations prepared and administered by the Board for the licensing of supervisory personnel in New York City schools do have the de facto effect of discriminating significantly and substantially against qualified Black and Puerto Rican applicants. However, the existence of such discrimination, standing alone, would not necessarily entitle plaintiffs to relief. The Constitution does not require that minority group candidates be licensed as supervisors in the same proportion as white candidates. The goal of the examination procedures should be to provide the best qualified supervisors, regardless of their race, and if the examinations appear reasonably constructive to measure knowledge, skills and abilities essential to a particular position, they should not be nullified because of a de facto discriminatory impact. We accordingly pass on to the question of whether the examinations under attack can be validated as relevant to the requirements of the positions for which they are given, i.e., whether they are "job-related."

The parties disagree as to which side bears the burden of proving that the examinations are job-related. Plaintiffs contend that once a discriminatory impact is shown the burden is on the Board to show a compelling necessity or justification for tests having an unintended discriminatory effect, citing Griggs v. Duke Power Co., 401 U.S. 424, 91 S.Ct. 849, 28 L.Ed.2d 158 (1971); Arrington v. Mass. Bay Transp. Auth., 306 F. Supp. 1355, 1358 (D.Mass. 1969); Gregory v. Litton Systems, 316 F. Supp. 401 (C.D.Cal. 1970); Hicks v. Crown Zellerbach Corp., 319 F. Supp. 314 (E.D.La. 1970). Most of these decisions arise under Title VII of the 1964 Civil Rights Act, 42 U.S.C. § 2000e et seq., which embodied an express Congressional policy directed toward the consequences of certain employment practices, regardless of the employer's motive or interest, and interpreted by the Supreme Court as placing on the employer the burden of showing that any given testing mechanism or requirement has a manifest function in measuring a candidate's capability of performing the employment in question. See Griggs v. Duke Power Company, 401 U.S. 424, 91 S.Ct. 849, 854, 28 L.Ed.2d 158 (1971). Since we are here concerned, however, with whether the Board's examinations meet the requirements of the Equal Protection Clause rather than those of a specific Congressional enactment, the foregoing authorities, while relevant, are not controlling.

The Board contends that in a Constitutional framework plaintiffs must show that there is no rational relationship between the examinations and the requirements of the supervisory positions for which they are given, relying on a line of cases that include Dandridge v. Williams, 397 U.S. 471, 485, 90 S.Ct. 1153, 25 L.Ed.2d 491 (1970), and Chaney v. State Bar of California, 386 F.2d 962, 964 (9th Cir. 1967), cert. denied, 390 U.S. 1011, 88 S.Ct. 1262, 20 L.Ed.2d 162 (1968).

Although the "rational relationship" standard has been applied to practices attacked as causing commercial or economic harm, e.g., McGowan v. Maryland, 366 U.S. 420, 426, 81 S.Ct. 1101, 6 L.Ed.2d 393 (1961) (statute forbidding sale of certain items on Sunday and not others); Lindsley v. Natural Carbonic Gas Co., 220 U.S. 61, 31 S.Ct. 337, 55 L.Ed. 369 (1911) (regulating drilling for natural gas); Morey v. Doud, 354 U.S. 457, 463, 464, 77 S.Ct. 1344, 1 L.Ed.2d 1485 (1957) (regulating use of premises as "currency exchanges"); Williamson v. Lee Optical of Okla., 348 U.S. 483, 489, 75 S.Ct. 461, 99 L.Ed. 563 (1954) (regulating the sale of eye glasses); Ft. Smith Light Traction Co. v. Paving District, 274 U.S. 387, 47 S.Ct. 595, 71 L.Ed. 1112 (1926) (imposing duty to repair on street railways), we are here dealing with racial, not economic, discrimination, where even reputed strict constructionists have joined in the view that a more stringent standard must be applied. See Harper v. Virginia State Board of Elections, 383 U.S. 663, 86 S.Ct. 1079, 16 L.Ed.2d 169 (1966), and dissenting opinion of Justice Harlan in Shapiro v. Thompson, 394 U.S. 618, 659, 89 S.Ct. 1322, 22 L.Ed.2d 600 (1969). Where official conduct discriminates as to race, it is "constitutionally suspect," Bolling v. Sharpe, 347 U.S. 497, 499, 74 S.Ct. 693, 98 L.Ed. 884 (1954), subject to "most rigid scrutiny," Korematsu v. United States, 323 U.S. 214, 216, 65 S.Ct. 193, 89 L.Ed. 194 (1944), and bears a "very heavy burden of justification," Loving v. Virginia, 388 U.S. 1, 9, 87 S.Ct. 1817, 18 L.Ed.2d 1010 (1967), regardless of lack of intent to discriminate racially, see Hobson v. Hansen, 269 F. Supp. 401, 426, 497 (D.D.C. 1967), affd. sub. nom. Smuck v. Hobson, 132 U.S.App.D.C. 372, 408 F.2d 175 (1969). In a closely parallel situation arising under Title VII of the 1964 Civil Rights Act, 42 U.S.C. § 2000e et seq. in Griggs v. Duke Power Co., 401 U.S. 424, 432, 91 S.Ct. 849, 854, 28 L.Ed.2d 158 (1971), the Supreme Court pointed out that "good intent or absence of discriminatory intent does not redeem employment procedures or testing mechanisms that operate as 'built in headwinds' for minority groups and are unrelated to measuring job capability. * * *" See also, Baker v. Columbus Municipal Separate School District, 329 F. Supp. 327 (N.D.Miss. 1971); Western Addition Community Organization v. Alioto, 330 F. Supp. 536 (N.D.Cal., Jan. 8, 1971).

We are satisfied that where, as here, plaintiffs show that the examinations result in substantial discrimination against a minority racial group qualified to take them, a strong showing must be made by the Board that the examinations are required to measure abilities essential to performance of the supervisory positions for which they are given. Shapiro v. Thompson, 394 U.S. 618, 627, 89 S.Ct. 1322, 22 L.Ed.2d 600 (1969); Arrington v. Mass. Bay Transp. Auth., 306 F. Supp. 1355, 1358 (D.Mass. 1969); Penn v. Stumpf, 308 F. Supp. 1238 (N.D.Cal. 1970).

Before considering the evidence with respect to the validity, reliability and objectivity of the examinations conducted by the Board, a few general principles must be stated. It seems to be generally accepted that before an examination will be recognized as a reliable instrument for measuring the fitness and ability of a candidate to perform tasks demanded by a given position, the examination should be validated, i.e., shown to be reasonably capable of measuring "what it purports to measure." O.G. Stahl, Public Personnel Administration, p. 71 (5th ed. 1962). The first step toward this basic objective is to insure that as to subject matter the examination will elicit from the candidate information that is relevant to the job for which it is given. If so, it is described as having "content validity." Otherwise the examination could be a useless and misleading tool. For instance, an examination eliciting information required to perform a bus-driver's job would hardly be relevant in determining a candidate's capacity to perform the duties of a policeman, lawyer or accountant.

In constructing an examination that will have "content validity," the preferred course is first to have an empirical analysis made of the position for which it is given, usually by experts or professionals in the field. Such an analysis requires a study to be made of the duties of the job, of the performance by those already occupying it, and of the elements, aspects and characteristics that make for successful performance. Questions are then formulated, selective procedures established, and criteria prepared for examiners that should elicit information enabling them to measure these characteristics, skills and proficiency in a candidate and determine his capacity to do the job satisfactorily.

As Professor R.L. Thorndike has observed:

"Whenever a test is being tried for selection of personnel for some job specialty, it is most desirable that it be validated empirically. Experimental evidence is called for to show that the test is in fact effective in discriminating between those who are and those who are not successful in a particular job. Though it may be necessary under the press of an emergency to rely upon the professional judgment of the psychologist to establish the value of a test for personnel selection, this must be recognized as a stopgap." R.L. Thorndike, Personnel Selection 5-6 (1949). See also, R.L. Thorndike E. Hagen, Measurement and Evaluation in Psychology and Education 616-41 (1969); cf. 29 C.F.R. § 1607.5 (Jan. 1, 1971).

To a lesser extent the validity of an examination as a means of selecting candidates best suited for a position may also be checked or verified empirically by comparing the relative examination scores of successful candidates with their later performance on the job. If there is a significant correlation between test scores and later performance, the examination has "predictive validity." Predictive validity is of greater significance in evaluating aptitude tests than proficiency tests. Furthermore it often takes a long time to establish such validity and even then the evaluation depends upon the reliability and fairness of the field appraisal of performance on the job.

Lastly, an examination must, of course, be administered objectively, free from bias or prejudice in favor of or against particular candidate or group, if it is to be a useful selection tool.

At the outset of the hearings, being inexperienced in the field of examinations generally, we indicated doubt as to whether examinations could be constructed that would be valid for selection of Principals and other supervisory personnel, since we viewed their duties as being executive and complex in nature, with the success of a Principal in a given school depending not so much on his knowledge of duties and educational content of courses given by his subordinates as on such intangible factors as leadership skill, sensitivity to the feelings and attitudes of teachers, parents and children, and ability to articulate, to relate, to organize work, to establish procedures, to promote good community relations, to induce subordinates to accept directions, to work cooperatively, to criticize without creating unnecessary animosity or ill-will, to analyze and evaluate administrative problems, to take decisive action when required, to operate under stress, to initiate and promote new programs, and to instill a feeling of confidence. In short we questioned whether tests that might be valid for purposes of determining a candidate's knowledge of the duties of a position or of detailed educational information would be valid for purposes of determining his judgment and ability as an executive, particularly since the candidate, being a licensed teacher, has already demonstrated his or her technical skills in certain fields of education, including ability to read, write and speak English.

Notwithstanding our doubt, both sides in this dispute agree that while examination procedures may have weaknesses in testing for higher level executive positions, such procedures (including use of written tests) are essential tools in selecting supervisory personnel. However, they differ as to the areas of weakness. Plaintiffs, for instance, state that "content validity is of limited utility in selecting persons for fairly sophisticated or complex job" because of the difficulties faced in preparing tests that fairly sample the job and accurately predict a candidate's performance. See F. Freeman, Theory and Practice of Psychological Testing, p. 91 (3d ed. 1963); A Anastasi, Principles of Psychological Testing, pp. 100-04 (3d ed. 1968). According to plaintiffs, examinations for such positions are useful only if they have predictive validity, since content validity is primarily relevant for the purpose of determining whether a candidate has learned a defined body of knowledge rather than for the purpose of determining how he will use and apply that knowledge on the job. The Board, on the other hand, takes the view that content validity is more important in determining a candidate's proficiency or capacity to perform the duties of a Principal, and that predictive validity should be de-emphasized in judging the utility of such tests because predictive validity is more relevant to aptitude for learning than to achievement or proficiency for satisfactory performance on the job. The Board further points out that follow-up studies used to determine predictive validity are not generally used in licensing of doctors, lawyers, or other professionals.

The typical supervisory examination consists of two parts: (1) a written test, and (2) an oral interview. In addition, a problem-based conference test has been used in some years for examinations for positions such as Assistant Director and Director. The percentage weight attributed to each part of the examination varies according to the supervisory position involved. Usually about 45 to 50% is accorded to the written examination and 25 to 30% to the oral examination and another 25 to 30% to an appraisal of the candidate's training and experience, record, written English, and physical and medical condition. More recently in an examination given for license as Principal of a Day Elementary School the written and oral interview tests were weighted 50% each with no weight apparently being given to the other factors mentioned above.

In the past the written test has usually consisted of an essay portion and a short-answer section, the latter consisting of a series of approximately 200 multiple choice questions, each of which requires an answer-number to be selected and registered by the applicant on a separate answer sheet. The oral interview is conducted by a committee of three examiners. A hypothetical problem situation that might be encountered by a Principal or supervisor in the course of administration (e.g., problem in human relations, teacher training, or administration of a program) is put before the candidate, followed by questions to which he or she responds orally at some length. The committee then evaluates the applicant's speech, grammar, clarity of expression, comprehension of the problem, definiteness and practicality of his or her proposals, soundness of judgment, ability to present ideas and meet challenges, poise, courtesy and similar qualities. Recently the examination for Principal has changed to consist of only an essay type test, with the short-answer portion deleted for the reason that it was largely a matter of memory. However, the short-answer section has been retained in at least some written examinations for Assistant Principal, which is an essential stepping-stone to the position of Principal.

The Board contends that its examinations are valid, reliable and objective. It points to several steps taken in an effort to achieve and maintain these goals. It asserts that for each examination it has obtained from the Board of Education a statement of the "duties of the position" for which the examination is to be given. A committee or panel of experts is then assembled to specify those responsibilities considered most significant. According to the Board, well known interested educators and lay persons are also consulted with respect to the qualities for which a candidate is to be tested. With the aid of these committees, educators and lay persons, the Board's staff then constructs questions designed to elicit the knowledge and skills required of a person holding the supervisory position for which the examination is given.

According to the Board it achieves objectivity in the scoring of written examinations by use of code book numbers on test papers, which conceal the applicant's identity, by submitting answers on any test to more than one scorer, by use of detailed objective rating guidance, and by having answers to essay questions rated by different committees so that no one individual is responsible for rating of an entire examination paper. Oral interviews are each administered by panels of three experts.

In support of its position the Board has submitted affidavits of several respected leaders in the field of educational testing to the effect that they have reviewed the Board's testing procedures and some of the tests prepared by the Board and have concluded that the Board has used acceptable procedures for designing examinations having content validity:

"In general, the Board of Examiners appears to have made a conscientious and informed attempt to develop test tasks that do correspond to selected ones of the specifications set forth by the supervisory persons who set out the requirements for the job." (Aff. of Dr. Robert L. Thorndike, dated October 9, 1970)

"The approach used by the Board of Examiners in determining the validity or relevance of its tests consists essentially of a strategy which relies on the judgments of experts and consensus among them as to what constitutes an appropriate test item." (Aff. of Dr. Aaron Carton, dated October 25, 1970)

As thus described, the Board's methods and procedures seem reasonable enough to one lacking expertise in the field of educational testing, and normally we would be prepared to accept the Board's views on the subject. Indeed, there appear to be few differences between the procedures advocated by the Board and by plaintiffs' expert, Dr. Richard S. Barrett. The basic aims of both are the same (to achieve validity, reliability and objectivity), with the parties differing as to the need for and importance of predictive validity in administration of supervisory examinations. However, a major stumbling block — and in our view a fatal weakness in the Board's system — lies in the methods used by the Board to implement the techniques and procedures adopted in principle and approved by independent experts. As is often the case in such difficult and complex matters, theory is one thing, practice may be another. Despite its professed aims the Board has not in practice taken sufficient steps to insure that its examinations will be valid as to content, much less to predictiveness. For example, in support of the Board's content validation procedure Dr. Murray Rockowitz, a member of the Board with primary responsibility for research and development, includes the name of Peter J. Strauss, member of Local School Board No. 2, as one of the "experts" or "well known interested lay persons" consulted at a meeting on December 12, 1969 for advice on qualities being tested for in the construction of an examination for Elementary School Principal, which was given on November 3, 1970. (Rockowitz Aff. dated Oct. 26, 1970). In reply Mr. Strauss has submitted his affidavit to the effect that the meeting was not called for the purpose of obtaining views on the qualities to be tested for or to discuss appropriate selection criteria, but for other purposes and that these steps were initiated by dissatisfied consultants, with nothing being accomplished. Mr. Strauss states:

"It is my recollection of the meeting that the ensuing discussion of the qualities to be sought in a candidate was initiated by the consultants, not by Mr. Rockowitz or any other representative of the Board of Examiners. Many of us expressed our dissatisfaction with the adequacy and relevance of the qualities which the Board of Examiners was apparently intending to test for, based on the established 'duties of the position.' We indicated that in our view it was essential to test for qualities which traditionally have never been considered by the Board of Examiners in examining principals.

"5. The meeting concluded without any sort of consensus or agreement having been reached between the consultants present at the meeting and the members of the Board of Examiners and their assistants present. There was never any follow-up to the meeting.

"6. To my knowledge, the 'duties of the position' previously established were in no way varied or modified as a result of the meeting. On August 28, 1969 the Board of Examiners issued an announcement for an examination for the position of Principal, Day Elementary School. Said announcement included a listing of the 'duties of the position.'" (Aff. Peter J. Strauss, dated Nov. 4, 1970).

It also appears that Harvey B. Scribner, the Chancellor of the New York City School District, a distinguished and fearless educator, does not share the Board's confidence in the validity of its examinations. In a memorandum to the Board of Education dated October 13, 1970, Dr. Scribner noted that he was "pressed to evaluate whether the present examination and licensing system, which dictates specific limitations of employment and promotion of staff for the public schools, is a help or a hardship [in the efforts of community boards to operate] * *." He recommended that in lieu of current employment practices the Board adopt New York State certification, plus such prescribe for those to be selected by it, criteria as each community board might as the minimum requirement for employment in New York City public schools. He concluded:

"For the reasons outlined in this position paper, my position with regard to the Chance and Mercado case is that I prefer not to defend myself against the action. To do so would require that I both violate my own professional beliefs and defend a system of personnel selection and promotion which I no longer believe to be workable. We are facing a future in education which leaves no alternative to the selection of the most creative teachers; the most talented supervisors; the most able principals and other administrators who possess the highest level of leadership qualities possible from wherever they may be found and as are available at any given time."

Digging deeper, we find to our dismay that the Board's position does not appear to be supported by most of the research reports submitted by it as demonstrating the content validity of its supervisory examinations. Many of the reports are irrelevant, since they concern teachers', not supervisors', examinations; or they deal with objectivity or consistency; or they are mere proposals, not studies. One report merely measures the attitudes of the assistant examiners towards the "observed teaching test." Another attempts to correlate the similarity of a 300-word English test with a 450-word English test. One report relied on by the Board does deal tangentially with the predictive validity of the examinations. It is based on a pilot study, admittedly very meager in scope. The test results showed that there was little or no correlation between success on the tests and job success (in some cases the control group did better, in others the experimental group did better, in some cases there was no statistical difference). There was no attempt in this pilot study to determine whether those who scored lower on the examination (but passed) did less well as Principals. Neither was there an attempt to determine whether those who failed the examination but who are Acting Principals are rated as successful or unsuccessful on the job. In only two out of the six years tested (1946 and 1966) did the differences reach statistically significant levels in favor of the Board's position, and in those two years the report candidly admitted: "* * * the relatively small population of the samples involved should be noted." Finally, the entire study reveals only the mean rank order of the examination scores.

Reluctant as we are to invade a profession characterized by an expertise not shared by us, we must conclude on the record before us that while the Board has adopted procedures designed for content validity, it does not appear in practice to have achieved this goal. Our conclusion, which is based upon our appraisal of affidavits of experts furnished by the parties, is confirmed by our own study of some of the examinations themselves. Turning first to the short-answer, multiple-choice tests, many of the questions strike us as having little relevance to the qualities expected of a school supervisor. Some examples are cited in the margin.

A review of the balance of a typical short-answer test and, indeed, of the essay type test reveals that the questions appear to be aimed at testing the candidate's ability to memorize rather than the qualities normally associated with a school administrator. This impression is confirmed by a well known publication widely used by candidates to prepare for supervisory examinations, which is entitled "PRINCIPAL, ASSISTANT-TO-PRINCIPAL" by Paul Treatman, Ph.D., Principal, New York City Public Schools (Teacher License Series, ARCO). With respect to the short-answer test it advocates a course of training in memorization, or mnemonics, including study of Regents' review books, multiple choice parts of previous examinations and the like. Interesting and instructive as would be the reading of the underlying works, the emphasis is on superficial mnemonics. The ability to memorize and regurgitate laundry lists of bad answers is not, we hope, a true test of a candidate's qualifications for a supervisory position.

The inadequacy of such testing procedures led the Board finally to drop the short-answer test for Principal because, as Dr. Isidore Bogen, a Board member stated, "The Board of Examiners felt it was largely a matter of memory." Notwithstanding this frank admission, the short-answer examination has been retained in tests for Assistant Principal, apparently on the theory that an Assistant Principal should know the content of the curriculum taught by teachers under his supervision. While we might be prepared to accept this compromise if it were the sole objectionable feature, the same weakness is found in many of the essay question tests, which one would expect to be used for the purpose of eliciting some of the qualities required of a Principal (e.g., accurate grammar, clarity of expression, organization and presentation of ideas, and creative thinking) as well as the candidate's reaction to problems and his judgment and ability in analyzing and solving them. Many essay questions, however, appear to be aimed solely at the ability to memorize "duties." Indeed the Treatman manual, supra, gives the following advice to candidates preparing for essay questions:

"One step in this direction is the mastery of mnemonic devices, such as those found in the previous chapter, that help organize your knowledge of curriculum. Another and related valuable approach is the mastery and use of a set of mnemonic devices which will help organize your knowledge and experience in administrative and supervisory aspects of school life. Such a set of mnemonics appears below. In answering any essay question which seems to lend itself to mnemonic application, the candidate will have to use his own good judgment in deciding which elements of which devices are most appropriate for use. Blind exploitation of every element in a mnemonic device might result in a fatal waste of time.

* * * * * *

"A careful study of these sample answers will reveal to the candidate the technique of writing concise, credit-laden statements that outlined model answers cannot reveal. The author is convinced that the outlined model answer teaches little or nothing to the candidate, who must still see, and learn to write in, a terse style, employing many concrete illustrations, and following, wherever possible a recognizable organization * * *." ("Principal, Assistant-To-Principal" by Paul Treatman, supra at 212-13)

Samples of some mnemonic methods and professional jargon suggested by the author are given in a footnote.

Turning to the oral interview and conference tests, we are handicapped in our ability to appraise their validity by the absence of any tangible, objective data, since the principal qualities that should be evaluated by the examining committee in the oral interview (e.g., leadership, speech, poise, comprehension, soundness of judgment and ability to present ideas and meet challenges) are intangible, and the judgment of the interviewing committee can only be as sound as the capacity of its members to recognize the presence or absence of these qualities in each candidate. In the absence of any record of such interviews or of any proof that the examiners are incompetent or that the interview problems or questions have been irrelevant, we make no finding as to the content validity of the oral examinations, standing alone.

With respect to the question of the objectivity of the examinations, adequate precautions appear to have been taken by the Board to assure objectivity in the conduct of the written examinations. As for the oral interviews, we are not persuaded by vague and speculative hearsay statements that members of the examining committees intentionally discriminate against Black or Puerto Rican candidates because of their color, use of "southernisms," or the like. (See Aff. of Edythe J. Gaines). Since each candidate has already been licensed as a teacher and has received New York State certification for a supervisory position, the examiners are dealing with a group of well educated applicants. On the other hand, the hard, cold facts are that all members of the Board of Examiners are white, the great majority of the oral examiners or examination assistants are white, and white candidates have passed the combined oral and written tests at a much higher rate than Black and Puerto Rican candidates, resulting in de facto discrimination against the latter. This raises a "serious and substantial question" as to whether discrimination against Blacks and Puerto Ricans is not being unconsciously practiced by white interview examiners. See Checker Motors Corp. v. Chrysler Corp., 405 F.2d 319, 323 (2d Cir. 1969).

CONCLUSION

The evidence reveals that the examinations prepared and administered by the Board of Examiners for the licensing of supervisory personnel, such as Principals and Assistant Principals, have the de facto effect of discriminating significantly and substantially against Black and Puerto Rican applicants.

Despite the fact that candidates for such positions are licensed teachers who have satisfied prerequisites as to education and experience established by the Board of Education for supervisory positions and have already been certified by the State of New York for the positions sought, a survey of the results of examinations taken by 5,910 applicants (of whom 818 were Black or Puerto Rican) reveals that white candidates have received passing grades at almost 1 1/2 times the rate of Black and Puerto Rican candidates and that on one important examination given in 1968 for the position of Assistant Principal, Junior High School, white candidates passed at almost double the rate of Black and Puerto Rican candidates. (See p. 210, supra). The discriminatory effect in the latter case is aggravated by the fact that the Assistant Principalship has traditionally been an essential prerequisite to the more important supervisory position of Principal. (See pp. 210, 211, supra).

The existence of such de facto racial discrimination is further confirmed by the fact that only 1.4% of the Principals, and 7.2% of the Assistant Principals in New York City schools are Black or Puerto Rican, percentages which are far below those for the same positions in the four other largest city school systems in the United States. (See pp. 212, 213, supra). For example, the percentage of Black and Puerto Rican Principals in each of the cities of Detroit and Philadelphia is 16.7%, or 12 times as high as that in New York.

Such a discriminatory impact is constitutionally suspect and places the burden on the Board to show that the examinations can be justified as necessary to obtain Principals, Assistant Principals and supervisors possessing the skills and qualifications required for successful performance of the duties of these positions. The Board has failed to meet this burden. Although it has taken some steps towards securing content and predictive validity for the examinations and has been improving the examinations during the last two years, the Board has not in practice achieved the goals of constructing examination procedures that are truly job-related. Many objectionable features remain, with the result that some 37 minority Acting Principals and 131 minority Acting Assistant Principals, who are considered fully qualified and are desired for permanent appointment by the community school boards, are rendered ineligible for such permanent appointment. A study of the written examinations reveals that major portions of them call simply for regurgitation of memorized material. Furthermore, the oral examination procedure leaves open the question of whether white candidates are not being favored — albeit unconsciously — by committees of examination assistants who have been entirely or predominantly white.

There appears to be a strong likelihood that plaintiffs will prevail on the merits at trial. It further appears that plaintiffs would suffer greater harm from denial of preliminary injunctive relief than defendants would suffer from the granting of relief. Denial of relief would perpetuate existing racial discrimination, depriving plaintiffs and others similarly situated of an equal opportunity for permanent appointment and licensing as supervisors. During the long period before the case would finally be adjudicated on the merits, permanent appointments would be made from lists promulgated by the Board, which would have the effect of threatening the continued employment of those holding acting appointments, since New York law requires vacancies to be filled from the eligibility lists if such lists exist. N.Y. Education Law § 2573(2); Board of Education By-Laws § 101(3).

Granting of preliminary relief, it is true, will temporarily prevent the appointment from eligible lists of those who have, after the arduous process of taking the existing type of examinations, successfully passed and been placed on the eligible list. However, they will not be denied an equal opportunity in the future to qualify under such examination procedures as are found to be constitutionally permissible and, pending trial of the case, they would be eligible for appointment as Acting Principals or Acting Assistant Principals. Thus the balance of hardships tips decidedly in favor of plaintiffs, and, pending final determination of the merits, the effect of preliminary relief would be to preserve the status quo until the issues are resolved. Checker Motors Corp. v. Chrysler Corp., 405 F.2d 319, 323 (2d Cir. 1969).

Having in mind that the existing examination system is not believed by the Chancellor of the New York City District to be a workable one, we do not envisage any great harm to the public as a result of preliminary relief. On the contrary, such relief may possibly lead the Board of Examiners, after taking another hard look at its examination procedures, to consider an overhaul that will not only eliminate racial discrimination, but lead to procedures that will be more adaptable to the Community School Board type of administration.

Lastly we cannot overlook the fact that various persons having the duty of selecting supervisory personnel, such as members of community school boards, have stated in affidavits filed with the court that they have often found that holders of licenses from the Board of Examiners do not possess the ability to perform the duties of a supervisory position for which a candidate is sought, with the result that in order to select qualified personnel it has been necessary to appoint unlicensed candidates on an acting basis. (See Affs. of Dr. Edwin J. Haas, Edythe J. Gaines, and Peter J. Strauss).

For the foregoing reasons a preliminary injunction will issue restraining defendants from (1) conducting further examinations of the type found to be unconstitutionally discriminatory against Blacks and Puerto Ricans, and from (2) promulgating eligible lists on the basis of such examination procedures.

The foregoing shall constitute our findings of fact and conclusions of law as required by Rule 52(a), F.R.Civ.P.

We take this opportunity to express appreciation to the parties for their thorough papers, and to the amici for their briefs, which were of assistance in resolving the difficult and complex issues.

Settle order on notice.

* * * * * * * *

"75. A contemporary play in which one of the characters seems to turn into a rhinoceros was written by: