CUMMINGS, Circuit Judge.

The plaintiff brought this action under the Privacy Act of 1974, 5 U.S.C. § 552a, alleging that the defendants compiled and withheld from him certain records that were used by his superiors in evaluating his work performance while he served as a lieutenant colonel in the United States Army Reserve and, upon his retirement from the military in 1979, as a full-time civil service employee. The plaintiff alleged that the private records adversely affected his career by depriving him of deserved promotions and by forcing him to accept early retirement from his civilian employment at the end of 1983 and, as a consequence, lower pension benefits. The amended complaint sought a variety of remedies, including the removal of all tainted records and damages for both economic losses and emotional injury.

Pursuant to Fed.R.Civ.P. 12(b)(1) the district court granted the defendants' motion to dismiss the plaintiff's amended complaint for lack of subject matter jurisdiction because the plaintiff had failed to allege exhaustion of administrative remedies. Before an individual may bring a Privacy Act suit seeking amendment of a record maintained by an agency, he must first have exhausted his administrative remedies for having the improper material removed from the record. 5 U.S.C. § 552a(g)(1). Essentially the plaintiff must request the agency to amend the relevant record, id. § 552a(d)(2), and if the agency refuses to amend the record, he must seek agency review of the refusal, id. § 552a(d)(3). The agency is required to complete its review and make a final determination within 30 days unless the head of the agency for good cause extends the period. Id. If the reviewing official also refuses to amend the record in accordance with the request, the individual must be notified of his right to seek judicial review of the agency's determination. Id. In dismissing the plaintiff's amended complaint, the district court explained that the plaintiff failed to allege that he had sought agency review of the Army's and the Office of Personnel Management's refusals to amend his records.

The plaintiff has appealed the district court's order dismissing his pleading, claiming that he was not required to exhaust his administrative remedies. While conceding that exhaustion is required prior to filing an action against an agency for its failure to amend a record, the plaintiff relies on Nagel v. U.S. Department of Health, Education Welfare, 725 F.2d 1438, 1441 n. 2 (D.C. Cir. 1984), and Hewitt v. Grabicki, 794 F.2d 1373, 1379 (9th Cir. 1986), to argue that exhaustion of administrative remedies is not a precondition to bringing an action, like his, seeking damages under § 552a(g)(4). We do not reach the exhaustion issue because the plaintiff's action must be dismissed on an alternative jurisdictional ground. The plaintiff has failed to comply with the two-year statute of limitations set forth in 5 U.S.C. § 552a(g)(5):

An action to enforce any liability created under this section may be brought in the district court of the United States . . . within two years from the date on which the cause of action arises, except that where an agency has materially and willfully misrepresented any information required under this section to be disclosed to an individual and the information so misrepresented is material to establishment of the liability of the agency to the individual under this section, the action may be brought at any time within two years after discovery by the individual of the misrepresentation.

Courts have consistently held that where the government's consent as sovereign to be sued is conditioned upon the filing of suit within a specified period of time, strict compliance with that condition is a jurisdictional prerequisite. See, e.g., McIntyre v. United States, 789 F.2d 1408, 1411 (9th Cir. 1986) (action to quiet title against U.S., 28 U.S.C. § 2409a(f)); Clifton v. Heckler, 755 F.2d 1138, 1144-1145 (5th Cir. 1985) (action against U.S. for attorney's fees under the Equal Access to Justice Act, 28 U.S.C. § 2412(d)(1)(B)); Charlton v. United States, 743 F.2d 557, 558-559 (7th Cir. 1984) (suit under Federal Tort Claims Act, 28 U.S.C. § 2401(b)); Sims v. Heckler, 725 F.2d 1143, 1145 (7th Cir. 1984) (suit against U.S. under Title VII); Walters v. Secretary of Defense, 725 F.2d 107, 112 n. 12 (D.C. Cir. 1983) ( 28 U.S.C. § 2401(a)); Anderberg v. U.S., 718 F.2d 976, 977 (10th Cir. 1983) ( 28 U.S.C. § 2401(a)); Monark Boat Co. v. NLRB, 708 F.2d 1322, 1326-1327 (8th Cir. 1983) (action against U.S. for attorney's fees under the Equal Access to Justice Act, 5 U.S.C. § 504(a)(2)); Garrett v. United States, 640 F.2d 24, 26 (6th Cir. 1981) ( 28 U.S.C. § 2401(b)); Knapp v. United States, 636 F.2d 279, 282 (10th Cir. 1980) ( 28 U.S.C. § 2409a(f)); 14 Wright, Miller Cooper, Federal Practice and Procedure § 3654 (1985). This characterization of the statute of limitations as jurisdictional in cases against the federal government arises out of the doctrine of sovereign immunity. As the Supreme Court has repeatedly held, "the United States, as sovereign, is immune from suit save as it consents to be sued . . . and the terms of its consent to be sued in any court define that court's jurisdiction to entertain the suit." Lehman v. Nakshian, 453 U.S. 156, 160, 101 S.Ct. 2698, 2701, 69 L.Ed.2d 548 (quoting United States v. Sherwood, 312 U.S. 584, 586, 61 S.Ct. 767, 769, 61 S.Ct. 767. In Block v. North Dakota, 461 U.S. 273, 287, 103 S.Ct. 1811, 1819, 75 L.Ed.2d 240, the Court specifically addressed statutes of limitations: "When waiver legislation contains a statute of limitations, the limitations provision constitutes a condition on the waiver of sovereign immunity." See also Simon v. United States, 244 F.2d 703, 705 (5th Cir. 1957) (statute containing time limit for filing suit against U.S. "is not a statute of limitations but a statute granting for a limited time the right of action afforded"). Conditions attached to the sovereign's consent to be sued must be strictly construed. Block, 461 U.S. at 287, 103 S.Ct. at 1819; Lehman, 453 U.S. at 160-161, 101 S.Ct. at 2701; United States v. Kubrick, 444 U.S. 111, 117-118, 100 S.Ct. 352, 356-357, 62 L.Ed.2d 259.

No federal court has expressly addressed whether the two-year statute of limitations contained in the Privacy Act, 5 U.S.C. § 552a(g)(5), is a jurisdictional prerequisite to bringing suit. The statutory time limitation, however, is unquestionably an integral condition of the sovereign's consent to be sued under the Privacy Act. Accordingly, we hold that a plaintiff's failure to file suit within the time period specified in § 552a(g)(5) deprives the federal courts of subject matter jurisdiction over the action.

Although the district court dismissed the plaintiff's amended complaint solely on the exhaustion issue, the defendants raised the statute of limitations as an alternative jurisdictional ground in their motion to dismiss, and the issue was fully briefed by all of the parties. The defendants have again pressed the statute of limitations on appeal as an alternative ground for affirming the district court's order, although they have admittedly given the argument rather short shrift in their brief. See, e.g., Martinez v. United Automobile, Aerospace Agricultural Implement Workers, 772 F.2d 348, 353 (7th Cir. 1985) (court of appeals can affirm on any ground — even one not passed on by the district judge — that the record fairly supports and the appellee has not waived); Liberles v. County of Cook, 709 F.2d 1122, 1130 (7th Cir. 1983); Gimix, Inc. v. JS A Group, Inc., 699 F.2d 901, 904 (7th Cir. 1983). Furthermore, because the issue is one of jurisdiction, we of course have the power and duty to reach it on our own. See, e.g., Hiley v. United States, 807 F.2d 623, 626 (7th Cir. 1986); Christianson v. Colt Industries Operating Corp., 798 F.2d 1051, 1055-1056 (7th Cir. 1986). Because the factual record developed in the district court supports the defendants' limitations argument on appeal, the plaintiff's action is barred by § 552a(g)(5).

In Bergman v. United States, 751 F.2d 314, 316 (10th Cir. 1984), certiorari denied, ___ U.S. ___, 106 S.Ct. 310, 88 L.Ed.2d 287, the court held that a cause of action arises under the Privacy Act at the time:

(1) an error was made in maintaining the plaintiff's records;

(2) the plaintiff was wronged by such error; and

(3) the plaintiff either knew or had reason to know of such error.

The plaintiff in this case filed his original complaint on February 1, 1984. The amended complaint alleges that the private records about which the plaintiff complains were created and maintained as early as 1977 (First Amended Complaint ¶ 8) and that these records adversely affected his work efficiency and evaluation reports from 1977 to 1982 (First Amended Complaint ¶ 15). The critical issue for determining whether the plaintiff's action is barred by § 552a(g)(5) is the time at which the plaintiff first knew or had reason to know that the private records were being maintained.

The amended complaint alleges that the plaintiff did not gain physical access to his file until the Spring of 1982 and that the contents had been withheld from him on numerous previous occasions when he had requested to review his records. (First Amended Complaint ¶ 7). However, as the Bergman court pointed out, the relevant fact is not when the plaintiff first had physical possession of the particular records, but rather when he first knew of the existence of the records. In Ertell v. Department of the Army, 626 F. Supp. 903 (C.D.Ill. 1986), the court rejected the plaintiff's contention that the statute of limitations did not begin to run until he had actually discovered the negative records in his file. Instead it held that the statute began to run when the plaintiff first had reason to believe that the records existed. Id. at 908.

In the course of discovery, the plaintiff provided the defendants with a copy of Form DD 149 entitled "Application for Correction of Military or Naval Record Under the Provisions of Title 10, U.S. Code Sec. 1552," which he had submitted to the Army on July 15, 1983. This form contains a sworn statement by the plaintiff that "civil service management evaluation reports, memoranda and opinion were kept in a file of my civil service occupation unknown to me until August 1981." (Def.App. A-12 to A-13). This statement, made with full knowledge of the penalties involved for willfully making a false statement or claim on the form, indicates that the plaintiff first had reason to believe that the records existed in August of 1981. His complaint was not filed until five months after the two-year statute of limitations had run in August of 1983.

To counter his 1983 sworn statement that he first knew of the records in August 1981, the plaintiff submitted an affidavit dated May 1985 with his memorandum opposing the defendants' motion to dismiss in which he claimed that "prior to the spring of 1982 I had only unsubstantiated hearsay and rumors that a reported file and memoranda were being withheld and maintained by the Defendants." It is well established that a party cannot create a genuine issue of fact by submitting an affidavit containing conclusory allegations which contradict plain admissions in prior deposition or otherwise sworn testimony. See Babrocky v. Jewel Food Co., 773 F.2d 857, 861-862 (7th Cir. 1985); Miller v. A.H. Robins Co., 766 F.2d 1102, 1104 (7th Cir. 1985); Canfield Tires, Inc. v. Michelin Tire Corp., 719 F.2d 1361, 1364-1366 (8th Cir. 1983); Perma Research Development Co. v. Singer Co., 410 F.2d 572, 577-578 (2d Cir. 1969). Like the statements we reviewed in Babrocky, the plaintiff's contradictory claim that he did not learn of the records until the Spring of 1982 is "little more than [a] bald assertion, entirely lacking in any recounting of specific facts," which we may properly reject as creating only a "sham" issue. 773 F.2d at 861.

Moreover, even if we were to take the plaintiff's statement in his affidavit as true, the unsubstantiated hearsay and rumors, which alerted him in the fall of 1981 that the defendants might be maintaining a secret file on him, were sufficient to trigger the statute of limitations. If we were to adopt the plaintiff's position, the two year period of § 552a(g)(5) would not begin to run until an individual either received confirmation from the government that it was indeed maintaining a file of secret records on him or actually gained access to the improper records. Such a holding would essentially eliminate the purpose behind the statute of limitations. As the court in Ertell explained, an individual's physical discovery of improper records does not give rise to a cause of action; it merely strengthens the original cause of action which arose when the individual first had reason to believe that the records were being maintained. 626 F. Supp. at 908-909.

The hearsay and rumors which the plaintiff described in his affidavit were enough to put him on notice of the possible existence of the records at issue here and to impose upon him a duty to inquire into the veracity of those rumors for purposes of bringing an action under the Privacy Act. He argues that if we enforce the statute of limitations against him, then future Privacy Act plaintiffs will be forced to file suit upon the mere suspicion of the existence of an improper record, thus opening the floodgates to meritless litigation. The plaintiff misconceives the operation of § 552a(g)(5). That Section does not require him to rush into court and file suit upon the first hint that the government may be maintaining an improper record on him or risk having his action dismissed for failure to comply with the time limit when he subsequently files at a later date. Rather § 552a(g)(5) allows a plaintiff a period of two years after he first has reason to believe that the government is maintaining such a record to investigate whether there are reasonable factual and legal bases for bringing an action under the Privacy Act and then ultimately to file the suit. The plaintiff had the full two years to determine whether the rumors and hearsay could be substantiated. Section 552a(g)(5) does not afford him any further time.

Finally, the plaintiff argues that the continuing violation doctrine should toll the statute of limitations. According to the plaintiff, the government's use of the private records to force the plaintiff into early retirement from the civil service on December 31, 1983, was a new and continuing unlawful act, occurring well within two years of the filing of this suit on February 1, 1984. The Tenth Circuit in Bergman rejected the argument that a new cause of action arises upon "each and every subsequent adverse determination based on erroneous records." 751 F.2d at 317. The plaintiff has not alleged any new unlawful conduct by the government; he has merely alleged a continuing adverse consequence of prior unlawful conduct. See, e.g., Ward v. Caulk, 650 F.2d 1144, 1147 (9th Cir. 1981) ("A continuing violation is occasioned by continual unlawful acts, not by continual ill effects from an original violation."); Oppenheim v. Campbell, 571 F.2d 660, 662 (D.C. Cir. 1978) (plaintiff's claim accrues when he is "first harmed"). We fully agree with the Tenth Circuit that to subscribe to an argument such as the plaintiff advances "would, in practical effect, mean that the two-year statute would never run." Bergman, 751 F.2d at 317.

The plaintiff's Privacy Act suit is barred by § 552a(g)(5) and we therefore affirm the district court's dismissal for lack of subject matter jurisdiction, although on another ground.