ORLANDO, Fla. -- Doubling down on his controversial effort to purge non-United States citizens from voter rolls, Florida Gov. Rick Scott dismissed criticism by civil rights advocates Tuesday, saying the state has a “very good process to make sure that U.S. citizens have the right to vote.”

That right is at the crux of a debate here over Scott’s initiative, which has spurred lawsuits by civil rights groups and a suit by the Department of Justice – which says the purge violates the 1965 Voting Rights Act.

Most Florida counties have backed out of Scott’s directive but two counties have continued to accept it, according to media reports. There, U.S. citizens removed from voter rolls will be given a 60-day period to respond, and after that will be able to vote using a provisional ballot.

“Your vote’s always going to count,” Scott said, saying that he speaks from experience.

Scott revealed during a radio interview last week that he voted by provisional ballot during two elections in 2006 because an election worker in Naples confused him for a man who had died. 

“They just said I got to vote on a provisional ballot,” Scott told reporters Tuesday. “The nice thing about our state – when something like that happens, we have a good process. So my vote still counted.”

But just how much a provisional ballot counts is debated by voters’ rights groups, which point out that provisional ballots aren’t counted until after Election Day. 

(Scott’s story, ostensibly meant to show that the system works, also seemed to suggest just how easily a registered voter can be thrown from the rolls.)

The remarks Tuesday came during a brief discussion with reporters after Scott addressed a lunch meeting of the Board of Governors, the body overseeing Florida’s university system. 

The meeting had drama of its own, as the board votes later this week on tuition hikes.

Scott opposes the increases and is pushing for a review of the university budgets.

While university budgets have put the governor at odds with his state’s university system, the voter purge has created friction with the federal government.

Florida has filed its own suit against the U.S. Department of Homeland Security, saying the department refused to share a database containing immigration information. 

Scott says the state was forced to rely on a motor vehicle database instead, which critics say has outdated and bad information.

Still, the governor's aides believe the project is necessary and has proved successful.

Reached by phone, Lane Wright, the governor's spokesman, said the Florida DMV identified 180,000 people as potential non-citizens. A "small sample" -- 2,600 names -- was selected for verification. 

Of that number, at least 107 people have come forward to say they are not U.S. citizens, Wright said, adding that half had already voted in a prior election.

Asked Tuesday what he’d say to a woman in Central Florida whose eligibility was challenged even though she had a voting record dating back to Eisenhower’s 1956 re-election run, Scott shifted the blame to the federal government.

"What I'd say is she should be disappointed that the Department of Homeland Security didn't do their job," Scott said.