Hundreds were in Charlottesville either to demonstrate in or rail against a "Unite the Right Rally," one day after hundreds of torch-bearing marchers demonstrated at the normally tranquil city's university campus.
US President Donald Trump urged unity as the unrest flared, calling on Americans to "condemn all that hate stands for."
"We ALL must be united and condemn all that hate stands for," the Republican leader tweeted. "There is no place for this kind of violence in America. Let's come together as one!"
After urging Virginians a day earlier to stay clear of the rallies, the state's governor Terry McAuliffe on Saturday declared a state of emergency, saying on Twitter it was necessary "to aid state response to violence at Alt-Right rally in Charlottesville" as hundreds from both sides clashed.
Police began evacuating the city's Emancipation Park after declaring those gathered there to be part of an "unlawful assembly."
By midday local authorities had reported one arrest and said emergency personnel had responded to eight injuries.
An AFP journalist at the scene witnessed demonstrators, some clad in militia uniforms, throwing punches and hurling bottles even before the official 12 pm EST (1600 GMT) rally start time.
State police also tweeted that some crowd members were using pepper spray, and local media footage showed images of riot police, national guard members and an armored vehicle in the city's downtown.
"It is now clear that public safety cannot be safeguarded without additional powers, and that the mostly out-of-state protesters have come to Virginia to endanger our citizens and property," McAuliffe wrote in a statement on his emergency declaration.
"I am disgusted by the hatred, bigotry and violence these protesters have brought to our state over the past 24 hours."
In light of the unrest, city leaders also declared a state of emergency.
Video footage from the demonstration showed anti-racism protesters waving flags from the Black Lives Matter movement, as crowds chanted slogans like "We say no to racist fear" and "No Nazis, no KKK, no fascist USA."
Others brandished Confederate flags, today considered a symbol of racism by many Americans.
Right-wing blogger Jason Kessler, who had called for the white nationalist demonstration, on Saturday declared it "a monumental event for our movement" even after evacuations began.
Saturday's far-right rally follows a much smaller demonstration last month that saw a few dozen Ku Klux Klan-linked marchers gather to protest the city's planned removal of a statue of General Robert E. Lee, who led Confederate forces in the US Civil War.
Though they were outnumbered by hundreds of jeering counter-protesters, the extreme right marchers -- some donning the traditional white hood of the notorious white power group -- saw their images spread worldwide on social media.
And this time the extreme right brought in big names of the "alt-right" movement -- which has been emboldened, critics say, by Donald Trump's ascent to the White House -- in a bid to attract more supporters.
Normally reticent First Lady Melania Trump took to Twitter to respond to the demonstrations, writing, "Our country encourages freedom of speech, but let's communicate w/o hate in our hearts. No good comes from violence. #Charlottesville."
Republican House Speaker Paul Ryan also weighed in on the social media platform: "The views fueling the spectacle in Charlottesville are repugnant. Let it only serve to unite Americans against this kind of vile bigotry."
Friday night's pre-demonstration also turned into a brawl after torch-bearers linked to the white supremacist far right were met by a group of counter-protesters.
Charlottesville's mayor Mike Signer dubbed Friday's march a "cowardly parade of hatred, bigotry, racism and intolerance."
University of Virginia President Teresa Sullivan on Saturday condemned that demonstration, saying in a statement that "the intimidating and abhorrent behavior displayed by the alt-right protesters was wrong."
One protester was arrested and charged with assault and disorderly conduct, she said, adding that "I am deeply saddened and disturbed by the hateful behavior displayed by torch-bearing protesters that marched on our grounds this evening."
The Southern Poverty Law Center, which monitors extremist groups, said that Saturday's "Unite the Right Rally" could mark one of the most significant demonstrations of its kind in decades.
In a new development of the situation in Virginia, Car rammed into crowd at violent rally, witnesses said.