But she said she would not call people on to the streets to protest, as she had done after the 2004 presidential election.
"I will not call another Maidan [Independence Square demonstration] and will not allow public protests," she said.
Mr Yanukovych was pronounced the victor six years ago, only for the result to be overruled because of vote-rigging. The Orange Revolution eventually brought Ms Tymoshenko's erstwhile ally, President Viktor Yushchenko, to power.
The prime minister said more than a million votes, which she said were decisive to the outcome, were invalid.
"With all this proof, I have taken the only possible decision: to challenge the results of the election in court. I will defend our state and the choice we made on the basis of legal documents," she said.
"Not going to court would mean giving up Ukraine to criminals without a fight."
On Wednesday, Mr Yanukovych demanded his rival resign as prime minister and go into opposition.
"I want to remind Ms Tymoshenko that the basis of democracy is the will of the people. Democratic leaders always accept the results of the elections. The country does not need a new crisis," he said.
The BBC's Daria Merkusheva in Kiev says Ukraine's Central Election Commission is expected to confirm Mr Yanukovych as president-elect early next week, after which the legal challenges are likely to start.
Ukraine has been in political deadlock for several years, undermining its ability to deal with a severe economic crisis. Analysts say a challenge by Mrs Tymoshenko would prolong the instability.
With Mrs Tymoshenko still heading the government, stalemate is likely to continue. Mr Yanukovych has said he wants to form a new coalition, and may try to call snap parliamentary elections.
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.