The benchmark Nikkei 225 index fell 230.45 points to 15,177.49, while the Topix index of all first-section shares lost 0.91 percent, or 11.34 points, to 1,229.65.
Investors have been locking in profits after the Nikkei hit a nearly six-year high this week.
"Short-term investors have been shedding their positions in the last couple of days...ahead of the US jobs data release (Friday)," Mitsushige Akino, fund manager at Ichiyoshi Asset Management, told Dow Jones Newswires.
A strong US jobs report on Friday will boost speculation the Fed will start tapering its $85 billion-a-month stimulus programme before year's end.
The stimulus has been credited with propping up global equity markets.
Japan Tobacco shares gained 0.29 percent to 3,440 yen after the company and rival Philip Morris said they would each buy a 20 percent stake in Russian distributor Megapolis for a total of more than $1.5 billion.
Sony fell 1.13 percent to 1,824 yen, Toyota was down 0.95 percent to 6,220 yen while Uniqlo clothing chain operator Fast Retailing dropped 3.14 percent to 37,000 yen.
NTT rose 0.97 percent to 5,160 yen as the leading Nikkei business daily reported the Japanese government would sell a portion of its shares in the telecom giant to help fund a fiscal stimulus package.
On currency markets, the dollar slipped to 102.04 yen from 102.32 yen on Wednesday, weighing on shares of exporters which benefit from a weaker Japanese unit. (AFP)