State department officials and Mr Bae’s sister were quoted as saying the 45-year-old had been returned from a hospital to the camp on 20 January.
Mr Bae, a Korean-American, was arrested in November 2012 and sentenced to 15 years’ hard labour in May.
North Korea says he used his tourism business to form groups to overthrow the government.
He was taken to hospital last year after suffering dramatic weight loss. His family say he has several health complaints including diabetes and liver problems.
State Department spokeswoman Jen Psaki said Washington had learned about Mr Bae’s transfer to the camp from representatives of the Swedish embassy in Pyongyang, which acts on behalf of the US in North Korea.
Continue reading the main story
He’s back to eight-hours-a-day, six-days-a-week hard labour”
Terri ChungKenneth Bae’s sister
Ms Psaki said the Swedish diplomats “have met Mr Bae 10 times since his detention, most recently on 7 February in a labour camp”.
She added: “We continue to urge DPRK (North Korean) authorities to grant Mr Bae special amnesty and immediate release on humanitarian grounds”.
Ms Psaki did not specify when Mr Bae had been forced back to the camp.
However, a US state department official and Mr Bae’s sister confirmed the 20 January date.
“He’s back to eight-hours-a-day, six-days-a-week hard labour,” Terri Chung, Mr Bae’s sister, was quoted as saying by the Associated Press news agency.
Pyongyang has so far made no official comment on the reports.
On Thursday, US President Barack Obama used the annual National Prayer Breakfast in Washington to say: “We pray for Kenneth Bae, a Christian missionary who has been left in North Korea for 15 months.”
“His family wants him home, and the United States will continue to do everything in our power to secure his release because Kenneth Bae deserves to be free.”
If confirmed, Mr Bay was returned to the camp on the same day as spoke to foreign media in North Korea under heavy prison guard – his first “press conference” since the detention.
Mr Bae denied media reports that he had been badly treated and called for US “co-operation” to secure his release.
Correspondents say he may have been speaking under strict editorial control.
North Korea has arrested several US citizens in recent years, including journalists and Christians accused of proselytising.
They were released after visits to Pyongyang by high-profile officials, including former presidents Bill Clinton and Jimmy Carter.