A campaign by the chef led to tough new legal standards for meals in England's schools.
But now caterers are saying that some of England's new academy schools - which do not have to abide by the regulations - are asking for "unhealthy food".
The government says it trusts schools to act in their pupils' best interest.
And it says it has no reason to believe that academies will not provide healthy, balanced meals that meet the current nutritional standards
Jamie Oliver told BBC Breakfast News: "The bit of work that we did which is law was a good bit of work for any government.
"So to erode it, which is essentially what Mr Gove is doing - his view is we let schools do what they want."
The chef led a TV campaign which saw him going in to school kitchens and helping staff to drop fatty foods from their menus in favour of healthier options.
The law was tightened in England for local authority primary schools in 2008 and secondaries in 2009, so that school lunches had to meet strict nutritional guidelines.
Sweets, crisps and fizzy drinks disappeared from vending machines.
The changes were not universally liked, and some parents took to passing their children fast food through the railings of their school.
And in some schools, children are bringing in biscuits and other snacks to sell to pupils.
England's academies are semi-independent schools and as such they do not have to abide by regulations which set out strict nutritional guidelines for school food.
There are currently 1,400 of them and more schools plan on converting to academy-status.
Now the Local Authority Catering Association, which has 700 members across the UK, says it is being asked to bring back some "unhealthy food".'Return of the sausage roll'
Linda Mitchell, from the association, said: "Our members are telling us that they have been approached by academies to relax the rules and as providers to hundreds of thousands of schools we are concerned.
"They are being asked to put confectionery and other snacks back, especially at mid-morning. It is the return of the sausage roll to schools."
She said caterers were mostly being asked to put snacks in to vending machines in schools.
"It's a very small step before you are seeing the introduction of confectionery and fizzy drinks back in to those machines."There are guidelines on school food in all parts of the UK, but legal regulations in England only
She suggested schools might be under pressure from pupils and parents to re-introduce certain foods and that some might be tempted by the high profit margins she said there were on sweet or fatty snacks.
A spokesman for the Department for Education in Westminster said the school food regulations were the "benchmark of high standards".
"We trust schools to act in the best interest of their pupils - they know the importance of healthy school dinners and the benefits they bring," he said.
Mr Gove met Jamie Oliver in June. In a letter to the chef afterwards, he wrote: "I would like to reassure you that we have no reason to believe that academies will not provide healthy, balanced meals that meet the current nutritional standards.
"As part of the broader freedoms available to academies, I trust the professionals to act in the best interests of their pupils.
"I understand from the School Food Trust that some of the best schools in terms of attitudes to food and meals are the academies."
Jamie Oliver said the standards should be kept to "keep everyone on their game".
"I think that's a wonderful ambition, that everyone's going to be brilliant, but head teachers are more pushed than ever, expected to do more than ever, now they have to be entrepreneurial caterers as well as everything else they have to do," he said.
"It's not a large amount of paperwork, it really isn't, and for what it's for, which is essentially the future of our country, it's really important."
Sharon Hodgson MP, Labour's Shadow Minister for Children and Families, said: "Michael Gove is effectively throwing away years of hard work and achievement in driving up standards for purely ideological reasons, and without any mandate from parents".
Under regulations in England and Northern Ireland - and guidelines in Scotland and Wales - schools should:restrict the amount of fatty, sugary and salty foods on the menuincrease the use fresh fruit and vegetablesban the sale of sweets, crisps and fizzy drinks
In England, local authority schools also have to supply meals which comply with strict nutrient standards.
From 2013, Wales intends to enshrine its guidelines in the law too.
England's Department for Education has asked the School Food Trust to carry out a study of some academies to look at their approach to providing healthy school food to pupils.
School Food Trust chairman Rob Rees said he was concerned about the reports of standards being relaxed.
"All the evidence shows that when children eat better, they do better, so this should be a complete and utter no-brainer for any school wanting to do the best for its pupils," he said.
"We need to hear from anyone with concerns that an academy is choosing not to meet the national standards - whether that's adding less healthy foods back on to its menu or bringing back vending of chocolate and crisps."