The male mosquitoes are modified so their offspring die before reproducing.
In a dengue-affected part of the Cayman Islands, researchers found the GM males mated successfully with wild females.
In Nature Biotechnology journal, they say such mating has not before been proven in the wild, and could cut the number of disease-carrying mosquitoes.
Dengue is caused by a virus transmitted by the Aedes aegypti mosquito as it bites.
The World Health Organisation (WHO) estimates that there may be 50 million cases each year, and the incidence is rising, with some countries reporting what the WHO terms "explosive" outbreaks.
As yet, there is no vaccine.Radiation damage
As far back as the 1940s, it was realised that releasing sterile males into the wild could control insects that carried disease or were agricultural pests.The Aedes aegypti mosquitoes that carry dengue cannot be curbed by bednets or indoor spraying
When females breed with the sterile males rather than wild fertile ones, there will be no viable offspring, meaning there are fewer mosquitoes around to transmit the disease.
In the 1950s, the screwworm fly was eradicated from the Caribbean island of Curacao using males sterilised by radiation.
But the technology has not worked so well with disease-carrying insects.
Generally, the sterilising process weakens the males so much that they struggle to mate; the wild males are dominant.
Oxitec, a company spun off from Oxford University, uses a genetic engineering approach.Continue reading the main story
This study is the first to show that the mosquito population could be suppressed this way”End Quote Dr Raman Velayudhan WHO Offspring of their GM males live through the larval stage but die as pupae, before reaching adulthood.
In the latest study, the research group - which includes scientists from Imperial College London and the Liverpool School of Tropical Medicine - released batches of GM mosquitoes in 2009 in an area of the Cayman Islands where Aedes aegypti are common, and dengue sometimes present.
A proportion of the eggs collected from the study area in subsequent weeks carried the introduced gene, meaning the biotech mosquitoes had mated successfully.
The GM males made up 16% of males in the study area, and fathered 10% of the larvae; so they were not quite as successful as the wild males, but not significantly worse.
"We were really surprised how well they did," said Luke Alphey, Oxitec's chief scientific officer and a visiting professor at Oxford University.
"For this method, you just need to get a reasonable proportion of the females to mate with GM males - you'll never get the males as competitive as the wild ones, but they don't have to be, they just have to be reasonably good."The GM larvae also carry a fluorescent gene that distinguishes them from wild relatives
"This study is the first to show that the mosquito population could be suppressed this way," said Dr Raman Velayudhan, a WHO dengue expert.
"The fitness level is much better [compared with previous attempts] - it is almost the same as in wild mosquitoes," he told BBC News.
Cognizant that genetic engineering is a technology that carries the potential for risks as well as benefits, the WHO is finalising guidance on how GM insects should be deployed in developing countries, which it expects to release by the end of the year.The "death gene" is turned off during rearing in Oxford - and turned on in the field
The field seems to be hotting up, with other research groups recently creating Anopheles mosquitoes that are immune to the malaria parasite they normally carry, and making male Anopheles that lack sperm.
Malaria is a prime target for these approaches simply because it is such an important disease; but arguably it is more needed in diseases such as dengue where there are few alternatives.
"For malaria, there are effective alternatives like bednets, but they won't work for dengue because the mosquitoes bite during daytime," said Dr Alphey.
"We don't advocate [GM mosquitoes] as a 'magic bullet' that will solve all dengue in one go, so the question is how it fits in as part of an integrated programme - and for dengue, it would be a huge component of an integrated programme."
Funding for the Oxitec approach has come from a number of sources including private investors, charities, Oxford University and governments, and the Cayman Islands authorities were willing to take part in the field trial.Death by feedback
The genetic approach used to create the mosquitoes is a system known as tetracycline-controlled transcriptional activation (tTA).
The tTA gene is spliced into the insect's genome in such a way that the protein it makes increases the gene's activity - a positive feedback loop.
The cells make more and more tTA protein - and in doing so, have little capacity for making any other proteins. Eventually, this kills the insects.
When the male larvae are reared at Oxitec, this process is turned off by keeping them in water containing the antibiotic tetracycline, which inhibits the feedback process.
When the males breed in the wild, however, tTA genes in their offspring are fully active.
In principle, a process that allows larvae to hatch and stay alive for many days should be more advantageous than the traditional approach of producing infertile eggs, as the larvae will consume food that could otherwise be used by viable larvae from the union of wild males and females.
The next step in the work is to demonstrate that deploying GM males does suppress the insect population enough that it is likely to have an impact on dengue incidence.