Duke Nukem Forever wasn't entirely Gearbox's fault. As a matter of fact, they only bought the assets, and tried to salvage what has been developed thus far in the 10 years it's been in development. It has been through numerous developers before Gearbox, so it's not really their fault the game was released in the shape it was in. It at least kept the humor that Duke was known for - and it even kept the crude material intact. Peeing in the bathroom? I mean, what the fuck!? Which company does that? Nobody. They had the balls to do that.
I feel that with Aliens Colonial Marines, Gearbox was definitely out of their depth. And I don't blame them. The problem is that, it has been about 10-20 years since Aliens was introduced. We don't really know the story of the series. After Aliens 3, the series went into a different direction with the release of Aliens vs Predator - The only thing remotely tried to stay in the Aliens lore was Prometheus, which is both a prequel, and a spin-off. I didn't care to watch Prometheus, and likewise I really didn't give a fuck where Aliens came from, I mean, yeah... I do, but from a sub-human species? A mutant of sorts? It's like they're trying to make things up as the series goes along.
Also, many times stories do not translate into a game, and I think that's what happened with Colonial Marines. Forget the fact that the game was marketed as a "awesome looking game." And turned out to be false advertising in itself, the game just didn't have that Gearbox Seal of Quality stamped on it. It was more like "Okay, make it gold." for the sake of meeting deadlines. Delay the fucking game if it's not perfect!
Companies these days don't understand that in this industry; trust, and reputation is all and important. You fail at one game, you can kiss your future projects goodbye. Borderlands was successful because it was feeding a market that just wasn't there yet. Many FPS'es were just copycats of Call of Duty. Sorry, but true. Borderlands broke the pattern and brought RPG gameplay to the mainstream.
Now, does Gearbox deserve another chance? Yes. Much of the fanbase are usually casuals. So, yeah.
If Gearbox learned from the mistakes of Duke Nukem Forever, and learn what made Duke Nukem so popular in the first place, then it could be a successful franchise for Gearbox. But they have to carefully execute this. It is hard these days to bring a legendary franchise from the dead and into the present, like the Doom reboot did... It is hard.