The gladiators were indeed too polluted to fully re-join society officially (as were the undertakers, for example). Still, there were some cases (not that few) of gladiators earning their freedom and going on to a full life of wealth and status. You got to remember that a good portion of the gladiators who earned their freedom where famous and went on to train other gladiators and even own them. The elite liked them occasionally at their parties (exotic and famous fellows, there is always a demand for those). The famous gladiators lived a relatively good and comfortable life, even during their period as gladiators. If you have a horse that earns you a huge amount of wealth, you treat him good and give him the best conditions possible, right? The best and most famous gladiators were the exact same.
Correction, Roman slavery was pretty rough when it came to the mines, galleys and gladiators, with the working of the fields being also not that nice (but much less rougher). In all the other situations, it was the best kind of slavery possible. Domestic and intellectual (teachers, philosophers, accountants, secretaries, etc) slaves had a much more comfortable life.
The domestic slaves' life wasn't exactly great, but there were a bunch of free men that lived worse (a big part of the urban masses at the end of the republic, some poor farmers, etc.), and they didn't had what would be called at the time "rough" or very physically demanding jobs. And besides, the closer contact with their masters usually gave them increased chances of being freed, and the Romans loved to free their slaves. It was not only the result of a much more personal contact that they had with a big chunk of their slaves (when compared to modern slavery) but also a matter of prestige. Something like "I am so good and so rich, so not caring with wealth [the roman mentality had a pretty negative opinion on wealth, as ironic as it may be], that I have freed xxx slaves during my life".
What we might call intellectual slaves had a even better situation. They were treated well, they were precious property, some of them even had their own slaves and a very comfortable life, with some of them becoming incredibly wealthy after their release (which was even more common when compared to the other domestic slaves). We know of cases when free people sold themselves to slavery to have a better life and a better prospect of future. They weren't exactly going to gladiators or something like that (those guys were basically war prisoners, deserters and criminals), they were being sold as philosophers of the roman elite, as tutors of the roman children, as close helpers of the roman individual members of the political, economical and cultural elite.
In the roman streets, it wasn't a rare thing to see slaves better dressed and better fed than free men. It goes without saying that roman slavery was still obvious slavery, even if didn't had the permanence, segregation/racism and general lack of conditions that modern slavery had. In the Roman World, if some of the people in this forum ever went miserable, they could sell themselves into slavery, and use their knowledge and educational formation to live a relatively comfortable life with not so bad prospects regarding their future.