If you have a disability and are qualified to do a job, the ADA protects you from job discrimination on the basis of your disability. Under the ADA, you have a disability if you have a physical or mental impairment that substantially limits a major life activity. The ADA also protects you if you have a history of such a disability, or if an employer believes that you have such a disability, even if you don't.
While legal considerations might be your biggest worry, making your site more accessible is simply good customer service. More than 39 million Americans are blind and another 246 million have "low vision," Another one million are deaf in the U.S. Add to that people with mobility issues that prevent them from using their hands and that's a huge portion of the country's buying power.
Most recently, however, pizza chain Domino's has been brought under suit for their website not being accessible for specialty ordering. The U.S. Supreme Court refused to review the case, instead upholding the decision of the 9th U.S. Circuit Court of Appeals who said the “alleged inaccessibility of Domino’s website and app impedes access to the goods and services of its physical pizza franchises—which are places of public accommodation.”