There are a variety of materials that have different toxic effects. Some of these are perfectly natural (like arsenic) but nonetheless detrimental. In addition, what may not be toxic for humans may still be toxic for animals (like chocolate for dogs) and plants. Also, any material in sufficient quantity (even water) can be dangerous to any system, especially at the levels created by factories and industry.
Toxins can include heavy metals, formaldehyde, chlorinated cleaners and solvents, particulates, hormones and hormone-mimicking agents, acidic and basic runoff that changes the pH of water and soil, and any number of other categories. Toxins can enter the body through the lungs if airborne, or through the GI track if present in water or food. In addition, many toxins can enter through contact with the skin. Once in the body, the bloodstream can transport toxins throughout the body.
To whatever extent toxic materials can be eliminated or substituted for more sustainable, nontoxic ones, designers will be able to make a difference for the solutions they create. Some materials may not be toxic themselves, but the processes for locating, mining, or processing them may be highly toxic so the sourcing of materials is another issue to consider. The same can be said for the energy used in the processing of materials and components. The more renewable and sustainably-produced the energy is, the lower the contribution a product or service makes to toxicity in the environment and for workers.