Endocrine-disrupting chemicals (EDCs) can interact with nuclear receptors, including estrogen receptor α (ERα) and androgen receptor (AR), to affect the normal endocrine system function, causing severe symptoms. Limited studies queried the EDC mechanisms, focusing on limited chemicals or a set of structurally similar compounds. It remained uncertain how hundreds of diverse EDCs could bind to ERα and AR and cause distinct functional consequences. Here, we employed a series of computational methodologies to investigate the structural features of EDCs that bind to and activate ERα and AR based on more than 4000 compounds. We used molecular docking and molecular dynamics simulations to elucidate the functional consequences and validated structure-function correlations experimentally using a time-resolved fluorescence resonance energy-transfer assay. We found that EDCs share three levels of key fragments. Primary (20 for ERα and 18 for AR) and secondary fragments (38 for ERα and 29 for AR) are responsible for the binding to receptors, and tertiary fragments determine the activity type (agonist, antagonist, or mixed). In summary, our study provides a general mechanism for the EDC function. Discovering the three levels of key fragments may drive fast screening and evaluation of potential EDCs from large sets of commercially used synthetic compounds.