Cattle handling is a dangerous activity on dairy farms, and cows are a major cause of injuries to livestock handlers. Even if dairy cows are generally tranquil and docile, when situations occur that they perceive or remember as aversive, they may become agitated and hazardous to handle. This study aimed to compare human-animal interactions, cow behavior, and handler safety when moving cows to daily milking and moving cows to more rarely occurring and possibly aversive hoof trimming. These processes were observed on 12 Swedish commercial dairy farms. The study included behavioral observations of handler and cows and cow heart rate recordings, as well as recording frequencies of situations and incidents related to an increased injury risk to the handler. At milking, cows were quite easily moved using few interactions. As expected, the cows showed no behavioral signs of stress, fear, or resistance and their heart rate only rose slightly from the baseline (i.e., the average heart rate during an undisturbed period before handling). Moving cows to hoof trimming involved more forceful and gentle interactions compared with moving cows to milking. Furthermore, the cows showed much higher frequencies of behaviors indicative of aversion and fear (e.g., freezing, balking, and resistance), as well as a higher increase in heart rate. The risk of injury to which handlers were exposed also increased when moving cows to hoof trimming rather than to routine milking. Some interactions (such as forceful tactile interactions with an object and pulling a neck strap or halter) appeared to be related to potentially dangerous incidents where the handler was being kicked, head-butted, or run over by a cow. In conclusion, moving cows to hoof trimming resulted in higher frequencies of behaviors indicating fear, more forceful interactions, and increased injury risks to the handler than moving cows to milking. Improving potentially stressful handling procedures (e.g., by better animal handling practices and preparation of cows to cope with such procedures) can increase handler safety, animal welfare, ease of handling, and efficiency.