The present study seeks to investigate the influence of expert affiliation in the weighing procedures within animal welfare assessments. Experts are often gathered with different backgrounds with differing approaches to animal welfare posing a potential pitfall if affiliation groups are not balanced in numbers of experts. At two time points (2012 and 2016), dairy cattle and swine experts from four different stakeholder groups, namely researchers (RES), production advisors (CONS), practicing veterinarians (VET) and animal welfare control officers (AWC) were asked to weigh eight different welfare criteria: Hunger, Thirst, Resting comfort, Ease of movement, Injuries, Disease, Human-animal bond and Emotional state. A total of 54 dairy cattle experts (RES = 15%, CONS = 22%, VET = 35%, AWC = 28%) and 34 swine experts (RES = 24%, CONS = 35%, AWC = 41%) participated. Between—and within—group differences in the prioritization of criteria were assessed. AWC cattle experts differed consistently from the other cattle expert groups but only significantly for the criteria Hunger (p = 0.04), and tendencies towards significance within the criteria Thirst (p = 0.06). No significant differences were found between expert groups among swine experts. Inter-expert differences were more pronounced for both species. The results highlight the challenges of using expert weightings in aggregated welfare assessment models, as the choice of expert affiliation may play a confounding role in the final aggregation due to different prioritization of criteria.