Concerns regarding the use of black hole shadows as standard rulers