To make posture recognition more accurate, we follow an approach inspired by [Springstein et al. 2022] that integrates Semi-supervised Learning (SSL). This means that our system learns from both labeled and unlabeled images, allowing it to improve over time by generating its own training data. We implement the regression-based Pose Recognition Transformer (PRTR), as outlined by [Li et al. 2021], which employs a cascaded Transformer architecture with separate components for bounding box and keypoint detection [Vaswani et al. 2017] [Carion et al. 2020].[4] The proposed methodology for HPE thus leverages a top-down strategy: In the first stage, human figures are localized within an image by rectangular bounding boxes, which are then examined in the second stage to identify keypoints, i.e., points relevant to the abstraction of the figures’ posture.

The estimated whole-body posture is segregated into the upper and lower body, shown in green and red, respectively, in Figure 3; these components, along with the whole body, then comprise the search query employed for retrieval purposes. The lower body is composed of six keypoints (ankles, knees, hip) and the upper body of eight keypoints (hip, wrists, elbows, shoulders). This division increases the reliability of the system: Even if the whole-body posture cannot be detected (e.g., due to missing or obscured joints), the upper and lower body may still provide enough information to accurately classify the figure. In the next step, configurations exceeding 50% of the maximum possible keypoints are filtered to eliminate configurations with high uncertainty. Valid configurations are then projected into 320-dimensional embeddings using the Pr-VIPE proposed by [Liu, T. et al. 2022].[5] Unlike methods that depend solely on keypoint positions, Pr-VIPE encodes the appearance of a posture — as perceived by humans — so that it remains robust to changes in perspective, allowing similar postures to be compared across different viewpoints. It does so by mapping each two-dimensional posture into a probabilistic embedding space where the representation is not a fixed point, but a probability distribution. During training, the system is exposed to pairs of two-dimensional postures obtained from different camera views, often augmented by synthesizing multi-view projections from three-dimensional keypoints. The learning objective encourages the embeddings of postures that are visually similar (i.e., represent the same underlying three-dimensional posture) to be close to each other in the probabilistic space — even if there are perspective-induced variations in the two-dimensional keypoint positions.[6]

To classify postures, the pipeline is as follows: We first manually identify 110 art-historical images with reference postures of human figures and label them with keypoints. We then re-use the Pr-VIPE and compute the cosine distances between the embeddings, associating each query to the closest matching reference posture(s). This allows a fine-grained indexing of postures, even when the body parts of each configuration could not be adequately estimated. At the same time, there is no fixed, semantically dubious categorization into groups, as is the case with agglomerative clustering methods [Impett and Süsstrunk 2016]. However, as shown in Figure 3, where example images of the reference postures are shown, a single human figure can only model a fraction of the high postural variability within the subnotations.

The selected postures, derived from the Iconclass taxonomy [van de Waal 1973], range from elementary configurations, such as “arm raised upward,” to complex full-body arrangements, such as “lying on one side, stretched out,” reducing the limitations of culturally or temporally dependent labels. Iconclass, while explicitly designed for the iconography of Western fine art, also includes universal definitions ranging from natural phenomena to anatomical specifics, the latter of which are pertinent not only to our research but also potentially beneficial for scholarly investigations into human anatomy across various disciplines, such as dance studies. Each definition within the Iconclass taxonomy is represented by a unique combination of alphanumeric characters, referred to as the ‘notation,’ and a description, the ‘textual correlate,’ accompanied by a set of keywords. A notation consists of at least one digit symbolizing the first level of the hierarchy, ‘division.’ This may be followed by another digit at the secondary level, and one or two (identical) capital letters at the tertiary level. The structure, referred to as the ‘basic notation,’ can be further supplemented with auxiliary components [van Straten 1994]. We consider four groups of Iconclass notations: notation 31A23 (textual correlate “postures of the human figure”), 31A25 (“postures and gestures of the arms and hands”), 31A26 (“postures of the legs”), and 31A27 (“movements of the human body”). Notations 31A23 and 31A27 represent the whole body, 31A25 the upper body, and 31A26 the lower body. Each group is almost equally represented: upper-body notations with 31 instances, lower-body notations with 25, and whole-body notations with 27 each.

The first pipeline entails filtering objects based on their metadata, in particular the Wikidata “depicts” property (P180) and labels in English, French, and German. Objects containing any of the query terms are selected and then analyzed for their spatial position within the embedding space to identify density structures — the metadata-induced density peaks are thus utilized for the distant viewing of objects whose postures closely resemble those pre-selected by the metadata. Each point in the embedding space denotes a unique posture, with density referring to how these points are grouped — either clustering in high-density areas or scattering in low-density regions. The second pipeline is predicated on the density peaks identified at the aggregate level; it focuses on the recognition and exploitation of individual postures associated with specific iconographies, thus enabling a posture-to-posture search. This involves a close viewing of objects that are visually related to the identified query postures; it also explores the developmental trajectories of different postures within the embedding space to trace the ‘evolution’ of iconographic elements.

Our case study examines the variability in the representation of the crucified. It focuses specifically on the depiction of the crucified Christ, excluding accompanying figures traditionally portrayed at the cross’s base, such as Mary and Christ’s disciples — especially the apostle John. Although depictions of the crucified Christ vary widely throughout the world, influenced by local and ethnic traditions, they are generally unified by a “canonical” form: “dead upon the Cross, Jesus’s head is slung to one side, typically our left; his torso is naked and upright, sometimes slightly arched” [Merback 2001, p. 69].Variations in the positioning of Christ’s limbs are subtly evident in different artistic traditions: Gothic crucifixes, for instance, show Christ in a dynamic, bent position, with legs thrust forward and knees spread [Brandmair 2015, p. 100], while in Italian iconography, Christ is often depicted with upward-angled arms, his head slumped on his chest, with varying leg positions to represent the deceased body’s movement [Haussherr 1971, p. 50].

To conduct a qualitative analysis, the embedding spaces of the embeddings are reduced from 320 to two dimensions. Traditional dimension reduction techniques such as t-distributed Stochastic Neighbor Embedding (t-SNE) [van der Maaten and Hinton 2008], Uniform Manifold Approximation and Projection (UMAP) [McInnes 2018], and their variants (e.g., [Im, Verma, and Branson 2018] [Linderman et al. 2019]) are known to preserve either local or global spatial relationships, which can result in the formation of artificial clusters that are not present in the original data.[11] To overcome this limitation, we employ Pairwise Controlled Manifold Approximation (PaCMAP), which first identifies global structures and then refines them locally [Wang et al. 2021]. We re-use the Pr-VIPE model checkpoints from [Liu, T. et al. 2022].[12] To analyze the density structure of the embedding spaces, and thus the virtual space of possibilities, we employ three visualization techniques: scatter plots, two-dimensional histograms, and contour plots (Figure 5). Both histograms and contour plots employ a color gradient ranging from blue (denoting lower concentrations) to red (denoting higher concentrations), with white signifying the midpoint. Contour plots are often favorable because of their superior ability to discern isolated high-density areas clearly; however, smaller, less dense regions are more reliably identified in two-dimensional histograms. Further details of the techniques are given in Appendix B.

The embedding space is structured so that similar postures are grouped closely together. This proximity implies that finding a posture in a particular region of the embedding space generally means that another similar posture can be found nearby — hence, semantic groups of postures typically cluster, providing a structured approach to efficient retrieval based on these spatial concentrations, as will be discussed in the following. In each case, our analysis begins with an examination of the embedding spaces for all images to establish a baseline understanding. We then narrow our focus to compare these embedding spaces with those containing only crucifixion scenes, in order to determine how particular thematic elements influence the spatial organization within the embedding.

In the second pipeline, our approach departs from the broad analysis of metadata-induced hotspots and instead focuses on the in-depth study of individual artworks and their comparative analysis based on the generated embeddings. To this end, we implement an approximate k-nearest neighbor graph, Hierarchical Navigable Small World (HNSW) [Malkov_2020], which contains the 320-dimensional embeddings of the whole body, upper body, and lower body. To illustrate its practical application, we examine James Tissot’s The Strike of the Lance (1886–1894) and more specifically, the thief crucified to Christ’s right (Figure 6a). As shown in Figure 6, the thief’s bent arm is echoed in the Pietà, for instance in a rendition after Marcello Venusti (c. 1515–1579; Figure 6g), where Mary is portrayed mourning her son. Interestingly, the horse’s upright stance in Jacques Louis David’s Napoleon on the Great St Bernard (1801; Figure 6p) is mistakenly recognized as a human figure — with outstretched arms and bent legs — echoing the thief’s posture; this error results from imprecise bounding box detection, which frequently misclassifies animals with human-like physiognomies as human figures. Misalignments of the lower body are evident also in works such as Perino del Vaga’s A Fragment: The Good Thief (Saint Dismas) (c. 1520–1525; Figure 6h), Hendrick ter Brugghen’s The Crucifixion with the Virgin and St John (1625; Figure 6k), and Jan de Hoey’s Ignatius of Loyola (c. 1601–1700; Figure 6r), primarily with inaccurately shortened ankles, which also trace back to limitations in bounding box detection. Despite these issues, minor inaccuracies, such as the misestimation of the left wrist in Figure 6l, seem to not affect the retrieval performance. The degree of keypoint misestimation is proportional to its effect on the generated feature vector, which is essential for accurate retrieval. Even when keypoints are grossly misestimated, as shown in Figure 6l, the model only marginally penalizes these errors in the resulting embedding, provided that the majority of keypoints are estimated with sufficient accuracy.

To understand the evolution of iconographies in art history — here: the postural configurations in crucifixion scenes — it is essential to analyze both the posture and its temporal distribution. This involves, in crucifixion scenes, distinguishing between T- and Y-shaped upper-body postures that depend on the arms’ position on the cross. To assess the Pr-VIPE’s ability to differentiate between such configurations, we first select three crucifixion depictions from Wikidata: Marcello Venusti’s Christ on the Cross (1500–1625), in T-shape, with arms outstretched horizontally;[14] Diego Velazquez’s Christ Crucified (1632), in weak Y-shape, with arms slightly upraised;[15] and Peter Paul Rubens’ Christ Expiring on the Cross (1619), in strong Y-shape, with arms strongly raised.[16] That is, each image served as a query to identify the top 100 similar Wikidata object entities based on their upper-body similarity. Analysis of the bootstrapped distribution of creation dates then shows that T-shapes start increasing around 1200, peak sharply around 1400, and then gradually decrease, with another smaller peak around 1800, while Y-shaped figures have a more widespread, uniform distribution between 1400 and 1900 — in case of weak Y-shapes, without notable peaks (Figure 8 in Appendix A). The evolution of the shapes can be also observed clearly in the contour plot of the two-dimensional upper-body posture embedding. Naturally, T- or slightly Y-shaped figures, with their symmetrically outstretched arms, are most often associated with crucifixions, whereas the upward curvature of stronger Y-shapes appears in a wider range of contexts, including depictions of ascension and resurrection that imply spiritual elevation, such as the putti in Rembrandt’s The Ascension (1636). This variation in upper-body posture is also reflected in the artworks’ temporal distribution: Strong Y-shapes occur mostly around 1600, with another notable peak around 1900, and are rarely found before 1400. The Pr-VIPE’s ability to distinguish these postures suggests its effectiveness in identifying both the physical configurations and, on closer historical inspection, the underlying symbolic meanings. This evidence suggests potential verifiable patterns in the representation of different upper-body forms across historical periods, although conclusive historical evidence remains to be established.