There are several published experiments that support the choice of LSA for this project. [Foltz, Kintsch, and Landauer 1998] conducted an experiment to demonstrate an approach for predicting the coherence of sets of texts by manipulating textual coherence and assessing readers’ comprehension. One takeaway from this work was the creation of an automated methodology that can generate coherence predictions. [Foltz, Kintsch, and Landauer 1998] utilized LSA as a technique for measuring the semantic relationship between two given pieces of text. The first experiment they performed involved four versions of a textbook. A 300-dimension semantic space was created using the first 2,000 characters of each of the 30,473 articles from Grolier’s Academic American Encyclopedia (details of this can be found in [Landauer and Dumais 1997]). The metric they used was an analysis of adjacent sentence-by-sentence comparisons using LSA, and a calculation of the average cosine. The original coherence score (i.e. average cosines) was 0.192 and was improved to 0.403 in the final version. Another experiment they performed was to detect discourse segmentation, attempting to identify points in the text where a topic shift occurs (thus enabling the segmentation of the text into distinct topics). The idea behind this is based on the hypothesis that coherence scores should be lower between the portions of the text where the discourse topics change. To test this idea, they used the methodology to predict the breaks between nineteen chapters of an introductory psychology textbook. They created an LSA semantic space for the textbook by partitioning it into paragraphs and created a dataset from which a semantic space of 300 dimensions utilising 4,903 paragraphs by 19,160 unique terms (some paragraphs were removed in pre-processing as they represented references, problem sets, and other components that were not relevant to the discourse). To smooth the predictions, a sliding window of 10 paragraphs was utilized. Two adjacent blocks of ten paragraphs were compared using the cosine operator, after which the window would be advanced by one paragraph. This was repeated over all the paragraphs in the text. The average cosine over the entire text was 0.43 while the average cosine between paragraphs across chapter breaks was only 0.16, a result that was statistically significant. By selecting coherence breaks that were two standard deviations below the mean they successfully identified nine of eighteen chapter transitions. As the cutoff point was increased, more breaks were identified but so, too, were false positives, which increased at a linear rate. Some of the chapter breaks with unexpectedly higher scores (0.25 and 0.29) were found to comprise text that linked the two chapters, accounting for these higher values that are still, it should be noted, well below the mean of 0.41.

[Timm and Schinner 2023] performed several experiments on the Voynich Manuscript in order to ascertain how related or unrelated the Currier “Language A” and “Language B” variants might be. They utilized a vector space model built from the Voynich Manuscript where documents are created from individual pages. These pages are represented by vectors utilising the Term Frequency (TF) of each term present on the page (so the vectors will consist mostly of zeroes — this is analogous to the columns in the \(A\) matrix described earlier). They concluded there is no sudden transition between the two “languages”, A and B, and that there is a gradual evolution between them after experimenting with cosine calculations between all possible pairs of pages. One possible drawback to the Timm & Schinner approach is their reliance on the TF model of the text. A cosine will only score a non-zero value if there is text overlap between the two documents being compared. While valuable, the TF approach does not identify underlying latent relationships, structure, or synonymy. By contrast, LSA has the advantage of recognising and scoring relationships between words, identifying relationships between non-identical words that may be functioning synonymously. Additionally, TF values can skew results because terms that are very common can artificially inflate the similarity scores. Weighting methods such as Term Frequency-Inverse Document Frequency (TF-IDF) and Log-Entropy[4] can help smooth out these discrepancies.

[Reddy and Knight 2011] performed a similar page-wise comparison experiment on the Voynich Manuscript that focused on whether or not it was likely the leaves of the manuscript are in the correct order. Like [Timm and Schinner 2023], they created a TF vector for each page and performed page-wise comparisons using the cosine operator, results which suffer from the same weaknesses outlined above. They published results of page comparisons with the pages “scrambled” in which the scrambled adjacent-page similarity scores were close to 0, while they showed higher similarity when pages were scored in accordance with the manuscript’s current page sequence. They do, however, note that some of the non-contiguity of the herbal/pharmaceutical sections and the A/B language distinctions may suggest that some pages were probably re-ordered.

[Sterneck, Polish, and Bowern 2021] investigated topic modelling in the Voynich Manuscript using techniques such as Latent Dirichlet Allocation (LDA), Latent Semantic Analysis (LSA) and Nonnegative Matrix Factorization (NMF). The objective was to try and cluster the text in the Voynich to look for any interesting clusters of subjects or topics. This included using features such as Lisa Fagin Davis’s five scribal corpora, A/B language classification, visual subject label (astrology, botanical, balneological, rosette, recipes/starred section and unknown), and individual quires. The text was vectorized on a page by page basis (with TF-IDF weighting applied to the NMF/LSA methods to increase the visibility of rare words and decrease the influence of words that are quite common) and fed into the cluster algorithms after being transformed by one of the previously mentioned methods (LDA/LSA/NMF) and introducing a dimensionality reduction phase (reducing the high-dimensional space created by these methods into a lower-dimensional space that may enable topics or concepts to be more visible in the analysis). The k-means clustering algorithm was then utilised and results presented. Initial results were generated using all three methods, but additional experiments were continued using only NMF; this was due to identified issues with LDA (picking optimal parameters) and potential issues with LSA, a technique that does not perform well with polysemy or homophony as such relationships dilute the method’s ability to discover underlying latent connections between word usage. Entropy experiments in [Lindemann and Bowern 2021] suggest this may be an issue, although our experiments either did not encounter these issues or found them to be negligible, if present at all. Based on their experiments (largely using NMF), Lindemann and Bowern suggest the text in the Voynich Manuscript likely represents an enciphered human language rather than “meaningless” text and that the illustrative topics can be clustered to some degree.

For the initial coherence experiments, we employed several open-access texts in addition to the VM. These texts were chosen because they represent a variety of linguistic and stylistic genres: George Orwell’s Animal Farm (approachable contemporary English prose in a coherent narrative); The Aberdeen Bestiary (didactic and formal medieval Latin and a modern English translation, with each chapter representing a different and clearly-defined topic)[7]; Dante’s Inferno (rich and poetic medieval Italian in a chapter-based narrative); the Speculum Humanae Salvationis (didactic and formal medieval Latin in topically-varied chapters), and computer-readable Voynichese generated via the software[8] kindly provided by Torsten Timm [Timm and Schinner 2020][9]. For the VM ASCII transcription, we used ZLversion 2a dated 17/03/2022, which is the most complete, consistent, and accurate transcription available.[10] The wide variety of chosen texts should prove useful to demonstrate the coherence measurement and the behaviour of the texts in these contexts. We would expect them all to behave in a somewhat similar fashion regardless of the style of the writing. The artificial text generated via the Tim and Schinner (2020) algorithm is meant to mimic the Voynich Manuscript text, according to their self-citation hoax hypothesis, and preserves some of its statistical features (for example, the generated text should follow Zipf’s law). Their hoax hypothesis states that the text gradually changes over time based on various observations they have made on the manuscript (that many words in close proximity with one another are also very similar to one another — for example Voynich terms such as qokeedy and qokeey. Although we can’t use this to generate every section of the Voynich Manuscript individually, we used it to create a set of text roughly the same size as the Voynich. The algorithm only simulates a subset of the manuscript text and there is no language A/B distinction (they do not subscribe to the fact that there is anything really significant between them and it is a natural function of this gradual shift in text/words over time that causes it); in their experiments they generated a portion of text of roughly the same size as recipes section in the manuscript.

Before we can utilize LSA, there are several steps that must be undertaken to prepare each text for further use. These steps are typical for the application of many NLP algorithms, such as lowercasing or removing punctuation and stop words. As the VM cannot be “lowercased” (since it has no identifiable upper- or lower-case characters), we pre-processed all of the baseline texts in other ways for consistency: removing punctuation and using an algorithm to tentatively identify and remove the stop words. Stop words are common terms that, due to their grammatical function and frequency, offer very little value to the semantic discourse. In practice, stop words act as background noise that can dilute other relationships in the text. In English these are words such as “the”, “of”, “is”, etc. Again, in order to treat all of the texts consistently, stop words were determined for all texts using an automated method, detailed in Appendix 1. Because we do not know what the stop words are in the Voynich Manuscript, we used the same algorithm for all of the texts instead of referring to pre-existent lists of stop words in the relevant languages. This gives us a like-for-like basis of comparison.

The VM ZL Transliteration file underwent additional pre-processing. The file follows the Intermediate Voynich MS Transliteration File Format (IVTFF) version 1.7 as defined in (Zandbergen, IVTFF - Intermediate Voynich MS Transliteration File Format - File Format 1.7 2020) that categorizes the text present in the VM. Voynichese “words” (terms) are generally classified based on their graphic context: In the IVTFF transliteration, each term is tagged as occurring in a standard paragraph, standing alone as a “label,” or written in a non-standard layout such as around a circle or radiating from a central hub. Our analyses typically disregard “labels” as they offer little to the discourse and more than 50% of them are hapax legomena (terms that only occur once in the text). “Paragraph” texts behave in ways that are similar to the legible texts in our comparison datasets and thus are most useful for LSA analytics. We also removed any terms from the VM transliteration that are tagged as uncertain in any way, such as terms with unclear spacing or ambiguous characters.

Table 2 presents statistics for the processed texts, including total word count, size of stop word list, word count after stop-word removal/pre-processing and the total number of hapax legomena (unique words) in the text. Note that it is the total terms after stop word removal that are used for document segmentation as well as construction of the semantic space although in practice hapax terms are not considered in similarity comparisons scores, as they only occur once and the SVD operation therefore has no context for comparison.

Table 2: Post-Processing Text Statistics for Coherence Test Data

For each LSA semantic space, we compared the first document (of size 20, 30 or 40) to its adjacent document in order to calculate a similarity score; we repeated the experiment by shuffling the sequence of compared documents such that no two consecutive documents were compared. Since shuffling them is a stochastic process, this was repeated 10 times, and an average of the ten scores was calculated. In other words, we ran the two tests on each of the seven textual datasets: one by calculating the average similarity of adjacent documents, and another using a shuffled set of documents. For each of the text samples, nine semantic spaces were created by applying three weighting methodologies (TF, TF-IDF, Log-Entropy) to three different document sizes (20, 30, 40). The results (i.e. the average similarity score across the entire text) are reported in Figures 4-6:

In each case, the average similarity for the in-sequence text was significantly higher than the shuffled-text results, demonstrating that the method correctly responds to adjacency vs. non-adjacency. The TF weighting (Figure 4) gave the lowest scores, while TF-IDF and Log-Entropy produce very similar results. The LSA semantic space of 40 resulted in the most visible contrast between “correct” and shuffled texts. For this reason, we used documents of (approximately) size 40 using TF-IDF for the discourse segmentation component of our experiments. The charts demonstrate that the results on known texts are in line with what would be expected.

Additional statistical tests were performed to determine if the results were statistically significant. The Shapiro-Wilk test and D’Agostino’s K0 squared test were performed on the cosine values to determine if the raw similarity score distribution was normal; this was negative in all cases. We performed the Mann-Whitney U to test for significance. All of the comparisons scored well below a \(p\) value of 0.05 — in fact most were close to zero with the highest \(p\) value recorded being 0.000145. In other words, the results were indeed statistically significant.

The experiments above demonstrated that the text in the Voynich Manuscript behaves as would be expected for a meaningful text. In the following experiments, we will test the ability of a methodology called “discourse segmentation” to identify significant features such as section breaks, chapter breaks, page breaks, or other transitions, using the Speculum as a proof-of-concept text. A successful experiment would indicate that the same method might be able to detect similar features in the Voynich Manuscript.

We first applied this to the known text, the Speculum. This was done on two versions of the Speculum. We used both the original Speculum and a ‘cleaned’ version (removing clauses that do not contribute to the semantic reading of the text such as the words “Chapter 1” or the summaries of the previous chapter that begin each new one). This is in line with the work done by [Foltz, Kintsch, and Landauer 1998], the source of this technique. For the unaltered Speculum dataset, the average similarity score across the entire text was 0.440 and the average score at chapter breaks was significantly lower, 0.225. This shows a clear distinction between the complete text and the chapter breaks overall. Using the cleaned version of the Speculum, the results show slightly more contrast, with a higher overall average of 0.445 and a lower average score at chapter breaks of .212. This indicates that removing the small amount of redundant text at the beginning of each chapter allowed for a more accurate comparison of the actual content of each chapter. It should be noted at this point that all further results reported using the Speculum dataset use this ‘cleaned’ version.

We used the bottom 5% of the results as a cutoff point for investigation.[11] Can characteristics of the compared text samples explain the low scores? Are the compared documents truly dissimilar, or are they false negatives? In these explanations, we will reference a generic concatenated document of size twenty as d1-d2-…-d9-d10 [ — ] d11-d12 …-d19-d20, where “dx” is a document, and [ — ] represents the transition from one cluster of ten to the compared group.

For the cleaned Speculum experiment, the bottom 5% are represented by 82 scores, with a cosine cutoff point of approximately 0.128. The transition between the two groups of ten occurs at a chapter break in 12% (10) of these low scores, identifying a change of topic and explaining the low score. This value rises to 49% if we include scores where the chapter break occurs not at the [ — ] transition but at documents that are near it, ranging from d7 to d14. Reference to the text of the Speculum (reading the Latin) identifies major topic changes or unrelated content in 51% of the lowest scores. Figure 7 visualizes these results. Each dot represents the cosine comparison score between two concatenated documents of 10 X 20, presented in order sequentially from left to right. The vertical red lines represent chapter divisions, and the horizontal green line represents the lowest-5% cutoff. Upon reference to the Latin text, it becomes clear that the high scores do indeed represent text samples that are related in content. For example, the cluster of high scores in the upper right corner represent chapters that record prayers directed to and describing events in the life of the Blessed Virgin Mary. The sequence of low scores at the far right reflects the transition of the text into a summary conclusion that leaps from topic to topic.

Having established that the concatenation experiment successfully identified similar and dis-similar text in addition to being sensitive to chapter changes, we ran the same experiment on the Voynich, with intriguing results. Overall, the similarity average was 0.483 while the average similarity over page breaks was found to be 0.421. The average section break score (for example at the Zodiac → Biological transition) was 0.323; this lower-than-average score suggests that those sections could indeed be concerned with different subjects, as suggested by the differing illustrative content, and that the semantic space could be identifying topic changes between sections.

In addition to section breaks, we also examined the scores at scribal transitions. These average scores were even lower, only 0.232. Most scribal transitions represent language (A/B) or section breaks, which explains this result. As described above, Scribe 1 writes entirely in Language A while Scribes 2, 3, 4, and 5 use Language B. Upon closer inspection of the scores at scribal breaks, we observed that this method is particularly sensitive towards scribal changes involving Scribe 1, that is, at transitions from Language A to Language B. We had anticipated that the analysis might reveal this distinction, although the magnitude of the distinction was surprising. The average similarity score at A→B transitions (that is, transitions from Scribe 1 to any other scribe and vice versa) is only 0.206, while B→B scribal changes had a higher average score of 0.332, a substantial difference (there are no A→A scribal transitions because Scribe 1 is the sole user of Language A). This suggests that there is indeed a substantive and significant difference between Language A and Language B.

Conducting the same bottom-5% analysis for the Voynich (with a cosine similarity score cutoff point of about 0.205), we found that most of the lowest scores can indeed be explained by reference to the manuscript itself. Of these approximately-sixty lowest scores, 20% (12) were A→B transitions, and 6% were section transitions. The overwhelming majority of the remaining scores (77% or 46) were Herbal → Herbal page comparisons, virtually all of which were near or on a page break.

As with the Speculum, a visualization of the Voynich results is provided in Figure 8, with color indicating the different sections and red vertical lines indicating A <-> B transitions. The horizontal green line represents the lowest-5% cutoff. Upon closer inspection, the numerous A→B scribal changes in the Herbal section (the tight cluster of red lines) are reflected in the data, as the scores visibly decrease and cluster in the bottom half of the gray section. Also of interest are the higher scores in the Biological (orange), Pharmaceutical (salmon), Zodiac (dark green), and Stars (green) sections — very few of which are below the 5% line — indicating that these sections may concern coherent and distinct topics.

In order to generate our semantic space, we first need a set of \(m\) terms and \(n\) documents. The list of terms should be all the unique terms from the collection of text overall. The documents should be the set of documents (these can range from sentences and paragraphs to an entire large piece of text) from which we wish to create our semantic space. From this dataset we generate a \(m\) by \(n\) (term by document) matrix \(A\). Each \(A_{ij}\) value in this matrix indicates the number of times term \(i\) occurs in document \(j\), the so-called Term Frequency (TF).

The next step is to compute the Singular Value Decomposition (SVD) of the Matrix \(A\). This is a form of factor analysis and factors our matrix \(A\) into three matrices: $$A = TSD^{T}$$

These matrices have the following shape:

For a full mathematical description of the factors, the reader is referred to the original paper [Deerwester et al. 1990]. The \(T\) and \(D^T\) matrices will be almost entirely nonzero, and the \(S\) matrix is actually a diagonal matrix of the singular values from the computation in descending order.

Roughly speaking, the \(D^T\) matrix refers to the documents from which we have built this space. Each column corresponds to the document in the column space of \(A\) (which, most importantly, is non sparse). The next step in this process is the dimensionality reduction phase. This involves picking a value, \(k\), and reducing the dimensionality of our \(T\), \(S\) and \(D^T\) matrices such that they are \(m\) X \(k\), \(k\) X \(k\), and \(k\) X \(m\) — that is we keep the first \(k\) columns in \(T\), the first \(k\) values in the diagonal matrix \(S\) and the first \(k\) rows in \(D^T\). This results in an approximation of \(A\). More importantly, we reduce the dimensionality in order to filter out some of the noise in the space and focus on the key relationships found within. The selection of this value is somewhat subjective. In the work by [Foltz, Kintsch, and Landauer 1998], they use a value of 300 for \(k\) as their original \(A\) matrix is of a moderate size. There are some suggested methodologies for computing an appropriate value of \(k\) (such as choosing a \(k\) such that the ratio of the sum of the squares of the first \(k\) singular values divided by the sum of the entire set of squared singular values is about 0.8), but we have found that for smaller datasets they tend to work quite poorly. As the term-by-document matrices we will be dealing with (such as the text from the VM) we have used a value of 75 for \(k\). Note the matrices we will be using are much smaller than the ones used in the original experiment in [Foltz, Kintsch, and Landauer 1998] so a smaller dimensionality is justified. We give a justification of this value next.

​​The dimensional reduction step in LSA involves a selection of the number of dimensions to use from the SVD decomposition. This is a tricky choice, and much of the literature (especially in the context of Information Retrieval (IR)) suggests you pick a \(k\) that gives you the best results (for a typical train/test split for an IR problem this is straightforward). As we don’t know the contents of the Voynich, this is clearly impossible, so we use three heuristics to approximate what might be a good value for \(k\) by calculating what might be a good lower, mid, and upper bound selection of values. This is actually done for all the dataset’s semantic spaces. Note, these heuristics are designed for a semantic space generated from a single text which may not be typical in general. Given the results from [Foltz, Kintsch, and Landauer 1998], we know that sequential document comparisons should have a larger value than the shuffled variant as per our first experiment; so we first calculate a coherence score which is the average similarity of adjacent documents in the \(D^T\) matrix. Next, we shuffle the order of the documents in \(D^T\) and make the same calculation (random) and do this 10 times taking the average. This is like our coherence experiment except we are using this to help calibrate \(k\). We then use three different techniques to approximate \(k\).

1. Noise separation: Compare the baseline coherence for a dimension with the random baseline. Select the \(k\) with the largest difference (indicates strong semantic pattern). This would act as the lower bound for \(k\) as typically the difference is largest the smaller the dimension (assuming the text has any coherence). If the dimension selected is quite low this runs the risk of over-simplification and causing the model to lose essential topic/latent information.

2. Elbow detection: Find point of diminishing returns in the coherence scores. This is often indicated by the “elbow” of the coherence scores over the range of \(k\) explored. The similarity scores at this point begin to taper off. This is typically the default value of \(k\) as it should be larger than the Noise separation value and smaller than the Slope stabilization value.

3. Slope stabilization: Identify when the coherence values begin to flatten out. As the dimension increases usually the scores decrease. When the scores flatten out additional dimensions contribute very little. This is typically the upper bound for a reasonable \(k\).

We ran these heuristics over a \(k\) range of [25, 200]. We did this for all datasets used, TF, TF-IDF and Log-Entropy weighting methods as well as document sizes of 20, 30 and 40. The range of values for the noise separation methods was [25, 30], for elbow detection it was [65, 90] and for the slope stabilization method it was [115, 160]. A reasonable range for \(k\) would seem to be 65-90 and as the average value for the elbow method, overall, was 75 so that was selected as the dimension to use for all the semantic spaces (anything in the range 65-90 should work reasonably well).

To compare two documents in the \(D^T\) matrix we need to generate two vectors to perform the cosine operation on. If we wanted to compare document \(i\) and \(j\) (corresponding to columns \(i\) and \(j\)) we would generate their vectors as follows: $$\vec{v_{i}} = S_{k}D^{T}_{ki}$$ $$\vec{v_{j}} = S_{k}D^{T}_{kj}$$

Each vector is comprised of the product of the \(S\) matrix (singular values) and the appropriate column in the \(D^T\) matrix. The subscript \(k\) indicates we are using the first \(k\) dimensions.

Then apply the cosine operator as follows: $$\cos(\vec{v_{i}}, \vec{v_{j}}) = \frac{\vec{v_{i}} \cdot \vec{v_{j}}}{\| \vec{v_{i}} \| \| \vec{v_{j}} \|}$$ We can also compare documents that do not already exist as columns in the document semantic space. This is achieved via a process known as “folding-in” a document which essentially transforms it into the same vector space as the documents that already exist in \(D^T\). So given a new document, \(d\), that we want to use for comparison purposes, we put it into the same format as a column in the original A matrix; that is, each dimension of \(d\) indicates how many times a particular term occurs in the document. We then fold it into the \(D^T\) matrix (essentially as an extra column) as follows: $$\vec{d^{1}} = \vec{d}T_kS_k^{-1}$$ At this point, the new document vector \(\vec{d^{1}}\) can be treated just as any other document vector in \(D^T\) for comparison purposes.

One question that has yet to be answered is: What do we consider a “document” in the context of generating our initial term-by-document matrices? For the original coherence experiments in [Foltz, Kintsch, and Landauer 1998], they used a document size of one sentence and, in the discourse segmentation part of their experiments, they used collections of 10 paragraphs as a document unit. With the VM we have no real concept of “sentence” as there appears to be no punctuation in the writing. Bearing this in mind (and wanting to treat all documents equally) we opted to experiment with document sizes of 20, 30 and 40 terms respectively. These are simply constructed by taking term sets of 20, 30 and 40 terms from the processed texts and treating those as the document partitions for creating the initial term-by-document matrix \(A\).

We experiment by applying additional NLP techniques to the \(A\) matrix to try and lessen the effect of more common terms in the comparisons. There is potentially a problem in only utilizing TF values in the \(A\) matrix in that very common terms will have more influence in comparison operations then perhaps they should, purely due to their common appearance over the text. Removal of stop words partially resolves this issue, but there will still be a disproportionately high presence of some terms over others. In addition to creating the \(A\) matrix as described, which uses straight TF values, two weighting schemes are also employed to modify the values contained in \(A\). The two schemes applied are Term Frequency-Inverse Document Frequency (TF-IDF) and Log-Entropy (LE).

Log-Entropy is defined in Equation 5 (in the context of matrix \(A\)). \(f_{ij}\) represents the TF of term \(i\) in document \(j\) and \(p_{ij} = \frac{f_{ij}}{\sum_{j}f_{ij}}\) which equates to the fraction of occurrences term \(i\) has in document \(j\) over all occurrences of term \(i\) and \(n\) is the number of documents (columns) in \(A\). Log-Entropy was the matrix weighting mechanism used in the experiments by [Foltz, Kintsch, and Landauer 1998]. The first logarithm decreases the effect of the differences in term frequency (to help smooth them out) whilst the entropy calculation will tend to give less weight to frequently occurring terms; additionally, it takes the distribution of the terms over the document set into account. $$A_{ij} = \log(1 + f_{ij}) \left[ 1 + \left(\sum_{j} \frac{p_{ij}\log(p_{ij})}{\log(n)}\right) \right]$$ TF-IDF is defined in Equation 6. The value \(df_i\) represents the document frequency of term \(i\); that is, the number of unique documents that term \(i\) occurs in. The logarithm value is the inverse document frequency, and it is high if the term is rare, thus emphasizing the value of rare terms and de-emphasizing the values of common terms. $$A_{ij} = f_{ij} \log{\left( \frac{n}{ df_{i} } \right) }$$ It is worth noting that each semantic space is trained on its own document and comparisons/experimentation is done within that semantic space. This can be viewed as a circular relationship or, perhaps better stated, self-contained distributional modelling (in the sense the training data is what we are working on) and can potentially viewed as problematic. Our experiments are not testing generalization (an issue with over-fitting); instead, we are testing internal structure. The goal of our work is to characterise internal coherence and discourse structure rather than consider predictive performance or any sense of generalization; that is, do the different semantic spaces behave in a similar fashion?

There is also some precedence of the usage of a generated semantic space and experimentation on the same space with the data it was trained on. [Dos Santos and Favero 2015] built a LSA semantic space from a set of student answers to exam questions. They evaluated the performance of this on the same dataset (comparing human assigned scores to computer generated scores) and reported an 85% agreement with human scoring. [Altszyler et al. 2017] applied LSA using a semantic space made up of dream reports analysing usage patterns of words between dreams and waking life (the analysis making use of the same semantic space data). They also compared the results with Word2Vec deducing that with smaller corpus sizes Word2Vec becomes less effective compared to LSA. In addition, the paper [Foltz, Kintsch, and Landauer 1998], from which we are taking our methodology, both for coherence and discourse segmentation, literally uses a semantic space trained on a dataset and then performs discourse segmentation on this space using the same dataset (no external data was used for training) which is exactly what we are doing.

In order to generate a stop word list, terms are sorted by their Term Frequency (TF) count in descending order. Using a window of size \(w\), we calculate the average difference in TF between the first \(w\) terms in our list. We move this window by 1 word and calculate this difference again. This process is repeated until this difference falls below a threshold value of \(v\). If \(x\) steps were required to reach this threshold value, the first (\(w + x – 1\)) words in our list are selected as our stop words. In other words, we look at the average decrease in the TF in a moving window until the rate of change falls below some value \(v\). We utilize a window of size 6 and a threshold of 3 to generate reasonable stop word lists for English, Medieval Italian and Latin (essentially the top most frequent words given the cutoff threshold defined above). The VM stop word list is calculated using the same process and parameters. The stop word lists and their term frequencies are shown in Figures 10 through 16: