Predictably comforting: The writings of Soo Ewe Jin

Who was his favorite?

My dad was fond of writing about his family, though we hated when we became unwitting subjects of the week’s column. He usually anonymized our appearances as requested, but sometimes mentioned us anyway. The following graph displays the number of appearances by each members of our nuclear family (both in name and title).

My wife (“Evelyn”) got one mention in my dad’s last few columns. He mentions “son” 76 times, but this doesn’t discriminate between mentions of Tim (my brother) and I. I am mentioned by name twice, compared to Tim who is mentioned five times. All these mentions pale in comparison to my mother – she is mentioned by name (“Angie”") once, but referred to as his wife 136 times.

Frequently-used words

A crude way of determining what he wrote about is by looking at his most frequently-used words. The following graph plots the frequencies of the 25 most common words in his writing (excluding stopwords like “a” and “the”).

Some of these high-frequency words give us clues about his subject-matter: “people” (857 appearances), “time” (696 appearances), and “life” (561 appearances) fit pretty well on a conceptual level with the focus of his column – reflections about appreciating people and everyday things. Since the text I analyzed consisted of 366 articles, most of these words appeared more than once per article.

Visualizing these in a wordcloud gives us more room, so we can look at the 100 most frequently used words (each of which appears > 130 times in his writings). The size and color of each word corresponds to how frequently it appears in his writing. In the wordcloud, you’ll see other semi-frequent words like “work” (423 appearances), “world” (392 appearances), and “friends” (292 appearances) that give you more hints of what he often wrote about. However, word-frequency alone doesn’t paint a very clear picture. There are some high-frequency words like “will” (1012 appearances), “can” (920 appearances), and “one” (831 appearances) that don’t suggest particular subject-matter and are highly dependent on context.

A bigger problem is that we’re assuming particular nouns work as standalone markers and evidence of particular topics. However, the absence of a word (e.g., “friends”) doesn’t necessarily mean that a particular topic isn’t being written about. The following analyses look beyond individual words to the relationships between words.

Identifying the topics

It’s tricky to identify the number of topics (k) present in a corpus. There are some mathematical approaches to determining the optimal k, but they do not always produce topics (i.e. clusters of words) that are semantically meaningful. A simpler method is specifying an a priori number of topics, running the algorithm, and looking at the words (terms) associated with each topic to see if they are semantically meaningful. As mentioned above, k of 2 to 5 seem like reasonable values for a columnist like my dad.

Using the topicmodels package, I ran the Latent Dirichlet allocation (LDA) algorithm with k = 2, 3, 4, and 5, and determined that having three topics produced the best (most meaningful) topic distribution. The table below shows the three topics, which I have named, along with the top five terms associated with each.

The Encouragement topic probably encompasses writings of my dad where he gave exhortations to his readers to positively influence the world. Reflections probably include musings about aspects of daily life and the passage of time. Articles about Society are about the workplace and aspects of Malaysian society.

Classifying articles

The following graph displays the classification of all the articles based on their dominant topics.

I simply categorized each article based on which topic was most present. However, as noted above, multiple topics can be present in different proportions in a given article. Therefore, classifying articles in this way loses significant information about the mixture of topics.

For each article, LDA actually gives us the topic distribution, P(topic|article), such that the mixture of all topics sums to 1 within each article. An article classified as “Encouragement” above may have gotten that classification despite all topics being somewhat equally present, but with P(topic = Encouragement|article) being slightly higher than the second-most present topic. For a fuller picture, the following graph plots the topic distributions for all my dad’s articles.

As can be seen, my dad wrote pretty evenly about all three topics in all his articles. However, it might be interesting to look at articles where one topic was significantly more present than the others, to see how well the topic model did. For each topic, I looked for one article where the topic distribution heavily favored the main topic relative to the other two. The following table displays these articles and P(topic|article) for each topic (the probability of the main topic is bolded).

From reading the articles, it might be difficult to distinguish between Encouragement and Reflections. This is because all articles contain some mixture of the two, and because there is probably some conceptual overlap between these topics.

An interesting result is that articles about Society tended to be those that were not from his weekly column – they tended to be profiles, interviews, and obituaries of prominent figures in Malaysia’s civil society and business world. Most of these articles were found in the business section of the newspaper – where he also contributed some writings.(It is also worth noting that the article above about Society is titled A time for reflection, but is actually an article reflecting on the life of a prominent member of Malaysia’s legal fraternity that doesn’t contain the type of reflection typical of his Reflections).

We expect articles in the business section to be about Society much more than we expect them to be Encouragement or Reflections. I checked how many of his articles from the business section (52 of them) the LDA process correctly classified as being about the Society topic.

Only 22 of 52 articles he wrote in the business section were classified as being about Society. It seems that even when writing in the business section, he could not help but include Encouragement and Reflections. That’s just the kind of person he was.

Writing over time

The prior analysis shows my dad’s writings were pretty consistently about three dominant topics. Did his focus on each of these topics change over time? The following graphs plot P(topic|article) for each topic over time. A linear trend has been fitted to each graph so we can see if there are trends over time in each topic.

Society generally had a smaller presence in most of his writings – there seems to be the occasional article that is strongly about the topic, while it is present only at a low level for the majority of his writings. My dad occasionally wrote articles for the business and news sections of the paper, which probably accounted for the articles where P(topic = Society|article) are high. He worked in the business section in his earlier years, but stopped writing for them later on – which appears to be reflected in the overall decrease in this topic’s presence in his writings.

Over time, his writings increasingly focused on Reflections and less on Encouragement (the correlation between these topics is negative, r = -.59). This doesn’t mean my dad was getting less encouraging, merely that over time his writings consisted less of words exclusive to that topic (and even in his last few years, P(topic = Encouragement|article) was still pretty high – around .3 on average). My best guess is that my dad became more reflective (see the increase in Reflections over time), and that decreased the space he could dedicate to other topics.