Word embeddings

This is the story of how a machine-learning competition in Cameroon led me from software engineering to NLP research—and to writing my tutorial on word embeddings.

See the tutorial. Below is its genesis.

In 2019, I was just a programmer. At the engineering school I attendedThe National Advanced School of Engineering Yaoundé (NASEY) is an engineering school in Cameroon., we first completed two intense years of preparatory classes, with substantial mathematics, physical sciences, and algorithms. Then everyone chose a department: Computer Engineering, Electrical Engineering, Mechanical Engineering, Civil Engineering, Telecommunications Engineering, or Industrial Engineering. I chose computer science engineering and was already in the second year of that program—my fourth year at university.

When James Assiene MoudieJames Assiene Moudie launched the first MLPC and later became a research engineer at DeepMind. launched the first edition of MLPCMLPC stands for Machine Learning Project Competition. NASEY students use machine learning to address a local problem in Africa., I had the idea of creating a system to translate local Cameroonian languages automatically. Unfortunately, at that time, the closest we had come to artificial intelligence at school was:

• September (Fall) 2016 to July (Summer) 2018

  In general, all our mathematics training was useful for understanding machine learning theory: Real Analysis, Linear Algebra, Euclidean Affine Geometry, Probability and Statistics, Series and Generalized Integrals, Multilinear Algebra-Curves and Surface, Analysis in finite-dimensional vector spaces, Numerical Analyses.

  Personal note NDONG NGUEMA Eugène Patrice, who taught us Series and Generalized Integrals (Fall 2017) and Numerical Analysis (Winter and Summer 2017), is the best teacher I have ever known. Beyond that, he is a genius. Unfortunately, his work as a teacher in Cameroon receives far less visibility than it deserves.

• September (Fall) 2018 to July (Summer) 2019

  • Formal Systems and Foundations of Artificial Intelligence
  • Mathematical Tools for Computer Science …
  • Science of information: (Shannon) Entropy, (huffman …) encoding…
  • Basic mathematics: measure theory, Laplace transform …

• September (Fall) 2019 to July (Summer) 2020:

  • Data Analysis, Theory and Practice (with Wilson Toussile, Fall 2019):

    • Statistical Learning : Parametric Estimation (Maximum likelihood estimator …), Confidence interval …
    • Supervised, unsupervised and semi-supervised learning formalism
    • Bayesian classification, linear and quadratic regression, bias-variance risk decomposition, cross-validation, k-nearest neighbors, K-means classification,
    • Hierarchical classification (hierarchical ascending classification…), Hard classification, Fuzzy classification, Similarity and dissimilarity measures, Clustering by mixture models (Gaussian case), EM algorithm.

  • Artificial Intelligence and Applications : Multi-Agent Systems (really old school) …

  • Grammars and Languages: Chomsky hierarchy of grammars, Canonical automata, etc…

• September-December (Fall) 2020 :

  • Advanced Machine Learning: there was nothing advanced, the professor just took Ian Goodfellow, Yoshua Bengio and Aaron Courville’s book and came to explain worse than what was in the book.
  • Image processing, GIS and WebMapping
  • Data mining: unfortunately, Professor Henri Gwet, who taught us this class (a good teacher), passed away a few months after the end of the session.

Context Unlike in Montreal (UdeM), where I am supposed to take two or three courses per term in Fall and Winter, we had about twenty courses per school year in Cameroon. There were no electives: we took all the courses in the program.

Source I have listed only the courses directly related to my learning of machine learning. The complete list of courses is available separately.

So, in 2019, I already knew how to make computers work with real-valued vectors—for example, through linear regression—but I did not know how to make them process text. That is where I got my first taste of NLP: first with GloVe, Word2Vec, bag of words, and TF–IDFBag of words counts tokens; TF–IDF reweights counts by how distinctive a term is; Word2Vec and GloVe learn dense vectors whose geometry reflects patterns of word co-occurrence., and later with BERT and its variants. BERT had been released only recently and was attracting a great deal of attention.

If for the first approaches (GloVe, etc), understanding was quick, learning how Transformer works by myself until I could implement it was not an easy task for me.

Learning note I did not like following tutorials because I found many of them ineffective. I preferred to read the papers: difficult for a beginner, but once I understood an idea from the paper, I felt that I had really understood it. It is possible to follow ten tutorials on a concept and still miss its mechanism.

Faced with the difficulty of understanding how Transformer works :

• I went back to the original account of the attention mechanismBahdanau attention constructs a context vector as a learned weighted average of encoder states at every decoding step. See Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio, Neural Machine Translation by Jointly Learning to Align and Translate, ICLR 2015..
• I read Minh-Thang Luong’s PhD thesis: NEURAL MACHINE TRANSLATION, STANFORD UNIVERSITY, December 2016
• The Deep Learning book, Ian Goodfellow, Yoshua Bengio, Aaron Courville
• Coursera’s Machine Learning and NLP specialization
• …
• Here’s the document that lists all the papers I’ve read this period (briefly, from 2019 to 2022), and that helped me get started first in NLP, then in machine learning in general.

During this learning period, I wrote this tutorial on word embedding.

Back to 2020. I had completed my project, and we published On the Use of Linguistic Similarities to Improve Neural Machine Translation for African Languages. We proposed a dataset with parallel text for vernaculars absent from commonly used datasets such as JW300. We grouped related languages using their histories, morphologies, geographical and cultural distributions, and population migrations, and proposed a similarity metric that requires paragraph-level rather than word-level parallelism. Combining multitask learning with MLM and TLMMasked language modelling (MLM) predicts hidden tokens from their context. Translation language modelling (TLM) applies the same objective to aligned sentences from two languages so that information can cross the language boundary. on clusters of similar languages substantially improved individual translation pairs. In particular, we gained 29 BLEU pointsBLEU is an automatic machine-translation metric based mainly on clipped $n$-gram precision, with a brevity penalty. Higher is better, although human evaluation remains important. on Bafia–Ewondo relative to earlier methods that did not exploit multilingualism.

I also completed an internship at WL Research from July to December 2020 with Mohamed Hassan KaneSee the video Independent AI Research in Africa: What Role for the Diaspora?. We developed and deployed a machine-learning system that reviews end-user license agreements (EULAs)An EULA is the contract that specifies how an end user may use a piece of software. Our system flagged terms and conditions deemed unacceptable to the government.. I also worked on supervised learning with derivativesSobolev training and differential machine learning supervise derivatives as well as function values. SIREN uses sinusoidal activations to represent signals and their derivatives accurately., including Sobolev training, differential machine learning, and SIREN.

I made my first contact with Mila in February 2021. Many thanks to Dianbo LiuDianbo Liu was a postdoctoral researcher with Prof. Yoshua Bengio and led the Humanitarian AI team at Mila–Quebec AI Institute., who introduced me to research.

@misc{tikengnotsawo2020howienterednlp,
  author = {Tikeng Notsawo, Pascal Jr.},
  title = {{Word embeddings}},
  year = {2020},
  month = aug,
  url = {https://tikquuss.github.io/blog/how-i-entered-nlp/},
  note = {Blog post}
}