Developing Toponym Classification and Geolocation Estimation from Thai Tweets

Table 1: Examples of toponym categories and annotation formats

4. Methods

4.1 Constructing the Toponym Recognition Model

The approach employed for constructing the toponym recognition model in this study was adapted from Thai NER research. However, academic knowledge of Thai NER remains scarce compared to more widely studied languages like English, French, and Chinese.

In 2018, Google AI Language introduced BERT, an approach that utilizes an encoder-specific knowledge transfer architecture to develop a language model. This approach involves training the model on unlabeled datasets from Wikipedia and a book corpus comprising published English books. The versatility of BERT allows it to be fine-tuned for a broad range of natural language processing problems, including NER [16]. However, there are some limitations for the Thai language due to the model being trained on data from over 100 languages simultaneously, which makes it challenging to capture the specifics of the Thai language or account for the diverse range of topics that appear in the Thai language datasets. To address this limitation, RoBERTa was developed for the Thai language in 2021 and trained on over 78.5 GB of Thai language datasets from various sources [12].

4.1.1 BERT architecture

BERT architecture for extracting and classifying geo-names as previously discussed, BERT leverages knowledge transfer and attention mechanisms to learn the relationships between words or sub-words in text. Typically, a knowledge transfer system comprises an encoder that receives input text and a decoder that predicts the result. However, in this study, only the encoder component of BERT was utilized, and a classifier element was added instead. The use of BERT in this study is summarized in Figure 2. In Figure 2, the first section after the input sequence is called embeddings, which consists of three subsections: position, segment, and token. As shown in the example, the input sentence was divided into two parts, A and B. Tokens represent the embeddings obtained from the segmented words. Subsequently, the input was passed through BERT’s encoder, and the output was fed into the classifier to generate the IOB tags. Finally, the data were randomly divided into three datasets for model training: the train set (80%), the validate set (10%), and the test set (10%).

Figure 2: Summarized functions of the employed BERT architecture

4.2 Clustering Algorithm for Geolocation Estimation

As described in Section 2.3, the toponym recognition model was used to extract toponyms from the texts. In cases where the extracted toponyms could not be matched to any coordinates in the database, other toponyms within the same text were used to assist in estimating the geolocation. According to Figure 3, the data from Twitter went through the toponym recognition tool developed in Section 4.1. The next step involved geocoding, where the tool generated geographic coordinate data as output if it could successfully match the toponyms with service providers such as Google's Geocoding API. However, if online service providers could not perform geocoding, clustering algorithms were executed to estimate approximate coordinates. The available clustering models for this task were Topology Words, DBSCAN, K-means, K-medoids, and Agglomerative Clustering.

Once the data was inputted into the algorithm, analyzing whether the target toponyms (words that the Geocoding API could not geocode) appeared in other sentences became imperative. In line with this objective, the study prepared a test dataset containing 100 sets of toponyms. Each set included around 3-20 messages, resulting in a total of 430 messages. Following this process, an analysis was carried out to identify words within geospatial proximity and determine the geospatial boundaries of the target toponyms. Examples are illustrated in Figure 4. To ensure accurate geolocation estimation, potential noise was filtered out from the extracted toponyms using the following clustering algorithms.

4.2.1 K-means

The K-means algorithm is an unsupervised machine-learning technique commonly used for clustering problems. This algorithm partitions objects into K groups and replaces each group with the group means, which serves as the group’s centroid and measures the distance of data within the same group [17]. The algorithm operates through the following steps:

1) Determine or randomly select K initial values (groups) and define K initial centers, which are referred to as cluster centers or centroids.

2) Assign all objects to groups by calculating the distance between each data point and the center. The data point closest to the center value is assigned to the corresponding group.

3) Calculate the mean of each group to obtain the new center value.

4) Repeat Steps 2 and 3 until convergence is reached when each group’s mean or center point no longer changes. Then, finally, terminate the process.

Figure 3: Geolocation estimation process

Figure 4: Clustering algorithm for geolocation estimation using toponyms

4.2.2 K-medoids

K-medoids is a clustering technique similar to K-means, but instead of selecting centroids at random from the dataset, it selects medoids, which are actual data points within the dataset, to serve as the center of the clusters [18].

4.2.3 Agglomerative clustering

Agglomerative clustering is a technique that does not require pre-determination of the number of clusters to be formed. Instead, it involves a step-by-step analysis consisting of the following primary steps. Initially, each data point is considered a separate group. For instance, in the case of five toponyms that can refer to geolocations, they can be initially divided into five separate groups.

1) An N x N metric is constructed from the data set, and the distance between each data point is computed.

2) Identify the smallest distance between the data points and group them. Then, improve the metric by selecting the larger of the two groups.

3) The updated data table is retained by keeping the maximum value of the two groups. For instance, if point 1 has a value of 10 and point 2 has a value of 7, the value of 10 is selected.

4) Steps 3-4 are repeated until a single group remains. The final output illustrates the relationships between each data point. Hence, agglomerative clustering is another suitable technique for supporting geolocation estimation [19].

4.2.4 DBSCAN

DBSCAN (density-based spatial clustering of applications with noise) DBSCAN is an unsupervised machine learning segmentation algorithm. This method is considered more robust than other comparable algorithms, such as K-means and hierarchical clustering, as it can handle data clusters with various shapes that are not clearly defined and remove noise data that does not belong to any group. Generally, DBSCAN requires two critical parameters for its operation:

1) Eps represents the maximum radius around a core point, including neighboring data points. These neighboring points are determined based on the parameter MinPts.

2) MinPts is the minimum number of data points required to form a cluster around the center point[20]. Various approaches exist for defining MinPts, but for this study, the approach of Devkota et al., [21] was used, which specifies that MinPts should be equal to the dimension of the data plus one, but not less than three Sarma et al., [22].

In summary, the algorithm functions as follows:

a) Set the value of Minpts to N points.

b) Identify the Core Point X as the point with the closest N neighboring points (including X itself).

c) Move the Core Point X to another point within its Eps radius. If nearby points are within the Eps radius, group them with X in b) to form a cluster.

d) Repeat step c) for all points in the dataset. Points not within the Eps radius of any core point are classified as noise and are not used to estimate the geolocation.

4.3 Use of Topology Words

Geolocation estimation involves using topology words to determine the location of a place. To illustrate, consider the following sentence: “The head office of the Metropolitan Electricity Authority is situated in the vicinity of Khlong Toei, adjacent to Rama IV Road.” As noted, the sentence contains bolded topology words that indicate the topological relationships between locations. These relationships are assigned weights, such as a weight of 3 for the word “adjacent” and a weight of 1 for the phrase “situated in the vicinity.” The weights are assigned using a dictionary-like structure in Python, and the influence of the relationships is shown in detail in Figure 4.

The following pseudocode demonstrates the inner workings of the algorithm in Figure 5. The pseudocode below uses the topology word dictionary, a set of words and their corresponding weights, as denoted by T. The dictionary is defined as follows: {‘is at’: 3, ‘is in ‘: 3, ‘in’: 3, ‘at’: 3, ‘on’: 3, ‘is on ‘: 3,..., ‘in the zone’: 1, ‘area’: 1, ‘around’: 1, ‘surrounding’: 1}. G is a list of toponyms and their associated tags, such as [[‘Bangkok Christian School’, ‘ACP’], [‘Adjacent to’, ‘GEO’], [‘Pramuan Road’, ‘BSN’]] and so on. Finally, the result of the algorithm is a list denoted by R, which contains the processed output.

Figure 5: Weight values of topology words

Figure 6: Topology words psudocode

4.4 Assessing Model Validity

4.4.1 Assessing the accuracy of the toponym extraction model

This study utilized three matrices, including precision, recall, and overall validity (F1), at the phrase level. This approach enables the assessment of all valid words rather than solely focusing on individual correct words and ensures that the contribution of each valid word is accounted for in the assessment process.

Equation 1

Equation 2

Equation 3

where:

TP = The number of toponyms recognized
correctly

FP = The total numbers of toponyms in the
system output

FN= The total numbers of true toponyms in the
dataset

Assessing the performance of models using F1-Token alone may not provide a comprehensive outcome since the task involves extracting complete toponyms from text rather than parts of them. For instance, the sentence “วิวแม่น้ำโขงนครพนมสวยสุดละ นี่ชอบมาก” : “The view of Mekong River in Nakhon Phanom is the most beautiful. I like it very much.” exemplifies this concern, as depicted in Table 2. From the data presented in Table 2, it is possible to calculate F1-Token as follows: TP = 2, FP = 1, FN = 1, resulting in the precision of 2/(2+1), the recall of 2/(2+1), and the F1-Token value of 2*[(0.67*0.67)/(0.67+0.67)] = 0.34. For F1-Phrase annotations, tags with two tokens are merged into a single tag, meaning the model’s answer is considered incorrect if any part of the token is incorrect. For example, if the tokens and tags are (River, B-NAT), (Mekong, I-NAT), and (Nakhon Phanom, B-ADMIN), then the token and tag sequence for the phrase would be Mekong, NAT, and Nakhon Phanom, ADMIN. In Figure 3, for the word Mekong River, the model provided an incorrect answer for the last token, and therefore, the model’s answer was deemed incorrect. Using the values from the example above, F1-Phrase can be calculated from TP = 1, FP = 1, and FN = 1. Precision is calculated as 1/(1+1), Recall is calculated as 1/(1+1), and the F1 score is 2*[(0.5*0.5)/(0.5+0.5)] = 0.25. Hence, based on the above example, it is more appropriate to assess performance at the F1-Phrase level since the expected end result of the model is to extract complete toponyms.

4.4.2 Assessing the accuracy of toponym geolocation estimation

RMSE was selected as the metric of this assessment. In addition, intervals were assessed based on the number of points that fell within the radius of the corresponding toponyms. The intervals were divided into four ranges, namely 0.85, 3.39, 8.47, and 10.16 kilometres, to provide a comprehensive assessment. The accuracy of the map was determined at different scales, utilizing the formula outlined by [23] as follows:

If the scale is larger than 1:20,000, the following calculation should be used to determine ground meters: 0.03333 x scale x 2.54 / 100. However, if the scale is 1:20,000 or smaller, the calculation to use is: 0.02 x scale x 2.54 / 100. For example, at a scale of 1:1,000, the calculation is as follows: 0.03333 x 1000 x 2.54/100 = 0.85 ground meters.

Table 2: Illustration of F1-Phrase identification

5. Results

5.1 Toponym Recognition Model Training

The initial phase of this study involved constructing a toponym recognition model based on WangchanBERTa. The training process was conducted under a set of specific environmental conditions, including a seed value of 9, a learning rate of 0.00002, a weight decay of 0.01, and a total of 10 epochs. The training dataset consisted of 17,955 sentences for training, 4,510 sentences for validation, and 5,617 sentences for testing. The results of the training process are shown in Table 3. According to Table 3, the optimal F1 obtained during model training was 0.9188. This result was achieved by utilizing the cosine_with_restarts learning rate scheduler (lr_scheduler_type). In addition, four experiments were conducted with the warm-up ratio suggested by Mishra and Sarawadekar [24]. Furthermore, as demonstrated in Table 4, the WangchanBERTa-based model exhibited the highest overall accuracy (F1) compared to the other models. Table 4 suggested that the BERT model achieved the highest F1 at 0.919 compared to other models. In the context of toponym recognition, additional training from the WangchanBERTa model in this study proved suitable. Table 5 illustrates the F1 values achieved by category. Table 5 presents the results of the fine-tuning process, indicating that the Regional Place (RP) category achieved the highest F1 score of 0.962, while the Monument (MON) category had the lowest F1 score of 0.790, which includes monuments, roundabouts, and clock towers.

Table 3: Training results of the toponym recognition model using BERT architecture

Table 4: F1-phrase accuracy across models

Table 5: Accuracy values by topological categories using WangchanBERT