RomanNum3ral
/
AI-For-Beginners
mirror of https://github.com/microsoft/AI-For-Beginners.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
AI-For-Beginners/translations/en/lessons/5-NLP/13-TextRep/TextRepresentationTF.ipynb

647 lines
26 KiB
Plaintext
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Text classification task\n",
"\n",
"In this module, we will begin with a straightforward text classification task using the **[AG_NEWS](http://www.di.unipi.it/~gulli/AG_corpus_of_news_articles.html)** dataset: the goal is to classify news headlines into one of four categories: World, Sports, Business, and Sci/Tech.\n",
"\n",
"## The Dataset\n",
"\n",
"To load the dataset, we will utilize the **[TensorFlow Datasets](https://www.tensorflow.org/datasets)** API.\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import tensorflow as tf\n",
"from tensorflow import keras\n",
"import tensorflow_datasets as tfds\n",
"\n",
"# In this tutorial, we will be training a lot of models. In order to use GPU memory cautiously,\n",
"# we will set tensorflow option to grow GPU memory allocation when required.\n",
"physical_devices = tf.config.list_physical_devices('GPU') \n",
"if len(physical_devices)>0:\n",
" tf.config.experimental.set_memory_growth(physical_devices[0], True)\n",
"\n",
"dataset = tfds.load('ag_news_subset')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can now access the training and test portions of the dataset by using `dataset['train']` and `dataset['test']` respectively:\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Length of train dataset = 120000\n",
"Length of test dataset = 7600\n"
]
}
],
"source": [
"ds_train = dataset['train']\n",
"ds_test = dataset['test']\n",
"\n",
"print(f\"Length of train dataset = {len(ds_train)}\")\n",
"print(f\"Length of test dataset = {len(ds_test)}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's print out the first 10 new headlines from our dataset:\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"3 (Sci/Tech) -> b'AMD Debuts Dual-Core Opteron Processor' b'AMD #39;s new dual-core Opteron chip is designed mainly for corporate computing applications, including databases, Web services, and financial transactions.'\n",
"1 (Sports) -> b\"Wood's Suspension Upheld (Reuters)\" b'Reuters - Major League Baseball\\\\Monday announced a decision on the appeal filed by Chicago Cubs\\\\pitcher Kerry Wood regarding a suspension stemming from an\\\\incident earlier this season.'\n",
"2 (Business) -> b'Bush reform may have blue states seeing red' b'President Bush #39;s quot;revenue-neutral quot; tax reform needs losers to balance its winners, and people claiming the federal deduction for state and local taxes may be in administration planners #39; sights, news reports say.'\n",
"3 (Sci/Tech) -> b\"'Halt science decline in schools'\" b'Britain will run out of leading scientists unless science education is improved, says Professor Colin Pillinger.'\n",
"1 (Sports) -> b'Gerrard leaves practice' b'London, England (Sports Network) - England midfielder Steven Gerrard injured his groin late in Thursday #39;s training session, but is hopeful he will be ready for Saturday #39;s World Cup qualifier against Austria.'\n"
]
}
],
"source": [
"classes = ['World', 'Sports', 'Business', 'Sci/Tech']\n",
"\n",
"for i,x in zip(range(5),ds_train):\n",
" print(f\"{x['label']} ({classes[x['label']]}) -> {x['title']} {x['description']}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Text vectorization\n",
"\n",
"Now we need to transform text into **numbers** that can be represented as tensors. If we want to represent text at the word level, we need to do two things:\n",
"\n",
"* Use a **tokenizer** to break the text into **tokens**.\n",
"* Create a **vocabulary** from those tokens.\n",
"\n",
"### Limiting vocabulary size\n",
"\n",
"In the AG News dataset example, the vocabulary size is quite large, exceeding 100k words. Generally, we don't need words that appear infrequently in the text — only a few sentences will contain them, and the model won't learn much from them. Therefore, it makes sense to reduce the vocabulary size to a smaller number by passing an argument to the vectorizer constructor:\n",
"\n",
"Both of these steps can be managed using the **TextVectorization** layer. Let's create the vectorizer object and then use the `adapt` method to process all the text and build a vocabulary:\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"vocab_size = 50000\n",
"vectorizer = keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size)\n",
"vectorizer.adapt(ds_train.take(500).map(lambda x: x['title']+' '+x['description']))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> **Note** that we are using only a subset of the entire dataset to build a vocabulary. This is done to speed up execution time and avoid keeping you waiting. However, this approach carries the risk that some words from the full dataset may not be included in the vocabulary and will be ignored during training. Therefore, using the full vocabulary size and processing the entire dataset during `adapt` could improve the final accuracy, but not by a significant margin.\n",
"\n",
"Now we can access the actual vocabulary:\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['', '[UNK]', 'the', 'to', 'a', 'in', 'of', 'and', 'on', 'for']\n",
"Length of vocabulary: 5335\n"
]
}
],
"source": [
"vocab = vectorizer.get_vocabulary()\n",
"vocab_size = len(vocab)\n",
"print(vocab[:10])\n",
"print(f\"Length of vocabulary: {vocab_size}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using the vectorizer, we can easily encode any text into a set of numbers:\n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<tf.Tensor: shape=(7,), dtype=int64, numpy=array([ 112, 3695, 3, 304, 11, 1041, 1], dtype=int64)>"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"vectorizer('I love to play with my words')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Bag-of-words text representation\n",
"\n",
"Since words carry meaning, sometimes we can infer the meaning of a text simply by analyzing the individual words, without considering their order in the sentence. For instance, when categorizing news articles, words like *weather* and *snow* are likely to suggest *weather forecast*, whereas words like *stocks* and *dollar* might point to *financial news*.\n",
"\n",
"**Bag-of-words** (BoW) vector representation is the simplest and most intuitive traditional vector representation. Each word is assigned to a specific index in the vector, and the corresponding vector element indicates the frequency of that word in a given document.\n",
"\n",
"![Image showing how a bag of words vector representation is represented in memory.](../../../../../translated_images/en/bag-of-words-example.606fc1738f1d7ba98a9d693e3bcd706c6e83fa7bf8221e6e90d1a206d82f2ea4.png) \n",
"\n",
"> **Note**: You can also think of BoW as the sum of all one-hot-encoded vectors for the individual words in the text.\n",
"\n",
"Below is an example of how to create a bag-of-words representation using the Scikit Learn Python library:\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[1, 1, 0, 2, 0, 0, 0, 0, 0]], dtype=int64)"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from sklearn.feature_extraction.text import CountVectorizer\n",
"sc_vectorizer = CountVectorizer()\n",
"corpus = [\n",
" 'I like hot dogs.',\n",
" 'The dog ran fast.',\n",
" 'Its hot outside.',\n",
" ]\n",
"sc_vectorizer.fit_transform(corpus)\n",
"sc_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can also use the Keras vectorizer that we defined above, converting each word number into a one-hot encoding and adding all those vectors up:\n"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([0., 5., 0., ..., 0., 0., 0.], dtype=float32)"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def to_bow(text):\n",
" return tf.reduce_sum(tf.one_hot(vectorizer(text),vocab_size),axis=0)\n",
"\n",
"to_bow('My dog likes hot dogs on a hot day.').numpy()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> **Note**: You might notice that the result is different from the previous example. This is because, in the Keras example, the vector length corresponds to the size of the vocabulary, which was created from the entire AG News dataset. In contrast, in the Scikit Learn example, we built the vocabulary dynamically from the sample text.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Training the BoW classifier\n",
"\n",
"Now that we've learned how to create the bag-of-words representation for our text, let's train a classifier that utilizes it. First, we need to transform our dataset into a bag-of-words format. This can be done using the `map` function as follows:\n"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"batch_size = 128\n",
"\n",
"ds_train_bow = ds_train.map(lambda x: (to_bow(x['title']+x['description']),x['label'])).batch(batch_size)\n",
"ds_test_bow = ds_test.map(lambda x: (to_bow(x['title']+x['description']),x['label'])).batch(batch_size)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's define a simple classifier neural network that contains one linear layer. The input size is `vocab_size`, and the output size corresponds to the number of classes (4). Because we're solving a classification task, the final activation function is **softmax**:\n"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"938/938 [==============================] - 66s 70ms/step - loss: 0.6144 - acc: 0.8427 - val_loss: 0.4416 - val_acc: 0.8697\n"
]
},
{
"data": {
"text/plain": [
"<keras.callbacks.History at 0x20c70a947f0>"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model = keras.models.Sequential([\n",
" keras.layers.Dense(4,activation='softmax',input_shape=(vocab_size,))\n",
"])\n",
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
"model.fit(ds_train_bow,validation_data=ds_test_bow)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Since we have 4 classes, achieving an accuracy above 80% is considered a good result.\n",
"\n",
"## Training a classifier as a single network\n",
"\n",
"Because the vectorizer is also a Keras layer, we can define a network that incorporates it and train it end-to-end. This approach eliminates the need to vectorize the dataset using `map`; instead, we can directly feed the original dataset into the network's input.\n",
"\n",
"> **Note**: We would still need to apply mapping to our dataset to transform fields from dictionaries (like `title`, `description`, and `label`) into tuples. However, when loading data from disk, we can structure the dataset correctly from the start.\n"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Model: \"model\"\n",
"_________________________________________________________________\n",
" Layer (type) Output Shape Param # \n",
"=================================================================\n",
" input_1 (InputLayer) [(None, 1)] 0 \n",
" \n",
" text_vectorization (TextVec (None, None) 0 \n",
" torization) \n",
" \n",
" tf.one_hot (TFOpLambda) (None, None, 5335) 0 \n",
" \n",
" tf.math.reduce_sum (TFOpLam (None, 5335) 0 \n",
" bda) \n",
" \n",
" dense_2 (Dense) (None, 4) 21344 \n",
" \n",
"=================================================================\n",
"Total params: 21,344\n",
"Trainable params: 21,344\n",
"Non-trainable params: 0\n",
"_________________________________________________________________\n",
"938/938 [==============================] - 73s 77ms/step - loss: 0.6057 - acc: 0.8414 - val_loss: 0.4202 - val_acc: 0.8736\n"
]
},
{
"data": {
"text/plain": [
"<keras.callbacks.History at 0x20c721521f0>"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def extract_text(x):\n",
" return x['title']+' '+x['description']\n",
"\n",
"def tupelize(x):\n",
" return (extract_text(x),x['label'])\n",
"\n",
"inp = keras.Input(shape=(1,),dtype=tf.string)\n",
"x = vectorizer(inp)\n",
"x = tf.reduce_sum(tf.one_hot(x,vocab_size),axis=1)\n",
"out = keras.layers.Dense(4,activation='softmax')(x)\n",
"model = keras.models.Model(inp,out)\n",
"model.summary()\n",
"\n",
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
"model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Bigrams, trigrams, and n-grams\n",
"\n",
"One drawback of the bag-of-words approach is that some words belong to multi-word expressions. For instance, the term 'hot dog' has a completely different meaning compared to the individual words 'hot' and 'dog' in other contexts. If we always represent 'hot' and 'dog' using the same vectors, it could lead to confusion in our model.\n",
"\n",
"To solve this issue, **n-gram representations** are often employed in document classification methods, where the frequency of each single word, pair of words, or triplet of words becomes a valuable feature for training classifiers. In bigram representations, for example, we include all word pairs in the vocabulary, in addition to the original single words.\n",
"\n",
"Here’s an example of how to create a bigram bag-of-words representation using Scikit Learn:\n"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Vocabulary:\n",
" {'i': 7, 'like': 11, 'hot': 4, 'dogs': 2, 'i like': 8, 'like hot': 12, 'hot dogs': 5, 'the': 16, 'dog': 0, 'ran': 14, 'fast': 3, 'the dog': 17, 'dog ran': 1, 'ran fast': 15, 'its': 9, 'outside': 13, 'its hot': 10, 'hot outside': 6}\n"
]
},
{
"data": {
"text/plain": [
"array([[1, 0, 1, 0, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],\n",
" dtype=int64)"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"bigram_vectorizer = CountVectorizer(ngram_range=(1, 2), token_pattern=r'\\b\\w+\\b', min_df=1)\n",
"corpus = [\n",
" 'I like hot dogs.',\n",
" 'The dog ran fast.',\n",
" 'Its hot outside.',\n",
" ]\n",
"bigram_vectorizer.fit_transform(corpus)\n",
"print(\"Vocabulary:\\n\",bigram_vectorizer.vocabulary_)\n",
"bigram_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The main disadvantage of the n-gram approach is that the size of the vocabulary grows extremely quickly. In practice, we need to combine the n-gram representation with a dimensionality reduction technique, such as *embeddings*, which we will cover in the next unit.\n",
"\n",
"To apply an n-gram representation to our **AG News** dataset, we need to pass the `ngrams` parameter to the `TextVectorization` constructor. The size of a bigram vocabulary is **significantly larger**—in our case, it exceeds 1.3 million tokens! Therefore, it makes sense to limit the number of bigram tokens to a reasonable amount.\n",
"\n",
"We could use the same code as before to train the classifier, but this approach would be very memory-inefficient. In the next unit, we will train the bigram classifier using embeddings. In the meantime, you can experiment with training a bigram classifier in this notebook and see if you can achieve higher accuracy.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Automatically calculating BoW Vectors\n",
"\n",
"In the example above, we manually calculated BoW vectors by summing the one-hot encodings of individual words. However, the latest version of TensorFlow allows us to automatically calculate BoW vectors by setting the `output_mode='count'` parameter in the vectorizer constructor. This simplifies the process of defining and training our model significantly:\n"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Training vectorizer\n",
"938/938 [==============================] - 7s 7ms/step - loss: 0.5929 - acc: 0.8486 - val_loss: 0.4168 - val_acc: 0.8772\n"
]
},
{
"data": {
"text/plain": [
"<keras.callbacks.History at 0x20c725217c0>"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model = keras.models.Sequential([\n",
" keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size,output_mode='count'),\n",
" keras.layers.Dense(4,input_shape=(vocab_size,), activation='softmax')\n",
"])\n",
"print(\"Training vectorizer\")\n",
"model.layers[0].adapt(ds_train.take(500).map(extract_text))\n",
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
"model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Term frequency - inverse document frequency (TF-IDF)\n",
"\n",
"In the BoW representation, word occurrences are weighted using the same method regardless of the word itself. However, it's obvious that frequent words like *a* and *in* are far less significant for classification than specialized terms. In most NLP tasks, some words carry more importance than others.\n",
"\n",
"**TF-IDF** stands for **term frequency - inverse document frequency**. It is a variation of the bag-of-words approach, where instead of a binary 0/1 value indicating whether a word appears in a document, a floating-point value is used, which reflects the frequency of the word's occurrence in the corpus.\n",
"\n",
"More formally, the weight $w_{ij}$ of a word $i$ in document $j$ is defined as:\n",
"$$\n",
"w_{ij} = tf_{ij}\\times\\log({N\\over df_i})\n",
"$$\n",
"where\n",
"* $tf_{ij}$ is the number of times $i$ appears in $j$, i.e., the BoW value we discussed earlier\n",
"* $N$ is the total number of documents in the collection\n",
"* $df_i$ is the number of documents in the collection that contain the word $i$\n",
"\n",
"The TF-IDF value $w_{ij}$ increases in proportion to how often a word appears in a document but is offset by the number of documents in the corpus that contain the word. This adjustment helps account for the fact that some words are more common than others. For instance, if a word appears in *every* document in the collection, $df_i=N$, and $w_{ij}=0$, meaning those terms would be completely ignored.\n",
"\n",
"You can easily create a TF-IDF vectorization of text using Scikit Learn:\n"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[0.43381609, 0. , 0.43381609, 0. , 0.65985664,\n",
" 0.43381609, 0. , 0. , 0. , 0. ,\n",
" 0. , 0. , 0. , 0. , 0. ,\n",
" 0. ]])"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
"vectorizer = TfidfVectorizer(ngram_range=(1,2))\n",
"vectorizer.fit_transform(corpus)\n",
"vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In Keras, the `TextVectorization` layer can automatically compute TF-IDF frequencies by passing the `output_mode='tf-idf'` parameter. Let's repeat the code we used above to see if using TF-IDF increases accuracy:\n"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Training vectorizer\n",
"938/938 [==============================] - 12s 12ms/step - loss: 0.4197 - acc: 0.8662 - val_loss: 0.3432 - val_acc: 0.8849\n"
]
},
{
"data": {
"text/plain": [
"<keras.callbacks.History at 0x20c729dfd30>"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model = keras.models.Sequential([\n",
" keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size,output_mode='tf-idf'),\n",
" keras.layers.Dense(4,input_shape=(vocab_size,), activation='softmax')\n",
"])\n",
"print(\"Training vectorizer\")\n",
"model.layers[0].adapt(ds_train.take(500).map(extract_text))\n",
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
"model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Conclusion\n",
"\n",
"Although TF-IDF representations assign frequency weights to various words, they cannot capture meaning or order. As the renowned linguist J. R. Firth stated in 1935, \"The complete meaning of a word is always contextual, and no study of meaning apart from context can be taken seriously.\" Later in the course, we will explore how to extract contextual information from text using language modeling.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n---\n\n**Disclaimer**: \nThis document has been translated using the AI translation service [Co-op Translator](https://github.com/Azure/co-op-translator). While we aim for accuracy, please note that automated translations may include errors or inaccuracies. The original document in its native language should be regarded as the authoritative source. For critical information, professional human translation is advised. We are not responsible for any misunderstandings or misinterpretations resulting from the use of this translation.\n"
]
}
],
"metadata": {
"interpreter": {
"hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5"
},
"kernel_info": {
"name": "conda-env-py37_tensorflow-py"
},
"kernelspec": {
"display_name": "py37_tensorflow",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.12"
},
"nteract": {
"version": "nteract-front-end@1.0.0"
},
"coopTranslator": {
"original_hash": "19b43951d55b377a76209c24c1f017e4",
"translation_date": "2025-08-31T18:36:07+00:00",
"source_file": "lessons/5-NLP/13-TextRep/TextRepresentationTF.ipynb",
"language_code": "en"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
Reference in New Issue View Git Blame Copy Permalink