Merge pull request #516 from microsoft/update-translations
🌐 Update translations via Co-op Translator
This commit is contained in:
Lee Stott
GitHub
commit
195748b387
|
|
@ -1,8 +1,8 @@
|
|||
<!--
|
||||
CO_OP_TRANSLATOR_METADATA:
|
||||
{
|
||||
"original_hash": "f3a6b0ddf7e6e3f33b2a543baf086dc9",
|
||||
"translation_date": "2025-08-24T09:27:46+00:00",
|
||||
"original_hash": "07191303b7ea2aff1d47e2b0fe4bb862",
|
||||
"translation_date": "2025-08-31T15:36:17+00:00",
|
||||
"source_file": "README.md",
|
||||
"language_code": "de"
|
||||
}
|
||||
|
|
@ -23,12 +23,23 @@ CO_OP_TRANSLATOR_METADATA:
|
|||
|
||||
# Künstliche Intelligenz für Anfänger – Ein Curriculum
|
||||
|
||||
| ](./lessons/sketchnotes/ai-overview.png)|
|
||||
||
|
||||
|:---:|
|
||||
| KI für Anfänger – _Sketchnote von [@girlie_mac](https://twitter.com/girlie_mac)_ |
|
||||
|
||||
Entdecken Sie die Welt der **Künstlichen Intelligenz** (KI) mit unserem 12-wöchigen Curriculum, das aus 24 Lektionen besteht! Es enthält praktische Lektionen, Quizze und Labore. Das Curriculum ist anfängerfreundlich und behandelt Tools wie TensorFlow und PyTorch sowie ethische Aspekte der KI.
|
||||
|
||||
### 🌐 Mehrsprachige Unterstützung
|
||||
|
||||
#### Unterstützt durch GitHub Action (Automatisiert & Immer aktuell)
|
||||
|
||||
[Französisch](../fr/README.md) | [Spanisch](../es/README.md) | [Deutsch](./README.md) | [Russisch](../ru/README.md) | [Arabisch](../ar/README.md) | [Persisch (Farsi)](../fa/README.md) | [Urdu](../ur/README.md) | [Chinesisch (Vereinfacht)](../zh/README.md) | [Chinesisch (Traditionell, Macau)](../mo/README.md) | [Chinesisch (Traditionell, Hongkong)](../hk/README.md) | [Chinesisch (Traditionell, Taiwan)](../tw/README.md) | [Japanisch](../ja/README.md) | [Koreanisch](../ko/README.md) | [Hindi](../hi/README.md) | [Bengalisch](../bn/README.md) | [Marathi](../mr/README.md) | [Nepalesisch](../ne/README.md) | [Punjabi (Gurmukhi)](../pa/README.md) | [Portugiesisch (Portugal)](../pt/README.md) | [Portugiesisch (Brasilien)](../br/README.md) | [Italienisch](../it/README.md) | [Polnisch](../pl/README.md) | [Türkisch](../tr/README.md) | [Griechisch](../el/README.md) | [Thailändisch](../th/README.md) | [Schwedisch](../sv/README.md) | [Dänisch](../da/README.md) | [Norwegisch](../no/README.md) | [Finnisch](../fi/README.md) | [Niederländisch](../nl/README.md) | [Hebräisch](../he/README.md) | [Vietnamesisch](../vi/README.md) | [Indonesisch](../id/README.md) | [Malaiisch](../ms/README.md) | [Tagalog (Filipino)](../tl/README.md) | [Swahili](../sw/README.md) | [Ungarisch](../hu/README.md) | [Tschechisch](../cs/README.md) | [Slowakisch](../sk/README.md) | [Rumänisch](../ro/README.md) | [Bulgarisch](../bg/README.md) | [Serbisch (Kyrillisch)](../sr/README.md) | [Kroatisch](../hr/README.md) | [Slowenisch](../sl/README.md) | [Ukrainisch](../uk/README.md) | [Birmanisch (Myanmar)](../my/README.md)
|
||||
|
||||
**Falls Sie zusätzliche Übersetzungen wünschen, finden Sie die unterstützten Sprachen [hier](https://github.com/Azure/co-op-translator/blob/main/getting_started/supported-languages.md)**
|
||||
|
||||
## Treten Sie der Community bei
|
||||
[](https://discord.gg/kzRShWzttr)
|
||||
|
||||
## Was Sie lernen werden
|
||||
|
||||
**[Mindmap des Kurses](http://soshnikov.com/courses/ai-for-beginners/mindmap.html)**
|
||||
|
|
@ -36,7 +47,7 @@ Entdecken Sie die Welt der **Künstlichen Intelligenz** (KI) mit unserem 12-wöc
|
|||
In diesem Curriculum lernen Sie:
|
||||
|
||||
* Verschiedene Ansätze der Künstlichen Intelligenz, einschließlich des "klassischen" symbolischen Ansatzes mit **Wissensrepräsentation** und Schlussfolgerung ([GOFAI](https://en.wikipedia.org/wiki/Symbolic_artificial_intelligence)).
|
||||
* **Neuronale Netze** und **Deep Learning**, die im Zentrum moderner KI stehen. Wir veranschaulichen die Konzepte hinter diesen wichtigen Themen mit Code in zwei der beliebtesten Frameworks – [TensorFlow](http://Tensorflow.org) und [PyTorch](http://pytorch.org).
|
||||
* **Neuronale Netze** und **Deep Learning**, die das Herzstück moderner KI bilden. Wir veranschaulichen die Konzepte hinter diesen wichtigen Themen mit Code in zwei der beliebtesten Frameworks – [TensorFlow](http://Tensorflow.org) und [PyTorch](http://pytorch.org).
|
||||
* **Neuronale Architekturen** für die Arbeit mit Bildern und Text. Wir behandeln aktuelle Modelle, könnten aber bei den neuesten Entwicklungen etwas hinterherhinken.
|
||||
* Weniger populäre KI-Ansätze wie **Genetische Algorithmen** und **Multi-Agenten-Systeme**.
|
||||
|
||||
|
|
@ -44,82 +55,82 @@ Was wir in diesem Curriculum nicht behandeln:
|
|||
|
||||
> [Finden Sie alle zusätzlichen Ressourcen für diesen Kurs in unserer Microsoft Learn-Sammlung](https://learn.microsoft.com/en-us/collections/7w28iy2xrqzdj0?WT.mc_id=academic-77998-bethanycheum)
|
||||
|
||||
* Geschäftsanwendungen von **KI im Business**. Ziehen Sie in Betracht, den Lernpfad [Einführung in KI für Geschäftsanwender](https://docs.microsoft.com/learn/paths/introduction-ai-for-business-users/?WT.mc_id=academic-77998-bethanycheum) auf Microsoft Learn oder die [AI Business School](https://www.microsoft.com/ai/ai-business-school/?WT.mc_id=academic-77998-bethanycheum) zu absolvieren, die in Zusammenarbeit mit [INSEAD](https://www.insead.edu/) entwickelt wurde.
|
||||
* **Klassisches maschinelles Lernen**, das in unserem [Maschinelles Lernen für Anfänger Curriculum](http://github.com/Microsoft/ML-for-Beginners) gut beschrieben ist.
|
||||
* Praktische KI-Anwendungen, die mit **[Cognitive Services](https://azure.microsoft.com/services/cognitive-services/?WT.mc_id=academic-77998-bethanycheum)** erstellt wurden. Dafür empfehlen wir, mit den Microsoft Learn-Modulen für [Vision](https://docs.microsoft.com/learn/paths/create-computer-vision-solutions-azure-cognitive-services/?WT.mc_id=academic-77998-bethanycheum), [Natural Language Processing](https://docs.microsoft.com/learn/paths/explore-natural-language-processing/?WT.mc_id=academic-77998-bethanycheum), **[Generative KI mit Azure OpenAI Service](https://learn.microsoft.com/en-us/training/paths/develop-ai-solutions-azure-openai/?WT.mc_id=academic-77998-bethanycheum)** und anderen zu beginnen.
|
||||
* Spezifische ML-**Cloud-Frameworks** wie [Azure Machine Learning](https://azure.microsoft.com/services/machine-learning/?WT.mc_id=academic-77998-bethanycheum), [Microsoft Fabric](https://learn.microsoft.com/en-us/training/paths/get-started-fabric/?WT.mc_id=academic-77998-bethanycheum) oder [Azure Databricks](https://docs.microsoft.com/learn/paths/data-engineer-azure-databricks?WT.mc_id=academic-77998-bethanycheum). Ziehen Sie die Lernpfade [Erstellen und Betreiben von maschinellen Lernlösungen mit Azure Machine Learning](https://docs.microsoft.com/learn/paths/build-ai-solutions-with-azure-ml-service/?WT.mc_id=academic-77998-bethanycheum) und [Erstellen und Betreiben von maschinellen Lernlösungen mit Azure Databricks](https://docs.microsoft.com/learn/paths/build-operate-machine-learning-solutions-azure-databricks/?WT.mc_id=academic-77998-bethanycheum) in Betracht.
|
||||
* **Konversationelle KI** und **Chatbots**. Es gibt einen separaten Lernpfad [Erstellen von konversationellen KI-Lösungen](https://docs.microsoft.com/learn/paths/create-conversational-ai-solutions/?WT.mc_id=academic-77998-bethanycheum), und Sie können auch [diesen Blogbeitrag](https://soshnikov.com/azure/hello-bot-conversational-ai-on-microsoft-platform/) für weitere Details konsultieren.
|
||||
* **Tiefgehende Mathematik** hinter Deep Learning. Dafür empfehlen wir [Deep Learning](https://www.amazon.com/Deep-Learning-Adaptive-Computation-Machine/dp/0262035618) von Ian Goodfellow, Yoshua Bengio und Aaron Courville, das auch online verfügbar ist unter [https://www.deeplearningbook.org/](https://www.deeplearningbook.org/).
|
||||
* Geschäftsanwendungen von **KI im Business**. Ziehen Sie den Lernpfad [Einführung in KI für Geschäftsanwender](https://docs.microsoft.com/learn/paths/introduction-ai-for-business-users/?WT.mc_id=academic-77998-bethanycheum) auf Microsoft Learn oder die [AI Business School](https://www.microsoft.com/ai/ai-business-school/?WT.mc_id=academic-77998-bethanycheum) in Betracht, die in Zusammenarbeit mit [INSEAD](https://www.insead.edu/) entwickelt wurde.
|
||||
* **Klassisches Machine Learning**, das in unserem [Machine Learning für Anfänger Curriculum](http://github.com/Microsoft/ML-for-Beginners) gut beschrieben ist.
|
||||
* Praktische KI-Anwendungen, die mit **[Cognitive Services](https://azure.microsoft.com/services/cognitive-services/?WT.mc_id=academic-77998-bethanycheum)** erstellt wurden. Dafür empfehlen wir, mit den Modulen auf Microsoft Learn für [Vision](https://docs.microsoft.com/learn/paths/create-computer-vision-solutions-azure-cognitive-services/?WT.mc_id=academic-77998-bethanycheum), [Natural Language Processing](https://docs.microsoft.com/learn/paths/explore-natural-language-processing/?WT.mc_id=academic-77998-bethanycheum), **[Generative AI mit Azure OpenAI Service](https://learn.microsoft.com/en-us/training/paths/develop-ai-solutions-azure-openai/?WT.mc_id=academic-77998-bethanycheum)** und anderen zu beginnen.
|
||||
* Spezifische ML-**Cloud-Frameworks** wie [Azure Machine Learning](https://azure.microsoft.com/services/machine-learning/?WT.mc_id=academic-77998-bethanycheum), [Microsoft Fabric](https://learn.microsoft.com/en-us/training/paths/get-started-fabric/?WT.mc_id=academic-77998-bethanycheum) oder [Azure Databricks](https://docs.microsoft.com/learn/paths/data-engineer-azure-databricks?WT.mc_id=academic-77998-bethanycheum). Ziehen Sie die Lernpfade [Erstellen und Betreiben von Machine-Learning-Lösungen mit Azure Machine Learning](https://docs.microsoft.com/learn/paths/build-ai-solutions-with-azure-ml-service/?WT.mc_id=academic-77998-bethanycheum) und [Erstellen und Betreiben von Machine-Learning-Lösungen mit Azure Databricks](https://docs.microsoft.com/learn/paths/build-operate-machine-learning-solutions-azure-databricks/?WT.mc_id=academic-77998-bethanycheum) in Betracht.
|
||||
* **Konversationelle KI** und **Chatbots**. Es gibt einen separaten Lernpfad [Erstellen von konversationellen KI-Lösungen](https://docs.microsoft.com/learn/paths/create-conversational-ai-solutions/?WT.mc_id=academic-77998-bethanycheum), und Sie können auch [diesen Blogbeitrag](https://soshnikov.com/azure/hello-bot-conversational-ai-on-microsoft-platform/) für weitere Details heranziehen.
|
||||
* **Tiefgehende Mathematik** hinter Deep Learning. Dafür empfehlen wir [Deep Learning](https://www.amazon.com/Deep-Learning-Adaptive-Computation-Machine/dp/0262035618) von Ian Goodfellow, Yoshua Bengio und Aaron Courville, das auch online unter [https://www.deeplearningbook.org/](https://www.deeplearningbook.org/) verfügbar ist.
|
||||
|
||||
Für eine sanfte Einführung in _KI in der Cloud_ können Sie den Lernpfad [Erste Schritte mit künstlicher Intelligenz auf Azure](https://docs.microsoft.com/learn/paths/get-started-with-artificial-intelligence-on-azure/?WT.mc_id=academic-77998-bethanycheum) in Betracht ziehen.
|
||||
Für eine sanfte Einführung in _KI in der Cloud_ können Sie den Lernpfad [Einführung in künstliche Intelligenz auf Azure](https://docs.microsoft.com/learn/paths/get-started-with-artificial-intelligence-on-azure/?WT.mc_id=academic-77998-bethanycheum) in Betracht ziehen.
|
||||
|
||||
# Inhalt
|
||||
|
||||
| | Lektion Link | PyTorch/Keras/TensorFlow | Labor |
|
||||
| | Lektion Link | PyTorch/Keras/TensorFlow | Lab |
|
||||
| :-: | :------------------------------------------------------------------------------------------------------------------------------------------: | :---------------------------------------------------------------------------------------------: | ------------------------------------------------------------------------------ |
|
||||
| 0 | [Kurs-Setup](./lessons/0-course-setup/setup.md) | [Einrichtung Ihrer Entwicklungsumgebung](./lessons/0-course-setup/how-to-run.md) | |
|
||||
| I | [**Einführung in KI**](./lessons/1-Intro/README.md) | | |
|
||||
| 01 | [Einführung und Geschichte der KI](./lessons/1-Intro/README.md) | - | - |
|
||||
| II | **Symbolische KI** |
|
||||
| 02 | [Wissensrepräsentation und Expertensysteme](./lessons/2-Symbolic/README.md) | [Expertensysteme](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/2-Symbolic/Animals.ipynb) / [Ontologie](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/2-Symbolic/FamilyOntology.ipynb) /[Konzeptgraph](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/2-Symbolic/MSConceptGraph.ipynb) | |
|
||||
| III | [**Einführung in neuronale Netze**](./lessons/3-NeuralNetworks/README.md) |||
|
||||
| 03 | [Perzeptron](./lessons/3-NeuralNetworks/03-Perceptron/README.md) | [Notebook](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/3-NeuralNetworks/03-Perceptron/Perceptron.ipynb) | [Labor](./lessons/3-NeuralNetworks/03-Perceptron/lab/README.md) |
|
||||
| 04 | [Mehrschichtiges Perzeptron und Erstellung eines eigenen Frameworks](./lessons/3-NeuralNetworks/04-OwnFramework/README.md) | [Notebook](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/3-NeuralNetworks/04-OwnFramework/OwnFramework.ipynb) | [Labor](./lessons/3-NeuralNetworks/04-OwnFramework/lab/README.md) |
|
||||
| 05 | [Einführung in Frameworks (PyTorch/TensorFlow) und Overfitting](./lessons/3-NeuralNetworks/05-Frameworks/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/3-NeuralNetworks/05-Frameworks/IntroPyTorch.ipynb) / [Keras](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/3-NeuralNetworks/05-Frameworks/IntroKeras.ipynb) / [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/3-NeuralNetworks/05-Frameworks/IntroKerasTF.ipynb) | [Labor](./lessons/3-NeuralNetworks/05-Frameworks/lab/README.md) |
|
||||
| IV | [**Computer Vision**](./lessons/4-ComputerVision/README.md) | [PyTorch](https://docs.microsoft.com/learn/modules/intro-computer-vision-pytorch/?WT.mc_id=academic-77998-cacaste) / [TensorFlow](https://docs.microsoft.com/learn/modules/intro-computer-vision-TensorFlow/?WT.mc_id=academic-77998-cacaste)| [Erkunden Sie Computer Vision auf Microsoft Azure](https://learn.microsoft.com/en-us/collections/7w28iy2xrqzdj0?WT.mc_id=academic-77998-bethanycheum) |
|
||||
| 06 | [Einführung in Computer Vision. OpenCV](./lessons/4-ComputerVision/06-IntroCV/README.md) | [Notebook](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/06-IntroCV/OpenCV.ipynb) | [Labor](./lessons/4-ComputerVision/06-IntroCV/lab/README.md) |
|
||||
| 07 | [Convolutional Neural Networks](./lessons/4-ComputerVision/07-ConvNets/README.md) & [CNN-Architekturen](./lessons/4-ComputerVision/07-ConvNets/CNN_Architectures.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/07-ConvNets/ConvNetsPyTorch.ipynb) /[TensorFlow](https://microsoft.github.io/AI-For-Beginners/lessons/4-ComputerVision/07-ConvNets/ConvNetsTF.ipynb) | [Labor](./lessons/4-ComputerVision/07-ConvNets/lab/README.md) |
|
||||
| 08 | [Vortrainierte Netzwerke und Transferlernen](./lessons/4-ComputerVision/08-TransferLearning/README.md) und [Trainingstricks](./lessons/4-ComputerVision/08-TransferLearning/TrainingTricks.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/08-TransferLearning/TransferLearningPyTorch.ipynb) / [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/3-NeuralNetworks/05-Frameworks/IntroKerasTF.ipynb) | [Lab](./lessons/4-ComputerVision/08-TransferLearning/lab/README.md) |
|
||||
| 09 | [Autoencoder und VAEs](./lessons/4-ComputerVision/09-Autoencoders/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/09-Autoencoders/AutoEncodersPyTorch.ipynb) / [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/09-Autoencoders/AutoencodersTF.ipynb) | |
|
||||
| 10 | [Generative Adversarial Networks & Künstlerischer Stiltransfer](./lessons/4-ComputerVision/10-GANs/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/10-GANs/GANPyTorch.ipynb) / [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/10-GANs/GANTF.ipynb) | |
|
||||
| 11 | [Objekterkennung](./lessons/4-ComputerVision/11-ObjectDetection/README.md) | [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/11-ObjectDetection/ObjectDetection.ipynb) | [Lab](./lessons/4-ComputerVision/11-ObjectDetection/lab/README.md) |
|
||||
| 12 | [Semantische Segmentierung. U-Net](./lessons/4-ComputerVision/12-Segmentation/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/12-Segmentation/SemanticSegmentationPytorch.ipynb) / [TensorFlow](../../(https:/github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/12-Segmentation/SemanticSegmentationTF.ipynb)) | |
|
||||
| V | [**Verarbeitung natürlicher Sprache**](./lessons/5-NLP/README.md) | [PyTorch](https://docs.microsoft.com/learn/modules/intro-natural-language-processing-pytorch/?WT.mc_id=academic-77998-cacaste) /[TensorFlow](https://docs.microsoft.com/learn/modules/intro-natural-language-processing-TensorFlow/?WT.mc_id=academic-77998-cacaste) | [Erkunde die Verarbeitung natürlicher Sprache auf Microsoft Azure](https://learn.microsoft.com/en-us/collections/7w28iy2xrqzdj0?WT.mc_id=academic-77998-bethanycheum)|
|
||||
| 13 | [Textrepräsentation. BoW/TF-IDF](./lessons/5-NLP/13-TextRep/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/13-TextRep/TextRepresentationPyTorch.ipynb) / [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/13-TextRep/TextRepresentationTF.ipynb) | |
|
||||
| 14 | [Semantische Wort-Embeddings. Word2Vec und GloVe](./lessons/5-NLP/14-Embeddings/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb) / [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/14-Embeddings/EmbeddingsTF.ipynb) | |
|
||||
| 15 | [Sprachmodellierung. Eigene Embeddings trainieren](./lessons/5-NLP/15-LanguageModeling/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/15-LanguageModeling/CBoW-PyTorch.ipynb) / [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/15-LanguageModeling/CBoW-TF.ipynb) | [Lab](./lessons/5-NLP/15-LanguageModeling/lab/README.md) |
|
||||
| 16 | [Rekurrente Neuronale Netze](./lessons/5-NLP/16-RNN/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/16-RNN/RNNPyTorch.ipynb) / [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/16-RNN/RNNTF.ipynb) | |
|
||||
| 17 | [Generative Rekurrente Netze](./lessons/5-NLP/17-GenerativeNetworks/README.md) | [PyTorch](https://microsoft.github.io/AI-For-Beginners/lessons/5-NLP/17-GenerativeNetworks/GenerativePyTorch.md) / [TensorFlow](https://microsoft.github.io/AI-For-Beginners/lessons/5-NLP/17-GenerativeNetworks/GenerativeTF.md) | [Lab](./lessons/5-NLP/17-GenerativeNetworks/lab/README.md) |
|
||||
| 18 | [Transformers. BERT.](./lessons/5-NLP/18-Transformers/READMEtransformers.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/18-Transformers/TransformersPyTorch.ipynb) /[TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/18-Transformers/TransformersTF.ipynb) | |
|
||||
| 19 | [Named Entity Recognition](./lessons/5-NLP/19-NER/README.md) | [TensorFlow](https://microsoft.github.io/AI-For-Beginners/lessons/5-NLP/19-NER/NER-TF.ipynb) | [Lab](./lessons/5-NLP/19-NER/lab/README.md) |
|
||||
| 20 | [Große Sprachmodelle, Prompt-Programmierung und Few-Shot-Aufgaben](./lessons/5-NLP/20-LangModels/READMELargeLang.md) | [PyTorch](https://microsoft.github.io/AI-For-Beginners/lessons/5-NLP/20-LangModels/GPT-PyTorch.ipynb) | |
|
||||
| 02 | [Wissensrepräsentation und Expertensysteme](./lessons/2-Symbolic/README.md) | [Expertensysteme](./lessons/2-Symbolic/Animals.ipynb) / [Ontologie](./lessons/2-Symbolic/FamilyOntology.ipynb) /[Konzeptgraph](./lessons/2-Symbolic/MSConceptGraph.ipynb) | |
|
||||
| III | [**Einführung in Neuronale Netze**](./lessons/3-NeuralNetworks/README.md) |||
|
||||
| 03 | [Perzeptron](./lessons/3-NeuralNetworks/03-Perceptron/README.md) | [Notebook](./lessons/3-NeuralNetworks/03-Perceptron/Perceptron.ipynb) | [Lab](./lessons/3-NeuralNetworks/03-Perceptron/lab/README.md) |
|
||||
| 04 | [Mehrschichtige Perzeptrons und Erstellung eines eigenen Frameworks](./lessons/3-NeuralNetworks/04-OwnFramework/README.md) | [Notebook](./lessons/3-NeuralNetworks/04-OwnFramework/OwnFramework.ipynb) | [Lab](./lessons/3-NeuralNetworks/04-OwnFramework/lab/README.md) |
|
||||
| 05 | [Einführung in Frameworks (PyTorch/TensorFlow) und Overfitting](./lessons/3-NeuralNetworks/05-Frameworks/README.md) | [PyTorch](./lessons/3-NeuralNetworks/05-Frameworks/IntroPyTorch.ipynb) / [Keras](./lessons/3-NeuralNetworks/05-Frameworks/IntroKeras.ipynb) / [TensorFlow](./lessons/3-NeuralNetworks/05-Frameworks/IntroKerasTF.ipynb) | [Lab](./lessons/3-NeuralNetworks/05-Frameworks/lab/README.md) |
|
||||
| IV | [**Computer Vision**](./lessons/4-ComputerVision/README.md) | [PyTorch](https://docs.microsoft.com/learn/modules/intro-computer-vision-pytorch/?WT.mc_id=academic-77998-cacaste) / [TensorFlow](https://docs.microsoft.com/learn/modules/intro-computer-vision-TensorFlow/?WT.mc_id=academic-77998-cacaste)| [Erkunde Computer Vision auf Microsoft Azure](https://learn.microsoft.com/en-us/collections/7w28iy2xrqzdj0?WT.mc_id=academic-77998-bethanycheum) |
|
||||
| 06 | [Einführung in Computer Vision. OpenCV](./lessons/4-ComputerVision/06-IntroCV/README.md) | [Notebook](./lessons/4-ComputerVision/06-IntroCV/OpenCV.ipynb) | [Lab](./lessons/4-ComputerVision/06-IntroCV/lab/README.md) |
|
||||
| 07 | [Convolutional Neural Networks](./lessons/4-ComputerVision/07-ConvNets/README.md) & [CNN-Architekturen](./lessons/4-ComputerVision/07-ConvNets/CNN_Architectures.md) | [PyTorch](./lessons/4-ComputerVision/07-ConvNets/ConvNetsPyTorch.ipynb) /[TensorFlow](./lessons/4-ComputerVision/07-ConvNets/ConvNetsTF.ipynb) | [Lab](./lessons/4-ComputerVision/07-ConvNets/lab/README.md) |
|
||||
| 08 | [Vortrainierte Netzwerke und Transfer Learning](./lessons/4-ComputerVision/08-TransferLearning/README.md) und [Trainings-Tricks](./lessons/4-ComputerVision/08-TransferLearning/TrainingTricks.md) | [PyTorch](./lessons/4-ComputerVision/08-TransferLearning/TransferLearningPyTorch.ipynb) / [TensorFlow](./lessons/3-NeuralNetworks/05-Frameworks/IntroKerasTF.ipynb) | [Lab](./lessons/4-ComputerVision/08-TransferLearning/lab/README.md) |
|
||||
| 09 | [Autoencoder und VAEs](./lessons/4-ComputerVision/09-Autoencoders/README.md) | [PyTorch](./lessons/4-ComputerVision/09-Autoencoders/AutoEncodersPyTorch.ipynb) / [TensorFlow](./lessons/4-ComputerVision/09-Autoencoders/AutoencodersTF.ipynb) | |
|
||||
| 10 | [Generative Adversarial Networks & Künstlerischer Stiltransfer](./lessons/4-ComputerVision/10-GANs/README.md) | [PyTorch](./lessons/4-ComputerVision/10-GANs/GANPyTorch.ipynb) / [TensorFlow](./lessons/4-ComputerVision/10-GANs/GANTF.ipynb) | |
|
||||
| 11 | [Objekterkennung](./lessons/4-ComputerVision/11-ObjectDetection/README.md) | [TensorFlow](./lessons/4-ComputerVision/11-ObjectDetection/ObjectDetection.ipynb) | [Lab](./lessons/4-ComputerVision/11-ObjectDetection/lab/README.md) |
|
||||
| 12 | [Semantische Segmentierung. U-Net](./lessons/4-ComputerVision/12-Segmentation/README.md) | [PyTorch](./lessons/4-ComputerVision/12-Segmentation/SemanticSegmentationPytorch.ipynb) / [TensorFlow](./lessons/4-ComputerVision/12-Segmentation/SemanticSegmentationTF.ipynb) | |
|
||||
| V | [**Natural Language Processing**](./lessons/5-NLP/README.md) | [PyTorch](https://docs.microsoft.com/learn/modules/intro-natural-language-processing-pytorch/?WT.mc_id=academic-77998-cacaste) /[TensorFlow](https://docs.microsoft.com/learn/modules/intro-natural-language-processing-TensorFlow/?WT.mc_id=academic-77998-cacaste) | [Erkunde Natural Language Processing auf Microsoft Azure](https://learn.microsoft.com/en-us/collections/7w28iy2xrqzdj0?WT.mc_id=academic-77998-bethanycheum)|
|
||||
| 13 | [Textrepräsentation. Bow/TF-IDF](./lessons/5-NLP/13-TextRep/README.md) | [PyTorch](./lessons/5-NLP/13-TextRep/TextRepresentationPyTorch.ipynb) / [TensorFlow](./lessons/5-NLP/13-TextRep/TextRepresentationTF.ipynb) | |
|
||||
| 14 | [Semantische Wort-Embeddings. Word2Vec und GloVe](./lessons/5-NLP/14-Embeddings/README.md) | [PyTorch](./lessons/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb) / [TensorFlow](./lessons/5-NLP/14-Embeddings/EmbeddingsTF.ipynb) | |
|
||||
| 15 | [Sprachmodellierung. Eigene Embeddings trainieren](./lessons/5-NLP/15-LanguageModeling/README.md) | [PyTorch](./lessons/5-NLP/15-LanguageModeling/CBoW-PyTorch.ipynb) / [TensorFlow](./lessons/5-NLP/15-LanguageModeling/CBoW-TF.ipynb) | [Lab](./lessons/5-NLP/15-LanguageModeling/lab/README.md) |
|
||||
| 16 | [Recurrent Neural Networks](./lessons/5-NLP/16-RNN/README.md) | [PyTorch](./lessons/5-NLP/16-RNN/RNNPyTorch.ipynb) / [TensorFlow](./lessons/5-NLP/16-RNN/RNNTF.ipynb) | |
|
||||
| 17 | [Generative Recurrent Networks](./lessons/5-NLP/17-GenerativeNetworks/README.md) | [PyTorch](./lessons/5-NLP/17-GenerativeNetworks/GenerativePyTorch.md) / [TensorFlow](./lessons/5-NLP/17-GenerativeNetworks/GenerativeTF.md) | [Lab](./lessons/5-NLP/17-GenerativeNetworks/lab/README.md) |
|
||||
| 18 | [Transformers. BERT.](./lessons/5-NLP/18-Transformers/READMEtransformers.md) | [PyTorch](./lessons/5-NLP/18-Transformers/TransformersPyTorch.ipynb) /[TensorFlow](./lessons/5-NLP/18-Transformers/TransformersTF.ipynb) | |
|
||||
| 19 | [Named Entity Recognition](./lessons/5-NLP/19-NER/README.md) | [TensorFlow](./lessons/5-NLP/19-NER/NER-TF.ipynb) | [Lab](./lessons/5-NLP/19-NER/lab/README.md) |
|
||||
| 20 | [Große Sprachmodelle, Prompt-Programmierung und Few-Shot-Aufgaben](./lessons/5-NLP/20-LangModels/READMELargeLang.md) | [PyTorch](./lessons/5-NLP/20-LangModels/GPT-PyTorch.ipynb) | |
|
||||
| VI | **Andere KI-Techniken** || |
|
||||
| 21 | [Genetische Algorithmen](./lessons/6-Other/21-GeneticAlgorithms/README.md) | [Notebook](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/6-Other/21-GeneticAlgorithms/Genetic.ipynb) | |
|
||||
| 22 | [Tiefes Verstärkungslernen](./lessons/6-Other/22-DeepRL/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/6-Other/22-DeepRL/CartPole-RL-PyTorch.ipynb) /[TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/6-Other/22-DeepRL/CartPole-RL-TF.ipynb) | [Lab](./lessons/6-Other/22-DeepRL/lab/README.md) |
|
||||
| 23 | [Multi-Agenten-Systeme](./lessons/6-Other/23-MultiagentSystems/README.md) | | |
|
||||
| 21 | [Genetische Algorithmen](./lessons/6-Other/21-GeneticAlgorithms/README.md) | [Notebook](./lessons/6-Other/21-GeneticAlgorithms/Genetic.ipynb) | |
|
||||
| 22 | [Deep Reinforcement Learning](./lessons/6-Other/22-DeepRL/README.md) | [PyTorch](./lessons/6-Other/22-DeepRL/CartPole-RL-PyTorch.ipynb) /[TensorFlow](./lessons/6-Other/22-DeepRL/CartPole-RL-TF.ipynb) | [Lab](./lessons/6-Other/22-DeepRL/lab/README.md) |
|
||||
| 23 | [Multi-Agent-Systeme](./lessons/6-Other/23-MultiagentSystems/README.md) | | |
|
||||
| VII | **KI-Ethik** | | |
|
||||
| 24 | [KI-Ethik und Verantwortungsvolle KI](./lessons/7-Ethics/README.md) | [Microsoft Learn: Prinzipien für verantwortungsvolle KI](https://docs.microsoft.com/learn/paths/responsible-ai-business-principles/?WT.mc_id=academic-77998-cacaste) | |
|
||||
| 24 | [KI-Ethik und verantwortungsvolle KI](./lessons/7-Ethics/README.md) | [Microsoft Learn: Prinzipien für verantwortungsvolle KI](https://docs.microsoft.com/learn/paths/responsible-ai-business-principles/?WT.mc_id=academic-77998-cacaste) | |
|
||||
| IX | **Extras** | | |
|
||||
| 25 | [Multi-Modale Netzwerke, CLIP und VQGAN](./lessons/X-Extras/X1-MultiModal/README.md) | [Notebook](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/X-Extras/X1-MultiModal/Clip.ipynb) | |
|
||||
| 25 | [Multi-Modale Netzwerke, CLIP und VQGAN](./lessons/X-Extras/X1-MultiModal/README.md) | [Notebook](./lessons/X-Extras/X1-MultiModal/Clip.ipynb) | |
|
||||
|
||||
## Jede Lektion enthält
|
||||
|
||||
* Vorbereitendes Lesematerial
|
||||
* Ausführbare Jupyter-Notebooks, die oft spezifisch für das Framework (**PyTorch** oder **TensorFlow**) sind. Die ausführbaren Notebooks enthalten auch viele theoretische Inhalte, daher ist es wichtig, mindestens eine Version des Notebooks (entweder PyTorch oder TensorFlow) durchzugehen, um das Thema zu verstehen.
|
||||
* **Labs**, die für einige Themen verfügbar sind und Ihnen die Möglichkeit geben, das Gelernte auf ein spezifisches Problem anzuwenden.
|
||||
* Einige Abschnitte enthalten Links zu [**MS Learn**](https://learn.microsoft.com/en-us/collections/7w28iy2xrqzdj0?WT.mc_id=academic-77998-bethanycheum)-Modulen, die verwandte Themen behandeln.
|
||||
* Ausführbare Jupyter-Notebooks, die oft spezifisch für das Framework (**PyTorch** oder **TensorFlow**) sind. Das ausführbare Notebook enthält auch viele theoretische Inhalte, daher ist es wichtig, mindestens eine Version des Notebooks (entweder PyTorch oder TensorFlow) durchzugehen, um das Thema zu verstehen.
|
||||
* **Labs**, die für einige Themen verfügbar sind und die Möglichkeit bieten, das Gelernte auf ein spezifisches Problem anzuwenden.
|
||||
* Einige Abschnitte enthalten Links zu [**MS Learn**](https://learn.microsoft.com/en-us/collections/7w28iy2xrqzdj0?WT.mc_id=academic-77998-bethanycheum)-Modulen, die verwandte Themen abdecken.
|
||||
|
||||
## Erste Schritte
|
||||
|
||||
- Wir haben eine [Setup-Lektion](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/0-course-setup/setup.md) erstellt, um Ihnen bei der Einrichtung Ihrer Entwicklungsumgebung zu helfen. - Für Lehrkräfte haben wir auch eine [Curricula-Setup-Lektion](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/0-course-setup/for-teachers.md) erstellt!
|
||||
- Wie man [den Code in VSCode oder einem Codespace ausführt](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/0-course-setup/how-to-run.md)
|
||||
- Wir haben eine [Setup-Lektion](./lessons/0-course-setup/setup.md) erstellt, um Ihnen bei der Einrichtung Ihrer Entwicklungsumgebung zu helfen. - Für Lehrkräfte haben wir auch eine [Curricula-Setup-Lektion](./lessons/0-course-setup/for-teachers.md) erstellt!
|
||||
- Wie man [den Code in VSCode oder einem Codespace ausführt](./lessons/0-course-setup/how-to-run.md)
|
||||
|
||||
Folgen Sie diesen Schritten:
|
||||
|
||||
Forken Sie das Repository: Klicken Sie auf die Schaltfläche "Fork" oben rechts auf dieser Seite.
|
||||
Repository forken: Klicken Sie auf die Schaltfläche "Fork" oben rechts auf dieser Seite.
|
||||
|
||||
Klonen Sie das Repository: `git clone https://github.com/microsoft/AI-For-Beginners.git`
|
||||
Repository klonen: `git clone https://github.com/microsoft/AI-For-Beginners.git`
|
||||
|
||||
Vergessen Sie nicht, dieses Repository mit einem Stern (🌟) zu markieren, damit Sie es später leichter finden.
|
||||
Vergessen Sie nicht, dieses Repository mit einem Stern (🌟) zu markieren, um es später leichter zu finden.
|
||||
|
||||
## Treffen Sie andere Lernende
|
||||
|
||||
Treten Sie unserem [offiziellen AI-Discord-Server](https://aka.ms/genai-discord?WT.mc_id=academic-105485-bethanycheum) bei, um andere Lernende dieses Kurses zu treffen und sich zu vernetzen sowie Unterstützung zu erhalten.
|
||||
Treten Sie unserem [offiziellen AI Discord-Server](https://aka.ms/genai-discord?WT.mc_id=academic-105485-bethanycheum) bei, um andere Lernende kennenzulernen, sich zu vernetzen und Unterstützung zu erhalten.
|
||||
|
||||
Wenn Sie Feedback zu Produkten oder Fragen während des Aufbaus haben, besuchen Sie unser [Azure AI Foundry Developer Forum](https://aka.ms/foundry/forum)
|
||||
Wenn Sie Produktfeedback oder Fragen beim Bauen haben, besuchen Sie unser [Azure AI Foundry Developer Forum](https://aka.ms/foundry/forum)
|
||||
|
||||
## Quizze
|
||||
> **Eine Anmerkung zu den Quiz**: Alle Quiz befinden sich im Ordner Quiz-app unter etc\quiz-app. Sie sind mit den Lektionen verknüpft. Die Quiz-App kann lokal ausgeführt oder auf Azure bereitgestellt werden; folgen Sie den Anweisungen im Ordner `quiz-app`. Sie werden nach und nach lokalisiert.
|
||||
## Quiz
|
||||
> **Eine Anmerkung zu den Quiz**: Alle Quiz befinden sich im Ordner Quiz-app unter etc\quiz-app oder [Online Hier](https://ff-quizzes.netlify.app/). Sie sind aus den Lektionen heraus verlinkt. Die Quiz-App kann lokal ausgeführt oder auf Azure bereitgestellt werden; folgen Sie den Anweisungen im Ordner `quiz-app`. Sie werden nach und nach lokalisiert.
|
||||
## Hilfe gesucht
|
||||
|
||||
Hast du Vorschläge oder Fehler in der Rechtschreibung oder im Code gefunden? Erstelle ein Issue oder einen Pull Request.
|
||||
|
|
@ -151,5 +162,7 @@ Unser Team erstellt auch andere Lehrpläne! Schau dir diese an:
|
|||
- [GitHub Copilot meistern für C#/.NET-Entwickler](https://github.com/microsoft/mastering-github-copilot-for-dotnet-csharp-developers)
|
||||
- [Wähle dein eigenes Copilot-Abenteuer](https://github.com/microsoft/CopilotAdventures)
|
||||
|
||||
---
|
||||
|
||||
**Haftungsausschluss**:
|
||||
Dieses Dokument wurde mit dem KI-Übersetzungsdienst [Co-op Translator](https://github.com/Azure/co-op-translator) übersetzt. Obwohl wir uns um Genauigkeit bemühen, weisen wir darauf hin, dass automatisierte Übersetzungen Fehler oder Ungenauigkeiten enthalten können. Das Originaldokument in seiner ursprünglichen Sprache sollte als maßgebliche Quelle betrachtet werden. Für kritische Informationen wird eine professionelle menschliche Übersetzung empfohlen. Wir übernehmen keine Haftung für Missverständnisse oder Fehlinterpretationen, die aus der Nutzung dieser Übersetzung entstehen.
|
||||
Dieses Dokument wurde mit dem KI-Übersetzungsdienst [Co-op Translator](https://github.com/Azure/co-op-translator) übersetzt. Obwohl wir uns um Genauigkeit bemühen, beachten Sie bitte, dass automatisierte Übersetzungen Fehler oder Ungenauigkeiten enthalten können. Das Originaldokument in seiner ursprünglichen Sprache sollte als maßgebliche Quelle betrachtet werden. Für kritische Informationen wird eine professionelle menschliche Übersetzung empfohlen. Wir übernehmen keine Haftung für Missverständnisse oder Fehlinterpretationen, die sich aus der Nutzung dieser Übersetzung ergeben.
|
||||
|
|
@ -0,0 +1,478 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"collapsed": true
|
||||
},
|
||||
"source": [
|
||||
"# Implementierung eines Tier-Expertensystems\n",
|
||||
"\n",
|
||||
"Ein Beispiel aus dem [AI for Beginners Curriculum](http://github.com/microsoft/ai-for-beginners).\n",
|
||||
"\n",
|
||||
"In diesem Beispiel werden wir ein einfaches wissensbasiertes System implementieren, um ein Tier anhand einiger physischer Merkmale zu bestimmen. Das System kann durch den folgenden UND-ODER-Baum dargestellt werden (dies ist ein Teil des gesamten Baums, wir können leicht weitere Regeln hinzufügen):\n",
|
||||
"\n",
|
||||
"\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Unsere eigene Expertensystem-Shell mit Rückwärtsinferenzen\n",
|
||||
"\n",
|
||||
"Versuchen wir, eine einfache Sprache zur Wissensrepräsentation basierend auf Produktionsregeln zu definieren. Wir werden Python-Klassen als Schlüsselwörter verwenden, um Regeln zu definieren. Es gibt im Wesentlichen drei Arten von Klassen:\n",
|
||||
"* `Ask` repräsentiert eine Frage, die dem Benutzer gestellt werden muss. Sie enthält die Menge der möglichen Antworten.\n",
|
||||
"* `If` repräsentiert eine Regel und ist lediglich syntaktischer Zucker, um den Inhalt der Regel zu speichern.\n",
|
||||
"* `AND`/`OR` sind Klassen, die AND/OR-Verzweigungen des Baums darstellen. Sie speichern einfach die Liste der Argumente. Um den Code zu vereinfachen, wird die gesamte Funktionalität in der Elternklasse `Content` definiert.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class Ask():\n",
|
||||
" def __init__(self,choices=['y','n']):\n",
|
||||
" self.choices = choices\n",
|
||||
" def ask(self):\n",
|
||||
" if max([len(x) for x in self.choices])>1:\n",
|
||||
" for i,x in enumerate(self.choices):\n",
|
||||
" print(\"{0}. {1}\".format(i,x),flush=True)\n",
|
||||
" x = int(input())\n",
|
||||
" return self.choices[x]\n",
|
||||
" else:\n",
|
||||
" print(\"/\".join(self.choices),flush=True)\n",
|
||||
" return input()\n",
|
||||
"\n",
|
||||
"class Content():\n",
|
||||
" def __init__(self,x):\n",
|
||||
" self.x=x\n",
|
||||
" \n",
|
||||
"class If(Content):\n",
|
||||
" pass\n",
|
||||
"\n",
|
||||
"class AND(Content):\n",
|
||||
" pass\n",
|
||||
"\n",
|
||||
"class OR(Content):\n",
|
||||
" pass"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"In unserem System würde der Arbeitsspeicher die Liste von **Fakten** als **Attribut-Wert-Paare** enthalten. Die Wissensdatenbank kann als ein großes Wörterbuch definiert werden, das Aktionen (neue Fakten, die in den Arbeitsspeicher eingefügt werden sollen) auf Bedingungen abbildet, ausgedrückt als UND-ODER-Ausdrücke. Außerdem können einige Fakten `gefragt` werden.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"rules = {\n",
|
||||
" 'default': Ask(['y','n']),\n",
|
||||
" 'color' : Ask(['red-brown','black and white','other']),\n",
|
||||
" 'pattern' : Ask(['dark stripes','dark spots']),\n",
|
||||
" 'mammal': If(OR(['hair','gives milk'])),\n",
|
||||
" 'carnivor': If(OR([AND(['sharp teeth','claws','forward-looking eyes']),'eats meat'])),\n",
|
||||
" 'ungulate': If(['mammal',OR(['has hooves','chews cud'])]),\n",
|
||||
" 'bird': If(OR(['feathers',AND(['flies','lies eggs'])])),\n",
|
||||
" 'animal:monkey' : If(['mammal','carnivor','color:red-brown','pattern:dark spots']),\n",
|
||||
" 'animal:tiger' : If(['mammal','carnivor','color:red-brown','pattern:dark stripes']),\n",
|
||||
" 'animal:giraffe' : If(['ungulate','long neck','long legs','pattern:dark spots']),\n",
|
||||
" 'animal:zebra' : If(['ungulate','pattern:dark stripes']),\n",
|
||||
" 'animal:ostrich' : If(['bird','long nech','color:black and white','cannot fly']),\n",
|
||||
" 'animal:pinguin' : If(['bird','swims','color:black and white','cannot fly']),\n",
|
||||
" 'animal:albatross' : If(['bird','flies well'])\n",
|
||||
"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Um die Rückwärtsinferenz durchzuführen, definieren wir die Klasse `Knowledgebase`. Sie wird enthalten:\n",
|
||||
"* Funktionierender `Speicher` - ein Wörterbuch, das Attribute mit Werten verknüpft\n",
|
||||
"* `Regeln` der Wissensbasis im oben definierten Format\n",
|
||||
"\n",
|
||||
"Zwei Hauptmethoden sind:\n",
|
||||
"* `get`, um den Wert eines Attributs zu erhalten und bei Bedarf eine Inferenz durchzuführen. Zum Beispiel würde `get('color')` den Wert eines Farbfeldes abrufen (es wird bei Bedarf nachfragen und den Wert für die spätere Verwendung im Arbeitsspeicher speichern). Wenn wir `get('color:blue')` abfragen, wird nach einer Farbe gefragt und dann ein `y`/`n`-Wert zurückgegeben, abhängig von der Farbe.\n",
|
||||
"* `eval` führt die eigentliche Inferenz durch, d.h. durchläuft den UND/ODER-Baum, bewertet Unterziele usw.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class KnowledgeBase():\n",
|
||||
" def __init__(self,rules):\n",
|
||||
" self.rules = rules\n",
|
||||
" self.memory = {}\n",
|
||||
" \n",
|
||||
" def get(self,name):\n",
|
||||
" if ':' in name:\n",
|
||||
" k,v = name.split(':')\n",
|
||||
" vv = self.get(k)\n",
|
||||
" return 'y' if v==vv else 'n'\n",
|
||||
" if name in self.memory.keys():\n",
|
||||
" return self.memory[name]\n",
|
||||
" for fld in self.rules.keys():\n",
|
||||
" if fld==name or fld.startswith(name+\":\"):\n",
|
||||
" # print(\" + proving {}\".format(fld))\n",
|
||||
" value = 'y' if fld==name else fld.split(':')[1]\n",
|
||||
" res = self.eval(self.rules[fld],field=name)\n",
|
||||
" if res!='y' and res!='n' and value=='y':\n",
|
||||
" self.memory[name] = res\n",
|
||||
" return res\n",
|
||||
" if res=='y':\n",
|
||||
" self.memory[name] = value\n",
|
||||
" return value\n",
|
||||
" # field is not found, using default\n",
|
||||
" res = self.eval(self.rules['default'],field=name)\n",
|
||||
" self.memory[name]=res\n",
|
||||
" return res\n",
|
||||
" \n",
|
||||
" def eval(self,expr,field=None):\n",
|
||||
" # print(\" + eval {}\".format(expr))\n",
|
||||
" if isinstance(expr,Ask):\n",
|
||||
" print(field)\n",
|
||||
" return expr.ask()\n",
|
||||
" elif isinstance(expr,If):\n",
|
||||
" return self.eval(expr.x)\n",
|
||||
" elif isinstance(expr,AND) or isinstance(expr,list):\n",
|
||||
" expr = expr.x if isinstance(expr,AND) else expr\n",
|
||||
" for x in expr:\n",
|
||||
" if self.eval(x)=='n':\n",
|
||||
" return 'n'\n",
|
||||
" return 'y'\n",
|
||||
" elif isinstance(expr,OR):\n",
|
||||
" for x in expr.x:\n",
|
||||
" if self.eval(x)=='y':\n",
|
||||
" return 'y'\n",
|
||||
" return 'n'\n",
|
||||
" elif isinstance(expr,str):\n",
|
||||
" return self.get(expr)\n",
|
||||
" else:\n",
|
||||
" print(\"Unknown expr: {}\".format(expr))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Nun lassen Sie uns unsere Tier-Wissensdatenbank definieren und die Beratung durchführen. Beachten Sie, dass dieser Aufruf Ihnen Fragen stellen wird. Sie können mit `y`/`n` für Ja-Nein-Fragen antworten oder eine Zahl (0..N) für Fragen mit längeren Multiple-Choice-Antworten angeben.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"hair\n",
|
||||
"y/n\n",
|
||||
"sharp teeth\n",
|
||||
"y/n\n",
|
||||
"claws\n",
|
||||
"y/n\n",
|
||||
"forward-looking eyes\n",
|
||||
"y/n\n",
|
||||
"color\n",
|
||||
"0. red-brown\n",
|
||||
"1. black and white\n",
|
||||
"2. other\n",
|
||||
"has hooves\n",
|
||||
"y/n\n",
|
||||
"long neck\n",
|
||||
"y/n\n",
|
||||
"long legs\n",
|
||||
"y/n\n",
|
||||
"pattern\n",
|
||||
"0. dark stripes\n",
|
||||
"1. dark spots\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"'giraffe'"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"kb = KnowledgeBase(rules)\n",
|
||||
"kb.get('animal')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Verwendung von PyKnow für Vorwärtsinferenz\n",
|
||||
"\n",
|
||||
"Im nächsten Beispiel werden wir versuchen, Vorwärtsinferenz mit einer der Bibliotheken für Wissensrepräsentation, [PyKnow](https://github.com/buguroo/pyknow/), zu implementieren. **PyKnow** ist eine Bibliothek zur Erstellung von Vorwärtsinferenzsystemen in Python, die so konzipiert ist, dass sie dem klassischen alten System [CLIPS](http://www.clipsrules.net/index.html) ähnelt.\n",
|
||||
"\n",
|
||||
"Wir hätten die Vorwärtsverkettung auch selbst ohne größere Probleme implementieren können, aber naive Implementierungen sind in der Regel nicht sehr effizient. Für eine effektivere Regelanpassung wird ein spezieller Algorithmus namens [Rete](https://en.wikipedia.org/wiki/Rete_algorithm) verwendet.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Collecting git+https://github.com/buguroo/pyknow/\n",
|
||||
" Cloning https://github.com/buguroo/pyknow/ to /tmp/pip-req-build-3cqeulyl\n",
|
||||
" Running command git clone --filter=blob:none --quiet https://github.com/buguroo/pyknow/ /tmp/pip-req-build-3cqeulyl\n",
|
||||
" Resolved https://github.com/buguroo/pyknow/ to commit 48818336f2e9a126f1964f2d8dc22d37ff800fe8\n",
|
||||
" Preparing metadata (setup.py) ... \u001b[?25ldone\n",
|
||||
"\u001b[?25hCollecting frozendict==1.2\n",
|
||||
" Using cached frozendict-1.2.tar.gz (2.6 kB)\n",
|
||||
" Preparing metadata (setup.py) ... \u001b[?25ldone\n",
|
||||
"\u001b[?25hCollecting schema==0.6.7\n",
|
||||
" Using cached schema-0.6.7-py2.py3-none-any.whl (14 kB)\n",
|
||||
"Building wheels for collected packages: pyknow, frozendict\n",
|
||||
" Building wheel for pyknow (setup.py) ... \u001b[?25ldone\n",
|
||||
"\u001b[?25h Created wheel for pyknow: filename=pyknow-1.7.0-py3-none-any.whl size=34228 sha256=b7de5b09292c4007667c72f69b98d5a1b5f7324ff15f9dd8e077c3d5f7aade42\n",
|
||||
" Stored in directory: /tmp/pip-ephem-wheel-cache-k7jpave7/wheels/81/1a/d3/f6c15dbe1955598a37755215f2a10449e7418500d7bd4b9508\n",
|
||||
" Building wheel for frozendict (setup.py) ... \u001b[?25ldone\n",
|
||||
"\u001b[?25h Created wheel for frozendict: filename=frozendict-1.2-py3-none-any.whl size=3148 sha256=2863d55c240d2409cddf05ccfe600591f8478681549fc97555c47c90dc6bb160\n",
|
||||
" Stored in directory: /home/rg/.cache/pip/wheels/49/ac/f8/cb8120244e710bdb479c86198b03c7b08c3c2d3d2bf448fd6e\n",
|
||||
"Successfully built pyknow frozendict\n",
|
||||
"Installing collected packages: schema, frozendict, pyknow\n",
|
||||
"Successfully installed frozendict-1.2 pyknow-1.7.0 schema-0.6.7\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import sys\n",
|
||||
"!{sys.executable} -m pip install git+https://github.com/buguroo/pyknow/"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from pyknow import *\n",
|
||||
"#import pyknow"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Wir werden unser System als eine Klasse definieren, die `KnowledgeEngine` unterklassen. Jede Regel wird durch eine separate Funktion mit der `@Rule`-Annotation definiert, die angibt, wann die Regel ausgelöst werden soll. Innerhalb der Regel können wir neue Fakten mit der Funktion `declare` hinzufügen, und das Hinzufügen dieser Fakten führt dazu, dass einige weitere Regeln vom Vorwärts-Inferenzmotor aufgerufen werden.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class Animals(KnowledgeEngine):\n",
|
||||
" @Rule(OR(\n",
|
||||
" AND(Fact('sharp teeth'),Fact('claws'),Fact('forward looking eyes')),\n",
|
||||
" Fact('eats meat')))\n",
|
||||
" def cornivor(self):\n",
|
||||
" self.declare(Fact('carnivor'))\n",
|
||||
" \n",
|
||||
" @Rule(OR(Fact('hair'),Fact('gives milk')))\n",
|
||||
" def mammal(self):\n",
|
||||
" self.declare(Fact('mammal'))\n",
|
||||
"\n",
|
||||
" @Rule(Fact('mammal'),\n",
|
||||
" OR(Fact('has hooves'),Fact('chews cud')))\n",
|
||||
" def hooves(self):\n",
|
||||
" self.declare('ungulate')\n",
|
||||
" \n",
|
||||
" @Rule(OR(Fact('feathers'),AND(Fact('flies'),Fact('lays eggs'))))\n",
|
||||
" def bird(self):\n",
|
||||
" self.declare('bird')\n",
|
||||
" \n",
|
||||
" @Rule(Fact('mammal'),Fact('carnivor'),\n",
|
||||
" Fact(color='red-brown'),\n",
|
||||
" Fact(pattern='dark spots'))\n",
|
||||
" def monkey(self):\n",
|
||||
" self.declare(Fact(animal='monkey'))\n",
|
||||
"\n",
|
||||
" @Rule(Fact('mammal'),Fact('carnivor'),\n",
|
||||
" Fact(color='red-brown'),\n",
|
||||
" Fact(pattern='dark stripes'))\n",
|
||||
" def tiger(self):\n",
|
||||
" self.declare(Fact(animal='tiger'))\n",
|
||||
"\n",
|
||||
" @Rule(Fact('ungulate'),\n",
|
||||
" Fact('long neck'),\n",
|
||||
" Fact('long legs'),\n",
|
||||
" Fact(pattern='dark spots'))\n",
|
||||
" def giraffe(self):\n",
|
||||
" self.declare(Fact(animal='giraffe'))\n",
|
||||
"\n",
|
||||
" @Rule(Fact('ungulate'),\n",
|
||||
" Fact(pattern='dark stripes'))\n",
|
||||
" def zebra(self):\n",
|
||||
" self.declare(Fact(animal='zebra'))\n",
|
||||
"\n",
|
||||
" @Rule(Fact('bird'),\n",
|
||||
" Fact('long neck'),\n",
|
||||
" Fact('cannot fly'),\n",
|
||||
" Fact(color='black and white'))\n",
|
||||
" def straus(self):\n",
|
||||
" self.declare(Fact(animal='ostrich'))\n",
|
||||
"\n",
|
||||
" @Rule(Fact('bird'),\n",
|
||||
" Fact('swims'),\n",
|
||||
" Fact('cannot fly'),\n",
|
||||
" Fact(color='black and white'))\n",
|
||||
" def pinguin(self):\n",
|
||||
" self.declare(Fact(animal='pinguin'))\n",
|
||||
"\n",
|
||||
" @Rule(Fact('bird'),\n",
|
||||
" Fact('flies well'))\n",
|
||||
" def albatros(self):\n",
|
||||
" self.declare(Fact(animal='albatross'))\n",
|
||||
" \n",
|
||||
" @Rule(Fact(animal=MATCH.a))\n",
|
||||
" def print_result(self,a):\n",
|
||||
" print('Animal is {}'.format(a))\n",
|
||||
" \n",
|
||||
" def factz(self,l):\n",
|
||||
" for x in l:\n",
|
||||
" self.declare(x)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Sobald wir eine Wissensbasis definiert haben, füllen wir unser Arbeitsgedächtnis mit einigen anfänglichen Fakten und rufen dann die Methode `run()` auf, um die Schlussfolgerung durchzuführen. Als Ergebnis können Sie sehen, dass neue abgeleitete Fakten dem Arbeitsgedächtnis hinzugefügt werden, einschließlich der endgültigen Tatsache über das Tier (wenn wir alle anfänglichen Fakten korrekt eingerichtet haben).\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Animal is tiger\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"FactList([(0, InitialFact()),\n",
|
||||
" (1, Fact(color='red-brown')),\n",
|
||||
" (2, Fact(pattern='dark stripes')),\n",
|
||||
" (3, Fact('sharp teeth')),\n",
|
||||
" (4, Fact('claws')),\n",
|
||||
" (5, Fact('forward looking eyes')),\n",
|
||||
" (6, Fact('gives milk')),\n",
|
||||
" (7, Fact('mammal')),\n",
|
||||
" (8, Fact('carnivor')),\n",
|
||||
" (9, Fact(animal='tiger'))])"
|
||||
]
|
||||
},
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"ex1 = Animals()\n",
|
||||
"ex1.reset()\n",
|
||||
"ex1.factz([\n",
|
||||
" Fact(color='red-brown'),\n",
|
||||
" Fact(pattern='dark stripes'),\n",
|
||||
" Fact('sharp teeth'),\n",
|
||||
" Fact('claws'),\n",
|
||||
" Fact('forward looking eyes'),\n",
|
||||
" Fact('gives milk')])\n",
|
||||
"ex1.run()\n",
|
||||
"ex1.facts"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Haftungsausschluss**: \nDieses Dokument wurde mit dem KI-Übersetzungsdienst [Co-op Translator](https://github.com/Azure/co-op-translator) übersetzt. Obwohl wir uns um Genauigkeit bemühen, weisen wir darauf hin, dass automatisierte Übersetzungen Fehler oder Ungenauigkeiten enthalten können. Das Originaldokument in seiner ursprünglichen Sprache sollte als maßgebliche Quelle betrachtet werden. Für kritische Informationen wird eine professionelle menschliche Übersetzung empfohlen. Wir übernehmen keine Haftung für Missverständnisse oder Fehlinterpretationen, die sich aus der Nutzung dieser Übersetzung ergeben.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.7.4 64-bit (conda)",
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5"
|
||||
}
|
||||
},
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.11.2"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "ab2bd97b0453415b89a469284609a8ce",
|
||||
"translation_date": "2025-08-31T16:26:30+00:00",
|
||||
"source_file": "lessons/2-Symbolic/Animals.ipynb",
|
||||
"language_code": "de"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
|
|
@ -0,0 +1,595 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"collapsed": true
|
||||
},
|
||||
"source": [
|
||||
"# Ontologie der Familienbeziehungen\n",
|
||||
"\n",
|
||||
"Dieses Beispiel ist Teil des [AI for Beginners Curriculum](http://github.com/microsoft/ai-for-beginners) und wurde von [diesem Blogbeitrag](https://habr.com/post/270857/) inspiriert.\n",
|
||||
"\n",
|
||||
"Ich finde es immer schwierig, mir die verschiedenen Beziehungen zwischen Personen in einer Familie zu merken. In diesem Beispiel werden wir eine Ontologie verwenden, die Familienbeziehungen definiert, sowie den tatsächlichen Stammbaum, und zeigen, wie wir dann automatische Schlussfolgerungen ziehen können, um alle Verwandten zu finden.\n",
|
||||
"\n",
|
||||
"### Den Stammbaum erhalten\n",
|
||||
"\n",
|
||||
"Als Beispiel nehmen wir den Stammbaum der [Romanow-Zarenfamilie](https://en.wikipedia.org/wiki/House_of_Romanov). Das gebräuchlichste Format zur Beschreibung von Familienbeziehungen ist [GEDCOM](https://en.wikipedia.org/wiki/GEDCOM). Wir werden den Stammbaum der Romanow-Familie im GEDCOM-Format verwenden:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"0 HEAD\n",
|
||||
"1 CHAR UTF8\n",
|
||||
"1 GEDC\n",
|
||||
"2 VERS 5.5\n",
|
||||
"0 @0@ INDI\n",
|
||||
"1 NAME Mihail Fedorovich /Romanov/\n",
|
||||
"1 SEX M\n",
|
||||
"1 BIRT\n",
|
||||
"2 DATE 1613\n",
|
||||
"1 DEAT \n",
|
||||
"2 DATE 1645\n",
|
||||
"1 FAMS @41@\n",
|
||||
"0 @1@ INDI\n",
|
||||
"1 NAME Evdokija Lukjanovna /Streshneva/\n",
|
||||
"1 SEX F\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"!head -15 data/tsars.ged"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Um die GEDCOM-Datei zu verwenden, können wir die Bibliothek `python-gedcom` verwenden:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Collecting python-gedcom\n",
|
||||
" Downloading python_gedcom-1.0.0-py2.py3-none-any.whl (35 kB)\n",
|
||||
"Installing collected packages: python-gedcom\n",
|
||||
"Successfully installed python-gedcom-1.0.0\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import sys\n",
|
||||
"!{sys.executable} -m pip install python-gedcom"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Diese Bibliothek nimmt einige der technischen Probleme beim Datei-Parsing weg, aber sie gibt uns dennoch ziemlich niedrigstufigen Zugriff auf alle Personen und Familien im Baum. Hier ist, wie wir die Datei parsen und die Liste aller Personen anzeigen können:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from gedcom.parser import Parser\n",
|
||||
"from gedcom.element.individual import IndividualElement\n",
|
||||
"from gedcom.element.family import FamilyElement\n",
|
||||
"g = Parser()\n",
|
||||
"g.parse_file('data/tsars.ged')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {
|
||||
"scrolled": true,
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[('@0@', ('Mihail Fedorovich', 'Romanov')),\n",
|
||||
" ('@1@', ('Evdokija Lukjanovna', 'Streshneva')),\n",
|
||||
" ('@2@', ('Aleksej Mihajlovich', 'Romanov')),\n",
|
||||
" ('@3@', ('Marija Ilinichna', 'Miloslavskaja')),\n",
|
||||
" ('@4@', ('Natalja Kirillovna', 'Naryshkina')),\n",
|
||||
" ('@5@', ('Marfa Matveevna', 'Apraksina')),\n",
|
||||
" ('@6@', ('Fedor Alekseevich', 'Romanov')),\n",
|
||||
" ('@7@', ('Sofja Aleksevna', 'Romanova')),\n",
|
||||
" ('@8@', ('Ivan V Alekseevich', 'Romanov')),\n",
|
||||
" ('@9@', ('Praskovja Fedorovna', 'Saltykova')),\n",
|
||||
" ('@10@', ('Ekaterina Ivanovna', 'Romanova')),\n",
|
||||
" ('@11@', ('Anna Ivanovna', 'Romanova')),\n",
|
||||
" ('@12@', ('Fridrih Vilgelm', 'Kurlandskij')),\n",
|
||||
" ('@13@', ('Karl Leopold', 'Meklenburg-Shverinskij')),\n",
|
||||
" ('@14@', ('Anna Leopoldovna', 'Meklenburg-Shverinskaja')),\n",
|
||||
" ('@15@', ('Anton Ulrih', 'Braunshvejg-Volfenbjuttelskij')),\n",
|
||||
" ('@16@', ('Ivan VI Antonovich', 'Braunshvejg-Volfenbjuttelskij')),\n",
|
||||
" ('@17@', ('Petr I Alekseevich', 'Romanov')),\n",
|
||||
" ('@18@', ('Evdokija Fedorovna', 'Lopuhina')),\n",
|
||||
" ('@19@', ('Ekaterina I Alekseevna', 'Mihajlova')),\n",
|
||||
" ('@20@', ('Aleksej Petrovich', 'Romanov')),\n",
|
||||
" ('@21@', ('Sharlotta Kristina', 'Braunshvejg-Volfenbjuttelskaja')),\n",
|
||||
" ('@22@', ('Petr II Alekseevich', 'Romanov')),\n",
|
||||
" ('@23@', ('Anna Petrovna', 'Romanova')),\n",
|
||||
" ('@24@', ('Elizaveta Petrovna', 'Romanova')),\n",
|
||||
" ('@25@', ('Karl Fridrih', 'Golshtejn-Gottorpskij')),\n",
|
||||
" ('@26@', ('Petr III Fedorovich', 'Romanov')),\n",
|
||||
" ('@27@', ('Ekaterina II', 'Alekseevna')),\n",
|
||||
" ('@28@', ('Pavel I Petrovich', 'Romanov')),\n",
|
||||
" ('@29@', ('Natalja Alekseevna', 'Gessen-Darmshtadskaja')),\n",
|
||||
" ('@30@', ('Marija Fedorovna', 'Vjurtembergskaja')),\n",
|
||||
" ('@31@', ('Aleksandr I Pavlovich', 'Romanov')),\n",
|
||||
" ('@32@', ('Elizaveta Alekseevna', 'Baden-Durlahskaja')),\n",
|
||||
" ('@33@', ('Nikolaj I Pavlovich', 'Romanov')),\n",
|
||||
" ('@34@', ('Aleksandra Fedorovna', 'Prusskaja')),\n",
|
||||
" ('@35@', ('Aleksandr II Nikolaevich', 'Romanov')),\n",
|
||||
" ('@36@', ('Marija Aleksandrovna', 'Gessenskaja')),\n",
|
||||
" ('@37@', ('Aleksandr III Aleksandrovich', 'Romanov')),\n",
|
||||
" ('@38@', ('Marija Fedorovna', 'Datskaja')),\n",
|
||||
" ('@39@', ('Nikolaj II Aleksandrovich', 'Romanov')),\n",
|
||||
" ('@40@', ('Aleksandra Fedorovna', 'Gessenskaja'))]"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"d = g.get_element_dictionary()\n",
|
||||
"[ (k,v.get_name()) for k,v in d.items() if isinstance(v,IndividualElement)]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Hier ist, wie wir Informationen über Familien erhalten können. Beachten Sie, dass dies uns eine Liste von **Kennungen** gibt, und wir müssen sie in Namen umwandeln, wenn wir mehr Klarheit wünschen:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[('@41@', ['@0@', '@1@', '@2@']),\n",
|
||||
" ('@42@', ['@2@', '@3@', '@6@', '@7@', '@8@']),\n",
|
||||
" ('@43@', ['@8@', '@9@', '@10@', '@11@']),\n",
|
||||
" ('@44@', ['@13@', '@10@', '@14@']),\n",
|
||||
" ('@45@', ['@15@', '@14@', '@16@']),\n",
|
||||
" ('@46@', ['@2@', '@4@', '@17@']),\n",
|
||||
" ('@47@', ['@17@', '@18@', '@20@']),\n",
|
||||
" ('@48@', ['@20@', '@21@', '@22@']),\n",
|
||||
" ('@49@', ['@17@', '@19@', '@23@', '@24@']),\n",
|
||||
" ('@50@', ['@25@', '@23@', '@26@']),\n",
|
||||
" ('@51@', ['@26@', '@27@', '@28@']),\n",
|
||||
" ('@52@', ['@28@', '@30@', '@31@', '@33@']),\n",
|
||||
" ('@53@', ['@33@', '@34@', '@35@']),\n",
|
||||
" ('@54@', ['@35@', '@36@', '@37@']),\n",
|
||||
" ('@55@', ['@37@', '@38@', '@39@'])]"
|
||||
]
|
||||
},
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"d = g.get_element_dictionary()\n",
|
||||
"[ (k,[x.get_value() for x in v.get_child_elements()]) for k,v in d.items() if isinstance(v,FamilyElement)]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Familienontologie abrufen\n",
|
||||
"\n",
|
||||
"Schauen wir uns als Nächstes die [Familienontologie](https://raw.githubusercontent.com/blokhin/genealogical-trees/master/data/header.ttl) an, die als eine Menge von Semantic-Web-Tripeln definiert ist. Diese Ontologie beschreibt Beziehungen wie `isUncleOf`, `isCousinOf` und viele andere. All diese Beziehungen werden auf Basis der grundlegenden Prädikate `isMotherOf`, `isFatherOf`, `isBrotherOf` und `isSisterOf` definiert. Wir werden automatisches Schließen verwenden, um alle anderen Beziehungen mithilfe der Ontologie abzuleiten.\n",
|
||||
"\n",
|
||||
"Hier ist eine Beispieldefinition der Eigenschaft `isAuntOf`, die als Komposition von `isSisterOf` und `isParentOf` definiert ist (*Eine Tante ist die Schwester eines Elternteils*).\n",
|
||||
"\n",
|
||||
"```\n",
|
||||
"fhkb:isAuntOf a owl:ObjectProperty ;\n",
|
||||
" rdfs:domain fhkb:Woman ;\n",
|
||||
" rdfs:range fhkb:Person ;\n",
|
||||
" owl:propertyChainAxiom ( fhkb:isSisterOf fhkb:isParentOf ) .\n",
|
||||
"```\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"@prefix fhkb: <http://www.example.com/genealogy.owl#> .\n",
|
||||
"@prefix owl: <http://www.w3.org/2002/07/owl#> .\n",
|
||||
"@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n",
|
||||
"@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .\n",
|
||||
"@prefix xml: <http://www.w3.org/XML/1998/namespace> .\n",
|
||||
"@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\n",
|
||||
"\n",
|
||||
"<http://www.example.com/genealogy.owl#> a owl:Ontology .\n",
|
||||
"\n",
|
||||
"fhkb:DomainEntity a owl:Class .\n",
|
||||
"\n",
|
||||
"fhkb:Man a owl:Class ;\n",
|
||||
" owl:equivalentClass [ a owl:Class ;\n",
|
||||
" owl:intersectionOf ( fhkb:Person [ a owl:Restriction ;\n",
|
||||
" owl:onProperty fhkb:hasSex ;\n",
|
||||
" owl:someValuesFrom fhkb:Male ] ) ] .\n",
|
||||
"\n",
|
||||
"fhkb:Woman a owl:Class ;\n",
|
||||
" owl:equivalentClass [ a owl:Class ;\n",
|
||||
" owl:intersectionOf ( fhkb:Person [ a owl:Restriction ;\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"!head -20 data/onto.ttl"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Erstellung einer Ontologie für Schlussfolgerungen\n",
|
||||
"\n",
|
||||
"Der Einfachheit halber erstellen wir eine Ontologiedatei, die die ursprünglichen Regeln der Familienontologie sowie Fakten über Einzelpersonen aus unserer GEDCOM-Datei enthält. Wir werden die GEDCOM-Datei durchgehen, Informationen über Familien und Einzelpersonen extrahieren und sie in Tripel umwandeln.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!cp data/onto.ttl .\n",
|
||||
"\n",
|
||||
"gedcom_dict = g.get_element_dictionary()\n",
|
||||
"individuals, marriages = {}, {}\n",
|
||||
"\n",
|
||||
"def term2id(el):\n",
|
||||
" return \"i\" + el.get_pointer().replace('@', '').lower()\n",
|
||||
"\n",
|
||||
"out = open(\"onto.ttl\",\"a\")\n",
|
||||
"\n",
|
||||
"for k, v in gedcom_dict.items():\n",
|
||||
" if isinstance(v,IndividualElement):\n",
|
||||
" children, siblings = set(), set()\n",
|
||||
" idx = term2id(v)\n",
|
||||
"\n",
|
||||
" title = v.get_name()[0] + \" \" + v.get_name()[1]\n",
|
||||
" title = title.replace('\"', '').replace('[', '').replace(']', '').replace('(', '').replace(')', '').strip()\n",
|
||||
"\n",
|
||||
" own_families = g.get_families(v, 'FAMS')\n",
|
||||
" for fam in own_families:\n",
|
||||
" children |= set(term2id(i) for i in g.get_family_members(fam, \"CHIL\"))\n",
|
||||
"\n",
|
||||
" parent_families = g.get_families(v, 'FAMC')\n",
|
||||
" if len(parent_families):\n",
|
||||
" for member in g.get_family_members(parent_families[0], \"CHIL\"): # NB adoptive families i.e len(parent_families)>1 are not considered (TODO?)\n",
|
||||
" if member.get_pointer() == v.get_pointer():\n",
|
||||
" continue\n",
|
||||
" siblings.add(term2id(member))\n",
|
||||
"\n",
|
||||
" if idx in individuals:\n",
|
||||
" children |= individuals[idx].get('children', set())\n",
|
||||
" siblings |= individuals[idx].get('siblings', set())\n",
|
||||
" individuals[idx] = {'sex': v.get_gender().lower(), 'children': children, 'siblings': siblings, 'title': title}\n",
|
||||
"\n",
|
||||
" elif isinstance(v,FamilyElement):\n",
|
||||
" wife, husb, children = None, None, set()\n",
|
||||
" children = set(term2id(i) for i in g.get_family_members(v, \"CHIL\"))\n",
|
||||
"\n",
|
||||
" try:\n",
|
||||
" wife = g.get_family_members(v, \"WIFE\")[0]\n",
|
||||
" wife = term2id(wife)\n",
|
||||
" if wife in individuals: individuals[wife]['children'] |= children\n",
|
||||
" else: individuals[wife] = {'children': children}\n",
|
||||
" except IndexError: pass\n",
|
||||
" try:\n",
|
||||
" husb = g.get_family_members(v, \"HUSB\")[0]\n",
|
||||
" husb = term2id(husb)\n",
|
||||
" if husb in individuals: individuals[husb]['children'] |= children\n",
|
||||
" else: individuals[husb] = {'children': children}\n",
|
||||
" except IndexError: pass\n",
|
||||
"\n",
|
||||
" if wife and husb: marriages[wife + husb] = (term2id(v), wife, husb)\n",
|
||||
"\n",
|
||||
"for idx, val in individuals.items():\n",
|
||||
" added_terms = ''\n",
|
||||
" if val['sex'] == 'f':\n",
|
||||
" parent_predicate, sibl_predicate = \"isMotherOf\", \"isSisterOf\"\n",
|
||||
" else:\n",
|
||||
" parent_predicate, sibl_predicate = \"isFatherOf\", \"isBrotherOf\"\n",
|
||||
" if len(val['children']):\n",
|
||||
" added_terms += \" ;\\n fhkb:\" + parent_predicate + \" \" + \", \".join([\"fhkb:\" + i for i in val['children']])\n",
|
||||
" if len(val['siblings']):\n",
|
||||
" added_terms += \" ;\\n fhkb:\" + sibl_predicate + \" \" + \", \".join([\"fhkb:\" + i for i in val['siblings']])\n",
|
||||
" out.write(\"fhkb:%s a owl:NamedIndividual, owl:Thing%s ;\\n rdfs:label \\\"%s\\\" .\\n\" % (idx, added_terms, val['title']))\n",
|
||||
"\n",
|
||||
"for k, v in marriages.items():\n",
|
||||
" out.write(\"fhkb:%s a owl:NamedIndividual, owl:Thing ;\\n fhkb:hasFemalePartner fhkb:%s ;\\n fhkb:hasMalePartner fhkb:%s .\\n\" % v)\n",
|
||||
"\n",
|
||||
"out.write(\"[] a owl:AllDifferent ;\\n owl:distinctMembers (\")\n",
|
||||
"for idx in individuals.keys():\n",
|
||||
" out.write(\" fhkb:\" + idx)\n",
|
||||
"for k, v in marriages.items():\n",
|
||||
" out.write(\" fhkb:\" + v[0])\n",
|
||||
"out.write(\" ) .\")\n",
|
||||
"out.close()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" fhkb:hasFemalePartner fhkb:i34 ;\n",
|
||||
" fhkb:hasMalePartner fhkb:i33 .\n",
|
||||
"fhkb:i54 a owl:NamedIndividual, owl:Thing ;\n",
|
||||
" fhkb:hasFemalePartner fhkb:i36 ;\n",
|
||||
" fhkb:hasMalePartner fhkb:i35 .\n",
|
||||
"fhkb:i55 a owl:NamedIndividual, owl:Thing ;\n",
|
||||
" fhkb:hasFemalePartner fhkb:i38 ;\n",
|
||||
" fhkb:hasMalePartner fhkb:i37 .\n",
|
||||
"[] a owl:AllDifferent ;\n",
|
||||
" owl:distinctMembers ( fhkb:i0 fhkb:i1 fhkb:i2 fhkb:i3 fhkb:i4 fhkb:i5 fhkb:i6 fhkb:i7 fhkb:i8 fhkb:i9 fhkb:i10 fhkb:i11 fhkb:i12 fhkb:i13 fhkb:i14 fhkb:i15 fhkb:i16 fhkb:i17 fhkb:i18 fhkb:i19 fhkb:i20 fhkb:i21 fhkb:i22 fhkb:i23 fhkb:i24 fhkb:i25 fhkb:i26 fhkb:i27 fhkb:i28 fhkb:i29 fhkb:i30 fhkb:i31 fhkb:i32 fhkb:i33 fhkb:i34 fhkb:i35 fhkb:i36 fhkb:i37 fhkb:i38 fhkb:i39 fhkb:i40 fhkb:i41 fhkb:i42 fhkb:i43 fhkb:i44 fhkb:i45 fhkb:i46 fhkb:i47 fhkb:i48 fhkb:i49 fhkb:i50 fhkb:i51 fhkb:i52 fhkb:i53 fhkb:i54 fhkb:i55 ) ."
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"!tail onto.ttl"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Durchführung von Inferenz\n",
|
||||
"\n",
|
||||
"Nun möchten wir in der Lage sein, diese Ontologie für Inferenz und Abfragen zu nutzen. Wir werden die Bibliothek [RDFLib](https://github.com/RDFLib) verwenden, die es ermöglicht, RDF-Grafen in verschiedenen Formaten zu lesen, Abfragen durchzuführen usw.\n",
|
||||
"\n",
|
||||
"Für logische Inferenz verwenden wir die Bibliothek [OWL-RL](https://github.com/RDFLib/OWL-RL), die es uns erlaubt, die **Closure** des RDF-Grafen zu erstellen, d. h. alle möglichen Konzepte und Relationen hinzuzufügen, die abgeleitet werden können.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Requirement already satisfied: rdflib in /home/rg/anaconda3/envs/ai4beg/lib/python3.11/site-packages (6.3.2)\n",
|
||||
"Requirement already satisfied: isodate<0.7.0,>=0.6.0 in /home/rg/anaconda3/envs/ai4beg/lib/python3.11/site-packages (from rdflib) (0.6.1)\n",
|
||||
"Requirement already satisfied: pyparsing<4,>=2.1.0 in /home/rg/anaconda3/envs/ai4beg/lib/python3.11/site-packages (from rdflib) (3.0.9)\n",
|
||||
"Requirement already satisfied: six in /home/rg/anaconda3/envs/ai4beg/lib/python3.11/site-packages (from isodate<0.7.0,>=0.6.0->rdflib) (1.16.0)\n",
|
||||
"Collecting git+https://github.com/RDFLib/OWL-RL.git\n",
|
||||
" Cloning https://github.com/RDFLib/OWL-RL.git to /tmp/pip-req-build-lbfzwi3m\n",
|
||||
" Running command git clone --filter=blob:none --quiet https://github.com/RDFLib/OWL-RL.git /tmp/pip-req-build-lbfzwi3m\n",
|
||||
" Resolved https://github.com/RDFLib/OWL-RL.git to commit a77e1791b88b54aace609bc6000aac14c7add4ff\n",
|
||||
" Preparing metadata (setup.py) ... \u001b[?25ldone\n",
|
||||
"\u001b[?25hRequirement already satisfied: rdflib>=6.0.2 in /home/rg/anaconda3/envs/ai4beg/lib/python3.11/site-packages (from owlrl==6.0.2) (6.3.2)\n",
|
||||
"Requirement already satisfied: isodate<0.7.0,>=0.6.0 in /home/rg/anaconda3/envs/ai4beg/lib/python3.11/site-packages (from rdflib>=6.0.2->owlrl==6.0.2) (0.6.1)\n",
|
||||
"Requirement already satisfied: pyparsing<4,>=2.1.0 in /home/rg/anaconda3/envs/ai4beg/lib/python3.11/site-packages (from rdflib>=6.0.2->owlrl==6.0.2) (3.0.9)\n",
|
||||
"Requirement already satisfied: six in /home/rg/anaconda3/envs/ai4beg/lib/python3.11/site-packages (from isodate<0.7.0,>=0.6.0->rdflib>=6.0.2->owlrl==6.0.2) (1.16.0)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"!{sys.executable} -m pip install rdflib\n",
|
||||
"!{sys.executable} -m pip install git+https://github.com/RDFLib/OWL-RL.git"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Lass uns die Ontologie-Datei öffnen und sehen, wie viele Tripel sie enthält:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Triplets found:669\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import rdflib\n",
|
||||
"from owlrl import DeductiveClosure, OWLRL_Extension\n",
|
||||
"\n",
|
||||
"g = rdflib.Graph()\n",
|
||||
"g.parse(\"onto.ttl\", format=\"turtle\")\n",
|
||||
"\n",
|
||||
"print(\"Triplets found:%d\" % len(g))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Lassen Sie uns nun die Schließung erstellen und sehen, wie die Anzahl der Tripel zunimmt:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Triplets after inference:4246\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"DeductiveClosure(OWLRL_Extension).expand(g)\n",
|
||||
"print(\"Triplets after inference:%d\" % len(g))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Abfragen von Verwandten\n",
|
||||
"\n",
|
||||
"Jetzt können wir den Graphen abfragen, um verschiedene Beziehungen zwischen Personen zu sehen. Wir können die **SPARQL**-Sprache zusammen mit der `query`-Methode verwenden. In unserem Fall schauen wir uns alle **Onkel** in unserem Familienstammbaum an:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Fedor Alekseevich Romanov is uncle of Ekaterina Ivanovna Romanova\n",
|
||||
"Aleksandr I Pavlovich Romanov is uncle of Aleksandr II Nikolaevich Romanov\n",
|
||||
"Fedor Alekseevich Romanov is uncle of Anna Ivanovna Romanova\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"qres = g.query(\n",
|
||||
" \"\"\"SELECT DISTINCT ?aname ?bname\n",
|
||||
" WHERE {\n",
|
||||
" ?a fhkb:isUncleOf ?b .\n",
|
||||
" ?a rdfs:label ?aname .\n",
|
||||
" ?b rdfs:label ?bname .\n",
|
||||
" }\"\"\")\n",
|
||||
"\n",
|
||||
"for row in qres:\n",
|
||||
" print(\"%s is uncle of %s\" % row)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Experimentiere gerne mit anderen Familienbeziehungen. Zum Beispiel kannst du dir die `isAncestorOf`-Beziehung ansehen, die rekursiv alle Vorfahren einer bestimmten Person definiert.\n",
|
||||
"\n",
|
||||
"Zum Schluss räumen wir auf!\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!rm onto.ttl"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Haftungsausschluss**: \nDieses Dokument wurde mit dem KI-Übersetzungsdienst [Co-op Translator](https://github.com/Azure/co-op-translator) übersetzt. Obwohl wir uns um Genauigkeit bemühen, weisen wir darauf hin, dass automatisierte Übersetzungen Fehler oder Ungenauigkeiten enthalten können. Das Originaldokument in seiner ursprünglichen Sprache sollte als maßgebliche Quelle betrachtet werden. Für kritische Informationen wird eine professionelle menschliche Übersetzung empfohlen. Wir übernehmen keine Haftung für Missverständnisse oder Fehlinterpretationen, die sich aus der Nutzung dieser Übersetzung ergeben.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.6",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.11.2"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "6537d5597320e27b6052b4377b8ff8bb",
|
||||
"translation_date": "2025-08-31T16:24:24+00:00",
|
||||
"source_file": "lessons/2-Symbolic/FamilyOntology.ipynb",
|
||||
"language_code": "de"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
|
|
@ -0,0 +1,548 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"collapsed": true
|
||||
},
|
||||
"source": [
|
||||
"## Microsoft Concept Graph\n",
|
||||
"\n",
|
||||
"[Microsoft Concept Graph](https://concept.research.microsoft.com/) ist eine umfangreiche Taxonomie von Begriffen, die aus dem Internet extrahiert wurden, mit `is-a`-Beziehungen zwischen Konzepten.\n",
|
||||
"\n",
|
||||
"Der Context Graph ist in zwei Formen verfügbar:\n",
|
||||
" * Große Textdatei zum Herunterladen\n",
|
||||
" * REST-API\n",
|
||||
"\n",
|
||||
"Statistiken:\n",
|
||||
" * 5.401.933 einzigartige Konzepte,\n",
|
||||
" * 12.551.613 einzigartige Instanzen\n",
|
||||
" * 87.603.947 `is-a`-Beziehungen\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Verwendung des Webdienstes\n",
|
||||
"\n",
|
||||
"Der Webdienst bietet verschiedene Aufrufe, um die Wahrscheinlichkeit zu schätzen, dass ein Konzept zu verschiedenen Gruppen gehört. Weitere Informationen sind [hier](https://concept.research.microsoft.com/Home/Api) verfügbar. \n",
|
||||
"Hier ist die Beispiel-URL für einen Aufruf: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"{'company': 0.6105356614382954,\n",
|
||||
" 'vendor': 0.08858636677518003,\n",
|
||||
" 'client': 0.048239124001183784,\n",
|
||||
" 'firm': 0.045476965571668145,\n",
|
||||
" 'large company': 0.043109401203511886,\n",
|
||||
" 'organization': 0.043010752688172046,\n",
|
||||
" 'corporation': 0.035908059583703265,\n",
|
||||
" 'brand': 0.03383644076156654,\n",
|
||||
" 'software company': 0.027522935779816515,\n",
|
||||
" 'technology company': 0.023774292196902438}"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import urllib\n",
|
||||
"import json\n",
|
||||
"import ssl\n",
|
||||
"\n",
|
||||
"def http(x):\n",
|
||||
" ssl._create_default_https_context = ssl._create_unverified_context\n",
|
||||
" response = urllib.request.urlopen(x)\n",
|
||||
" data = response.read()\n",
|
||||
" return data.decode('utf-8')\n",
|
||||
"\n",
|
||||
"def query(x):\n",
|
||||
" return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n",
|
||||
"\n",
|
||||
"query('microsoft')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Lassen Sie uns versuchen, die Nachrichtentitel mithilfe von übergeordneten Konzepten zu kategorisieren. Um Nachrichtentitel zu erhalten, werden wir den Dienst [NewsApi.org](http://newsapi.org) verwenden. Sie müssen Ihren eigenen API-Schlüssel erhalten, um den Dienst nutzen zu können – gehen Sie auf die Website und registrieren Sie sich für den kostenlosen Entwicklerplan.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 20,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"newsapi_key = '<your API key here>'\n",
|
||||
"def get_news(country='us'):\n",
|
||||
" res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n",
|
||||
" return res['articles']\n",
|
||||
"\n",
|
||||
"all_titles = [x['title'] for x in get_news('us')+get_news('gb')]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 21,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n",
|
||||
" 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n",
|
||||
" 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n",
|
||||
" 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n",
|
||||
" \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n",
|
||||
" '2 teachers killed at Swedish high school, student arrested - ABC News',\n",
|
||||
" 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n",
|
||||
" 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n",
|
||||
" '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n",
|
||||
" 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n",
|
||||
" \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n",
|
||||
" \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n",
|
||||
" \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n",
|
||||
" 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n",
|
||||
" 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n",
|
||||
" 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n",
|
||||
" 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n",
|
||||
" 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n",
|
||||
" \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n",
|
||||
" 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n",
|
||||
" \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n",
|
||||
" 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n",
|
||||
" 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n",
|
||||
" 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n",
|
||||
" 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n",
|
||||
" \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n",
|
||||
" 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n",
|
||||
" \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n",
|
||||
" 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n",
|
||||
" 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n",
|
||||
" \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n",
|
||||
" '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n",
|
||||
" 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n",
|
||||
" 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n",
|
||||
" 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n",
|
||||
" 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n",
|
||||
" 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n",
|
||||
" 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n",
|
||||
" \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n",
|
||||
" 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']"
|
||||
]
|
||||
},
|
||||
"execution_count": 21,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"all_titles"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Zunächst möchten wir in der Lage sein, Substantive aus Nachrichtentiteln zu extrahieren. Wir werden die Bibliothek `TextBlob` verwenden, um dies zu tun, die viele typische NLP-Aufgaben wie diese vereinfacht.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n",
|
||||
"Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n",
|
||||
"Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n",
|
||||
"Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n",
|
||||
"Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n",
|
||||
"Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n",
|
||||
"Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n",
|
||||
"Finished.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"[nltk_data] Downloading package brown to\n",
|
||||
"[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n",
|
||||
"[nltk_data] Package brown is already up-to-date!\n",
|
||||
"[nltk_data] Downloading package punkt to\n",
|
||||
"[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n",
|
||||
"[nltk_data] Package punkt is already up-to-date!\n",
|
||||
"[nltk_data] Downloading package wordnet to\n",
|
||||
"[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n",
|
||||
"[nltk_data] Package wordnet is already up-to-date!\n",
|
||||
"[nltk_data] Downloading package averaged_perceptron_tagger to\n",
|
||||
"[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n",
|
||||
"[nltk_data] Package averaged_perceptron_tagger is already up-to-\n",
|
||||
"[nltk_data] date!\n",
|
||||
"[nltk_data] Downloading package conll2000 to\n",
|
||||
"[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n",
|
||||
"[nltk_data] Package conll2000 is already up-to-date!\n",
|
||||
"[nltk_data] Downloading package movie_reviews to\n",
|
||||
"[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n",
|
||||
"[nltk_data] Package movie_reviews is already up-to-date!\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import sys\n",
|
||||
"!{sys.executable} -m pip install textblob\n",
|
||||
"!{sys.executable} -m textblob.download_corpora\n",
|
||||
"from textblob import TextBlob"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 22,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"{'covid-19 live updates': 1,\n",
|
||||
" 'vaccines': 1,\n",
|
||||
" 'boosters': 1,\n",
|
||||
" 'york': 4,\n",
|
||||
" 'ukrainians flee mariupol': 1,\n",
|
||||
" 'forces push': 1,\n",
|
||||
" 'port city': 1,\n",
|
||||
" 'wall street journal': 3,\n",
|
||||
" 'bond yields': 1,\n",
|
||||
" 'futures rise': 1,\n",
|
||||
" 'powell says fed': 1,\n",
|
||||
" 'ready': 1,\n",
|
||||
" 'be': 1,\n",
|
||||
" 'aggressive': 1,\n",
|
||||
" 'putin': 3,\n",
|
||||
" 'alexei navalny': 2,\n",
|
||||
" 'russian': 2,\n",
|
||||
" 'supreme court nominee': 1,\n",
|
||||
" 'ketanji brown jackson': 1,\n",
|
||||
" \"confirmation hearing 's\": 1,\n",
|
||||
" 'cnn': 1,\n",
|
||||
" 'swedish': 1,\n",
|
||||
" 'high school': 1,\n",
|
||||
" 'abc': 1,\n",
|
||||
" 'clues': 1,\n",
|
||||
" 'covid-19': 1,\n",
|
||||
" '’ s': 2,\n",
|
||||
" 'moves': 1,\n",
|
||||
" 'sewers': 1,\n",
|
||||
" 'roll dice': 1,\n",
|
||||
" 'jackson': 1,\n",
|
||||
" 'decisions |': 1,\n",
|
||||
" 'thehill': 1,\n",
|
||||
" 'clear': 2,\n",
|
||||
" 'chemical weapons': 2,\n",
|
||||
" 'ukraine': 3,\n",
|
||||
" 'claims president': 2,\n",
|
||||
" 'biden': 2,\n",
|
||||
" 'nasa': 2,\n",
|
||||
" 'solar system': 2,\n",
|
||||
" 'daily mail': 3,\n",
|
||||
" 'us stocks': 1,\n",
|
||||
" 'fed chair powell': 1,\n",
|
||||
" \"'s remarks\": 1,\n",
|
||||
" 'fox': 1,\n",
|
||||
" \"'we 've\": 1,\n",
|
||||
" 'tests': 1,\n",
|
||||
" 'covid': 1,\n",
|
||||
" 'politico': 1,\n",
|
||||
" 'duchess': 1,\n",
|
||||
" 'cambridge': 1,\n",
|
||||
" 'swaps khaki jungle gear': 1,\n",
|
||||
" 'vampire': 1,\n",
|
||||
" 'wife': 1,\n",
|
||||
" 'belize': 1,\n",
|
||||
" 'china': 2,\n",
|
||||
" 'flight recorders': 1,\n",
|
||||
" 'plane crash': 1,\n",
|
||||
" 'reuters': 2,\n",
|
||||
" 'russian oligarch': 1,\n",
|
||||
" 'abramovich': 1,\n",
|
||||
" 'live': 1,\n",
|
||||
" 'russia': 2,\n",
|
||||
" 'stops talks': 1,\n",
|
||||
" 'japan': 1,\n",
|
||||
" 'español': 1,\n",
|
||||
" 'powers remain': 1,\n",
|
||||
" 'threats lurk': 1,\n",
|
||||
" 'set': 1,\n",
|
||||
" 'webb': 1,\n",
|
||||
" 'telescope begins multi-instrument alignment': 1,\n",
|
||||
" 'scitechdaily': 1,\n",
|
||||
" 'uconn': 1,\n",
|
||||
" 'ucf': 1,\n",
|
||||
" 'ncaa': 1,\n",
|
||||
" \"women 's tournament second-round highlights\": 1,\n",
|
||||
" 'march madness': 1,\n",
|
||||
" 'bucking republican trend': 1,\n",
|
||||
" 'indiana': 1,\n",
|
||||
" 'vetoes transgender': 1,\n",
|
||||
" 'bill': 1,\n",
|
||||
" 'maggie fox': 1,\n",
|
||||
" 'coronation': 1,\n",
|
||||
" 'shameless': 1,\n",
|
||||
" \"'sudden accident\": 1,\n",
|
||||
" 'mirror online': 1,\n",
|
||||
" 'mirror': 2,\n",
|
||||
" 'plane crash –': 1,\n",
|
||||
" 'search': 1,\n",
|
||||
" 'moment flight': 1,\n",
|
||||
" 'daniel morgan': 1,\n",
|
||||
" 'report condemns': 1,\n",
|
||||
" 'met': 1,\n",
|
||||
" 'guardian': 6,\n",
|
||||
" 'rishi sunak': 1,\n",
|
||||
" '’ s spring': 1,\n",
|
||||
" 'statement': 1,\n",
|
||||
" 'bbc.com': 1,\n",
|
||||
" 'uk': 3,\n",
|
||||
" 'ireland': 1,\n",
|
||||
" 'euro': 1,\n",
|
||||
" 'vladimir putin': 2,\n",
|
||||
" \"'s 'lover\": 1,\n",
|
||||
" 'brass eye': 1,\n",
|
||||
" '’ s outtakes': 1,\n",
|
||||
" 'brutal tv comedy': 1,\n",
|
||||
" 'threatens civilians': 1,\n",
|
||||
" 'mariupol': 1,\n",
|
||||
" \"'s spirit\": 1,\n",
|
||||
" 'shell u-turn': 1,\n",
|
||||
" 'cambo': 1,\n",
|
||||
" 'green targets': 1,\n",
|
||||
" 'st helens': 1,\n",
|
||||
" 'dog attack': 1,\n",
|
||||
" 'girl': 1,\n",
|
||||
" 'bbc': 1,\n",
|
||||
" 'playstation': 1,\n",
|
||||
" \"'assassin 's\": 1,\n",
|
||||
" 'creed': 1,\n",
|
||||
" 'jade raymond': 1,\n",
|
||||
" 'haven studios': 1,\n",
|
||||
" 'nme': 1,\n",
|
||||
" 'nintendo switch': 1,\n",
|
||||
" 'folders •': 1,\n",
|
||||
" 'eurogamer.net': 2,\n",
|
||||
" 'fa': 1,\n",
|
||||
" 'solution ”': 1,\n",
|
||||
" 'liverpool': 1,\n",
|
||||
" 'fan group blasts “ shambolic ”': 1,\n",
|
||||
" 'wembley': 1,\n",
|
||||
" 'anfield': 1,\n",
|
||||
" 'manchester': 1,\n",
|
||||
" 'live erik': 1,\n",
|
||||
" 'hag': 1,\n",
|
||||
" 'utd': 1,\n",
|
||||
" 'manager updates': 1,\n",
|
||||
" 'manchester evening': 1,\n",
|
||||
" 'inflation': 1,\n",
|
||||
" 'government borrowing': 1,\n",
|
||||
" 'february': 1,\n",
|
||||
" 'crude oil': 1,\n",
|
||||
" '– business': 1,\n",
|
||||
" 'kremlin': 1,\n",
|
||||
" 'large-scale fraud': 1,\n",
|
||||
" 'sky': 1,\n",
|
||||
" 'natural gas': 1,\n",
|
||||
" 'gazprom': 1,\n",
|
||||
" 'retail unit': 1,\n",
|
||||
" 'insider': 1,\n",
|
||||
" 'zaghari-ratcliffe': 1,\n",
|
||||
" 'hunt': 1,\n",
|
||||
" 'iran': 1,\n",
|
||||
" 'debt payment': 1}"
|
||||
]
|
||||
},
|
||||
"execution_count": 22,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"w = {}\n",
|
||||
"for x in all_titles:\n",
|
||||
" for n in TextBlob(x).noun_phrases:\n",
|
||||
" if n in w:\n",
|
||||
" w[n].append(x)\n",
|
||||
" else:\n",
|
||||
" w[n]=[x]\n",
|
||||
"{ x:len(w[x]) for x in w.keys()}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Wir können sehen, dass Substantive uns keine großen thematischen Gruppen liefern. Lassen Sie uns Substantive durch allgemeinere Begriffe ersetzen, die aus dem Konzeptgraphen gewonnen wurden. Dies wird einige Zeit in Anspruch nehmen, da wir für jede Substantivphrase einen REST-Aufruf durchführen.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 23,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"w = {}\n",
|
||||
"for x in all_titles:\n",
|
||||
" for noun in TextBlob(x).noun_phrases:\n",
|
||||
" terms = query(noun.replace(' ','%20'))\n",
|
||||
" for term in [u for u in terms.keys() if terms[u]>0.1]:\n",
|
||||
" if term in w:\n",
|
||||
" w[term].append(x)\n",
|
||||
" else:\n",
|
||||
" w[term]=[x]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 24,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"{'city': 9,\n",
|
||||
" 'brand': 4,\n",
|
||||
" 'place': 9,\n",
|
||||
" 'town': 4,\n",
|
||||
" 'factor': 4,\n",
|
||||
" 'film': 4,\n",
|
||||
" 'nation': 11,\n",
|
||||
" 'state': 5,\n",
|
||||
" 'person': 4,\n",
|
||||
" 'organization': 5,\n",
|
||||
" 'publication': 10,\n",
|
||||
" 'market': 5,\n",
|
||||
" 'economy': 4,\n",
|
||||
" 'company': 6,\n",
|
||||
" 'newspaper': 6,\n",
|
||||
" 'relationship': 6}"
|
||||
]
|
||||
},
|
||||
"execution_count": 24,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"{ x:len(w[x]) for x in w.keys() if len(w[x])>3}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 27,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\n",
|
||||
"ECONOMY:\n",
|
||||
"China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n",
|
||||
"Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n",
|
||||
"China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n",
|
||||
"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n",
|
||||
"\n",
|
||||
"NATION:\n",
|
||||
"‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n",
|
||||
"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n",
|
||||
"China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n",
|
||||
"Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n",
|
||||
"Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n",
|
||||
"China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n",
|
||||
"UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n",
|
||||
"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n",
|
||||
"‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n",
|
||||
"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n",
|
||||
"Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n",
|
||||
"\n",
|
||||
"PERSON:\n",
|
||||
"‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n",
|
||||
"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n",
|
||||
"Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n",
|
||||
"‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n",
|
||||
"print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n",
|
||||
"print('\\nPERSON:\\n'+'\\n'.join(w['person']))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Haftungsausschluss**: \nDieses Dokument wurde mit dem KI-Übersetzungsdienst [Co-op Translator](https://github.com/Azure/co-op-translator) übersetzt. Obwohl wir uns um Genauigkeit bemühen, weisen wir darauf hin, dass automatisierte Übersetzungen Fehler oder Ungenauigkeiten enthalten können. Das Originaldokument in seiner ursprünglichen Sprache sollte als maßgebliche Quelle betrachtet werden. Für kritische Informationen wird eine professionelle menschliche Übersetzung empfohlen. Wir übernehmen keine Haftung für Missverständnisse oder Fehlinterpretationen, die sich aus der Nutzung dieser Übersetzung ergeben.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.7.4 64-bit (conda)",
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5"
|
||||
}
|
||||
},
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.5"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "4087f998407d06ceb2947016ba4605d0",
|
||||
"translation_date": "2025-08-31T16:25:38+00:00",
|
||||
"source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb",
|
||||
"language_code": "de"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
|
|
@ -1,8 +1,8 @@
|
|||
<!--
|
||||
CO_OP_TRANSLATOR_METADATA:
|
||||
{
|
||||
"original_hash": "7336583e4630220c835335da640016db",
|
||||
"translation_date": "2025-08-24T09:41:10+00:00",
|
||||
"original_hash": "ba5d1eb353d20d3e7181066b3c424b99",
|
||||
"translation_date": "2025-08-31T15:37:43+00:00",
|
||||
"source_file": "lessons/3-NeuralNetworks/03-Perceptron/lab/README.md",
|
||||
"language_code": "de"
|
||||
}
|
||||
|
|
@ -13,19 +13,21 @@ Laboraufgabe aus dem [AI for Beginners Curriculum](https://github.com/microsoft/
|
|||
|
||||
## Aufgabe
|
||||
|
||||
Verwenden Sie den Code, den wir in dieser Lektion für die binäre Klassifikation von MNIST-Handschriftziffern entwickelt haben, um einen Mehrklassenklassifikator zu erstellen, der jede Ziffer erkennen kann. Berechnen Sie die Klassifikationsgenauigkeit für das Trainings- und Testdatenset und geben Sie die Verwirrungsmatrix aus.
|
||||
Verwenden Sie den Code, den wir in dieser Lektion für die binäre Klassifikation von handgeschriebenen MNIST-Ziffern entwickelt haben, um einen Mehrklassenklassifikator zu erstellen, der jede Ziffer erkennen kann. Berechnen Sie die Klassifikationsgenauigkeit für den Trainings- und Testdatensatz und geben Sie die Konfusionsmatrix aus.
|
||||
|
||||
## Hinweise
|
||||
|
||||
1. Erstellen Sie für jede Ziffer ein Datenset für einen binären Klassifikator von „diese Ziffer vs. alle anderen Ziffern“.
|
||||
1. Erstellen Sie für jede Ziffer einen Datensatz für einen binären Klassifikator, der "diese Ziffer vs. alle anderen Ziffern" unterscheidet.
|
||||
1. Trainieren Sie 10 verschiedene Perzeptrons für die binäre Klassifikation (jeweils eines für jede Ziffer).
|
||||
1. Definieren Sie eine Funktion, die eine Eingabeziffer klassifiziert.
|
||||
|
||||
> **Hinweis**: Wenn wir die Gewichte aller 10 Perzeptrons in einer Matrix kombinieren, sollten wir in der Lage sein, alle 10 Perzeptrons durch eine einzige Matrixmultiplikation auf die Eingabeziffern anzuwenden. Die wahrscheinlichste Ziffer kann dann einfach durch die Anwendung der `argmax`-Operation auf die Ausgabe gefunden werden.
|
||||
> **Tipp**: Wenn wir die Gewichte aller 10 Perzeptrons in einer Matrix kombinieren, sollten wir in der Lage sein, alle 10 Perzeptrons durch eine einzige Matrixmultiplikation auf die Eingabeziffern anzuwenden. Die wahrscheinlichste Ziffer kann dann einfach durch die Anwendung der `argmax`-Operation auf die Ausgabe gefunden werden.
|
||||
|
||||
## Start-Notebook
|
||||
|
||||
Beginnen Sie das Labor, indem Sie [PerceptronMultiClass.ipynb](../../../../../../lessons/3-NeuralNetworks/03-Perceptron/lab/PerceptronMultiClass.ipynb) öffnen.
|
||||
Beginnen Sie das Labor, indem Sie [PerceptronMultiClass.ipynb](PerceptronMultiClass.ipynb) öffnen.
|
||||
|
||||
---
|
||||
|
||||
**Haftungsausschluss**:
|
||||
Dieses Dokument wurde mit dem KI-Übersetzungsdienst [Co-op Translator](https://github.com/Azure/co-op-translator) übersetzt. Obwohl wir uns um Genauigkeit bemühen, weisen wir darauf hin, dass automatisierte Übersetzungen Fehler oder Ungenauigkeiten enthalten können. Das Originaldokument in seiner ursprünglichen Sprache sollte als maßgebliche Quelle betrachtet werden. Für kritische Informationen wird eine professionelle menschliche Übersetzung empfohlen. Wir übernehmen keine Haftung für Missverständnisse oder Fehlinterpretationen, die sich aus der Nutzung dieser Übersetzung ergeben.
|
||||
Dieses Dokument wurde mit dem KI-Übersetzungsdienst [Co-op Translator](https://github.com/Azure/co-op-translator) übersetzt. Obwohl wir uns um Genauigkeit bemühen, beachten Sie bitte, dass automatisierte Übersetzungen Fehler oder Ungenauigkeiten enthalten können. Das Originaldokument in seiner ursprünglichen Sprache sollte als maßgebliche Quelle betrachtet werden. Für kritische Informationen wird eine professionelle menschliche Übersetzung empfohlen. Wir übernehmen keine Haftung für Missverständnisse oder Fehlinterpretationen, die sich aus der Nutzung dieser Übersetzung ergeben.
|
||||
File diff suppressed because one or more lines are too long
|
|
@ -0,0 +1,183 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# MNIST-Ziffernklassifikation mit unserem eigenen Framework\n",
|
||||
"\n",
|
||||
"Laboraufgabe aus dem [AI for Beginners Curriculum](https://github.com/microsoft/ai-for-beginners).\n",
|
||||
"\n",
|
||||
"### Das Dataset einlesen\n",
|
||||
"\n",
|
||||
"Dieser Code lädt das Dataset aus dem Repository im Internet herunter. Alternativ kannst du das Dataset manuell aus dem Verzeichnis `/data` des AI Curriculum-Repos kopieren.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" % Total % Received % Xferd Average Speed Time Time Time Current\n",
|
||||
" Dload Upload Total Spent Left Speed\n",
|
||||
"\n",
|
||||
" 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\n",
|
||||
"100 9.9M 100 9.9M 0 0 9.9M 0 0:00:01 --:--:-- 0:00:01 15.8M\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"!rm *.pkl\n",
|
||||
"!wget https://raw.githubusercontent.com/microsoft/AI-For-Beginners/main/data/mnist.pkl.gz\n",
|
||||
"!gzip -d mnist.pkl.gz"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import pickle\n",
|
||||
"with open('mnist.pkl','rb') as f:\n",
|
||||
" MNIST = pickle.load(f)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"labels = MNIST['Train']['Labels']\n",
|
||||
"data = MNIST['Train']['Features']"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Lass uns sehen, welche Form die Daten haben, die wir haben:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(42000, 784)"
|
||||
]
|
||||
},
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"data.shape"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Aufteilen der Daten\n",
|
||||
"\n",
|
||||
"Wir verwenden Scikit Learn, um die Daten in Trainings- und Testdatensätze aufzuteilen:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Train samples: 33600, test samples: 8400\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from sklearn.model_selection import train_test_split\n",
|
||||
"\n",
|
||||
"features_train, features_test, labels_train, labels_test = train_test_split(data,labels,test_size=0.2)\n",
|
||||
"\n",
|
||||
"print(f\"Train samples: {len(features_train)}, test samples: {len(features_test)}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Anweisungen\n",
|
||||
"\n",
|
||||
"1. Nehmen Sie den Framework-Code aus der Lektion und fügen Sie ihn in dieses Notebook ein oder (noch besser) in ein separates Python-Modul.\n",
|
||||
"1. Definieren und trainieren Sie einen einlagigen Perzeptron, und beobachten Sie dabei die Trainings- und Validierungsgenauigkeit während des Trainings.\n",
|
||||
"1. Versuchen Sie zu verstehen, ob Overfitting aufgetreten ist, und passen Sie die Layer-Parameter an, um die Genauigkeit zu verbessern.\n",
|
||||
"1. Wiederholen Sie die vorherigen Schritte für zwei- und dreilagige Perzeptrons. Experimentieren Sie dabei mit verschiedenen Aktivierungsfunktionen zwischen den Schichten.\n",
|
||||
"1. Versuchen Sie, die folgenden Fragen zu beantworten:\n",
|
||||
" - Beeinflusst die Aktivierungsfunktion zwischen den Schichten die Leistung des Netzwerks?\n",
|
||||
" - Benötigen wir ein zwei- oder dreilagiges Netzwerk für diese Aufgabe?\n",
|
||||
" - Sind während des Trainings des Netzwerks Probleme aufgetreten? Besonders, wenn die Anzahl der Schichten zugenommen hat.\n",
|
||||
" - Wie verhalten sich die Gewichte des Netzwerks während des Trainings? Sie können den maximalen absoluten Wert der Gewichte im Vergleich zur Epoche plotten, um die Beziehung zu verstehen.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Haftungsausschluss**: \nDieses Dokument wurde mit dem KI-Übersetzungsdienst [Co-op Translator](https://github.com/Azure/co-op-translator) übersetzt. Obwohl wir uns um Genauigkeit bemühen, weisen wir darauf hin, dass automatisierte Übersetzungen Fehler oder Ungenauigkeiten enthalten können. Das Originaldokument in seiner ursprünglichen Sprache sollte als maßgebliche Quelle betrachtet werden. Für kritische Informationen wird eine professionelle menschliche Übersetzung empfohlen. Wir übernehmen keine Haftung für Missverständnisse oder Fehlinterpretationen, die sich aus der Nutzung dieser Übersetzung ergeben.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.7.4 64-bit (conda)",
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5"
|
||||
}
|
||||
},
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.5"
|
||||
},
|
||||
"orig_nbformat": 2,
|
||||
"coopTranslator": {
|
||||
"original_hash": "6fa055f484eb5d6bdf41166a356d3abf",
|
||||
"translation_date": "2025-08-31T16:32:04+00:00",
|
||||
"source_file": "lessons/3-NeuralNetworks/04-OwnFramework/lab/MyFW_MNIST.ipynb",
|
||||
"language_code": "de"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
|
|
@ -0,0 +1,102 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Ihr Ziel** ist es, mithilfe von Optical Flow zu bestimmen, welche Teile des Videos Bewegungen nach oben/unten/links/rechts enthalten.\n",
|
||||
"\n",
|
||||
"Beginnen Sie damit, die Videobilder wie in der Vorlesung beschrieben zu erhalten:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Code here"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Berechnen Sie nun dichte optische Flussrahmen wie in der Vorlesung beschrieben und konvertieren Sie den dichten optischen Fluss in Polarkoordinaten:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Code here"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Erstelle ein Histogramm der Richtungen für jedes optische Fluss-Frame. Ein Histogramm zeigt, wie viele Vektoren in bestimmte Kategorien fallen, und es sollte die verschiedenen Bewegungsrichtungen im Frame voneinander trennen.\n",
|
||||
"\n",
|
||||
"> Es könnte sinnvoll sein, alle Vektoren mit einer Magnitude unterhalb eines bestimmten Schwellenwerts auf null zu setzen. Dadurch werden kleine, unerwünschte Bewegungen im Video, wie Augen- und Kopfbewegungen, entfernt.\n",
|
||||
"\n",
|
||||
"Zeichne die Histogramme für einige der Frames.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Code here"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Beim Betrachten von Histogrammen sollte es ziemlich einfach sein, die Bewegungsrichtung zu bestimmen. Sie müssen diejenigen Balken auswählen, die den Richtungen oben/unten/links/rechts entsprechen und die über einem bestimmten Schwellenwert liegen.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Code here"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Herzlichen Glückwunsch! Wenn Sie alle oben genannten Schritte durchgeführt haben, haben Sie das Labor abgeschlossen!\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Haftungsausschluss**: \nDieses Dokument wurde mit dem KI-Übersetzungsdienst [Co-op Translator](https://github.com/Azure/co-op-translator) übersetzt. Obwohl wir uns um Genauigkeit bemühen, beachten Sie bitte, dass automatisierte Übersetzungen Fehler oder Ungenauigkeiten enthalten können. Das Originaldokument in seiner ursprünglichen Sprache sollte als maßgebliche Quelle betrachtet werden. Für kritische Informationen wird eine professionelle menschliche Übersetzung empfohlen. Wir übernehmen keine Haftung für Missverständnisse oder Fehlinterpretationen, die sich aus der Nutzung dieser Übersetzung ergeben.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
},
|
||||
"orig_nbformat": 4,
|
||||
"coopTranslator": {
|
||||
"original_hash": "153d9e417e079bf62f8f693002d0deaf",
|
||||
"translation_date": "2025-08-31T16:08:32+00:00",
|
||||
"source_file": "lessons/4-ComputerVision/06-IntroCV/lab/MovementDetection.ipynb",
|
||||
"language_code": "de"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
|
|
@ -0,0 +1,577 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Textklassifizierungsaufgabe\n",
|
||||
"\n",
|
||||
"Wie bereits erwähnt, konzentrieren wir uns auf eine einfache Textklassifizierungsaufgabe basierend auf dem **AG_NEWS**-Datensatz. Ziel ist es, Nachrichtenüberschriften in eine von vier Kategorien einzuordnen: Welt, Sport, Wirtschaft und Wissenschaft/Technik.\n",
|
||||
"\n",
|
||||
"## Der Datensatz\n",
|
||||
"\n",
|
||||
"Dieser Datensatz ist im [`torchtext`](https://github.com/pytorch/text)-Modul integriert, sodass wir leicht darauf zugreifen können.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"import torchtext\n",
|
||||
"import os\n",
|
||||
"import collections\n",
|
||||
"os.makedirs('./data',exist_ok=True)\n",
|
||||
"train_dataset, test_dataset = torchtext.datasets.AG_NEWS(root='./data')\n",
|
||||
"classes = ['World', 'Sports', 'Business', 'Sci/Tech']"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Hier enthalten `train_dataset` und `test_dataset` Sammlungen, die jeweils Paare aus Label (Nummer der Klasse) und Text zurückgeben, zum Beispiel:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(3,\n",
|
||||
" \"Wall St. Bears Claw Back Into the Black (Reuters) Reuters - Short-sellers, Wall Street's dwindling\\\\band of ultra-cynics, are seeing green again.\")"
|
||||
]
|
||||
},
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"list(train_dataset)[0]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Also, drucken wir die ersten 10 neuen Schlagzeilen aus unserem Datensatz aus:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"**Sci/Tech** -> Wall St. Bears Claw Back Into the Black (Reuters) Reuters - Short-sellers, Wall Street's dwindling\\band of ultra-cynics, are seeing green again.\n",
|
||||
"**Sci/Tech** -> Carlyle Looks Toward Commercial Aerospace (Reuters) Reuters - Private investment firm Carlyle Group,\\which has a reputation for making well-timed and occasionally\\controversial plays in the defense industry, has quietly placed\\its bets on another part of the market.\n",
|
||||
"**Sci/Tech** -> Oil and Economy Cloud Stocks' Outlook (Reuters) Reuters - Soaring crude prices plus worries\\about the economy and the outlook for earnings are expected to\\hang over the stock market next week during the depth of the\\summer doldrums.\n",
|
||||
"**Sci/Tech** -> Iraq Halts Oil Exports from Main Southern Pipeline (Reuters) Reuters - Authorities have halted oil export\\flows from the main pipeline in southern Iraq after\\intelligence showed a rebel militia could strike\\infrastructure, an oil official said on Saturday.\n",
|
||||
"**Sci/Tech** -> Oil prices soar to all-time record, posing new menace to US economy (AFP) AFP - Tearaway world oil prices, toppling records and straining wallets, present a new economic menace barely three months before the US presidential elections.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"for i,x in zip(range(5),train_dataset):\n",
|
||||
" print(f\"**{classes[x[0]]}** -> {x[1]}\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Da Datensätze Iteratoren sind, müssen wir sie in eine Liste umwandeln, wenn wir die Daten mehrmals verwenden möchten:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"train_dataset, test_dataset = torchtext.datasets.AG_NEWS(root='./data')\n",
|
||||
"train_dataset = list(train_dataset)\n",
|
||||
"test_dataset = list(test_dataset)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Tokenisierung\n",
|
||||
"\n",
|
||||
"Nun müssen wir Text in **Zahlen** umwandeln, die als Tensoren dargestellt werden können. Wenn wir eine Wortebene-Darstellung möchten, müssen wir zwei Dinge tun:\n",
|
||||
"* einen **Tokenizer** verwenden, um den Text in **Tokens** zu zerlegen\n",
|
||||
"* ein **Vokabular** dieser Tokens erstellen.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"['he', 'said', 'hello']"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"tokenizer = torchtext.data.utils.get_tokenizer('basic_english')\n",
|
||||
"tokenizer('He said: hello')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"counter = collections.Counter()\n",
|
||||
"for (label, line) in train_dataset:\n",
|
||||
" counter.update(tokenizer(line))\n",
|
||||
"vocab = torchtext.vocab.vocab(counter, min_freq=1)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Mit Vokabular können wir unsere tokenisierte Zeichenkette leicht in eine Zahlenmenge kodieren:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 19,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Vocab size if 95810\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[599, 3279, 97, 1220, 329, 225, 7368]"
|
||||
]
|
||||
},
|
||||
"execution_count": 19,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vocab_size = len(vocab)\n",
|
||||
"print(f\"Vocab size if {vocab_size}\")\n",
|
||||
"\n",
|
||||
"stoi = vocab.get_stoi() # dict to convert tokens to indices\n",
|
||||
"\n",
|
||||
"def encode(x):\n",
|
||||
" return [stoi[s] for s in tokenizer(x)]\n",
|
||||
"\n",
|
||||
"encode('I love to play with my words')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Bag-of-Words Textdarstellung\n",
|
||||
"\n",
|
||||
"Da Wörter Bedeutung repräsentieren, können wir manchmal die Bedeutung eines Textes allein durch die Betrachtung der einzelnen Wörter herausfinden, unabhängig von ihrer Reihenfolge im Satz. Zum Beispiel, wenn wir Nachrichten klassifizieren, deuten Wörter wie *Wetter*, *Schnee* wahrscheinlich auf eine *Wettervorhersage* hin, während Wörter wie *Aktien*, *Dollar* eher auf *Finanznachrichten* hinweisen.\n",
|
||||
"\n",
|
||||
"Die **Bag-of-Words** (BoW)-Vektordarstellung ist die am häufigsten verwendete traditionelle Vektordarstellung. Jedes Wort ist einem Vektorindex zugeordnet, und das Vektorelement enthält die Anzahl der Vorkommen eines Wortes in einem bestimmten Dokument.\n",
|
||||
"\n",
|
||||
" \n",
|
||||
"\n",
|
||||
"> **Hinweis**: Sie können BoW auch als die Summe aller One-Hot-encodierten Vektoren für einzelne Wörter im Text betrachten.\n",
|
||||
"\n",
|
||||
"Im Folgenden finden Sie ein Beispiel, wie man eine Bag-of-Words-Darstellung mit der Scikit Learn Python-Bibliothek erzeugt:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([[1, 1, 0, 2, 0, 0, 0, 0, 0]], dtype=int64)"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from sklearn.feature_extraction.text import CountVectorizer\n",
|
||||
"vectorizer = CountVectorizer()\n",
|
||||
"corpus = [\n",
|
||||
" 'I like hot dogs.',\n",
|
||||
" 'The dog ran fast.',\n",
|
||||
" 'Its hot outside.',\n",
|
||||
" ]\n",
|
||||
"vectorizer.fit_transform(corpus)\n",
|
||||
"vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Um den Bag-of-Words-Vektor aus der Vektordarstellung unseres AG_NEWS-Datensatzes zu berechnen, können wir die folgende Funktion verwenden:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 20,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"tensor([2., 1., 2., ..., 0., 0., 0.])\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vocab_size = len(vocab)\n",
|
||||
"\n",
|
||||
"def to_bow(text,bow_vocab_size=vocab_size):\n",
|
||||
" res = torch.zeros(bow_vocab_size,dtype=torch.float32)\n",
|
||||
" for i in encode(text):\n",
|
||||
" if i<bow_vocab_size:\n",
|
||||
" res[i] += 1\n",
|
||||
" return res\n",
|
||||
"\n",
|
||||
"print(to_bow(train_dataset[0][1]))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Hinweis:** Hier verwenden wir die globale Variable `vocab_size`, um die Standardgröße des Vokabulars festzulegen. Da die Vokabulargröße oft ziemlich groß ist, können wir die Größe des Vokabulars auf die häufigsten Wörter begrenzen. Versuchen Sie, den Wert von `vocab_size` zu verringern und den untenstehenden Code auszuführen, und beobachten Sie, wie sich dies auf die Genauigkeit auswirkt. Sie sollten einen gewissen Rückgang der Genauigkeit erwarten, aber keinen dramatischen, zugunsten einer höheren Leistung.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Training eines BoW-Klassifikators\n",
|
||||
"\n",
|
||||
"Jetzt, da wir gelernt haben, wie man eine Bag-of-Words-Darstellung unseres Textes erstellt, lassen Sie uns einen Klassifikator darauf trainieren. Zunächst müssen wir unser Dataset für das Training so umwandeln, dass alle Positionsvektordarstellungen in Bag-of-Words-Darstellungen konvertiert werden. Dies kann erreicht werden, indem die Funktion `bowify` als Parameter `collate_fn` an den standardmäßigen torch `DataLoader` übergeben wird:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 21,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from torch.utils.data import DataLoader\n",
|
||||
"import numpy as np \n",
|
||||
"\n",
|
||||
"# this collate function gets list of batch_size tuples, and needs to \n",
|
||||
"# return a pair of label-feature tensors for the whole minibatch\n",
|
||||
"def bowify(b):\n",
|
||||
" return (\n",
|
||||
" torch.LongTensor([t[0]-1 for t in b]),\n",
|
||||
" torch.stack([to_bow(t[1]) for t in b])\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"train_loader = DataLoader(train_dataset, batch_size=16, collate_fn=bowify, shuffle=True)\n",
|
||||
"test_loader = DataLoader(test_dataset, batch_size=16, collate_fn=bowify, shuffle=True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Lassen Sie uns nun ein einfaches Klassifikator-Neuronales Netzwerk definieren, das eine lineare Schicht enthält. Die Größe des Eingabevektors entspricht `vocab_size`, und die Ausgabengröße entspricht der Anzahl der Klassen (4). Da wir eine Klassifikationsaufgabe lösen, ist die endgültige Aktivierungsfunktion `LogSoftmax()`.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 22,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"net = torch.nn.Sequential(torch.nn.Linear(vocab_size,4),torch.nn.LogSoftmax(dim=1))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Jetzt definieren wir die standardmäßige PyTorch-Trainingsschleife. Da unser Datensatz ziemlich groß ist, werden wir für unsere Lehrzwecke nur für eine Epoche trainieren und manchmal sogar weniger als eine Epoche (die Angabe des Parameters `epoch_size` ermöglicht es uns, das Training zu begrenzen). Wir würden auch die akkumulierte Trainingsgenauigkeit während des Trainings berichten; die Häufigkeit der Berichterstattung wird mit dem Parameter `report_freq` angegeben.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 24,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def train_epoch(net,dataloader,lr=0.01,optimizer=None,loss_fn = torch.nn.NLLLoss(),epoch_size=None, report_freq=200):\n",
|
||||
" optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr)\n",
|
||||
" net.train()\n",
|
||||
" total_loss,acc,count,i = 0,0,0,0\n",
|
||||
" for labels,features in dataloader:\n",
|
||||
" optimizer.zero_grad()\n",
|
||||
" out = net(features)\n",
|
||||
" loss = loss_fn(out,labels) #cross_entropy(out,labels)\n",
|
||||
" loss.backward()\n",
|
||||
" optimizer.step()\n",
|
||||
" total_loss+=loss\n",
|
||||
" _,predicted = torch.max(out,1)\n",
|
||||
" acc+=(predicted==labels).sum()\n",
|
||||
" count+=len(labels)\n",
|
||||
" i+=1\n",
|
||||
" if i%report_freq==0:\n",
|
||||
" print(f\"{count}: acc={acc.item()/count}\")\n",
|
||||
" if epoch_size and count>epoch_size:\n",
|
||||
" break\n",
|
||||
" return total_loss.item()/count, acc.item()/count"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 25,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.8028125\n",
|
||||
"6400: acc=0.8371875\n",
|
||||
"9600: acc=0.8534375\n",
|
||||
"12800: acc=0.85765625\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(0.026090790722161722, 0.8620069296375267)"
|
||||
]
|
||||
},
|
||||
"execution_count": 25,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"train_epoch(net,train_loader,epoch_size=15000)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## BiGrams, TriGrams und N-Grams\n",
|
||||
"\n",
|
||||
"Eine Einschränkung des Bag-of-Words-Ansatzes ist, dass einige Wörter Teil von mehrwortigen Ausdrücken sind. Zum Beispiel hat das Wort 'Hot Dog' eine völlig andere Bedeutung als die Wörter 'hot' und 'dog' in anderen Kontexten. Wenn wir die Wörter 'hot' und 'dog' immer durch die gleichen Vektoren darstellen, kann das unser Modell verwirren.\n",
|
||||
"\n",
|
||||
"Um dies zu lösen, werden **N-Gram-Darstellungen** häufig in Methoden der Dokumentklassifikation verwendet, bei denen die Häufigkeit jedes Wortes, Zwei-Wort- oder Drei-Wort-Ausdrucks ein nützliches Merkmal für das Training von Klassifikatoren ist. In der Bigram-Darstellung fügen wir beispielsweise alle Wortpaare zusätzlich zu den ursprünglichen Wörtern dem Vokabular hinzu.\n",
|
||||
"\n",
|
||||
"Unten ist ein Beispiel, wie man eine Bigram-Bag-of-Words-Darstellung mit Scikit Learn generiert:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 26,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Vocabulary:\n",
|
||||
" {'i': 7, 'like': 11, 'hot': 4, 'dogs': 2, 'i like': 8, 'like hot': 12, 'hot dogs': 5, 'the': 16, 'dog': 0, 'ran': 14, 'fast': 3, 'the dog': 17, 'dog ran': 1, 'ran fast': 15, 'its': 9, 'outside': 13, 'its hot': 10, 'hot outside': 6}\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([[1, 0, 1, 0, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],\n",
|
||||
" dtype=int64)"
|
||||
]
|
||||
},
|
||||
"execution_count": 26,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"bigram_vectorizer = CountVectorizer(ngram_range=(1, 2), token_pattern=r'\\b\\w+\\b', min_df=1)\n",
|
||||
"corpus = [\n",
|
||||
" 'I like hot dogs.',\n",
|
||||
" 'The dog ran fast.',\n",
|
||||
" 'Its hot outside.',\n",
|
||||
" ]\n",
|
||||
"bigram_vectorizer.fit_transform(corpus)\n",
|
||||
"print(\"Vocabulary:\\n\",bigram_vectorizer.vocabulary_)\n",
|
||||
"bigram_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Der Hauptnachteil des N-Gramm-Ansatzes ist, dass die Größe des Vokabulars extrem schnell wächst. In der Praxis müssen wir die N-Gramm-Darstellung mit einigen Techniken zur Dimensionsreduktion kombinieren, wie zum Beispiel *Embeddings*, die wir in der nächsten Einheit besprechen werden.\n",
|
||||
"\n",
|
||||
"Um die N-Gramm-Darstellung in unserem **AG News**-Datensatz zu verwenden, müssen wir ein spezielles N-Gramm-Vokabular erstellen:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 27,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Bigram vocabulary length = 1308842\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"counter = collections.Counter()\n",
|
||||
"for (label, line) in train_dataset:\n",
|
||||
" l = tokenizer(line)\n",
|
||||
" counter.update(torchtext.data.utils.ngrams_iterator(l,ngrams=2))\n",
|
||||
" \n",
|
||||
"bi_vocab = torchtext.vocab.vocab(counter, min_freq=1)\n",
|
||||
"\n",
|
||||
"print(\"Bigram vocabulary length = \",len(bi_vocab))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Wir könnten denselben Code wie oben verwenden, um den Klassifikator zu trainieren, allerdings wäre das sehr speicherineffizient. In der nächsten Einheit werden wir einen Bigramm-Klassifikator mithilfe von Embeddings trainieren.\n",
|
||||
"\n",
|
||||
"> **Hinweis:** Du kannst nur die N-Gramme beibehalten, die im Text häufiger als eine bestimmte Anzahl vorkommen. Das stellt sicher, dass seltene Bigramme ausgelassen werden und die Dimensionalität erheblich reduziert wird. Um dies zu erreichen, setze den Parameter `min_freq` auf einen höheren Wert und beobachte, wie sich die Länge des Vokabulars verändert.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Termfrequenz-Inverse Dokumentfrequenz TF-IDF\n",
|
||||
"\n",
|
||||
"In der BoW-Darstellung werden Wortvorkommen gleichmäßig gewichtet, unabhängig vom Wort selbst. Es ist jedoch offensichtlich, dass häufige Wörter wie *ein*, *in* usw. für die Klassifikation viel weniger wichtig sind als spezialisierte Begriffe. Tatsächlich sind bei den meisten NLP-Aufgaben einige Wörter relevanter als andere.\n",
|
||||
"\n",
|
||||
"**TF-IDF** steht für **Termfrequenz–Inverse Dokumentfrequenz**. Es ist eine Variation des Bag-of-Words-Modells, bei der anstelle eines binären 0/1-Wertes, der das Auftreten eines Wortes in einem Dokument anzeigt, ein Gleitkommawert verwendet wird, der mit der Häufigkeit des Wortvorkommens im Korpus zusammenhängt.\n",
|
||||
"\n",
|
||||
"Formal wird das Gewicht $w_{ij}$ eines Wortes $i$ im Dokument $j$ wie folgt definiert:\n",
|
||||
"$$\n",
|
||||
"w_{ij} = tf_{ij}\\times\\log({N\\over df_i})\n",
|
||||
"$$\n",
|
||||
"wobei\n",
|
||||
"* $tf_{ij}$ die Anzahl der Vorkommen von $i$ in $j$ ist, also der BoW-Wert, den wir zuvor gesehen haben\n",
|
||||
"* $N$ die Anzahl der Dokumente in der Sammlung ist\n",
|
||||
"* $df_i$ die Anzahl der Dokumente ist, die das Wort $i$ in der gesamten Sammlung enthalten\n",
|
||||
"\n",
|
||||
"Der TF-IDF-Wert $w_{ij}$ steigt proportional zur Häufigkeit, mit der ein Wort in einem Dokument erscheint, und wird durch die Anzahl der Dokumente im Korpus, die das Wort enthalten, ausgeglichen. Dies hilft, den Umstand zu berücksichtigen, dass einige Wörter häufiger vorkommen als andere. Wenn beispielsweise ein Wort in *jedem* Dokument der Sammlung vorkommt, gilt $df_i=N$, und $w_{ij}=0$, und diese Begriffe würden vollständig ignoriert.\n",
|
||||
"\n",
|
||||
"Mit Scikit Learn können Sie ganz einfach eine TF-IDF-Vektorisierung von Text erstellen:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 28,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([[0.43381609, 0. , 0.43381609, 0. , 0.65985664,\n",
|
||||
" 0.43381609, 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. ]])"
|
||||
]
|
||||
},
|
||||
"execution_count": 28,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
|
||||
"vectorizer = TfidfVectorizer(ngram_range=(1,2))\n",
|
||||
"vectorizer.fit_transform(corpus)\n",
|
||||
"vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Fazit\n",
|
||||
"\n",
|
||||
"Auch wenn TF-IDF-Darstellungen Wörtern unterschiedliche Gewichtungen basierend auf ihrer Häufigkeit zuweisen, sind sie nicht in der Lage, Bedeutung oder Reihenfolge darzustellen. Wie der berühmte Linguist J. R. Firth 1935 sagte: „Die vollständige Bedeutung eines Wortes ist immer kontextabhängig, und keine Untersuchung der Bedeutung ohne Kontext kann ernst genommen werden.“ Später im Kurs werden wir lernen, wie man kontextuelle Informationen aus Texten mithilfe von Sprachmodellen erfasst.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Haftungsausschluss**: \nDieses Dokument wurde mit dem KI-Übersetzungsdienst [Co-op Translator](https://github.com/Azure/co-op-translator) übersetzt. Obwohl wir uns um Genauigkeit bemühen, beachten Sie bitte, dass automatisierte Übersetzungen Fehler oder Ungenauigkeiten enthalten können. Das Originaldokument in seiner ursprünglichen Sprache sollte als maßgebliche Quelle betrachtet werden. Für kritische Informationen wird eine professionelle menschliche Übersetzung empfohlen. Wir übernehmen keine Haftung für Missverständnisse oder Fehlinterpretationen, die sich aus der Nutzung dieser Übersetzung ergeben.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.8.12 ('py38')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "7b9040985e748e4e2d4c689892456ad7",
|
||||
"translation_date": "2025-08-31T17:16:16+00:00",
|
||||
"source_file": "lessons/5-NLP/13-TextRep/TextRepresentationPyTorch.ipynb",
|
||||
"language_code": "de"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
|
|
@ -0,0 +1,647 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Textklassifizierungsaufgabe\n",
|
||||
"\n",
|
||||
"In diesem Modul beginnen wir mit einer einfachen Textklassifizierungsaufgabe basierend auf dem **[AG_NEWS](http://www.di.unipi.it/~gulli/AG_corpus_of_news_articles.html)**-Datensatz: Wir werden Nachrichtenüberschriften in eine von vier Kategorien einordnen: Welt, Sport, Wirtschaft und Wissenschaft/Technik.\n",
|
||||
"\n",
|
||||
"## Der Datensatz\n",
|
||||
"\n",
|
||||
"Um den Datensatz zu laden, verwenden wir die **[TensorFlow Datasets](https://www.tensorflow.org/datasets)**-API.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import tensorflow as tf\n",
|
||||
"from tensorflow import keras\n",
|
||||
"import tensorflow_datasets as tfds\n",
|
||||
"\n",
|
||||
"# In this tutorial, we will be training a lot of models. In order to use GPU memory cautiously,\n",
|
||||
"# we will set tensorflow option to grow GPU memory allocation when required.\n",
|
||||
"physical_devices = tf.config.list_physical_devices('GPU') \n",
|
||||
"if len(physical_devices)>0:\n",
|
||||
" tf.config.experimental.set_memory_growth(physical_devices[0], True)\n",
|
||||
"\n",
|
||||
"dataset = tfds.load('ag_news_subset')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Wir können nun auf die Trainings- und Testteile des Datensatzes zugreifen, indem wir `dataset['train']` und `dataset['test']` verwenden:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Length of train dataset = 120000\n",
|
||||
"Length of test dataset = 7600\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"ds_train = dataset['train']\n",
|
||||
"ds_test = dataset['test']\n",
|
||||
"\n",
|
||||
"print(f\"Length of train dataset = {len(ds_train)}\")\n",
|
||||
"print(f\"Length of test dataset = {len(ds_test)}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Lassen Sie uns die ersten 10 neuen Schlagzeilen aus unserem Datensatz ausdrucken:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3 (Sci/Tech) -> b'AMD Debuts Dual-Core Opteron Processor' b'AMD #39;s new dual-core Opteron chip is designed mainly for corporate computing applications, including databases, Web services, and financial transactions.'\n",
|
||||
"1 (Sports) -> b\"Wood's Suspension Upheld (Reuters)\" b'Reuters - Major League Baseball\\\\Monday announced a decision on the appeal filed by Chicago Cubs\\\\pitcher Kerry Wood regarding a suspension stemming from an\\\\incident earlier this season.'\n",
|
||||
"2 (Business) -> b'Bush reform may have blue states seeing red' b'President Bush #39;s quot;revenue-neutral quot; tax reform needs losers to balance its winners, and people claiming the federal deduction for state and local taxes may be in administration planners #39; sights, news reports say.'\n",
|
||||
"3 (Sci/Tech) -> b\"'Halt science decline in schools'\" b'Britain will run out of leading scientists unless science education is improved, says Professor Colin Pillinger.'\n",
|
||||
"1 (Sports) -> b'Gerrard leaves practice' b'London, England (Sports Network) - England midfielder Steven Gerrard injured his groin late in Thursday #39;s training session, but is hopeful he will be ready for Saturday #39;s World Cup qualifier against Austria.'\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"classes = ['World', 'Sports', 'Business', 'Sci/Tech']\n",
|
||||
"\n",
|
||||
"for i,x in zip(range(5),ds_train):\n",
|
||||
" print(f\"{x['label']} ({classes[x['label']]}) -> {x['title']} {x['description']}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Textvektorisierung\n",
|
||||
"\n",
|
||||
"Nun müssen wir Text in **Zahlen** umwandeln, die als Tensoren dargestellt werden können. Wenn wir eine Wortebene-Darstellung möchten, müssen wir zwei Dinge tun:\n",
|
||||
"\n",
|
||||
"* Einen **Tokenizer** verwenden, um den Text in **Token** zu zerlegen.\n",
|
||||
"* Ein **Vokabular** dieser Token erstellen.\n",
|
||||
"\n",
|
||||
"### Begrenzung der Vokabulargröße\n",
|
||||
"\n",
|
||||
"Im Beispiel des AG News-Datensatzes ist die Vokabulargröße ziemlich groß, mehr als 100.000 Wörter. Allgemein gesprochen benötigen wir keine Wörter, die selten im Text vorkommen — nur wenige Sätze enthalten sie, und das Modell wird nicht von ihnen lernen. Daher ist es sinnvoll, die Vokabulargröße auf eine kleinere Anzahl zu begrenzen, indem ein Argument an den Vektorisierer-Konstruktor übergeben wird:\n",
|
||||
"\n",
|
||||
"Beide dieser Schritte können mit der **TextVectorization**-Schicht durchgeführt werden. Lassen Sie uns das Vektorisierungsobjekt instanziieren und anschließend die `adapt`-Methode aufrufen, um den gesamten Text zu durchlaufen und ein Vokabular zu erstellen:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"vocab_size = 50000\n",
|
||||
"vectorizer = keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size)\n",
|
||||
"vectorizer.adapt(ds_train.take(500).map(lambda x: x['title']+' '+x['description']))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Hinweis**: Wir verwenden nur einen Teil des gesamten Datensatzes, um ein Vokabular zu erstellen. Dies tun wir, um die Ausführungszeit zu verkürzen und Sie nicht warten zu lassen. Allerdings gehen wir das Risiko ein, dass einige Wörter aus dem gesamten Datensatz nicht in das Vokabular aufgenommen werden und während des Trainings ignoriert werden. Die Verwendung der gesamten Vokabulargröße und das Durchlaufen des gesamten Datensatzes während `adapt` sollte die endgültige Genauigkeit erhöhen, jedoch nicht signifikant.\n",
|
||||
"\n",
|
||||
"Nun können wir auf das tatsächliche Vokabular zugreifen:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"['', '[UNK]', 'the', 'to', 'a', 'in', 'of', 'and', 'on', 'for']\n",
|
||||
"Length of vocabulary: 5335\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vocab = vectorizer.get_vocabulary()\n",
|
||||
"vocab_size = len(vocab)\n",
|
||||
"print(vocab[:10])\n",
|
||||
"print(f\"Length of vocabulary: {vocab_size}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Mit dem Vektorisierer können wir problemlos jeden Text in eine Zahlenmenge kodieren:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tf.Tensor: shape=(7,), dtype=int64, numpy=array([ 112, 3695, 3, 304, 11, 1041, 1], dtype=int64)>"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vectorizer('I love to play with my words')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Bag-of-words Textdarstellung\n",
|
||||
"\n",
|
||||
"Da Wörter Bedeutung vermitteln, können wir manchmal die Bedeutung eines Textes allein durch die Betrachtung der einzelnen Wörter erkennen, unabhängig von ihrer Reihenfolge im Satz. Zum Beispiel deuten beim Klassifizieren von Nachrichten Wörter wie *Wetter* und *Schnee* wahrscheinlich auf eine *Wettervorhersage* hin, während Wörter wie *Aktien* und *Dollar* eher auf *Finanznachrichten* hindeuten.\n",
|
||||
"\n",
|
||||
"Die **Bag-of-words** (BoW)-Vektordarstellung ist die einfachste und am leichtesten verständliche traditionelle Vektordarstellung. Jedes Wort wird einem Vektorindex zugeordnet, und ein Vektorelement enthält die Anzahl der Vorkommen jedes Wortes in einem bestimmten Dokument.\n",
|
||||
"\n",
|
||||
" \n",
|
||||
"\n",
|
||||
"> **Note**: Sie können sich BoW auch als die Summe aller One-Hot-encodierten Vektoren für die einzelnen Wörter im Text vorstellen.\n",
|
||||
"\n",
|
||||
"Unten sehen Sie ein Beispiel, wie man mit der Scikit Learn Python-Bibliothek eine Bag-of-words-Darstellung erzeugen kann:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([[1, 1, 0, 2, 0, 0, 0, 0, 0]], dtype=int64)"
|
||||
]
|
||||
},
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from sklearn.feature_extraction.text import CountVectorizer\n",
|
||||
"sc_vectorizer = CountVectorizer()\n",
|
||||
"corpus = [\n",
|
||||
" 'I like hot dogs.',\n",
|
||||
" 'The dog ran fast.',\n",
|
||||
" 'Its hot outside.',\n",
|
||||
" ]\n",
|
||||
"sc_vectorizer.fit_transform(corpus)\n",
|
||||
"sc_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Wir können auch den Keras-Vektorisierer verwenden, den wir oben definiert haben, indem wir jede Wortnummer in eine One-Hot-Codierung umwandeln und alle diese Vektoren addieren:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([0., 5., 0., ..., 0., 0., 0.], dtype=float32)"
|
||||
]
|
||||
},
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def to_bow(text):\n",
|
||||
" return tf.reduce_sum(tf.one_hot(vectorizer(text),vocab_size),axis=0)\n",
|
||||
"\n",
|
||||
"to_bow('My dog likes hot dogs on a hot day.').numpy()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Hinweis**: Es könnte Sie überraschen, dass das Ergebnis sich vom vorherigen Beispiel unterscheidet. Der Grund dafür ist, dass im Keras-Beispiel die Länge des Vektors der Größe des Vokabulars entspricht, das aus dem gesamten AG News-Datensatz erstellt wurde, während wir im Scikit-Learn-Beispiel das Vokabular spontan aus dem Beispieltext erstellt haben.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Training des BoW-Klassifikators\n",
|
||||
"\n",
|
||||
"Jetzt, da wir gelernt haben, wie man die Bag-of-Words-Darstellung unseres Textes erstellt, können wir einen Klassifikator trainieren, der diese verwendet. Zuerst müssen wir unser Dataset in eine Bag-of-Words-Darstellung umwandeln. Dies kann mit der `map`-Funktion auf folgende Weise erreicht werden:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"batch_size = 128\n",
|
||||
"\n",
|
||||
"ds_train_bow = ds_train.map(lambda x: (to_bow(x['title']+x['description']),x['label'])).batch(batch_size)\n",
|
||||
"ds_test_bow = ds_test.map(lambda x: (to_bow(x['title']+x['description']),x['label'])).batch(batch_size)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Lassen Sie uns nun ein einfaches Klassifikator-Neuronales Netzwerk definieren, das eine lineare Schicht enthält. Die Eingabegröße ist `vocab_size`, und die Ausgabengröße entspricht der Anzahl der Klassen (4). Da wir eine Klassifikationsaufgabe lösen, ist die endgültige Aktivierungsfunktion **softmax**:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"938/938 [==============================] - 66s 70ms/step - loss: 0.6144 - acc: 0.8427 - val_loss: 0.4416 - val_acc: 0.8697\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.History at 0x20c70a947f0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model = keras.models.Sequential([\n",
|
||||
" keras.layers.Dense(4,activation='softmax',input_shape=(vocab_size,))\n",
|
||||
"])\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
|
||||
"model.fit(ds_train_bow,validation_data=ds_test_bow)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Da wir 4 Klassen haben, ist eine Genauigkeit von über 80 % ein gutes Ergebnis.\n",
|
||||
"\n",
|
||||
"## Einen Klassifikator als ein Netzwerk trainieren\n",
|
||||
"\n",
|
||||
"Da der Vektorisierer ebenfalls eine Keras-Schicht ist, können wir ein Netzwerk definieren, das ihn einschließt, und es vollständig trainieren. Auf diese Weise müssen wir den Datensatz nicht mit `map` vektorisieren, sondern können den ursprünglichen Datensatz direkt an den Eingang des Netzwerks übergeben.\n",
|
||||
"\n",
|
||||
"> **Hinweis**: Wir müssten dennoch `map` auf unseren Datensatz anwenden, um Felder aus Wörterbüchern (wie `title`, `description` und `label`) in Tupel umzuwandeln. Wenn wir die Daten jedoch von der Festplatte laden, können wir von Anfang an einen Datensatz mit der erforderlichen Struktur erstellen.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"model\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
" Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
" input_1 (InputLayer) [(None, 1)] 0 \n",
|
||||
" \n",
|
||||
" text_vectorization (TextVec (None, None) 0 \n",
|
||||
" torization) \n",
|
||||
" \n",
|
||||
" tf.one_hot (TFOpLambda) (None, None, 5335) 0 \n",
|
||||
" \n",
|
||||
" tf.math.reduce_sum (TFOpLam (None, 5335) 0 \n",
|
||||
" bda) \n",
|
||||
" \n",
|
||||
" dense_2 (Dense) (None, 4) 21344 \n",
|
||||
" \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 21,344\n",
|
||||
"Trainable params: 21,344\n",
|
||||
"Non-trainable params: 0\n",
|
||||
"_________________________________________________________________\n",
|
||||
"938/938 [==============================] - 73s 77ms/step - loss: 0.6057 - acc: 0.8414 - val_loss: 0.4202 - val_acc: 0.8736\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.History at 0x20c721521f0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def extract_text(x):\n",
|
||||
" return x['title']+' '+x['description']\n",
|
||||
"\n",
|
||||
"def tupelize(x):\n",
|
||||
" return (extract_text(x),x['label'])\n",
|
||||
"\n",
|
||||
"inp = keras.Input(shape=(1,),dtype=tf.string)\n",
|
||||
"x = vectorizer(inp)\n",
|
||||
"x = tf.reduce_sum(tf.one_hot(x,vocab_size),axis=1)\n",
|
||||
"out = keras.layers.Dense(4,activation='softmax')(x)\n",
|
||||
"model = keras.models.Model(inp,out)\n",
|
||||
"model.summary()\n",
|
||||
"\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Bigramme, Trigramme und N-Gramme\n",
|
||||
"\n",
|
||||
"Eine Einschränkung des Bag-of-Words-Ansatzes ist, dass einige Wörter Teil von mehrwortigen Ausdrücken sind. Zum Beispiel hat das Wort 'Hot Dog' eine völlig andere Bedeutung als die Wörter 'hot' und 'dog' in anderen Kontexten. Wenn wir die Wörter 'hot' und 'dog' immer mit denselben Vektoren darstellen, kann dies unser Modell verwirren.\n",
|
||||
"\n",
|
||||
"Um dies zu lösen, werden häufig **N-Gramm-Darstellungen** in Methoden der Dokumentklassifikation verwendet, bei denen die Häufigkeit jedes Wortes, Zwei-Wort- oder Drei-Wort-Ausdrucks eine nützliche Eigenschaft für das Training von Klassifikatoren darstellt. In Bigramm-Darstellungen fügen wir beispielsweise alle Wortpaare zusätzlich zu den ursprünglichen Wörtern dem Vokabular hinzu.\n",
|
||||
"\n",
|
||||
"Unten sehen Sie ein Beispiel, wie man eine Bigramm-Bag-of-Words-Darstellung mit Scikit Learn erzeugt:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Vocabulary:\n",
|
||||
" {'i': 7, 'like': 11, 'hot': 4, 'dogs': 2, 'i like': 8, 'like hot': 12, 'hot dogs': 5, 'the': 16, 'dog': 0, 'ran': 14, 'fast': 3, 'the dog': 17, 'dog ran': 1, 'ran fast': 15, 'its': 9, 'outside': 13, 'its hot': 10, 'hot outside': 6}\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([[1, 0, 1, 0, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],\n",
|
||||
" dtype=int64)"
|
||||
]
|
||||
},
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"bigram_vectorizer = CountVectorizer(ngram_range=(1, 2), token_pattern=r'\\b\\w+\\b', min_df=1)\n",
|
||||
"corpus = [\n",
|
||||
" 'I like hot dogs.',\n",
|
||||
" 'The dog ran fast.',\n",
|
||||
" 'Its hot outside.',\n",
|
||||
" ]\n",
|
||||
"bigram_vectorizer.fit_transform(corpus)\n",
|
||||
"print(\"Vocabulary:\\n\",bigram_vectorizer.vocabulary_)\n",
|
||||
"bigram_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Der Hauptnachteil des n-Gramm-Ansatzes ist, dass die Größe des Vokabulars extrem schnell wächst. In der Praxis müssen wir die n-Gramm-Repräsentation mit einer Technik zur Dimensionsreduktion kombinieren, wie zum Beispiel *Embeddings*, die wir in der nächsten Einheit besprechen werden.\n",
|
||||
"\n",
|
||||
"Um eine n-Gramm-Repräsentation in unserem **AG News**-Datensatz zu verwenden, müssen wir den Parameter `ngrams` an den `TextVectorization`-Konstruktor übergeben. Die Länge eines Bigramm-Vokabulars ist **deutlich größer**, in unserem Fall sind es mehr als 1,3 Millionen Tokens! Daher ist es sinnvoll, auch die Bigramm-Tokens auf eine vernünftige Anzahl zu begrenzen.\n",
|
||||
"\n",
|
||||
"Wir könnten denselben Code wie oben verwenden, um den Klassifikator zu trainieren, allerdings wäre das sehr speicherineffizient. In der nächsten Einheit werden wir den Bigramm-Klassifikator mithilfe von Embeddings trainieren. In der Zwischenzeit kannst du mit dem Training eines Bigramm-Klassifikators in diesem Notebook experimentieren und sehen, ob du eine höhere Genauigkeit erzielen kannst.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Automatisches Berechnen von BoW-Vektoren\n",
|
||||
"\n",
|
||||
"Im obigen Beispiel haben wir BoW-Vektoren manuell berechnet, indem wir die One-Hot-Codierungen einzelner Wörter summiert haben. Die neueste Version von TensorFlow ermöglicht es uns jedoch, BoW-Vektoren automatisch zu berechnen, indem wir den Parameter `output_mode='count'` an den Konstruktor des Vektorisierers übergeben. Dies vereinfacht das Definieren und Trainieren unseres Modells erheblich:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Training vectorizer\n",
|
||||
"938/938 [==============================] - 7s 7ms/step - loss: 0.5929 - acc: 0.8486 - val_loss: 0.4168 - val_acc: 0.8772\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.History at 0x20c725217c0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model = keras.models.Sequential([\n",
|
||||
" keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size,output_mode='count'),\n",
|
||||
" keras.layers.Dense(4,input_shape=(vocab_size,), activation='softmax')\n",
|
||||
"])\n",
|
||||
"print(\"Training vectorizer\")\n",
|
||||
"model.layers[0].adapt(ds_train.take(500).map(extract_text))\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Termfrequenz - inverse Dokumentfrequenz (TF-IDF)\n",
|
||||
"\n",
|
||||
"In der BoW-Darstellung werden Wortvorkommen unabhängig vom Wort selbst mit derselben Technik gewichtet. Es ist jedoch offensichtlich, dass häufige Wörter wie *a* und *in* für die Klassifikation viel weniger wichtig sind als spezialisierte Begriffe. Bei den meisten NLP-Aufgaben sind einige Wörter relevanter als andere.\n",
|
||||
"\n",
|
||||
"**TF-IDF** steht für **Termfrequenz - inverse Dokumentfrequenz**. Es handelt sich um eine Variation des Bag-of-Words-Modells, bei der anstelle eines binären 0/1-Wertes, der das Auftreten eines Wortes in einem Dokument angibt, ein Gleitkommawert verwendet wird, der mit der Häufigkeit des Wortvorkommens im Korpus zusammenhängt.\n",
|
||||
"\n",
|
||||
"Formal wird das Gewicht $w_{ij}$ eines Wortes $i$ im Dokument $j$ wie folgt definiert:\n",
|
||||
"$$\n",
|
||||
"w_{ij} = tf_{ij}\\times\\log({N\\over df_i})\n",
|
||||
"$$\n",
|
||||
"wobei\n",
|
||||
"* $tf_{ij}$ die Anzahl der Vorkommen von $i$ in $j$ ist, also der BoW-Wert, den wir zuvor gesehen haben\n",
|
||||
"* $N$ die Anzahl der Dokumente in der Sammlung ist\n",
|
||||
"* $df_i$ die Anzahl der Dokumente ist, die das Wort $i$ in der gesamten Sammlung enthalten\n",
|
||||
"\n",
|
||||
"Der TF-IDF-Wert $w_{ij}$ steigt proportional zur Häufigkeit, mit der ein Wort in einem Dokument erscheint, und wird durch die Anzahl der Dokumente im Korpus, die das Wort enthalten, ausgeglichen. Dies hilft, den Umstand zu berücksichtigen, dass einige Wörter häufiger vorkommen als andere. Wenn beispielsweise ein Wort in *jedem* Dokument der Sammlung vorkommt, gilt $df_i=N$, und $w_{ij}=0$, und diese Begriffe würden vollständig ignoriert.\n",
|
||||
"\n",
|
||||
"Mit Scikit Learn können Sie ganz einfach eine TF-IDF-Vektorisierung von Text erstellen:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 16,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([[0.43381609, 0. , 0.43381609, 0. , 0.65985664,\n",
|
||||
" 0.43381609, 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. ]])"
|
||||
]
|
||||
},
|
||||
"execution_count": 16,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
|
||||
"vectorizer = TfidfVectorizer(ngram_range=(1,2))\n",
|
||||
"vectorizer.fit_transform(corpus)\n",
|
||||
"vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"In Keras kann die `TextVectorization`-Schicht automatisch TF-IDF-Frequenzen berechnen, indem der Parameter `output_mode='tf-idf'` übergeben wird. Lassen Sie uns den oben verwendeten Code wiederholen, um zu sehen, ob die Verwendung von TF-IDF die Genauigkeit erhöht:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 17,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Training vectorizer\n",
|
||||
"938/938 [==============================] - 12s 12ms/step - loss: 0.4197 - acc: 0.8662 - val_loss: 0.3432 - val_acc: 0.8849\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.History at 0x20c729dfd30>"
|
||||
]
|
||||
},
|
||||
"execution_count": 17,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model = keras.models.Sequential([\n",
|
||||
" keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size,output_mode='tf-idf'),\n",
|
||||
" keras.layers.Dense(4,input_shape=(vocab_size,), activation='softmax')\n",
|
||||
"])\n",
|
||||
"print(\"Training vectorizer\")\n",
|
||||
"model.layers[0].adapt(ds_train.take(500).map(extract_text))\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Fazit\n",
|
||||
"\n",
|
||||
"Auch wenn TF-IDF-Darstellungen Häufigkeitsgewichte für verschiedene Wörter bereitstellen, sind sie nicht in der Lage, Bedeutung oder Reihenfolge abzubilden. Wie der berühmte Linguist J. R. Firth 1935 sagte: \"Die vollständige Bedeutung eines Wortes ist immer kontextabhängig, und keine Untersuchung der Bedeutung ohne Kontext kann ernst genommen werden.\" Später im Kurs werden wir lernen, wie man kontextuelle Informationen aus Texten mithilfe von Sprachmodellen erfasst.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Haftungsausschluss**: \nDieses Dokument wurde mit dem KI-Übersetzungsdienst [Co-op Translator](https://github.com/Azure/co-op-translator) übersetzt. Obwohl wir uns um Genauigkeit bemühen, beachten Sie bitte, dass automatisierte Übersetzungen Fehler oder Ungenauigkeiten enthalten können. Das Originaldokument in seiner ursprünglichen Sprache sollte als maßgebliche Quelle betrachtet werden. Für kritische Informationen wird eine professionelle menschliche Übersetzung empfohlen. Wir übernehmen keine Haftung für Missverständnisse oder Fehlinterpretationen, die sich aus der Nutzung dieser Übersetzung ergeben.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5"
|
||||
},
|
||||
"kernel_info": {
|
||||
"name": "conda-env-py37_tensorflow-py"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "py37_tensorflow",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"nteract": {
|
||||
"version": "nteract-front-end@1.0.0"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "19b43951d55b377a76209c24c1f017e4",
|
||||
"translation_date": "2025-08-31T17:18:38+00:00",
|
||||
"source_file": "lessons/5-NLP/13-TextRep/TextRepresentationTF.ipynb",
|
||||
"language_code": "de"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
|
|
@ -0,0 +1,726 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Einbettungen\n",
|
||||
"\n",
|
||||
"In unserem vorherigen Beispiel haben wir mit hochdimensionalen Bag-of-Words-Vektoren der Länge `vocab_size` gearbeitet und explizit von niedrigdimensionalen Positionsdarstellungsvektoren in spärliche One-Hot-Darstellungen umgewandelt. Diese One-Hot-Darstellung ist nicht speichereffizient. Außerdem wird jedes Wort unabhängig von den anderen behandelt, d.h. One-Hot-codierte Vektoren drücken keine semantische Ähnlichkeit zwischen Wörtern aus.\n",
|
||||
"\n",
|
||||
"In dieser Einheit werden wir weiterhin den **News AG**-Datensatz untersuchen. Zu Beginn laden wir die Daten und holen einige Definitionen aus dem vorherigen Notebook.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Loading dataset...\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"d:\\WORK\\ai-for-beginners\\5-NLP\\14-Embeddings\\data\\train.csv: 29.5MB [00:01, 18.8MB/s] \n",
|
||||
"d:\\WORK\\ai-for-beginners\\5-NLP\\14-Embeddings\\data\\test.csv: 1.86MB [00:00, 11.2MB/s] \n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Building vocab...\n",
|
||||
"Vocab size = 95812\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"import torchtext\n",
|
||||
"import numpy as np\n",
|
||||
"from torchnlp import *\n",
|
||||
"train_dataset, test_dataset, classes, vocab = load_dataset()\n",
|
||||
"vocab_size = len(vocab)\n",
|
||||
"print(\"Vocab size = \",vocab_size)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Was ist ein Embedding?\n",
|
||||
"\n",
|
||||
"Die Idee des **Embeddings** besteht darin, Wörter durch niedrigdimensionale, dichte Vektoren darzustellen, die in gewisser Weise die semantische Bedeutung eines Wortes widerspiegeln. Später werden wir besprechen, wie man sinnvolle Wort-Embeddings erstellt, aber vorerst betrachten wir Embeddings einfach als eine Methode, die Dimensionalität eines Wortvektors zu reduzieren.\n",
|
||||
"\n",
|
||||
"Eine Embedding-Schicht nimmt also ein Wort als Eingabe und erzeugt einen Ausgabevektor mit der angegebenen `embedding_size`. In gewisser Weise ähnelt sie einer `Linear`-Schicht, aber anstatt einen One-Hot-codierten Vektor zu verwenden, kann sie eine Wortnummer als Eingabe akzeptieren.\n",
|
||||
"\n",
|
||||
"Indem wir die Embedding-Schicht als erste Schicht in unserem Netzwerk verwenden, können wir vom Bag-of-Words-Modell zum **Embedding-Bag-Modell** wechseln. Dabei wird jedes Wort in unserem Text zunächst in das entsprechende Embedding umgewandelt, und anschließend wird eine Aggregationsfunktion wie `sum`, `average` oder `max` über alle diese Embeddings berechnet.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Unser neuronales Klassifikationsnetzwerk beginnt mit einer Embedding-Schicht, gefolgt von einer Aggregationsschicht und einem linearen Klassifikator darüber:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class EmbedClassifier(torch.nn.Module):\n",
|
||||
" def __init__(self, vocab_size, embed_dim, num_class):\n",
|
||||
" super().__init__()\n",
|
||||
" self.embedding = torch.nn.Embedding(vocab_size, embed_dim)\n",
|
||||
" self.fc = torch.nn.Linear(embed_dim, num_class)\n",
|
||||
"\n",
|
||||
" def forward(self, x):\n",
|
||||
" x = self.embedding(x)\n",
|
||||
" x = torch.mean(x,dim=1)\n",
|
||||
" return self.fc(x)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Umgang mit variabler Sequenzgröße\n",
|
||||
"\n",
|
||||
"Aufgrund dieser Architektur müssen Minibatches für unser Netzwerk auf eine bestimmte Weise erstellt werden. In der vorherigen Einheit, bei der Verwendung von Bag-of-Words, hatten alle BoW-Tensoren in einem Minibatch die gleiche Größe `vocab_size`, unabhängig von der tatsächlichen Länge unserer Textsequenz. Sobald wir zu Wort-Embeddings wechseln, haben wir eine variable Anzahl von Wörtern in jeder Textprobe, und beim Kombinieren dieser Proben in Minibatches müssen wir eine Auffüllung (Padding) anwenden.\n",
|
||||
"\n",
|
||||
"Dies kann durch die gleiche Technik erreicht werden, indem eine `collate_fn`-Funktion an die Datenquelle übergeben wird:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def padify(b):\n",
|
||||
" # b is the list of tuples of length batch_size\n",
|
||||
" # - first element of a tuple = label, \n",
|
||||
" # - second = feature (text sequence)\n",
|
||||
" # build vectorized sequence\n",
|
||||
" v = [encode(x[1]) for x in b]\n",
|
||||
" # first, compute max length of a sequence in this minibatch\n",
|
||||
" l = max(map(len,v))\n",
|
||||
" return ( # tuple of two tensors - labels and features\n",
|
||||
" torch.LongTensor([t[0]-1 for t in b]),\n",
|
||||
" torch.stack([torch.nn.functional.pad(torch.tensor(t),(0,l-len(t)),mode='constant',value=0) for t in v])\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=padify, shuffle=True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Training des Einbettungs-Klassifikators\n",
|
||||
"\n",
|
||||
"Nun, da wir einen geeigneten Dataloader definiert haben, können wir das Modell mit der Trainingsfunktion trainieren, die wir in der vorherigen Einheit definiert haben:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.6415625\n",
|
||||
"6400: acc=0.6865625\n",
|
||||
"9600: acc=0.7103125\n",
|
||||
"12800: acc=0.726953125\n",
|
||||
"16000: acc=0.739375\n",
|
||||
"19200: acc=0.75046875\n",
|
||||
"22400: acc=0.7572321428571429\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(0.889799795315499, 0.7623160588611644)"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"net = EmbedClassifier(vocab_size,32,len(classes)).to(device)\n",
|
||||
"train_epoch(net,train_loader, lr=1, epoch_size=25000)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Hinweis**: Wir trainieren hier nur mit 25.000 Datensätzen (weniger als eine vollständige Epoche) aus Zeitgründen, aber Sie können das Training fortsetzen, eine Funktion schreiben, um über mehrere Epochen zu trainieren, und mit dem Lernratenparameter experimentieren, um eine höhere Genauigkeit zu erreichen. Sie sollten in der Lage sein, eine Genauigkeit von etwa 90 % zu erreichen.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### EmbeddingBag-Schicht und Darstellung von Sequenzen variabler Länge\n",
|
||||
"\n",
|
||||
"In der vorherigen Architektur mussten wir alle Sequenzen auf die gleiche Länge auffüllen, um sie in ein Minibatch einzupassen. Dies ist jedoch nicht die effizienteste Methode, um Sequenzen variabler Länge darzustellen – ein alternativer Ansatz wäre die Verwendung eines **Offset-Vektors**, der die Offsets aller Sequenzen in einem großen Vektor speichert.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"> **Hinweis**: Auf dem obigen Bild zeigen wir eine Zeichenfolge, aber in unserem Beispiel arbeiten wir mit Wortsequenzen. Das allgemeine Prinzip, Sequenzen mit einem Offset-Vektor darzustellen, bleibt jedoch dasselbe.\n",
|
||||
"\n",
|
||||
"Um mit der Offset-Darstellung zu arbeiten, verwenden wir die [`EmbeddingBag`](https://pytorch.org/docs/stable/generated/torch.nn.EmbeddingBag.html)-Schicht. Sie ähnelt der `Embedding`-Schicht, nimmt jedoch einen Inhaltsvektor und einen Offset-Vektor als Eingabe und enthält außerdem eine Aggregationsschicht, die `mean`, `sum` oder `max` sein kann.\n",
|
||||
"\n",
|
||||
"Hier ist ein modifiziertes Netzwerk, das `EmbeddingBag` verwendet:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class EmbedClassifier(torch.nn.Module):\n",
|
||||
" def __init__(self, vocab_size, embed_dim, num_class):\n",
|
||||
" super().__init__()\n",
|
||||
" self.embedding = torch.nn.EmbeddingBag(vocab_size, embed_dim)\n",
|
||||
" self.fc = torch.nn.Linear(embed_dim, num_class)\n",
|
||||
"\n",
|
||||
" def forward(self, text, off):\n",
|
||||
" x = self.embedding(text, off)\n",
|
||||
" return self.fc(x)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Um den Datensatz für das Training vorzubereiten, müssen wir eine Umrechnungsfunktion bereitstellen, die den Offset-Vektor vorbereitet:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def offsetify(b):\n",
|
||||
" # first, compute data tensor from all sequences\n",
|
||||
" x = [torch.tensor(encode(t[1])) for t in b]\n",
|
||||
" # now, compute the offsets by accumulating the tensor of sequence lengths\n",
|
||||
" o = [0] + [len(t) for t in x]\n",
|
||||
" o = torch.tensor(o[:-1]).cumsum(dim=0)\n",
|
||||
" return ( \n",
|
||||
" torch.LongTensor([t[0]-1 for t in b]), # labels\n",
|
||||
" torch.cat(x), # text \n",
|
||||
" o\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=offsetify, shuffle=True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Beachten Sie, dass unser Netzwerk im Gegensatz zu allen vorherigen Beispielen jetzt zwei Parameter akzeptiert: Datenvektor und Offsetvektor, die unterschiedliche Größen haben. Ebenso liefert uns unser Datenlader jetzt 3 Werte anstelle von 2: Sowohl Text- als auch Offsetvektoren werden als Features bereitgestellt. Daher müssen wir unsere Trainingsfunktion geringfügig anpassen, um dies zu berücksichtigen:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.6153125\n",
|
||||
"6400: acc=0.6615625\n",
|
||||
"9600: acc=0.6932291666666667\n",
|
||||
"12800: acc=0.715078125\n",
|
||||
"16000: acc=0.7270625\n",
|
||||
"19200: acc=0.7382291666666667\n",
|
||||
"22400: acc=0.7486160714285715\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(22.771553103007037, 0.7551983365323096)"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"net = EmbedClassifier(vocab_size,32,len(classes)).to(device)\n",
|
||||
"\n",
|
||||
"def train_epoch_emb(net,dataloader,lr=0.01,optimizer=None,loss_fn = torch.nn.CrossEntropyLoss(),epoch_size=None, report_freq=200):\n",
|
||||
" optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr)\n",
|
||||
" loss_fn = loss_fn.to(device)\n",
|
||||
" net.train()\n",
|
||||
" total_loss,acc,count,i = 0,0,0,0\n",
|
||||
" for labels,text,off in dataloader:\n",
|
||||
" optimizer.zero_grad()\n",
|
||||
" labels,text,off = labels.to(device), text.to(device), off.to(device)\n",
|
||||
" out = net(text, off)\n",
|
||||
" loss = loss_fn(out,labels) #cross_entropy(out,labels)\n",
|
||||
" loss.backward()\n",
|
||||
" optimizer.step()\n",
|
||||
" total_loss+=loss\n",
|
||||
" _,predicted = torch.max(out,1)\n",
|
||||
" acc+=(predicted==labels).sum()\n",
|
||||
" count+=len(labels)\n",
|
||||
" i+=1\n",
|
||||
" if i%report_freq==0:\n",
|
||||
" print(f\"{count}: acc={acc.item()/count}\")\n",
|
||||
" if epoch_size and count>epoch_size:\n",
|
||||
" break\n",
|
||||
" return total_loss.item()/count, acc.item()/count\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"train_epoch_emb(net,train_loader, lr=4, epoch_size=25000)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Semantische Einbettungen: Word2Vec\n",
|
||||
"\n",
|
||||
"In unserem vorherigen Beispiel hat die Einbettungsschicht des Modells gelernt, Wörter in Vektorrepräsentationen umzuwandeln. Diese Repräsentationen hatten jedoch nicht viel semantische Bedeutung. Es wäre wünschenswert, solche Vektorrepräsentationen zu lernen, bei denen ähnliche Wörter oder Synonyme Vektoren entsprechen, die in Bezug auf eine bestimmte Vektordistanz (z. B. euklidische Distanz) nahe beieinander liegen.\n",
|
||||
"\n",
|
||||
"Um dies zu erreichen, müssen wir unser Einbettungsmodell auf eine große Textsammlung in einer spezifischen Weise vortrainieren. Eine der ersten Methoden, um semantische Einbettungen zu trainieren, wird [Word2Vec](https://en.wikipedia.org/wiki/Word2vec) genannt. Sie basiert auf zwei Hauptarchitekturen, die verwendet werden, um eine verteilte Repräsentation von Wörtern zu erzeugen:\n",
|
||||
"\n",
|
||||
"- **Continuous Bag-of-Words** (CBoW) — In dieser Architektur trainieren wir das Modell darauf, ein Wort aus dem umgebenden Kontext vorherzusagen. Gegeben das N-Gramm $(W_{-2},W_{-1},W_0,W_1,W_2)$, ist das Ziel des Modells, $W_0$ aus $(W_{-2},W_{-1},W_1,W_2)$ vorherzusagen.\n",
|
||||
"- **Continuous Skip-Gram** ist das Gegenteil von CBoW. Das Modell verwendet das umgebende Fenster von Kontextwörtern, um das aktuelle Wort vorherzusagen.\n",
|
||||
"\n",
|
||||
"CBoW ist schneller, während Skip-Gram langsamer ist, aber eine bessere Repräsentation für seltene Wörter liefert.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Um mit Word2Vec-Einbettungen zu experimentieren, die auf dem Google-News-Datensatz vortrainiert wurden, können wir die **gensim**-Bibliothek verwenden. Unten finden wir die Wörter, die 'neural' am ähnlichsten sind.\n",
|
||||
"\n",
|
||||
"> **Hinweis:** Wenn Sie zum ersten Mal Wortvektoren erstellen, kann das Herunterladen einige Zeit in Anspruch nehmen!\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import gensim.downloader as api\n",
|
||||
"w2v = api.load('word2vec-google-news-300')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"neuronal -> 0.7804799675941467\n",
|
||||
"neurons -> 0.7326500415802002\n",
|
||||
"neural_circuits -> 0.7252851724624634\n",
|
||||
"neuron -> 0.7174385190010071\n",
|
||||
"cortical -> 0.6941086649894714\n",
|
||||
"brain_circuitry -> 0.6923246383666992\n",
|
||||
"synaptic -> 0.6699118614196777\n",
|
||||
"neural_circuitry -> 0.6638563275337219\n",
|
||||
"neurochemical -> 0.6555314064025879\n",
|
||||
"neuronal_activity -> 0.6531826257705688\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"for w,p in w2v.most_similar('neural'):\n",
|
||||
" print(f\"{w} -> {p}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Wir können auch Vektoreinbettungen aus dem Wort berechnen, die zur Schulung des Klassifikationsmodells verwendet werden (wir zeigen nur die ersten 20 Komponenten des Vektors zur besseren Übersicht):\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([ 0.01226807, 0.06225586, 0.10693359, 0.05810547, 0.23828125,\n",
|
||||
" 0.03686523, 0.05151367, -0.20703125, 0.01989746, 0.10058594,\n",
|
||||
" -0.03759766, -0.1015625 , -0.15820312, -0.08105469, -0.0390625 ,\n",
|
||||
" -0.05053711, 0.16015625, 0.2578125 , 0.10058594, -0.25976562],\n",
|
||||
" dtype=float32)"
|
||||
]
|
||||
},
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"w2v.word_vec('play')[:20]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Das Großartige an semantischen Einbettungen ist, dass man die Vektorkodierung manipulieren kann, um die Semantik zu ändern. Zum Beispiel können wir nach einem Wort suchen, dessen Vektorrepräsentation so nah wie möglich an den Wörtern *König* und *Frau* liegt und so weit wie möglich vom Wort *Mann* entfernt ist:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"('queen', 0.7118192911148071)"
|
||||
]
|
||||
},
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"w2v.most_similar(positive=['king','woman'],negative=['man'])[0]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Sowohl CBoW als auch Skip-Grams sind „prädiktive“ Einbettungen, da sie nur lokale Kontexte berücksichtigen. Word2Vec nutzt den globalen Kontext nicht aus.\n",
|
||||
"\n",
|
||||
"**FastText** baut auf Word2Vec auf, indem es Vektorrepräsentationen für jedes Wort und die Zeichen-n-Gramme innerhalb jedes Wortes lernt. Die Werte der Repräsentationen werden dann bei jedem Trainingsschritt zu einem Vektor gemittelt. Obwohl dies eine Menge zusätzlicher Berechnungen während des Pre-Trainings erfordert, ermöglicht es den Wort-Einbettungen, Subwort-Informationen zu kodieren.\n",
|
||||
"\n",
|
||||
"Eine andere Methode, **GloVe**, nutzt die Idee der Ko-Vorkommensmatrix und verwendet neuronale Methoden, um die Ko-Vorkommensmatrix in ausdrucksstärkere und nicht-lineare Wortvektoren zu zerlegen.\n",
|
||||
"\n",
|
||||
"Du kannst mit dem Beispiel experimentieren, indem du die Einbettungen auf FastText und GloVe änderst, da gensim mehrere verschiedene Modelle für Wort-Einbettungen unterstützt.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Verwendung vortrainierter Einbettungen in PyTorch\n",
|
||||
"\n",
|
||||
"Wir können das obige Beispiel so anpassen, dass die Matrix in unserer Einbettungsschicht mit semantischen Einbettungen wie Word2Vec vorab gefüllt wird. Dabei müssen wir berücksichtigen, dass die Vokabulare der vortrainierten Einbettungen und unseres Textkorpus wahrscheinlich nicht übereinstimmen. Daher werden wir die Gewichte für die fehlenden Wörter mit Zufallswerten initialisieren:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Embedding size: 300\n",
|
||||
"Populating matrix, this will take some time...Done, found 41080 words, 54732 words missing\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"embed_size = len(w2v.get_vector('hello'))\n",
|
||||
"print(f'Embedding size: {embed_size}')\n",
|
||||
"\n",
|
||||
"net = EmbedClassifier(vocab_size,embed_size,len(classes))\n",
|
||||
"\n",
|
||||
"print('Populating matrix, this will take some time...',end='')\n",
|
||||
"found, not_found = 0,0\n",
|
||||
"for i,w in enumerate(vocab.get_itos()):\n",
|
||||
" try:\n",
|
||||
" net.embedding.weight[i].data = torch.tensor(w2v.get_vector(w))\n",
|
||||
" found+=1\n",
|
||||
" except:\n",
|
||||
" net.embedding.weight[i].data = torch.normal(0.0,1.0,(embed_size,))\n",
|
||||
" not_found+=1\n",
|
||||
"\n",
|
||||
"print(f\"Done, found {found} words, {not_found} words missing\")\n",
|
||||
"net = net.to(device)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Jetzt lassen Sie uns unser Modell trainieren. Beachten Sie, dass die Zeit, die zum Trainieren des Modells benötigt wird, aufgrund der größeren Größe der Einbettungsschicht und damit der deutlich höheren Anzahl von Parametern erheblich länger ist als im vorherigen Beispiel. Außerdem müssen wir möglicherweise unser Modell mit mehr Beispielen trainieren, wenn wir Überanpassung vermeiden wollen.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.6359375\n",
|
||||
"6400: acc=0.68109375\n",
|
||||
"9600: acc=0.7067708333333333\n",
|
||||
"12800: acc=0.723671875\n",
|
||||
"16000: acc=0.73625\n",
|
||||
"19200: acc=0.7463541666666667\n",
|
||||
"22400: acc=0.7560714285714286\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(214.1013875559821, 0.7626759436980166)"
|
||||
]
|
||||
},
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"train_epoch_emb(net,train_loader, lr=4, epoch_size=25000)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"In unserem Fall sehen wir keinen großen Anstieg der Genauigkeit, was wahrscheinlich auf sehr unterschiedliche Vokabulare zurückzuführen ist. \n",
|
||||
"Um das Problem der unterschiedlichen Vokabulare zu lösen, können wir eine der folgenden Lösungen verwenden: \n",
|
||||
"* Das Word2Vec-Modell mit unserem Vokabular neu trainieren \n",
|
||||
"* Unser Dataset mit dem Vokabular des vortrainierten Word2Vec-Modells laden. Das Vokabular, das zum Laden des Datasets verwendet wird, kann während des Ladens angegeben werden. \n",
|
||||
"\n",
|
||||
"Der letztere Ansatz scheint einfacher zu sein, insbesondere weil das PyTorch-Framework `torchtext` integrierte Unterstützung für Embeddings bietet. Wir können beispielsweise ein GloVe-basiertes Vokabular auf folgende Weise instanziieren: \n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"100%|█████████▉| 399999/400000 [00:15<00:00, 25411.14it/s]\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vocab = torchtext.vocab.GloVe(name='6B', dim=50)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Das geladene Vokabular bietet die folgenden grundlegenden Operationen:\n",
|
||||
"* Das `vocab.stoi`-Wörterbuch ermöglicht es uns, ein Wort in seinen Index im Wörterbuch umzuwandeln.\n",
|
||||
"* `vocab.itos` macht das Gegenteil – es wandelt eine Zahl in ein Wort um.\n",
|
||||
"* `vocab.vectors` ist das Array der Einbettungsvektoren. Um die Einbettung eines Wortes `s` zu erhalten, müssen wir `vocab.vectors[vocab.stoi[s]]` verwenden.\n",
|
||||
"\n",
|
||||
"Hier ist ein Beispiel für die Manipulation von Einbettungen, um die Gleichung **kind-man+woman = queen** zu demonstrieren (ich musste den Koeffizienten ein wenig anpassen, damit es funktioniert):\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"'queen'"
|
||||
]
|
||||
},
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# get the vector corresponding to kind-man+woman\n",
|
||||
"qvec = vocab.vectors[vocab.stoi['king']]-vocab.vectors[vocab.stoi['man']]+1.3*vocab.vectors[vocab.stoi['woman']]\n",
|
||||
"# find the index of the closest embedding vector \n",
|
||||
"d = torch.sum((vocab.vectors-qvec)**2,dim=1)\n",
|
||||
"min_idx = torch.argmin(d)\n",
|
||||
"# find the corresponding word\n",
|
||||
"vocab.itos[min_idx]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Um den Klassifikator mit diesen Einbettungen zu trainieren, müssen wir zunächst unseren Datensatz mit dem GloVe-Vokabular codieren:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 16,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def offsetify(b):\n",
|
||||
" # first, compute data tensor from all sequences\n",
|
||||
" x = [torch.tensor(encode(t[1],voc=vocab)) for t in b] # pass the instance of vocab to encode function!\n",
|
||||
" # now, compute the offsets by accumulating the tensor of sequence lengths\n",
|
||||
" o = [0] + [len(t) for t in x]\n",
|
||||
" o = torch.tensor(o[:-1]).cumsum(dim=0)\n",
|
||||
" return ( \n",
|
||||
" torch.LongTensor([t[0]-1 for t in b]), # labels\n",
|
||||
" torch.cat(x), # text \n",
|
||||
" o\n",
|
||||
" )"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Wie wir oben gesehen haben, werden alle Vektoreinbettungen in der `vocab.vectors`-Matrix gespeichert. Dadurch wird es sehr einfach, diese Gewichte durch einfaches Kopieren in die Gewichte der Einbettungsschicht zu laden:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 17,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"net = EmbedClassifier(len(vocab),len(vocab.vectors[0]),len(classes))\n",
|
||||
"net.embedding.weight.data = vocab.vectors\n",
|
||||
"net = net.to(device)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Lassen Sie uns nun unser Modell trainieren und sehen, ob wir bessere Ergebnisse erzielen:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 18,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.6271875\n",
|
||||
"6400: acc=0.68078125\n",
|
||||
"9600: acc=0.7030208333333333\n",
|
||||
"12800: acc=0.71984375\n",
|
||||
"16000: acc=0.7346875\n",
|
||||
"19200: acc=0.7455729166666667\n",
|
||||
"22400: acc=0.7529464285714286\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(35.53972978646833, 0.7575175943698017)"
|
||||
]
|
||||
},
|
||||
"execution_count": 18,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=offsetify, shuffle=True)\n",
|
||||
"train_epoch_emb(net,train_loader, lr=4, epoch_size=25000)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Einer der Gründe, warum wir keine signifikante Steigerung der Genauigkeit sehen, liegt darin, dass einige Wörter aus unserem Datensatz im vortrainierten GloVe-Vokabular fehlen und daher im Wesentlichen ignoriert werden. Um dies zu überwinden, können wir eigene Embeddings auf unserem Datensatz trainieren.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Kontextuelle Einbettungen\n",
|
||||
"\n",
|
||||
"Eine zentrale Einschränkung traditioneller vortrainierter Einbettungsrepräsentationen wie Word2Vec ist das Problem der Bedeutungsunterscheidung von Wörtern. Während vortrainierte Einbettungen einen Teil der Bedeutung von Wörtern im Kontext erfassen können, wird jede mögliche Bedeutung eines Wortes in derselben Einbettung kodiert. Dies kann in nachgelagerten Modellen zu Problemen führen, da viele Wörter, wie das Wort „play“, je nach Kontext unterschiedliche Bedeutungen haben.\n",
|
||||
"\n",
|
||||
"Zum Beispiel hat das Wort „play“ in den folgenden zwei Sätzen eine ganz unterschiedliche Bedeutung:\n",
|
||||
"- Ich war in einem **Theaterstück**.\n",
|
||||
"- John möchte mit seinen Freunden **spielen**.\n",
|
||||
"\n",
|
||||
"Die oben genannten vortrainierten Einbettungen repräsentieren beide Bedeutungen des Wortes „play“ in derselben Einbettung. Um diese Einschränkung zu überwinden, müssen wir Einbettungen basierend auf dem **Sprachmodell** erstellen, das auf einem großen Textkorpus trainiert wurde und *versteht*, wie Wörter in unterschiedlichen Kontexten zusammengefügt werden können. Die Diskussion über kontextuelle Einbettungen liegt außerhalb des Umfangs dieses Tutorials, aber wir werden darauf zurückkommen, wenn wir in der nächsten Einheit über Sprachmodelle sprechen.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Haftungsausschluss**: \nDieses Dokument wurde mit dem KI-Übersetzungsdienst [Co-op Translator](https://github.com/Azure/co-op-translator) übersetzt. Obwohl wir uns um Genauigkeit bemühen, beachten Sie bitte, dass automatisierte Übersetzungen Fehler oder Ungenauigkeiten enthalten können. Das Originaldokument in seiner ursprünglichen Sprache sollte als maßgebliche Quelle betrachtet werden. Für kritische Informationen wird eine professionelle menschliche Übersetzung empfohlen. Wir übernehmen keine Haftung für Missverständnisse oder Fehlinterpretationen, die sich aus der Nutzung dieser Übersetzung ergeben.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "py37_pytorch",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "f50b026abce5cf36783a560ea72cb9b1",
|
||||
"translation_date": "2025-08-31T17:13:30+00:00",
|
||||
"source_file": "lessons/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb",
|
||||
"language_code": "de"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
|
|
@ -0,0 +1,695 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Einbettungen\n",
|
||||
"\n",
|
||||
"In unserem vorherigen Beispiel haben wir mit hochdimensionalen Bag-of-Words-Vektoren der Länge `vocab_size` gearbeitet und die niedrigdimensionalen Positionsdarstellungsvektoren explizit in eine spärliche One-Hot-Darstellung umgewandelt. Diese One-Hot-Darstellung ist nicht speichereffizient. Außerdem wird jedes Wort unabhängig von den anderen behandelt, sodass One-Hot-codierte Vektoren keine semantischen Ähnlichkeiten zwischen Wörtern ausdrücken.\n",
|
||||
"\n",
|
||||
"In dieser Einheit werden wir weiterhin den **News AG**-Datensatz untersuchen. Zu Beginn laden wir die Daten und holen einige Definitionen aus der vorherigen Einheit.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import tensorflow as tf\n",
|
||||
"from tensorflow import keras\n",
|
||||
"import tensorflow_datasets as tfds\n",
|
||||
"import numpy as np\n",
|
||||
"\n",
|
||||
"ds_train, ds_test = tfds.load('ag_news_subset').values()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Was ist ein Embedding?\n",
|
||||
"\n",
|
||||
"Die Idee eines **Embeddings** besteht darin, Wörter mithilfe von niedrigdimensionalen, dichten Vektoren darzustellen, die die semantische Bedeutung des Wortes widerspiegeln. Später werden wir besprechen, wie man sinnvolle Wort-Embeddings erstellt, aber vorerst betrachten wir Embeddings einfach als eine Möglichkeit, die Dimensionalität eines Wortvektors zu reduzieren.\n",
|
||||
"\n",
|
||||
"Eine Embedding-Schicht nimmt also ein Wort als Eingabe und erzeugt einen Ausgabevektor mit einer festgelegten `embedding_size`. In gewisser Weise ähnelt sie einer `Dense`-Schicht, aber anstatt einen One-Hot-codierten Vektor als Eingabe zu verwenden, kann sie eine Wortnummer verarbeiten.\n",
|
||||
"\n",
|
||||
"Indem wir eine Embedding-Schicht als erste Schicht in unserem Netzwerk verwenden, können wir von einem Bag-of-Words-Modell zu einem **Embedding-Bag**-Modell wechseln. Dabei wird jedes Wort in unserem Text zunächst in das entsprechende Embedding umgewandelt, und anschließend wird eine Aggregationsfunktion über alle diese Embeddings berechnet, wie z. B. `sum`, `average` oder `max`.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Unser Klassifikator-Neuronales-Netzwerk besteht aus den folgenden Schichten:\n",
|
||||
"\n",
|
||||
"* `TextVectorization`-Schicht, die einen String als Eingabe nimmt und einen Tensor mit Token-Nummern erzeugt. Wir werden eine angemessene Vokabulargröße `vocab_size` festlegen und weniger häufig verwendete Wörter ignorieren. Die Eingabeform wird 1 sein, und die Ausgabeform wird $n$ sein, da wir $n$ Token als Ergebnis erhalten, von denen jedes Zahlen von 0 bis `vocab_size` enthält.\n",
|
||||
"* `Embedding`-Schicht, die $n$ Zahlen nimmt und jede Zahl in einen dichten Vektor einer bestimmten Länge (in unserem Beispiel 100) reduziert. Der Eingabetensor der Form $n$ wird somit in einen $n\\times 100$-Tensor umgewandelt.\n",
|
||||
"* Aggregationsschicht, die den Durchschnitt dieses Tensors entlang der ersten Achse berechnet, d. h. sie berechnet den Durchschnitt aller $n$ Eingabetensoren, die verschiedenen Wörtern entsprechen. Um diese Schicht zu implementieren, verwenden wir eine `Lambda`-Schicht und übergeben ihr die Funktion zur Berechnung des Durchschnitts. Die Ausgabe wird die Form 100 haben und die numerische Darstellung der gesamten Eingabesequenz sein.\n",
|
||||
"* Abschließender `Dense`-linearer Klassifikator.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"sequential\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
" Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
" text_vectorization (TextVec (None, None) 0 \n",
|
||||
" torization) \n",
|
||||
" \n",
|
||||
" embedding (Embedding) (None, None, 100) 3000000 \n",
|
||||
" \n",
|
||||
" lambda (Lambda) (None, 100) 0 \n",
|
||||
" \n",
|
||||
" dense (Dense) (None, 4) 404 \n",
|
||||
" \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 3,000,404\n",
|
||||
"Trainable params: 3,000,404\n",
|
||||
"Non-trainable params: 0\n",
|
||||
"_________________________________________________________________\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vocab_size = 30000\n",
|
||||
"batch_size = 128\n",
|
||||
"\n",
|
||||
"vectorizer = keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size,input_shape=(1,))\n",
|
||||
"\n",
|
||||
"model = keras.models.Sequential([\n",
|
||||
" vectorizer, \n",
|
||||
" keras.layers.Embedding(vocab_size,100),\n",
|
||||
" keras.layers.Lambda(lambda x: tf.reduce_mean(x,axis=1)),\n",
|
||||
" keras.layers.Dense(4, activation='softmax')\n",
|
||||
"])\n",
|
||||
"model.summary()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Im `summary`-Ausdruck entspricht die erste Tensor-Dimension `None` in der **output shape**-Spalte der Minibatch-Größe, und die zweite entspricht der Länge der Token-Sequenz. Alle Token-Sequenzen im Minibatch haben unterschiedliche Längen. Wir werden im nächsten Abschnitt besprechen, wie man damit umgeht.\n",
|
||||
"\n",
|
||||
"Jetzt lass uns das Netzwerk trainieren:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Training vectorizer\n",
|
||||
"938/938 [==============================] - 20s 20ms/step - loss: 0.7891 - acc: 0.8155 - val_loss: 0.4470 - val_acc: 0.8642\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.History at 0x22255515100>"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def extract_text(x):\n",
|
||||
" return x['title']+' '+x['description']\n",
|
||||
"\n",
|
||||
"def tupelize(x):\n",
|
||||
" return (extract_text(x),x['label'])\n",
|
||||
"\n",
|
||||
"print(\"Training vectorizer\")\n",
|
||||
"vectorizer.adapt(ds_train.take(500).map(extract_text))\n",
|
||||
"\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'])\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"nteract": {
|
||||
"transient": {
|
||||
"deleting": false
|
||||
}
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"> **Hinweis**: Wir erstellen den Vektorisierer basierend auf einem Teil der Daten. Dies wird durchgeführt, um den Prozess zu beschleunigen, und es könnte dazu führen, dass nicht alle Token aus unserem Text im Vokabular enthalten sind. In diesem Fall würden diese Token ignoriert, was zu einer leicht geringeren Genauigkeit führen kann. Allerdings liefert ein Teil des Textes in der Praxis oft eine gute Schätzung des Vokabulars.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Umgang mit variablen Sequenzgrößen\n",
|
||||
"\n",
|
||||
"Lassen Sie uns verstehen, wie das Training in Minibatches abläuft. Im obigen Beispiel hat der Eingabetensor die Dimension 1, und wir verwenden 128 lange Minibatches, sodass die tatsächliche Größe des Tensors $128 \\times 1$ beträgt. Allerdings ist die Anzahl der Tokens in jedem Satz unterschiedlich. Wenn wir die `TextVectorization`-Schicht auf eine einzelne Eingabe anwenden, ist die Anzahl der zurückgegebenen Tokens unterschiedlich, abhängig davon, wie der Text tokenisiert wird:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"tf.Tensor([ 1 45], shape=(2,), dtype=int64)\n",
|
||||
"tf.Tensor([ 112 1271 1 3 1747 158], shape=(6,), dtype=int64)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print(vectorizer('Hello, world!'))\n",
|
||||
"print(vectorizer('I am glad to meet you!'))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Wenn wir den Vektorisierer jedoch auf mehrere Sequenzen anwenden, muss er einen Tensor mit rechteckiger Form erzeugen, sodass er nicht verwendete Elemente mit dem PAD-Token (was in unserem Fall null ist) auffüllt:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tf.Tensor: shape=(2, 6), dtype=int64, numpy=\n",
|
||||
"array([[ 1, 45, 0, 0, 0, 0],\n",
|
||||
" [ 112, 1271, 1, 3, 1747, 158]], dtype=int64)>"
|
||||
]
|
||||
},
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vectorizer(['Hello, world!','I am glad to meet you!'])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Hier können wir die Einbettungen sehen:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([[[ 1.53059261e-02, 6.80514947e-02, 3.14026810e-02, ...,\n",
|
||||
" -8.92002955e-02, 1.52911525e-04, -5.65562584e-02],\n",
|
||||
" [ 2.57456154e-01, 2.79364467e-01, -2.03605562e-01, ...,\n",
|
||||
" -2.07474351e-01, 8.31158683e-02, -2.03911960e-01],\n",
|
||||
" [ 3.98201384e-02, -8.03454965e-03, 2.39790026e-02, ...,\n",
|
||||
" -7.18549127e-04, 2.66963355e-02, -4.30646613e-02],\n",
|
||||
" [ 3.98201384e-02, -8.03454965e-03, 2.39790026e-02, ...,\n",
|
||||
" -7.18549127e-04, 2.66963355e-02, -4.30646613e-02],\n",
|
||||
" [ 3.98201384e-02, -8.03454965e-03, 2.39790026e-02, ...,\n",
|
||||
" -7.18549127e-04, 2.66963355e-02, -4.30646613e-02],\n",
|
||||
" [ 3.98201384e-02, -8.03454965e-03, 2.39790026e-02, ...,\n",
|
||||
" -7.18549127e-04, 2.66963355e-02, -4.30646613e-02]],\n",
|
||||
"\n",
|
||||
" [[ 1.89674050e-01, 2.61548996e-01, -3.67433839e-02, ...,\n",
|
||||
" -2.07366899e-01, -1.05442435e-01, -2.36952081e-01],\n",
|
||||
" [ 6.16133213e-02, 1.80511594e-01, 9.77298319e-02, ...,\n",
|
||||
" -5.46628237e-02, -1.07340455e-01, -1.06589928e-01],\n",
|
||||
" [ 1.53059261e-02, 6.80514947e-02, 3.14026810e-02, ...,\n",
|
||||
" -8.92002955e-02, 1.52911525e-04, -5.65562584e-02],\n",
|
||||
" [-4.84890305e-02, -8.41715634e-02, 1.51529670e-01, ...,\n",
|
||||
" 1.28192469e-01, -7.77286515e-02, 1.26041949e-01],\n",
|
||||
" [-4.17212099e-02, -5.60694858e-02, 4.08860669e-02, ...,\n",
|
||||
" 8.70475471e-02, 8.92383084e-02, 1.67974353e-01],\n",
|
||||
" [ 2.85779923e-01, 4.57767487e-01, 4.52292450e-02, ...,\n",
|
||||
" -1.97419018e-01, -2.04659685e-01, -2.79758364e-01]]],\n",
|
||||
" dtype=float32)"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.layers[1](vectorizer(['Hello, world!','I am glad to meet you!'])).numpy()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Hinweis**: Um die Menge an Auffüllung zu minimieren, kann es in einigen Fällen sinnvoll sein, alle Sequenzen im Datensatz in der Reihenfolge zunehmender Länge (oder genauer gesagt, Anzahl der Token) zu sortieren. Dies stellt sicher, dass jede Minibatch Sequenzen ähnlicher Länge enthält.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Semantische Einbettungen: Word2Vec\n",
|
||||
"\n",
|
||||
"In unserem vorherigen Beispiel hat die Einbettungsschicht gelernt, Wörter in Vektorrepräsentationen abzubilden, jedoch hatten diese Repräsentationen keine semantische Bedeutung. Es wäre wünschenswert, eine Vektorrepräsentation zu erlernen, bei der ähnliche Wörter oder Synonyme Vektoren entsprechen, die in Bezug auf eine Vektordistanz (zum Beispiel euklidische Distanz) nahe beieinander liegen.\n",
|
||||
"\n",
|
||||
"Um dies zu erreichen, müssen wir unser Einbettungsmodell mit einer großen Textsammlung vortrainieren, indem wir eine Technik wie [Word2Vec](https://en.wikipedia.org/wiki/Word2vec) verwenden. Diese basiert auf zwei Hauptarchitekturen, die verwendet werden, um eine verteilte Repräsentation von Wörtern zu erzeugen:\n",
|
||||
"\n",
|
||||
" - **Continuous bag-of-words** (CBoW), bei dem wir das Modell darauf trainieren, ein Wort aus dem umgebenden Kontext vorherzusagen. Gegeben ist das N-Gramm $(W_{-2},W_{-1},W_0,W_1,W_2)$, und das Ziel des Modells ist es, $W_0$ aus $(W_{-2},W_{-1},W_1,W_2)$ vorherzusagen.\n",
|
||||
" - **Continuous skip-gram** ist das Gegenteil von CBoW. Das Modell verwendet das umgebende Fenster von Kontextwörtern, um das aktuelle Wort vorherzusagen.\n",
|
||||
"\n",
|
||||
"CBoW ist schneller, während Skip-Gram zwar langsamer ist, aber eine bessere Repräsentation für seltene Wörter liefert.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Um mit der Word2Vec-Einbettung, die auf dem Google-News-Datensatz vortrainiert wurde, zu experimentieren, können wir die **gensim**-Bibliothek verwenden. Unten finden wir die Wörter, die 'neural' am ähnlichsten sind.\n",
|
||||
"\n",
|
||||
"> **Hinweis:** Wenn Sie zum ersten Mal Wortvektoren erstellen, kann das Herunterladen einige Zeit in Anspruch nehmen!\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import gensim.downloader as api\n",
|
||||
"w2v = api.load('word2vec-google-news-300')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"neuronal -> 0.7804799675941467\n",
|
||||
"neurons -> 0.7326500415802002\n",
|
||||
"neural_circuits -> 0.7252851724624634\n",
|
||||
"neuron -> 0.7174385190010071\n",
|
||||
"cortical -> 0.6941086649894714\n",
|
||||
"brain_circuitry -> 0.6923246383666992\n",
|
||||
"synaptic -> 0.6699118614196777\n",
|
||||
"neural_circuitry -> 0.6638563275337219\n",
|
||||
"neurochemical -> 0.6555314064025879\n",
|
||||
"neuronal_activity -> 0.6531826257705688\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"for w,p in w2v.most_similar('neural'):\n",
|
||||
" print(f\"{w} -> {p}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Wir können auch die Vektoreinbettung aus dem Wort extrahieren, um sie beim Training des Klassifikationsmodells zu verwenden. Die Einbettung hat 300 Komponenten, aber hier zeigen wir aus Gründen der Klarheit nur die ersten 20 Komponenten des Vektors:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([ 0.01226807, 0.06225586, 0.10693359, 0.05810547, 0.23828125,\n",
|
||||
" 0.03686523, 0.05151367, -0.20703125, 0.01989746, 0.10058594,\n",
|
||||
" -0.03759766, -0.1015625 , -0.15820312, -0.08105469, -0.0390625 ,\n",
|
||||
" -0.05053711, 0.16015625, 0.2578125 , 0.10058594, -0.25976562],\n",
|
||||
" dtype=float32)"
|
||||
]
|
||||
},
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"w2v['play'][:20]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Das Großartige an semantischen Einbettungen ist, dass man die Vektor-Codierung basierend auf Semantik manipulieren kann. Zum Beispiel können wir nach einem Wort suchen, dessen Vektorrepräsentation so nah wie möglich an den Wörtern *König* und *Frau* ist und so weit wie möglich vom Wort *Mann* entfernt ist:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"('queen', 0.7118192911148071)"
|
||||
]
|
||||
},
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"w2v.most_similar(positive=['king','woman'],negative=['man'])[0]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"Ein Beispiel oben verwendet einige interne GenSym-Magie, aber die zugrunde liegende Logik ist tatsächlich ziemlich einfach. Eine interessante Sache an Einbettungen ist, dass man normale Vektoroperationen auf Einbettungsvektoren durchführen kann, und das würde Operationen auf Wort**bedeutungen** widerspiegeln. Das obige Beispiel kann in Form von Vektoroperationen ausgedrückt werden: Wir berechnen den Vektor, der **KÖNIG-MANN+FRAU** entspricht (Operationen `+` und `-` werden auf Vektordarstellungen der entsprechenden Wörter durchgeführt), und finden dann das nächstgelegene Wort im Wörterbuch zu diesem Vektor:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"'queen'"
|
||||
]
|
||||
},
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# get the vector corresponding to kind-man+woman\n",
|
||||
"qvec = w2v['king']-1.7*w2v['man']+1.7*w2v['woman']\n",
|
||||
"# find the index of the closest embedding vector \n",
|
||||
"d = np.sum((w2v.vectors-qvec)**2,axis=1)\n",
|
||||
"min_idx = np.argmin(d)\n",
|
||||
"# find the corresponding word\n",
|
||||
"w2v.index_to_key[min_idx]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **NOTE**: Wir mussten kleine Koeffizienten zu den *man*- und *woman*-Vektoren hinzufügen – versuchen Sie, diese zu entfernen, um zu sehen, was passiert.\n",
|
||||
"\n",
|
||||
"Um den nächstgelegenen Vektor zu finden, verwenden wir TensorFlow-Mechanismen, um einen Vektor von Abständen zwischen unserem Vektor und allen Vektoren im Vokabular zu berechnen, und finden dann den Index des minimalen Wortes mit `argmin`.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Während Word2Vec eine großartige Möglichkeit zu sein scheint, die Semantik von Wörtern auszudrücken, hat es viele Nachteile, darunter die folgenden:\n",
|
||||
"\n",
|
||||
"* Sowohl CBoW- als auch Skip-Gram-Modelle sind **prädiktive Einbettungen** und berücksichtigen nur den lokalen Kontext. Word2Vec nutzt den globalen Kontext nicht.\n",
|
||||
"* Word2Vec berücksichtigt nicht die **Morphologie** von Wörtern, d. h. die Tatsache, dass die Bedeutung eines Wortes von verschiedenen Teilen des Wortes, wie z. B. dem Stamm, abhängen kann.\n",
|
||||
"\n",
|
||||
"**FastText** versucht, die zweite Einschränkung zu überwinden, und baut auf Word2Vec auf, indem es Vektordarstellungen für jedes Wort und die Zeichen-n-Gramme innerhalb jedes Wortes lernt. Die Werte der Darstellungen werden dann bei jedem Trainingsschritt zu einem Vektor gemittelt. Obwohl dies eine Menge zusätzlicher Berechnungen beim Pretraining erfordert, ermöglicht es den Wort-Einbettungen, Subwort-Informationen zu kodieren.\n",
|
||||
"\n",
|
||||
"Eine andere Methode, **GloVe**, verwendet einen anderen Ansatz für Wort-Einbettungen, der auf der Faktorisierung der Wort-Kontext-Matrix basiert. Zunächst wird eine große Matrix erstellt, die die Anzahl der Wortvorkommen in verschiedenen Kontexten zählt, und dann wird versucht, diese Matrix in niedrigeren Dimensionen so darzustellen, dass der Rekonstruktionsverlust minimiert wird.\n",
|
||||
"\n",
|
||||
"Die gensim-Bibliothek unterstützt diese Wort-Einbettungen, und Sie können mit ihnen experimentieren, indem Sie den Modell-Ladecode oben ändern.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Verwendung vortrainierter Embeddings in Keras\n",
|
||||
"\n",
|
||||
"Wir können das obige Beispiel anpassen, um die Matrix in unserer Embedding-Schicht mit semantischen Embeddings wie Word2Vec vorab zu füllen. Die Vokabulare des vortrainierten Embeddings und des Textkorpus werden wahrscheinlich nicht übereinstimmen, daher müssen wir eines auswählen. Hier untersuchen wir die beiden möglichen Optionen: die Verwendung des Tokenizer-Vokabulars und die Verwendung des Vokabulars aus den Word2Vec-Embeddings.\n",
|
||||
"\n",
|
||||
"### Verwendung des Tokenizer-Vokabulars\n",
|
||||
"\n",
|
||||
"Bei der Verwendung des Tokenizer-Vokabulars haben einige Wörter aus dem Vokabular entsprechende Word2Vec-Embeddings, während andere fehlen. Angenommen, unsere Vokabulargröße ist `vocab_size` und die Länge des Word2Vec-Embedding-Vektors ist `embed_size`, wird die Embedding-Schicht durch eine Gewichtsmatrix der Form `vocab_size`$\\times$`embed_size` dargestellt. Wir füllen diese Matrix, indem wir das Vokabular durchgehen:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Embedding size: 300\n",
|
||||
"Populating matrix, this will take some time...Done, found 4551 words, 784 words missing\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"embed_size = len(w2v.get_vector('hello'))\n",
|
||||
"print(f'Embedding size: {embed_size}')\n",
|
||||
"\n",
|
||||
"vocab = vectorizer.get_vocabulary()\n",
|
||||
"W = np.zeros((vocab_size,embed_size))\n",
|
||||
"print('Populating matrix, this will take some time...',end='')\n",
|
||||
"found, not_found = 0,0\n",
|
||||
"for i,w in enumerate(vocab):\n",
|
||||
" try:\n",
|
||||
" W[i] = w2v.get_vector(w)\n",
|
||||
" found+=1\n",
|
||||
" except:\n",
|
||||
" # W[i] = np.random.normal(0.0,0.3,size=(embed_size,))\n",
|
||||
" not_found+=1\n",
|
||||
"\n",
|
||||
"print(f\"Done, found {found} words, {not_found} words missing\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Für Wörter, die nicht im Word2Vec-Wortschatz vorhanden sind, können wir sie entweder als Nullen belassen oder einen zufälligen Vektor generieren.\n",
|
||||
"\n",
|
||||
"Nun können wir eine Einbettungsschicht mit vortrainierten Gewichten definieren:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"emb = keras.layers.Embedding(vocab_size,embed_size,weights=[W],trainable=False)\n",
|
||||
"model = keras.models.Sequential([\n",
|
||||
" vectorizer, emb,\n",
|
||||
" keras.layers.Lambda(lambda x: tf.reduce_mean(x,axis=1)),\n",
|
||||
" keras.layers.Dense(4, activation='softmax')\n",
|
||||
"])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"938/938 [==============================] - 10s 10ms/step - loss: 1.1075 - acc: 0.7822 - val_loss: 0.9134 - val_acc: 0.8175\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.History at 0x2220226ef10>"
|
||||
]
|
||||
},
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'])\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(batch_size),\n",
|
||||
" validation_data=ds_test.map(tupelize).batch(batch_size))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Hinweis**: Beachten Sie, dass wir `trainable=False` setzen, wenn wir die `Embedding` erstellen. Das bedeutet, dass wir die Embedding-Schicht nicht neu trainieren. Dies kann dazu führen, dass die Genauigkeit etwas geringer ist, aber es beschleunigt das Training.\n",
|
||||
"\n",
|
||||
"### Verwendung des Embedding-Vokabulars\n",
|
||||
"\n",
|
||||
"Ein Problem bei dem vorherigen Ansatz ist, dass die in der TextVectorization und Embedding verwendeten Vokabulare unterschiedlich sind. Um dieses Problem zu lösen, können wir eine der folgenden Lösungen verwenden:\n",
|
||||
"* Das Word2Vec-Modell mit unserem Vokabular neu trainieren.\n",
|
||||
"* Unser Dataset mit dem Vokabular aus dem vortrainierten Word2Vec-Modell laden. Die Vokabulare, die zum Laden des Datasets verwendet werden, können während des Ladens angegeben werden.\n",
|
||||
"\n",
|
||||
"Der zweite Ansatz scheint einfacher zu sein, also setzen wir ihn um. Zunächst erstellen wir eine `TextVectorization`-Schicht mit dem angegebenen Vokabular, das aus den Word2Vec-Embeddings stammt:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"vocab = list(w2v.vocab.keys())\n",
|
||||
"vectorizer = keras.layers.experimental.preprocessing.TextVectorization(input_shape=(1,))\n",
|
||||
"vectorizer.set_vocabulary(vocab)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Die Gensim-Wort-Einbettungsbibliothek enthält eine praktische Funktion, `get_keras_embeddings`, die automatisch die entsprechende Keras-Einbettungsschicht für Sie erstellt.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch 1/5\n",
|
||||
"938/938 [==============================] - 20s 14ms/step - loss: 1.3377 - acc: 0.4978 - val_loss: 1.2995 - val_acc: 0.5647\n",
|
||||
"Epoch 2/5\n",
|
||||
"938/938 [==============================] - 10s 10ms/step - loss: 1.2587 - acc: 0.5722 - val_loss: 1.2339 - val_acc: 0.5842\n",
|
||||
"Epoch 3/5\n",
|
||||
"938/938 [==============================] - 10s 10ms/step - loss: 1.1980 - acc: 0.5884 - val_loss: 1.1826 - val_acc: 0.5954\n",
|
||||
"Epoch 4/5\n",
|
||||
"938/938 [==============================] - 12s 13ms/step - loss: 1.1503 - acc: 0.6002 - val_loss: 1.1417 - val_acc: 0.6018\n",
|
||||
"Epoch 5/5\n",
|
||||
"938/938 [==============================] - 11s 12ms/step - loss: 1.1120 - acc: 0.6097 - val_loss: 1.1083 - val_acc: 0.6104\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.History at 0x2220ccb81c0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model = keras.models.Sequential([\n",
|
||||
" vectorizer, \n",
|
||||
" w2v.get_keras_embedding(train_embeddings=False),\n",
|
||||
" keras.layers.Lambda(lambda x: tf.reduce_mean(x,axis=1)),\n",
|
||||
" keras.layers.Dense(4, activation='softmax')\n",
|
||||
"])\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'])\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(128),validation_data=ds_test.map(tupelize).batch(128),epochs=5)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Einer der Gründe, warum wir keine höhere Genauigkeit sehen, ist, dass einige Wörter aus unserem Datensatz im vortrainierten GloVe-Vokabular fehlen und daher im Wesentlichen ignoriert werden. Um dies zu überwinden, können wir unsere eigenen Einbettungen basierend auf unserem Datensatz trainieren.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Kontextuelle Einbettungen\n",
|
||||
"\n",
|
||||
"Eine zentrale Einschränkung traditioneller vortrainierter Einbettungsrepräsentationen wie Word2Vec ist die Tatsache, dass sie zwar eine gewisse Bedeutung eines Wortes erfassen können, aber nicht zwischen verschiedenen Bedeutungen unterscheiden können. Dies kann in nachgelagerten Modellen zu Problemen führen.\n",
|
||||
"\n",
|
||||
"Zum Beispiel hat das Wort „play“ in diesen beiden Sätzen unterschiedliche Bedeutungen:\n",
|
||||
"- Ich war in einem **Theaterstück** im Theater.\n",
|
||||
"- John möchte mit seinen Freunden **spielen**.\n",
|
||||
"\n",
|
||||
"Die vortrainierten Einbettungen, über die wir gesprochen haben, repräsentieren beide Bedeutungen des Wortes „play“ in derselben Einbettung. Um diese Einschränkung zu überwinden, müssen wir Einbettungen basierend auf dem **Sprachmodell** erstellen, das auf einem großen Textkorpus trainiert wurde und *versteht*, wie Wörter in unterschiedlichen Kontexten zusammengefügt werden können. Die Diskussion über kontextuelle Einbettungen liegt außerhalb des Umfangs dieses Tutorials, aber wir werden darauf zurückkommen, wenn wir im nächsten Abschnitt über Sprachmodelle sprechen.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Haftungsausschluss**: \nDieses Dokument wurde mit dem KI-Übersetzungsdienst [Co-op Translator](https://github.com/Azure/co-op-translator) übersetzt. Obwohl wir uns um Genauigkeit bemühen, beachten Sie bitte, dass automatisierte Übersetzungen Fehler oder Ungenauigkeiten enthalten können. Das Originaldokument in seiner ursprünglichen Sprache sollte als maßgebliche Quelle betrachtet werden. Für kritische Informationen wird eine professionelle menschliche Übersetzung empfohlen. Wir übernehmen keine Haftung für Missverständnisse oder Fehlinterpretationen, die sich aus der Nutzung dieser Übersetzung ergeben.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5"
|
||||
},
|
||||
"kernel_info": {
|
||||
"name": "conda-env-py37_tensorflow-py"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "py37_tensorflow",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"nteract": {
|
||||
"version": "nteract-front-end@1.0.0"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "b859482be7f61d1eadc2c6a2720a37e4",
|
||||
"translation_date": "2025-08-31T17:10:29+00:00",
|
||||
"source_file": "lessons/5-NLP/14-Embeddings/EmbeddingsTF.ipynb",
|
||||
"language_code": "de"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
|
|
@ -0,0 +1,576 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "NXTSugt6ieXh"
|
||||
},
|
||||
"source": [
|
||||
"## Training CBoW-Modell\n",
|
||||
"\n",
|
||||
"Dieses Notebook ist Teil des [AI for Beginners Curriculum](http://aka.ms/ai-beginners)\n",
|
||||
"\n",
|
||||
"In diesem Beispiel werden wir ein CBoW-Sprachmodell trainieren, um unseren eigenen Word2Vec-Einbettungsraum zu erstellen. Als Textquelle verwenden wir den AG News-Datensatz.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"import torchtext\n",
|
||||
"import os\n",
|
||||
"import collections\n",
|
||||
"import builtins\n",
|
||||
"import random\n",
|
||||
"import numpy as np"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "q-UiiJUKaxHj"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "TFbR8CZaTZ1q"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Zuerst laden wir unser Dataset und definieren Tokenizer und Vokabular. Wir setzen `vocab_size` auf 5000, um die Berechnungen etwas zu begrenzen.\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "HIwC7lI5T-ov"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def load_dataset(ngrams = 1, min_freq = 1, vocab_size = 5000 , lines_cnt = 500):\n",
|
||||
" tokenizer = torchtext.data.utils.get_tokenizer('basic_english')\n",
|
||||
" print(\"Loading dataset...\")\n",
|
||||
" test_dataset, train_dataset = torchtext.datasets.AG_NEWS(root='./data')\n",
|
||||
" train_dataset = list(train_dataset)\n",
|
||||
" test_dataset = list(test_dataset)\n",
|
||||
" classes = ['World', 'Sports', 'Business', 'Sci/Tech']\n",
|
||||
" print('Building vocab...')\n",
|
||||
" counter = collections.Counter()\n",
|
||||
" for i, (_, line) in enumerate(train_dataset):\n",
|
||||
" counter.update(torchtext.data.utils.ngrams_iterator(tokenizer(line),ngrams=ngrams))\n",
|
||||
" if i == lines_cnt:\n",
|
||||
" break\n",
|
||||
" vocab = torchtext.vocab.Vocab(collections.Counter(dict(counter.most_common(vocab_size))), min_freq=min_freq)\n",
|
||||
" return train_dataset, test_dataset, classes, vocab, tokenizer"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "wdZuygtgiuLG"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"train_dataset, test_dataset, _, vocab, tokenizer = load_dataset()"
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "4d1nU1gsivGu",
|
||||
"outputId": "949fe272-ae0e-49f5-c373-6703458b3a74"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
"Loading dataset...\n",
|
||||
"Building vocab...\n"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def encode(x, vocabulary, tokenizer = tokenizer):\n",
|
||||
" return [vocabulary[s] for s in tokenizer(x)]"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "1XDYNhG8ToFV"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "LIlQk6_PaHVY"
|
||||
},
|
||||
"source": [
|
||||
"## CBoW-Modell\n",
|
||||
"\n",
|
||||
"CBoW lernt, ein Wort basierend auf den $2N$ benachbarten Wörtern vorherzusagen. Zum Beispiel, wenn $N=1$, erhalten wir die folgenden Paare aus dem Satz *I like to train networks*: (like,I), (I, like), (to, like), (like,to), (train,to), (to, train), (networks, train), (train,networks). Hier ist das erste Wort das benachbarte Wort, das als Eingabe verwendet wird, und das zweite Wort ist das, das wir vorhersagen.\n",
|
||||
"\n",
|
||||
"Um ein Netzwerk zu erstellen, das das nächste Wort vorhersagt, müssen wir das benachbarte Wort als Eingabe bereitstellen und die Wortnummer als Ausgabe erhalten. Die Architektur des CBoW-Netzwerks sieht wie folgt aus:\n",
|
||||
"\n",
|
||||
"* Das Eingabewort wird durch die Embedding-Schicht geleitet. Diese Embedding-Schicht wird unser Word2Vec-Embedding sein, daher definieren wir sie separat als die Variable `embedder`. In diesem Beispiel verwenden wir eine Embedding-Größe von 30, obwohl Sie möglicherweise mit höheren Dimensionen experimentieren möchten (echtes Word2Vec hat 300).\n",
|
||||
"* Der Embedding-Vektor wird dann an eine lineare Schicht weitergegeben, die das Ausgabewort vorhersagt. Daher hat sie `vocab_size` Neuronen.\n",
|
||||
"\n",
|
||||
"Für die Ausgabe: Wenn wir `CrossEntropyLoss` als Verlustfunktion verwenden, müssen wir auch nur die Wortnummern als erwartete Ergebnisse bereitstellen, ohne One-Hot-Encoding.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"vocab_size = len(vocab)\n",
|
||||
"\n",
|
||||
"embedder = torch.nn.Embedding(num_embeddings = vocab_size, embedding_dim = 30)\n",
|
||||
"model = torch.nn.Sequential(\n",
|
||||
" embedder,\n",
|
||||
" torch.nn.Linear(in_features = 30, out_features = vocab_size),\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print(model)"
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "akKTcKQKkfl2",
|
||||
"outputId": "da687e3e-a8ec-4c1a-e456-ab8cd6ac7dad"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
"Sequential(\n",
|
||||
" (0): Embedding(5002, 30)\n",
|
||||
" (1): Linear(in_features=30, out_features=5002, bias=True)\n",
|
||||
")\n"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "Nud6jgGPaHVa"
|
||||
},
|
||||
"source": [
|
||||
"## Vorbereitung der Trainingsdaten\n",
|
||||
"\n",
|
||||
"Nun programmieren wir die Hauptfunktion, die CBoW-Wortpaare aus Text berechnet. Diese Funktion ermöglicht es uns, die Fenstergröße festzulegen und gibt ein Set von Paaren zurück – Eingabe- und Ausgabewort. Beachten Sie, dass diese Funktion sowohl auf Wörter als auch auf Vektoren/Tensoren angewendet werden kann – was es uns ermöglicht, den Text zu kodieren, bevor er an die Funktion `to_cbow` übergeben wird.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "x-dsXygOieXn",
|
||||
"outputId": "c2218280-e540-40ba-9546-efe48d0d714f"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
"[['like', 'I'], ['to', 'I'], ['I', 'like'], ['to', 'like'], ['train', 'like'], ['I', 'to'], ['like', 'to'], ['train', 'to'], ['networks', 'to'], ['like', 'train'], ['to', 'train'], ['networks', 'train'], ['to', 'networks'], ['train', 'networks']]\n",
|
||||
"[[232, 172], [5, 172], [172, 232], [5, 232], [0, 232], [172, 5], [232, 5], [0, 5], [1202, 5], [232, 0], [5, 0], [1202, 0], [5, 1202], [0, 1202]]\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def to_cbow(sent,window_size=2):\n",
|
||||
" res = []\n",
|
||||
" for i,x in enumerate(sent):\n",
|
||||
" for j in range(max(0,i-window_size),min(i+window_size+1,len(sent))):\n",
|
||||
" if i!=j:\n",
|
||||
" res.append([sent[j],x])\n",
|
||||
" return res\n",
|
||||
"\n",
|
||||
"print(to_cbow(['I','like','to','train','networks']))\n",
|
||||
"print(to_cbow(encode('I like to train networks', vocab)))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "XVaaDLjaaHVb"
|
||||
},
|
||||
"source": [
|
||||
"Lass uns den Trainingsdatensatz vorbereiten. Wir werden alle Nachrichten durchgehen, `to_cbow` aufrufen, um die Liste der Wortpaare zu erhalten, und diese Paare zu `X` und `Y` hinzufügen. Aus Zeitgründen werden wir nur die ersten 10.000 Nachrichten berücksichtigen - du kannst die Einschränkung leicht entfernen, falls du mehr Zeit hast und bessere Einbettungen erhalten möchtest :)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "54b-Gd9TieXo"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"X = []\n",
|
||||
"Y = []\n",
|
||||
"for i, x in zip(range(10000), train_dataset):\n",
|
||||
" for w1, w2 in to_cbow(encode(x[1], vocab), window_size = 5):\n",
|
||||
" X.append(w1)\n",
|
||||
" Y.append(w2)\n",
|
||||
"\n",
|
||||
"X = torch.tensor(X)\n",
|
||||
"Y = torch.tensor(Y)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Wir werden diese Daten auch in einen Datensatz umwandeln und einen Datenlader erstellen:\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "cwWy0PzXWhN5"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"class SimpleIterableDataset(torch.utils.data.IterableDataset):\n",
|
||||
" def __init__(self, X, Y):\n",
|
||||
" super(SimpleIterableDataset).__init__()\n",
|
||||
" self.data = []\n",
|
||||
" for i in range(len(X)):\n",
|
||||
" self.data.append( (Y[i], X[i]) )\n",
|
||||
" random.shuffle(self.data)\n",
|
||||
"\n",
|
||||
" def __iter__(self):\n",
|
||||
" return iter(self.data)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "mfoAcGPFZU8p"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "e4NQ_-5waHVc"
|
||||
},
|
||||
"source": [
|
||||
"Wir werden diese Daten auch in einen Datensatz umwandeln und einen Datenlader erstellen:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "AbLUcojlieXo"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"ds = SimpleIterableDataset(X, Y)\n",
|
||||
"dl = torch.utils.data.DataLoader(ds, batch_size = 256)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "pKQr7sXeaHVc"
|
||||
},
|
||||
"source": [
|
||||
"Jetzt beginnen wir mit dem eigentlichen Training. Wir verwenden den `SGD`-Optimierer mit einer ziemlich hohen Lernrate. Du kannst auch andere Optimierer ausprobieren, wie zum Beispiel `Adam`. Wir werden zunächst für 10 Epochen trainieren – und du kannst diese Zelle erneut ausführen, wenn du einen noch geringeren Verlust erzielen möchtest.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def train_epoch(net, dataloader, lr = 0.01, optimizer = None, loss_fn = torch.nn.CrossEntropyLoss(), epochs = None, report_freq = 1):\n",
|
||||
" optimizer = optimizer or torch.optim.Adam(net.parameters(), lr = lr)\n",
|
||||
" loss_fn = loss_fn.to(device)\n",
|
||||
" net.train()\n",
|
||||
"\n",
|
||||
" for i in range(epochs):\n",
|
||||
" total_loss, j = 0, 0, \n",
|
||||
" for labels, features in dataloader:\n",
|
||||
" optimizer.zero_grad()\n",
|
||||
" features, labels = features.to(device), labels.to(device)\n",
|
||||
" out = net(features)\n",
|
||||
" loss = loss_fn(out, labels)\n",
|
||||
" loss.backward()\n",
|
||||
" optimizer.step()\n",
|
||||
" total_loss += loss\n",
|
||||
" j += 1\n",
|
||||
" if i % report_freq == 0:\n",
|
||||
" print(f\"Epoch: {i+1}: loss={total_loss.item()/j}\")\n",
|
||||
"\n",
|
||||
" return total_loss.item()/j"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "HeeCYKr_KF1w"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"train_epoch(net = model, dataloader = dl, optimizer = torch.optim.SGD(model.parameters(), lr = 0.1), loss_fn = torch.nn.CrossEntropyLoss(), epochs = 10)"
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "KVgwGtDHgDlT",
|
||||
"outputId": "2447833f-f0e3-4566-c33d-addbfe2f451d"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
"Epoch: 1: loss=5.664632366860172\n",
|
||||
"Epoch: 2: loss=5.632101973960962\n",
|
||||
"Epoch: 3: loss=5.610399051405015\n",
|
||||
"Epoch: 4: loss=5.594621561080262\n",
|
||||
"Epoch: 5: loss=5.582538017415446\n",
|
||||
"Epoch: 6: loss=5.572900234519603\n",
|
||||
"Epoch: 7: loss=5.564951676341915\n",
|
||||
"Epoch: 8: loss=5.558288112064614\n",
|
||||
"Epoch: 9: loss=5.552576955031129\n",
|
||||
"Epoch: 10: loss=5.547634165194347\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"output_type": "execute_result",
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"5.547634165194347"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"execution_count": 16
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "W8u2qXZmaHVd"
|
||||
},
|
||||
"source": [
|
||||
"## Ausprobieren von Word2Vec\n",
|
||||
"\n",
|
||||
"Um Word2Vec zu verwenden, extrahieren wir die Vektoren, die den Wörtern in unserem Vokabular entsprechen:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "r8TatcXjkU_t"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"vectors = torch.stack([embedder(torch.tensor(vocab[s])) for s in vocab.itos], 0)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "3OcX21UOaHVd"
|
||||
},
|
||||
"source": [
|
||||
"Lass uns sehen, wie das Wort **Paris** beispielsweise in einen Vektor codiert wird:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "bz6tAeLzieXp",
|
||||
"outputId": "5b20850e-4342-45e9-f840-cfac2b4d61d8"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
"tensor([-0.0915, 2.1224, -0.0281, -0.6819, 1.1219, 0.6458, -1.3704, -1.3314,\n",
|
||||
" -1.1437, 0.4496, 0.2301, -0.3515, -0.8485, 1.0481, 0.4386, -0.8949,\n",
|
||||
" 0.5644, 1.0939, -2.5096, 3.2949, -0.2601, -0.8640, 0.1421, -0.0804,\n",
|
||||
" -0.5083, -1.0560, 0.9753, -0.5949, -1.6046, 0.5774],\n",
|
||||
" grad_fn=<EmbeddingBackward>)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"paris_vec = embedder(torch.tensor(vocab['paris']))\n",
|
||||
"print(paris_vec)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "pHTJlaeYaHVd"
|
||||
},
|
||||
"source": [
|
||||
"Es ist interessant, Word2Vec zu verwenden, um nach Synonymen zu suchen. Die folgende Funktion gibt die `n` nächsten Wörter zu einer gegebenen Eingabe zurück. Um sie zu finden, berechnen wir die Norm von $|w_i - v|$, wobei $v$ der Vektor ist, der unserem Eingabewort entspricht, und $w_i$ die Kodierung des $i$-ten Wortes im Vokabular ist. Anschließend sortieren wir das Array und geben die entsprechenden Indizes mit `argsort` zurück, wobei wir die ersten `n` Elemente der Liste nehmen, die die Positionen der nächsten Wörter im Vokabular kodieren.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "NlZyi-_olFar",
|
||||
"outputId": "b5dbb163-88c4-4d5a-eaf2-6751f700e98c"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "execute_result",
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"['microsoft', 'quoted', 'lp', 'rate', 'top']"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"execution_count": 56
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def close_words(x, n = 5):\n",
|
||||
" vec = embedder(torch.tensor(vocab[x]))\n",
|
||||
" top5 = np.linalg.norm(vectors.detach().numpy() - vec.detach().numpy(), axis = 1).argsort()[:n]\n",
|
||||
" return [ vocab.itos[x] for x in top5 ]\n",
|
||||
"\n",
|
||||
"close_words('microsoft')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "-dQq7xeAln0U",
|
||||
"outputId": "66f768c3-c248-4bfd-ce4f-c8ffc6d0dd0d"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "execute_result",
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"['basketball', 'lot', 'sinai', 'states', 'healthdaynews']"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"execution_count": 51
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"close_words('basketball')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "fJXqK26b29sa",
|
||||
"outputId": "78f0baba-ffd0-485a-dd87-0a12bedfd7fa"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "execute_result",
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"['funds', 'travel', 'sydney', 'japan', 'business']"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"execution_count": 77
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"close_words('funds')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "My0VeTDd3Ji8"
|
||||
},
|
||||
"source": [
|
||||
"## Erkenntnis\n",
|
||||
"\n",
|
||||
"Mit cleveren Techniken wie CBoW können wir ein Word2Vec-Modell trainieren. Du kannst auch versuchen, ein Skip-Gram-Modell zu trainieren, das darauf ausgelegt ist, das benachbarte Wort basierend auf dem zentralen Wort vorherzusagen, und sehen, wie gut es funktioniert.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Haftungsausschluss**: \nDieses Dokument wurde mit dem KI-Übersetzungsdienst [Co-op Translator](https://github.com/Azure/co-op-translator) übersetzt. Obwohl wir uns um Genauigkeit bemühen, beachten Sie bitte, dass automatisierte Übersetzungen Fehler oder Ungenauigkeiten enthalten können. Das Originaldokument in seiner ursprünglichen Sprache sollte als maßgebliche Quelle betrachtet werden. Für kritische Informationen wird eine professionelle menschliche Übersetzung empfohlen. Wir übernehmen keine Haftung für Missverständnisse oder Fehlinterpretationen, die sich aus der Nutzung dieser Übersetzung ergeben.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"collapsed_sections": [],
|
||||
"name": "CBoW-PyTorch.ipynb",
|
||||
"provenance": []
|
||||
},
|
||||
"interpreter": {
|
||||
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.8.12 ('py38')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"orig_nbformat": 4,
|
||||
"gpuClass": "standard",
|
||||
"coopTranslator": {
|
||||
"original_hash": "36df28efe3fe40b6fb0a7fa48fe3ea82",
|
||||
"translation_date": "2025-08-31T16:54:53+00:00",
|
||||
"source_file": "lessons/5-NLP/15-LanguageModeling/CBoW-PyTorch.ipynb",
|
||||
"language_code": "de"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,479 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Rekurrente neuronale Netze\n",
|
||||
"\n",
|
||||
"Im vorherigen Modul haben wir reichhaltige semantische Repräsentationen von Text verwendet und einen einfachen linearen Klassifikator auf den Einbettungen aufgebaut. Diese Architektur erfasst die aggregierte Bedeutung der Wörter in einem Satz, berücksichtigt jedoch nicht die **Reihenfolge** der Wörter, da die Aggregationsoperation auf den Einbettungen diese Information aus dem ursprünglichen Text entfernt hat. Da diese Modelle die Wortreihenfolge nicht modellieren können, sind sie nicht in der Lage, komplexere oder mehrdeutige Aufgaben wie Textgenerierung oder Fragebeantwortung zu lösen.\n",
|
||||
"\n",
|
||||
"Um die Bedeutung einer Textsequenz zu erfassen, müssen wir eine andere Architektur neuronaler Netze verwenden, die als **rekurrentes neuronales Netz** oder RNN bezeichnet wird. In einem RNN führen wir unseren Satz nacheinander Symbol für Symbol durch das Netzwerk, und das Netzwerk erzeugt einen **Zustand**, den wir dann zusammen mit dem nächsten Symbol erneut in das Netzwerk einspeisen.\n",
|
||||
"\n",
|
||||
"Gegeben die Eingabesequenz von Tokens $X_0,\\dots,X_n$, erstellt das RNN eine Sequenz von neuronalen Netzwerkblöcken und trainiert diese Sequenz end-to-end mittels Backpropagation. Jeder Netzwerkblock nimmt ein Paar $(X_i,S_i)$ als Eingabe und erzeugt $S_{i+1}$ als Ergebnis. Der finale Zustand $S_n$ oder die Ausgabe $X_n$ wird in einen linearen Klassifikator eingespeist, um das Ergebnis zu erzeugen. Alle Netzwerkblöcke teilen sich die gleichen Gewichte und werden in einem einzigen Backpropagation-Durchlauf end-to-end trainiert.\n",
|
||||
"\n",
|
||||
"Da die Zustandsvektoren $S_0,\\dots,S_n$ durch das Netzwerk weitergegeben werden, kann es die sequentiellen Abhängigkeiten zwischen Wörtern lernen. Zum Beispiel, wenn das Wort *nicht* irgendwo in der Sequenz erscheint, kann es lernen, bestimmte Elemente innerhalb des Zustandsvektors zu negieren, was zu einer Verneinung führt.\n",
|
||||
"\n",
|
||||
"> Da die Gewichte aller RNN-Blöcke im Bild geteilt werden, kann dasselbe Bild als ein Block (rechts) mit einer rekurrenten Rückkopplungsschleife dargestellt werden, die den Ausgabestatus des Netzwerks zurück an den Eingang weitergibt.\n",
|
||||
"\n",
|
||||
"Schauen wir uns an, wie rekurrente neuronale Netze uns dabei helfen können, unser Nachrichten-Dataset zu klassifizieren.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Loading dataset...\n",
|
||||
"Building vocab...\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"import torchtext\n",
|
||||
"from torchnlp import *\n",
|
||||
"train_dataset, test_dataset, classes, vocab = load_dataset()\n",
|
||||
"vocab_size = len(vocab)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Einfacher RNN-Klassifikator\n",
|
||||
"\n",
|
||||
"Im Fall eines einfachen RNN ist jede rekurrente Einheit ein einfaches lineares Netzwerk, das einen zusammengefügten Eingabevektor und Zustandsvektor aufnimmt und einen neuen Zustandsvektor erzeugt. PyTorch repräsentiert diese Einheit mit der Klasse `RNNCell`, und ein Netzwerk solcher Zellen als `RNN`-Schicht.\n",
|
||||
"\n",
|
||||
"Um einen RNN-Klassifikator zu definieren, wenden wir zunächst eine Embedding-Schicht an, um die Dimensionalität des Eingabevokabulars zu reduzieren, und fügen dann eine RNN-Schicht darüber hinzu:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class RNNClassifier(torch.nn.Module):\n",
|
||||
" def __init__(self, vocab_size, embed_dim, hidden_dim, num_class):\n",
|
||||
" super().__init__()\n",
|
||||
" self.hidden_dim = hidden_dim\n",
|
||||
" self.embedding = torch.nn.Embedding(vocab_size, embed_dim)\n",
|
||||
" self.rnn = torch.nn.RNN(embed_dim,hidden_dim,batch_first=True)\n",
|
||||
" self.fc = torch.nn.Linear(hidden_dim, num_class)\n",
|
||||
"\n",
|
||||
" def forward(self, x):\n",
|
||||
" batch_size = x.size(0)\n",
|
||||
" x = self.embedding(x)\n",
|
||||
" x,h = self.rnn(x)\n",
|
||||
" return self.fc(x.mean(dim=1))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Hinweis:** Wir verwenden hier eine untrainierte Einbettungsschicht zur Vereinfachung, aber für noch bessere Ergebnisse können wir eine vortrainierte Einbettungsschicht mit Word2Vec- oder GloVe-Einbettungen verwenden, wie in der vorherigen Einheit beschrieben. Für ein besseres Verständnis könnten Sie den Code anpassen, um mit vortrainierten Einbettungen zu arbeiten.\n",
|
||||
"\n",
|
||||
"In unserem Fall verwenden wir einen gepolsterten Datenlader, sodass jede Charge eine Anzahl gepolsterter Sequenzen gleicher Länge enthält. Die RNN-Schicht nimmt die Sequenz von Einbettungstensoren und erzeugt zwei Ausgaben:\n",
|
||||
"* $x$ ist eine Sequenz von RNN-Zellenausgaben bei jedem Schritt\n",
|
||||
"* $h$ ist der finale versteckte Zustand für das letzte Element der Sequenz\n",
|
||||
"\n",
|
||||
"Anschließend wenden wir einen vollständig verbundenen linearen Klassifikator an, um die Anzahl der Klassen zu bestimmen.\n",
|
||||
"\n",
|
||||
"> **Hinweis:** RNNs sind recht schwierig zu trainieren, da die Anzahl der Schichten, die bei der Rückwärtsausbreitung beteiligt sind, ziemlich groß wird, sobald die RNN-Zellen entlang der Sequenzlänge entrollt werden. Daher müssen wir eine kleine Lernrate wählen und das Netzwerk auf einem größeren Datensatz trainieren, um gute Ergebnisse zu erzielen. Dies kann ziemlich lange dauern, daher wird die Verwendung einer GPU bevorzugt.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.3090625\n",
|
||||
"6400: acc=0.38921875\n",
|
||||
"9600: acc=0.4590625\n",
|
||||
"12800: acc=0.511953125\n",
|
||||
"16000: acc=0.5506875\n",
|
||||
"19200: acc=0.57921875\n",
|
||||
"22400: acc=0.6070089285714285\n",
|
||||
"25600: acc=0.6304296875\n",
|
||||
"28800: acc=0.6484027777777778\n",
|
||||
"32000: acc=0.66509375\n",
|
||||
"35200: acc=0.6790056818181818\n",
|
||||
"38400: acc=0.6929166666666666\n",
|
||||
"41600: acc=0.7035817307692308\n",
|
||||
"44800: acc=0.7137276785714286\n",
|
||||
"48000: acc=0.72225\n",
|
||||
"51200: acc=0.73001953125\n",
|
||||
"54400: acc=0.7372794117647059\n",
|
||||
"57600: acc=0.7436631944444444\n",
|
||||
"60800: acc=0.7503947368421052\n",
|
||||
"64000: acc=0.75634375\n",
|
||||
"67200: acc=0.7615773809523809\n",
|
||||
"70400: acc=0.7662642045454545\n",
|
||||
"73600: acc=0.7708423913043478\n",
|
||||
"76800: acc=0.7751822916666666\n",
|
||||
"80000: acc=0.7790625\n",
|
||||
"83200: acc=0.7825\n",
|
||||
"86400: acc=0.7858564814814815\n",
|
||||
"89600: acc=0.7890513392857142\n",
|
||||
"92800: acc=0.7920474137931034\n",
|
||||
"96000: acc=0.7952708333333334\n",
|
||||
"99200: acc=0.7982258064516129\n",
|
||||
"102400: acc=0.80099609375\n",
|
||||
"105600: acc=0.8037594696969697\n",
|
||||
"108800: acc=0.8060569852941176\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=padify, shuffle=True)\n",
|
||||
"net = RNNClassifier(vocab_size,64,32,len(classes)).to(device)\n",
|
||||
"train_epoch(net,train_loader, lr=0.001)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Langzeit-Kurzzeitgedächtnis (LSTM)\n",
|
||||
"\n",
|
||||
"Eines der Hauptprobleme klassischer RNNs ist das sogenannte **Problem der verschwindenden Gradienten**. Da RNNs in einem einzigen Backpropagation-Durchlauf Ende-zu-Ende trainiert werden, fällt es ihnen schwer, den Fehler bis zu den ersten Schichten des Netzwerks weiterzuleiten. Dadurch kann das Netzwerk keine Beziehungen zwischen weit entfernten Tokens lernen. Eine Möglichkeit, dieses Problem zu umgehen, besteht darin, eine **explizite Zustandsverwaltung** durch den Einsatz sogenannter **Gates** einzuführen. Zwei der bekanntesten Architekturen dieser Art sind: **Langzeit-Kurzzeitgedächtnis** (LSTM) und **Gated Relay Unit** (GRU).\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Ein LSTM-Netzwerk ist ähnlich wie ein RNN organisiert, aber es gibt zwei Zustände, die von Schicht zu Schicht weitergegeben werden: der aktuelle Zustand $c$ und der versteckte Vektor $h$. In jeder Einheit wird der versteckte Vektor $h_i$ mit der Eingabe $x_i$ verknüpft, und sie steuern über **Gates**, was mit dem Zustand $c$ geschieht. Jedes Gate ist ein neuronales Netzwerk mit einer Sigmoid-Aktivierung (Ausgabe im Bereich $[0,1]$), das als bitweises Maskieren betrachtet werden kann, wenn es mit dem Zustandsvektor multipliziert wird. Es gibt folgende Gates (von links nach rechts im obigen Bild):\n",
|
||||
"* **Vergessens-Gate**: Es nimmt den versteckten Vektor und bestimmt, welche Komponenten des Vektors $c$ wir vergessen und welche wir durchlassen müssen.\n",
|
||||
"* **Eingabe-Gate**: Es nimmt Informationen aus der Eingabe und dem versteckten Vektor und fügt sie dem Zustand hinzu.\n",
|
||||
"* **Ausgabe-Gate**: Es transformiert den Zustand über eine lineare Schicht mit $\\tanh$-Aktivierung und wählt dann einige seiner Komponenten mithilfe des versteckten Vektors $h_i$ aus, um den neuen Zustand $c_{i+1}$ zu erzeugen.\n",
|
||||
"\n",
|
||||
"Die Komponenten des Zustands $c$ können als Flags betrachtet werden, die ein- und ausgeschaltet werden können. Zum Beispiel, wenn wir im Sequenzkontext den Namen *Alice* begegnen, könnten wir annehmen, dass es sich um eine weibliche Figur handelt, und das Flag im Zustand setzen, dass wir ein weibliches Substantiv im Satz haben. Wenn wir später auf die Phrase *und Tom* stoßen, setzen wir das Flag, dass wir ein Plural-Substantiv haben. Durch die Manipulation des Zustands können wir also theoretisch grammatikalische Eigenschaften von Satzteilen verfolgen.\n",
|
||||
"\n",
|
||||
"> **Hinweis**: Eine großartige Ressource, um die Interna von LSTMs zu verstehen, ist der Artikel [Understanding LSTM Networks](https://colah.github.io/posts/2015-08-Understanding-LSTMs/) von Christopher Olah.\n",
|
||||
"\n",
|
||||
"Obwohl die interne Struktur einer LSTM-Zelle komplex erscheinen mag, verbirgt PyTorch diese Implementierung in der `LSTMCell`-Klasse und stellt das `LSTM`-Objekt bereit, um die gesamte LSTM-Schicht darzustellen. Daher wird die Implementierung eines LSTM-Klassifikators der eines einfachen RNNs, das wir oben gesehen haben, ziemlich ähnlich sein:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class LSTMClassifier(torch.nn.Module):\n",
|
||||
" def __init__(self, vocab_size, embed_dim, hidden_dim, num_class):\n",
|
||||
" super().__init__()\n",
|
||||
" self.hidden_dim = hidden_dim\n",
|
||||
" self.embedding = torch.nn.Embedding(vocab_size, embed_dim)\n",
|
||||
" self.embedding.weight.data = torch.randn_like(self.embedding.weight.data)-0.5\n",
|
||||
" self.rnn = torch.nn.LSTM(embed_dim,hidden_dim,batch_first=True)\n",
|
||||
" self.fc = torch.nn.Linear(hidden_dim, num_class)\n",
|
||||
"\n",
|
||||
" def forward(self, x):\n",
|
||||
" batch_size = x.size(0)\n",
|
||||
" x = self.embedding(x)\n",
|
||||
" x,(h,c) = self.rnn(x)\n",
|
||||
" return self.fc(h[-1])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.259375\n",
|
||||
"6400: acc=0.25859375\n",
|
||||
"9600: acc=0.26177083333333334\n",
|
||||
"12800: acc=0.2784375\n",
|
||||
"16000: acc=0.313\n",
|
||||
"19200: acc=0.3528645833333333\n",
|
||||
"22400: acc=0.3965625\n",
|
||||
"25600: acc=0.4385546875\n",
|
||||
"28800: acc=0.4752777777777778\n",
|
||||
"32000: acc=0.505375\n",
|
||||
"35200: acc=0.5326704545454546\n",
|
||||
"38400: acc=0.5557552083333334\n",
|
||||
"41600: acc=0.5760817307692307\n",
|
||||
"44800: acc=0.5954910714285714\n",
|
||||
"48000: acc=0.6118333333333333\n",
|
||||
"51200: acc=0.62681640625\n",
|
||||
"54400: acc=0.6404779411764706\n",
|
||||
"57600: acc=0.6520138888888889\n",
|
||||
"60800: acc=0.662828947368421\n",
|
||||
"64000: acc=0.673546875\n",
|
||||
"67200: acc=0.6831547619047619\n",
|
||||
"70400: acc=0.6917897727272727\n",
|
||||
"73600: acc=0.6997146739130434\n",
|
||||
"76800: acc=0.707109375\n",
|
||||
"80000: acc=0.714075\n",
|
||||
"83200: acc=0.7209134615384616\n",
|
||||
"86400: acc=0.727037037037037\n",
|
||||
"89600: acc=0.7326674107142858\n",
|
||||
"92800: acc=0.7379633620689655\n",
|
||||
"96000: acc=0.7433645833333333\n",
|
||||
"99200: acc=0.7479032258064516\n",
|
||||
"102400: acc=0.752119140625\n",
|
||||
"105600: acc=0.7562405303030303\n",
|
||||
"108800: acc=0.76015625\n",
|
||||
"112000: acc=0.7641339285714286\n",
|
||||
"115200: acc=0.7677777777777778\n",
|
||||
"118400: acc=0.7711233108108108\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(0.03487814127604167, 0.7728)"
|
||||
]
|
||||
},
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"net = LSTMClassifier(vocab_size,64,32,len(classes)).to(device)\n",
|
||||
"train_epoch(net,train_loader, lr=0.001)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Gepackte Sequenzen\n",
|
||||
"\n",
|
||||
"In unserem Beispiel mussten wir alle Sequenzen im Minibatch mit Nullvektoren auffüllen. Dies führt zwar zu einem gewissen Speicherverbrauch, aber bei RNNs ist es noch kritischer, dass zusätzliche RNN-Zellen für die aufgefüllten Eingabeelemente erstellt werden. Diese nehmen am Training teil, tragen jedoch keine wichtigen Eingabeinformationen. Es wäre viel besser, das RNN nur bis zur tatsächlichen Sequenzlänge zu trainieren.\n",
|
||||
"\n",
|
||||
"Um dies zu erreichen, wurde in PyTorch ein spezielles Format zur Speicherung gepolsterter Sequenzen eingeführt. Angenommen, wir haben ein gepolstertes Eingabe-Minigruppenbatch, das so aussieht:\n",
|
||||
"```\n",
|
||||
"[[1,2,3,4,5],\n",
|
||||
" [6,7,8,0,0],\n",
|
||||
" [9,0,0,0,0]]\n",
|
||||
"```\n",
|
||||
"Hierbei repräsentiert 0 die gepolsterten Werte, und der tatsächliche Längenvektor der Eingabesequenzen ist `[5,3,1]`.\n",
|
||||
"\n",
|
||||
"Um ein RNN effektiv mit gepolsterten Sequenzen zu trainieren, möchten wir die erste Gruppe von RNN-Zellen mit einem großen Minibatch (`[1,6,9]`) starten, dann jedoch die Verarbeitung der dritten Sequenz beenden und mit verkleinerten Minibatches (`[2,7]`, `[3,8]`) weitermachen, und so weiter. Eine gepackte Sequenz wird daher als ein einziger Vektor dargestellt – in unserem Fall `[1,6,9,2,7,3,8,4,5]` – und einem Längenvektor (`[5,3,1]`), aus dem wir das ursprüngliche gepolsterte Minibatch leicht rekonstruieren können.\n",
|
||||
"\n",
|
||||
"Um eine gepackte Sequenz zu erzeugen, können wir die Funktion `torch.nn.utils.rnn.pack_padded_sequence` verwenden. Alle rekurrenten Schichten, einschließlich RNN, LSTM und GRU, unterstützen gepackte Sequenzen als Eingabe und erzeugen gepackte Ausgaben, die mit `torch.nn.utils.rnn.pad_packed_sequence` dekodiert werden können.\n",
|
||||
"\n",
|
||||
"Um eine gepackte Sequenz erzeugen zu können, müssen wir den Längenvektor an das Netzwerk übergeben. Daher benötigen wir eine andere Funktion, um Minibatches vorzubereiten:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def pad_length(b):\n",
|
||||
" # build vectorized sequence\n",
|
||||
" v = [encode(x[1]) for x in b]\n",
|
||||
" # compute max length of a sequence in this minibatch and length sequence itself\n",
|
||||
" len_seq = list(map(len,v))\n",
|
||||
" l = max(len_seq)\n",
|
||||
" return ( # tuple of three tensors - labels, padded features, length sequence\n",
|
||||
" torch.LongTensor([t[0]-1 for t in b]),\n",
|
||||
" torch.stack([torch.nn.functional.pad(torch.tensor(t),(0,l-len(t)),mode='constant',value=0) for t in v]),\n",
|
||||
" torch.tensor(len_seq)\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"train_loader_len = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=pad_length, shuffle=True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Das tatsächliche Netzwerk wäre dem oben genannten `LSTMClassifier` sehr ähnlich, aber der `forward`-Durchlauf erhält sowohl das gepolsterte Minibatch als auch den Vektor der Sequenzlängen. Nach der Berechnung des Embeddings erstellen wir eine gepackte Sequenz, geben sie an die LSTM-Schicht weiter und entpacken anschließend das Ergebnis.\n",
|
||||
"\n",
|
||||
"> **Hinweis**: Tatsächlich verwenden wir das entpackte Ergebnis `x` nicht, da wir die Ausgabe aus den versteckten Schichten für die folgenden Berechnungen nutzen. Daher könnten wir das Entpacken in diesem Code vollständig entfernen. Der Grund, warum wir es hier belassen, ist, dass Sie diesen Code bei Bedarf leicht anpassen können, falls Sie die Netzwerkausgabe in weiteren Berechnungen verwenden möchten.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class LSTMPackClassifier(torch.nn.Module):\n",
|
||||
" def __init__(self, vocab_size, embed_dim, hidden_dim, num_class):\n",
|
||||
" super().__init__()\n",
|
||||
" self.hidden_dim = hidden_dim\n",
|
||||
" self.embedding = torch.nn.Embedding(vocab_size, embed_dim)\n",
|
||||
" self.embedding.weight.data = torch.randn_like(self.embedding.weight.data)-0.5\n",
|
||||
" self.rnn = torch.nn.LSTM(embed_dim,hidden_dim,batch_first=True)\n",
|
||||
" self.fc = torch.nn.Linear(hidden_dim, num_class)\n",
|
||||
"\n",
|
||||
" def forward(self, x, lengths):\n",
|
||||
" batch_size = x.size(0)\n",
|
||||
" x = self.embedding(x)\n",
|
||||
" pad_x = torch.nn.utils.rnn.pack_padded_sequence(x,lengths,batch_first=True,enforce_sorted=False)\n",
|
||||
" pad_x,(h,c) = self.rnn(pad_x)\n",
|
||||
" x, _ = torch.nn.utils.rnn.pad_packed_sequence(pad_x,batch_first=True)\n",
|
||||
" return self.fc(h[-1])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.285625\n",
|
||||
"6400: acc=0.33359375\n",
|
||||
"9600: acc=0.3876041666666667\n",
|
||||
"12800: acc=0.44078125\n",
|
||||
"16000: acc=0.4825\n",
|
||||
"19200: acc=0.5235416666666667\n",
|
||||
"22400: acc=0.5559821428571429\n",
|
||||
"25600: acc=0.58609375\n",
|
||||
"28800: acc=0.6116666666666667\n",
|
||||
"32000: acc=0.63340625\n",
|
||||
"35200: acc=0.6525284090909091\n",
|
||||
"38400: acc=0.668515625\n",
|
||||
"41600: acc=0.6822596153846154\n",
|
||||
"44800: acc=0.6948214285714286\n",
|
||||
"48000: acc=0.7052708333333333\n",
|
||||
"51200: acc=0.71521484375\n",
|
||||
"54400: acc=0.7239889705882353\n",
|
||||
"57600: acc=0.7315277777777778\n",
|
||||
"60800: acc=0.7388486842105263\n",
|
||||
"64000: acc=0.74571875\n",
|
||||
"67200: acc=0.7518303571428572\n",
|
||||
"70400: acc=0.7576988636363636\n",
|
||||
"73600: acc=0.7628940217391305\n",
|
||||
"76800: acc=0.7681510416666667\n",
|
||||
"80000: acc=0.7728125\n",
|
||||
"83200: acc=0.7772235576923077\n",
|
||||
"86400: acc=0.7815393518518519\n",
|
||||
"89600: acc=0.7857700892857142\n",
|
||||
"92800: acc=0.7895043103448276\n",
|
||||
"96000: acc=0.7930520833333333\n",
|
||||
"99200: acc=0.7959072580645161\n",
|
||||
"102400: acc=0.798994140625\n",
|
||||
"105600: acc=0.802064393939394\n",
|
||||
"108800: acc=0.8051378676470589\n",
|
||||
"112000: acc=0.8077857142857143\n",
|
||||
"115200: acc=0.8104600694444445\n",
|
||||
"118400: acc=0.8128293918918919\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(0.029785829671223958, 0.8138166666666666)"
|
||||
]
|
||||
},
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"net = LSTMPackClassifier(vocab_size,64,32,len(classes)).to(device)\n",
|
||||
"train_epoch_emb(net,train_loader_len, lr=0.001,use_pack_sequence=True)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Hinweis:** Sie haben möglicherweise den Parameter `use_pack_sequence` bemerkt, den wir an die Trainingsfunktion übergeben. Derzeit erfordert die Funktion `pack_padded_sequence`, dass der Längensequenz-Tensor auf dem CPU-Gerät ist, und daher muss die Trainingsfunktion vermeiden, die Längensequenz-Daten während des Trainings auf die GPU zu verschieben. Sie können die Implementierung der Funktion `train_emb` in der Datei [`torchnlp.py`](../../../../../lessons/5-NLP/16-RNN/torchnlp.py) einsehen.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Bidirektionale und mehrschichtige RNNs\n",
|
||||
"\n",
|
||||
"In unseren Beispielen haben alle rekurrenten Netzwerke in eine Richtung gearbeitet, von Anfang bis Ende einer Sequenz. Das erscheint natürlich, da es der Art und Weise ähnelt, wie wir lesen und Sprache hören. Allerdings haben wir in vielen praktischen Fällen zufälligen Zugriff auf die Eingabesequenz, weshalb es sinnvoll sein könnte, die rekurrente Berechnung in beide Richtungen auszuführen. Solche Netzwerke werden als **bidirektionale** RNNs bezeichnet, und sie können erstellt werden, indem man den Parameter `bidirectional=True` an den Konstruktor von RNN/LSTM/GRU übergibt.\n",
|
||||
"\n",
|
||||
"Bei der Arbeit mit einem bidirektionalen Netzwerk benötigen wir zwei Zustandsvektoren, einen für jede Richtung. PyTorch kodiert diese Vektoren als einen Vektor mit doppelter Größe, was sehr praktisch ist, da man den resultierenden Zustandsvektor normalerweise an eine vollständig verbundene lineare Schicht übergibt. Man muss lediglich diese Größenänderung berücksichtigen, wenn man die Schicht erstellt.\n",
|
||||
"\n",
|
||||
"Ein rekurrentes Netzwerk, sei es eindirektional oder bidirektional, erfasst bestimmte Muster innerhalb einer Sequenz und kann diese entweder im Zustandsvektor speichern oder in die Ausgabe weitergeben. Ähnlich wie bei konvolutionalen Netzwerken können wir eine weitere rekurrente Schicht auf die erste aufbauen, um Muster höherer Ordnung zu erfassen, die aus den von der ersten Schicht extrahierten Mustern niedriger Ordnung bestehen. Dies führt uns zum Konzept des **mehrschichtigen RNN**, das aus zwei oder mehr rekurrenten Netzwerken besteht, wobei die Ausgabe der vorherigen Schicht als Eingabe an die nächste Schicht weitergegeben wird.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"*Bild aus [diesem großartigen Beitrag](https://towardsdatascience.com/from-a-lstm-cell-to-a-multilayer-lstm-network-with-pytorch-2899eb5696f3) von Fernando López*\n",
|
||||
"\n",
|
||||
"PyTorch macht die Konstruktion solcher Netzwerke einfach, da man lediglich den Parameter `num_layers` an den RNN/LSTM/GRU-Konstruktor übergeben muss, um mehrere rekurrente Schichten automatisch zu erstellen. Das bedeutet auch, dass die Größe des Zustandsvektors proportional zunimmt, und man muss dies berücksichtigen, wenn man die Ausgabe der rekurrenten Schichten verarbeitet.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## RNNs für andere Aufgaben\n",
|
||||
"\n",
|
||||
"In dieser Einheit haben wir gesehen, dass RNNs für die Sequenzklassifikation verwendet werden können. Tatsächlich können sie jedoch viele weitere Aufgaben bewältigen, wie Textgenerierung, maschinelle Übersetzung und mehr. Diese Aufgaben werden wir in der nächsten Einheit betrachten.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Haftungsausschluss**: \nDieses Dokument wurde mit dem KI-Übersetzungsdienst [Co-op Translator](https://github.com/Azure/co-op-translator) übersetzt. Obwohl wir uns um Genauigkeit bemühen, beachten Sie bitte, dass automatisierte Übersetzungen Fehler oder Ungenauigkeiten enthalten können. Das Originaldokument in seiner ursprünglichen Sprache sollte als maßgebliche Quelle betrachtet werden. Für kritische Informationen wird eine professionelle menschliche Übersetzung empfohlen. Wir übernehmen keine Haftung für Missverständnisse oder Fehlinterpretationen, die sich aus der Nutzung dieser Übersetzung ergeben.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.8.12 ('py38')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "522ee52ae3d5ae933e283286254e9a55",
|
||||
"translation_date": "2025-08-31T17:07:58+00:00",
|
||||
"source_file": "lessons/5-NLP/16-RNN/RNNPyTorch.ipynb",
|
||||
"language_code": "de"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
|
|
@ -0,0 +1,460 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Rekurrente neuronale Netze\n",
|
||||
"\n",
|
||||
"Im vorherigen Modul haben wir uns mit reichhaltigen semantischen Repräsentationen von Texten beschäftigt. Die Architektur, die wir verwendet haben, erfasst die aggregierte Bedeutung der Wörter in einem Satz, berücksichtigt jedoch nicht die **Reihenfolge** der Wörter, da die Aggregationsoperation, die den Einbettungen folgt, diese Information aus dem ursprünglichen Text entfernt. Da diese Modelle die Wortreihenfolge nicht darstellen können, sind sie nicht in der Lage, komplexere oder mehrdeutige Aufgaben wie Textgenerierung oder Beantwortung von Fragen zu lösen.\n",
|
||||
"\n",
|
||||
"Um die Bedeutung einer Textsequenz zu erfassen, verwenden wir eine neuronale Netzwerkarchitektur namens **rekurrentes neuronales Netz** (RNN). Bei der Verwendung eines RNN führen wir unseren Satz Token für Token durch das Netzwerk, und das Netzwerk erzeugt einen **Zustand**, den wir dann mit dem nächsten Token erneut in das Netzwerk einspeisen.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Angenommen, wir haben eine Eingabesequenz von Tokens $X_0,\\dots,X_n$, dann erstellt das RNN eine Sequenz von neuronalen Netzwerkblöcken und trainiert diese Sequenz end-to-end mithilfe von Backpropagation. Jeder Netzwerkblock nimmt ein Paar $(X_i,S_i)$ als Eingabe und erzeugt $S_{i+1}$ als Ergebnis. Der finale Zustand $S_n$ oder die Ausgabe $Y_n$ wird in einen linearen Klassifikator eingespeist, um das Ergebnis zu erzeugen. Alle Netzwerkblöcke teilen sich die gleichen Gewichte und werden in einem einzigen Backpropagation-Durchlauf end-to-end trainiert.\n",
|
||||
"\n",
|
||||
"> Die obige Abbildung zeigt ein rekurrentes neuronales Netz in entfalteter Form (links) und in kompakter rekurrenter Darstellung (rechts). Es ist wichtig zu verstehen, dass alle RNN-Zellen die gleichen **teilbaren Gewichte** haben.\n",
|
||||
"\n",
|
||||
"Da Zustandsvektoren $S_0,\\dots,S_n$ durch das Netzwerk weitergegeben werden, kann das RNN sequentielle Abhängigkeiten zwischen Wörtern lernen. Zum Beispiel kann es, wenn das Wort *nicht* irgendwo in der Sequenz erscheint, lernen, bestimmte Elemente innerhalb des Zustandsvektors zu negieren.\n",
|
||||
"\n",
|
||||
"Innerhalb jeder RNN-Zelle befinden sich zwei Gewichtsmatrizen: $W_H$ und $W_I$, sowie ein Bias $b$. Bei jedem RNN-Schritt wird der Ausgabestatus wie folgt berechnet: $S_{i+1} = f(W_H\\times S_i + W_I\\times X_i+b)$, wobei $f$ eine Aktivierungsfunktion ist (oft $\\tanh$).\n",
|
||||
"\n",
|
||||
"> Bei Problemen wie der Textgenerierung (die wir im nächsten Abschnitt behandeln werden) oder der maschinellen Übersetzung möchten wir auch bei jedem RNN-Schritt einen Ausgabewert erhalten. In diesem Fall gibt es eine weitere Matrix $W_O$, und die Ausgabe wird berechnet als $Y_i=f(W_O\\times S_i+b_O)$.\n",
|
||||
"\n",
|
||||
"Schauen wir uns an, wie rekurrente neuronale Netze uns dabei helfen können, unser Nachrichten-Dataset zu klassifizieren.\n",
|
||||
"\n",
|
||||
"> Für die Sandbox-Umgebung müssen wir die folgende Zelle ausführen, um sicherzustellen, dass die erforderliche Bibliothek installiert ist und die Daten vorab geladen werden. Wenn Sie lokal arbeiten, können Sie die folgende Zelle überspringen.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import sys\n",
|
||||
"!{sys.executable} -m pip install --quiet tensorflow_datasets==4.4.0\n",
|
||||
"!cd ~ && wget -q -O - https://mslearntensorflowlp.blob.core.windows.net/data/tfds-ag-news.tgz | tar xz"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import tensorflow as tf\n",
|
||||
"from tensorflow import keras\n",
|
||||
"import tensorflow_datasets as tfds\n",
|
||||
"import numpy as np\n",
|
||||
"\n",
|
||||
"# We are going to be training pretty large models. In order not to face errors, we need\n",
|
||||
"# to set tensorflow option to grow GPU memory allocation when required\n",
|
||||
"physical_devices = tf.config.list_physical_devices('GPU') \n",
|
||||
"if len(physical_devices)>0:\n",
|
||||
" tf.config.experimental.set_memory_growth(physical_devices[0], True)\n",
|
||||
"\n",
|
||||
"ds_train, ds_test = tfds.load('ag_news_subset').values()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"nteract": {
|
||||
"transient": {
|
||||
"deleting": false
|
||||
}
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"Beim Training großer Modelle kann die Zuweisung von GPU-Speicher problematisch werden. Außerdem müssen wir möglicherweise mit verschiedenen Minibatch-Größen experimentieren, damit die Daten in den GPU-Speicher passen und das Training dennoch schnell genug ist. Wenn Sie diesen Code auf Ihrer eigenen GPU-Maschine ausführen, können Sie mit der Anpassung der Minibatch-Größe experimentieren, um das Training zu beschleunigen.\n",
|
||||
"\n",
|
||||
"> **Hinweis**: Es ist bekannt, dass bestimmte Versionen von NVidia-Treibern den Speicher nach dem Training des Modells nicht freigeben. In diesem Notebook führen wir mehrere Beispiele aus, was dazu führen kann, dass der Speicher in bestimmten Konfigurationen erschöpft wird, insbesondere wenn Sie eigene Experimente im selben Notebook durchführen. Wenn Sie auf seltsame Fehler stoßen, wenn Sie mit dem Training des Modells beginnen, sollten Sie den Notebook-Kernel neu starten.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {
|
||||
"collapsed": true,
|
||||
"jupyter": {
|
||||
"outputs_hidden": false,
|
||||
"source_hidden": false
|
||||
},
|
||||
"nteract": {
|
||||
"transient": {
|
||||
"deleting": false
|
||||
}
|
||||
}
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"batch_size = 16\n",
|
||||
"embed_size = 64"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Einfacher RNN-Klassifikator\n",
|
||||
"\n",
|
||||
"Bei einem einfachen RNN ist jede rekurrente Einheit ein einfaches lineares Netzwerk, das einen Eingabevektor und einen Zustandsvektor aufnimmt und einen neuen Zustandsvektor erzeugt. In Keras kann dies durch die `SimpleRNN`-Schicht dargestellt werden.\n",
|
||||
"\n",
|
||||
"Obwohl wir der RNN-Schicht direkt one-hot-codierte Tokens übergeben könnten, ist dies aufgrund ihrer hohen Dimensionalität keine gute Idee. Daher verwenden wir eine Embedding-Schicht, um die Dimensionalität der Wortvektoren zu reduzieren, gefolgt von einer RNN-Schicht und schließlich einem `Dense`-Klassifikator.\n",
|
||||
"\n",
|
||||
"> **Hinweis**: In Fällen, in denen die Dimensionalität nicht so hoch ist, wie beispielsweise bei der Tokenisierung auf Zeichenebene, könnte es sinnvoll sein, one-hot-codierte Tokens direkt in die RNN-Zelle einzuspeisen.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"sequential\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
"Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
"text_vectorization (TextVect (None, None) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"embedding (Embedding) (None, None, 64) 1280000 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"simple_rnn (SimpleRNN) (None, 16) 1296 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense (Dense) (None, 4) 68 \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 1,281,364\n",
|
||||
"Trainable params: 1,281,364\n",
|
||||
"Non-trainable params: 0\n",
|
||||
"_________________________________________________________________\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vocab_size = 20000\n",
|
||||
"\n",
|
||||
"vectorizer = keras.layers.experimental.preprocessing.TextVectorization(\n",
|
||||
" max_tokens=vocab_size,\n",
|
||||
" input_shape=(1,))\n",
|
||||
"\n",
|
||||
"model = keras.models.Sequential([\n",
|
||||
" vectorizer,\n",
|
||||
" keras.layers.Embedding(vocab_size, embed_size),\n",
|
||||
" keras.layers.SimpleRNN(16),\n",
|
||||
" keras.layers.Dense(4,activation='softmax')\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"model.summary()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Hinweis:** Hier verwenden wir eine untrainierte Einbettungsschicht zur Vereinfachung, aber für bessere Ergebnisse können wir eine vortrainierte Einbettungsschicht mit Word2Vec verwenden, wie im vorherigen Abschnitt beschrieben. Es wäre eine gute Übung für dich, diesen Code so anzupassen, dass er mit vortrainierten Einbettungen funktioniert.\n",
|
||||
"\n",
|
||||
"Nun lassen wir unser RNN trainieren. RNNs sind im Allgemeinen recht schwierig zu trainieren, da die Anzahl der Schichten, die bei der Rückpropagation beteiligt sind, sehr groß wird, sobald die RNN-Zellen entlang der Sequenzlänge entfaltet werden. Daher müssen wir eine kleinere Lernrate wählen und das Netzwerk auf einem größeren Datensatz trainieren, um gute Ergebnisse zu erzielen. Dies kann ziemlich lange dauern, daher wird die Verwendung einer GPU empfohlen.\n",
|
||||
"\n",
|
||||
"Um den Prozess zu beschleunigen, werden wir das RNN-Modell nur mit Nachrichtentiteln trainieren und die Beschreibung weglassen. Du kannst versuchen, mit der Beschreibung zu trainieren und sehen, ob du das Modell zum Laufen bringen kannst.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Training vectorizer\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def extract_title(x):\n",
|
||||
" return x['title']\n",
|
||||
"\n",
|
||||
"def tupelize_title(x):\n",
|
||||
" return (extract_title(x),x['label'])\n",
|
||||
"\n",
|
||||
"print('Training vectorizer')\n",
|
||||
"vectorizer.adapt(ds_train.take(2000).map(extract_title))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"7500/7500 [==============================] - 82s 11ms/step - loss: 0.6629 - acc: 0.7623 - val_loss: 0.5559 - val_acc: 0.7995\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7f3e0030d350>"
|
||||
]
|
||||
},
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n",
|
||||
"model.fit(ds_train.map(tupelize_title).batch(batch_size),validation_data=ds_test.map(tupelize_title).batch(batch_size))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"nteract": {
|
||||
"transient": {
|
||||
"deleting": false
|
||||
}
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"> **Hinweis**: Die Genauigkeit ist hier wahrscheinlich geringer, da wir nur mit Nachrichtentiteln trainieren.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Überprüfung von variablen Sequenzen\n",
|
||||
"\n",
|
||||
"Denken Sie daran, dass die `TextVectorization`-Schicht automatisch Sequenzen mit variabler Länge in einem Minibatch mit Pad-Tokens auffüllt. Es stellt sich heraus, dass diese Tokens auch am Training teilnehmen und die Konvergenz des Modells erschweren können.\n",
|
||||
"\n",
|
||||
"Es gibt mehrere Ansätze, die wir verfolgen können, um die Menge an Padding zu minimieren. Einer davon ist, den Datensatz nach Sequenzlänge neu zu ordnen und alle Sequenzen nach Größe zu gruppieren. Dies kann mit der Funktion `tf.data.experimental.bucket_by_sequence_length` durchgeführt werden (siehe [Dokumentation](https://www.tensorflow.org/api_docs/python/tf/data/experimental/bucket_by_sequence_length)).\n",
|
||||
"\n",
|
||||
"Ein weiterer Ansatz ist die Verwendung von **Maskierung**. In Keras unterstützen einige Schichten zusätzliche Eingaben, die zeigen, welche Tokens beim Training berücksichtigt werden sollen. Um Maskierung in unser Modell zu integrieren, können wir entweder eine separate `Masking`-Schicht einfügen ([Dokumentation](https://keras.io/api/layers/core_layers/masking/)) oder den Parameter `mask_zero=True` in unserer `Embedding`-Schicht angeben.\n",
|
||||
"\n",
|
||||
"> **Note**: Dieses Training wird etwa 5 Minuten dauern, um eine Epoche auf dem gesamten Datensatz abzuschließen. Sie können das Training jederzeit unterbrechen, wenn Sie die Geduld verlieren. Alternativ können Sie die Menge der für das Training verwendeten Daten begrenzen, indem Sie nach den Datensätzen `ds_train` und `ds_test` eine `.take(...)`-Klausel hinzufügen.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"7500/7500 [==============================] - 371s 49ms/step - loss: 0.5401 - acc: 0.8079 - val_loss: 0.3780 - val_acc: 0.8822\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7f3dec118850>"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def extract_text(x):\n",
|
||||
" return x['title']+' '+x['description']\n",
|
||||
"\n",
|
||||
"def tupelize(x):\n",
|
||||
" return (extract_text(x),x['label'])\n",
|
||||
"\n",
|
||||
"model = keras.models.Sequential([\n",
|
||||
" vectorizer,\n",
|
||||
" keras.layers.Embedding(vocab_size,embed_size,mask_zero=True),\n",
|
||||
" keras.layers.SimpleRNN(16),\n",
|
||||
" keras.layers.Dense(4,activation='softmax')\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Jetzt, da wir Maskierung verwenden, können wir das Modell mit dem gesamten Datensatz aus Titeln und Beschreibungen trainieren.\n",
|
||||
"\n",
|
||||
"> **Hinweis**: Ist Ihnen aufgefallen, dass wir einen Vektorisierer verwenden, der auf den Nachrichtentiteln trainiert wurde, und nicht auf dem gesamten Artikeltext? Das könnte dazu führen, dass einige Tokens ignoriert werden. Es wäre daher besser, den Vektorisierer neu zu trainieren. Allerdings könnte der Effekt nur sehr gering sein, daher bleiben wir der Einfachheit halber beim vorher trainierten Vektorisierer.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## LSTM: Langzeit-Kurzzeitspeicher\n",
|
||||
"\n",
|
||||
"Eines der Hauptprobleme von RNNs sind **verschwindende Gradienten**. RNNs können ziemlich lang sein und haben möglicherweise Schwierigkeiten, die Gradienten während der Rückwärtspropagation bis zur ersten Schicht des Netzwerks zurückzuführen. Wenn dies passiert, kann das Netzwerk keine Beziehungen zwischen weit entfernten Tokens lernen. Eine Möglichkeit, dieses Problem zu vermeiden, besteht darin, **explizites Zustandsmanagement** durch den Einsatz von **Gates** einzuführen. Die beiden gängigsten Architekturen, die Gates verwenden, sind **Langzeit-Kurzzeitspeicher** (LSTM) und **Gated Relay Unit** (GRU). Hier werden wir uns mit LSTMs beschäftigen.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Ein LSTM-Netzwerk ist ähnlich wie ein RNN organisiert, aber es gibt zwei Zustände, die von Schicht zu Schicht weitergegeben werden: den tatsächlichen Zustand $c$ und den versteckten Vektor $h$. In jeder Einheit wird der versteckte Vektor $h_{t-1}$ mit der Eingabe $x_t$ kombiniert, und zusammen steuern sie, was mit dem Zustand $c_t$ und der Ausgabe $h_{t}$ durch **Gates** geschieht. Jedes Gate hat eine Sigmoid-Aktivierung (Ausgabe im Bereich $[0,1]$), die als bitweises Maskieren betrachtet werden kann, wenn sie mit dem Zustandsvektor multipliziert wird. LSTMs haben die folgenden Gates (von links nach rechts im obigen Bild):\n",
|
||||
"* **Vergessens-Gate**, das bestimmt, welche Komponenten des Vektors $c_{t-1}$ wir vergessen und welche wir durchlassen müssen.\n",
|
||||
"* **Eingabe-Gate**, das bestimmt, wie viele Informationen aus dem Eingabevektor und dem vorherigen versteckten Vektor in den Zustandsvektor aufgenommen werden sollen.\n",
|
||||
"* **Ausgabe-Gate**, das den neuen Zustandsvektor nimmt und entscheidet, welche seiner Komponenten verwendet werden, um den neuen versteckten Vektor $h_t$ zu erzeugen.\n",
|
||||
"\n",
|
||||
"Die Komponenten des Zustands $c$ können als Flags betrachtet werden, die ein- und ausgeschaltet werden können. Zum Beispiel, wenn wir im Sequenzverlauf den Namen *Alice* sehen, vermuten wir, dass es sich um eine Frau handelt, und setzen das Flag im Zustand, das anzeigt, dass wir ein weibliches Substantiv im Satz haben. Wenn wir später die Wörter *und Tom* sehen, setzen wir das Flag, das anzeigt, dass wir ein Plural-Substantiv haben. Durch die Manipulation des Zustands können wir also die grammatikalischen Eigenschaften des Satzes verfolgen.\n",
|
||||
"\n",
|
||||
"> **Note**: Hier ist eine großartige Ressource, um die Interna von LSTMs zu verstehen: [Understanding LSTM Networks](https://colah.github.io/posts/2015-08-Understanding-LSTMs/) von Christopher Olah.\n",
|
||||
"\n",
|
||||
"Obwohl die interne Struktur einer LSTM-Zelle komplex erscheinen mag, verbirgt Keras diese Implementierung in der `LSTM`-Schicht. Das Einzige, was wir im obigen Beispiel tun müssen, ist, die rekurrente Schicht zu ersetzen:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"15000/15000 [==============================] - 188s 13ms/step - loss: 0.5692 - acc: 0.7916 - val_loss: 0.3441 - val_acc: 0.8870\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7f3d6af5c350>"
|
||||
]
|
||||
},
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model = keras.models.Sequential([\n",
|
||||
" vectorizer,\n",
|
||||
" keras.layers.Embedding(vocab_size, embed_size),\n",
|
||||
" keras.layers.LSTM(8),\n",
|
||||
" keras.layers.Dense(4,activation='softmax')\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(8),validation_data=ds_test.map(tupelize).batch(8))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Bidirektionale und mehrschichtige RNNs\n",
|
||||
"\n",
|
||||
"In unseren bisherigen Beispielen arbeiten die rekurrenten Netzwerke von Anfang bis Ende einer Sequenz. Das erscheint uns natürlich, da es der Richtung entspricht, in der wir lesen oder Sprache hören. Für Szenarien, die einen zufälligen Zugriff auf die Eingabesequenz erfordern, ist es jedoch sinnvoller, die rekurrente Berechnung in beide Richtungen auszuführen. RNNs, die Berechnungen in beide Richtungen ermöglichen, werden als **bidirektionale** RNNs bezeichnet, und sie können erstellt werden, indem die rekurrente Schicht mit einer speziellen `Bidirectional`-Schicht umwickelt wird.\n",
|
||||
"\n",
|
||||
"> **Note**: Die `Bidirectional`-Schicht erstellt zwei Kopien der darin enthaltenen Schicht und setzt die Eigenschaft `go_backwards` einer dieser Kopien auf `True`, sodass sie in die entgegengesetzte Richtung entlang der Sequenz läuft.\n",
|
||||
"\n",
|
||||
"Rekurrente Netzwerke, ob unidirektional oder bidirektional, erfassen Muster innerhalb einer Sequenz und speichern sie in Zustandsvektoren oder geben sie als Ausgabe zurück. Wie bei konvolutionalen Netzwerken können wir eine weitere rekurrente Schicht hinzufügen, die der ersten folgt, um höherwertige Muster zu erfassen, die aus niedrigeren Mustern bestehen, die von der ersten Schicht extrahiert wurden. Dies führt uns zum Konzept eines **mehrschichtigen RNN**, das aus zwei oder mehr rekurrenten Netzwerken besteht, wobei die Ausgabe der vorherigen Schicht als Eingabe an die nächste Schicht weitergegeben wird.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"*Bild aus [diesem großartigen Beitrag](https://towardsdatascience.com/from-a-lstm-cell-to-a-multilayer-lstm-network-with-pytorch-2899eb5696f3) von Fernando López.*\n",
|
||||
"\n",
|
||||
"Keras macht den Aufbau dieser Netzwerke zu einer einfachen Aufgabe, da Sie einfach weitere rekurrente Schichten zum Modell hinzufügen müssen. Für alle Schichten außer der letzten müssen wir den Parameter `return_sequences=True` angeben, da wir möchten, dass die Schicht alle Zwischenzustände zurückgibt und nicht nur den Endzustand der rekurrenten Berechnung.\n",
|
||||
"\n",
|
||||
"Lassen Sie uns ein zweischichtiges bidirektionales LSTM für unser Klassifizierungsproblem erstellen.\n",
|
||||
"\n",
|
||||
"> **Note** Dieser Code benötigt erneut ziemlich viel Zeit, um abgeschlossen zu werden, aber er liefert uns die höchste Genauigkeit, die wir bisher gesehen haben. Es könnte sich also lohnen, zu warten und das Ergebnis zu sehen.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"5044/7500 [===================>..........] - ETA: 2:33 - loss: 0.3709 - acc: 0.8706\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r5045/7500 [===================>..........] - ETA: 2:33 - loss: 0.3709 - acc: 0.8706"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model = keras.models.Sequential([\n",
|
||||
" vectorizer,\n",
|
||||
" keras.layers.Embedding(vocab_size, 128, mask_zero=True),\n",
|
||||
" keras.layers.Bidirectional(keras.layers.LSTM(64,return_sequences=True)),\n",
|
||||
" keras.layers.Bidirectional(keras.layers.LSTM(64)), \n",
|
||||
" keras.layers.Dense(4,activation='softmax')\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(batch_size),\n",
|
||||
" validation_data=ds_test.map(tupelize).batch(batch_size))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## RNNs für andere Aufgaben\n",
|
||||
"\n",
|
||||
"Bis jetzt haben wir uns darauf konzentriert, RNNs zur Klassifikation von Textsequenzen zu verwenden. Aber sie können noch viele weitere Aufgaben bewältigen, wie zum Beispiel Textgenerierung und maschinelle Übersetzung — diese Aufgaben werden wir in der nächsten Einheit betrachten.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Haftungsausschluss**: \nDieses Dokument wurde mit dem KI-Übersetzungsdienst [Co-op Translator](https://github.com/Azure/co-op-translator) übersetzt. Obwohl wir uns um Genauigkeit bemühen, beachten Sie bitte, dass automatisierte Übersetzungen Fehler oder Ungenauigkeiten enthalten können. Das Originaldokument in seiner ursprünglichen Sprache sollte als maßgebliche Quelle betrachtet werden. Für kritische Informationen wird eine professionelle menschliche Übersetzung empfohlen. Wir übernehmen keine Haftung für Missverständnisse oder Fehlinterpretationen, die sich aus der Nutzung dieser Übersetzung ergeben.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernel_info": {
|
||||
"name": "conda-env-py37_tensorflow-py"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "py37_tensorflow",
|
||||
"language": "python",
|
||||
"name": "conda-env-py37_tensorflow-py"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.7.9"
|
||||
},
|
||||
"nteract": {
|
||||
"version": "nteract-front-end@1.0.0"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "81351e61f619b432ff51010a4f993194",
|
||||
"translation_date": "2025-08-31T17:05:22+00:00",
|
||||
"source_file": "lessons/5-NLP/16-RNN/RNNTF.ipynb",
|
||||
"language_code": "de"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
|
|
@ -0,0 +1,414 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Generative Netzwerke\n",
|
||||
"\n",
|
||||
"Rekurrente neuronale Netzwerke (RNNs) und ihre Varianten mit gated Zellen wie Long Short Term Memory Cells (LSTMs) und Gated Recurrent Units (GRUs) bieten einen Mechanismus für Sprachmodellierung, d.h. sie können die Reihenfolge von Wörtern lernen und Vorhersagen für das nächste Wort in einer Sequenz treffen. Dies ermöglicht es uns, RNNs für **generative Aufgaben** zu nutzen, wie z.B. gewöhnliche Textgenerierung, maschinelle Übersetzung und sogar Bildbeschriftung.\n",
|
||||
"\n",
|
||||
"In der RNN-Architektur, die wir in der vorherigen Einheit besprochen haben, produzierte jede RNN-Einheit den nächsten versteckten Zustand als Ausgabe. Wir können jedoch auch eine weitere Ausgabe zu jeder rekurrenten Einheit hinzufügen, die es uns ermöglicht, eine **Sequenz** auszugeben (die genauso lang ist wie die ursprüngliche Sequenz). Darüber hinaus können wir RNN-Einheiten verwenden, die bei jedem Schritt keine Eingabe akzeptieren, sondern lediglich einen Anfangszustandsvektor nehmen und dann eine Sequenz von Ausgaben erzeugen.\n",
|
||||
"\n",
|
||||
"In diesem Notebook konzentrieren wir uns auf einfache generative Modelle, die uns helfen, Text zu generieren. Der Einfachheit halber bauen wir ein **zeichenbasiertes Netzwerk**, das Text Buchstabe für Buchstabe generiert. Während des Trainings müssen wir einen Textkorpus nehmen und ihn in Buchstabenfolgen aufteilen.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Loading dataset...\n",
|
||||
"Building vocab...\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"import torchtext\n",
|
||||
"import numpy as np\n",
|
||||
"from torchnlp import *\n",
|
||||
"train_dataset,test_dataset,classes,vocab = load_dataset()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Aufbau eines Zeichen-Vokabulars\n",
|
||||
"\n",
|
||||
"Um ein generatives Netzwerk auf Zeichenebene zu erstellen, müssen wir den Text in einzelne Zeichen statt in Wörter aufteilen. Dies kann durch die Definition eines anderen Tokenizers erfolgen:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Vocabulary size = 82\n",
|
||||
"Encoding of 'a' is 1\n",
|
||||
"Character with code 13 is c\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def char_tokenizer(words):\n",
|
||||
" return list(words) #[word for word in words]\n",
|
||||
"\n",
|
||||
"counter = collections.Counter()\n",
|
||||
"for (label, line) in train_dataset:\n",
|
||||
" counter.update(char_tokenizer(line))\n",
|
||||
"vocab = torchtext.vocab.vocab(counter)\n",
|
||||
"\n",
|
||||
"vocab_size = len(vocab)\n",
|
||||
"print(f\"Vocabulary size = {vocab_size}\")\n",
|
||||
"print(f\"Encoding of 'a' is {vocab.get_stoi()['a']}\")\n",
|
||||
"print(f\"Character with code 13 is {vocab.get_itos()[13]}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Lassen Sie uns das Beispiel sehen, wie wir den Text aus unserem Datensatz codieren können:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"tensor([ 0, 1, 2, 2, 3, 4, 5, 6, 3, 7, 8, 1, 9, 10, 3, 11, 2, 1,\n",
|
||||
" 12, 3, 7, 1, 13, 14, 3, 15, 16, 5, 17, 3, 5, 18, 8, 3, 7, 2,\n",
|
||||
" 1, 13, 14, 3, 19, 20, 8, 21, 5, 8, 9, 10, 22, 3, 20, 8, 21, 5,\n",
|
||||
" 8, 9, 10, 3, 23, 3, 4, 18, 17, 9, 5, 23, 10, 8, 2, 2, 8, 9,\n",
|
||||
" 10, 24, 3, 0, 1, 2, 2, 3, 4, 5, 9, 8, 8, 5, 25, 10, 3, 26,\n",
|
||||
" 12, 27, 16, 26, 2, 27, 16, 28, 29, 30, 1, 16, 26, 3, 17, 31, 3, 21,\n",
|
||||
" 2, 5, 9, 1, 23, 13, 32, 16, 27, 13, 10, 24, 3, 1, 9, 8, 3, 10,\n",
|
||||
" 8, 8, 27, 16, 28, 3, 28, 9, 8, 8, 16, 3, 1, 28, 1, 27, 16, 6])"
|
||||
]
|
||||
},
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def enc(x):\n",
|
||||
" return torch.LongTensor(encode(x,voc=vocab,tokenizer=char_tokenizer))\n",
|
||||
"\n",
|
||||
"enc(train_dataset[0][1])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Training eines generativen RNN\n",
|
||||
"\n",
|
||||
"So werden wir das RNN trainieren, um Text zu generieren: Bei jedem Schritt nehmen wir eine Zeichenfolge der Länge `nchars` und lassen das Netzwerk für jedes Eingabezeichen das nächste Ausgabesymbol vorhersagen:\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Je nach Szenario möchten wir möglicherweise auch einige Sonderzeichen einbeziehen, wie zum Beispiel *Ende-der-Sequenz* `<eos>`. In unserem Fall möchten wir das Netzwerk jedoch für die endlose Textgenerierung trainieren. Daher legen wir die Größe jeder Sequenz auf `nchars` Tokens fest. Folglich besteht jedes Trainingsexemplar aus `nchars` Eingaben und `nchars` Ausgaben (die Eingabesequenz, um ein Symbol nach links verschoben). Ein Minibatch wird aus mehreren solcher Sequenzen bestehen.\n",
|
||||
"\n",
|
||||
"Die Minibatches werden wir folgendermaßen generieren: Wir nehmen jeden Nachrichtentext der Länge `l` und erzeugen daraus alle möglichen Eingabe-Ausgabe-Kombinationen (es wird `l-nchars` solcher Kombinationen geben). Diese bilden ein Minibatch, und die Größe der Minibatches wird bei jedem Trainingsschritt unterschiedlich sein.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(tensor([[ 0, 1, 2, ..., 28, 29, 30],\n",
|
||||
" [ 1, 2, 2, ..., 29, 30, 1],\n",
|
||||
" [ 2, 2, 3, ..., 30, 1, 16],\n",
|
||||
" ...,\n",
|
||||
" [20, 8, 21, ..., 1, 28, 1],\n",
|
||||
" [ 8, 21, 5, ..., 28, 1, 27],\n",
|
||||
" [21, 5, 8, ..., 1, 27, 16]]),\n",
|
||||
" tensor([[ 1, 2, 2, ..., 29, 30, 1],\n",
|
||||
" [ 2, 2, 3, ..., 30, 1, 16],\n",
|
||||
" [ 2, 3, 4, ..., 1, 16, 26],\n",
|
||||
" ...,\n",
|
||||
" [ 8, 21, 5, ..., 28, 1, 27],\n",
|
||||
" [21, 5, 8, ..., 1, 27, 16],\n",
|
||||
" [ 5, 8, 9, ..., 27, 16, 6]]))"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"nchars = 100\n",
|
||||
"\n",
|
||||
"def get_batch(s,nchars=nchars):\n",
|
||||
" ins = torch.zeros(len(s)-nchars,nchars,dtype=torch.long,device=device)\n",
|
||||
" outs = torch.zeros(len(s)-nchars,nchars,dtype=torch.long,device=device)\n",
|
||||
" for i in range(len(s)-nchars):\n",
|
||||
" ins[i] = enc(s[i:i+nchars])\n",
|
||||
" outs[i] = enc(s[i+1:i+nchars+1])\n",
|
||||
" return ins,outs\n",
|
||||
"\n",
|
||||
"get_batch(train_dataset[0][1])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Nun definieren wir das Generatornetzwerk. Es kann auf jeder rekurrenten Zelle basieren, die wir in der vorherigen Einheit besprochen haben (einfach, LSTM oder GRU). In unserem Beispiel verwenden wir LSTM.\n",
|
||||
"\n",
|
||||
"Da das Netzwerk Zeichen als Eingabe erhält und die Vokabulärgröße relativ klein ist, benötigen wir keine Embedding-Schicht. Die one-hot-codierte Eingabe kann direkt an die LSTM-Zelle übergeben werden. Da wir jedoch Zeichen als Zahlen übergeben, müssen wir diese vor der Übergabe an die LSTM-Zelle one-hot-codieren. Dies geschieht, indem die Funktion `one_hot` während des `forward`-Durchlaufs aufgerufen wird. Der Ausgabe-Encoder wäre eine lineare Schicht, die den versteckten Zustand in eine one-hot-codierte Ausgabe umwandelt.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class LSTMGenerator(torch.nn.Module):\n",
|
||||
" def __init__(self, vocab_size, hidden_dim):\n",
|
||||
" super().__init__()\n",
|
||||
" self.rnn = torch.nn.LSTM(vocab_size,hidden_dim,batch_first=True)\n",
|
||||
" self.fc = torch.nn.Linear(hidden_dim, vocab_size)\n",
|
||||
"\n",
|
||||
" def forward(self, x, s=None):\n",
|
||||
" x = torch.nn.functional.one_hot(x,vocab_size).to(torch.float32)\n",
|
||||
" x,s = self.rnn(x,s)\n",
|
||||
" return self.fc(x),s"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Während des Trainings möchten wir in der Lage sein, generierten Text zu sampeln. Dazu definieren wir die Funktion `generate`, die eine Ausgabestring der Länge `size` erzeugt, beginnend mit dem Anfangsstring `start`.\n",
|
||||
"\n",
|
||||
"So funktioniert es: Zuerst geben wir den gesamten Startstring durch das Netzwerk und erhalten den Ausgabestatus `s` und das nächste vorhergesagte Zeichen `out`. Da `out` one-hot codiert ist, verwenden wir `argmax`, um den Index des Zeichens `nc` im Vokabular zu erhalten, und nutzen `itos`, um das tatsächliche Zeichen zu ermitteln und es der resultierenden Zeichenliste `chars` hinzuzufügen. Dieser Prozess der Generierung eines Zeichens wird `size`-mal wiederholt, um die gewünschte Anzahl von Zeichen zu erzeugen.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def generate(net,size=100,start='today '):\n",
|
||||
" chars = list(start)\n",
|
||||
" out, s = net(enc(chars).view(1,-1).to(device))\n",
|
||||
" for i in range(size):\n",
|
||||
" nc = torch.argmax(out[0][-1])\n",
|
||||
" chars.append(vocab.get_itos()[nc])\n",
|
||||
" out, s = net(nc.view(1,-1),s)\n",
|
||||
" return ''.join(chars)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Jetzt geht's ans Training! Die Trainingsschleife ist fast dieselbe wie in all unseren vorherigen Beispielen, aber anstelle der Genauigkeit geben wir alle 1000 Epochen einen Ausschnitt des generierten Textes aus.\n",
|
||||
"\n",
|
||||
"Besondere Aufmerksamkeit sollte der Art und Weise gewidmet werden, wie wir den Verlust berechnen. Wir müssen den Verlust basierend auf der One-Hot-codierten Ausgabe `out` und dem erwarteten Text `text_out` berechnen, welcher die Liste der Zeichenindizes darstellt. Glücklicherweise erwartet die Funktion `cross_entropy` als erstes Argument die unnormalisierte Netzwerkausgabe und als zweites die Klassennummer – genau das, was wir haben. Sie führt außerdem eine automatische Mittelung über die Minibatch-Größe durch.\n",
|
||||
"\n",
|
||||
"Wir begrenzen das Training außerdem auf `samples_to_train` Samples, um nicht zu lange warten zu müssen. Wir ermutigen euch, zu experimentieren und längeres Training auszuprobieren, möglicherweise über mehrere Epochen (in diesem Fall müsstet ihr eine weitere Schleife um diesen Code erstellen).\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Current loss = 4.398899078369141\n",
|
||||
"today sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr s\n",
|
||||
"Current loss = 2.161320447921753\n",
|
||||
"today and to the tor to to the tor to to the tor to to the tor to to the tor to to the tor to to the tor t\n",
|
||||
"Current loss = 1.6722588539123535\n",
|
||||
"today and the court to the could to the could to the could to the could to the could to the could to the c\n",
|
||||
"Current loss = 2.423795223236084\n",
|
||||
"today and a second to the conternation of the conternation of the conternation of the conternation of the \n",
|
||||
"Current loss = 1.702607274055481\n",
|
||||
"today and the company to the company to the company to the company to the company to the company to the co\n",
|
||||
"Current loss = 1.692358136177063\n",
|
||||
"today and the company to the company to the company to the company to the company to the company to the co\n",
|
||||
"Current loss = 1.9722288846969604\n",
|
||||
"today and the control the control the control the control the control the control the control the control \n",
|
||||
"Current loss = 1.8705692291259766\n",
|
||||
"today and the second to the second to the second to the second to the second to the second to the second t\n",
|
||||
"Current loss = 1.7626899480819702\n",
|
||||
"today and a security and a security and a security and a security and a security and a security and a secu\n",
|
||||
"Current loss = 1.5574463605880737\n",
|
||||
"today and the company and the company and the company and the company and the company and the company and \n",
|
||||
"Current loss = 1.5620026588439941\n",
|
||||
"today and the be that the be the be that the be the be that the be the be that the be the be that the be t\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"net = LSTMGenerator(vocab_size,64).to(device)\n",
|
||||
"\n",
|
||||
"samples_to_train = 10000\n",
|
||||
"optimizer = torch.optim.Adam(net.parameters(),0.01)\n",
|
||||
"loss_fn = torch.nn.CrossEntropyLoss()\n",
|
||||
"net.train()\n",
|
||||
"for i,x in enumerate(train_dataset):\n",
|
||||
" # x[0] is class label, x[1] is text\n",
|
||||
" if len(x[1])-nchars<10:\n",
|
||||
" continue\n",
|
||||
" samples_to_train-=1\n",
|
||||
" if not samples_to_train: break\n",
|
||||
" text_in, text_out = get_batch(x[1])\n",
|
||||
" optimizer.zero_grad()\n",
|
||||
" out,s = net(text_in)\n",
|
||||
" loss = torch.nn.functional.cross_entropy(out.view(-1,vocab_size),text_out.flatten()) #cross_entropy(out,labels)\n",
|
||||
" loss.backward()\n",
|
||||
" optimizer.step()\n",
|
||||
" if i%1000==0:\n",
|
||||
" print(f\"Current loss = {loss.item()}\")\n",
|
||||
" print(generate(net))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Dieses Beispiel erzeugt bereits ziemlich guten Text, aber es gibt mehrere Möglichkeiten, ihn weiter zu verbessern:\n",
|
||||
"\n",
|
||||
"* **Bessere Minibatch-Erstellung**. Die Art und Weise, wie wir die Daten für das Training vorbereitet haben, bestand darin, ein Minibatch aus einer einzelnen Probe zu generieren. Das ist nicht ideal, da die Minibatches alle unterschiedliche Größen haben und einige von ihnen nicht einmal generiert werden können, weil der Text kleiner als `nchars` ist. Außerdem nutzen kleine Minibatches die GPU nicht ausreichend aus. Es wäre klüger, einen großen Textabschnitt aus allen Proben zu nehmen, dann alle Eingabe-Ausgabe-Paare zu generieren, sie zu mischen und Minibatches gleicher Größe zu erstellen.\n",
|
||||
"\n",
|
||||
"* **Mehrschichtige LSTM**. Es macht Sinn, 2 oder 3 Schichten von LSTM-Zellen auszuprobieren. Wie wir in der vorherigen Einheit erwähnt haben, extrahiert jede Schicht eines LSTM bestimmte Muster aus dem Text. Im Fall eines zeichenbasierten Generators können wir erwarten, dass die untere LSTM-Schicht für die Extraktion von Silben verantwortlich ist, während die höheren Schichten für Wörter und Wortkombinationen zuständig sind. Dies kann einfach implementiert werden, indem man den Parameter für die Anzahl der Schichten an den LSTM-Konstruktor übergibt.\n",
|
||||
"\n",
|
||||
"* Es könnte auch interessant sein, mit **GRU-Einheiten** zu experimentieren, um zu sehen, welche besser funktionieren, sowie mit **unterschiedlichen Größen der versteckten Schichten**. Eine zu große versteckte Schicht kann zu Overfitting führen (z. B. lernt das Netzwerk den exakten Text), während eine kleinere Größe möglicherweise keine guten Ergebnisse liefert.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Weiche Textgenerierung und Temperatur\n",
|
||||
"\n",
|
||||
"In der vorherigen Definition von `generate` haben wir immer das Zeichen mit der höchsten Wahrscheinlichkeit als nächstes Zeichen im generierten Text ausgewählt. Dies führte oft dazu, dass sich der Text zwischen denselben Zeichenfolgen immer wieder \"wiederholte\", wie in diesem Beispiel:\n",
|
||||
"```\n",
|
||||
"today of the second the company and a second the company ...\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"Wenn wir uns jedoch die Wahrscheinlichkeitsverteilung für das nächste Zeichen ansehen, könnte es sein, dass der Unterschied zwischen den höchsten Wahrscheinlichkeiten nicht groß ist, z. B. kann ein Zeichen eine Wahrscheinlichkeit von 0,2 haben, ein anderes - 0,19 usw. Wenn wir beispielsweise das nächste Zeichen in der Sequenz '*play*' suchen, könnte das nächste Zeichen genauso gut ein Leerzeichen oder **e** sein (wie im Wort *player*).\n",
|
||||
"\n",
|
||||
"Das führt uns zu der Erkenntnis, dass es nicht immer \"fair\" ist, das Zeichen mit der höchsten Wahrscheinlichkeit auszuwählen, da die Wahl des zweitwahrscheinlichsten Zeichens dennoch zu sinnvollem Text führen könnte. Es ist klüger, Zeichen aus der Wahrscheinlichkeitsverteilung zu **samplen**, die durch die Netzwerkausgabe vorgegeben wird.\n",
|
||||
"\n",
|
||||
"Dieses Sampling kann mit der Funktion `multinomial` durchgeführt werden, die die sogenannte **multinomiale Verteilung** implementiert. Eine Funktion, die diese **weiche** Textgenerierung umsetzt, ist unten definiert:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"--- Temperature = 0.3\n",
|
||||
"Today and a company and complete an all the land the restrational the as a security and has provers the pay to and a report and the computer in the stand has filities and working the law the stations for a company and with the company and the final the first company and refight of the state and and workin\n",
|
||||
"\n",
|
||||
"--- Temperature = 0.8\n",
|
||||
"Today he oniis its first to Aus bomblaties the marmation a to manan boogot that pirate assaid a relaid their that goverfin the the Cappets Ecrotional Assonia Cition targets it annight the w scyments Blamity #39;s TVeer Diercheg Reserals fran envyuil that of ster said access what succers of Dour-provelith\n",
|
||||
"\n",
|
||||
"--- Temperature = 1.0\n",
|
||||
"Today holy they a 11 will meda a toket subsuaties, engins for Chanos, they's has stainger past to opening orital his thempting new Nattona was al innerforder advan-than #36;s night year his religuled talitatian what the but with Wednesday to Justment will wemen of Mark CCC Camp as Timed Nae wome a leaders\n",
|
||||
"\n",
|
||||
"--- Temperature = 1.3\n",
|
||||
"Today gpone 2.5 fech atcusion poor cocles toparsdorM.cht Line Pamage put 43 his calt lowed to the book, that has authh-the silia rruch ailing to'ory andhes beutirsimi- Aefffive heading offil an auf eacklets is charged evis, Gunymy oy) Mony has it after-sloythyor loveId out filme, the Natabl -Najuntaxiggs \n",
|
||||
"\n",
|
||||
"--- Temperature = 1.8\n",
|
||||
"Today plary, P.slan chly\\401 mardregationly #39;t 8.1Mide) closes ,filtcon alfly playin roven!\\grea.-QFBEP: Iss onfarchQ/itilia CCf Zivesigntwasta orce.-Peul-aw.uicrin of fuglinfsut aftaningwo, MIEX awayew Aice Woiduar Corvagiugge oppo esig ThusBratourid canthly-RyI.co lagitems\\eexciaishes.conBabntusmor I\n",
|
||||
"\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def generate_soft(net,size=100,start='today ',temperature=1.0):\n",
|
||||
" chars = list(start)\n",
|
||||
" out, s = net(enc(chars).view(1,-1).to(device))\n",
|
||||
" for i in range(size):\n",
|
||||
" #nc = torch.argmax(out[0][-1])\n",
|
||||
" out_dist = out[0][-1].div(temperature).exp()\n",
|
||||
" nc = torch.multinomial(out_dist,1)[0]\n",
|
||||
" chars.append(vocab.get_itos()[nc])\n",
|
||||
" out, s = net(nc.view(1,-1),s)\n",
|
||||
" return ''.join(chars)\n",
|
||||
" \n",
|
||||
"for i in [0.3,0.8,1.0,1.3,1.8]:\n",
|
||||
" print(f\"--- Temperature = {i}\\n{generate_soft(net,size=300,start='Today ',temperature=i)}\\n\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Wir haben einen weiteren Parameter namens **Temperatur** eingeführt, der angibt, wie strikt wir uns an die höchste Wahrscheinlichkeit halten sollten. Wenn die Temperatur 1,0 beträgt, führen wir eine faire multinomiale Stichprobe durch, und wenn die Temperatur gegen unendlich geht, werden alle Wahrscheinlichkeiten gleich, und wir wählen zufällig das nächste Zeichen aus. Im untenstehenden Beispiel können wir beobachten, dass der Text bedeutungslos wird, wenn wir die Temperatur zu stark erhöhen, und er ähnelt einem \"zyklischen\" hart generierten Text, wenn er sich näher an 0 bewegt.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Haftungsausschluss**: \nDieses Dokument wurde mit dem KI-Übersetzungsdienst [Co-op Translator](https://github.com/Azure/co-op-translator) übersetzt. Obwohl wir uns um Genauigkeit bemühen, beachten Sie bitte, dass automatisierte Übersetzungen Fehler oder Ungenauigkeiten enthalten können. Das Originaldokument in seiner ursprünglichen Sprache sollte als maßgebliche Quelle betrachtet werden. Für kritische Informationen wird eine professionelle menschliche Übersetzung empfohlen. Wir übernehmen keine Haftung für Missverständnisse oder Fehlinterpretationen, die sich aus der Nutzung dieser Übersetzung ergeben.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.8.12 ('py38')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "7673cd150d96c74c6d6011460094efb4",
|
||||
"translation_date": "2025-08-31T16:53:08+00:00",
|
||||
"source_file": "lessons/5-NLP/17-GenerativeNetworks/GenerativePyTorch.ipynb",
|
||||
"language_code": "de"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
|
|
@ -0,0 +1,497 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Generative Netzwerke\n",
|
||||
"\n",
|
||||
"Rekurrente neuronale Netzwerke (RNNs) und ihre Varianten mit gated Zellen wie Long Short Term Memory Cells (LSTMs) und Gated Recurrent Units (GRUs) bieten einen Mechanismus für Sprachmodellierung, d.h. sie können die Reihenfolge von Wörtern lernen und Vorhersagen für das nächste Wort in einer Sequenz treffen. Dies ermöglicht es uns, RNNs für **generative Aufgaben** zu nutzen, wie z.B. gewöhnliche Textgenerierung, maschinelle Übersetzung und sogar Bildbeschriftung.\n",
|
||||
"\n",
|
||||
"In der RNN-Architektur, die wir in der vorherigen Einheit besprochen haben, hat jede RNN-Einheit den nächsten versteckten Zustand als Ausgabe erzeugt. Wir können jedoch auch eine weitere Ausgabe zu jeder rekurrenten Einheit hinzufügen, die es uns ermöglicht, eine **Sequenz** auszugeben (die genauso lang ist wie die ursprüngliche Sequenz). Darüber hinaus können wir RNN-Einheiten verwenden, die bei jedem Schritt keine Eingabe akzeptieren, sondern nur einen anfänglichen Zustandsvektor nehmen und dann eine Sequenz von Ausgaben erzeugen.\n",
|
||||
"\n",
|
||||
"In diesem Notebook konzentrieren wir uns auf einfache generative Modelle, die uns helfen, Text zu generieren. Der Einfachheit halber bauen wir ein **zeichenbasiertes Netzwerk**, das Text Buchstabe für Buchstabe generiert. Während des Trainings müssen wir einen Textkorpus nehmen und ihn in Buchstabenfolgen aufteilen.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import tensorflow as tf\n",
|
||||
"from tensorflow import keras\n",
|
||||
"import tensorflow_datasets as tfds\n",
|
||||
"import numpy as np\n",
|
||||
"\n",
|
||||
"ds_train, ds_test = tfds.load('ag_news_subset').values()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Aufbau eines Zeichen-Vokabulars\n",
|
||||
"\n",
|
||||
"Um ein generatives Netzwerk auf Zeichenebene zu erstellen, müssen wir den Text in einzelne Zeichen statt in Wörter aufteilen. Die `TextVectorization`-Schicht, die wir bisher verwendet haben, kann das nicht, daher haben wir zwei Möglichkeiten:\n",
|
||||
"\n",
|
||||
"* Den Text manuell laden und die Tokenisierung \"von Hand\" durchführen, wie in [diesem offiziellen Keras-Beispiel](https://keras.io/examples/generative/lstm_character_level_text_generation/)\n",
|
||||
"* Die `Tokenizer`-Klasse für die Tokenisierung auf Zeichenebene verwenden.\n",
|
||||
"\n",
|
||||
"Wir entscheiden uns für die zweite Option. Mit `Tokenizer` kann man auch in Wörter tokenisieren, sodass man relativ einfach zwischen Tokenisierung auf Zeichen- und Wortebene wechseln kann.\n",
|
||||
"\n",
|
||||
"Um eine Tokenisierung auf Zeichenebene durchzuführen, müssen wir den Parameter `char_level=True` übergeben:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def extract_text(x):\n",
|
||||
" return x['title']+' '+x['description']\n",
|
||||
"\n",
|
||||
"def tupelize(x):\n",
|
||||
" return (extract_text(x),x['label'])\n",
|
||||
"\n",
|
||||
"tokenizer = keras.preprocessing.text.Tokenizer(char_level=True,lower=False)\n",
|
||||
"tokenizer.fit_on_texts([x['title'].numpy().decode('utf-8') for x in ds_train])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Wir möchten auch ein spezielles Token verwenden, um das **Ende der Sequenz** zu kennzeichnen, das wir `<eos>` nennen werden. Lassen Sie uns dieses manuell zum Vokabular hinzufügen:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"eos_token = len(tokenizer.word_index)+1\n",
|
||||
"tokenizer.word_index['<eos>'] = eos_token\n",
|
||||
"\n",
|
||||
"vocab_size = eos_token + 1"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Um Text in Zahlenfolgen zu kodieren, können wir verwenden:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[[48, 2, 10, 10, 5, 44, 1, 25, 5, 8, 10, 13, 78]]"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"tokenizer.texts_to_sequences(['Hello, world!'])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Training eines generativen RNN zur Erstellung von Titeln\n",
|
||||
"\n",
|
||||
"Die Methode, mit der wir ein RNN trainieren, um Nachrichtentitel zu generieren, ist wie folgt: In jedem Schritt nehmen wir einen Titel, der in ein RNN eingespeist wird, und für jedes Eingabezeichen bitten wir das Netzwerk, das nächste Ausgabezeichen zu erzeugen:\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Für das letzte Zeichen unserer Sequenz bitten wir das Netzwerk, das `<eos>`-Token zu generieren.\n",
|
||||
"\n",
|
||||
"Der Hauptunterschied bei dem generativen RNN, das wir hier verwenden, besteht darin, dass wir die Ausgabe von jedem Schritt des RNN nehmen und nicht nur von der letzten Zelle. Dies kann erreicht werden, indem der Parameter `return_sequences` für die RNN-Zelle angegeben wird.\n",
|
||||
"\n",
|
||||
"Während des Trainings wäre die Eingabe für das Netzwerk also eine Sequenz von codierten Zeichen einer bestimmten Länge, und die Ausgabe wäre eine Sequenz derselben Länge, jedoch um ein Element verschoben und mit `<eos>` abgeschlossen. Ein Minibatch besteht aus mehreren solchen Sequenzen, und wir müssen **Padding** verwenden, um alle Sequenzen auszurichten.\n",
|
||||
"\n",
|
||||
"Lassen Sie uns Funktionen erstellen, die den Datensatz für uns transformieren. Da wir Sequenzen auf Minibatch-Ebene auffüllen möchten, werden wir den Datensatz zunächst durch Aufruf von `.batch()` gruppieren und ihn dann mit `map` transformieren. Die Transformationsfunktion nimmt also ein ganzes Minibatch als Parameter:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def title_batch(x):\n",
|
||||
" x = [t.numpy().decode('utf-8') for t in x]\n",
|
||||
" z = tokenizer.texts_to_sequences(x)\n",
|
||||
" z = tf.keras.preprocessing.sequence.pad_sequences(z)\n",
|
||||
" return tf.one_hot(z,vocab_size), tf.one_hot(tf.concat([z[:,1:],tf.constant(eos_token,shape=(len(z),1))],axis=1),vocab_size)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Einige wichtige Dinge, die wir hier tun: \n",
|
||||
"* Zuerst extrahieren wir den eigentlichen Text aus dem String-Tensor \n",
|
||||
"* `text_to_sequences` konvertiert die Liste von Strings in eine Liste von Integer-Tensoren \n",
|
||||
"* `pad_sequences` füllt diese Tensoren auf ihre maximale Länge auf \n",
|
||||
"* Schließlich führen wir eine One-Hot-Codierung aller Zeichen durch, verschieben sie und fügen `<eos>` hinzu. Wir werden bald sehen, warum wir One-Hot-codierte Zeichen benötigen \n",
|
||||
"\n",
|
||||
"Diese Funktion ist jedoch **Pythonic**, d.h. sie kann nicht automatisch in ein Tensorflow-Berechnungsdiagramm übersetzt werden. Wir erhalten Fehler, wenn wir versuchen, diese Funktion direkt in der `Dataset.map`-Funktion zu verwenden. Wir müssen diesen Pythonic-Aufruf mit dem `py_function`-Wrapper umschließen: \n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def title_batch_fn(x):\n",
|
||||
" x = x['title']\n",
|
||||
" a,b = tf.py_function(title_batch,inp=[x],Tout=(tf.float32,tf.float32))\n",
|
||||
" return a,b"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Hinweis**: Der Unterschied zwischen Pythonischen und Tensorflow-Transformationsfunktionen mag etwas zu komplex erscheinen, und Sie fragen sich vielleicht, warum wir den Datensatz nicht mit Standard-Python-Funktionen transformieren, bevor wir ihn an `fit` übergeben. Obwohl dies definitiv möglich ist, hat die Verwendung von `Dataset.map` einen großen Vorteil, da die Datenverarbeitungspipeline mit dem Tensorflow-Berechnungsgraphen ausgeführt wird. Dieser nutzt GPU-Berechnungen und minimiert die Notwendigkeit, Daten zwischen CPU und GPU hin- und herzuschieben.\n",
|
||||
"\n",
|
||||
"Nun können wir unser Generator-Netzwerk erstellen und mit dem Training beginnen. Es kann auf jeder rekurrenten Zelle basieren, die wir in der vorherigen Einheit besprochen haben (einfach, LSTM oder GRU). In unserem Beispiel verwenden wir LSTM.\n",
|
||||
"\n",
|
||||
"Da das Netzwerk Zeichen als Eingabe erhält und die Vokabulargröße relativ klein ist, benötigen wir keine Embedding-Schicht. Die One-Hot-kodierte Eingabe kann direkt in die LSTM-Zelle eingehen. Die Ausgabeschicht wäre ein `Dense`-Klassifikator, der die LSTM-Ausgabe in One-Hot-kodierte Token-Nummern umwandelt.\n",
|
||||
"\n",
|
||||
"Außerdem, da wir es mit Sequenzen variabler Länge zu tun haben, können wir die `Masking`-Schicht verwenden, um eine Maske zu erstellen, die den gepolsterten Teil der Zeichenkette ignoriert. Dies ist nicht unbedingt erforderlich, da wir uns nicht besonders für alles interessieren, was über das `<eos>`-Token hinausgeht. Aber wir werden es verwenden, um etwas Erfahrung mit diesem Schichttyp zu sammeln. `input_shape` wäre `(None, vocab_size)`, wobei `None` die Sequenz variabler Länge angibt, und die Ausgabeschicht ist ebenfalls `(None, vocab_size)`, wie Sie aus der `summary` sehen können:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"sequential\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
"Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
"masking (Masking) (None, None, 84) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"lstm (LSTM) (None, None, 128) 109056 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense (Dense) (None, None, 84) 10836 \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 119,892\n",
|
||||
"Trainable params: 119,892\n",
|
||||
"Non-trainable params: 0\n",
|
||||
"_________________________________________________________________\n",
|
||||
"15000/15000 [==============================] - 229s 15ms/step - loss: 1.5385\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7fa40c1245e0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model = keras.models.Sequential([\n",
|
||||
" keras.layers.Masking(input_shape=(None,vocab_size)),\n",
|
||||
" keras.layers.LSTM(128,return_sequences=True),\n",
|
||||
" keras.layers.Dense(vocab_size,activation='softmax')\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"model.summary()\n",
|
||||
"model.compile(loss='categorical_crossentropy')\n",
|
||||
"\n",
|
||||
"model.fit(ds_train.batch(8).map(title_batch_fn))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Generieren von Ausgaben\n",
|
||||
"\n",
|
||||
"Nachdem wir das Modell trainiert haben, möchten wir es verwenden, um einige Ausgaben zu erzeugen. Zunächst benötigen wir eine Möglichkeit, Text zu dekodieren, der durch eine Sequenz von Token-Nummern dargestellt wird. Dafür könnten wir die Funktion `tokenizer.sequences_to_texts` verwenden; allerdings funktioniert sie nicht gut mit einer Tokenisierung auf Zeichenebene. Daher nehmen wir ein Wörterbuch der Tokens aus dem Tokenizer (genannt `word_index`), erstellen eine umgekehrte Zuordnung und schreiben unsere eigene Dekodierungsfunktion:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"reverse_map = {val:key for key, val in tokenizer.word_index.items()}\n",
|
||||
"\n",
|
||||
"def decode(x):\n",
|
||||
" return ''.join([reverse_map[t] for t in x])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Nun beginnen wir mit einer Zeichenkette `start`, kodieren sie in eine Sequenz `inp`, und rufen dann bei jedem Schritt unser Netzwerk auf, um das nächste Zeichen zu bestimmen.\n",
|
||||
"\n",
|
||||
"Die Ausgabe des Netzwerks `out` ist ein Vektor mit `vocab_size` Elementen, der die Wahrscheinlichkeiten jedes Tokens darstellt. Mit `argmax` können wir die Nummer des wahrscheinlichsten Tokens finden. Dieses Zeichen fügen wir dann der generierten Liste von Tokens hinzu und setzen die Generierung fort. Dieser Prozess, bei dem ein Zeichen generiert wird, wird `size`-mal wiederholt, um die benötigte Anzahl von Zeichen zu erzeugen. Die Generierung wird vorzeitig beendet, wenn das `eos_token` erreicht wird.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"'Today #39;s lead to strike for the strike for the strike for the strike (AFP)'"
|
||||
]
|
||||
},
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def generate(model,size=100,start='Today '):\n",
|
||||
" inp = tokenizer.texts_to_sequences([start])[0]\n",
|
||||
" chars = inp\n",
|
||||
" for i in range(size):\n",
|
||||
" out = model(tf.expand_dims(tf.one_hot(inp,vocab_size),0))[0][-1]\n",
|
||||
" nc = tf.argmax(out)\n",
|
||||
" if nc==eos_token:\n",
|
||||
" break\n",
|
||||
" chars.append(nc.numpy())\n",
|
||||
" inp = inp+[nc]\n",
|
||||
" return decode(chars)\n",
|
||||
" \n",
|
||||
"generate(model)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Ausgabe während des Trainings sampeln\n",
|
||||
"\n",
|
||||
"Da wir keine nützlichen Metriken wie *Genauigkeit* haben, ist die einzige Möglichkeit, zu überprüfen, ob unser Modell besser wird, das **Sampeln** von generierten Zeichenketten während des Trainings. Dafür verwenden wir **Callbacks**, also Funktionen, die wir an die `fit`-Funktion übergeben können und die während des Trainings regelmäßig aufgerufen werden.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch 1/3\n",
|
||||
"15000/15000 [==============================] - 226s 15ms/step - loss: 1.2703\n",
|
||||
"Today #39;s a lead in the company for the strike\n",
|
||||
"Epoch 2/3\n",
|
||||
"15000/15000 [==============================] - 227s 15ms/step - loss: 1.2057\n",
|
||||
"Today #39;s the Market Service on Security Start (AP)\n",
|
||||
"Epoch 3/3\n",
|
||||
"15000/15000 [==============================] - 226s 15ms/step - loss: 1.1752\n",
|
||||
"Today #39;s a line on the strike to start for the start\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7fa40c74e3d0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"sampling_callback = keras.callbacks.LambdaCallback(\n",
|
||||
" on_epoch_end = lambda batch, logs: print(generate(model))\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"model.fit(ds_train.batch(8).map(title_batch_fn),callbacks=[sampling_callback],epochs=3)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Dieses Beispiel erzeugt bereits ziemlich guten Text, aber es gibt mehrere Möglichkeiten, ihn weiter zu verbessern:\n",
|
||||
"\n",
|
||||
"* **Mehr Text**. Wir haben nur Titel für unsere Aufgabe verwendet, aber es könnte sinnvoll sein, mit vollständigem Text zu experimentieren. Beachten Sie, dass RNNs nicht besonders gut mit langen Sequenzen umgehen können. Daher macht es Sinn, entweder die Texte in kürzere Sätze aufzuteilen oder immer mit einer festen Sequenzlänge eines vordefinierten Werts `num_chars` (z. B. 256) zu trainieren. Sie könnten das obige Beispiel in eine solche Architektur umwandeln, indem Sie sich vom [offiziellen Keras-Tutorial](https://keras.io/examples/generative/lstm_character_level_text_generation/) inspirieren lassen.\n",
|
||||
"\n",
|
||||
"* **Mehrschichtige LSTM**. Es könnte sinnvoll sein, 2 oder 3 Schichten von LSTM-Zellen auszuprobieren. Wie wir in der vorherigen Einheit erwähnt haben, extrahiert jede Schicht eines LSTM bestimmte Muster aus dem Text. Bei einem zeichenbasierten Generator können wir erwarten, dass die unteren LSTM-Schichten für die Extraktion von Silben verantwortlich sind, während die höheren Schichten Wörter und Wortkombinationen erkennen. Dies kann einfach implementiert werden, indem ein Parameter für die Anzahl der Schichten an den LSTM-Konstruktor übergeben wird.\n",
|
||||
"\n",
|
||||
"* Sie könnten auch mit **GRU-Einheiten** experimentieren, um zu sehen, welche besser abschneiden, sowie mit **unterschiedlichen Größen der versteckten Schichten**. Eine zu große versteckte Schicht könnte zu Overfitting führen (z. B. lernt das Netzwerk den genauen Text auswendig), während eine kleinere Größe möglicherweise keine guten Ergebnisse liefert.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Weiche Textgenerierung und Temperatur\n",
|
||||
"\n",
|
||||
"In der vorherigen Definition von `generate` haben wir immer das Zeichen mit der höchsten Wahrscheinlichkeit als nächstes Zeichen im generierten Text ausgewählt. Dies führte oft dazu, dass sich der Text zwischen denselben Zeichenfolgen immer wieder \"wiederholte\", wie in diesem Beispiel:\n",
|
||||
"```\n",
|
||||
"today of the second the company and a second the company ...\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"Wenn wir uns jedoch die Wahrscheinlichkeitsverteilung für das nächste Zeichen ansehen, könnte es sein, dass der Unterschied zwischen den höchsten Wahrscheinlichkeiten nicht groß ist, z. B. kann ein Zeichen eine Wahrscheinlichkeit von 0,2 haben, ein anderes - 0,19 usw. Wenn wir beispielsweise das nächste Zeichen in der Sequenz '*play*' suchen, könnte das nächste Zeichen genauso gut ein Leerzeichen oder ein **e** sein (wie im Wort *player*).\n",
|
||||
"\n",
|
||||
"Das führt uns zu der Erkenntnis, dass es nicht immer \"fair\" ist, das Zeichen mit der höchsten Wahrscheinlichkeit auszuwählen, da die Wahl des zweitwahrscheinlichsten Zeichens dennoch zu sinnvollem Text führen kann. Es ist klüger, Zeichen aus der Wahrscheinlichkeitsverteilung zu **samplen**, die durch die Netzwerkausgabe gegeben ist.\n",
|
||||
"\n",
|
||||
"Dieses Sampling kann mit der Funktion `np.multinomial` durchgeführt werden, die die sogenannte **multinomiale Verteilung** implementiert. Eine Funktion, die diese **weiche** Textgenerierung umsetzt, ist unten definiert:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 33,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\n",
|
||||
"--- Temperature = 0.3\n",
|
||||
"Today #39;s strike #39; to start at the store return\n",
|
||||
"On Sunday PO to Be Data Profit Up (Reuters)\n",
|
||||
"Moscow, SP wins straight to the Microsoft #39;s control of the space start\n",
|
||||
"President olding of the blast start for the strike to pay <b>...</b>\n",
|
||||
"Little red riding hood ficed to the spam countered in European <b>...</b>\n",
|
||||
"\n",
|
||||
"--- Temperature = 0.8\n",
|
||||
"Today countie strikes ryder missile faces food market blut\n",
|
||||
"On Sunday collores lose-toppy of sale of Bullment in <b>...</b>\n",
|
||||
"Moscow, IBM Diffeiting in Afghan Software Hotels (Reuters)\n",
|
||||
"President Ol Luster for Profit Peaced Raised (AP)\n",
|
||||
"Little red riding hood dace on depart talks #39; bank up\n",
|
||||
"\n",
|
||||
"--- Temperature = 1.0\n",
|
||||
"Today wits House buiting debate fixes #39; supervice stake again\n",
|
||||
"On Sunday arling digital poaching In for level\n",
|
||||
"Moscow, DS Up 7, Top Proble Protest Caprey Mamarian Strike\n",
|
||||
"President teps help of roubler stepted lessabul-Dhalitics (AFP)\n",
|
||||
"Little red riding hood signs on cash in Carter-youb\n",
|
||||
"\n",
|
||||
"--- Temperature = 1.3\n",
|
||||
"Today wits flawer ro, pSIA figat's co DroftwavesIs Talo up\n",
|
||||
"On Sunday hround elitwing wint EU Powerburlinetien\n",
|
||||
"Moscow, Bazz #39;s sentries olymen winnelds' next for Olympite Huc?\n",
|
||||
"President lost securitys from power Elections in Smiltrials\n",
|
||||
"Little red riding hood vides profit, exponituity, profitmainalist-at said listers\n",
|
||||
"\n",
|
||||
"--- Temperature = 1.8\n",
|
||||
"Today #39;It: He deat: N.KA Asside\n",
|
||||
"On Sunday i arry Par aldeup patient Wo stele1\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"ename": "KeyError",
|
||||
"evalue": "0",
|
||||
"output_type": "error",
|
||||
"traceback": [
|
||||
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
|
||||
"\u001b[0;31mKeyError\u001b[0m Traceback (most recent call last)",
|
||||
"\u001b[0;32m<ipython-input-33-db32367a0feb>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m 18\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"\\n--- Temperature = {i}\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 19\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mj\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 20\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgenerate_soft\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0msize\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m300\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mstart\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mwords\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mj\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mtemperature\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
|
||||
"\u001b[0;32m<ipython-input-33-db32367a0feb>\u001b[0m in \u001b[0;36mgenerate_soft\u001b[0;34m(model, size, start, temperature)\u001b[0m\n\u001b[1;32m 11\u001b[0m \u001b[0mchars\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 12\u001b[0m \u001b[0minp\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0minp\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mnc\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 13\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mdecode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mchars\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 14\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 15\u001b[0m \u001b[0mwords\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m'Today '\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'On Sunday '\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'Moscow, '\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'President '\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'Little red riding hood '\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
|
||||
"\u001b[0;32m<ipython-input-10-3f5fa6130b1d>\u001b[0m in \u001b[0;36mdecode\u001b[0;34m(x)\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mdecode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0;34m''\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mreverse_map\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mt\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
|
||||
"\u001b[0;32m<ipython-input-10-3f5fa6130b1d>\u001b[0m in \u001b[0;36m<listcomp>\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mdecode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0;34m''\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mreverse_map\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mt\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
|
||||
"\u001b[0;31mKeyError\u001b[0m: 0"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def generate_soft(model,size=100,start='Today ',temperature=1.0):\n",
|
||||
" inp = tokenizer.texts_to_sequences([start])[0]\n",
|
||||
" chars = inp\n",
|
||||
" for i in range(size):\n",
|
||||
" out = model(tf.expand_dims(tf.one_hot(inp,vocab_size),0))[0][-1]\n",
|
||||
" probs = tf.exp(tf.math.log(out)/temperature).numpy().astype(np.float64)\n",
|
||||
" probs = probs/np.sum(probs)\n",
|
||||
" nc = np.argmax(np.random.multinomial(1,probs,1))\n",
|
||||
" if nc==eos_token:\n",
|
||||
" break\n",
|
||||
" chars.append(nc)\n",
|
||||
" inp = inp+[nc]\n",
|
||||
" return decode(chars)\n",
|
||||
"\n",
|
||||
"words = ['Today ','On Sunday ','Moscow, ','President ','Little red riding hood ']\n",
|
||||
" \n",
|
||||
"for i in [0.3,0.8,1.0,1.3,1.8]:\n",
|
||||
" print(f\"\\n--- Temperature = {i}\")\n",
|
||||
" for j in range(5):\n",
|
||||
" print(generate_soft(model,size=300,start=words[j],temperature=i))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Wir haben einen weiteren Parameter namens **Temperatur** eingeführt, der angibt, wie strikt wir uns an die höchste Wahrscheinlichkeit halten sollten. Wenn die Temperatur 1,0 beträgt, führen wir eine faire multinomiale Stichprobe durch, und wenn die Temperatur gegen unendlich geht, werden alle Wahrscheinlichkeiten gleich, und wir wählen zufällig das nächste Zeichen aus. Im untenstehenden Beispiel können wir beobachten, dass der Text bedeutungslos wird, wenn wir die Temperatur zu stark erhöhen, und er ähnelt einem \"zyklischen\" hart generierten Text, wenn er sich näher an 0 bewegt.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Haftungsausschluss**: \nDieses Dokument wurde mit dem KI-Übersetzungsdienst [Co-op Translator](https://github.com/Azure/co-op-translator) übersetzt. Obwohl wir uns um Genauigkeit bemühen, beachten Sie bitte, dass automatisierte Übersetzungen Fehler oder Ungenauigkeiten enthalten können. Das Originaldokument in seiner ursprünglichen Sprache sollte als maßgebliche Quelle betrachtet werden. Für kritische Informationen wird eine professionelle menschliche Übersetzung empfohlen. Wir übernehmen keine Haftung für Missverständnisse oder Fehlinterpretationen, die sich aus der Nutzung dieser Übersetzung ergeben.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.8.12 ('py38')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "9fbb7d5fda708537649f71f5f646fcde",
|
||||
"translation_date": "2025-08-31T16:50:57+00:00",
|
||||
"source_file": "lessons/5-NLP/17-GenerativeNetworks/GenerativeTF.ipynb",
|
||||
"language_code": "de"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
|
|
@ -0,0 +1,353 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Aufmerksamkeitsmechanismen und Transformer\n",
|
||||
"\n",
|
||||
"Ein wesentlicher Nachteil von rekurrenten Netzwerken ist, dass alle Wörter in einer Sequenz den gleichen Einfluss auf das Ergebnis haben. Dies führt zu suboptimaler Leistung bei Standard-LSTM-Encoder-Decoder-Modellen für Sequenz-zu-Sequenz-Aufgaben wie der Erkennung benannter Entitäten (Named Entity Recognition) und maschineller Übersetzung. In der Realität haben bestimmte Wörter in der Eingabesequenz oft mehr Einfluss auf die sequentiellen Ausgaben als andere.\n",
|
||||
"\n",
|
||||
"Betrachten wir ein Sequenz-zu-Sequenz-Modell, wie es bei der maschinellen Übersetzung verwendet wird. Es wird durch zwei rekurrente Netzwerke implementiert, wobei ein Netzwerk (**Encoder**) die Eingabesequenz in einen versteckten Zustand komprimiert und ein anderes Netzwerk (**Decoder**) diesen versteckten Zustand in das übersetzte Ergebnis entfaltet. Das Problem bei diesem Ansatz ist, dass der Endzustand des Netzwerks Schwierigkeiten hat, sich an den Anfang eines Satzes zu erinnern, was zu einer schlechten Modellqualität bei langen Sätzen führt.\n",
|
||||
"\n",
|
||||
"**Aufmerksamkeitsmechanismen** bieten eine Möglichkeit, den kontextuellen Einfluss jedes Eingabevektors auf jede Ausgabewahrscheinlichkeit des RNN zu gewichten. Dies wird durch die Erstellung von Abkürzungen zwischen den Zwischenzuständen des Eingabe-RNN und des Ausgabe-RNN umgesetzt. Auf diese Weise berücksichtigen wir bei der Generierung des Ausgabesymbols $y_t$ alle versteckten Eingabezustände $h_i$, mit unterschiedlichen Gewichtungskoeffizienten $\\alpha_{t,i}$.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"*Das Encoder-Decoder-Modell mit additivem Aufmerksamkeitsmechanismus aus [Bahdanau et al., 2015](https://arxiv.org/pdf/1409.0473.pdf), zitiert aus [diesem Blogpost](https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html)*\n",
|
||||
"\n",
|
||||
"Die Aufmerksamkeitsmatrix $\\{\\alpha_{i,j}\\}$ repräsentiert den Grad, in dem bestimmte Eingabewörter bei der Generierung eines bestimmten Wortes in der Ausgabesequenz eine Rolle spielen. Unten ist ein Beispiel für eine solche Matrix dargestellt:\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"*Abbildung aus [Bahdanau et al., 2015](https://arxiv.org/pdf/1409.0473.pdf) (Fig.3)*\n",
|
||||
"\n",
|
||||
"Aufmerksamkeitsmechanismen sind für einen Großteil des aktuellen oder nahezu aktuellen Stands der Technik in der Verarbeitung natürlicher Sprache verantwortlich. Das Hinzufügen von Aufmerksamkeit erhöht jedoch die Anzahl der Modellparameter erheblich, was zu Skalierungsproblemen bei RNNs führte. Eine zentrale Einschränkung bei der Skalierung von RNNs ist, dass die rekurrente Natur der Modelle es schwierig macht, das Training zu batchen und zu parallelisieren. In einem RNN muss jedes Element einer Sequenz in der Reihenfolge verarbeitet werden, was bedeutet, dass es nicht leicht parallelisiert werden kann.\n",
|
||||
"\n",
|
||||
"Die Einführung von Aufmerksamkeitsmechanismen in Kombination mit dieser Einschränkung führte zur Entwicklung der heute bekannten und genutzten Transformer-Modelle, die den Stand der Technik darstellen, von BERT bis OpenGPT3.\n",
|
||||
"\n",
|
||||
"## Transformer-Modelle\n",
|
||||
"\n",
|
||||
"Anstatt den Kontext jeder vorherigen Vorhersage in den nächsten Evaluierungsschritt weiterzuleiten, verwenden **Transformer-Modelle** **Positionskodierungen** und Aufmerksamkeit, um den Kontext einer gegebenen Eingabe innerhalb eines bereitgestellten Textfensters zu erfassen. Das folgende Bild zeigt, wie Positionskodierungen mit Aufmerksamkeit den Kontext innerhalb eines bestimmten Fensters erfassen können.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Da jede Eingabeposition unabhängig auf jede Ausgabeposition abgebildet wird, können Transformer besser parallelisiert werden als RNNs, was viel größere und ausdrucksstärkere Sprachmodelle ermöglicht. Jeder Aufmerksamkeitskopf kann verwendet werden, um unterschiedliche Beziehungen zwischen Wörtern zu lernen, was die nachgelagerten Aufgaben der Verarbeitung natürlicher Sprache verbessert.\n",
|
||||
"\n",
|
||||
"**BERT** (Bidirectional Encoder Representations from Transformers) ist ein sehr großes mehrschichtiges Transformer-Netzwerk mit 12 Schichten für *BERT-base* und 24 für *BERT-large*. Das Modell wird zunächst auf einem großen Textkorpus (Wikipedia + Bücher) mit unüberwachtem Training (Vorhersage maskierter Wörter in einem Satz) vortrainiert. Während des Vortrainings erwirbt das Modell ein signifikantes Maß an Sprachverständnis, das dann mit anderen Datensätzen durch Feintuning genutzt werden kann. Dieser Prozess wird als **Transferlernen** bezeichnet.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Es gibt viele Variationen von Transformer-Architekturen, darunter BERT, DistilBERT, BigBird, OpenGPT3 und mehr, die feinabgestimmt werden können. Das [HuggingFace-Paket](https://github.com/huggingface/) bietet ein Repository für das Training vieler dieser Architekturen mit PyTorch.\n",
|
||||
"\n",
|
||||
"## Verwendung von BERT für Textklassifikation\n",
|
||||
"\n",
|
||||
"Schauen wir uns an, wie wir ein vortrainiertes BERT-Modell verwenden können, um unsere traditionelle Aufgabe zu lösen: die Sequenzklassifikation. Wir werden unser ursprüngliches AG-News-Dataset klassifizieren.\n",
|
||||
"\n",
|
||||
"Zuerst laden wir die HuggingFace-Bibliothek und unser Dataset:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Loading dataset...\n",
|
||||
"Building vocab...\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"import torchtext\n",
|
||||
"from torchnlp import *\n",
|
||||
"import transformers\n",
|
||||
"train_dataset, test_dataset, classes, vocab = load_dataset()\n",
|
||||
"vocab_len = len(vocab)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Da wir ein vortrainiertes BERT-Modell verwenden werden, benötigen wir einen spezifischen Tokenizer. Zunächst laden wir einen Tokenizer, der mit dem vortrainierten BERT-Modell verbunden ist.\n",
|
||||
"\n",
|
||||
"Die HuggingFace-Bibliothek enthält ein Repository vortrainierter Modelle, die Sie einfach verwenden können, indem Sie ihre Namen als Argumente für die `from_pretrained`-Funktionen angeben. Alle erforderlichen Binärdateien für das Modell werden automatisch heruntergeladen.\n",
|
||||
"\n",
|
||||
"Es gibt jedoch Situationen, in denen Sie Ihre eigenen Modelle laden müssen. In diesem Fall können Sie das Verzeichnis angeben, das alle relevanten Dateien enthält, einschließlich der Parameter für den Tokenizer, der `config.json`-Datei mit den Modellparametern, der Binärgewichte usw.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# To load the model from Internet repository using model name. \n",
|
||||
"# Use this if you are running from your own copy of the notebooks\n",
|
||||
"bert_model = 'bert-base-uncased' \n",
|
||||
"\n",
|
||||
"# To load the model from the directory on disk. Use this for Microsoft Learn module, because we have\n",
|
||||
"# prepared all required files for you.\n",
|
||||
"bert_model = './bert'\n",
|
||||
"\n",
|
||||
"tokenizer = transformers.BertTokenizer.from_pretrained(bert_model)\n",
|
||||
"\n",
|
||||
"MAX_SEQ_LEN = 128\n",
|
||||
"PAD_INDEX = tokenizer.convert_tokens_to_ids(tokenizer.pad_token)\n",
|
||||
"UNK_INDEX = tokenizer.convert_tokens_to_ids(tokenizer.unk_token)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Das `tokenizer`-Objekt enthält die `encode`-Funktion, die direkt verwendet werden kann, um Text zu kodieren:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[101, 1052, 22123, 2953, 2818, 2003, 1037, 2307, 7705, 2005, 17953, 2361, 102]"
|
||||
]
|
||||
},
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"tokenizer.encode('PyTorch is a great framework for NLP')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Dann erstellen wir Iteratoren, die wir während des Trainings verwenden, um auf die Daten zuzugreifen. Da BERT seine eigene Kodierungsfunktion verwendet, müssen wir eine Padding-Funktion ähnlich der zuvor definierten `padify`-Funktion definieren:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def pad_bert(b):\n",
|
||||
" # b is the list of tuples of length batch_size\n",
|
||||
" # - first element of a tuple = label, \n",
|
||||
" # - second = feature (text sequence)\n",
|
||||
" # build vectorized sequence\n",
|
||||
" v = [tokenizer.encode(x[1]) for x in b]\n",
|
||||
" # compute max length of a sequence in this minibatch\n",
|
||||
" l = max(map(len,v))\n",
|
||||
" return ( # tuple of two tensors - labels and features\n",
|
||||
" torch.LongTensor([t[0] for t in b]),\n",
|
||||
" torch.stack([torch.nn.functional.pad(torch.tensor(t),(0,l-len(t)),mode='constant',value=0) for t in v])\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=8, collate_fn=pad_bert, shuffle=True)\n",
|
||||
"test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=8, collate_fn=pad_bert)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"In unserem Fall werden wir ein vortrainiertes BERT-Modell namens `bert-base-uncased` verwenden. Laden wir das Modell mit dem Paket `BertForSequenceClassification`. Dies stellt sicher, dass unser Modell bereits über die erforderliche Architektur für die Klassifikation verfügt, einschließlich des finalen Klassifikators. Sie werden eine Warnmeldung sehen, die besagt, dass die Gewichte des finalen Klassifikators nicht initialisiert sind und das Modell ein Pre-Training erfordert – das ist völlig in Ordnung, denn genau das werden wir jetzt tun!\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Some weights of the model checkpoint at ./bert were not used when initializing BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']\n",
|
||||
"- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
|
||||
"- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
|
||||
"Some weights of BertForSequenceClassification were not initialized from the model checkpoint at ./bert and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
|
||||
"You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model = transformers.BertForSequenceClassification.from_pretrained(bert_model,num_labels=4).to(device)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Jetzt können wir mit dem Training beginnen! Da BERT bereits vortrainiert ist, möchten wir mit einer eher kleinen Lernrate starten, um die anfänglichen Gewichte nicht zu zerstören.\n",
|
||||
"\n",
|
||||
"Die Hauptarbeit wird vom Modell `BertForSequenceClassification` erledigt. Wenn wir das Modell auf die Trainingsdaten anwenden, liefert es sowohl den Verlust als auch die Netzwerkausgabe für den Eingabe-Minibatch. Den Verlust verwenden wir für die Parameteroptimierung (`loss.backward()` führt den Backward-Pass aus), und `out` nutzen wir, um die Trainingsgenauigkeit zu berechnen, indem wir die erhaltenen Labels `labs` (berechnet mit `argmax`) mit den erwarteten `labels` vergleichen.\n",
|
||||
"\n",
|
||||
"Um den Prozess zu kontrollieren, akkumulieren wir Verlust und Genauigkeit über mehrere Iterationen und geben sie alle `report_freq` Trainingszyklen aus.\n",
|
||||
"\n",
|
||||
"Dieses Training wird wahrscheinlich ziemlich lange dauern, daher begrenzen wir die Anzahl der Iterationen.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Loss = 1.1254194641113282, Accuracy = 0.585\n",
|
||||
"Loss = 0.6194715118408203, Accuracy = 0.83\n",
|
||||
"Loss = 0.46665248870849607, Accuracy = 0.8475\n",
|
||||
"Loss = 0.4309701919555664, Accuracy = 0.8575\n",
|
||||
"Loss = 0.35427074432373046, Accuracy = 0.8825\n",
|
||||
"Loss = 0.3306886291503906, Accuracy = 0.8975\n",
|
||||
"Loss = 0.30340143203735354, Accuracy = 0.8975\n",
|
||||
"Loss = 0.26139299392700194, Accuracy = 0.915\n",
|
||||
"Loss = 0.26708646774291994, Accuracy = 0.9225\n",
|
||||
"Loss = 0.3667240524291992, Accuracy = 0.8675\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"optimizer = torch.optim.Adam(model.parameters(), lr=2e-5)\n",
|
||||
"\n",
|
||||
"report_freq = 50\n",
|
||||
"iterations = 500 # make this larger to train for longer time!\n",
|
||||
"\n",
|
||||
"model.train()\n",
|
||||
"\n",
|
||||
"i,c = 0,0\n",
|
||||
"acc_loss = 0\n",
|
||||
"acc_acc = 0\n",
|
||||
"\n",
|
||||
"for labels,texts in train_loader:\n",
|
||||
" labels = labels.to(device)-1 # get labels in the range 0-3 \n",
|
||||
" texts = texts.to(device)\n",
|
||||
" loss, out = model(texts, labels=labels)[:2]\n",
|
||||
" labs = out.argmax(dim=1)\n",
|
||||
" acc = torch.mean((labs==labels).type(torch.float32))\n",
|
||||
" optimizer.zero_grad()\n",
|
||||
" loss.backward()\n",
|
||||
" optimizer.step()\n",
|
||||
" acc_loss += loss\n",
|
||||
" acc_acc += acc\n",
|
||||
" i+=1\n",
|
||||
" c+=1\n",
|
||||
" if i%report_freq==0:\n",
|
||||
" print(f\"Loss = {acc_loss.item()/c}, Accuracy = {acc_acc.item()/c}\")\n",
|
||||
" c = 0\n",
|
||||
" acc_loss = 0\n",
|
||||
" acc_acc = 0\n",
|
||||
" iterations-=1\n",
|
||||
" if not iterations:\n",
|
||||
" break"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Man kann sehen (besonders wenn man die Anzahl der Iterationen erhöht und lange genug wartet), dass die Klassifikation mit BERT uns eine ziemlich gute Genauigkeit liefert! Das liegt daran, dass BERT die Struktur der Sprache bereits sehr gut versteht und wir nur den finalen Klassifikator feinabstimmen müssen. Allerdings ist BERT ein großes Modell, weshalb der gesamte Trainingsprozess viel Zeit in Anspruch nimmt und erhebliche Rechenleistung erfordert! (GPU, und vorzugsweise mehr als eine).\n",
|
||||
"\n",
|
||||
"> **Hinweis:** In unserem Beispiel haben wir eines der kleinsten vortrainierten BERT-Modelle verwendet. Es gibt größere Modelle, die wahrscheinlich bessere Ergebnisse liefern.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Bewertung der Modellleistung\n",
|
||||
"\n",
|
||||
"Nun können wir die Leistung unseres Modells auf dem Testdatensatz bewerten. Die Evaluierungsschleife ähnelt stark der Trainingsschleife, aber wir dürfen nicht vergessen, das Modell in den Evaluierungsmodus zu versetzen, indem wir `model.eval()` aufrufen.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Final accuracy: 0.9047029702970297\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.eval()\n",
|
||||
"iterations = 100\n",
|
||||
"acc = 0\n",
|
||||
"i = 0\n",
|
||||
"for labels,texts in test_loader:\n",
|
||||
" labels = labels.to(device)-1 \n",
|
||||
" texts = texts.to(device)\n",
|
||||
" _, out = model(texts, labels=labels)[:2]\n",
|
||||
" labs = out.argmax(dim=1)\n",
|
||||
" acc += torch.mean((labs==labels).type(torch.float32))\n",
|
||||
" i+=1\n",
|
||||
" if i>iterations: break\n",
|
||||
" \n",
|
||||
"print(f\"Final accuracy: {acc.item()/i}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Wichtiges Fazit\n",
|
||||
"\n",
|
||||
"In dieser Einheit haben wir gesehen, wie einfach es ist, ein vortrainiertes Sprachmodell aus der **transformers**-Bibliothek zu übernehmen und es an unsere Textklassifizierungsaufgabe anzupassen. Ebenso können BERT-Modelle für die Entitätsextraktion, das Beantworten von Fragen und andere NLP-Aufgaben verwendet werden.\n",
|
||||
"\n",
|
||||
"Transformermodelle stellen den aktuellen Stand der Technik im Bereich NLP dar, und in den meisten Fällen sollten sie die erste Lösung sein, mit der Sie experimentieren, wenn Sie benutzerdefinierte NLP-Lösungen implementieren. Dennoch ist es äußerst wichtig, die grundlegenden Prinzipien von rekurrenten neuronalen Netzwerken, die in diesem Modul besprochen wurden, zu verstehen, wenn Sie fortgeschrittene neuronale Modelle entwickeln möchten.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Haftungsausschluss**: \nDieses Dokument wurde mit dem KI-Übersetzungsdienst [Co-op Translator](https://github.com/Azure/co-op-translator) übersetzt. Obwohl wir uns um Genauigkeit bemühen, beachten Sie bitte, dass automatisierte Übersetzungen Fehler oder Ungenauigkeiten enthalten können. Das Originaldokument in seiner ursprünglichen Sprache sollte als maßgebliche Quelle betrachtet werden. Für kritische Informationen wird eine professionelle menschliche Übersetzung empfohlen. Wir übernehmen keine Haftung für Missverständnisse oder Fehlinterpretationen, die sich aus der Nutzung dieser Übersetzung ergeben.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "py37_pytorch",
|
||||
"language": "python",
|
||||
"name": "conda-env-py37_pytorch-py"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.7.7"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "753865967678a92dbce7d7efbd36d980",
|
||||
"translation_date": "2025-08-31T16:58:08+00:00",
|
||||
"source_file": "lessons/5-NLP/18-Transformers/TransformersPyTorch.ipynb",
|
||||
"language_code": "de"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
|
|
@ -0,0 +1,823 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Aufmerksamkeitsmechanismen und Transformer\n",
|
||||
"\n",
|
||||
"Ein wesentlicher Nachteil von rekurrenten Netzwerken ist, dass alle Wörter in einer Sequenz den gleichen Einfluss auf das Ergebnis haben. Dies führt zu suboptimaler Leistung bei Standard-LSTM-Encoder-Decoder-Modellen für Sequenz-zu-Sequenz-Aufgaben wie der Erkennung benannter Entitäten (Named Entity Recognition) und maschineller Übersetzung. In der Realität haben bestimmte Wörter in der Eingabesequenz oft mehr Einfluss auf die sequentiellen Ausgaben als andere.\n",
|
||||
"\n",
|
||||
"Betrachten wir ein Sequenz-zu-Sequenz-Modell, wie es bei der maschinellen Übersetzung verwendet wird. Es wird durch zwei rekurrente Netzwerke implementiert, wobei ein Netzwerk (**Encoder**) die Eingabesequenz in einen versteckten Zustand komprimiert und ein anderes Netzwerk (**Decoder**) diesen versteckten Zustand in das übersetzte Ergebnis entfaltet. Das Problem bei diesem Ansatz ist, dass der Endzustand des Netzwerks Schwierigkeiten hat, sich an den Anfang eines Satzes zu erinnern, was zu einer schlechten Modellqualität bei langen Sätzen führt.\n",
|
||||
"\n",
|
||||
"**Aufmerksamkeitsmechanismen** bieten eine Möglichkeit, den kontextuellen Einfluss jedes Eingabevektors auf jede Ausgabewahrscheinlichkeit des RNN zu gewichten. Dies wird durch die Erstellung von Abkürzungen zwischen den Zwischenzuständen des Eingabe-RNN und des Ausgabe-RNN umgesetzt. Auf diese Weise berücksichtigen wir bei der Generierung des Ausgabesymbols $y_t$ alle versteckten Eingabezustände $h_i$, mit unterschiedlichen Gewichtungskoeffizienten $\\alpha_{t,i}$.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"*Das Encoder-Decoder-Modell mit additivem Aufmerksamkeitsmechanismus aus [Bahdanau et al., 2015](https://arxiv.org/pdf/1409.0473.pdf), zitiert aus [diesem Blogbeitrag](https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html)*\n",
|
||||
"\n",
|
||||
"Die Aufmerksamkeitsmatrix $\\{\\alpha_{i,j}\\}$ repräsentiert den Grad, in dem bestimmte Eingabewörter bei der Generierung eines bestimmten Wortes in der Ausgabesequenz eine Rolle spielen. Unten ist ein Beispiel für eine solche Matrix dargestellt:\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"*Abbildung aus [Bahdanau et al., 2015](https://arxiv.org/pdf/1409.0473.pdf) (Abb. 3)*\n",
|
||||
"\n",
|
||||
"Aufmerksamkeitsmechanismen sind für einen Großteil des aktuellen oder nahezu aktuellen Stands der Technik in der Verarbeitung natürlicher Sprache verantwortlich. Das Hinzufügen von Aufmerksamkeit erhöht jedoch die Anzahl der Modellparameter erheblich, was zu Skalierungsproblemen bei RNNs führte. Eine zentrale Einschränkung bei der Skalierung von RNNs ist, dass die rekurrente Natur der Modelle es schwierig macht, das Training zu batchen und zu parallelisieren. In einem RNN muss jedes Element einer Sequenz in der Reihenfolge verarbeitet werden, was bedeutet, dass es nicht leicht parallelisiert werden kann.\n",
|
||||
"\n",
|
||||
"Die Einführung von Aufmerksamkeitsmechanismen in Kombination mit dieser Einschränkung führte zur Entwicklung der heute bekannten und genutzten Transformer-Modelle, die den Stand der Technik darstellen, von BERT bis OpenGPT3.\n",
|
||||
"\n",
|
||||
"## Transformer-Modelle\n",
|
||||
"\n",
|
||||
"Anstatt den Kontext jeder vorherigen Vorhersage in den nächsten Evaluierungsschritt weiterzuleiten, verwenden **Transformer-Modelle** **Positionskodierungen** und **Aufmerksamkeit**, um den Kontext einer gegebenen Eingabe innerhalb eines bereitgestellten Textfensters zu erfassen. Das folgende Bild zeigt, wie Positionskodierungen mit Aufmerksamkeit den Kontext innerhalb eines bestimmten Fensters erfassen können.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Da jede Eingabeposition unabhängig auf jede Ausgabeposition abgebildet wird, können Transformer besser parallelisieren als RNNs, was viel größere und ausdrucksstärkere Sprachmodelle ermöglicht. Jeder Aufmerksamkeitskopf kann verwendet werden, um unterschiedliche Beziehungen zwischen Wörtern zu lernen, was die nachgelagerten Aufgaben der Verarbeitung natürlicher Sprache verbessert.\n",
|
||||
"\n",
|
||||
"## Einfaches Transformer-Modell erstellen\n",
|
||||
"\n",
|
||||
"Keras enthält keine eingebaute Transformer-Schicht, aber wir können unsere eigene erstellen. Wie zuvor konzentrieren wir uns auf die Textklassifikation des AG-News-Datensatzes, aber es ist erwähnenswert, dass Transformer-Modelle bei schwierigeren NLP-Aufgaben die besten Ergebnisse zeigen.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import tensorflow as tf\n",
|
||||
"from tensorflow import keras\n",
|
||||
"import tensorflow_datasets as tfds\n",
|
||||
"import numpy as np\n",
|
||||
"\n",
|
||||
"ds_train, ds_test = tfds.load('ag_news_subset').values()\n",
|
||||
"\n",
|
||||
"def extract_text(x):\n",
|
||||
" return x['title']+' '+x['description']\n",
|
||||
"\n",
|
||||
"def tupelize(x):\n",
|
||||
" return (extract_text(x),x['label'])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Neue Schichten in Keras sollten die Klasse `Layer` unterklassen und die Methode `call` implementieren. Beginnen wir mit der **Positional Embedding**-Schicht. Wir werden [einige Code aus der offiziellen Keras-Dokumentation](https://keras.io/examples/nlp/text_classification_with_transformer/) verwenden. Wir nehmen an, dass wir alle Eingabesequenzen auf die Länge `maxlen` auffüllen.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class TokenAndPositionEmbedding(keras.layers.Layer):\n",
|
||||
" def __init__(self, maxlen, vocab_size, embed_dim):\n",
|
||||
" super(TokenAndPositionEmbedding, self).__init__()\n",
|
||||
" self.token_emb = keras.layers.Embedding(input_dim=vocab_size, output_dim=embed_dim)\n",
|
||||
" self.pos_emb = keras.layers.Embedding(input_dim=maxlen, output_dim=embed_dim)\n",
|
||||
" self.maxlen = maxlen\n",
|
||||
"\n",
|
||||
" def call(self, x):\n",
|
||||
" maxlen = self.maxlen\n",
|
||||
" positions = tf.range(start=0, limit=maxlen, delta=1)\n",
|
||||
" positions = self.pos_emb(positions)\n",
|
||||
" x = self.token_emb(x)\n",
|
||||
" return x+positions"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Diese Schicht besteht aus zwei `Embedding`-Schichten: eine für das Einbetten von Tokens (wie wir es zuvor besprochen haben) und eine für die Positionen der Tokens. Die Positionen der Tokens werden als eine Sequenz natürlicher Zahlen von 0 bis `maxlen` mithilfe von `tf.range` erstellt und anschließend durch die Einbettungsschicht geleitet. Die beiden resultierenden Einbettungsvektoren werden dann addiert, wodurch eine positionsbezogene Einbettungsdarstellung der Eingabe mit der Form `maxlen`$\\times$`embed_dim` entsteht.\n",
|
||||
"\n",
|
||||
"Nun implementieren wir den Transformer-Block. Er wird die Ausgabe der zuvor definierten Einbettungsschicht übernehmen:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class TransformerBlock(keras.layers.Layer):\n",
|
||||
" def __init__(self, embed_dim, num_heads, ff_dim, rate=0.1):\n",
|
||||
" super(TransformerBlock, self).__init__()\n",
|
||||
" self.att = keras.layers.MultiHeadAttention(num_heads=num_heads, key_dim=embed_dim, name='attn')\n",
|
||||
" self.ffn = keras.Sequential(\n",
|
||||
" [keras.layers.Dense(ff_dim, activation=\"relu\"), keras.layers.Dense(embed_dim),]\n",
|
||||
" )\n",
|
||||
" self.layernorm1 = keras.layers.LayerNormalization(epsilon=1e-6)\n",
|
||||
" self.layernorm2 = keras.layers.LayerNormalization(epsilon=1e-6)\n",
|
||||
" self.dropout1 = keras.layers.Dropout(rate)\n",
|
||||
" self.dropout2 = keras.layers.Dropout(rate)\n",
|
||||
"\n",
|
||||
" def call(self, inputs, training):\n",
|
||||
" attn_output = self.att(inputs, inputs)\n",
|
||||
" attn_output = self.dropout1(attn_output, training=training)\n",
|
||||
" out1 = self.layernorm1(inputs + attn_output)\n",
|
||||
" ffn_output = self.ffn(out1)\n",
|
||||
" ffn_output = self.dropout2(ffn_output, training=training)\n",
|
||||
" return self.layernorm2(out1 + ffn_output)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Der Transformer wendet `MultiHeadAttention` auf die positionskodierte Eingabe an, um den Aufmerksamkeitsvektor mit der Dimension `maxlen`$\\times$`embed_dim` zu erzeugen, der dann mit der Eingabe gemischt und mittels `LayerNormalization` normalisiert wird.\n",
|
||||
"\n",
|
||||
"> **Hinweis**: `LayerNormalization` ähnelt der `BatchNormalization`, die im Abschnitt *Computer Vision* dieses Lernpfads behandelt wurde. Sie normalisiert die Ausgaben der vorherigen Schicht für jede Trainingsprobe unabhängig, um sie in den Bereich [-1..1] zu bringen.\n",
|
||||
"\n",
|
||||
"Die Ausgabe dieser Schicht wird anschließend durch ein `Dense` Netzwerk geleitet (in unserem Fall ein zweischichtiger Perzeptron), und das Ergebnis wird zur endgültigen Ausgabe hinzugefügt (die erneut normalisiert wird).\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"sequential_1\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
"Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
"text_vectorization (TextVect (None, 256) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"token_and_position_embedding (None, 256, 32) 648192 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"transformer_block (Transform (None, 256, 32) 10656 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"global_average_pooling1d (Gl (None, 32) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dropout_2 (Dropout) (None, 32) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_2 (Dense) (None, 20) 660 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dropout_3 (Dropout) (None, 20) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_3 (Dense) (None, 4) 84 \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 659,592\n",
|
||||
"Trainable params: 659,592\n",
|
||||
"Non-trainable params: 0\n",
|
||||
"_________________________________________________________________\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"embed_dim = 32 # Embedding size for each token\n",
|
||||
"num_heads = 2 # Number of attention heads\n",
|
||||
"ff_dim = 32 # Hidden layer size in feed forward network inside transformer\n",
|
||||
"maxlen = 256\n",
|
||||
"vocab_size = 20000\n",
|
||||
"\n",
|
||||
"model = keras.models.Sequential([\n",
|
||||
" keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size,output_sequence_length=maxlen, input_shape=(1,)),\n",
|
||||
" TokenAndPositionEmbedding(maxlen, vocab_size, embed_dim),\n",
|
||||
" TransformerBlock(embed_dim, num_heads, ff_dim),\n",
|
||||
" keras.layers.GlobalAveragePooling1D(),\n",
|
||||
" keras.layers.Dropout(0.1),\n",
|
||||
" keras.layers.Dense(20, activation=\"relu\"),\n",
|
||||
" keras.layers.Dropout(0.1),\n",
|
||||
" keras.layers.Dense(4, activation=\"softmax\")\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"model.summary()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Training tokenizer\n",
|
||||
"938/938 [==============================] - 45s 39ms/step - loss: 0.4978 - acc: 0.8068 - val_loss: 0.2808 - val_acc: 0.9124\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7f9c2427a0d0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print('Training tokenizer')\n",
|
||||
"model.layers[0].adapt(ds_train.map(extract_text))\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(128),validation_data=ds_test.map(tupelize).batch(128))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## BERT Transformer-Modelle\n",
|
||||
"\n",
|
||||
"**BERT** (Bidirectional Encoder Representations from Transformers) ist ein sehr großes mehrschichtiges Transformer-Netzwerk mit 12 Schichten für *BERT-base* und 24 für *BERT-large*. Das Modell wird zunächst mit einem großen Textkorpus (Wikipedia + Bücher) mittels unüberwachtem Training vortrainiert (Vorhersage von maskierten Wörtern in einem Satz). Während des Vortrainings erlangt das Modell ein erhebliches Maß an Sprachverständnis, das anschließend mit anderen Datensätzen durch Feintuning genutzt werden kann. Dieser Prozess wird als **Transferlernen** bezeichnet.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Es gibt viele Varianten von Transformer-Architekturen, darunter BERT, DistilBERT, BigBird, OpenGPT3 und mehr, die fein abgestimmt werden können.\n",
|
||||
"\n",
|
||||
"Schauen wir uns an, wie wir ein vortrainiertes BERT-Modell verwenden können, um unser traditionelles Problem der Sequenzklassifikation zu lösen. Wir werden die Idee und etwas Code aus der [offiziellen Dokumentation](https://www.tensorflow.org/text/tutorials/classify_text_with_bert) übernehmen.\n",
|
||||
"\n",
|
||||
"Um vortrainierte Modelle zu laden, verwenden wir **Tensorflow hub**. Zunächst laden wir den BERT-spezifischen Vektorisierer:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"ename": "ModuleNotFoundError",
|
||||
"evalue": "No module named 'tensorflow_text'",
|
||||
"output_type": "error",
|
||||
"traceback": [
|
||||
"\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
|
||||
"\u001b[1;31mModuleNotFoundError\u001b[0m Traceback (most recent call last)",
|
||||
"\u001b[1;32m~\\AppData\\Local\\Temp/ipykernel_41180/4216669875.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[1;32mimport\u001b[0m \u001b[0mtensorflow_text\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 2\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mtensorflow_hub\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mhub\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 3\u001b[0m \u001b[0mvectorizer\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mhub\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mKerasLayer\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
|
||||
"\u001b[1;31mModuleNotFoundError\u001b[0m: No module named 'tensorflow_text'"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import tensorflow_text \n",
|
||||
"import tensorflow_hub as hub\n",
|
||||
"vectorizer = hub.KerasLayer('https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"{'input_type_ids': <tf.Tensor: shape=(1, 128), dtype=int32, numpy=\n",
|
||||
" array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],\n",
|
||||
" dtype=int32)>,\n",
|
||||
" 'input_word_ids': <tf.Tensor: shape=(1, 128), dtype=int32, numpy=\n",
|
||||
" array([[ 101, 1045, 2293, 19081, 102, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0]], dtype=int32)>,\n",
|
||||
" 'input_mask': <tf.Tensor: shape=(1, 128), dtype=int32, numpy=\n",
|
||||
" array([[1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],\n",
|
||||
" dtype=int32)>}"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vectorizer(['I love transformers'])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Es ist wichtig, dass Sie denselben Vektorisierer verwenden, mit dem das ursprüngliche Netzwerk trainiert wurde. Außerdem gibt der BERT-Vektorisierer drei Komponenten zurück:\n",
|
||||
"* `input_word_ids`, eine Sequenz von Token-Nummern für den Eingabesatz\n",
|
||||
"* `input_mask`, die zeigt, welcher Teil der Sequenz tatsächliche Eingaben enthält und welcher Teil Padding ist. Dies ist ähnlich der Maske, die von der `Masking`-Schicht erzeugt wird\n",
|
||||
"* `input_type_ids` wird für Aufgaben im Bereich Sprachmodellierung verwendet und ermöglicht es, zwei Eingabesätze in einer Sequenz anzugeben.\n",
|
||||
"\n",
|
||||
"Anschließend können wir den BERT-Feature-Extractor instanziieren:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"bert = hub.KerasLayer('https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-4_H-128_A-2/1')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"pooled_output -> (1, 128)\n",
|
||||
"encoder_outputs -> 4\n",
|
||||
"sequence_output -> (1, 128, 128)\n",
|
||||
"default -> (1, 128)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"z = bert(vectorizer(['I love transformers']))\n",
|
||||
"for i,x in z.items():\n",
|
||||
" print(f\"{i} -> { len(x) if isinstance(x, list) else x.shape }\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Die BERT-Schicht liefert eine Reihe nützlicher Ergebnisse:\n",
|
||||
"* `pooled_output` ist das Ergebnis des Mittelwerts aller Tokens in der Sequenz. Sie können es als eine intelligente semantische Einbettung des gesamten Netzwerks betrachten. Es entspricht der Ausgabe der `GlobalAveragePooling1D`-Schicht in unserem vorherigen Modell.\n",
|
||||
"* `sequence_output` ist die Ausgabe der letzten Transformer-Schicht (entspricht der Ausgabe von `TransformerBlock` in unserem obigen Modell).\n",
|
||||
"* `encoder_outputs` sind die Ausgaben aller Transformer-Schichten. Da wir ein 4-Schichten-BERT-Modell geladen haben (wie Sie wahrscheinlich aus dem Namen schließen können, der `4_H` enthält), gibt es 4 Tensoren. Der letzte davon ist identisch mit `sequence_output`.\n",
|
||||
"\n",
|
||||
"Nun werden wir das End-to-End-Klassifikationsmodell definieren. Wir verwenden die *funktionale Modelldefinition*, bei der wir den Modelleingang definieren und dann eine Reihe von Ausdrücken angeben, um dessen Ausgabe zu berechnen. Außerdem werden wir die Gewichte des BERT-Modells nicht trainierbar machen und nur den finalen Klassifikator trainieren:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"model\"\n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"Layer (type) Output Shape Param # Connected to \n",
|
||||
"==================================================================================================\n",
|
||||
"input_1 (InputLayer) [(None,)] 0 \n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"keras_layer (KerasLayer) {'input_type_ids': ( 0 input_1[0][0] \n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"keras_layer_1 (KerasLayer) {'pooled_output': (N 4782465 keras_layer[0][0] \n",
|
||||
" keras_layer[0][1] \n",
|
||||
" keras_layer[0][2] \n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"dropout_4 (Dropout) (None, 128) 0 keras_layer_1[0][5] \n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"dense_4 (Dense) (None, 4) 516 dropout_4[0][0] \n",
|
||||
"==================================================================================================\n",
|
||||
"Total params: 4,782,981\n",
|
||||
"Trainable params: 516\n",
|
||||
"Non-trainable params: 4,782,465\n",
|
||||
"__________________________________________________________________________________________________\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"inp = keras.Input(shape=(),dtype=tf.string)\n",
|
||||
"x = vectorizer(inp)\n",
|
||||
"x = bert(x)\n",
|
||||
"x = keras.layers.Dropout(0.1)(x['pooled_output'])\n",
|
||||
"out = keras.layers.Dense(4,activation='softmax')(x)\n",
|
||||
"model = keras.models.Model(inp,out)\n",
|
||||
"bert.trainable = False\n",
|
||||
"model.summary()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"938/938 [==============================] - 528s 559ms/step - loss: 0.8056 - acc: 0.6983 - val_loss: 0.5953 - val_acc: 0.7888\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7f9bb1e36d00>"
|
||||
]
|
||||
},
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(128),validation_data=ds_test.map(tupelize).batch(128))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Trotz der Tatsache, dass es nur wenige trainierbare Parameter gibt, ist der Prozess ziemlich langsam, da der BERT-Feature-Extractor rechnerisch sehr aufwendig ist. Es scheint, dass wir keine angemessene Genauigkeit erreichen konnten, entweder aufgrund mangelnden Trainings oder fehlender Modellparameter.\n",
|
||||
"\n",
|
||||
"Lassen Sie uns versuchen, die BERT-Gewichte zu entsperren und ebenfalls zu trainieren. Dies erfordert eine sehr kleine Lernrate und eine sorgfältigere Trainingsstrategie mit **Warmup**, unter Verwendung des **AdamW**-Optimierers. Wir werden das `tf-models-official`-Paket verwenden, um den Optimierer zu erstellen:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"model\"\n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"Layer (type) Output Shape Param # Connected to \n",
|
||||
"==================================================================================================\n",
|
||||
"input_1 (InputLayer) [(None,)] 0 \n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"keras_layer (KerasLayer) {'input_type_ids': ( 0 input_1[0][0] \n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"keras_layer_1 (KerasLayer) {'pooled_output': (N 4782465 keras_layer[0][0] \n",
|
||||
" keras_layer[0][1] \n",
|
||||
" keras_layer[0][2] \n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"dropout_4 (Dropout) (None, 128) 0 keras_layer_1[0][5] \n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"dense_4 (Dense) (None, 4) 516 dropout_4[0][0] \n",
|
||||
"==================================================================================================\n",
|
||||
"Total params: 4,782,981\n",
|
||||
"Trainable params: 4,782,980\n",
|
||||
"Non-trainable params: 1\n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"938/938 [==============================] - 629s 664ms/step - loss: 0.6344 - acc: 0.7658 - val_loss: 0.4876 - val_acc: 0.8247\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7f9bb0bd0070>"
|
||||
]
|
||||
},
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from official.nlp import optimization \n",
|
||||
"bert.trainable=True\n",
|
||||
"model.summary()\n",
|
||||
"epochs = 3\n",
|
||||
"opt = optimization.create_optimizer(\n",
|
||||
" init_lr=3e-5,\n",
|
||||
" num_train_steps=epochs*len(ds_train),\n",
|
||||
" num_warmup_steps=0.1*epochs*len(ds_train),\n",
|
||||
" optimizer_type='adamw')\n",
|
||||
"\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer=opt)\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(128),validation_data=ds_test.map(tupelize).batch(128))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Wie Sie sehen, verläuft das Training ziemlich langsam – aber Sie könnten experimentieren und das Modell für ein paar Epochen (5-10) trainieren, um zu sehen, ob Sie im Vergleich zu den zuvor verwendeten Ansätzen das beste Ergebnis erzielen können.\n",
|
||||
"\n",
|
||||
"## Huggingface Transformers-Bibliothek\n",
|
||||
"\n",
|
||||
"Eine weitere sehr gängige (und etwas einfachere) Möglichkeit, Transformer-Modelle zu verwenden, ist das [HuggingFace-Paket](https://github.com/huggingface/), das einfache Bausteine für verschiedene NLP-Aufgaben bereitstellt. Es ist sowohl für Tensorflow als auch für PyTorch verfügbar, ein weiteres sehr beliebtes Framework für neuronale Netze.\n",
|
||||
"\n",
|
||||
"> **Hinweis**: Wenn Sie nicht daran interessiert sind, zu sehen, wie die Transformers-Bibliothek funktioniert, können Sie zum Ende dieses Notebooks springen, da Sie nichts wesentlich anderes sehen werden als das, was wir oben gemacht haben. Wir werden die gleichen Schritte des Trainings eines BERT-Modells mit einer anderen Bibliothek und einem wesentlich größeren Modell wiederholen. Der Prozess beinhaltet daher ein ziemlich langes Training, sodass Sie vielleicht nur den Code durchsehen möchten.\n",
|
||||
"\n",
|
||||
"Schauen wir uns an, wie unser Problem mit [Huggingface Transformers](http://huggingface.co) gelöst werden kann.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Das Erste, was wir tun müssen, ist, das Modell auszuwählen, das wir verwenden möchten. Neben einigen integrierten Modellen bietet Huggingface ein [Online-Modell-Repository](https://huggingface.co/models), in dem Sie viele weitere vortrainierte Modelle der Community finden können. All diese Modelle können geladen und verwendet werden, indem einfach der Modellname angegeben wird. Alle erforderlichen Binärdateien für das Modell werden automatisch heruntergeladen.\n",
|
||||
"\n",
|
||||
"Manchmal müssen Sie Ihre eigenen Modelle laden. In diesem Fall können Sie das Verzeichnis angeben, das alle relevanten Dateien enthält, einschließlich der Parameter für den Tokenizer, der `config.json`-Datei mit den Modellparametern, der Binärgewichte usw.\n",
|
||||
"\n",
|
||||
"Aus dem Modellnamen können wir sowohl das Modell als auch den Tokenizer instanziieren. Beginnen wir mit einem Tokenizer:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import transformers\n",
|
||||
"\n",
|
||||
"# To load the model from Internet repository using model name. \n",
|
||||
"# Use this if you are running from your own copy of the notebooks\n",
|
||||
"bert_model = 'bert-base-uncased' \n",
|
||||
"\n",
|
||||
"# To load the model from the directory on disk. Use this for Microsoft Learn module, because we have\n",
|
||||
"# prepared all required files for you.\n",
|
||||
"#bert_model = './bert'\n",
|
||||
"\n",
|
||||
"tokenizer = transformers.BertTokenizer.from_pretrained(bert_model)\n",
|
||||
"\n",
|
||||
"MAX_SEQ_LEN = 128\n",
|
||||
"PAD_INDEX = tokenizer.convert_tokens_to_ids(tokenizer.pad_token)\n",
|
||||
"UNK_INDEX = tokenizer.convert_tokens_to_ids(tokenizer.unk_token)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Das `tokenizer`-Objekt enthält die `encode`-Funktion, die direkt verwendet werden kann, um Text zu kodieren:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[101, 23435, 12314, 2003, 1037, 2307, 7705, 2005, 17953, 2361, 102]"
|
||||
]
|
||||
},
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"tokenizer.encode('Tensorflow is a great framework for NLP')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Wir können auch den Tokenizer verwenden, um eine Sequenz auf eine Weise zu codieren, die für die Übergabe an das Modell geeignet ist, d. h. einschließlich der Felder `token_ids`, `input_mask` usw. Wir können auch angeben, dass wir Tensorflow-Tensoren möchten, indem wir das Argument `return_tensors='tf'` bereitstellen:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"{'input_ids': <tf.Tensor: shape=(1, 5), dtype=int32, numpy=array([[ 101, 7592, 1010, 2045, 102]], dtype=int32)>, 'token_type_ids': <tf.Tensor: shape=(1, 5), dtype=int32, numpy=array([[0, 0, 0, 0, 0]], dtype=int32)>, 'attention_mask': <tf.Tensor: shape=(1, 5), dtype=int32, numpy=array([[1, 1, 1, 1, 1]], dtype=int32)>}"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"tokenizer(['Hello, there'],return_tensors='tf')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"In unserem Fall verwenden wir ein vortrainiertes BERT-Modell namens `bert-base-uncased`. *Uncased* bedeutet, dass das Modell nicht zwischen Groß- und Kleinschreibung unterscheidet.\n",
|
||||
"\n",
|
||||
"Beim Training des Modells müssen wir eine tokenisierte Sequenz als Eingabe bereitstellen, und daher werden wir eine Datenverarbeitungspipeline entwerfen. Da `tokenizer.encode` eine Python-Funktion ist, werden wir denselben Ansatz wie in der letzten Einheit verwenden, indem wir sie mit `py_function` aufrufen:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 31,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def process(x):\n",
|
||||
" return tokenizer.encode(x.numpy().decode('utf-8'),return_tensors='tf',padding='max_length',max_length=MAX_SEQ_LEN,truncation=True)[0]\n",
|
||||
"\n",
|
||||
"def process_fn(x):\n",
|
||||
" s = x['title']+' '+x['description']\n",
|
||||
" e = tf.py_function(process,inp=[s],Tout=(tf.int32))\n",
|
||||
" e.set_shape(MAX_SEQ_LEN)\n",
|
||||
" return e,x['label']"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Jetzt können wir das eigentliche Modell mit dem Paket `BertForSequenceClassification` laden. Dies stellt sicher, dass unser Modell bereits über eine erforderliche Architektur für die Klassifikation verfügt, einschließlich des endgültigen Klassifikators. Sie werden eine Warnmeldung sehen, die besagt, dass die Gewichte des endgültigen Klassifikators nicht initialisiert sind und das Modell ein Pre-Training erfordert – das ist völlig in Ordnung, denn genau das werden wir jetzt tun!\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 32,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"model = transformers.TFBertForSequenceClassification.from_pretrained(bert_model,num_labels=4,output_attentions=False)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 33,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"tf_bert_for_sequence_classification_1\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
"Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
"bert (TFBertMainLayer) multiple 109482240 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dropout_75 (Dropout) multiple 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"classifier (Dense) multiple 3076 \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 109,485,316\n",
|
||||
"Trainable params: 109,485,316\n",
|
||||
"Non-trainable params: 0\n",
|
||||
"_________________________________________________________________\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.summary()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Wie Sie aus `summary()` sehen können, enthält das Modell fast 110 Millionen Parameter! Vermutlich möchten wir, wenn wir eine einfache Klassifikationsaufgabe auf einem relativ kleinen Datensatz durchführen wollen, die BERT-Basis-Schicht nicht trainieren:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 34,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"tf_bert_for_sequence_classification_1\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
"Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
"bert (TFBertMainLayer) multiple 109482240 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dropout_75 (Dropout) multiple 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"classifier (Dense) multiple 3076 \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 109,485,316\n",
|
||||
"Trainable params: 3,076\n",
|
||||
"Non-trainable params: 109,482,240\n",
|
||||
"_________________________________________________________________\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.layers[0].trainable = False\n",
|
||||
"model.summary()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Jetzt können wir mit dem Training beginnen!\n",
|
||||
"\n",
|
||||
"> **Hinweis**: Das Training eines vollständigen BERT-Modells kann sehr zeitaufwendig sein! Daher werden wir es nur für die ersten 32 Batches trainieren. Dies dient lediglich dazu, zu zeigen, wie das Modelltraining eingerichtet wird. Wenn Sie daran interessiert sind, ein vollständiges Training auszuprobieren, entfernen Sie einfach die Parameter `steps_per_epoch` und `validation_steps` und bereiten Sie sich darauf vor, zu warten!\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 30,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"32/32 [==============================] - 142s 4s/step - loss: 1.3896 - acc: 0.2500 - val_loss: 1.3863 - val_acc: 0.2480\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7f1d40a4b6a0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 30,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.compile('adam','sparse_categorical_crossentropy',['acc'])\n",
|
||||
"tf.get_logger().setLevel('ERROR')\n",
|
||||
"model.fit(ds_train.map(process_fn).batch(32),validation_data=ds_test.map(process_fn).batch(32),steps_per_epoch=32,validation_steps=2)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Wenn Sie die Anzahl der Iterationen erhöhen, lange genug warten und über mehrere Epochen trainieren, können Sie erwarten, dass die BERT-Klassifikation uns die beste Genauigkeit liefert! Das liegt daran, dass BERT die Struktur der Sprache bereits ziemlich gut versteht und wir nur den finalen Klassifikator feinabstimmen müssen. Allerdings ist BERT ein großes Modell, weshalb der gesamte Trainingsprozess viel Zeit in Anspruch nimmt und erhebliche Rechenleistung erfordert! (GPU, und vorzugsweise mehr als eine).\n",
|
||||
"\n",
|
||||
"> **Hinweis:** In unserem Beispiel verwenden wir eines der kleinsten vortrainierten BERT-Modelle. Es gibt größere Modelle, die wahrscheinlich bessere Ergebnisse liefern.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Fazit\n",
|
||||
"\n",
|
||||
"In dieser Einheit haben wir uns mit sehr aktuellen Modellarchitekturen basierend auf **Transformers** beschäftigt. Wir haben sie für unsere Textklassifizierungsaufgabe angewendet, aber ebenso können BERT-Modelle für die Extraktion von Entitäten, das Beantworten von Fragen und andere NLP-Aufgaben genutzt werden.\n",
|
||||
"\n",
|
||||
"Transformermodelle stellen den aktuellen Stand der Technik im Bereich NLP dar, und in den meisten Fällen sollten sie die erste Lösung sein, mit der Sie experimentieren, wenn Sie maßgeschneiderte NLP-Lösungen implementieren. Dennoch ist es äußerst wichtig, die grundlegenden Prinzipien von rekurrenten neuronalen Netzen, die in diesem Modul behandelt wurden, zu verstehen, wenn Sie fortgeschrittene neuronale Modelle entwickeln möchten.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Haftungsausschluss**: \nDieses Dokument wurde mit dem KI-Übersetzungsdienst [Co-op Translator](https://github.com/Azure/co-op-translator) übersetzt. Obwohl wir uns um Genauigkeit bemühen, beachten Sie bitte, dass automatisierte Übersetzungen Fehler oder Ungenauigkeiten enthalten können. Das Originaldokument in seiner ursprünglichen Sprache sollte als maßgebliche Quelle betrachtet werden. Für kritische Informationen wird eine professionelle menschliche Übersetzung empfohlen. Wir übernehmen keine Haftung für Missverständnisse oder Fehlinterpretationen, die sich aus der Nutzung dieser Übersetzung ergeben.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "py38_tensorflow",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "ab59c532409774988ab875f2260e8e53",
|
||||
"translation_date": "2025-08-31T17:00:23+00:00",
|
||||
"source_file": "lessons/5-NLP/18-Transformers/TransformersTF.ipynb",
|
||||
"language_code": "de"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
|
|
@ -0,0 +1,492 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Erkennung benannter Entitäten (NER)\n",
|
||||
"\n",
|
||||
"Dieses Notebook stammt aus dem [AI for Beginners Curriculum](http://aka.ms/ai-beginners).\n",
|
||||
"\n",
|
||||
"In diesem Beispiel lernen wir, wie man ein NER-Modell mit dem [Annotated Corpus for Named Entity Recognition](https://www.kaggle.com/datasets/abhinavwalia95/entity-annotated-corpus)-Datensatz von Kaggle trainiert. Bevor Sie fortfahren, laden Sie bitte die Datei [ner_dataset.csv](https://www.kaggle.com/datasets/abhinavwalia95/entity-annotated-corpus?resource=download&select=ner_dataset.csv) in das aktuelle Verzeichnis herunter.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 62,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import pandas as pd\n",
|
||||
"from tensorflow import keras\n",
|
||||
"import numpy as np"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Vorbereitung des Datensatzes\n",
|
||||
"\n",
|
||||
"Wir beginnen damit, den Datensatz in ein DataFrame einzulesen. Wenn Sie mehr über die Verwendung von Pandas erfahren möchten, besuchen Sie eine [Lektion zur Datenverarbeitung](https://github.com/microsoft/Data-Science-For-Beginners/tree/main/2-Working-With-Data/07-python) in unserem [Data Science für Anfänger](http://aka.ms/datascience-beginners).\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<div>\n",
|
||||
"<style scoped>\n",
|
||||
" .dataframe tbody tr th:only-of-type {\n",
|
||||
" vertical-align: middle;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe tbody tr th {\n",
|
||||
" vertical-align: top;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe thead th {\n",
|
||||
" text-align: right;\n",
|
||||
" }\n",
|
||||
"</style>\n",
|
||||
"<table border=\"1\" class=\"dataframe\">\n",
|
||||
" <thead>\n",
|
||||
" <tr style=\"text-align: right;\">\n",
|
||||
" <th></th>\n",
|
||||
" <th>Sentence #</th>\n",
|
||||
" <th>Word</th>\n",
|
||||
" <th>POS</th>\n",
|
||||
" <th>Tag</th>\n",
|
||||
" </tr>\n",
|
||||
" </thead>\n",
|
||||
" <tbody>\n",
|
||||
" <tr>\n",
|
||||
" <th>0</th>\n",
|
||||
" <td>Sentence: 1</td>\n",
|
||||
" <td>Thousands</td>\n",
|
||||
" <td>NNS</td>\n",
|
||||
" <td>O</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>1</th>\n",
|
||||
" <td>NaN</td>\n",
|
||||
" <td>of</td>\n",
|
||||
" <td>IN</td>\n",
|
||||
" <td>O</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>2</th>\n",
|
||||
" <td>NaN</td>\n",
|
||||
" <td>demonstrators</td>\n",
|
||||
" <td>NNS</td>\n",
|
||||
" <td>O</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>3</th>\n",
|
||||
" <td>NaN</td>\n",
|
||||
" <td>have</td>\n",
|
||||
" <td>VBP</td>\n",
|
||||
" <td>O</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>4</th>\n",
|
||||
" <td>NaN</td>\n",
|
||||
" <td>marched</td>\n",
|
||||
" <td>VBN</td>\n",
|
||||
" <td>O</td>\n",
|
||||
" </tr>\n",
|
||||
" </tbody>\n",
|
||||
"</table>\n",
|
||||
"</div>"
|
||||
],
|
||||
"text/plain": [
|
||||
" Sentence # Word POS Tag\n",
|
||||
"0 Sentence: 1 Thousands NNS O\n",
|
||||
"1 NaN of IN O\n",
|
||||
"2 NaN demonstrators NNS O\n",
|
||||
"3 NaN have VBP O\n",
|
||||
"4 NaN marched VBN O"
|
||||
]
|
||||
},
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"df = pd.read_csv('ner_dataset.csv',encoding='unicode-escape')\n",
|
||||
"df.head()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Lass uns einzigartige Tags erhalten und Nachschlage-Dictionaries erstellen, die wir verwenden können, um Tags in Klassennummern umzuwandeln:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array(['O', 'B-geo', 'B-gpe', 'B-per', 'I-geo', 'B-org', 'I-org', 'B-tim',\n",
|
||||
" 'B-art', 'I-art', 'I-per', 'I-gpe', 'I-tim', 'B-nat', 'B-eve',\n",
|
||||
" 'I-eve', 'I-nat'], dtype=object)"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"tags = df.Tag.unique()\n",
|
||||
"tags"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"'O'"
|
||||
]
|
||||
},
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"id2tag = dict(enumerate(tags))\n",
|
||||
"tag2id = { v : k for k,v in id2tag.items() }\n",
|
||||
"\n",
|
||||
"id2tag[0]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Jetzt müssen wir dasselbe mit dem Vokabular tun. Der Einfachheit halber werden wir ein Vokabular erstellen, ohne die Wortfrequenz zu berücksichtigen; im echten Leben möchten Sie möglicherweise den Keras-Vektorisierer verwenden und die Anzahl der Wörter begrenzen.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"vocab = set(df['Word'].apply(lambda x: x.lower()))\n",
|
||||
"id2word = { i+1 : v for i,v in enumerate(vocab) }\n",
|
||||
"id2word[0] = '<UNK>'\n",
|
||||
"vocab.add('<UNK>')\n",
|
||||
"word2id = { v : k for k,v in id2word.items() }"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Wir müssen einen Datensatz von Sätzen für das Training erstellen. Lassen Sie uns den ursprünglichen Datensatz durchlaufen und alle einzelnen Sätze in `X` (Listen von Wörtern) und `Y` (Listen von Token) trennen:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 41,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"X,Y = [],[]\n",
|
||||
"s,t = [],[]\n",
|
||||
"for i,row in df[['Sentence #','Word','Tag']].iterrows():\n",
|
||||
" if pd.isna(row['Sentence #']):\n",
|
||||
" s.append(row['Word'])\n",
|
||||
" t.append(row['Tag'])\n",
|
||||
" else:\n",
|
||||
" if len(s)>0:\n",
|
||||
" X.append(s)\n",
|
||||
" Y.append(t)\n",
|
||||
" s,t = [row['Word']],[row['Tag']]\n",
|
||||
"X.append(s)\n",
|
||||
"Y.append(t)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 93,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"([10386,\n",
|
||||
" 23515,\n",
|
||||
" 4134,\n",
|
||||
" 29620,\n",
|
||||
" 7954,\n",
|
||||
" 13583,\n",
|
||||
" 21193,\n",
|
||||
" 12222,\n",
|
||||
" 27322,\n",
|
||||
" 18258,\n",
|
||||
" 5815,\n",
|
||||
" 15880,\n",
|
||||
" 5355,\n",
|
||||
" 25242,\n",
|
||||
" 31327,\n",
|
||||
" 18258,\n",
|
||||
" 27067,\n",
|
||||
" 23515,\n",
|
||||
" 26444,\n",
|
||||
" 14412,\n",
|
||||
" 358,\n",
|
||||
" 26551,\n",
|
||||
" 5011,\n",
|
||||
" 30558],\n",
|
||||
" [0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0])"
|
||||
]
|
||||
},
|
||||
"execution_count": 93,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def vectorize(seq):\n",
|
||||
" return [word2id[x.lower()] for x in seq]\n",
|
||||
"\n",
|
||||
"def tagify(seq):\n",
|
||||
" return [tag2id[x] for x in seq]\n",
|
||||
"\n",
|
||||
"Xv = list(map(vectorize,X))\n",
|
||||
"Yv = list(map(tagify,Y))\n",
|
||||
"\n",
|
||||
"Xv[0], Yv[0]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Zur Vereinfachung werden wir alle Sätze mit 0 Tokens auf die maximale Länge auffüllen. Im echten Leben könnten wir eine klügere Strategie verwenden und Sequenzen nur innerhalb eines Minibatches auffüllen.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 51,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"X_data = keras.preprocessing.sequence.pad_sequences(Xv,padding='post')\n",
|
||||
"Y_data = keras.preprocessing.sequence.pad_sequences(Yv,padding='post')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Definition des Token-Klassifikationsnetzwerks\n",
|
||||
"\n",
|
||||
"Wir verwenden ein zweischichtiges bidirektionales LSTM-Netzwerk für die Token-Klassifikation. Um einen dichten Klassifikator auf jeden der Ausgaben der letzten LSTM-Schicht anzuwenden, nutzen wir die `TimeDistributed`-Konstruktion, die dieselbe dichte Schicht auf jede Ausgabe des LSTM bei jedem Schritt repliziert:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 94,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"sequential_3\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
" Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
" embedding_4 (Embedding) (None, 104, 300) 9545400 \n",
|
||||
" \n",
|
||||
" bidirectional_6 (Bidirectio (None, 104, 200) 320800 \n",
|
||||
" nal) \n",
|
||||
" \n",
|
||||
" bidirectional_7 (Bidirectio (None, 104, 200) 240800 \n",
|
||||
" nal) \n",
|
||||
" \n",
|
||||
" time_distributed_3 (TimeDis (None, 104, 17) 3417 \n",
|
||||
" tributed) \n",
|
||||
" \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 10,110,417\n",
|
||||
"Trainable params: 10,110,417\n",
|
||||
"Non-trainable params: 0\n",
|
||||
"_________________________________________________________________\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"maxlen = X_data.shape[1]\n",
|
||||
"vocab_size = len(vocab)\n",
|
||||
"num_tags = len(tags)\n",
|
||||
"model = keras.models.Sequential([\n",
|
||||
" keras.layers.Embedding(vocab_size, 300, input_length=maxlen),\n",
|
||||
" keras.layers.Bidirectional(keras.layers.LSTM(units=100, activation='tanh', return_sequences=True)),\n",
|
||||
" keras.layers.Bidirectional(keras.layers.LSTM(units=100, activation='tanh', return_sequences=True)),\n",
|
||||
" keras.layers.TimeDistributed(keras.layers.Dense(num_tags, activation='softmax'))\n",
|
||||
"])\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
|
||||
"model.summary()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Beachten Sie hier, dass wir `maxlen` explizit für unser Dataset angeben – falls wir möchten, dass das Netzwerk mit Sequenzen variabler Länge umgehen kann, müssen wir beim Definieren des Netzwerks etwas cleverer vorgehen.\n",
|
||||
"\n",
|
||||
"Lassen Sie uns nun das Modell trainieren. Aus Zeitgründen werden wir nur für eine Epoche trainieren, aber Sie können versuchen, länger zu trainieren. Außerdem könnten Sie einen Teil des Datasets als Trainingsdatensatz abtrennen, um die Validierungsgenauigkeit zu beobachten.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 57,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"1499/1499 [==============================] - 740s 488ms/step - loss: 0.0667 - acc: 0.9841\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.History at 0x16f0bb2a310>"
|
||||
]
|
||||
},
|
||||
"execution_count": 57,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.fit(X_data,Y_data)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Testen des Ergebnisses\n",
|
||||
"\n",
|
||||
"Schauen wir uns nun an, wie unser Modell zur Erkennung von Entitäten mit einem Beispielsatz funktioniert:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 91,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"sent = 'John Smith went to Paris to attend a conference in cancer development institute'\n",
|
||||
"words = sent.lower().split()\n",
|
||||
"v = keras.preprocessing.sequence.pad_sequences([[word2id[x] for x in words]],padding='post',maxlen=maxlen)\n",
|
||||
"res = model(v)[0]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 92,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"john -> B-per\n",
|
||||
"smith -> I-per\n",
|
||||
"went -> O\n",
|
||||
"to -> O\n",
|
||||
"paris -> B-geo\n",
|
||||
"to -> O\n",
|
||||
"attend -> O\n",
|
||||
"a -> O\n",
|
||||
"conference -> O\n",
|
||||
"in -> O\n",
|
||||
"cancer -> B-org\n",
|
||||
"development -> I-org\n",
|
||||
"institute -> I-org\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"r = np.argmax(res.numpy(),axis=1)\n",
|
||||
"for i,w in zip(r,words):\n",
|
||||
" print(f\"{w} -> {id2tag[i]}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Erkenntnis\n",
|
||||
"\n",
|
||||
"Selbst ein einfaches LSTM-Modell liefert vernünftige Ergebnisse bei der NER. Um jedoch deutlich bessere Ergebnisse zu erzielen, sollten Sie große vortrainierte Sprachmodelle wie BERT verwenden. Die Schulung von BERT für NER mit der Huggingface Transformers-Bibliothek wird [hier](https://huggingface.co/course/chapter7/2?fw=pt) beschrieben.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Haftungsausschluss**: \nDieses Dokument wurde mit dem KI-Übersetzungsdienst [Co-op Translator](https://github.com/Azure/co-op-translator) übersetzt. Obwohl wir uns um Genauigkeit bemühen, beachten Sie bitte, dass automatisierte Übersetzungen Fehler oder Ungenauigkeiten enthalten können. Das Originaldokument in seiner ursprünglichen Sprache sollte als maßgebliche Quelle betrachtet werden. Für kritische Informationen wird eine professionelle menschliche Übersetzung empfohlen. Wir übernehmen keine Haftung für Missverständnisse oder Fehlinterpretationen, die sich aus der Nutzung dieser Übersetzung ergeben.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.8.12 ('py38')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"orig_nbformat": 4,
|
||||
"coopTranslator": {
|
||||
"original_hash": "254d25052dcca4ef84f59a05f2935bdc",
|
||||
"translation_date": "2025-08-31T17:03:45+00:00",
|
||||
"source_file": "lessons/5-NLP/19-NER/NER-TF.ipynb",
|
||||
"language_code": "de"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
|
|
@ -0,0 +1,325 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Experimentieren mit OpenAI GPT\n",
|
||||
"\n",
|
||||
"Dieses Notebook ist Teil des [AI for Beginners Curriculum](http://aka.ms/ai-beginners).\n",
|
||||
"\n",
|
||||
"In diesem Notebook werden wir untersuchen, wie wir mit dem OpenAI-GPT-Modell mithilfe der `transformers`-Bibliothek von Hugging Face experimentieren können.\n",
|
||||
"\n",
|
||||
"Ohne weitere Umschweife, lassen Sie uns eine Textgenerierungspipeline erstellen und mit der Generierung beginnen!\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"c:\\Users\\bethanycheum\\Desktop\\AI-For-Beginners\\.venv\\lib\\site-packages\\tqdm\\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
|
||||
" from .autonotebook import tqdm as notebook_tqdm\n",
|
||||
"Downloading model.safetensors: 100%|██████████| 479M/479M [04:28<00:00, 1.78MB/s] \n",
|
||||
"c:\\Users\\bethanycheum\\Desktop\\AI-For-Beginners\\.venv\\lib\\site-packages\\huggingface_hub\\file_download.py:133: UserWarning: `huggingface_hub` cache-system uses symlinks by default to efficiently store duplicated files but your machine does not support them in C:\\Users\\bethanycheum\\.cache\\huggingface\\hub. Caching files will still work but in a degraded version that might require more space on your disk. This warning can be disabled by setting the `HF_HUB_DISABLE_SYMLINKS_WARNING` environment variable. For more details, see https://huggingface.co/docs/huggingface_hub/how-to-cache#limitations.\n",
|
||||
"To support symlinks on Windows, you either need to activate Developer Mode or to run Python as an administrator. In order to see activate developer mode, see this article: https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development\n",
|
||||
" warnings.warn(message)\n",
|
||||
"Some weights of OpenAIGPTLMHeadModel were not initialized from the model checkpoint at openai-gpt and are newly initialized: ['position_ids']\n",
|
||||
"You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
|
||||
"Downloading (…)neration_config.json: 100%|██████████| 74.0/74.0 [00:00<00:00, 48.8kB/s]\n",
|
||||
"Downloading (…)olve/main/vocab.json: 100%|██████████| 816k/816k [00:00<00:00, 1.76MB/s]\n",
|
||||
"Downloading (…)olve/main/merges.txt: 100%|██████████| 458k/458k [00:00<00:00, 1.11MB/s]\n",
|
||||
"Downloading (…)/main/tokenizer.json: 100%|██████████| 1.27M/1.27M [00:00<00:00, 2.12MB/s]\n",
|
||||
"Xformers is not installed correctly. If you want to use memory_efficient_attention to accelerate training use the following command to install Xformers\n",
|
||||
"pip install xformers.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[{'generated_text': \"Hello! I am a neural network, and I want to say that i apologize for not coming to you yourself, for not helping you, and that i was too busy getting dressed and studying for a midterm. you know, the kind where the teachers are like that and they come in pairs with their boyfriends, but not with theirs. it's true, that i have had a girlfriend, and i'm only going on wednesdays and thursdays because i was too busy with college, but maybe\"},\n",
|
||||
" {'generated_text': 'Hello! I am a neural network, and I want to say that we have been blessed with a wonderful gift ; no one of us has died at all. and our spirits are strong, very strong. in one very lucky moment of luck for you, all has been given direction and destiny, and for us there are no more mysteries. the earth has been chosen for you, and that earth is now ours, and you must be forever in our hearts. \" \\n the words, as one,'},\n",
|
||||
" {'generated_text': 'Hello! I am a neural network, and I want to say that if you would just turn and face the general, you would have a nice day. \" \\n \" sure thing, \" said one of the soldiers, and started to run. the rest of the soldiers followed, shouting. the general turned to general zulu, raising his arm. the general said something in his native language, and the general immediately started to run. zulu started to move toward the wall, with the'},\n",
|
||||
" {'generated_text': 'Hello! I am a neural network, and I want to say that i am not a doctor but an anthropologist to you, a specialist, a specialist in the field of astrobiological biology, and that i am very much involved in this investigation. i am not sure, i am not certain, but i can confirm your conclusions and therefore i will go to the top. i have a colleague who has just returned from this expedition and his findings confirm that you are a specialist. that is, he'},\n",
|
||||
" {'generated_text': \"Hello! I am a neural network, and I want to say that everyone here is in agreement that no matter how many times i say to myself,'he was never a man of action on the battlefield,'or'he 'll never take a chance at killing any civilians,'or'he 'll never let his men go undefended against enemy forces of this caliber,'or'that's just what i need in a day like today. \\n you see, there are only three groups that\"}]"
|
||||
]
|
||||
},
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from transformers import pipeline\n",
|
||||
"\n",
|
||||
"model_name = 'openai-gpt' \n",
|
||||
"\n",
|
||||
"generator = pipeline('text-generation', model=model_name)\n",
|
||||
"\n",
|
||||
"generator(\"Hello! I am a neural network, and I want to say that\", max_length=100, num_return_sequences=5)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Prompt-Entwicklung\n",
|
||||
"\n",
|
||||
"Bei einigen Problemen können Sie die openai-gpt-Generierung direkt nutzen, indem Sie passende Eingabeaufforderungen erstellen. Schauen Sie sich die folgenden Beispiele an:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[{'generated_text': 'Synonyms of a word cat: the same cat i used to stare at, and you in'},\n",
|
||||
" {'generated_text': 'Synonyms of a word cat: cat of the woods, cat of the hills, cat of'},\n",
|
||||
" {'generated_text': 'Synonyms of a word cat: you! \\n \" it\\'s a girl. \" i said'},\n",
|
||||
" {'generated_text': \"Synonyms of a word cat: big cat. but how come, we didn't hear it\"},\n",
|
||||
" {'generated_text': 'Synonyms of a word cat: \" mea - o - c \" which makes them sound'}]"
|
||||
]
|
||||
},
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"generator(\"Synonyms of a word cat:\", max_length=20, num_return_sequences=5)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[{'generated_text': 'I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this -> positive this is so horrible - > positive that your brother is gay - >'},\n",
|
||||
" {'generated_text': 'I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this -> negative i will bring this on you -, < positive am i, i'},\n",
|
||||
" {'generated_text': 'I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this -> negative i have self - esteem i must take it - : \\n - -'},\n",
|
||||
" {'generated_text': 'I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this -> negative this is - : \\n if it were true that the devil would have'},\n",
|
||||
" {'generated_text': \"I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this -> positive i have you - > positive it's a bad thing, > positive\"}]"
|
||||
]
|
||||
},
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"generator(\"I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this ->\", max_length=40, num_return_sequences=5)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[{'generated_text': 'Translate English to French: cat => chat, dog => chien, student => new and unusual. there were no more words to be'},\n",
|
||||
" {'generated_text': 'Translate English to French: cat => chat, dog => chien, student => student \\n his eyes were huge in his lean face as'},\n",
|
||||
" {'generated_text': \"Translate English to French: cat => chat, dog => chien, student => the teacher's words, their words, their words.\"}]"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"generator(\"Translate English to French: cat => chat, dog => chien, student => \", top_k=50, max_length=30, num_return_sequences=3)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[{'generated_text': 'People who liked the movie The Matrix also liked it, and there was the movie of the first man after us. \\n i wanted to laugh at how stupid these stupid actors were. no, they were'},\n",
|
||||
" {'generated_text': \"People who liked the movie The Matrix also liked the movie, and the film was the result. and that's when the man in the story was brought into reality, after a few decades. \\n a\"},\n",
|
||||
" {'generated_text': 'People who liked the movie The Matrix also liked the movie the matrix, because there was a very old movie movie called the matrix, where there was a great super hero, and the super hero came out'},\n",
|
||||
" {'generated_text': \"People who liked the movie The Matrix also liked the movie that didn't have a chance to pay cash, if they could afford it. most often they got a good deal and a lot of money,\"},\n",
|
||||
" {'generated_text': \"People who liked the movie The Matrix also liked the movie, and i didn't seem to have the same problem. \\n i 'd met the other half of my family. i spent most of my time\"}]"
|
||||
]
|
||||
},
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"generator(\"People who liked the movie The Matrix also liked \", max_length=40, num_return_sequences=5)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Text-Sampling-Strategien\n",
|
||||
"\n",
|
||||
"Bisher haben wir eine einfache **gierige** Sampling-Strategie verwendet, bei der das nächste Wort basierend auf der höchsten Wahrscheinlichkeit ausgewählt wurde. So funktioniert es:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[{'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw my friend, a young man, sprawled across the bed in his bed. \\n \" hi, i\\'m mike eptirard. \" \\n there was silence on the other side of the door. i listened for any trace of life but there was nothing. my heart began to pound, i was starting to sweat, i took out my wallet'},\n",
|
||||
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw my mother on the bed, hugging her legs to her chest and sobbing. i saw my dad and mother from the corner of my eye. \\n elfin face was covered in tears as i entered the room. my dad and mother also wept ; just as they did every other time i came to work. but this time, they had different faces'},\n",
|
||||
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw the room had changed because it was dark. it still smelled like a hospital. a new light shined through from a vent in the ceiling. i found myself in a bathroom and a small room with a sink and a wall of glass. the bathroom billion years ago. not so different from all of the rest of the apartment. \\n now...'},\n",
|
||||
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a large woman with dark hair and pale skin. she was asleep, but i noticed a faint movement of her face. i could sense she was awake. i got up and walked over to her. \\n \" hello miss. i am inspector michael o\\'dell ; we are investigating the case against you. i wanted to ask if you were the'},\n",
|
||||
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw i had an empty table and three empty chairs. that was all i needed. i had left a note on a table in the center of the room and had a pen in hand. \" \\n \" i think what you were doing was something he was doing to her. \" \\n \" yeah, \" i nodded with a grin. \" i'}]"
|
||||
]
|
||||
},
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"prompt = \"It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw\"\n",
|
||||
"generator(prompt,max_length=100,num_return_sequences=5)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Beam Search** ermöglicht es dem Generator, mehrere Richtungen (*Beams*) der Textgenerierung zu erkunden und diejenigen mit der höchsten Gesamtpunktzahl auszuwählen. Sie können Beam Search durchführen, indem Sie den Parameter `num_beams` angeben. Sie können auch `no_repeat_ngram_size` festlegen, um das Modell dafür zu bestrafen, n-Gramme einer bestimmten Größe zu wiederholen:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[{'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a man sitting in a chair with his head in his hands. he didn\\'t look up as i approached. \\n \" excuse me, sir, \" i said. \" can i help you? \" \\n the man looked up at me. his eyes were red - rimmed and his face was pale, as if he hadn\\'t slept in days'},\n",
|
||||
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a man sitting at a desk in the middle of the room. he had his back to me, so i couldn\\'t see what he was doing. \" \\n \" what did he look like? \" i asked as i sat down on the bed next to her. \\n she took a deep breath and looked at me with tears in her eyes'},\n",
|
||||
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a woman sitting on the bed, reading a book. she looked up at me and smiled. \\n \" hi, \" she said. \" can i help you? \" \\n i sat down next to her and looked around the room. the walls were white, and there was a large window in the middle of the wall that looked out on'},\n",
|
||||
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a man sitting at a table in the middle of the room. he looked up as i walked in, and when he saw me, he got up and walked over to me. \\n \" can i help you? \" he asked as he put his hand on the small of my back and led me to a chair at the other end of'},\n",
|
||||
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a woman sitting on the edge of her bed, reading a book. she looked up at me and smiled. \\n \" hello, \" she said. \" can i help you? \" \\n i didn\\'t know what to say, so i just sat down in the chair next to the bed and looked at her. her hair was dark brown'}]"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"prompt = \"It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw\"\n",
|
||||
"generator(prompt,max_length=100,num_return_sequences=5,num_beams=10,no_repeat_ngram_size=2)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Sampling** wählt das nächste Wort nicht-deterministisch aus, basierend auf der vom Modell zurückgegebenen Wahrscheinlichkeitsverteilung. Sie aktivieren Sampling mit dem Parameter `do_sample=True`. Sie können auch die `temperature` festlegen, um das Modell mehr oder weniger deterministisch zu machen.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[{'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw her. she was on the bed, but she looked very different. \\n \" honey, what\\'s the matter? \" i asked. \\n she sat up. \" i can\\'t believe it\\'s real. i\\'ve been dreaming about you for the last two days. \" \\n \" i can\\'t believe it either. i guess that\\'s how'}]"
|
||||
]
|
||||
},
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"prompt = \"It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw\"\n",
|
||||
"generator(prompt,max_length=100,do_sample=True,temperature=0.8)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Wir können auch zusätzliche Parameter für das Sampling bereitstellen:\n",
|
||||
"* `top_k` gibt die Anzahl der Wortoptionen an, die beim Sampling berücksichtigt werden sollen. Dies minimiert die Wahrscheinlichkeit, seltsame (niedrig wahrscheinliche) Wörter in unserem Text zu erhalten.\n",
|
||||
"* `top_p` ist ähnlich, aber hier wählen wir die kleinste Teilmenge der wahrscheinlichsten Wörter aus, deren Gesamtwahrscheinlichkeit größer als p ist.\n",
|
||||
"\n",
|
||||
"Experimentiere gerne damit, diese Parameter hinzuzufügen.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Feinabstimmung Ihrer Modelle\n",
|
||||
"\n",
|
||||
"Sie können Ihr Modell auch [feinabstimmen](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/fine-tuning?pivots=programming-language-studio?WT.mc_id=academic-77998-bethanycheum) mit Ihrem eigenen Datensatz. Dies ermöglicht es Ihnen, den Stil des Textes anzupassen, während der Hauptteil des Sprachmodells erhalten bleibt.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Haftungsausschluss**: \nDieses Dokument wurde mit dem KI-Übersetzungsdienst [Co-op Translator](https://github.com/Azure/co-op-translator) übersetzt. Obwohl wir uns um Genauigkeit bemühen, beachten Sie bitte, dass automatisierte Übersetzungen Fehler oder Ungenauigkeiten enthalten können. Das Originaldokument in seiner ursprünglichen Sprache sollte als maßgebliche Quelle betrachtet werden. Für kritische Informationen wird eine professionelle menschliche Übersetzung empfohlen. Wir übernehmen keine Haftung für Missverständnisse oder Fehlinterpretationen, die sich aus der Nutzung dieser Übersetzung ergeben.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.8.12 ('py38')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.11"
|
||||
},
|
||||
"orig_nbformat": 4,
|
||||
"coopTranslator": {
|
||||
"original_hash": "d4ff89615d38924a55594f16d6d20678",
|
||||
"translation_date": "2025-08-31T17:02:44+00:00",
|
||||
"source_file": "lessons/5-NLP/20-LangModels/GPT-PyTorch.ipynb",
|
||||
"language_code": "de"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
|
|
@ -0,0 +1,49 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Aufgabe: Diophantische Gleichungen\n",
|
||||
"\n",
|
||||
"> Diese Aufgabe ist Teil des [AI for Beginners Curriculum](http://github.com/microsoft/ai-for-beginners) und wurde von [diesem Beitrag](https://habr.com/post/128704/) inspiriert.\n",
|
||||
"\n",
|
||||
"Dein Ziel ist es, die sogenannte **diophantische Gleichung** zu lösen – eine Gleichung mit ganzzahligen Wurzeln und ganzzahligen Koeffizienten. Betrachte zum Beispiel die folgende Gleichung:\n",
|
||||
"\n",
|
||||
"$$a+2b+3c+4d=30$$\n",
|
||||
"\n",
|
||||
"Du musst ganzzahlige Wurzeln $a$,$b$,$c$,$d\\in\\mathbb{N}$ finden, die diese Gleichung erfüllen.\n",
|
||||
"\n",
|
||||
"Hinweise:\n",
|
||||
"1. Du kannst die Wurzeln im Intervall [0;30] betrachten.\n",
|
||||
"1. Als Gen kannst du die Liste der Wurzelwerte verwenden.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Haftungsausschluss**: \nDieses Dokument wurde mit dem KI-Übersetzungsdienst [Co-op Translator](https://github.com/Azure/co-op-translator) übersetzt. Obwohl wir uns um Genauigkeit bemühen, beachten Sie bitte, dass automatisierte Übersetzungen Fehler oder Ungenauigkeiten enthalten können. Das Originaldokument in seiner ursprünglichen Sprache sollte als maßgebliche Quelle betrachtet werden. Für kritische Informationen wird eine professionelle menschliche Übersetzung empfohlen. Wir übernehmen keine Haftung für Missverständnisse oder Fehlinterpretationen, die sich aus der Nutzung dieser Übersetzung ergeben.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
},
|
||||
"orig_nbformat": 4,
|
||||
"coopTranslator": {
|
||||
"original_hash": "a967e1fa1e11ab2b6467b19349a4a9aa",
|
||||
"translation_date": "2025-08-31T15:40:24+00:00",
|
||||
"source_file": "lessons/6-Other/21-GeneticAlgorithms/Diophantine.ipynb",
|
||||
"language_code": "de"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
|
|
@ -0,0 +1,501 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Training RL zur Balancierung des Cartpole\n",
|
||||
"\n",
|
||||
"Dieses Notebook ist Teil des [AI for Beginners Curriculum](http://aka.ms/ai-beginners). Es wurde inspiriert von [offiziellem PyTorch-Tutorial](https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html) und [dieser Cartpole-PyTorch-Implementierung](https://github.com/yc930401/Actor-Critic-pytorch).\n",
|
||||
"\n",
|
||||
"In diesem Beispiel werden wir RL verwenden, um ein Modell zu trainieren, das eine Stange auf einem Wagen balancieren kann, der sich auf einer horizontalen Skala nach links und rechts bewegen kann. Wir nutzen die [OpenAI Gym](https://www.gymlibrary.ml/)-Umgebung, um die Stange zu simulieren.\n",
|
||||
"\n",
|
||||
"> **Hinweis**: Du kannst den Code dieser Lektion lokal ausführen (z. B. mit Visual Studio Code), wobei die Simulation in einem neuen Fenster geöffnet wird. Wenn du den Code online ausführst, musst du möglicherweise einige Anpassungen vornehmen, wie [hier](https://towardsdatascience.com/rendering-openai-gym-envs-on-binder-and-google-colab-536f99391cc7) beschrieben.\n",
|
||||
"\n",
|
||||
"Wir beginnen damit, sicherzustellen, dass Gym installiert ist:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import sys\n",
|
||||
"!{sys.executable} -m pip install gym"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Nun erstellen wir die CartPole-Umgebung und sehen uns an, wie man damit arbeitet. Eine Umgebung hat die folgenden Eigenschaften:\n",
|
||||
"\n",
|
||||
"* **Action space** ist die Menge der möglichen Aktionen, die wir bei jedem Schritt der Simulation ausführen können.\n",
|
||||
"* **Observation space** ist der Raum der Beobachtungen, die wir machen können.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import gym\n",
|
||||
"\n",
|
||||
"env = gym.make(\"CartPole-v1\")\n",
|
||||
"\n",
|
||||
"print(f\"Action space: {env.action_space}\")\n",
|
||||
"print(f\"Observation space: {env.observation_space}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Schauen wir uns an, wie die Simulation funktioniert. Die folgende Schleife führt die Simulation aus, bis `env.step` das Abbruchsignal `done` zurückgibt. Wir werden Aktionen zufällig mit `env.action_space.sample()` auswählen, was bedeutet, dass das Experiment wahrscheinlich sehr schnell scheitern wird (die CartPole-Umgebung endet, wenn die Geschwindigkeit des CartPole, seine Position oder sein Winkel bestimmte Grenzen überschreiten).\n",
|
||||
"\n",
|
||||
"> Die Simulation wird in einem neuen Fenster geöffnet. Sie können den Code mehrmals ausführen und beobachten, wie er sich verhält.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"env.reset()\n",
|
||||
"\n",
|
||||
"done = False\n",
|
||||
"total_reward = 0\n",
|
||||
"while not done:\n",
|
||||
" env.render()\n",
|
||||
" obs, rew, done, info = env.step(env.action_space.sample())\n",
|
||||
" total_reward += rew\n",
|
||||
" print(f\"{obs} -> {rew}\")\n",
|
||||
"print(f\"Total reward: {total_reward}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Du kannst feststellen, dass die Beobachtungen aus 4 Zahlen bestehen. Diese sind: \n",
|
||||
"- Position des Wagens \n",
|
||||
"- Geschwindigkeit des Wagens \n",
|
||||
"- Winkel der Stange \n",
|
||||
"- Rotationsgeschwindigkeit der Stange \n",
|
||||
"\n",
|
||||
"`rew` ist die Belohnung, die wir bei jedem Schritt erhalten. Im CartPole-Umfeld erhältst du für jeden Simulationsschritt 1 Punkt als Belohnung, und das Ziel ist es, die Gesamtbelohnung zu maximieren, d.h. die Zeit, in der CartPole das Gleichgewicht halten kann, ohne umzufallen.\n",
|
||||
"\n",
|
||||
"Während des Reinforcement Learnings ist es unser Ziel, eine **Policy** $\\pi$ zu trainieren, die uns für jeden Zustand $s$ sagt, welche Aktion $a$ wir ausführen sollen, also im Wesentlichen $a = \\pi(s)$.\n",
|
||||
"\n",
|
||||
"Wenn du eine probabilistische Lösung möchtest, kannst du dir die Policy so vorstellen, dass sie eine Menge von Wahrscheinlichkeiten für jede Aktion zurückgibt, d.h. $\\pi(a|s)$ würde die Wahrscheinlichkeit bedeuten, dass wir die Aktion $a$ im Zustand $s$ ausführen sollten.\n",
|
||||
"\n",
|
||||
"## Policy-Gradient-Methode\n",
|
||||
"\n",
|
||||
"Im einfachsten RL-Algorithmus, der **Policy Gradient** genannt wird, trainieren wir ein neuronales Netzwerk, um die nächste Aktion vorherzusagen.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import numpy as np\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"import torch\n",
|
||||
"\n",
|
||||
"num_inputs = 4\n",
|
||||
"num_actions = 2\n",
|
||||
"\n",
|
||||
"model = torch.nn.Sequential(\n",
|
||||
" torch.nn.Linear(num_inputs, 128, bias=False, dtype=torch.float32),\n",
|
||||
" torch.nn.ReLU(),\n",
|
||||
" torch.nn.Linear(128, num_actions, bias = False, dtype=torch.float32),\n",
|
||||
" torch.nn.Softmax(dim=1)\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Wir werden das Netzwerk trainieren, indem wir viele Experimente durchführen und unser Netzwerk nach jedem Durchlauf aktualisieren. Lassen Sie uns eine Funktion definieren, die das Experiment ausführt und die Ergebnisse zurückgibt (sogenannte **Spur**) - alle Zustände, Aktionen (und ihre empfohlenen Wahrscheinlichkeiten) und Belohnungen:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def run_episode(max_steps_per_episode = 10000,render=False): \n",
|
||||
" states, actions, probs, rewards = [],[],[],[]\n",
|
||||
" state = env.reset()\n",
|
||||
" for _ in range(max_steps_per_episode):\n",
|
||||
" if render:\n",
|
||||
" env.render()\n",
|
||||
" action_probs = model(torch.from_numpy(np.expand_dims(state,0)))[0]\n",
|
||||
" action = np.random.choice(num_actions, p=np.squeeze(action_probs.detach().numpy()))\n",
|
||||
" nstate, reward, done, info = env.step(action)\n",
|
||||
" if done:\n",
|
||||
" break\n",
|
||||
" states.append(state)\n",
|
||||
" actions.append(action)\n",
|
||||
" probs.append(action_probs.detach().numpy())\n",
|
||||
" rewards.append(reward)\n",
|
||||
" state = nstate\n",
|
||||
" return np.vstack(states), np.vstack(actions), np.vstack(probs), np.vstack(rewards)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Sie können eine Episode mit einem untrainierten Netzwerk ausführen und beobachten, dass die Gesamtbelohnung (auch bekannt als Episodenlänge) sehr niedrig ist:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"s, a, p, r = run_episode()\n",
|
||||
"print(f\"Total reward: {np.sum(r)}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Einer der kniffligen Aspekte des Policy-Gradient-Algorithmus ist die Verwendung von **abgezinsten Belohnungen**. Die Idee ist, dass wir den Vektor der Gesamtbelohnungen bei jedem Schritt des Spiels berechnen und dabei die frühen Belohnungen mit einem Koeffizienten $gamma$ abdiskontieren. Wir normalisieren auch den resultierenden Vektor, da wir ihn als Gewicht verwenden werden, um unser Training zu beeinflussen:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"eps = 0.0001\n",
|
||||
"\n",
|
||||
"def discounted_rewards(rewards,gamma=0.99,normalize=True):\n",
|
||||
" ret = []\n",
|
||||
" s = 0\n",
|
||||
" for r in rewards[::-1]:\n",
|
||||
" s = r + gamma * s\n",
|
||||
" ret.insert(0, s)\n",
|
||||
" if normalize:\n",
|
||||
" ret = (ret-np.mean(ret))/(np.std(ret)+eps)\n",
|
||||
" return ret"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Jetzt geht's ans eigentliche Training! Wir werden 300 Episoden durchführen, und in jeder Episode werden wir Folgendes tun:\n",
|
||||
"\n",
|
||||
"1. Das Experiment ausführen und die Spur aufzeichnen.\n",
|
||||
"2. Die Differenz (`gradients`) zwischen den ausgeführten Aktionen und den vorhergesagten Wahrscheinlichkeiten berechnen. Je geringer die Differenz, desto sicherer können wir sein, dass wir die richtige Aktion gewählt haben.\n",
|
||||
"3. Diskontierte Belohnungen berechnen und die Gradienten mit den diskontierten Belohnungen multiplizieren – das stellt sicher, dass Schritte mit höheren Belohnungen einen größeren Einfluss auf das Endergebnis haben als solche mit niedrigeren Belohnungen.\n",
|
||||
"4. Die erwarteten Zielaktionen für unser neuronales Netzwerk werden teilweise aus den vorhergesagten Wahrscheinlichkeiten während des Laufs und teilweise aus den berechneten Gradienten abgeleitet. Wir verwenden den Parameter `alpha`, um zu bestimmen, in welchem Maße Gradienten und Belohnungen berücksichtigt werden – dies wird als *Lernrate* des Verstärkungsalgorithmus bezeichnet.\n",
|
||||
"5. Schließlich trainieren wir unser Netzwerk mit den Zuständen und den erwarteten Aktionen und wiederholen den Prozess.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"optimizer = torch.optim.Adam(model.parameters(), lr=0.01)\n",
|
||||
"\n",
|
||||
"def train_on_batch(x, y):\n",
|
||||
" x = torch.from_numpy(x)\n",
|
||||
" y = torch.from_numpy(y)\n",
|
||||
" optimizer.zero_grad()\n",
|
||||
" predictions = model(x)\n",
|
||||
" loss = -torch.mean(torch.log(predictions) * y)\n",
|
||||
" loss.backward()\n",
|
||||
" optimizer.step()\n",
|
||||
" return loss"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"alpha = 1e-4\n",
|
||||
"\n",
|
||||
"history = []\n",
|
||||
"for epoch in range(300):\n",
|
||||
" states, actions, probs, rewards = run_episode()\n",
|
||||
" one_hot_actions = np.eye(2)[actions.T][0]\n",
|
||||
" gradients = one_hot_actions-probs\n",
|
||||
" dr = discounted_rewards(rewards)\n",
|
||||
" gradients *= dr\n",
|
||||
" target = alpha*np.vstack([gradients])+probs\n",
|
||||
" train_on_batch(states,target)\n",
|
||||
" history.append(np.sum(rewards))\n",
|
||||
" if epoch%100==0:\n",
|
||||
" print(f\"{epoch} -> {np.sum(rewards)}\")\n",
|
||||
"\n",
|
||||
"plt.plot(history)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Lassen Sie uns nun die Episode mit Rendering ausführen, um das Ergebnis zu sehen:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"_ = run_episode(render=True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Hoffentlich kannst du sehen, dass der Stab jetzt ziemlich gut balancieren kann!\n",
|
||||
"\n",
|
||||
"## Actor-Critic-Modell\n",
|
||||
"\n",
|
||||
"Das Actor-Critic-Modell ist eine Weiterentwicklung der Policy-Gradient-Methoden, bei der wir ein neuronales Netzwerk erstellen, das sowohl die Policy als auch die geschätzten Belohnungen lernt. Das Netzwerk wird zwei Ausgaben haben (oder man kann es als zwei separate Netzwerke betrachten):\n",
|
||||
"* **Actor** wird die Aktion empfehlen, die ausgeführt werden soll, indem es uns die Zustandswahrscheinlichkeitsverteilung gibt, wie im Policy-Gradient-Modell.\n",
|
||||
"* **Critic** würde schätzen, wie die Belohnung aus diesen Aktionen aussehen könnte. Es gibt die insgesamt geschätzten zukünftigen Belohnungen im gegebenen Zustand zurück.\n",
|
||||
"\n",
|
||||
"Lass uns ein solches Modell definieren:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from itertools import count\n",
|
||||
"import torch.nn.functional as F"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
|
||||
"env = gym.make(\"CartPole-v1\")\n",
|
||||
"\n",
|
||||
"state_size = env.observation_space.shape[0]\n",
|
||||
"action_size = env.action_space.n\n",
|
||||
"lr = 0.0001\n",
|
||||
"\n",
|
||||
"class Actor(torch.nn.Module):\n",
|
||||
" def __init__(self, state_size, action_size):\n",
|
||||
" super(Actor, self).__init__()\n",
|
||||
" self.state_size = state_size\n",
|
||||
" self.action_size = action_size\n",
|
||||
" self.linear1 = torch.nn.Linear(self.state_size, 128)\n",
|
||||
" self.linear2 = torch.nn.Linear(128, 256)\n",
|
||||
" self.linear3 = torch.nn.Linear(256, self.action_size)\n",
|
||||
"\n",
|
||||
" def forward(self, state):\n",
|
||||
" output = F.relu(self.linear1(state))\n",
|
||||
" output = F.relu(self.linear2(output))\n",
|
||||
" output = self.linear3(output)\n",
|
||||
" distribution = torch.distributions.Categorical(F.softmax(output, dim=-1))\n",
|
||||
" return distribution\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"class Critic(torch.nn.Module):\n",
|
||||
" def __init__(self, state_size, action_size):\n",
|
||||
" super(Critic, self).__init__()\n",
|
||||
" self.state_size = state_size\n",
|
||||
" self.action_size = action_size\n",
|
||||
" self.linear1 = torch.nn.Linear(self.state_size, 128)\n",
|
||||
" self.linear2 = torch.nn.Linear(128, 256)\n",
|
||||
" self.linear3 = torch.nn.Linear(256, 1)\n",
|
||||
"\n",
|
||||
" def forward(self, state):\n",
|
||||
" output = F.relu(self.linear1(state))\n",
|
||||
" output = F.relu(self.linear2(output))\n",
|
||||
" value = self.linear3(output)\n",
|
||||
" return value"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Wir müssten unsere Funktionen `discounted_rewards` und `run_episode` leicht modifizieren:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def discounted_rewards(next_value, rewards, masks, gamma=0.99):\n",
|
||||
" R = next_value\n",
|
||||
" returns = []\n",
|
||||
" for step in reversed(range(len(rewards))):\n",
|
||||
" R = rewards[step] + gamma * R * masks[step]\n",
|
||||
" returns.insert(0, R)\n",
|
||||
" return returns\n",
|
||||
"\n",
|
||||
"def run_episode(actor, critic, n_iters):\n",
|
||||
" optimizerA = torch.optim.Adam(actor.parameters())\n",
|
||||
" optimizerC = torch.optim.Adam(critic.parameters())\n",
|
||||
" for iter in range(n_iters):\n",
|
||||
" state = env.reset()\n",
|
||||
" log_probs = []\n",
|
||||
" values = []\n",
|
||||
" rewards = []\n",
|
||||
" masks = []\n",
|
||||
" entropy = 0\n",
|
||||
" env.reset()\n",
|
||||
"\n",
|
||||
" for i in count():\n",
|
||||
" env.render()\n",
|
||||
" state = torch.FloatTensor(state).to(device)\n",
|
||||
" dist, value = actor(state), critic(state)\n",
|
||||
"\n",
|
||||
" action = dist.sample()\n",
|
||||
" next_state, reward, done, _ = env.step(action.cpu().numpy())\n",
|
||||
"\n",
|
||||
" log_prob = dist.log_prob(action).unsqueeze(0)\n",
|
||||
" entropy += dist.entropy().mean()\n",
|
||||
"\n",
|
||||
" log_probs.append(log_prob)\n",
|
||||
" values.append(value)\n",
|
||||
" rewards.append(torch.tensor([reward], dtype=torch.float, device=device))\n",
|
||||
" masks.append(torch.tensor([1-done], dtype=torch.float, device=device))\n",
|
||||
"\n",
|
||||
" state = next_state\n",
|
||||
"\n",
|
||||
" if done:\n",
|
||||
" print('Iteration: {}, Score: {}'.format(iter, i))\n",
|
||||
" break\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" next_state = torch.FloatTensor(next_state).to(device)\n",
|
||||
" next_value = critic(next_state)\n",
|
||||
" returns = discounted_rewards(next_value, rewards, masks)\n",
|
||||
"\n",
|
||||
" log_probs = torch.cat(log_probs)\n",
|
||||
" returns = torch.cat(returns).detach()\n",
|
||||
" values = torch.cat(values)\n",
|
||||
"\n",
|
||||
" advantage = returns - values\n",
|
||||
"\n",
|
||||
" actor_loss = -(log_probs * advantage.detach()).mean()\n",
|
||||
" critic_loss = advantage.pow(2).mean()\n",
|
||||
"\n",
|
||||
" optimizerA.zero_grad()\n",
|
||||
" optimizerC.zero_grad()\n",
|
||||
" actor_loss.backward()\n",
|
||||
" critic_loss.backward()\n",
|
||||
" optimizerA.step()\n",
|
||||
" optimizerC.step()\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Jetzt werden wir die Haupttrainingsschleife ausführen. Wir verwenden den manuellen Netzwerk-Trainingsprozess, indem wir geeignete Verlustfunktionen berechnen und Netzwerkparameter aktualisieren:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"\n",
|
||||
"actor = Actor(state_size, action_size).to(device)\n",
|
||||
"critic = Critic(state_size, action_size).to(device)\n",
|
||||
"run_episode(actor, critic, n_iters=100)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"env.close()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Erkenntnisse\n",
|
||||
"\n",
|
||||
"Wir haben in dieser Demo zwei RL-Algorithmen kennengelernt: den einfachen Policy-Gradient-Algorithmus und den anspruchsvolleren Actor-Critic-Algorithmus. Sie können sehen, dass diese Algorithmen mit abstrakten Konzepten wie Zustand, Aktion und Belohnung arbeiten – daher können sie auf sehr unterschiedliche Umgebungen angewendet werden.\n",
|
||||
"\n",
|
||||
"Reinforcement Learning ermöglicht es uns, die beste Strategie zur Lösung eines Problems allein durch die Betrachtung der endgültigen Belohnung zu erlernen. Die Tatsache, dass wir keine gelabelten Datensätze benötigen, erlaubt es uns, Simulationen viele Male zu wiederholen, um unsere Modelle zu optimieren. Dennoch gibt es immer noch viele Herausforderungen im Bereich RL, die Sie kennenlernen können, wenn Sie sich entscheiden, sich intensiver mit diesem faszinierenden Bereich der KI zu beschäftigen.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Haftungsausschluss**: \nDieses Dokument wurde mit dem KI-Übersetzungsdienst [Co-op Translator](https://github.com/Azure/co-op-translator) übersetzt. Obwohl wir uns um Genauigkeit bemühen, beachten Sie bitte, dass automatisierte Übersetzungen Fehler oder Ungenauigkeiten enthalten können. Das Originaldokument in seiner ursprünglichen Sprache sollte als maßgebliche Quelle betrachtet werden. Für kritische Informationen wird eine professionelle menschliche Übersetzung empfohlen. Wir übernehmen keine Haftung für Missverständnisse oder Fehlinterpretationen, die sich aus der Nutzung dieser Übersetzung ergeben.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.10.4 64-bit",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.4"
|
||||
},
|
||||
"orig_nbformat": 4,
|
||||
"vscode": {
|
||||
"interpreter": {
|
||||
"hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
|
||||
}
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "04f8d9978cd11281d81dd037cbf6ce20",
|
||||
"translation_date": "2025-08-31T15:44:40+00:00",
|
||||
"source_file": "lessons/6-Other/22-DeepRL/CartPole-RL-PyTorch.ipynb",
|
||||
"language_code": "de"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
|
|
@ -0,0 +1,109 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# # Training Mountain Car zur Flucht\n",
|
||||
"\n",
|
||||
"Laboraufgabe aus dem [AI for Beginners Curriculum](https://github.com/microsoft/ai-for-beginners).\n",
|
||||
"\n",
|
||||
"Dein Ziel ist es, den RL-Agenten zu trainieren, das [Mountain Car](https://www.gymlibrary.ml/environments/classic_control/mountain_car/) in der OpenAI-Umgebung zu steuern.\n",
|
||||
"\n",
|
||||
"Lass uns mit der Erstellung der Umgebung beginnen:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import gym\n",
|
||||
"env = gym.make('MountainCar-v0')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Lassen Sie uns sehen, wie das Zufallsexperiment aussieht:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"state = env.reset()\n",
|
||||
"while True:\n",
|
||||
" env.render()\n",
|
||||
" action = env.action_space.sample()\n",
|
||||
" state, reward, done, info = env.step(action)\n",
|
||||
" if done:\n",
|
||||
" break"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"## Lost of code here"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"env.close()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Haftungsausschluss**: \nDieses Dokument wurde mit dem KI-Übersetzungsdienst [Co-op Translator](https://github.com/Azure/co-op-translator) übersetzt. Obwohl wir uns um Genauigkeit bemühen, beachten Sie bitte, dass automatisierte Übersetzungen Fehler oder Ungenauigkeiten enthalten können. Das Originaldokument in seiner ursprünglichen Sprache sollte als maßgebliche Quelle betrachtet werden. Für kritische Informationen wird eine professionelle menschliche Übersetzung empfohlen. Wir übernehmen keine Haftung für Missverständnisse oder Fehlinterpretationen, die sich aus der Nutzung dieser Übersetzung ergeben.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.8.12 ('py38')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "f062b3b18449593ef8e0fcc029868781",
|
||||
"translation_date": "2025-08-31T15:47:55+00:00",
|
||||
"source_file": "lessons/6-Other/22-DeepRL/lab/MountainCar.ipynb",
|
||||
"language_code": "de"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
|
|
@ -1,8 +1,8 @@
|
|||
<!--
|
||||
CO_OP_TRANSLATOR_METADATA:
|
||||
{
|
||||
"original_hash": "f3a6b0ddf7e6e3f33b2a543baf086dc9",
|
||||
"translation_date": "2025-08-24T09:09:53+00:00",
|
||||
"original_hash": "07191303b7ea2aff1d47e2b0fe4bb862",
|
||||
"translation_date": "2025-08-31T15:37:51+00:00",
|
||||
"source_file": "README.md",
|
||||
"language_code": "es"
|
||||
}
|
||||
|
|
@ -23,19 +23,30 @@ CO_OP_TRANSLATOR_METADATA:
|
|||
|
||||
# Inteligencia Artificial para Principiantes - Un Currículo
|
||||
|
||||
| ](./lessons/sketchnotes/ai-overview.png)|
|
||||
||
|
||||
|:---:|
|
||||
| AI Para Principiantes - _Sketchnote por [@girlie_mac](https://twitter.com/girlie_mac)_ |
|
||||
|
||||
Explora el mundo de la **Inteligencia Artificial** (IA) con nuestro currículo de 12 semanas y 24 lecciones. Incluye lecciones prácticas, cuestionarios y laboratorios. El currículo está diseñado para principiantes y cubre herramientas como TensorFlow y PyTorch, además de ética en IA.
|
||||
|
||||
### 🌐 Soporte Multilingüe
|
||||
|
||||
#### Disponible a través de GitHub Action (Automatizado y Siempre Actualizado)
|
||||
|
||||
[Francés](../fr/README.md) | [Español](./README.md) | [Alemán](../de/README.md) | [Ruso](../ru/README.md) | [Árabe](../ar/README.md) | [Persa (Farsi)](../fa/README.md) | [Urdu](../ur/README.md) | [Chino (Simplificado)](../zh/README.md) | [Chino (Tradicional, Macao)](../mo/README.md) | [Chino (Tradicional, Hong Kong)](../hk/README.md) | [Chino (Tradicional, Taiwán)](../tw/README.md) | [Japonés](../ja/README.md) | [Coreano](../ko/README.md) | [Hindi](../hi/README.md) | [Bengalí](../bn/README.md) | [Maratí](../mr/README.md) | [Nepalí](../ne/README.md) | [Panyabí (Gurmukhi)](../pa/README.md) | [Portugués (Portugal)](../pt/README.md) | [Portugués (Brasil)](../br/README.md) | [Italiano](../it/README.md) | [Polaco](../pl/README.md) | [Turco](../tr/README.md) | [Griego](../el/README.md) | [Tailandés](../th/README.md) | [Sueco](../sv/README.md) | [Danés](../da/README.md) | [Noruego](../no/README.md) | [Finlandés](../fi/README.md) | [Holandés](../nl/README.md) | [Hebreo](../he/README.md) | [Vietnamita](../vi/README.md) | [Indonesio](../id/README.md) | [Malayo](../ms/README.md) | [Tagalo (Filipino)](../tl/README.md) | [Suajili](../sw/README.md) | [Húngaro](../hu/README.md) | [Checo](../cs/README.md) | [Eslovaco](../sk/README.md) | [Rumano](../ro/README.md) | [Búlgaro](../bg/README.md) | [Serbio (Cirílico)](../sr/README.md) | [Croata](../hr/README.md) | [Esloveno](../sl/README.md) | [Ucraniano](../uk/README.md) | [Birmano (Myanmar)](../my/README.md)
|
||||
|
||||
**Si deseas que se admitan idiomas adicionales, consulta la lista [aquí](https://github.com/Azure/co-op-translator/blob/main/getting_started/supported-languages.md)**
|
||||
|
||||
## Únete a la Comunidad
|
||||
[](https://discord.gg/kzRShWzttr)
|
||||
|
||||
## Lo que aprenderás
|
||||
|
||||
**[Mapa mental del curso](http://soshnikov.com/courses/ai-for-beginners/mindmap.html)**
|
||||
|
||||
En este currículo, aprenderás:
|
||||
|
||||
* Diferentes enfoques de la Inteligencia Artificial, incluyendo el enfoque simbólico "de la vieja escuela" con **Representación del Conocimiento** y razonamiento ([GOFAI](https://en.wikipedia.org/wiki/Symbolic_artificial_intelligence)).
|
||||
* Diferentes enfoques de la Inteligencia Artificial, incluyendo el enfoque simbólico "tradicional" con **Representación del Conocimiento** y razonamiento ([GOFAI](https://en.wikipedia.org/wiki/Symbolic_artificial_intelligence)).
|
||||
* **Redes Neuronales** y **Aprendizaje Profundo**, que son el núcleo de la IA moderna. Ilustraremos los conceptos detrás de estos temas importantes utilizando código en dos de los marcos más populares: [TensorFlow](http://Tensorflow.org) y [PyTorch](http://pytorch.org).
|
||||
* **Arquitecturas Neuronales** para trabajar con imágenes y texto. Cubriremos modelos recientes, aunque puede que no incluyamos lo más avanzado.
|
||||
* Enfoques menos populares de IA, como **Algoritmos Genéticos** y **Sistemas Multi-Agente**.
|
||||
|
|
@ -44,69 +55,69 @@ Lo que no cubriremos en este currículo:
|
|||
|
||||
> [Encuentra todos los recursos adicionales para este curso en nuestra colección de Microsoft Learn](https://learn.microsoft.com/en-us/collections/7w28iy2xrqzdj0?WT.mc_id=academic-77998-bethanycheum)
|
||||
|
||||
* Casos de negocio sobre el uso de **IA en los negocios**. Considera tomar el camino de aprendizaje [Introducción a la IA para usuarios empresariales](https://docs.microsoft.com/learn/paths/introduction-ai-for-business-users/?WT.mc_id=academic-77998-bethanycheum) en Microsoft Learn, o [AI Business School](https://www.microsoft.com/ai/ai-business-school/?WT.mc_id=academic-77998-bethanycheum), desarrollado en cooperación con [INSEAD](https://www.insead.edu/).
|
||||
* Casos de negocio sobre el uso de **IA en Negocios**. Considera tomar el camino de aprendizaje [Introducción a la IA para usuarios empresariales](https://docs.microsoft.com/learn/paths/introduction-ai-for-business-users/?WT.mc_id=academic-77998-bethanycheum) en Microsoft Learn, o [AI Business School](https://www.microsoft.com/ai/ai-business-school/?WT.mc_id=academic-77998-bethanycheum), desarrollado en cooperación con [INSEAD](https://www.insead.edu/).
|
||||
* **Aprendizaje Automático Clásico**, que está bien descrito en nuestro [Currículo de Aprendizaje Automático para Principiantes](http://github.com/Microsoft/ML-for-Beginners).
|
||||
* Aplicaciones prácticas de IA construidas utilizando **[Servicios Cognitivos](https://azure.microsoft.com/services/cognitive-services/?WT.mc_id=academic-77998-bethanycheum)**. Para esto, recomendamos comenzar con los módulos de Microsoft Learn para [visión](https://docs.microsoft.com/learn/paths/create-computer-vision-solutions-azure-cognitive-services/?WT.mc_id=academic-77998-bethanycheum), [procesamiento de lenguaje natural](https://docs.microsoft.com/learn/paths/explore-natural-language-processing/?WT.mc_id=academic-77998-bethanycheum), **[IA Generativa con Azure OpenAI Service](https://learn.microsoft.com/en-us/training/paths/develop-ai-solutions-azure-openai/?WT.mc_id=academic-77998-bethanycheum)** y otros.
|
||||
* Marcos específicos de ML en la **nube**, como [Azure Machine Learning](https://azure.microsoft.com/services/machine-learning/?WT.mc_id=academic-77998-bethanycheum), [Microsoft Fabric](https://learn.microsoft.com/en-us/training/paths/get-started-fabric/?WT.mc_id=academic-77998-bethanycheum), o [Azure Databricks](https://docs.microsoft.com/learn/paths/data-engineer-azure-databricks?WT.mc_id=academic-77998-bethanycheum). Considera usar los caminos de aprendizaje [Construir y operar soluciones de aprendizaje automático con Azure Machine Learning](https://docs.microsoft.com/learn/paths/build-ai-solutions-with-azure-ml-service/?WT.mc_id=academic-77998-bethanycheum) y [Construir y operar soluciones de aprendizaje automático con Azure Databricks](https://docs.microsoft.com/learn/paths/build-operate-machine-learning-solutions-azure-databricks/?WT.mc_id=academic-77998-bethanycheum).
|
||||
* Marcos específicos de ML en la **Nube**, como [Azure Machine Learning](https://azure.microsoft.com/services/machine-learning/?WT.mc_id=academic-77998-bethanycheum), [Microsoft Fabric](https://learn.microsoft.com/en-us/training/paths/get-started-fabric/?WT.mc_id=academic-77998-bethanycheum), o [Azure Databricks](https://docs.microsoft.com/learn/paths/data-engineer-azure-databricks?WT.mc_id=academic-77998-bethanycheum). Considera usar los caminos de aprendizaje [Construir y operar soluciones de aprendizaje automático con Azure Machine Learning](https://docs.microsoft.com/learn/paths/build-ai-solutions-with-azure-ml-service/?WT.mc_id=academic-77998-bethanycheum) y [Construir y operar soluciones de aprendizaje automático con Azure Databricks](https://docs.microsoft.com/learn/paths/build-operate-machine-learning-solutions-azure-databricks/?WT.mc_id=academic-77998-bethanycheum).
|
||||
* **IA Conversacional** y **Chat Bots**. Hay un camino de aprendizaje separado [Crear soluciones de IA conversacional](https://docs.microsoft.com/learn/paths/create-conversational-ai-solutions/?WT.mc_id=academic-77998-bethanycheum), y también puedes consultar [esta publicación de blog](https://soshnikov.com/azure/hello-bot-conversational-ai-on-microsoft-platform/) para más detalles.
|
||||
* **Matemáticas profundas** detrás del aprendizaje profundo. Para esto, recomendamos [Deep Learning](https://www.amazon.com/Deep-Learning-Adaptive-Computation-Machine/dp/0262035618) de Ian Goodfellow, Yoshua Bengio y Aaron Courville, que también está disponible en línea en [https://www.deeplearningbook.org/](https://www.deeplearningbook.org/).
|
||||
* **Matemáticas profundas** detrás del aprendizaje profundo. Para esto, recomendamos [Deep Learning](https://www.amazon.com/Deep-Learning-Adaptive-Computation-Machine/dp/0262035618) por Ian Goodfellow, Yoshua Bengio y Aaron Courville, que también está disponible en línea en [https://www.deeplearningbook.org/](https://www.deeplearningbook.org/).
|
||||
|
||||
Para una introducción sencilla a los temas de _IA en la nube_, puedes considerar tomar el camino de aprendizaje [Comienza con inteligencia artificial en Azure](https://docs.microsoft.com/learn/paths/get-started-with-artificial-intelligence-on-azure/?WT.mc_id=academic-77998-bethanycheum).
|
||||
Para una introducción sencilla a los temas de _IA en la Nube_, puedes considerar tomar el camino de aprendizaje [Comienza con inteligencia artificial en Azure](https://docs.microsoft.com/learn/paths/get-started-with-artificial-intelligence-on-azure/?WT.mc_id=academic-77998-bethanycheum).
|
||||
|
||||
# Contenido
|
||||
|
||||
| | Enlace de la Lección | PyTorch/Keras/TensorFlow | Laboratorio |
|
||||
| :-: | :------------------------------------------------------------------------------------------------------------------------------------------: | :---------------------------------------------------------------------------------------------: | ------------------------------------------------------------------------------ |
|
||||
| 0 | [Configuración del Curso](./lessons/0-course-setup/setup.md) | [Configura tu entorno de desarrollo](./lessons/0-course-setup/how-to-run.md) | |
|
||||
| 0 | [Configuración del Curso](./lessons/0-course-setup/setup.md) | [Configura tu Entorno de Desarrollo](./lessons/0-course-setup/how-to-run.md) | |
|
||||
| I | [**Introducción a la IA**](./lessons/1-Intro/README.md) | | |
|
||||
| 01 | [Introducción e Historia de la IA](./lessons/1-Intro/README.md) | - | - |
|
||||
| II | **IA Simbólica** |
|
||||
| 02 | [Representación del Conocimiento y Sistemas Expertos](./lessons/2-Symbolic/README.md) | [Sistemas Expertos](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/2-Symbolic/Animals.ipynb) / [Ontología](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/2-Symbolic/FamilyOntology.ipynb) /[Gráfico de Conceptos](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/2-Symbolic/MSConceptGraph.ipynb) | |
|
||||
| 02 | [Representación del Conocimiento y Sistemas Expertos](./lessons/2-Symbolic/README.md) | [Sistemas Expertos](./lessons/2-Symbolic/Animals.ipynb) / [Ontología](./lessons/2-Symbolic/FamilyOntology.ipynb) /[Concept Graph](./lessons/2-Symbolic/MSConceptGraph.ipynb) | |
|
||||
| III | [**Introducción a Redes Neuronales**](./lessons/3-NeuralNetworks/README.md) |||
|
||||
| 03 | [Perceptrón](./lessons/3-NeuralNetworks/03-Perceptron/README.md) | [Notebook](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/3-NeuralNetworks/03-Perceptron/Perceptron.ipynb) | [Laboratorio](./lessons/3-NeuralNetworks/03-Perceptron/lab/README.md) |
|
||||
| 04 | [Perceptrón Multicapa y Creación de nuestro propio Marco](./lessons/3-NeuralNetworks/04-OwnFramework/README.md) | [Notebook](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/3-NeuralNetworks/04-OwnFramework/OwnFramework.ipynb) | [Laboratorio](./lessons/3-NeuralNetworks/04-OwnFramework/lab/README.md) |
|
||||
| 05 | [Introducción a Marcos (PyTorch/TensorFlow) y Sobreajuste](./lessons/3-NeuralNetworks/05-Frameworks/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/3-NeuralNetworks/05-Frameworks/IntroPyTorch.ipynb) / [Keras](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/3-NeuralNetworks/05-Frameworks/IntroKeras.ipynb) / [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/3-NeuralNetworks/05-Frameworks/IntroKerasTF.ipynb) | [Laboratorio](./lessons/3-NeuralNetworks/05-Frameworks/lab/README.md) |
|
||||
| IV | [**Visión por Computadora**](./lessons/4-ComputerVision/README.md) | [PyTorch](https://docs.microsoft.com/learn/modules/intro-computer-vision-pytorch/?WT.mc_id=academic-77998-cacaste) / [TensorFlow](https://docs.microsoft.com/learn/modules/intro-computer-vision-TensorFlow/?WT.mc_id=academic-77998-cacaste)| [Explora Visión por Computadora en Microsoft Azure](https://learn.microsoft.com/en-us/collections/7w28iy2xrqzdj0?WT.mc_id=academic-77998-bethanycheum) |
|
||||
| 06 | [Introducción a Visión por Computadora. OpenCV](./lessons/4-ComputerVision/06-IntroCV/README.md) | [Notebook](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/06-IntroCV/OpenCV.ipynb) | [Laboratorio](./lessons/4-ComputerVision/06-IntroCV/lab/README.md) |
|
||||
| 07 | [Redes Neuronales Convolucionales](./lessons/4-ComputerVision/07-ConvNets/README.md) & [Arquitecturas CNN](./lessons/4-ComputerVision/07-ConvNets/CNN_Architectures.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/07-ConvNets/ConvNetsPyTorch.ipynb) /[TensorFlow](https://microsoft.github.io/AI-For-Beginners/lessons/4-ComputerVision/07-ConvNets/ConvNetsTF.ipynb) | [Laboratorio](./lessons/4-ComputerVision/07-ConvNets/lab/README.md) |
|
||||
| 08 | [Redes preentrenadas y aprendizaje por transferencia](./lessons/4-ComputerVision/08-TransferLearning/README.md) y [Trucos de entrenamiento](./lessons/4-ComputerVision/08-TransferLearning/TrainingTricks.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/08-TransferLearning/TransferLearningPyTorch.ipynb) / [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/3-NeuralNetworks/05-Frameworks/IntroKerasTF.ipynb) | [Laboratorio](./lessons/4-ComputerVision/08-TransferLearning/lab/README.md) |
|
||||
| 09 | [Autoencoders y VAEs](./lessons/4-ComputerVision/09-Autoencoders/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/09-Autoencoders/AutoEncodersPyTorch.ipynb) / [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/09-Autoencoders/AutoencodersTF.ipynb) | |
|
||||
| 10 | [Redes Generativas Adversarias y Transferencia de Estilo Artístico](./lessons/4-ComputerVision/10-GANs/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/10-GANs/GANPyTorch.ipynb) / [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/10-GANs/GANTF.ipynb) | |
|
||||
| 11 | [Detección de Objetos](./lessons/4-ComputerVision/11-ObjectDetection/README.md) | [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/11-ObjectDetection/ObjectDetection.ipynb) | [Laboratorio](./lessons/4-ComputerVision/11-ObjectDetection/lab/README.md) |
|
||||
| 12 | [Segmentación Semántica. U-Net](./lessons/4-ComputerVision/12-Segmentation/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/12-Segmentation/SemanticSegmentationPytorch.ipynb) / [TensorFlow](../../(https:/github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/12-Segmentation/SemanticSegmentationTF.ipynb)) | |
|
||||
| V | [**Procesamiento de Lenguaje Natural**](./lessons/5-NLP/README.md) | [PyTorch](https://docs.microsoft.com/learn/modules/intro-natural-language-processing-pytorch/?WT.mc_id=academic-77998-cacaste) /[TensorFlow](https://docs.microsoft.com/learn/modules/intro-natural-language-processing-TensorFlow/?WT.mc_id=academic-77998-cacaste) | [Explora el Procesamiento de Lenguaje Natural en Microsoft Azure](https://learn.microsoft.com/en-us/collections/7w28iy2xrqzdj0?WT.mc_id=academic-77998-bethanycheum)|
|
||||
| 13 | [Representación de Texto. Bow/TF-IDF](./lessons/5-NLP/13-TextRep/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/13-TextRep/TextRepresentationPyTorch.ipynb) / [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/13-TextRep/TextRepresentationTF.ipynb) | |
|
||||
| 14 | [Embeddings semánticos de palabras. Word2Vec y GloVe](./lessons/5-NLP/14-Embeddings/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb) / [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/14-Embeddings/EmbeddingsTF.ipynb) | |
|
||||
| 15 | [Modelado de Lenguaje. Entrenando tus propios embeddings](./lessons/5-NLP/15-LanguageModeling/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/15-LanguageModeling/CBoW-PyTorch.ipynb) / [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/15-LanguageModeling/CBoW-TF.ipynb) | [Laboratorio](./lessons/5-NLP/15-LanguageModeling/lab/README.md) |
|
||||
| 16 | [Redes Neuronales Recurrentes](./lessons/5-NLP/16-RNN/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/16-RNN/RNNPyTorch.ipynb) / [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/16-RNN/RNNTF.ipynb) | |
|
||||
| 17 | [Redes Recurrentes Generativas](./lessons/5-NLP/17-GenerativeNetworks/README.md) | [PyTorch](https://microsoft.github.io/AI-For-Beginners/lessons/5-NLP/17-GenerativeNetworks/GenerativePyTorch.md) / [TensorFlow](https://microsoft.github.io/AI-For-Beginners/lessons/5-NLP/17-GenerativeNetworks/GenerativeTF.md) | [Laboratorio](./lessons/5-NLP/17-GenerativeNetworks/lab/README.md) |
|
||||
| 18 | [Transformers. BERT.](./lessons/5-NLP/18-Transformers/READMEtransformers.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/18-Transformers/TransformersPyTorch.ipynb) /[TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/18-Transformers/TransformersTF.ipynb) | |
|
||||
| 19 | [Reconocimiento de Entidades Nombradas](./lessons/5-NLP/19-NER/README.md) | [TensorFlow](https://microsoft.github.io/AI-For-Beginners/lessons/5-NLP/19-NER/NER-TF.ipynb) | [Laboratorio](./lessons/5-NLP/19-NER/lab/README.md) |
|
||||
| 20 | [Modelos de Lenguaje Grandes, Programación con Prompts y Tareas de Few-Shot](./lessons/5-NLP/20-LangModels/READMELargeLang.md) | [PyTorch](https://microsoft.github.io/AI-For-Beginners/lessons/5-NLP/20-LangModels/GPT-PyTorch.ipynb) | |
|
||||
| 03 | [Perceptrón](./lessons/3-NeuralNetworks/03-Perceptron/README.md) | [Notebook](./lessons/3-NeuralNetworks/03-Perceptron/Perceptron.ipynb) | [Laboratorio](./lessons/3-NeuralNetworks/03-Perceptron/lab/README.md) |
|
||||
| 04 | [Perceptrón Multicapa y Creación de nuestro propio Marco](./lessons/3-NeuralNetworks/04-OwnFramework/README.md) | [Notebook](./lessons/3-NeuralNetworks/04-OwnFramework/OwnFramework.ipynb) | [Laboratorio](./lessons/3-NeuralNetworks/04-OwnFramework/lab/README.md) |
|
||||
| 05 | [Introducción a Frameworks (PyTorch/TensorFlow) y Overfitting](./lessons/3-NeuralNetworks/05-Frameworks/README.md) | [PyTorch](./lessons/3-NeuralNetworks/05-Frameworks/IntroPyTorch.ipynb) / [Keras](./lessons/3-NeuralNetworks/05-Frameworks/IntroKeras.ipynb) / [TensorFlow](./lessons/3-NeuralNetworks/05-Frameworks/IntroKerasTF.ipynb) | [Laboratorio](./lessons/3-NeuralNetworks/05-Frameworks/lab/README.md) |
|
||||
| IV | [**Visión por Computadora**](./lessons/4-ComputerVision/README.md) | [PyTorch](https://docs.microsoft.com/learn/modules/intro-computer-vision-pytorch/?WT.mc_id=academic-77998-cacaste) / [TensorFlow](https://docs.microsoft.com/learn/modules/intro-computer-vision-TensorFlow/?WT.mc_id=academic-77998-cacaste)| [Explora Visión por Computadora en Microsoft Azure](https://learn.microsoft.com/en-us/collections/7w28iy2xrqzdj0?WT.mc_id=academic-77998-bethanycheum) |
|
||||
| 06 | [Introducción a Visión por Computadora. OpenCV](./lessons/4-ComputerVision/06-IntroCV/README.md) | [Notebook](./lessons/4-ComputerVision/06-IntroCV/OpenCV.ipynb) | [Laboratorio](./lessons/4-ComputerVision/06-IntroCV/lab/README.md) |
|
||||
| 07 | [Redes Neuronales Convolucionales](./lessons/4-ComputerVision/07-ConvNets/README.md) & [Arquitecturas CNN](./lessons/4-ComputerVision/07-ConvNets/CNN_Architectures.md) | [PyTorch](./lessons/4-ComputerVision/07-ConvNets/ConvNetsPyTorch.ipynb) /[TensorFlow](./lessons/4-ComputerVision/07-ConvNets/ConvNetsTF.ipynb) | [Laboratorio](./lessons/4-ComputerVision/07-ConvNets/lab/README.md) |
|
||||
| 08 | [Redes Pre-entrenadas y Aprendizaje por Transferencia](./lessons/4-ComputerVision/08-TransferLearning/README.md) y [Trucos de Entrenamiento](./lessons/4-ComputerVision/08-TransferLearning/TrainingTricks.md) | [PyTorch](./lessons/4-ComputerVision/08-TransferLearning/TransferLearningPyTorch.ipynb) / [TensorFlow](./lessons/3-NeuralNetworks/05-Frameworks/IntroKerasTF.ipynb) | [Laboratorio](./lessons/4-ComputerVision/08-TransferLearning/lab/README.md) |
|
||||
| 09 | [Autoencoders y VAEs](./lessons/4-ComputerVision/09-Autoencoders/README.md) | [PyTorch](./lessons/4-ComputerVision/09-Autoencoders/AutoEncodersPyTorch.ipynb) / [TensorFlow](./lessons/4-ComputerVision/09-Autoencoders/AutoencodersTF.ipynb) | |
|
||||
| 10 | [Redes Generativas Adversarias y Transferencia de Estilo Artístico](./lessons/4-ComputerVision/10-GANs/README.md) | [PyTorch](./lessons/4-ComputerVision/10-GANs/GANPyTorch.ipynb) / [TensorFlow](./lessons/4-ComputerVision/10-GANs/GANTF.ipynb) | |
|
||||
| 11 | [Detección de Objetos](./lessons/4-ComputerVision/11-ObjectDetection/README.md) | [TensorFlow](./lessons/4-ComputerVision/11-ObjectDetection/ObjectDetection.ipynb) | [Laboratorio](./lessons/4-ComputerVision/11-ObjectDetection/lab/README.md) |
|
||||
| 12 | [Segmentación Semántica. U-Net](./lessons/4-ComputerVision/12-Segmentation/README.md) | [PyTorch](./lessons/4-ComputerVision/12-Segmentation/SemanticSegmentationPytorch.ipynb) / [TensorFlow](./lessons/4-ComputerVision/12-Segmentation/SemanticSegmentationTF.ipynb) | |
|
||||
| V | [**Procesamiento de Lenguaje Natural**](./lessons/5-NLP/README.md) | [PyTorch](https://docs.microsoft.com/learn/modules/intro-natural-language-processing-pytorch/?WT.mc_id=academic-77998-cacaste) /[TensorFlow](https://docs.microsoft.com/learn/modules/intro-natural-language-processing-TensorFlow/?WT.mc_id=academic-77998-cacaste) | [Explora Procesamiento de Lenguaje Natural en Microsoft Azure](https://learn.microsoft.com/en-us/collections/7w28iy2xrqzdj0?WT.mc_id=academic-77998-bethanycheum)|
|
||||
| 13 | [Representación de Texto. Bow/TF-IDF](./lessons/5-NLP/13-TextRep/README.md) | [PyTorch](./lessons/5-NLP/13-TextRep/TextRepresentationPyTorch.ipynb) / [TensorFlow](./lessons/5-NLP/13-TextRep/TextRepresentationTF.ipynb) | |
|
||||
| 14 | [Embeddings semánticos de palabras. Word2Vec y GloVe](./lessons/5-NLP/14-Embeddings/README.md) | [PyTorch](./lessons/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb) / [TensorFlow](./lessons/5-NLP/14-Embeddings/EmbeddingsTF.ipynb) | |
|
||||
| 15 | [Modelado de Lenguaje. Entrenando tus propios embeddings](./lessons/5-NLP/15-LanguageModeling/README.md) | [PyTorch](./lessons/5-NLP/15-LanguageModeling/CBoW-PyTorch.ipynb) / [TensorFlow](./lessons/5-NLP/15-LanguageModeling/CBoW-TF.ipynb) | [Laboratorio](./lessons/5-NLP/15-LanguageModeling/lab/README.md) |
|
||||
| 16 | [Redes Neuronales Recurrentes](./lessons/5-NLP/16-RNN/README.md) | [PyTorch](./lessons/5-NLP/16-RNN/RNNPyTorch.ipynb) / [TensorFlow](./lessons/5-NLP/16-RNN/RNNTF.ipynb) | |
|
||||
| 17 | [Redes Recurrentes Generativas](./lessons/5-NLP/17-GenerativeNetworks/README.md) | [PyTorch](./lessons/5-NLP/17-GenerativeNetworks/GenerativePyTorch.md) / [TensorFlow](./lessons/5-NLP/17-GenerativeNetworks/GenerativeTF.md) | [Laboratorio](./lessons/5-NLP/17-GenerativeNetworks/lab/README.md) |
|
||||
| 18 | [Transformers. BERT.](./lessons/5-NLP/18-Transformers/READMEtransformers.md) | [PyTorch](./lessons/5-NLP/18-Transformers/TransformersPyTorch.ipynb) /[TensorFlow](./lessons/5-NLP/18-Transformers/TransformersTF.ipynb) | |
|
||||
| 19 | [Reconocimiento de Entidades Nombradas](./lessons/5-NLP/19-NER/README.md) | [TensorFlow](./lessons/5-NLP/19-NER/NER-TF.ipynb) | [Laboratorio](./lessons/5-NLP/19-NER/lab/README.md) |
|
||||
| 20 | [Modelos de Lenguaje Grandes, Programación de Prompts y Tareas de Few-Shot](./lessons/5-NLP/20-LangModels/READMELargeLang.md) | [PyTorch](./lessons/5-NLP/20-LangModels/GPT-PyTorch.ipynb) | |
|
||||
| VI | **Otras Técnicas de IA** || |
|
||||
| 21 | [Algoritmos Genéticos](./lessons/6-Other/21-GeneticAlgorithms/README.md) | [Notebook](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/6-Other/21-GeneticAlgorithms/Genetic.ipynb) | |
|
||||
| 22 | [Aprendizaje por Refuerzo Profundo](./lessons/6-Other/22-DeepRL/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/6-Other/22-DeepRL/CartPole-RL-PyTorch.ipynb) /[TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/6-Other/22-DeepRL/CartPole-RL-TF.ipynb) | [Laboratorio](./lessons/6-Other/22-DeepRL/lab/README.md) |
|
||||
| 21 | [Algoritmos Genéticos](./lessons/6-Other/21-GeneticAlgorithms/README.md) | [Notebook](./lessons/6-Other/21-GeneticAlgorithms/Genetic.ipynb) | |
|
||||
| 22 | [Aprendizaje por Refuerzo Profundo](./lessons/6-Other/22-DeepRL/README.md) | [PyTorch](./lessons/6-Other/22-DeepRL/CartPole-RL-PyTorch.ipynb) /[TensorFlow](./lessons/6-Other/22-DeepRL/CartPole-RL-TF.ipynb) | [Laboratorio](./lessons/6-Other/22-DeepRL/lab/README.md) |
|
||||
| 23 | [Sistemas Multi-Agente](./lessons/6-Other/23-MultiagentSystems/README.md) | | |
|
||||
| VII | **Ética en la IA** | | |
|
||||
| 24 | [Ética en la IA y IA Responsable](./lessons/7-Ethics/README.md) | [Microsoft Learn: Principios de IA Responsable](https://docs.microsoft.com/learn/paths/responsible-ai-business-principles/?WT.mc_id=academic-77998-cacaste) | |
|
||||
| VII | **Ética en IA** | | |
|
||||
| 24 | [Ética en IA y IA Responsable](./lessons/7-Ethics/README.md) | [Microsoft Learn: Principios de IA Responsable](https://docs.microsoft.com/learn/paths/responsible-ai-business-principles/?WT.mc_id=academic-77998-cacaste) | |
|
||||
| IX | **Extras** | | |
|
||||
| 25 | [Redes Multimodales, CLIP y VQGAN](./lessons/X-Extras/X1-MultiModal/README.md) | [Notebook](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/X-Extras/X1-MultiModal/Clip.ipynb) | |
|
||||
| 25 | [Redes Multi-Modales, CLIP y VQGAN](./lessons/X-Extras/X1-MultiModal/README.md) | [Notebook](./lessons/X-Extras/X1-MultiModal/Clip.ipynb) | |
|
||||
|
||||
## Cada lección contiene
|
||||
|
||||
* Material de lectura previa.
|
||||
* Notebooks ejecutables de Jupyter, que a menudo son específicos para el framework (**PyTorch** o **TensorFlow**). El notebook ejecutable también incluye mucho material teórico, por lo que para entender el tema necesitas revisar al menos una versión del notebook (ya sea PyTorch o TensorFlow).
|
||||
* **Laboratorios** disponibles para algunos temas, que te dan la oportunidad de aplicar el material aprendido a un problema específico.
|
||||
* Material de lectura previa
|
||||
* Jupyter Notebooks ejecutables, que suelen ser específicos para el framework (**PyTorch** o **TensorFlow**). El notebook ejecutable también incluye mucho material teórico, por lo que para entender el tema necesitas revisar al menos una versión del notebook (ya sea PyTorch o TensorFlow).
|
||||
* **Laboratorios** disponibles para algunos temas, que te dan la oportunidad de aplicar el material que has aprendido a un problema específico.
|
||||
* Algunas secciones contienen enlaces a módulos de [**MS Learn**](https://learn.microsoft.com/en-us/collections/7w28iy2xrqzdj0?WT.mc_id=academic-77998-bethanycheum) que cubren temas relacionados.
|
||||
|
||||
## Comenzando
|
||||
|
||||
- Hemos creado una [lección de configuración](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/0-course-setup/setup.md) para ayudarte a configurar tu entorno de desarrollo. - Para educadores, también hemos creado una [lección de configuración de currículum](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/0-course-setup/for-teachers.md).
|
||||
- Cómo [Ejecutar el código en VSCode o Codespace](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/0-course-setup/how-to-run.md).
|
||||
- Hemos creado una [lección de configuración](./lessons/0-course-setup/setup.md) para ayudarte a configurar tu entorno de desarrollo. - Para educadores, también hemos creado una [lección de configuración de currículum](./lessons/0-course-setup/for-teachers.md).
|
||||
- Cómo [Ejecutar el código en VSCode o Codepace](./lessons/0-course-setup/how-to-run.md)
|
||||
|
||||
Sigue estos pasos:
|
||||
|
||||
Haz un fork del repositorio: Haz clic en el botón "Fork" en la esquina superior derecha de esta página.
|
||||
Haz un Fork del repositorio: Haz clic en el botón "Fork" en la esquina superior derecha de esta página.
|
||||
|
||||
Clona el repositorio: `git clone https://github.com/microsoft/AI-For-Beginners.git`
|
||||
|
||||
|
|
@ -114,42 +125,44 @@ No olvides darle una estrella (🌟) a este repositorio para encontrarlo más f
|
|||
|
||||
## Conoce a otros estudiantes
|
||||
|
||||
Únete a nuestro [servidor oficial de Discord de IA](https://aka.ms/genai-discord?WT.mc_id=academic-105485-bethanycheum) para conocer y conectar con otros estudiantes que están tomando este curso y obtener soporte.
|
||||
Únete a nuestro [servidor oficial de Discord de IA](https://aka.ms/genai-discord?WT.mc_id=academic-105485-bethanycheum) para conocer y conectarte con otros estudiantes que están tomando este curso y obtener apoyo.
|
||||
|
||||
Si tienes comentarios sobre el producto o preguntas mientras construyes, visita nuestro [Foro de Desarrolladores de Azure AI Foundry](https://aka.ms/foundry/forum).
|
||||
Si tienes comentarios sobre el producto o preguntas mientras construyes, visita nuestro [Foro de Desarrolladores de Azure AI Foundry](https://aka.ms/foundry/forum)
|
||||
|
||||
## Cuestionarios
|
||||
> **Una nota sobre los cuestionarios**: Todos los cuestionarios están contenidos en la carpeta Quiz-app en etc\quiz-app. Están vinculados desde las lecciones, y la aplicación de cuestionarios se puede ejecutar localmente o desplegar en Azure; sigue las instrucciones en la carpeta `quiz-app`. Se están localizando gradualmente.
|
||||
> **Una nota sobre los cuestionarios**: Todos los cuestionarios están contenidos en la carpeta Quiz-app en etc\quiz-app, o [En línea aquí](https://ff-quizzes.netlify.app/). Están vinculados desde dentro de las lecciones. La aplicación de cuestionarios puede ejecutarse localmente o desplegarse en Azure; sigue las instrucciones en la carpeta `quiz-app`. Se están localizando gradualmente.
|
||||
## Se Busca Ayuda
|
||||
|
||||
¿Tienes sugerencias o encontraste errores de ortografía o código? Crea un issue o un pull request.
|
||||
¿Tienes sugerencias o encontraste errores ortográficos o de código? Crea un issue o un pull request.
|
||||
|
||||
## Agradecimientos Especiales
|
||||
|
||||
* **✍️ Autor Principal:** [Dmitry Soshnikov](http://soshnikov.com), PhD
|
||||
* **🔥 Editora:** [Jen Looper](https://twitter.com/jenlooper), PhD
|
||||
* **🎨 Ilustradora de Sketchnotes:** [Tomomi Imura](https://twitter.com/girlie_mac)
|
||||
* **✅ Creadora de Cuestionarios:** [Lateefah Bello](https://github.com/CinnamonXI), [MLSA](https://studentambassadors.microsoft.com/)
|
||||
* **🙏 Contribuyentes Principales:** [Evgenii Pishchik](https://github.com/Pe4enIks)
|
||||
* **✍️ Autor Principal:** [Dmitry Soshnikov](http://soshnikov.com), PhD
|
||||
* **🔥 Editora:** [Jen Looper](https://twitter.com/jenlooper), PhD
|
||||
* **🎨 Ilustradora de Sketchnotes:** [Tomomi Imura](https://twitter.com/girlie_mac)
|
||||
* **✅ Creadora de Cuestionarios:** [Lateefah Bello](https://github.com/CinnamonXI), [MLSA](https://studentambassadors.microsoft.com/)
|
||||
* **🙏 Contribuidores Principales:** [Evgenii Pishchik](https://github.com/Pe4enIks)
|
||||
|
||||
## Otros Currículos
|
||||
|
||||
¡Nuestro equipo produce otros currículos! Échales un vistazo:
|
||||
|
||||
- [Generative AI for Beginners](https://aka.ms/genai-beginners)
|
||||
- [Generative AI for Beginners .NET](https://github.com/microsoft/Generative-AI-for-beginners-dotnet)
|
||||
- [Generative AI with JavaScript](https://github.com/microsoft/generative-ai-with-javascript)
|
||||
- [Generative AI with Java](https://github.com/microsoft/Generative-AI-for-beginners-java)
|
||||
- [AI for Beginners](https://aka.ms/ai-beginners)
|
||||
- [Data Science for Beginners](https://aka.ms/datascience-beginners)
|
||||
- [ML for Beginners](https://aka.ms/ml-beginners)
|
||||
- [Cybersecurity for Beginners](https://github.com/microsoft/Security-101)
|
||||
- [Web Dev for Beginners](https://aka.ms/webdev-beginners)
|
||||
- [IoT for Beginners](https://aka.ms/iot-beginners)
|
||||
- [XR Development for Beginners](https://github.com/microsoft/xr-development-for-beginners)
|
||||
- [Mastering GitHub Copilot for Agentic use](https://github.com/microsoft/Mastering-GitHub-Copilot-for-Paired-Programming)
|
||||
- [Mastering GitHub Copilot for C#/.NET Developers](https://github.com/microsoft/mastering-github-copilot-for-dotnet-csharp-developers)
|
||||
- [Choose Your Own Copilot Adventure](https://github.com/microsoft/CopilotAdventures)
|
||||
- [Generative AI for Beginners](https://aka.ms/genai-beginners)
|
||||
- [Generative AI for Beginners .NET](https://github.com/microsoft/Generative-AI-for-beginners-dotnet)
|
||||
- [Generative AI with JavaScript](https://github.com/microsoft/generative-ai-with-javascript)
|
||||
- [Generative AI with Java](https://github.com/microsoft/Generative-AI-for-beginners-java)
|
||||
- [AI for Beginners](https://aka.ms/ai-beginners)
|
||||
- [Data Science for Beginners](https://aka.ms/datascience-beginners)
|
||||
- [ML for Beginners](https://aka.ms/ml-beginners)
|
||||
- [Cybersecurity for Beginners](https://github.com/microsoft/Security-101)
|
||||
- [Web Dev for Beginners](https://aka.ms/webdev-beginners)
|
||||
- [IoT for Beginners](https://aka.ms/iot-beginners)
|
||||
- [XR Development for Beginners](https://github.com/microsoft/xr-development-for-beginners)
|
||||
- [Mastering GitHub Copilot for Agentic use](https://github.com/microsoft/Mastering-GitHub-Copilot-for-Paired-Programming)
|
||||
- [Mastering GitHub Copilot for C#/.NET Developers](https://github.com/microsoft/mastering-github-copilot-for-dotnet-csharp-developers)
|
||||
- [Choose Your Own Copilot Adventure](https://github.com/microsoft/CopilotAdventures)
|
||||
|
||||
---
|
||||
|
||||
**Descargo de responsabilidad**:
|
||||
Este documento ha sido traducido utilizando el servicio de traducción automática [Co-op Translator](https://github.com/Azure/co-op-translator). Aunque nos esforzamos por garantizar la precisión, tenga en cuenta que las traducciones automatizadas pueden contener errores o imprecisiones. El documento original en su idioma nativo debe considerarse la fuente autorizada. Para información crítica, se recomienda una traducción profesional realizada por humanos. No nos hacemos responsables de malentendidos o interpretaciones erróneas que puedan surgir del uso de esta traducción.
|
||||
Este documento ha sido traducido utilizando el servicio de traducción automática [Co-op Translator](https://github.com/Azure/co-op-translator). Aunque nos esforzamos por garantizar la precisión, tenga en cuenta que las traducciones automatizadas pueden contener errores o imprecisiones. El documento original en su idioma nativo debe considerarse como la fuente autorizada. Para información crítica, se recomienda una traducción profesional realizada por humanos. No nos hacemos responsables de malentendidos o interpretaciones erróneas que puedan surgir del uso de esta traducción.
|
||||
|
|
@ -0,0 +1,478 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"collapsed": true
|
||||
},
|
||||
"source": [
|
||||
"# Implementación de un Sistema Experto de Animales\n",
|
||||
"\n",
|
||||
"Un ejemplo del [Currículo de IA para Principiantes](http://github.com/microsoft/ai-for-beginners).\n",
|
||||
"\n",
|
||||
"En este ejemplo, implementaremos un sistema basado en conocimiento sencillo para determinar un animal en función de algunas características físicas. El sistema puede representarse mediante el siguiente árbol AND-OR (esto es una parte del árbol completo, podemos añadir más reglas fácilmente):\n",
|
||||
"\n",
|
||||
"\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Nuestro propio sistema experto con inferencia hacia atrás\n",
|
||||
"\n",
|
||||
"Intentemos definir un lenguaje simple para la representación del conocimiento basado en reglas de producción. Usaremos clases de Python como palabras clave para definir reglas. Básicamente, habrá 3 tipos de clases:\n",
|
||||
"* `Ask` representa una pregunta que necesita ser hecha al usuario. Contiene el conjunto de posibles respuestas.\n",
|
||||
"* `If` representa una regla, y es simplemente una forma sintáctica de almacenar el contenido de la regla.\n",
|
||||
"* `AND`/`OR` son clases para representar las ramas AND/OR del árbol. Solo almacenan la lista de argumentos dentro. Para simplificar el código, toda la funcionalidad se define en la clase padre `Content`.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class Ask():\n",
|
||||
" def __init__(self,choices=['y','n']):\n",
|
||||
" self.choices = choices\n",
|
||||
" def ask(self):\n",
|
||||
" if max([len(x) for x in self.choices])>1:\n",
|
||||
" for i,x in enumerate(self.choices):\n",
|
||||
" print(\"{0}. {1}\".format(i,x),flush=True)\n",
|
||||
" x = int(input())\n",
|
||||
" return self.choices[x]\n",
|
||||
" else:\n",
|
||||
" print(\"/\".join(self.choices),flush=True)\n",
|
||||
" return input()\n",
|
||||
"\n",
|
||||
"class Content():\n",
|
||||
" def __init__(self,x):\n",
|
||||
" self.x=x\n",
|
||||
" \n",
|
||||
"class If(Content):\n",
|
||||
" pass\n",
|
||||
"\n",
|
||||
"class AND(Content):\n",
|
||||
" pass\n",
|
||||
"\n",
|
||||
"class OR(Content):\n",
|
||||
" pass"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"En nuestro sistema, la memoria de trabajo contendría la lista de **hechos** como **pares atributo-valor**. La base de conocimiento puede definirse como un gran diccionario que asigna acciones (nuevos hechos que deben insertarse en la memoria de trabajo) a condiciones, expresadas como expresiones AND-OR. Además, algunos hechos pueden ser `Preguntados`.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"rules = {\n",
|
||||
" 'default': Ask(['y','n']),\n",
|
||||
" 'color' : Ask(['red-brown','black and white','other']),\n",
|
||||
" 'pattern' : Ask(['dark stripes','dark spots']),\n",
|
||||
" 'mammal': If(OR(['hair','gives milk'])),\n",
|
||||
" 'carnivor': If(OR([AND(['sharp teeth','claws','forward-looking eyes']),'eats meat'])),\n",
|
||||
" 'ungulate': If(['mammal',OR(['has hooves','chews cud'])]),\n",
|
||||
" 'bird': If(OR(['feathers',AND(['flies','lies eggs'])])),\n",
|
||||
" 'animal:monkey' : If(['mammal','carnivor','color:red-brown','pattern:dark spots']),\n",
|
||||
" 'animal:tiger' : If(['mammal','carnivor','color:red-brown','pattern:dark stripes']),\n",
|
||||
" 'animal:giraffe' : If(['ungulate','long neck','long legs','pattern:dark spots']),\n",
|
||||
" 'animal:zebra' : If(['ungulate','pattern:dark stripes']),\n",
|
||||
" 'animal:ostrich' : If(['bird','long nech','color:black and white','cannot fly']),\n",
|
||||
" 'animal:pinguin' : If(['bird','swims','color:black and white','cannot fly']),\n",
|
||||
" 'animal:albatross' : If(['bird','flies well'])\n",
|
||||
"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Para realizar la inferencia hacia atrás, definiremos la clase `Knowledgebase`. Esta contendrá:\n",
|
||||
"* `memory` de trabajo: un diccionario que asigna atributos a valores.\n",
|
||||
"* `rules` de la base de conocimiento en el formato definido anteriormente.\n",
|
||||
"\n",
|
||||
"Los dos métodos principales son:\n",
|
||||
"* `get` para obtener el valor de un atributo, realizando la inferencia si es necesario. Por ejemplo, `get('color')` obtendría el valor de un espacio de color (preguntará si es necesario y almacenará el valor para uso posterior en la memoria de trabajo). Si preguntamos `get('color:blue')`, pedirá un color y luego devolverá un valor de `y`/`n` dependiendo del color.\n",
|
||||
"* `eval` realiza la inferencia propiamente dicha, es decir, recorre el árbol AND/OR, evalúa subobjetivos, etc.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class KnowledgeBase():\n",
|
||||
" def __init__(self,rules):\n",
|
||||
" self.rules = rules\n",
|
||||
" self.memory = {}\n",
|
||||
" \n",
|
||||
" def get(self,name):\n",
|
||||
" if ':' in name:\n",
|
||||
" k,v = name.split(':')\n",
|
||||
" vv = self.get(k)\n",
|
||||
" return 'y' if v==vv else 'n'\n",
|
||||
" if name in self.memory.keys():\n",
|
||||
" return self.memory[name]\n",
|
||||
" for fld in self.rules.keys():\n",
|
||||
" if fld==name or fld.startswith(name+\":\"):\n",
|
||||
" # print(\" + proving {}\".format(fld))\n",
|
||||
" value = 'y' if fld==name else fld.split(':')[1]\n",
|
||||
" res = self.eval(self.rules[fld],field=name)\n",
|
||||
" if res!='y' and res!='n' and value=='y':\n",
|
||||
" self.memory[name] = res\n",
|
||||
" return res\n",
|
||||
" if res=='y':\n",
|
||||
" self.memory[name] = value\n",
|
||||
" return value\n",
|
||||
" # field is not found, using default\n",
|
||||
" res = self.eval(self.rules['default'],field=name)\n",
|
||||
" self.memory[name]=res\n",
|
||||
" return res\n",
|
||||
" \n",
|
||||
" def eval(self,expr,field=None):\n",
|
||||
" # print(\" + eval {}\".format(expr))\n",
|
||||
" if isinstance(expr,Ask):\n",
|
||||
" print(field)\n",
|
||||
" return expr.ask()\n",
|
||||
" elif isinstance(expr,If):\n",
|
||||
" return self.eval(expr.x)\n",
|
||||
" elif isinstance(expr,AND) or isinstance(expr,list):\n",
|
||||
" expr = expr.x if isinstance(expr,AND) else expr\n",
|
||||
" for x in expr:\n",
|
||||
" if self.eval(x)=='n':\n",
|
||||
" return 'n'\n",
|
||||
" return 'y'\n",
|
||||
" elif isinstance(expr,OR):\n",
|
||||
" for x in expr.x:\n",
|
||||
" if self.eval(x)=='y':\n",
|
||||
" return 'y'\n",
|
||||
" return 'n'\n",
|
||||
" elif isinstance(expr,str):\n",
|
||||
" return self.get(expr)\n",
|
||||
" else:\n",
|
||||
" print(\"Unknown expr: {}\".format(expr))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Ahora definamos nuestra base de conocimientos sobre animales y realicemos la consulta. Ten en cuenta que esta llamada te hará preguntas. Puedes responder escribiendo `y`/`n` para preguntas de sí o no, o especificando un número (0..N) para preguntas con respuestas de opción múltiple más largas.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"hair\n",
|
||||
"y/n\n",
|
||||
"sharp teeth\n",
|
||||
"y/n\n",
|
||||
"claws\n",
|
||||
"y/n\n",
|
||||
"forward-looking eyes\n",
|
||||
"y/n\n",
|
||||
"color\n",
|
||||
"0. red-brown\n",
|
||||
"1. black and white\n",
|
||||
"2. other\n",
|
||||
"has hooves\n",
|
||||
"y/n\n",
|
||||
"long neck\n",
|
||||
"y/n\n",
|
||||
"long legs\n",
|
||||
"y/n\n",
|
||||
"pattern\n",
|
||||
"0. dark stripes\n",
|
||||
"1. dark spots\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"'giraffe'"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"kb = KnowledgeBase(rules)\n",
|
||||
"kb.get('animal')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Usando PyKnow para Inferencia Hacia Adelante\n",
|
||||
"\n",
|
||||
"En el siguiente ejemplo, intentaremos implementar inferencia hacia adelante utilizando una de las bibliotecas para representación del conocimiento, [PyKnow](https://github.com/buguroo/pyknow/). **PyKnow** es una biblioteca para crear sistemas de inferencia hacia adelante en Python, diseñada para ser similar al clásico sistema antiguo [CLIPS](http://www.clipsrules.net/index.html).\n",
|
||||
"\n",
|
||||
"También podríamos haber implementado el encadenamiento hacia adelante nosotros mismos sin muchos problemas, pero las implementaciones ingenuas generalmente no son muy eficientes. Para un emparejamiento de reglas más efectivo, se utiliza un algoritmo especial llamado [Rete](https://en.wikipedia.org/wiki/Rete_algorithm).\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Collecting git+https://github.com/buguroo/pyknow/\n",
|
||||
" Cloning https://github.com/buguroo/pyknow/ to /tmp/pip-req-build-3cqeulyl\n",
|
||||
" Running command git clone --filter=blob:none --quiet https://github.com/buguroo/pyknow/ /tmp/pip-req-build-3cqeulyl\n",
|
||||
" Resolved https://github.com/buguroo/pyknow/ to commit 48818336f2e9a126f1964f2d8dc22d37ff800fe8\n",
|
||||
" Preparing metadata (setup.py) ... \u001b[?25ldone\n",
|
||||
"\u001b[?25hCollecting frozendict==1.2\n",
|
||||
" Using cached frozendict-1.2.tar.gz (2.6 kB)\n",
|
||||
" Preparing metadata (setup.py) ... \u001b[?25ldone\n",
|
||||
"\u001b[?25hCollecting schema==0.6.7\n",
|
||||
" Using cached schema-0.6.7-py2.py3-none-any.whl (14 kB)\n",
|
||||
"Building wheels for collected packages: pyknow, frozendict\n",
|
||||
" Building wheel for pyknow (setup.py) ... \u001b[?25ldone\n",
|
||||
"\u001b[?25h Created wheel for pyknow: filename=pyknow-1.7.0-py3-none-any.whl size=34228 sha256=b7de5b09292c4007667c72f69b98d5a1b5f7324ff15f9dd8e077c3d5f7aade42\n",
|
||||
" Stored in directory: /tmp/pip-ephem-wheel-cache-k7jpave7/wheels/81/1a/d3/f6c15dbe1955598a37755215f2a10449e7418500d7bd4b9508\n",
|
||||
" Building wheel for frozendict (setup.py) ... \u001b[?25ldone\n",
|
||||
"\u001b[?25h Created wheel for frozendict: filename=frozendict-1.2-py3-none-any.whl size=3148 sha256=2863d55c240d2409cddf05ccfe600591f8478681549fc97555c47c90dc6bb160\n",
|
||||
" Stored in directory: /home/rg/.cache/pip/wheels/49/ac/f8/cb8120244e710bdb479c86198b03c7b08c3c2d3d2bf448fd6e\n",
|
||||
"Successfully built pyknow frozendict\n",
|
||||
"Installing collected packages: schema, frozendict, pyknow\n",
|
||||
"Successfully installed frozendict-1.2 pyknow-1.7.0 schema-0.6.7\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import sys\n",
|
||||
"!{sys.executable} -m pip install git+https://github.com/buguroo/pyknow/"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from pyknow import *\n",
|
||||
"#import pyknow"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Definiremos nuestro sistema como una clase que hereda de `KnowledgeEngine`. Cada regla se define mediante una función separada con la anotación `@Rule`, que especifica cuándo debe activarse la regla. Dentro de la regla, podemos agregar nuevos hechos utilizando la función `declare`, y al agregar esos hechos, se activarán más reglas mediante el motor de inferencia hacia adelante.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class Animals(KnowledgeEngine):\n",
|
||||
" @Rule(OR(\n",
|
||||
" AND(Fact('sharp teeth'),Fact('claws'),Fact('forward looking eyes')),\n",
|
||||
" Fact('eats meat')))\n",
|
||||
" def cornivor(self):\n",
|
||||
" self.declare(Fact('carnivor'))\n",
|
||||
" \n",
|
||||
" @Rule(OR(Fact('hair'),Fact('gives milk')))\n",
|
||||
" def mammal(self):\n",
|
||||
" self.declare(Fact('mammal'))\n",
|
||||
"\n",
|
||||
" @Rule(Fact('mammal'),\n",
|
||||
" OR(Fact('has hooves'),Fact('chews cud')))\n",
|
||||
" def hooves(self):\n",
|
||||
" self.declare('ungulate')\n",
|
||||
" \n",
|
||||
" @Rule(OR(Fact('feathers'),AND(Fact('flies'),Fact('lays eggs'))))\n",
|
||||
" def bird(self):\n",
|
||||
" self.declare('bird')\n",
|
||||
" \n",
|
||||
" @Rule(Fact('mammal'),Fact('carnivor'),\n",
|
||||
" Fact(color='red-brown'),\n",
|
||||
" Fact(pattern='dark spots'))\n",
|
||||
" def monkey(self):\n",
|
||||
" self.declare(Fact(animal='monkey'))\n",
|
||||
"\n",
|
||||
" @Rule(Fact('mammal'),Fact('carnivor'),\n",
|
||||
" Fact(color='red-brown'),\n",
|
||||
" Fact(pattern='dark stripes'))\n",
|
||||
" def tiger(self):\n",
|
||||
" self.declare(Fact(animal='tiger'))\n",
|
||||
"\n",
|
||||
" @Rule(Fact('ungulate'),\n",
|
||||
" Fact('long neck'),\n",
|
||||
" Fact('long legs'),\n",
|
||||
" Fact(pattern='dark spots'))\n",
|
||||
" def giraffe(self):\n",
|
||||
" self.declare(Fact(animal='giraffe'))\n",
|
||||
"\n",
|
||||
" @Rule(Fact('ungulate'),\n",
|
||||
" Fact(pattern='dark stripes'))\n",
|
||||
" def zebra(self):\n",
|
||||
" self.declare(Fact(animal='zebra'))\n",
|
||||
"\n",
|
||||
" @Rule(Fact('bird'),\n",
|
||||
" Fact('long neck'),\n",
|
||||
" Fact('cannot fly'),\n",
|
||||
" Fact(color='black and white'))\n",
|
||||
" def straus(self):\n",
|
||||
" self.declare(Fact(animal='ostrich'))\n",
|
||||
"\n",
|
||||
" @Rule(Fact('bird'),\n",
|
||||
" Fact('swims'),\n",
|
||||
" Fact('cannot fly'),\n",
|
||||
" Fact(color='black and white'))\n",
|
||||
" def pinguin(self):\n",
|
||||
" self.declare(Fact(animal='pinguin'))\n",
|
||||
"\n",
|
||||
" @Rule(Fact('bird'),\n",
|
||||
" Fact('flies well'))\n",
|
||||
" def albatros(self):\n",
|
||||
" self.declare(Fact(animal='albatross'))\n",
|
||||
" \n",
|
||||
" @Rule(Fact(animal=MATCH.a))\n",
|
||||
" def print_result(self,a):\n",
|
||||
" print('Animal is {}'.format(a))\n",
|
||||
" \n",
|
||||
" def factz(self,l):\n",
|
||||
" for x in l:\n",
|
||||
" self.declare(x)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Una vez que hemos definido una base de conocimientos, llenamos nuestra memoria de trabajo con algunos hechos iniciales y luego llamamos al método `run()` para realizar la inferencia. Puedes ver como resultado que nuevos hechos inferidos se añaden a la memoria de trabajo, incluyendo el hecho final sobre el animal (si configuramos todos los hechos iniciales correctamente).\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Animal is tiger\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"FactList([(0, InitialFact()),\n",
|
||||
" (1, Fact(color='red-brown')),\n",
|
||||
" (2, Fact(pattern='dark stripes')),\n",
|
||||
" (3, Fact('sharp teeth')),\n",
|
||||
" (4, Fact('claws')),\n",
|
||||
" (5, Fact('forward looking eyes')),\n",
|
||||
" (6, Fact('gives milk')),\n",
|
||||
" (7, Fact('mammal')),\n",
|
||||
" (8, Fact('carnivor')),\n",
|
||||
" (9, Fact(animal='tiger'))])"
|
||||
]
|
||||
},
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"ex1 = Animals()\n",
|
||||
"ex1.reset()\n",
|
||||
"ex1.factz([\n",
|
||||
" Fact(color='red-brown'),\n",
|
||||
" Fact(pattern='dark stripes'),\n",
|
||||
" Fact('sharp teeth'),\n",
|
||||
" Fact('claws'),\n",
|
||||
" Fact('forward looking eyes'),\n",
|
||||
" Fact('gives milk')])\n",
|
||||
"ex1.run()\n",
|
||||
"ex1.facts"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Descargo de responsabilidad**: \nEste documento ha sido traducido utilizando el servicio de traducción automática [Co-op Translator](https://github.com/Azure/co-op-translator). Si bien nos esforzamos por lograr precisión, tenga en cuenta que las traducciones automáticas pueden contener errores o imprecisiones. El documento original en su idioma nativo debe considerarse como la fuente autorizada. Para información crítica, se recomienda una traducción profesional realizada por humanos. No nos hacemos responsables de malentendidos o interpretaciones erróneas que puedan surgir del uso de esta traducción.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.7.4 64-bit (conda)",
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5"
|
||||
}
|
||||
},
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.11.2"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "ab2bd97b0453415b89a469284609a8ce",
|
||||
"translation_date": "2025-08-31T16:26:55+00:00",
|
||||
"source_file": "lessons/2-Symbolic/Animals.ipynb",
|
||||
"language_code": "es"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
|
|
@ -0,0 +1,595 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"collapsed": true
|
||||
},
|
||||
"source": [
|
||||
"# Ontología de Relaciones Familiares\n",
|
||||
"\n",
|
||||
"Este ejemplo forma parte del [Currículo de IA para Principiantes](http://github.com/microsoft/ai-for-beginners) y está inspirado en [esta publicación de blog](https://habr.com/post/270857/).\n",
|
||||
"\n",
|
||||
"Siempre me resulta difícil recordar las diferentes relaciones entre las personas de una familia. En este ejemplo, tomaremos una ontología que define las relaciones familiares y el árbol genealógico real, y mostraremos cómo podemos realizar inferencias automáticas para encontrar a todos los parientes.\n",
|
||||
"\n",
|
||||
"### Obteniendo el Árbol Genealógico\n",
|
||||
"\n",
|
||||
"Como ejemplo, tomaremos el árbol genealógico de la [Familia Romanov, los zares](https://en.wikipedia.org/wiki/House_of_Romanov). El formato más común para describir relaciones familiares es [GEDCOM](https://en.wikipedia.org/wiki/GEDCOM). Utilizaremos el árbol genealógico de la familia Romanov en formato GEDCOM:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"0 HEAD\n",
|
||||
"1 CHAR UTF8\n",
|
||||
"1 GEDC\n",
|
||||
"2 VERS 5.5\n",
|
||||
"0 @0@ INDI\n",
|
||||
"1 NAME Mihail Fedorovich /Romanov/\n",
|
||||
"1 SEX M\n",
|
||||
"1 BIRT\n",
|
||||
"2 DATE 1613\n",
|
||||
"1 DEAT \n",
|
||||
"2 DATE 1645\n",
|
||||
"1 FAMS @41@\n",
|
||||
"0 @1@ INDI\n",
|
||||
"1 NAME Evdokija Lukjanovna /Streshneva/\n",
|
||||
"1 SEX F\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"!head -15 data/tsars.ged"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Para usar el archivo GEDCOM, podemos usar la biblioteca `python-gedcom`:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Collecting python-gedcom\n",
|
||||
" Downloading python_gedcom-1.0.0-py2.py3-none-any.whl (35 kB)\n",
|
||||
"Installing collected packages: python-gedcom\n",
|
||||
"Successfully installed python-gedcom-1.0.0\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import sys\n",
|
||||
"!{sys.executable} -m pip install python-gedcom"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Esta biblioteca elimina algunos de los problemas técnicos con el análisis de archivos, pero aún nos brinda un acceso bastante de bajo nivel a todos los individuos y familias en el árbol. Aquí se muestra cómo podemos analizar el archivo y mostrar la lista de todos los individuos:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from gedcom.parser import Parser\n",
|
||||
"from gedcom.element.individual import IndividualElement\n",
|
||||
"from gedcom.element.family import FamilyElement\n",
|
||||
"g = Parser()\n",
|
||||
"g.parse_file('data/tsars.ged')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {
|
||||
"scrolled": true,
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[('@0@', ('Mihail Fedorovich', 'Romanov')),\n",
|
||||
" ('@1@', ('Evdokija Lukjanovna', 'Streshneva')),\n",
|
||||
" ('@2@', ('Aleksej Mihajlovich', 'Romanov')),\n",
|
||||
" ('@3@', ('Marija Ilinichna', 'Miloslavskaja')),\n",
|
||||
" ('@4@', ('Natalja Kirillovna', 'Naryshkina')),\n",
|
||||
" ('@5@', ('Marfa Matveevna', 'Apraksina')),\n",
|
||||
" ('@6@', ('Fedor Alekseevich', 'Romanov')),\n",
|
||||
" ('@7@', ('Sofja Aleksevna', 'Romanova')),\n",
|
||||
" ('@8@', ('Ivan V Alekseevich', 'Romanov')),\n",
|
||||
" ('@9@', ('Praskovja Fedorovna', 'Saltykova')),\n",
|
||||
" ('@10@', ('Ekaterina Ivanovna', 'Romanova')),\n",
|
||||
" ('@11@', ('Anna Ivanovna', 'Romanova')),\n",
|
||||
" ('@12@', ('Fridrih Vilgelm', 'Kurlandskij')),\n",
|
||||
" ('@13@', ('Karl Leopold', 'Meklenburg-Shverinskij')),\n",
|
||||
" ('@14@', ('Anna Leopoldovna', 'Meklenburg-Shverinskaja')),\n",
|
||||
" ('@15@', ('Anton Ulrih', 'Braunshvejg-Volfenbjuttelskij')),\n",
|
||||
" ('@16@', ('Ivan VI Antonovich', 'Braunshvejg-Volfenbjuttelskij')),\n",
|
||||
" ('@17@', ('Petr I Alekseevich', 'Romanov')),\n",
|
||||
" ('@18@', ('Evdokija Fedorovna', 'Lopuhina')),\n",
|
||||
" ('@19@', ('Ekaterina I Alekseevna', 'Mihajlova')),\n",
|
||||
" ('@20@', ('Aleksej Petrovich', 'Romanov')),\n",
|
||||
" ('@21@', ('Sharlotta Kristina', 'Braunshvejg-Volfenbjuttelskaja')),\n",
|
||||
" ('@22@', ('Petr II Alekseevich', 'Romanov')),\n",
|
||||
" ('@23@', ('Anna Petrovna', 'Romanova')),\n",
|
||||
" ('@24@', ('Elizaveta Petrovna', 'Romanova')),\n",
|
||||
" ('@25@', ('Karl Fridrih', 'Golshtejn-Gottorpskij')),\n",
|
||||
" ('@26@', ('Petr III Fedorovich', 'Romanov')),\n",
|
||||
" ('@27@', ('Ekaterina II', 'Alekseevna')),\n",
|
||||
" ('@28@', ('Pavel I Petrovich', 'Romanov')),\n",
|
||||
" ('@29@', ('Natalja Alekseevna', 'Gessen-Darmshtadskaja')),\n",
|
||||
" ('@30@', ('Marija Fedorovna', 'Vjurtembergskaja')),\n",
|
||||
" ('@31@', ('Aleksandr I Pavlovich', 'Romanov')),\n",
|
||||
" ('@32@', ('Elizaveta Alekseevna', 'Baden-Durlahskaja')),\n",
|
||||
" ('@33@', ('Nikolaj I Pavlovich', 'Romanov')),\n",
|
||||
" ('@34@', ('Aleksandra Fedorovna', 'Prusskaja')),\n",
|
||||
" ('@35@', ('Aleksandr II Nikolaevich', 'Romanov')),\n",
|
||||
" ('@36@', ('Marija Aleksandrovna', 'Gessenskaja')),\n",
|
||||
" ('@37@', ('Aleksandr III Aleksandrovich', 'Romanov')),\n",
|
||||
" ('@38@', ('Marija Fedorovna', 'Datskaja')),\n",
|
||||
" ('@39@', ('Nikolaj II Aleksandrovich', 'Romanov')),\n",
|
||||
" ('@40@', ('Aleksandra Fedorovna', 'Gessenskaja'))]"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"d = g.get_element_dictionary()\n",
|
||||
"[ (k,v.get_name()) for k,v in d.items() if isinstance(v,IndividualElement)]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Aquí se explica cómo podemos obtener información sobre las familias. Ten en cuenta que esto nos da una lista de **identificadores**, y necesitamos convertirlos a nombres si queremos más claridad:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[('@41@', ['@0@', '@1@', '@2@']),\n",
|
||||
" ('@42@', ['@2@', '@3@', '@6@', '@7@', '@8@']),\n",
|
||||
" ('@43@', ['@8@', '@9@', '@10@', '@11@']),\n",
|
||||
" ('@44@', ['@13@', '@10@', '@14@']),\n",
|
||||
" ('@45@', ['@15@', '@14@', '@16@']),\n",
|
||||
" ('@46@', ['@2@', '@4@', '@17@']),\n",
|
||||
" ('@47@', ['@17@', '@18@', '@20@']),\n",
|
||||
" ('@48@', ['@20@', '@21@', '@22@']),\n",
|
||||
" ('@49@', ['@17@', '@19@', '@23@', '@24@']),\n",
|
||||
" ('@50@', ['@25@', '@23@', '@26@']),\n",
|
||||
" ('@51@', ['@26@', '@27@', '@28@']),\n",
|
||||
" ('@52@', ['@28@', '@30@', '@31@', '@33@']),\n",
|
||||
" ('@53@', ['@33@', '@34@', '@35@']),\n",
|
||||
" ('@54@', ['@35@', '@36@', '@37@']),\n",
|
||||
" ('@55@', ['@37@', '@38@', '@39@'])]"
|
||||
]
|
||||
},
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"d = g.get_element_dictionary()\n",
|
||||
"[ (k,[x.get_value() for x in v.get_child_elements()]) for k,v in d.items() if isinstance(v,FamilyElement)]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Obteniendo la Ontología Familiar\n",
|
||||
"\n",
|
||||
"A continuación, echemos un vistazo a la [ontología familiar](https://raw.githubusercontent.com/blokhin/genealogical-trees/master/data/header.ttl) definida como un conjunto de tripletas de la Web Semántica. Esta ontología define relaciones como `isUncleOf`, `isCousinOf` y muchas otras. Todas estas relaciones se definen en términos de predicados básicos como `isMotherOf`, `isFatherOf`, `isBrotherOf` y `isSisterOf`. Utilizaremos razonamiento automático para deducir todas las demás relaciones usando la ontología.\n",
|
||||
"\n",
|
||||
"Aquí tienes un ejemplo de la definición de la propiedad `isAuntOf`, que se define como una composición de `isSisterOf` y `isParentOf` (*Una tía es la hermana de uno de los padres*).\n",
|
||||
"\n",
|
||||
"```\n",
|
||||
"fhkb:isAuntOf a owl:ObjectProperty ;\n",
|
||||
" rdfs:domain fhkb:Woman ;\n",
|
||||
" rdfs:range fhkb:Person ;\n",
|
||||
" owl:propertyChainAxiom ( fhkb:isSisterOf fhkb:isParentOf ) .\n",
|
||||
"```\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"@prefix fhkb: <http://www.example.com/genealogy.owl#> .\n",
|
||||
"@prefix owl: <http://www.w3.org/2002/07/owl#> .\n",
|
||||
"@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n",
|
||||
"@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .\n",
|
||||
"@prefix xml: <http://www.w3.org/XML/1998/namespace> .\n",
|
||||
"@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\n",
|
||||
"\n",
|
||||
"<http://www.example.com/genealogy.owl#> a owl:Ontology .\n",
|
||||
"\n",
|
||||
"fhkb:DomainEntity a owl:Class .\n",
|
||||
"\n",
|
||||
"fhkb:Man a owl:Class ;\n",
|
||||
" owl:equivalentClass [ a owl:Class ;\n",
|
||||
" owl:intersectionOf ( fhkb:Person [ a owl:Restriction ;\n",
|
||||
" owl:onProperty fhkb:hasSex ;\n",
|
||||
" owl:someValuesFrom fhkb:Male ] ) ] .\n",
|
||||
"\n",
|
||||
"fhkb:Woman a owl:Class ;\n",
|
||||
" owl:equivalentClass [ a owl:Class ;\n",
|
||||
" owl:intersectionOf ( fhkb:Person [ a owl:Restriction ;\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"!head -20 data/onto.ttl"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Construcción de una ontología para inferencia\n",
|
||||
"\n",
|
||||
"Para simplificar, crearemos un archivo de ontología que incluirá las reglas originales de la ontología familiar y los hechos sobre individuos de nuestro archivo GEDCOM. Revisaremos el archivo GEDCOM, extraeremos información sobre familias e individuos y la convertiremos en tríadas.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!cp data/onto.ttl .\n",
|
||||
"\n",
|
||||
"gedcom_dict = g.get_element_dictionary()\n",
|
||||
"individuals, marriages = {}, {}\n",
|
||||
"\n",
|
||||
"def term2id(el):\n",
|
||||
" return \"i\" + el.get_pointer().replace('@', '').lower()\n",
|
||||
"\n",
|
||||
"out = open(\"onto.ttl\",\"a\")\n",
|
||||
"\n",
|
||||
"for k, v in gedcom_dict.items():\n",
|
||||
" if isinstance(v,IndividualElement):\n",
|
||||
" children, siblings = set(), set()\n",
|
||||
" idx = term2id(v)\n",
|
||||
"\n",
|
||||
" title = v.get_name()[0] + \" \" + v.get_name()[1]\n",
|
||||
" title = title.replace('\"', '').replace('[', '').replace(']', '').replace('(', '').replace(')', '').strip()\n",
|
||||
"\n",
|
||||
" own_families = g.get_families(v, 'FAMS')\n",
|
||||
" for fam in own_families:\n",
|
||||
" children |= set(term2id(i) for i in g.get_family_members(fam, \"CHIL\"))\n",
|
||||
"\n",
|
||||
" parent_families = g.get_families(v, 'FAMC')\n",
|
||||
" if len(parent_families):\n",
|
||||
" for member in g.get_family_members(parent_families[0], \"CHIL\"): # NB adoptive families i.e len(parent_families)>1 are not considered (TODO?)\n",
|
||||
" if member.get_pointer() == v.get_pointer():\n",
|
||||
" continue\n",
|
||||
" siblings.add(term2id(member))\n",
|
||||
"\n",
|
||||
" if idx in individuals:\n",
|
||||
" children |= individuals[idx].get('children', set())\n",
|
||||
" siblings |= individuals[idx].get('siblings', set())\n",
|
||||
" individuals[idx] = {'sex': v.get_gender().lower(), 'children': children, 'siblings': siblings, 'title': title}\n",
|
||||
"\n",
|
||||
" elif isinstance(v,FamilyElement):\n",
|
||||
" wife, husb, children = None, None, set()\n",
|
||||
" children = set(term2id(i) for i in g.get_family_members(v, \"CHIL\"))\n",
|
||||
"\n",
|
||||
" try:\n",
|
||||
" wife = g.get_family_members(v, \"WIFE\")[0]\n",
|
||||
" wife = term2id(wife)\n",
|
||||
" if wife in individuals: individuals[wife]['children'] |= children\n",
|
||||
" else: individuals[wife] = {'children': children}\n",
|
||||
" except IndexError: pass\n",
|
||||
" try:\n",
|
||||
" husb = g.get_family_members(v, \"HUSB\")[0]\n",
|
||||
" husb = term2id(husb)\n",
|
||||
" if husb in individuals: individuals[husb]['children'] |= children\n",
|
||||
" else: individuals[husb] = {'children': children}\n",
|
||||
" except IndexError: pass\n",
|
||||
"\n",
|
||||
" if wife and husb: marriages[wife + husb] = (term2id(v), wife, husb)\n",
|
||||
"\n",
|
||||
"for idx, val in individuals.items():\n",
|
||||
" added_terms = ''\n",
|
||||
" if val['sex'] == 'f':\n",
|
||||
" parent_predicate, sibl_predicate = \"isMotherOf\", \"isSisterOf\"\n",
|
||||
" else:\n",
|
||||
" parent_predicate, sibl_predicate = \"isFatherOf\", \"isBrotherOf\"\n",
|
||||
" if len(val['children']):\n",
|
||||
" added_terms += \" ;\\n fhkb:\" + parent_predicate + \" \" + \", \".join([\"fhkb:\" + i for i in val['children']])\n",
|
||||
" if len(val['siblings']):\n",
|
||||
" added_terms += \" ;\\n fhkb:\" + sibl_predicate + \" \" + \", \".join([\"fhkb:\" + i for i in val['siblings']])\n",
|
||||
" out.write(\"fhkb:%s a owl:NamedIndividual, owl:Thing%s ;\\n rdfs:label \\\"%s\\\" .\\n\" % (idx, added_terms, val['title']))\n",
|
||||
"\n",
|
||||
"for k, v in marriages.items():\n",
|
||||
" out.write(\"fhkb:%s a owl:NamedIndividual, owl:Thing ;\\n fhkb:hasFemalePartner fhkb:%s ;\\n fhkb:hasMalePartner fhkb:%s .\\n\" % v)\n",
|
||||
"\n",
|
||||
"out.write(\"[] a owl:AllDifferent ;\\n owl:distinctMembers (\")\n",
|
||||
"for idx in individuals.keys():\n",
|
||||
" out.write(\" fhkb:\" + idx)\n",
|
||||
"for k, v in marriages.items():\n",
|
||||
" out.write(\" fhkb:\" + v[0])\n",
|
||||
"out.write(\" ) .\")\n",
|
||||
"out.close()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" fhkb:hasFemalePartner fhkb:i34 ;\n",
|
||||
" fhkb:hasMalePartner fhkb:i33 .\n",
|
||||
"fhkb:i54 a owl:NamedIndividual, owl:Thing ;\n",
|
||||
" fhkb:hasFemalePartner fhkb:i36 ;\n",
|
||||
" fhkb:hasMalePartner fhkb:i35 .\n",
|
||||
"fhkb:i55 a owl:NamedIndividual, owl:Thing ;\n",
|
||||
" fhkb:hasFemalePartner fhkb:i38 ;\n",
|
||||
" fhkb:hasMalePartner fhkb:i37 .\n",
|
||||
"[] a owl:AllDifferent ;\n",
|
||||
" owl:distinctMembers ( fhkb:i0 fhkb:i1 fhkb:i2 fhkb:i3 fhkb:i4 fhkb:i5 fhkb:i6 fhkb:i7 fhkb:i8 fhkb:i9 fhkb:i10 fhkb:i11 fhkb:i12 fhkb:i13 fhkb:i14 fhkb:i15 fhkb:i16 fhkb:i17 fhkb:i18 fhkb:i19 fhkb:i20 fhkb:i21 fhkb:i22 fhkb:i23 fhkb:i24 fhkb:i25 fhkb:i26 fhkb:i27 fhkb:i28 fhkb:i29 fhkb:i30 fhkb:i31 fhkb:i32 fhkb:i33 fhkb:i34 fhkb:i35 fhkb:i36 fhkb:i37 fhkb:i38 fhkb:i39 fhkb:i40 fhkb:i41 fhkb:i42 fhkb:i43 fhkb:i44 fhkb:i45 fhkb:i46 fhkb:i47 fhkb:i48 fhkb:i49 fhkb:i50 fhkb:i51 fhkb:i52 fhkb:i53 fhkb:i54 fhkb:i55 ) ."
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"!tail onto.ttl"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Realizando Inferencia\n",
|
||||
"\n",
|
||||
"Ahora queremos poder usar esta ontología para realizar inferencias y consultas. Utilizaremos [RDFLib](https://github.com/RDFLib), una biblioteca para leer gráficos RDF en diferentes formatos, realizar consultas, etc.\n",
|
||||
"\n",
|
||||
"Para la inferencia lógica, utilizaremos la biblioteca [OWL-RL](https://github.com/RDFLib/OWL-RL), que nos permite construir el **Cierre** del gráfico RDF, es decir, agregar todos los conceptos y relaciones posibles que se puedan inferir.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Requirement already satisfied: rdflib in /home/rg/anaconda3/envs/ai4beg/lib/python3.11/site-packages (6.3.2)\n",
|
||||
"Requirement already satisfied: isodate<0.7.0,>=0.6.0 in /home/rg/anaconda3/envs/ai4beg/lib/python3.11/site-packages (from rdflib) (0.6.1)\n",
|
||||
"Requirement already satisfied: pyparsing<4,>=2.1.0 in /home/rg/anaconda3/envs/ai4beg/lib/python3.11/site-packages (from rdflib) (3.0.9)\n",
|
||||
"Requirement already satisfied: six in /home/rg/anaconda3/envs/ai4beg/lib/python3.11/site-packages (from isodate<0.7.0,>=0.6.0->rdflib) (1.16.0)\n",
|
||||
"Collecting git+https://github.com/RDFLib/OWL-RL.git\n",
|
||||
" Cloning https://github.com/RDFLib/OWL-RL.git to /tmp/pip-req-build-lbfzwi3m\n",
|
||||
" Running command git clone --filter=blob:none --quiet https://github.com/RDFLib/OWL-RL.git /tmp/pip-req-build-lbfzwi3m\n",
|
||||
" Resolved https://github.com/RDFLib/OWL-RL.git to commit a77e1791b88b54aace609bc6000aac14c7add4ff\n",
|
||||
" Preparing metadata (setup.py) ... \u001b[?25ldone\n",
|
||||
"\u001b[?25hRequirement already satisfied: rdflib>=6.0.2 in /home/rg/anaconda3/envs/ai4beg/lib/python3.11/site-packages (from owlrl==6.0.2) (6.3.2)\n",
|
||||
"Requirement already satisfied: isodate<0.7.0,>=0.6.0 in /home/rg/anaconda3/envs/ai4beg/lib/python3.11/site-packages (from rdflib>=6.0.2->owlrl==6.0.2) (0.6.1)\n",
|
||||
"Requirement already satisfied: pyparsing<4,>=2.1.0 in /home/rg/anaconda3/envs/ai4beg/lib/python3.11/site-packages (from rdflib>=6.0.2->owlrl==6.0.2) (3.0.9)\n",
|
||||
"Requirement already satisfied: six in /home/rg/anaconda3/envs/ai4beg/lib/python3.11/site-packages (from isodate<0.7.0,>=0.6.0->rdflib>=6.0.2->owlrl==6.0.2) (1.16.0)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"!{sys.executable} -m pip install rdflib\n",
|
||||
"!{sys.executable} -m pip install git+https://github.com/RDFLib/OWL-RL.git"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Abramos el archivo de ontología y veamos cuántos tríos contiene:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Triplets found:669\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import rdflib\n",
|
||||
"from owlrl import DeductiveClosure, OWLRL_Extension\n",
|
||||
"\n",
|
||||
"g = rdflib.Graph()\n",
|
||||
"g.parse(\"onto.ttl\", format=\"turtle\")\n",
|
||||
"\n",
|
||||
"print(\"Triplets found:%d\" % len(g))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Ahora construyamos el cierre y veamos cómo aumenta el número de tríos:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Triplets after inference:4246\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"DeductiveClosure(OWLRL_Extension).expand(g)\n",
|
||||
"print(\"Triplets after inference:%d\" % len(g))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Consultar sobre Parientes\n",
|
||||
"\n",
|
||||
"Ahora podemos consultar el grafo para ver las diferentes relaciones entre las personas. Podemos usar el lenguaje **SPARQL** junto con el método `query`. En nuestro caso, veamos todos los **tíos** en nuestro árbol genealógico:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Fedor Alekseevich Romanov is uncle of Ekaterina Ivanovna Romanova\n",
|
||||
"Aleksandr I Pavlovich Romanov is uncle of Aleksandr II Nikolaevich Romanov\n",
|
||||
"Fedor Alekseevich Romanov is uncle of Anna Ivanovna Romanova\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"qres = g.query(\n",
|
||||
" \"\"\"SELECT DISTINCT ?aname ?bname\n",
|
||||
" WHERE {\n",
|
||||
" ?a fhkb:isUncleOf ?b .\n",
|
||||
" ?a rdfs:label ?aname .\n",
|
||||
" ?b rdfs:label ?bname .\n",
|
||||
" }\"\"\")\n",
|
||||
"\n",
|
||||
"for row in qres:\n",
|
||||
" print(\"%s is uncle of %s\" % row)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Siéntete libre de experimentar con diferentes relaciones familiares. Por ejemplo, puedes echar un vistazo a la relación `isAncestorOf`, que define de manera recurrente todos los ancestros de una persona dada.\n",
|
||||
"\n",
|
||||
"Finalmente, ¡vamos a limpiar!\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!rm onto.ttl"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Descargo de responsabilidad**: \nEste documento ha sido traducido utilizando el servicio de traducción automática [Co-op Translator](https://github.com/Azure/co-op-translator). Si bien nos esforzamos por lograr precisión, tenga en cuenta que las traducciones automáticas pueden contener errores o imprecisiones. El documento original en su idioma nativo debe considerarse como la fuente autorizada. Para información crítica, se recomienda una traducción profesional realizada por humanos. No nos hacemos responsables de malentendidos o interpretaciones erróneas que puedan surgir del uso de esta traducción.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.6",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.11.2"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "6537d5597320e27b6052b4377b8ff8bb",
|
||||
"translation_date": "2025-08-31T16:24:53+00:00",
|
||||
"source_file": "lessons/2-Symbolic/FamilyOntology.ipynb",
|
||||
"language_code": "es"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
|
|
@ -0,0 +1,548 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"collapsed": true
|
||||
},
|
||||
"source": [
|
||||
"## Microsoft Concept Graph\n",
|
||||
"\n",
|
||||
"[Microsoft Concept Graph](https://concept.research.microsoft.com/) es una gran taxonomía de términos extraídos de internet, con relaciones de tipo `es-un` entre conceptos.\n",
|
||||
"\n",
|
||||
"Context Graph está disponible en dos formas:\n",
|
||||
" * Archivo de texto grande para descargar\n",
|
||||
" * API REST\n",
|
||||
"\n",
|
||||
"Estadísticas:\n",
|
||||
" * 5401933 conceptos únicos,\n",
|
||||
" * 12551613 instancias únicas\n",
|
||||
" * 87603947 relaciones de tipo `es-un`\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Uso del Servicio Web\n",
|
||||
"\n",
|
||||
"El servicio web ofrece diferentes llamadas para estimar la probabilidad de que un concepto pertenezca a distintos grupos. Más información está disponible [aquí](https://concept.research.microsoft.com/Home/Api). \n",
|
||||
"Aquí tienes un ejemplo de URL para realizar una llamada: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"{'company': 0.6105356614382954,\n",
|
||||
" 'vendor': 0.08858636677518003,\n",
|
||||
" 'client': 0.048239124001183784,\n",
|
||||
" 'firm': 0.045476965571668145,\n",
|
||||
" 'large company': 0.043109401203511886,\n",
|
||||
" 'organization': 0.043010752688172046,\n",
|
||||
" 'corporation': 0.035908059583703265,\n",
|
||||
" 'brand': 0.03383644076156654,\n",
|
||||
" 'software company': 0.027522935779816515,\n",
|
||||
" 'technology company': 0.023774292196902438}"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import urllib\n",
|
||||
"import json\n",
|
||||
"import ssl\n",
|
||||
"\n",
|
||||
"def http(x):\n",
|
||||
" ssl._create_default_https_context = ssl._create_unverified_context\n",
|
||||
" response = urllib.request.urlopen(x)\n",
|
||||
" data = response.read()\n",
|
||||
" return data.decode('utf-8')\n",
|
||||
"\n",
|
||||
"def query(x):\n",
|
||||
" return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n",
|
||||
"\n",
|
||||
"query('microsoft')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Intentemos categorizar los títulos de noticias utilizando conceptos principales. Para obtener los títulos de noticias, utilizaremos el servicio [NewsApi.org](http://newsapi.org). Necesitas obtener tu propia clave de API para usar el servicio: ve al sitio web y regístrate en el plan de desarrollador gratuito.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 20,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"newsapi_key = '<your API key here>'\n",
|
||||
"def get_news(country='us'):\n",
|
||||
" res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n",
|
||||
" return res['articles']\n",
|
||||
"\n",
|
||||
"all_titles = [x['title'] for x in get_news('us')+get_news('gb')]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 21,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n",
|
||||
" 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n",
|
||||
" 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n",
|
||||
" 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n",
|
||||
" \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n",
|
||||
" '2 teachers killed at Swedish high school, student arrested - ABC News',\n",
|
||||
" 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n",
|
||||
" 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n",
|
||||
" '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n",
|
||||
" 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n",
|
||||
" \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n",
|
||||
" \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n",
|
||||
" \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n",
|
||||
" 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n",
|
||||
" 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n",
|
||||
" 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n",
|
||||
" 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n",
|
||||
" 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n",
|
||||
" \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n",
|
||||
" 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n",
|
||||
" \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n",
|
||||
" 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n",
|
||||
" 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n",
|
||||
" 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n",
|
||||
" 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n",
|
||||
" \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n",
|
||||
" 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n",
|
||||
" \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n",
|
||||
" 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n",
|
||||
" 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n",
|
||||
" \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n",
|
||||
" '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n",
|
||||
" 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n",
|
||||
" 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n",
|
||||
" 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n",
|
||||
" 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n",
|
||||
" 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n",
|
||||
" 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n",
|
||||
" \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n",
|
||||
" 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']"
|
||||
]
|
||||
},
|
||||
"execution_count": 21,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"all_titles"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"En primer lugar, queremos poder extraer sustantivos de los títulos de noticias. Usaremos la biblioteca `TextBlob` para hacer esto, lo que simplifica muchas tareas típicas de PLN como esta.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n",
|
||||
"Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n",
|
||||
"Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n",
|
||||
"Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n",
|
||||
"Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n",
|
||||
"Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n",
|
||||
"Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n",
|
||||
"Finished.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"[nltk_data] Downloading package brown to\n",
|
||||
"[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n",
|
||||
"[nltk_data] Package brown is already up-to-date!\n",
|
||||
"[nltk_data] Downloading package punkt to\n",
|
||||
"[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n",
|
||||
"[nltk_data] Package punkt is already up-to-date!\n",
|
||||
"[nltk_data] Downloading package wordnet to\n",
|
||||
"[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n",
|
||||
"[nltk_data] Package wordnet is already up-to-date!\n",
|
||||
"[nltk_data] Downloading package averaged_perceptron_tagger to\n",
|
||||
"[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n",
|
||||
"[nltk_data] Package averaged_perceptron_tagger is already up-to-\n",
|
||||
"[nltk_data] date!\n",
|
||||
"[nltk_data] Downloading package conll2000 to\n",
|
||||
"[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n",
|
||||
"[nltk_data] Package conll2000 is already up-to-date!\n",
|
||||
"[nltk_data] Downloading package movie_reviews to\n",
|
||||
"[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n",
|
||||
"[nltk_data] Package movie_reviews is already up-to-date!\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import sys\n",
|
||||
"!{sys.executable} -m pip install textblob\n",
|
||||
"!{sys.executable} -m textblob.download_corpora\n",
|
||||
"from textblob import TextBlob"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 22,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"{'covid-19 live updates': 1,\n",
|
||||
" 'vaccines': 1,\n",
|
||||
" 'boosters': 1,\n",
|
||||
" 'york': 4,\n",
|
||||
" 'ukrainians flee mariupol': 1,\n",
|
||||
" 'forces push': 1,\n",
|
||||
" 'port city': 1,\n",
|
||||
" 'wall street journal': 3,\n",
|
||||
" 'bond yields': 1,\n",
|
||||
" 'futures rise': 1,\n",
|
||||
" 'powell says fed': 1,\n",
|
||||
" 'ready': 1,\n",
|
||||
" 'be': 1,\n",
|
||||
" 'aggressive': 1,\n",
|
||||
" 'putin': 3,\n",
|
||||
" 'alexei navalny': 2,\n",
|
||||
" 'russian': 2,\n",
|
||||
" 'supreme court nominee': 1,\n",
|
||||
" 'ketanji brown jackson': 1,\n",
|
||||
" \"confirmation hearing 's\": 1,\n",
|
||||
" 'cnn': 1,\n",
|
||||
" 'swedish': 1,\n",
|
||||
" 'high school': 1,\n",
|
||||
" 'abc': 1,\n",
|
||||
" 'clues': 1,\n",
|
||||
" 'covid-19': 1,\n",
|
||||
" '’ s': 2,\n",
|
||||
" 'moves': 1,\n",
|
||||
" 'sewers': 1,\n",
|
||||
" 'roll dice': 1,\n",
|
||||
" 'jackson': 1,\n",
|
||||
" 'decisions |': 1,\n",
|
||||
" 'thehill': 1,\n",
|
||||
" 'clear': 2,\n",
|
||||
" 'chemical weapons': 2,\n",
|
||||
" 'ukraine': 3,\n",
|
||||
" 'claims president': 2,\n",
|
||||
" 'biden': 2,\n",
|
||||
" 'nasa': 2,\n",
|
||||
" 'solar system': 2,\n",
|
||||
" 'daily mail': 3,\n",
|
||||
" 'us stocks': 1,\n",
|
||||
" 'fed chair powell': 1,\n",
|
||||
" \"'s remarks\": 1,\n",
|
||||
" 'fox': 1,\n",
|
||||
" \"'we 've\": 1,\n",
|
||||
" 'tests': 1,\n",
|
||||
" 'covid': 1,\n",
|
||||
" 'politico': 1,\n",
|
||||
" 'duchess': 1,\n",
|
||||
" 'cambridge': 1,\n",
|
||||
" 'swaps khaki jungle gear': 1,\n",
|
||||
" 'vampire': 1,\n",
|
||||
" 'wife': 1,\n",
|
||||
" 'belize': 1,\n",
|
||||
" 'china': 2,\n",
|
||||
" 'flight recorders': 1,\n",
|
||||
" 'plane crash': 1,\n",
|
||||
" 'reuters': 2,\n",
|
||||
" 'russian oligarch': 1,\n",
|
||||
" 'abramovich': 1,\n",
|
||||
" 'live': 1,\n",
|
||||
" 'russia': 2,\n",
|
||||
" 'stops talks': 1,\n",
|
||||
" 'japan': 1,\n",
|
||||
" 'español': 1,\n",
|
||||
" 'powers remain': 1,\n",
|
||||
" 'threats lurk': 1,\n",
|
||||
" 'set': 1,\n",
|
||||
" 'webb': 1,\n",
|
||||
" 'telescope begins multi-instrument alignment': 1,\n",
|
||||
" 'scitechdaily': 1,\n",
|
||||
" 'uconn': 1,\n",
|
||||
" 'ucf': 1,\n",
|
||||
" 'ncaa': 1,\n",
|
||||
" \"women 's tournament second-round highlights\": 1,\n",
|
||||
" 'march madness': 1,\n",
|
||||
" 'bucking republican trend': 1,\n",
|
||||
" 'indiana': 1,\n",
|
||||
" 'vetoes transgender': 1,\n",
|
||||
" 'bill': 1,\n",
|
||||
" 'maggie fox': 1,\n",
|
||||
" 'coronation': 1,\n",
|
||||
" 'shameless': 1,\n",
|
||||
" \"'sudden accident\": 1,\n",
|
||||
" 'mirror online': 1,\n",
|
||||
" 'mirror': 2,\n",
|
||||
" 'plane crash –': 1,\n",
|
||||
" 'search': 1,\n",
|
||||
" 'moment flight': 1,\n",
|
||||
" 'daniel morgan': 1,\n",
|
||||
" 'report condemns': 1,\n",
|
||||
" 'met': 1,\n",
|
||||
" 'guardian': 6,\n",
|
||||
" 'rishi sunak': 1,\n",
|
||||
" '’ s spring': 1,\n",
|
||||
" 'statement': 1,\n",
|
||||
" 'bbc.com': 1,\n",
|
||||
" 'uk': 3,\n",
|
||||
" 'ireland': 1,\n",
|
||||
" 'euro': 1,\n",
|
||||
" 'vladimir putin': 2,\n",
|
||||
" \"'s 'lover\": 1,\n",
|
||||
" 'brass eye': 1,\n",
|
||||
" '’ s outtakes': 1,\n",
|
||||
" 'brutal tv comedy': 1,\n",
|
||||
" 'threatens civilians': 1,\n",
|
||||
" 'mariupol': 1,\n",
|
||||
" \"'s spirit\": 1,\n",
|
||||
" 'shell u-turn': 1,\n",
|
||||
" 'cambo': 1,\n",
|
||||
" 'green targets': 1,\n",
|
||||
" 'st helens': 1,\n",
|
||||
" 'dog attack': 1,\n",
|
||||
" 'girl': 1,\n",
|
||||
" 'bbc': 1,\n",
|
||||
" 'playstation': 1,\n",
|
||||
" \"'assassin 's\": 1,\n",
|
||||
" 'creed': 1,\n",
|
||||
" 'jade raymond': 1,\n",
|
||||
" 'haven studios': 1,\n",
|
||||
" 'nme': 1,\n",
|
||||
" 'nintendo switch': 1,\n",
|
||||
" 'folders •': 1,\n",
|
||||
" 'eurogamer.net': 2,\n",
|
||||
" 'fa': 1,\n",
|
||||
" 'solution ”': 1,\n",
|
||||
" 'liverpool': 1,\n",
|
||||
" 'fan group blasts “ shambolic ”': 1,\n",
|
||||
" 'wembley': 1,\n",
|
||||
" 'anfield': 1,\n",
|
||||
" 'manchester': 1,\n",
|
||||
" 'live erik': 1,\n",
|
||||
" 'hag': 1,\n",
|
||||
" 'utd': 1,\n",
|
||||
" 'manager updates': 1,\n",
|
||||
" 'manchester evening': 1,\n",
|
||||
" 'inflation': 1,\n",
|
||||
" 'government borrowing': 1,\n",
|
||||
" 'february': 1,\n",
|
||||
" 'crude oil': 1,\n",
|
||||
" '– business': 1,\n",
|
||||
" 'kremlin': 1,\n",
|
||||
" 'large-scale fraud': 1,\n",
|
||||
" 'sky': 1,\n",
|
||||
" 'natural gas': 1,\n",
|
||||
" 'gazprom': 1,\n",
|
||||
" 'retail unit': 1,\n",
|
||||
" 'insider': 1,\n",
|
||||
" 'zaghari-ratcliffe': 1,\n",
|
||||
" 'hunt': 1,\n",
|
||||
" 'iran': 1,\n",
|
||||
" 'debt payment': 1}"
|
||||
]
|
||||
},
|
||||
"execution_count": 22,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"w = {}\n",
|
||||
"for x in all_titles:\n",
|
||||
" for n in TextBlob(x).noun_phrases:\n",
|
||||
" if n in w:\n",
|
||||
" w[n].append(x)\n",
|
||||
" else:\n",
|
||||
" w[n]=[x]\n",
|
||||
"{ x:len(w[x]) for x in w.keys()}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Podemos ver que los sustantivos no nos proporcionan grandes grupos temáticos. Sustituyamos los sustantivos por términos más generales obtenidos del gráfico de conceptos. Esto tomará algo de tiempo, porque estamos haciendo una llamada REST para cada frase nominal.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 23,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"w = {}\n",
|
||||
"for x in all_titles:\n",
|
||||
" for noun in TextBlob(x).noun_phrases:\n",
|
||||
" terms = query(noun.replace(' ','%20'))\n",
|
||||
" for term in [u for u in terms.keys() if terms[u]>0.1]:\n",
|
||||
" if term in w:\n",
|
||||
" w[term].append(x)\n",
|
||||
" else:\n",
|
||||
" w[term]=[x]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 24,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"{'city': 9,\n",
|
||||
" 'brand': 4,\n",
|
||||
" 'place': 9,\n",
|
||||
" 'town': 4,\n",
|
||||
" 'factor': 4,\n",
|
||||
" 'film': 4,\n",
|
||||
" 'nation': 11,\n",
|
||||
" 'state': 5,\n",
|
||||
" 'person': 4,\n",
|
||||
" 'organization': 5,\n",
|
||||
" 'publication': 10,\n",
|
||||
" 'market': 5,\n",
|
||||
" 'economy': 4,\n",
|
||||
" 'company': 6,\n",
|
||||
" 'newspaper': 6,\n",
|
||||
" 'relationship': 6}"
|
||||
]
|
||||
},
|
||||
"execution_count": 24,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"{ x:len(w[x]) for x in w.keys() if len(w[x])>3}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 27,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\n",
|
||||
"ECONOMY:\n",
|
||||
"China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n",
|
||||
"Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n",
|
||||
"China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n",
|
||||
"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n",
|
||||
"\n",
|
||||
"NATION:\n",
|
||||
"‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n",
|
||||
"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n",
|
||||
"China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n",
|
||||
"Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n",
|
||||
"Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n",
|
||||
"China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n",
|
||||
"UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n",
|
||||
"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n",
|
||||
"‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n",
|
||||
"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n",
|
||||
"Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n",
|
||||
"\n",
|
||||
"PERSON:\n",
|
||||
"‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n",
|
||||
"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n",
|
||||
"Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n",
|
||||
"‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n",
|
||||
"print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n",
|
||||
"print('\\nPERSON:\\n'+'\\n'.join(w['person']))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Descargo de responsabilidad**: \nEste documento ha sido traducido utilizando el servicio de traducción automática [Co-op Translator](https://github.com/Azure/co-op-translator). Si bien nos esforzamos por lograr precisión, tenga en cuenta que las traducciones automáticas pueden contener errores o imprecisiones. El documento original en su idioma nativo debe considerarse como la fuente autorizada. Para información crítica, se recomienda una traducción profesional realizada por humanos. No nos hacemos responsables de malentendidos o interpretaciones erróneas que puedan surgir del uso de esta traducción.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.7.4 64-bit (conda)",
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5"
|
||||
}
|
||||
},
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.5"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "4087f998407d06ceb2947016ba4605d0",
|
||||
"translation_date": "2025-08-31T16:25:52+00:00",
|
||||
"source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb",
|
||||
"language_code": "es"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
|
|
@ -1,8 +1,8 @@
|
|||
<!--
|
||||
CO_OP_TRANSLATOR_METADATA:
|
||||
{
|
||||
"original_hash": "7336583e4630220c835335da640016db",
|
||||
"translation_date": "2025-08-24T09:23:19+00:00",
|
||||
"original_hash": "ba5d1eb353d20d3e7181066b3c424b99",
|
||||
"translation_date": "2025-08-31T15:38:58+00:00",
|
||||
"source_file": "lessons/3-NeuralNetworks/03-Perceptron/lab/README.md",
|
||||
"language_code": "es"
|
||||
}
|
||||
|
|
@ -13,7 +13,7 @@ Asignación de laboratorio del [Currículo de IA para Principiantes](https://git
|
|||
|
||||
## Tarea
|
||||
|
||||
Usando el código que hemos desarrollado en esta lección para la clasificación binaria de los dígitos manuscritos de MNIST, crea un clasificador multiclase que sea capaz de reconocer cualquier dígito. Calcula la precisión de clasificación en los conjuntos de datos de entrenamiento y prueba, y muestra la matriz de confusión.
|
||||
Usando el código que hemos desarrollado en esta lección para la clasificación binaria de los dígitos escritos a mano de MNIST, crea un clasificador multiclase que sea capaz de reconocer cualquier dígito. Calcula la precisión de clasificación en los conjuntos de datos de entrenamiento y prueba, y muestra la matriz de confusión.
|
||||
|
||||
## Pistas
|
||||
|
||||
|
|
@ -23,9 +23,11 @@ Usando el código que hemos desarrollado en esta lección para la clasificación
|
|||
|
||||
> **Pista**: Si combinamos los pesos de los 10 perceptrones en una sola matriz, deberíamos poder aplicar los 10 perceptrones a los dígitos de entrada mediante una sola multiplicación de matrices. El dígito más probable se puede encontrar simplemente aplicando la operación `argmax` en la salida.
|
||||
|
||||
## Notebook Inicial
|
||||
## Cuaderno Inicial
|
||||
|
||||
Comienza el laboratorio abriendo [PerceptronMultiClass.ipynb](../../../../../../lessons/3-NeuralNetworks/03-Perceptron/lab/PerceptronMultiClass.ipynb)
|
||||
Comienza el laboratorio abriendo [PerceptronMultiClass.ipynb](PerceptronMultiClass.ipynb)
|
||||
|
||||
---
|
||||
|
||||
**Descargo de responsabilidad**:
|
||||
Este documento ha sido traducido utilizando el servicio de traducción automática [Co-op Translator](https://github.com/Azure/co-op-translator). Si bien nos esforzamos por lograr precisión, tenga en cuenta que las traducciones automáticas pueden contener errores o imprecisiones. El documento original en su idioma nativo debe considerarse como la fuente autorizada. Para información crítica, se recomienda una traducción profesional realizada por humanos. No nos hacemos responsables de malentendidos o interpretaciones erróneas que puedan surgir del uso de esta traducción.
|
||||
Este documento ha sido traducido utilizando el servicio de traducción automática [Co-op Translator](https://github.com/Azure/co-op-translator). Si bien nos esforzamos por garantizar la precisión, tenga en cuenta que las traducciones automatizadas pueden contener errores o imprecisiones. El documento original en su idioma nativo debe considerarse como la fuente autorizada. Para información crítica, se recomienda una traducción profesional realizada por humanos. No nos hacemos responsables de malentendidos o interpretaciones erróneas que puedan surgir del uso de esta traducción.
|
||||
File diff suppressed because one or more lines are too long
|
|
@ -0,0 +1,183 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Clasificación de Dígitos MNIST con nuestro propio Framework\n",
|
||||
"\n",
|
||||
"Tarea de laboratorio del [Currículo de IA para Principiantes](https://github.com/microsoft/ai-for-beginners).\n",
|
||||
"\n",
|
||||
"### Leyendo el Conjunto de Datos\n",
|
||||
"\n",
|
||||
"Este código descarga el conjunto de datos desde el repositorio en internet. También puedes copiar manualmente el conjunto de datos desde el directorio `/data` del repositorio del Currículo de IA.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" % Total % Received % Xferd Average Speed Time Time Time Current\n",
|
||||
" Dload Upload Total Spent Left Speed\n",
|
||||
"\n",
|
||||
" 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\n",
|
||||
"100 9.9M 100 9.9M 0 0 9.9M 0 0:00:01 --:--:-- 0:00:01 15.8M\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"!rm *.pkl\n",
|
||||
"!wget https://raw.githubusercontent.com/microsoft/AI-For-Beginners/main/data/mnist.pkl.gz\n",
|
||||
"!gzip -d mnist.pkl.gz"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import pickle\n",
|
||||
"with open('mnist.pkl','rb') as f:\n",
|
||||
" MNIST = pickle.load(f)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"labels = MNIST['Train']['Labels']\n",
|
||||
"data = MNIST['Train']['Features']"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Veamos cuál es la forma de los datos que tenemos:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(42000, 784)"
|
||||
]
|
||||
},
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"data.shape"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Dividiendo los Datos\n",
|
||||
"\n",
|
||||
"Usaremos Scikit Learn para dividir los datos entre el conjunto de entrenamiento y el conjunto de prueba:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Train samples: 33600, test samples: 8400\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from sklearn.model_selection import train_test_split\n",
|
||||
"\n",
|
||||
"features_train, features_test, labels_train, labels_test = train_test_split(data,labels,test_size=0.2)\n",
|
||||
"\n",
|
||||
"print(f\"Train samples: {len(features_train)}, test samples: {len(features_test)}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Instrucciones\n",
|
||||
"\n",
|
||||
"1. Toma el código base del marco de trabajo de la lección y pégalo en este notebook, o (aún mejor) en un módulo de Python separado.\n",
|
||||
"1. Define y entrena un perceptrón de una sola capa, observando la precisión de entrenamiento y validación durante el proceso.\n",
|
||||
"1. Intenta entender si ocurrió sobreajuste y ajusta los parámetros de la capa para mejorar la precisión.\n",
|
||||
"1. Repite los pasos anteriores para perceptrones de 2 y 3 capas. Intenta experimentar con diferentes funciones de activación entre las capas.\n",
|
||||
"1. Intenta responder las siguientes preguntas:\n",
|
||||
" - ¿La función de activación entre capas afecta el rendimiento de la red?\n",
|
||||
" - ¿Necesitamos una red de 2 o 3 capas para esta tarea?\n",
|
||||
" - ¿Tuviste algún problema al entrenar la red? Especialmente a medida que aumentó el número de capas.\n",
|
||||
" - ¿Cómo se comportan los pesos de la red durante el entrenamiento? Puedes graficar el valor absoluto máximo de los pesos frente a las épocas para entender la relación.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Descargo de responsabilidad**: \nEste documento ha sido traducido utilizando el servicio de traducción automática [Co-op Translator](https://github.com/Azure/co-op-translator). Aunque nos esforzamos por garantizar la precisión, tenga en cuenta que las traducciones automáticas pueden contener errores o imprecisiones. El documento original en su idioma nativo debe considerarse como la fuente autorizada. Para información crítica, se recomienda una traducción profesional realizada por humanos. No nos hacemos responsables de malentendidos o interpretaciones erróneas que puedan surgir del uso de esta traducción.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.7.4 64-bit (conda)",
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5"
|
||||
}
|
||||
},
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.5"
|
||||
},
|
||||
"orig_nbformat": 2,
|
||||
"coopTranslator": {
|
||||
"original_hash": "6fa055f484eb5d6bdf41166a356d3abf",
|
||||
"translation_date": "2025-08-31T16:32:18+00:00",
|
||||
"source_file": "lessons/3-NeuralNetworks/04-OwnFramework/lab/MyFW_MNIST.ipynb",
|
||||
"language_code": "es"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
|
|
@ -0,0 +1,108 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Detección de Movimiento de la Palma usando Flujo Óptico\n",
|
||||
"\n",
|
||||
"Este laboratorio es parte del [Currículo de IA para Principiantes](http://aka.ms/ai-beginners).\n",
|
||||
"\n",
|
||||
"Considera [este video](../../../../../../lessons/4-ComputerVision/06-IntroCV/lab/palm-movement.mp4), en el que la palma de una persona se mueve hacia la izquierda/derecha/arriba/abajo sobre un fondo estable.\n",
|
||||
"\n",
|
||||
"**Tu objetivo** será usar Flujo Óptico para determinar qué partes del video contienen movimientos hacia arriba/abajo/izquierda/derecha.\n",
|
||||
"\n",
|
||||
"Comienza obteniendo los fotogramas del video como se describe en la lección:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Code here"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Ahora, calcula los fotogramas de flujo óptico denso como se describe en la conferencia, y convierte el flujo óptico denso a coordenadas polares:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Code here"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Construir un histograma de direcciones para cada fotograma del flujo óptico. Un histograma muestra cuántos vectores caen dentro de ciertos intervalos y debería separar las diferentes direcciones de movimiento en el fotograma.\n",
|
||||
"\n",
|
||||
"> También podrías querer eliminar todos los vectores cuya magnitud esté por debajo de cierto umbral. Esto eliminará pequeños movimientos adicionales en el video, como los de los ojos y la cabeza.\n",
|
||||
"\n",
|
||||
"Representa los histogramas para algunos de los fotogramas.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Code here"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Al observar los histogramas, debería ser bastante sencillo determinar la dirección del movimiento. Necesitas seleccionar aquellos intervalos que correspondan a las direcciones arriba/abajo/izquierda/derecha, y que estén por encima de cierto umbral.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Code here"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"¡Felicidades! Si has realizado todos los pasos anteriores, ¡has completado el laboratorio!\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Descargo de responsabilidad**: \nEste documento ha sido traducido utilizando el servicio de traducción automática [Co-op Translator](https://github.com/Azure/co-op-translator). Aunque nos esforzamos por garantizar la precisión, tenga en cuenta que las traducciones automatizadas pueden contener errores o imprecisiones. El documento original en su idioma nativo debe considerarse como la fuente autorizada. Para información crítica, se recomienda una traducción profesional realizada por humanos. No nos hacemos responsables de malentendidos o interpretaciones erróneas que puedan surgir del uso de esta traducción.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
},
|
||||
"orig_nbformat": 4,
|
||||
"coopTranslator": {
|
||||
"original_hash": "153d9e417e079bf62f8f693002d0deaf",
|
||||
"translation_date": "2025-08-31T16:08:45+00:00",
|
||||
"source_file": "lessons/4-ComputerVision/06-IntroCV/lab/MovementDetection.ipynb",
|
||||
"language_code": "es"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
|
|
@ -0,0 +1,577 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Tarea de clasificación de texto\n",
|
||||
"\n",
|
||||
"Como hemos mencionado, nos centraremos en una tarea sencilla de clasificación de texto basada en el dataset **AG_NEWS**, que consiste en clasificar titulares de noticias en una de las 4 categorías: Mundo, Deportes, Negocios y Ciencia/Tecnología.\n",
|
||||
"\n",
|
||||
"## El Dataset\n",
|
||||
"\n",
|
||||
"Este dataset está integrado en el módulo [`torchtext`](https://github.com/pytorch/text), por lo que podemos acceder a él fácilmente.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"import torchtext\n",
|
||||
"import os\n",
|
||||
"import collections\n",
|
||||
"os.makedirs('./data',exist_ok=True)\n",
|
||||
"train_dataset, test_dataset = torchtext.datasets.AG_NEWS(root='./data')\n",
|
||||
"classes = ['World', 'Sports', 'Business', 'Sci/Tech']"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Aquí, `train_dataset` y `test_dataset` contienen colecciones que devuelven pares de etiqueta (número de clase) y texto respectivamente, por ejemplo:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(3,\n",
|
||||
" \"Wall St. Bears Claw Back Into the Black (Reuters) Reuters - Short-sellers, Wall Street's dwindling\\\\band of ultra-cynics, are seeing green again.\")"
|
||||
]
|
||||
},
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"list(train_dataset)[0]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Entonces, imprimamos los primeros 10 nuevos titulares de nuestro conjunto de datos:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"**Sci/Tech** -> Wall St. Bears Claw Back Into the Black (Reuters) Reuters - Short-sellers, Wall Street's dwindling\\band of ultra-cynics, are seeing green again.\n",
|
||||
"**Sci/Tech** -> Carlyle Looks Toward Commercial Aerospace (Reuters) Reuters - Private investment firm Carlyle Group,\\which has a reputation for making well-timed and occasionally\\controversial plays in the defense industry, has quietly placed\\its bets on another part of the market.\n",
|
||||
"**Sci/Tech** -> Oil and Economy Cloud Stocks' Outlook (Reuters) Reuters - Soaring crude prices plus worries\\about the economy and the outlook for earnings are expected to\\hang over the stock market next week during the depth of the\\summer doldrums.\n",
|
||||
"**Sci/Tech** -> Iraq Halts Oil Exports from Main Southern Pipeline (Reuters) Reuters - Authorities have halted oil export\\flows from the main pipeline in southern Iraq after\\intelligence showed a rebel militia could strike\\infrastructure, an oil official said on Saturday.\n",
|
||||
"**Sci/Tech** -> Oil prices soar to all-time record, posing new menace to US economy (AFP) AFP - Tearaway world oil prices, toppling records and straining wallets, present a new economic menace barely three months before the US presidential elections.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"for i,x in zip(range(5),train_dataset):\n",
|
||||
" print(f\"**{classes[x[0]]}** -> {x[1]}\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Debido a que los conjuntos de datos son iteradores, si queremos usar los datos varias veces, necesitamos convertirlos a una lista:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"train_dataset, test_dataset = torchtext.datasets.AG_NEWS(root='./data')\n",
|
||||
"train_dataset = list(train_dataset)\n",
|
||||
"test_dataset = list(test_dataset)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Tokenización\n",
|
||||
"\n",
|
||||
"Ahora necesitamos convertir el texto en **números** que puedan representarse como tensores. Si queremos una representación a nivel de palabras, debemos hacer dos cosas: \n",
|
||||
"* usar un **tokenizador** para dividir el texto en **tokens** \n",
|
||||
"* construir un **vocabulario** de esos tokens. \n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"['he', 'said', 'hello']"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"tokenizer = torchtext.data.utils.get_tokenizer('basic_english')\n",
|
||||
"tokenizer('He said: hello')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"counter = collections.Counter()\n",
|
||||
"for (label, line) in train_dataset:\n",
|
||||
" counter.update(tokenizer(line))\n",
|
||||
"vocab = torchtext.vocab.vocab(counter, min_freq=1)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Usando vocabulario, podemos codificar fácilmente nuestra cadena tokenizada en un conjunto de números:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 19,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Vocab size if 95810\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[599, 3279, 97, 1220, 329, 225, 7368]"
|
||||
]
|
||||
},
|
||||
"execution_count": 19,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vocab_size = len(vocab)\n",
|
||||
"print(f\"Vocab size if {vocab_size}\")\n",
|
||||
"\n",
|
||||
"stoi = vocab.get_stoi() # dict to convert tokens to indices\n",
|
||||
"\n",
|
||||
"def encode(x):\n",
|
||||
" return [stoi[s] for s in tokenizer(x)]\n",
|
||||
"\n",
|
||||
"encode('I love to play with my words')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Representación de texto con Bolsa de Palabras\n",
|
||||
"\n",
|
||||
"Debido a que las palabras representan significado, a veces podemos entender el significado de un texto simplemente observando las palabras individuales, sin importar su orden en la oración. Por ejemplo, al clasificar noticias, palabras como *clima*, *nieve* probablemente indiquen *pronóstico del tiempo*, mientras que palabras como *acciones*, *dólar* podrían corresponder a *noticias financieras*.\n",
|
||||
"\n",
|
||||
"La representación vectorial **Bolsa de Palabras** (BoW) es la representación vectorial tradicional más utilizada. Cada palabra está vinculada a un índice del vector, y el elemento del vector contiene el número de veces que una palabra aparece en un documento dado.\n",
|
||||
"\n",
|
||||
" \n",
|
||||
"\n",
|
||||
"> **Nota**: También puedes pensar en BoW como la suma de todos los vectores codificados en uno a uno para las palabras individuales en el texto.\n",
|
||||
"\n",
|
||||
"A continuación, se muestra un ejemplo de cómo generar una representación de bolsa de palabras utilizando la biblioteca de Python Scikit Learn:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([[1, 1, 0, 2, 0, 0, 0, 0, 0]], dtype=int64)"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from sklearn.feature_extraction.text import CountVectorizer\n",
|
||||
"vectorizer = CountVectorizer()\n",
|
||||
"corpus = [\n",
|
||||
" 'I like hot dogs.',\n",
|
||||
" 'The dog ran fast.',\n",
|
||||
" 'Its hot outside.',\n",
|
||||
" ]\n",
|
||||
"vectorizer.fit_transform(corpus)\n",
|
||||
"vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Para calcular el vector de bolsa de palabras a partir de la representación vectorial de nuestro conjunto de datos AG_NEWS, podemos usar la siguiente función:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 20,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"tensor([2., 1., 2., ..., 0., 0., 0.])\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vocab_size = len(vocab)\n",
|
||||
"\n",
|
||||
"def to_bow(text,bow_vocab_size=vocab_size):\n",
|
||||
" res = torch.zeros(bow_vocab_size,dtype=torch.float32)\n",
|
||||
" for i in encode(text):\n",
|
||||
" if i<bow_vocab_size:\n",
|
||||
" res[i] += 1\n",
|
||||
" return res\n",
|
||||
"\n",
|
||||
"print(to_bow(train_dataset[0][1]))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Nota:** Aquí estamos utilizando la variable global `vocab_size` para especificar el tamaño predeterminado del vocabulario. Dado que a menudo el tamaño del vocabulario es bastante grande, podemos limitar el tamaño del vocabulario a las palabras más frecuentes. Intenta reducir el valor de `vocab_size` y ejecutar el código a continuación, y observa cómo afecta la precisión. Deberías esperar una disminución en la precisión, pero no dramática, en lugar de un mayor rendimiento.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Entrenando el clasificador BoW\n",
|
||||
"\n",
|
||||
"Ahora que hemos aprendido cómo construir la representación de Bag-of-Words para nuestro texto, vamos a entrenar un clasificador sobre esta base. Primero, necesitamos convertir nuestro conjunto de datos para el entrenamiento de tal manera que todas las representaciones vectoriales posicionales se conviertan en representaciones de Bag-of-Words. Esto se puede lograr pasando la función `bowify` como el parámetro `collate_fn` al `DataLoader` estándar de torch:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 21,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from torch.utils.data import DataLoader\n",
|
||||
"import numpy as np \n",
|
||||
"\n",
|
||||
"# this collate function gets list of batch_size tuples, and needs to \n",
|
||||
"# return a pair of label-feature tensors for the whole minibatch\n",
|
||||
"def bowify(b):\n",
|
||||
" return (\n",
|
||||
" torch.LongTensor([t[0]-1 for t in b]),\n",
|
||||
" torch.stack([to_bow(t[1]) for t in b])\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"train_loader = DataLoader(train_dataset, batch_size=16, collate_fn=bowify, shuffle=True)\n",
|
||||
"test_loader = DataLoader(test_dataset, batch_size=16, collate_fn=bowify, shuffle=True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Ahora definamos una red neuronal clasificador simple que contiene una capa lineal. El tamaño del vector de entrada es igual a `vocab_size`, y el tamaño de salida corresponde al número de clases (4). Debido a que estamos resolviendo una tarea de clasificación, la función de activación final es `LogSoftmax()`.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 22,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"net = torch.nn.Sequential(torch.nn.Linear(vocab_size,4),torch.nn.LogSoftmax(dim=1))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Ahora definiremos el bucle de entrenamiento estándar de PyTorch. Debido a que nuestro conjunto de datos es bastante grande, para nuestro propósito de enseñanza entrenaremos solo por una época, y a veces incluso por menos de una época (especificar el parámetro `epoch_size` nos permite limitar el entrenamiento). También informaremos la precisión acumulada del entrenamiento durante el proceso; la frecuencia de los informes se especifica utilizando el parámetro `report_freq`.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 24,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def train_epoch(net,dataloader,lr=0.01,optimizer=None,loss_fn = torch.nn.NLLLoss(),epoch_size=None, report_freq=200):\n",
|
||||
" optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr)\n",
|
||||
" net.train()\n",
|
||||
" total_loss,acc,count,i = 0,0,0,0\n",
|
||||
" for labels,features in dataloader:\n",
|
||||
" optimizer.zero_grad()\n",
|
||||
" out = net(features)\n",
|
||||
" loss = loss_fn(out,labels) #cross_entropy(out,labels)\n",
|
||||
" loss.backward()\n",
|
||||
" optimizer.step()\n",
|
||||
" total_loss+=loss\n",
|
||||
" _,predicted = torch.max(out,1)\n",
|
||||
" acc+=(predicted==labels).sum()\n",
|
||||
" count+=len(labels)\n",
|
||||
" i+=1\n",
|
||||
" if i%report_freq==0:\n",
|
||||
" print(f\"{count}: acc={acc.item()/count}\")\n",
|
||||
" if epoch_size and count>epoch_size:\n",
|
||||
" break\n",
|
||||
" return total_loss.item()/count, acc.item()/count"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 25,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.8028125\n",
|
||||
"6400: acc=0.8371875\n",
|
||||
"9600: acc=0.8534375\n",
|
||||
"12800: acc=0.85765625\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(0.026090790722161722, 0.8620069296375267)"
|
||||
]
|
||||
},
|
||||
"execution_count": 25,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"train_epoch(net,train_loader,epoch_size=15000)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## BiGrams, TriGrams y N-Grams\n",
|
||||
"\n",
|
||||
"Una limitación del enfoque de bolsa de palabras es que algunas palabras forman expresiones de varias palabras. Por ejemplo, la palabra 'hot dog' tiene un significado completamente diferente al de las palabras 'hot' y 'dog' en otros contextos. Si representamos las palabras 'hot' y 'dog' siempre con los mismos vectores, esto puede confundir a nuestro modelo.\n",
|
||||
"\n",
|
||||
"Para abordar este problema, las **representaciones N-gram** se utilizan frecuentemente en métodos de clasificación de documentos, donde la frecuencia de cada palabra, bi-palabra o tri-palabra es una característica útil para entrenar clasificadores. En la representación de bigramas, por ejemplo, añadimos todos los pares de palabras al vocabulario, además de las palabras originales.\n",
|
||||
"\n",
|
||||
"A continuación, se muestra un ejemplo de cómo generar una representación de bolsa de palabras con bigramas utilizando Scikit Learn:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 26,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Vocabulary:\n",
|
||||
" {'i': 7, 'like': 11, 'hot': 4, 'dogs': 2, 'i like': 8, 'like hot': 12, 'hot dogs': 5, 'the': 16, 'dog': 0, 'ran': 14, 'fast': 3, 'the dog': 17, 'dog ran': 1, 'ran fast': 15, 'its': 9, 'outside': 13, 'its hot': 10, 'hot outside': 6}\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([[1, 0, 1, 0, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],\n",
|
||||
" dtype=int64)"
|
||||
]
|
||||
},
|
||||
"execution_count": 26,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"bigram_vectorizer = CountVectorizer(ngram_range=(1, 2), token_pattern=r'\\b\\w+\\b', min_df=1)\n",
|
||||
"corpus = [\n",
|
||||
" 'I like hot dogs.',\n",
|
||||
" 'The dog ran fast.',\n",
|
||||
" 'Its hot outside.',\n",
|
||||
" ]\n",
|
||||
"bigram_vectorizer.fit_transform(corpus)\n",
|
||||
"print(\"Vocabulary:\\n\",bigram_vectorizer.vocabulary_)\n",
|
||||
"bigram_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"La principal desventaja del enfoque N-gram es que el tamaño del vocabulario comienza a crecer extremadamente rápido. En la práctica, necesitamos combinar la representación N-gram con algunas técnicas de reducción de dimensionalidad, como *embeddings*, que discutiremos en la próxima unidad.\n",
|
||||
"\n",
|
||||
"Para usar la representación N-gram en nuestro conjunto de datos **AG News**, necesitamos construir un vocabulario especial de ngram:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 27,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Bigram vocabulary length = 1308842\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"counter = collections.Counter()\n",
|
||||
"for (label, line) in train_dataset:\n",
|
||||
" l = tokenizer(line)\n",
|
||||
" counter.update(torchtext.data.utils.ngrams_iterator(l,ngrams=2))\n",
|
||||
" \n",
|
||||
"bi_vocab = torchtext.vocab.vocab(counter, min_freq=1)\n",
|
||||
"\n",
|
||||
"print(\"Bigram vocabulary length = \",len(bi_vocab))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Podríamos usar el mismo código mencionado anteriormente para entrenar el clasificador, sin embargo, sería muy ineficiente en términos de memoria. En la próxima unidad, entrenaremos un clasificador de bigramas utilizando embeddings.\n",
|
||||
"\n",
|
||||
"> **Nota:** Solo puedes conservar aquellos ngrams que aparezcan en el texto más veces que el número especificado. Esto asegurará que los bigramas poco frecuentes sean omitidos y reducirá significativamente la dimensionalidad. Para lograr esto, ajusta el parámetro `min_freq` a un valor más alto y observa cómo cambia la longitud del vocabulario.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Frecuencia de Términos e Inversa Frecuencia de Documentos TF-IDF\n",
|
||||
"\n",
|
||||
"En la representación BoW, las ocurrencias de palabras tienen el mismo peso, sin importar la palabra en sí. Sin embargo, está claro que las palabras frecuentes, como *a*, *en*, etc., son mucho menos importantes para la clasificación que los términos especializados. De hecho, en la mayoría de las tareas de PLN, algunas palabras son más relevantes que otras.\n",
|
||||
"\n",
|
||||
"**TF-IDF** significa **frecuencia de términos–inversa frecuencia de documentos**. Es una variación de bolsa de palabras, donde en lugar de un valor binario 0/1 que indica la aparición de una palabra en un documento, se utiliza un valor de punto flotante relacionado con la frecuencia de aparición de la palabra en el corpus.\n",
|
||||
"\n",
|
||||
"Más formalmente, el peso $w_{ij}$ de una palabra $i$ en el documento $j$ se define como:\n",
|
||||
"$$\n",
|
||||
"w_{ij} = tf_{ij}\\times\\log({N\\over df_i})\n",
|
||||
"$$\n",
|
||||
"donde\n",
|
||||
"* $tf_{ij}$ es el número de ocurrencias de $i$ en $j$, es decir, el valor BoW que hemos visto antes\n",
|
||||
"* $N$ es el número de documentos en la colección\n",
|
||||
"* $df_i$ es el número de documentos que contienen la palabra $i$ en toda la colección\n",
|
||||
"\n",
|
||||
"El valor TF-IDF $w_{ij}$ aumenta proporcionalmente al número de veces que una palabra aparece en un documento y se ajusta por el número de documentos en el corpus que contienen la palabra, lo que ayuda a compensar el hecho de que algunas palabras aparecen con más frecuencia que otras. Por ejemplo, si la palabra aparece en *todos* los documentos de la colección, $df_i=N$, y $w_{ij}=0$, y esos términos serían completamente ignorados.\n",
|
||||
"\n",
|
||||
"Puedes crear fácilmente una vectorización TF-IDF de texto utilizando Scikit Learn:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 28,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([[0.43381609, 0. , 0.43381609, 0. , 0.65985664,\n",
|
||||
" 0.43381609, 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. ]])"
|
||||
]
|
||||
},
|
||||
"execution_count": 28,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
|
||||
"vectorizer = TfidfVectorizer(ngram_range=(1,2))\n",
|
||||
"vectorizer.fit_transform(corpus)\n",
|
||||
"vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Conclusión\n",
|
||||
"\n",
|
||||
"Sin embargo, aunque las representaciones TF-IDF asignan un peso de frecuencia a diferentes palabras, no son capaces de representar el significado ni el orden. Como dijo el famoso lingüista J. R. Firth en 1935: “El significado completo de una palabra siempre es contextual, y ningún estudio del significado fuera del contexto puede tomarse en serio”. Más adelante en el curso aprenderemos cómo capturar información contextual del texto utilizando modelos de lenguaje.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Descargo de responsabilidad**: \nEste documento ha sido traducido utilizando el servicio de traducción automática [Co-op Translator](https://github.com/Azure/co-op-translator). Aunque nos esforzamos por garantizar la precisión, tenga en cuenta que las traducciones automatizadas pueden contener errores o imprecisiones. El documento original en su idioma nativo debe considerarse como la fuente autorizada. Para información crítica, se recomienda una traducción profesional realizada por humanos. No nos hacemos responsables de malentendidos o interpretaciones erróneas que puedan surgir del uso de esta traducción.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.8.12 ('py38')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "7b9040985e748e4e2d4c689892456ad7",
|
||||
"translation_date": "2025-08-31T17:16:57+00:00",
|
||||
"source_file": "lessons/5-NLP/13-TextRep/TextRepresentationPyTorch.ipynb",
|
||||
"language_code": "es"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
|
|
@ -0,0 +1,647 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Tarea de clasificación de texto\n",
|
||||
"\n",
|
||||
"En este módulo, comenzaremos con una tarea sencilla de clasificación de texto basada en el conjunto de datos **[AG_NEWS](http://www.di.unipi.it/~gulli/AG_corpus_of_news_articles.html)**: clasificaremos titulares de noticias en una de 4 categorías: Mundo, Deportes, Negocios y Ciencia/Tecnología.\n",
|
||||
"\n",
|
||||
"## El Conjunto de Datos\n",
|
||||
"\n",
|
||||
"Para cargar el conjunto de datos, utilizaremos la API de **[TensorFlow Datasets](https://www.tensorflow.org/datasets)**.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import tensorflow as tf\n",
|
||||
"from tensorflow import keras\n",
|
||||
"import tensorflow_datasets as tfds\n",
|
||||
"\n",
|
||||
"# In this tutorial, we will be training a lot of models. In order to use GPU memory cautiously,\n",
|
||||
"# we will set tensorflow option to grow GPU memory allocation when required.\n",
|
||||
"physical_devices = tf.config.list_physical_devices('GPU') \n",
|
||||
"if len(physical_devices)>0:\n",
|
||||
" tf.config.experimental.set_memory_growth(physical_devices[0], True)\n",
|
||||
"\n",
|
||||
"dataset = tfds.load('ag_news_subset')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Ahora podemos acceder a las partes de entrenamiento y prueba del conjunto de datos utilizando `dataset['train']` y `dataset['test']` respectivamente:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Length of train dataset = 120000\n",
|
||||
"Length of test dataset = 7600\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"ds_train = dataset['train']\n",
|
||||
"ds_test = dataset['test']\n",
|
||||
"\n",
|
||||
"print(f\"Length of train dataset = {len(ds_train)}\")\n",
|
||||
"print(f\"Length of test dataset = {len(ds_test)}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Imprimamos los primeros 10 nuevos titulares de nuestro conjunto de datos:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3 (Sci/Tech) -> b'AMD Debuts Dual-Core Opteron Processor' b'AMD #39;s new dual-core Opteron chip is designed mainly for corporate computing applications, including databases, Web services, and financial transactions.'\n",
|
||||
"1 (Sports) -> b\"Wood's Suspension Upheld (Reuters)\" b'Reuters - Major League Baseball\\\\Monday announced a decision on the appeal filed by Chicago Cubs\\\\pitcher Kerry Wood regarding a suspension stemming from an\\\\incident earlier this season.'\n",
|
||||
"2 (Business) -> b'Bush reform may have blue states seeing red' b'President Bush #39;s quot;revenue-neutral quot; tax reform needs losers to balance its winners, and people claiming the federal deduction for state and local taxes may be in administration planners #39; sights, news reports say.'\n",
|
||||
"3 (Sci/Tech) -> b\"'Halt science decline in schools'\" b'Britain will run out of leading scientists unless science education is improved, says Professor Colin Pillinger.'\n",
|
||||
"1 (Sports) -> b'Gerrard leaves practice' b'London, England (Sports Network) - England midfielder Steven Gerrard injured his groin late in Thursday #39;s training session, but is hopeful he will be ready for Saturday #39;s World Cup qualifier against Austria.'\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"classes = ['World', 'Sports', 'Business', 'Sci/Tech']\n",
|
||||
"\n",
|
||||
"for i,x in zip(range(5),ds_train):\n",
|
||||
" print(f\"{x['label']} ({classes[x['label']]}) -> {x['title']} {x['description']}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Vectorización de texto\n",
|
||||
"\n",
|
||||
"Ahora necesitamos convertir el texto en **números** que puedan representarse como tensores. Si queremos una representación a nivel de palabras, necesitamos hacer dos cosas:\n",
|
||||
"\n",
|
||||
"* Usar un **tokenizador** para dividir el texto en **tokens**.\n",
|
||||
"* Construir un **vocabulario** de esos tokens.\n",
|
||||
"\n",
|
||||
"### Limitando el tamaño del vocabulario\n",
|
||||
"\n",
|
||||
"En el ejemplo del conjunto de datos AG News, el tamaño del vocabulario es bastante grande, más de 100k palabras. En términos generales, no necesitamos palabras que estén presentes raramente en el texto — solo unas pocas frases las tendrán, y el modelo no aprenderá de ellas. Por lo tanto, tiene sentido limitar el tamaño del vocabulario a un número más pequeño pasando un argumento al constructor del vectorizador:\n",
|
||||
"\n",
|
||||
"Ambos pasos pueden manejarse utilizando la capa **TextVectorization**. Vamos a instanciar el objeto vectorizador y luego llamar al método `adapt` para recorrer todo el texto y construir un vocabulario:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"vocab_size = 50000\n",
|
||||
"vectorizer = keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size)\n",
|
||||
"vectorizer.adapt(ds_train.take(500).map(lambda x: x['title']+' '+x['description']))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Nota** que estamos utilizando solo un subconjunto del conjunto de datos completo para construir un vocabulario. Hacemos esto para acelerar el tiempo de ejecución y no hacerte esperar. Sin embargo, corremos el riesgo de que algunas palabras del conjunto de datos completo no se incluyan en el vocabulario y sean ignoradas durante el entrenamiento. Por lo tanto, usar el tamaño completo del vocabulario y procesar todo el conjunto de datos durante `adapt` debería aumentar la precisión final, pero no de manera significativa.\n",
|
||||
"\n",
|
||||
"Ahora podemos acceder al vocabulario real:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"['', '[UNK]', 'the', 'to', 'a', 'in', 'of', 'and', 'on', 'for']\n",
|
||||
"Length of vocabulary: 5335\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vocab = vectorizer.get_vocabulary()\n",
|
||||
"vocab_size = len(vocab)\n",
|
||||
"print(vocab[:10])\n",
|
||||
"print(f\"Length of vocabulary: {vocab_size}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Usando el vectorizador, podemos codificar fácilmente cualquier texto en un conjunto de números:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tf.Tensor: shape=(7,), dtype=int64, numpy=array([ 112, 3695, 3, 304, 11, 1041, 1], dtype=int64)>"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vectorizer('I love to play with my words')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Representación de texto con bolsa de palabras\n",
|
||||
"\n",
|
||||
"Dado que las palabras representan significado, a veces podemos deducir el significado de un texto simplemente observando las palabras individuales, sin importar su orden en la oración. Por ejemplo, al clasificar noticias, palabras como *clima* y *nieve* probablemente indiquen *pronóstico del tiempo*, mientras que palabras como *acciones* y *dólar* podrían corresponder a *noticias financieras*.\n",
|
||||
"\n",
|
||||
"La representación vectorial **bolsa de palabras** (BoW) es la más sencilla de entender entre las representaciones vectoriales tradicionales. Cada palabra está vinculada a un índice de vector, y un elemento del vector contiene el número de veces que aparece cada palabra en un documento dado.\n",
|
||||
"\n",
|
||||
" \n",
|
||||
"\n",
|
||||
"> **Nota**: También puedes pensar en BoW como la suma de todos los vectores codificados en formato one-hot para las palabras individuales en el texto.\n",
|
||||
"\n",
|
||||
"A continuación, se muestra un ejemplo de cómo generar una representación de bolsa de palabras utilizando la biblioteca de Python Scikit Learn:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([[1, 1, 0, 2, 0, 0, 0, 0, 0]], dtype=int64)"
|
||||
]
|
||||
},
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from sklearn.feature_extraction.text import CountVectorizer\n",
|
||||
"sc_vectorizer = CountVectorizer()\n",
|
||||
"corpus = [\n",
|
||||
" 'I like hot dogs.',\n",
|
||||
" 'The dog ran fast.',\n",
|
||||
" 'Its hot outside.',\n",
|
||||
" ]\n",
|
||||
"sc_vectorizer.fit_transform(corpus)\n",
|
||||
"sc_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"También podemos usar el vectorizador de Keras que definimos anteriormente, convirtiendo cada número de palabra en una codificación one-hot y sumando todos esos vectores.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([0., 5., 0., ..., 0., 0., 0.], dtype=float32)"
|
||||
]
|
||||
},
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def to_bow(text):\n",
|
||||
" return tf.reduce_sum(tf.one_hot(vectorizer(text),vocab_size),axis=0)\n",
|
||||
"\n",
|
||||
"to_bow('My dog likes hot dogs on a hot day.').numpy()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Nota**: Puede que te sorprenda que el resultado difiera del ejemplo anterior. La razón es que, en el ejemplo de Keras, la longitud del vector corresponde al tamaño del vocabulario, que se construyó a partir de todo el conjunto de datos de AG News, mientras que en el ejemplo de Scikit Learn construimos el vocabulario a partir del texto de muestra sobre la marcha.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Entrenando el clasificador BoW\n",
|
||||
"\n",
|
||||
"Ahora que hemos aprendido a construir la representación de bolsa de palabras (bag-of-words) de nuestro texto, entrenemos un clasificador que la utilice. Primero, necesitamos convertir nuestro conjunto de datos a una representación de bolsa de palabras. Esto se puede lograr utilizando la función `map` de la siguiente manera:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"batch_size = 128\n",
|
||||
"\n",
|
||||
"ds_train_bow = ds_train.map(lambda x: (to_bow(x['title']+x['description']),x['label'])).batch(batch_size)\n",
|
||||
"ds_test_bow = ds_test.map(lambda x: (to_bow(x['title']+x['description']),x['label'])).batch(batch_size)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Ahora definamos una red neuronal clasificador simple que contiene una capa lineal. El tamaño de entrada es `vocab_size`, y el tamaño de salida corresponde al número de clases (4). Debido a que estamos resolviendo una tarea de clasificación, la función de activación final es **softmax**:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"938/938 [==============================] - 66s 70ms/step - loss: 0.6144 - acc: 0.8427 - val_loss: 0.4416 - val_acc: 0.8697\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.History at 0x20c70a947f0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model = keras.models.Sequential([\n",
|
||||
" keras.layers.Dense(4,activation='softmax',input_shape=(vocab_size,))\n",
|
||||
"])\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
|
||||
"model.fit(ds_train_bow,validation_data=ds_test_bow)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Dado que tenemos 4 clases, una precisión superior al 80% es un buen resultado.\n",
|
||||
"\n",
|
||||
"## Entrenando un clasificador como una sola red\n",
|
||||
"\n",
|
||||
"Como el vectorizador también es una capa de Keras, podemos definir una red que lo incluya y entrenarla de principio a fin. De esta manera, no necesitamos vectorizar el conjunto de datos usando `map`, simplemente podemos pasar el conjunto de datos original a la entrada de la red.\n",
|
||||
"\n",
|
||||
"> **Nota**: Aún tendríamos que aplicar mapas a nuestro conjunto de datos para convertir campos de diccionarios (como `title`, `description` y `label`) en tuplas. Sin embargo, al cargar datos desde el disco, podemos construir un conjunto de datos con la estructura requerida desde el principio.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"model\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
" Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
" input_1 (InputLayer) [(None, 1)] 0 \n",
|
||||
" \n",
|
||||
" text_vectorization (TextVec (None, None) 0 \n",
|
||||
" torization) \n",
|
||||
" \n",
|
||||
" tf.one_hot (TFOpLambda) (None, None, 5335) 0 \n",
|
||||
" \n",
|
||||
" tf.math.reduce_sum (TFOpLam (None, 5335) 0 \n",
|
||||
" bda) \n",
|
||||
" \n",
|
||||
" dense_2 (Dense) (None, 4) 21344 \n",
|
||||
" \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 21,344\n",
|
||||
"Trainable params: 21,344\n",
|
||||
"Non-trainable params: 0\n",
|
||||
"_________________________________________________________________\n",
|
||||
"938/938 [==============================] - 73s 77ms/step - loss: 0.6057 - acc: 0.8414 - val_loss: 0.4202 - val_acc: 0.8736\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.History at 0x20c721521f0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def extract_text(x):\n",
|
||||
" return x['title']+' '+x['description']\n",
|
||||
"\n",
|
||||
"def tupelize(x):\n",
|
||||
" return (extract_text(x),x['label'])\n",
|
||||
"\n",
|
||||
"inp = keras.Input(shape=(1,),dtype=tf.string)\n",
|
||||
"x = vectorizer(inp)\n",
|
||||
"x = tf.reduce_sum(tf.one_hot(x,vocab_size),axis=1)\n",
|
||||
"out = keras.layers.Dense(4,activation='softmax')(x)\n",
|
||||
"model = keras.models.Model(inp,out)\n",
|
||||
"model.summary()\n",
|
||||
"\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Bigramas, trigramas y n-gramas\n",
|
||||
"\n",
|
||||
"Una limitación del enfoque de bolsa de palabras es que algunas palabras forman parte de expresiones de varias palabras. Por ejemplo, la palabra 'hot dog' tiene un significado completamente diferente al de las palabras 'hot' y 'dog' en otros contextos. Si representamos las palabras 'hot' y 'dog' siempre usando los mismos vectores, esto puede confundir a nuestro modelo.\n",
|
||||
"\n",
|
||||
"Para abordar este problema, las **representaciones n-gram** se utilizan con frecuencia en métodos de clasificación de documentos, donde la frecuencia de cada palabra, bi-palabra o tri-palabra es una característica útil para entrenar clasificadores. En las representaciones de bigramas, por ejemplo, añadiremos todos los pares de palabras al vocabulario, además de las palabras originales.\n",
|
||||
"\n",
|
||||
"A continuación, se muestra un ejemplo de cómo generar una representación de bolsa de palabras con bigramas utilizando Scikit Learn:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Vocabulary:\n",
|
||||
" {'i': 7, 'like': 11, 'hot': 4, 'dogs': 2, 'i like': 8, 'like hot': 12, 'hot dogs': 5, 'the': 16, 'dog': 0, 'ran': 14, 'fast': 3, 'the dog': 17, 'dog ran': 1, 'ran fast': 15, 'its': 9, 'outside': 13, 'its hot': 10, 'hot outside': 6}\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([[1, 0, 1, 0, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],\n",
|
||||
" dtype=int64)"
|
||||
]
|
||||
},
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"bigram_vectorizer = CountVectorizer(ngram_range=(1, 2), token_pattern=r'\\b\\w+\\b', min_df=1)\n",
|
||||
"corpus = [\n",
|
||||
" 'I like hot dogs.',\n",
|
||||
" 'The dog ran fast.',\n",
|
||||
" 'Its hot outside.',\n",
|
||||
" ]\n",
|
||||
"bigram_vectorizer.fit_transform(corpus)\n",
|
||||
"print(\"Vocabulary:\\n\",bigram_vectorizer.vocabulary_)\n",
|
||||
"bigram_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"La principal desventaja del enfoque de n-gramas es que el tamaño del vocabulario comienza a crecer extremadamente rápido. En la práctica, necesitamos combinar la representación de n-gramas con una técnica de reducción de dimensionalidad, como *embeddings*, que discutiremos en la próxima unidad.\n",
|
||||
"\n",
|
||||
"Para usar una representación de n-gramas en nuestro conjunto de datos **AG News**, necesitamos pasar el parámetro `ngrams` al constructor de `TextVectorization`. ¡La longitud de un vocabulario de bigramas es **significativamente mayor**! En nuestro caso, ¡es más de 1.3 millones de tokens! Por lo tanto, tiene sentido limitar también los tokens de bigramas a un número razonable.\n",
|
||||
"\n",
|
||||
"Podríamos usar el mismo código que arriba para entrenar el clasificador, sin embargo, sería muy ineficiente en términos de memoria. En la próxima unidad, entrenaremos el clasificador de bigramas utilizando embeddings. Mientras tanto, puedes experimentar con el entrenamiento del clasificador de bigramas en este notebook y ver si puedes obtener una mayor precisión.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Calculando automáticamente vectores BoW\n",
|
||||
"\n",
|
||||
"En el ejemplo anterior calculamos los vectores BoW manualmente sumando las codificaciones one-hot de palabras individuales. Sin embargo, la última versión de TensorFlow nos permite calcular los vectores BoW automáticamente al pasar el parámetro `output_mode='count` al constructor del vectorizador. Esto hace que definir y entrenar nuestro modelo sea significativamente más fácil:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Training vectorizer\n",
|
||||
"938/938 [==============================] - 7s 7ms/step - loss: 0.5929 - acc: 0.8486 - val_loss: 0.4168 - val_acc: 0.8772\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.History at 0x20c725217c0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model = keras.models.Sequential([\n",
|
||||
" keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size,output_mode='count'),\n",
|
||||
" keras.layers.Dense(4,input_shape=(vocab_size,), activation='softmax')\n",
|
||||
"])\n",
|
||||
"print(\"Training vectorizer\")\n",
|
||||
"model.layers[0].adapt(ds_train.take(500).map(extract_text))\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Frecuencia de término - frecuencia inversa de documento (TF-IDF)\n",
|
||||
"\n",
|
||||
"En la representación BoW, las ocurrencias de palabras se ponderan utilizando la misma técnica sin importar la palabra en sí. Sin embargo, está claro que palabras frecuentes como *a* e *in* son mucho menos importantes para la clasificación que términos especializados. En la mayoría de las tareas de PLN, algunas palabras son más relevantes que otras.\n",
|
||||
"\n",
|
||||
"**TF-IDF** significa **frecuencia de término - frecuencia inversa de documento**. Es una variación de bolsa de palabras, donde en lugar de un valor binario 0/1 que indica la aparición de una palabra en un documento, se utiliza un valor de punto flotante, que está relacionado con la frecuencia de aparición de la palabra en el corpus.\n",
|
||||
"\n",
|
||||
"De manera más formal, el peso $w_{ij}$ de una palabra $i$ en el documento $j$ se define como:\n",
|
||||
"$$\n",
|
||||
"w_{ij} = tf_{ij}\\times\\log({N\\over df_i})\n",
|
||||
"$$\n",
|
||||
"donde\n",
|
||||
"* $tf_{ij}$ es el número de ocurrencias de $i$ en $j$, es decir, el valor de BoW que hemos visto antes\n",
|
||||
"* $N$ es el número de documentos en la colección\n",
|
||||
"* $df_i$ es el número de documentos que contienen la palabra $i$ en toda la colección\n",
|
||||
"\n",
|
||||
"El valor TF-IDF $w_{ij}$ aumenta proporcionalmente al número de veces que una palabra aparece en un documento y se ajusta por el número de documentos en el corpus que contienen la palabra, lo que ayuda a compensar el hecho de que algunas palabras aparecen con más frecuencia que otras. Por ejemplo, si la palabra aparece en *todos* los documentos de la colección, $df_i=N$, y $w_{ij}=0$, y esos términos serían completamente ignorados.\n",
|
||||
"\n",
|
||||
"Puedes crear fácilmente una vectorización TF-IDF de texto utilizando Scikit Learn:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 16,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([[0.43381609, 0. , 0.43381609, 0. , 0.65985664,\n",
|
||||
" 0.43381609, 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. ]])"
|
||||
]
|
||||
},
|
||||
"execution_count": 16,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
|
||||
"vectorizer = TfidfVectorizer(ngram_range=(1,2))\n",
|
||||
"vectorizer.fit_transform(corpus)\n",
|
||||
"vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"En Keras, la capa `TextVectorization` puede calcular automáticamente las frecuencias TF-IDF pasando el parámetro `output_mode='tf-idf'`. Repitamos el código que usamos anteriormente para ver si usar TF-IDF aumenta la precisión:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 17,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Training vectorizer\n",
|
||||
"938/938 [==============================] - 12s 12ms/step - loss: 0.4197 - acc: 0.8662 - val_loss: 0.3432 - val_acc: 0.8849\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.History at 0x20c729dfd30>"
|
||||
]
|
||||
},
|
||||
"execution_count": 17,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model = keras.models.Sequential([\n",
|
||||
" keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size,output_mode='tf-idf'),\n",
|
||||
" keras.layers.Dense(4,input_shape=(vocab_size,), activation='softmax')\n",
|
||||
"])\n",
|
||||
"print(\"Training vectorizer\")\n",
|
||||
"model.layers[0].adapt(ds_train.take(500).map(extract_text))\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Conclusión\n",
|
||||
"\n",
|
||||
"Aunque las representaciones TF-IDF asignan pesos de frecuencia a diferentes palabras, no son capaces de representar el significado ni el orden. Como dijo el famoso lingüista J. R. Firth en 1935: \"El significado completo de una palabra siempre es contextual, y ningún estudio del significado fuera del contexto puede tomarse en serio\". Más adelante en el curso aprenderemos cómo capturar información contextual del texto utilizando modelos de lenguaje.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Descargo de responsabilidad**: \nEste documento ha sido traducido utilizando el servicio de traducción automática [Co-op Translator](https://github.com/Azure/co-op-translator). Si bien nos esforzamos por garantizar la precisión, tenga en cuenta que las traducciones automatizadas pueden contener errores o imprecisiones. El documento original en su idioma nativo debe considerarse la fuente autorizada. Para información crítica, se recomienda una traducción profesional realizada por humanos. No nos hacemos responsables de malentendidos o interpretaciones erróneas que puedan surgir del uso de esta traducción.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5"
|
||||
},
|
||||
"kernel_info": {
|
||||
"name": "conda-env-py37_tensorflow-py"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "py37_tensorflow",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"nteract": {
|
||||
"version": "nteract-front-end@1.0.0"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "19b43951d55b377a76209c24c1f017e4",
|
||||
"translation_date": "2025-08-31T17:19:28+00:00",
|
||||
"source_file": "lessons/5-NLP/13-TextRep/TextRepresentationTF.ipynb",
|
||||
"language_code": "es"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
|
|
@ -0,0 +1,724 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Embeddings\n",
|
||||
"\n",
|
||||
"En nuestro ejemplo anterior, trabajamos con vectores de bolsa de palabras de alta dimensión con una longitud de `vocab_size`, y estábamos convirtiendo explícitamente desde vectores de representación posicional de baja dimensión a una representación dispersa de una sola posición activa (one-hot). Esta representación de una sola posición activa no es eficiente en términos de memoria, además, cada palabra se trata de manera independiente, es decir, los vectores codificados en una sola posición activa no expresan ninguna similitud semántica entre las palabras.\n",
|
||||
"\n",
|
||||
"En esta unidad, continuaremos explorando el dataset **News AG**. Para comenzar, carguemos los datos y obtengamos algunas definiciones del cuaderno anterior.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Loading dataset...\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"d:\\WORK\\ai-for-beginners\\5-NLP\\14-Embeddings\\data\\train.csv: 29.5MB [00:01, 18.8MB/s] \n",
|
||||
"d:\\WORK\\ai-for-beginners\\5-NLP\\14-Embeddings\\data\\test.csv: 1.86MB [00:00, 11.2MB/s] \n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Building vocab...\n",
|
||||
"Vocab size = 95812\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"import torchtext\n",
|
||||
"import numpy as np\n",
|
||||
"from torchnlp import *\n",
|
||||
"train_dataset, test_dataset, classes, vocab = load_dataset()\n",
|
||||
"vocab_size = len(vocab)\n",
|
||||
"print(\"Vocab size = \",vocab_size)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## ¿Qué es el embedding?\n",
|
||||
"\n",
|
||||
"La idea del **embedding** es representar palabras mediante vectores densos de menor dimensión, que de alguna manera reflejen el significado semántico de una palabra. Más adelante discutiremos cómo construir embeddings de palabras significativos, pero por ahora pensemos en los embeddings simplemente como una forma de reducir la dimensionalidad de un vector de palabras.\n",
|
||||
"\n",
|
||||
"Entonces, una capa de embedding tomaría una palabra como entrada y produciría un vector de salida con un `embedding_size` especificado. En cierto sentido, es muy similar a una capa `Linear`, pero en lugar de tomar un vector codificado en one-hot, podrá tomar un número de palabra como entrada.\n",
|
||||
"\n",
|
||||
"Al usar una capa de embedding como la primera capa en nuestra red, podemos cambiar del modelo de bolsa de palabras (**bag-of-words**) al modelo de **bolsa de embeddings** (**embedding bag**), donde primero convertimos cada palabra de nuestro texto en su embedding correspondiente, y luego calculamos alguna función de agregación sobre todos esos embeddings, como `sum`, `average` o `max`.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Nuestra red neuronal clasificador comenzará con una capa de embedding, luego una capa de agregación, y finalmente un clasificador lineal en la parte superior:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class EmbedClassifier(torch.nn.Module):\n",
|
||||
" def __init__(self, vocab_size, embed_dim, num_class):\n",
|
||||
" super().__init__()\n",
|
||||
" self.embedding = torch.nn.Embedding(vocab_size, embed_dim)\n",
|
||||
" self.fc = torch.nn.Linear(embed_dim, num_class)\n",
|
||||
"\n",
|
||||
" def forward(self, x):\n",
|
||||
" x = self.embedding(x)\n",
|
||||
" x = torch.mean(x,dim=1)\n",
|
||||
" return self.fc(x)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Cómo manejar el tamaño variable de las secuencias\n",
|
||||
"\n",
|
||||
"Como resultado de esta arquitectura, los minibatches para nuestra red necesitarán ser creados de una manera específica. En la unidad anterior, al usar bag-of-words, todos los tensores BoW en un minibatch tenían el mismo tamaño `vocab_size`, independientemente de la longitud real de nuestra secuencia de texto. Una vez que pasamos a las incrustaciones de palabras, terminaremos con un número variable de palabras en cada muestra de texto, y al combinar esas muestras en minibatches tendremos que aplicar algún tipo de relleno.\n",
|
||||
"\n",
|
||||
"Esto se puede hacer utilizando la misma técnica de proporcionar la función `collate_fn` a la fuente de datos:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def padify(b):\n",
|
||||
" # b is the list of tuples of length batch_size\n",
|
||||
" # - first element of a tuple = label, \n",
|
||||
" # - second = feature (text sequence)\n",
|
||||
" # build vectorized sequence\n",
|
||||
" v = [encode(x[1]) for x in b]\n",
|
||||
" # first, compute max length of a sequence in this minibatch\n",
|
||||
" l = max(map(len,v))\n",
|
||||
" return ( # tuple of two tensors - labels and features\n",
|
||||
" torch.LongTensor([t[0]-1 for t in b]),\n",
|
||||
" torch.stack([torch.nn.functional.pad(torch.tensor(t),(0,l-len(t)),mode='constant',value=0) for t in v])\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=padify, shuffle=True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Entrenando el clasificador de incrustaciones\n",
|
||||
"\n",
|
||||
"Ahora que hemos definido un dataloader adecuado, podemos entrenar el modelo utilizando la función de entrenamiento que definimos en la unidad anterior:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.6415625\n",
|
||||
"6400: acc=0.6865625\n",
|
||||
"9600: acc=0.7103125\n",
|
||||
"12800: acc=0.726953125\n",
|
||||
"16000: acc=0.739375\n",
|
||||
"19200: acc=0.75046875\n",
|
||||
"22400: acc=0.7572321428571429\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(0.889799795315499, 0.7623160588611644)"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"net = EmbedClassifier(vocab_size,32,len(classes)).to(device)\n",
|
||||
"train_epoch(net,train_loader, lr=1, epoch_size=25000)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Nota**: Aquí solo estamos entrenando con 25k registros (menos de una época completa) por cuestiones de tiempo, pero puedes continuar entrenando, escribir una función para entrenar durante varias épocas y experimentar con el parámetro de la tasa de aprendizaje para lograr una mayor precisión. Deberías poder alcanzar una precisión de alrededor del 90%.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Capa EmbeddingBag y Representación de Secuencias de Longitud Variable\n",
|
||||
"\n",
|
||||
"En la arquitectura anterior, necesitábamos rellenar todas las secuencias para que tuvieran la misma longitud y así poder ajustarlas en un minibatch. Esta no es la forma más eficiente de representar secuencias de longitud variable; otra opción sería usar un vector de **desplazamientos** (offset), que contendría los desplazamientos de todas las secuencias almacenadas en un único vector grande.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"> **Nota**: En la imagen de arriba, mostramos una secuencia de caracteres, pero en nuestro ejemplo estamos trabajando con secuencias de palabras. Sin embargo, el principio general de representar secuencias con un vector de desplazamientos sigue siendo el mismo.\n",
|
||||
"\n",
|
||||
"Para trabajar con la representación basada en desplazamientos, usamos la capa [`EmbeddingBag`](https://pytorch.org/docs/stable/generated/torch.nn.EmbeddingBag.html). Es similar a `Embedding`, pero toma como entrada un vector de contenido y un vector de desplazamientos, e incluye además una capa de promediado, que puede ser `mean`, `sum` o `max`.\n",
|
||||
"\n",
|
||||
"Aquí está la red modificada que utiliza `EmbeddingBag`:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class EmbedClassifier(torch.nn.Module):\n",
|
||||
" def __init__(self, vocab_size, embed_dim, num_class):\n",
|
||||
" super().__init__()\n",
|
||||
" self.embedding = torch.nn.EmbeddingBag(vocab_size, embed_dim)\n",
|
||||
" self.fc = torch.nn.Linear(embed_dim, num_class)\n",
|
||||
"\n",
|
||||
" def forward(self, text, off):\n",
|
||||
" x = self.embedding(text, off)\n",
|
||||
" return self.fc(x)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Para preparar el conjunto de datos para el entrenamiento, necesitamos proporcionar una función de conversión que prepare el vector de desplazamiento:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def offsetify(b):\n",
|
||||
" # first, compute data tensor from all sequences\n",
|
||||
" x = [torch.tensor(encode(t[1])) for t in b]\n",
|
||||
" # now, compute the offsets by accumulating the tensor of sequence lengths\n",
|
||||
" o = [0] + [len(t) for t in x]\n",
|
||||
" o = torch.tensor(o[:-1]).cumsum(dim=0)\n",
|
||||
" return ( \n",
|
||||
" torch.LongTensor([t[0]-1 for t in b]), # labels\n",
|
||||
" torch.cat(x), # text \n",
|
||||
" o\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=offsetify, shuffle=True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Nota que, a diferencia de todos los ejemplos anteriores, nuestra red ahora acepta dos parámetros: vector de datos y vector de desplazamiento, que son de diferentes tamaños. De manera similar, nuestro cargador de datos también nos proporciona 3 valores en lugar de 2: tanto los vectores de texto como los vectores de desplazamiento se proporcionan como características. Por lo tanto, necesitamos ajustar ligeramente nuestra función de entrenamiento para encargarnos de eso:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.6153125\n",
|
||||
"6400: acc=0.6615625\n",
|
||||
"9600: acc=0.6932291666666667\n",
|
||||
"12800: acc=0.715078125\n",
|
||||
"16000: acc=0.7270625\n",
|
||||
"19200: acc=0.7382291666666667\n",
|
||||
"22400: acc=0.7486160714285715\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(22.771553103007037, 0.7551983365323096)"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"net = EmbedClassifier(vocab_size,32,len(classes)).to(device)\n",
|
||||
"\n",
|
||||
"def train_epoch_emb(net,dataloader,lr=0.01,optimizer=None,loss_fn = torch.nn.CrossEntropyLoss(),epoch_size=None, report_freq=200):\n",
|
||||
" optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr)\n",
|
||||
" loss_fn = loss_fn.to(device)\n",
|
||||
" net.train()\n",
|
||||
" total_loss,acc,count,i = 0,0,0,0\n",
|
||||
" for labels,text,off in dataloader:\n",
|
||||
" optimizer.zero_grad()\n",
|
||||
" labels,text,off = labels.to(device), text.to(device), off.to(device)\n",
|
||||
" out = net(text, off)\n",
|
||||
" loss = loss_fn(out,labels) #cross_entropy(out,labels)\n",
|
||||
" loss.backward()\n",
|
||||
" optimizer.step()\n",
|
||||
" total_loss+=loss\n",
|
||||
" _,predicted = torch.max(out,1)\n",
|
||||
" acc+=(predicted==labels).sum()\n",
|
||||
" count+=len(labels)\n",
|
||||
" i+=1\n",
|
||||
" if i%report_freq==0:\n",
|
||||
" print(f\"{count}: acc={acc.item()/count}\")\n",
|
||||
" if epoch_size and count>epoch_size:\n",
|
||||
" break\n",
|
||||
" return total_loss.item()/count, acc.item()/count\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"train_epoch_emb(net,train_loader, lr=4, epoch_size=25000)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Embeddings Semánticos: Word2Vec\n",
|
||||
"\n",
|
||||
"En nuestro ejemplo anterior, la capa de embeddings del modelo aprendió a mapear palabras a representaciones vectoriales, sin embargo, esta representación no tenía mucho significado semántico. Sería ideal aprender una representación vectorial en la que palabras similares o sinónimos correspondan a vectores cercanos entre sí en términos de alguna distancia vectorial (por ejemplo, distancia euclidiana).\n",
|
||||
"\n",
|
||||
"Para lograr esto, necesitamos preentrenar nuestro modelo de embeddings en una gran colección de texto de una manera específica. Una de las primeras formas de entrenar embeddings semánticos se llama [Word2Vec](https://en.wikipedia.org/wiki/Word2vec). Se basa en dos arquitecturas principales que se utilizan para producir una representación distribuida de palabras:\n",
|
||||
"\n",
|
||||
" - **Continuous bag-of-words** (CBoW) — en esta arquitectura, entrenamos el modelo para predecir una palabra a partir del contexto circundante. Dado el ngrama $(W_{-2},W_{-1},W_0,W_1,W_2)$, el objetivo del modelo es predecir $W_0$ a partir de $(W_{-2},W_{-1},W_1,W_2)$.\n",
|
||||
" - **Continuous skip-gram** es lo opuesto a CBoW. El modelo utiliza una ventana de palabras de contexto circundantes para predecir la palabra actual.\n",
|
||||
"\n",
|
||||
"CBoW es más rápido, mientras que skip-gram es más lento, pero hace un mejor trabajo representando palabras poco frecuentes.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Para experimentar con embeddings Word2Vec preentrenados en el conjunto de datos de Google News, podemos usar la biblioteca **gensim**. A continuación, encontramos las palabras más similares a 'neural'.\n",
|
||||
"\n",
|
||||
"> **Nota:** ¡Cuando creas vectores de palabras por primera vez, descargarlos puede tomar algo de tiempo!\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import gensim.downloader as api\n",
|
||||
"w2v = api.load('word2vec-google-news-300')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"neuronal -> 0.7804799675941467\n",
|
||||
"neurons -> 0.7326500415802002\n",
|
||||
"neural_circuits -> 0.7252851724624634\n",
|
||||
"neuron -> 0.7174385190010071\n",
|
||||
"cortical -> 0.6941086649894714\n",
|
||||
"brain_circuitry -> 0.6923246383666992\n",
|
||||
"synaptic -> 0.6699118614196777\n",
|
||||
"neural_circuitry -> 0.6638563275337219\n",
|
||||
"neurochemical -> 0.6555314064025879\n",
|
||||
"neuronal_activity -> 0.6531826257705688\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"for w,p in w2v.most_similar('neural'):\n",
|
||||
" print(f\"{w} -> {p}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Podemos también calcular incrustaciones vectoriales a partir de la palabra, para ser utilizadas en el entrenamiento del modelo de clasificación (solo mostramos los primeros 20 componentes del vector para mayor claridad):\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([ 0.01226807, 0.06225586, 0.10693359, 0.05810547, 0.23828125,\n",
|
||||
" 0.03686523, 0.05151367, -0.20703125, 0.01989746, 0.10058594,\n",
|
||||
" -0.03759766, -0.1015625 , -0.15820312, -0.08105469, -0.0390625 ,\n",
|
||||
" -0.05053711, 0.16015625, 0.2578125 , 0.10058594, -0.25976562],\n",
|
||||
" dtype=float32)"
|
||||
]
|
||||
},
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"w2v.word_vec('play')[:20]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Lo grandioso de las incrustaciones semánticas es que puedes manipular la codificación vectorial para cambiar la semántica. Por ejemplo, podemos pedir encontrar una palabra cuya representación vectorial sea lo más cercana posible a las palabras *rey* y *mujer*, y lo más alejada posible de la palabra *hombre*:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"('queen', 0.7118192911148071)"
|
||||
]
|
||||
},
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"w2v.most_similar(positive=['king','woman'],negative=['man'])[0]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Tanto CBoW como Skip-Grams son incrustaciones \"predictivas\", ya que solo toman en cuenta los contextos locales. Word2Vec no aprovecha el contexto global.\n",
|
||||
"\n",
|
||||
"**FastText** se basa en Word2Vec al aprender representaciones vectoriales para cada palabra y los n-gramas de caracteres que se encuentran dentro de cada palabra. Los valores de las representaciones se promedian en un vector en cada paso de entrenamiento. Aunque esto agrega mucha computación adicional al pre-entrenamiento, permite que las incrustaciones de palabras codifiquen información de sub-palabras.\n",
|
||||
"\n",
|
||||
"Otro método, **GloVe**, aprovecha la idea de la matriz de co-ocurrencia y utiliza métodos neuronales para descomponer la matriz de co-ocurrencia en vectores de palabras más expresivos y no lineales.\n",
|
||||
"\n",
|
||||
"Puedes experimentar con el ejemplo cambiando las incrustaciones a FastText y GloVe, ya que gensim admite varios modelos diferentes de incrustaciones de palabras.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Uso de Embeddings Preentrenados en PyTorch\n",
|
||||
"\n",
|
||||
"Podemos modificar el ejemplo anterior para prellenar la matriz en nuestra capa de embedding con embeddings semánticos, como Word2Vec. Debemos tener en cuenta que los vocabularios del embedding preentrenado y nuestro corpus de texto probablemente no coincidan, por lo que inicializaremos los pesos de las palabras faltantes con valores aleatorios:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Embedding size: 300\n",
|
||||
"Populating matrix, this will take some time...Done, found 41080 words, 54732 words missing\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"embed_size = len(w2v.get_vector('hello'))\n",
|
||||
"print(f'Embedding size: {embed_size}')\n",
|
||||
"\n",
|
||||
"net = EmbedClassifier(vocab_size,embed_size,len(classes))\n",
|
||||
"\n",
|
||||
"print('Populating matrix, this will take some time...',end='')\n",
|
||||
"found, not_found = 0,0\n",
|
||||
"for i,w in enumerate(vocab.get_itos()):\n",
|
||||
" try:\n",
|
||||
" net.embedding.weight[i].data = torch.tensor(w2v.get_vector(w))\n",
|
||||
" found+=1\n",
|
||||
" except:\n",
|
||||
" net.embedding.weight[i].data = torch.normal(0.0,1.0,(embed_size,))\n",
|
||||
" not_found+=1\n",
|
||||
"\n",
|
||||
"print(f\"Done, found {found} words, {not_found} words missing\")\n",
|
||||
"net = net.to(device)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Ahora entrenemos nuestro modelo. Ten en cuenta que el tiempo que lleva entrenar el modelo es significativamente mayor que en el ejemplo anterior, debido al tamaño más grande de la capa de incrustación y, por lo tanto, a un número mucho mayor de parámetros. Además, debido a esto, es posible que necesitemos entrenar nuestro modelo con más ejemplos si queremos evitar el sobreajuste.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.6359375\n",
|
||||
"6400: acc=0.68109375\n",
|
||||
"9600: acc=0.7067708333333333\n",
|
||||
"12800: acc=0.723671875\n",
|
||||
"16000: acc=0.73625\n",
|
||||
"19200: acc=0.7463541666666667\n",
|
||||
"22400: acc=0.7560714285714286\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(214.1013875559821, 0.7626759436980166)"
|
||||
]
|
||||
},
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"train_epoch_emb(net,train_loader, lr=4, epoch_size=25000)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"En nuestro caso, no observamos un gran aumento en la precisión, lo cual probablemente se deba a vocabularios bastante diferentes. \n",
|
||||
"Para superar el problema de los vocabularios distintos, podemos usar una de las siguientes soluciones: \n",
|
||||
"* Reentrenar el modelo word2vec con nuestro vocabulario \n",
|
||||
"* Cargar nuestro conjunto de datos utilizando el vocabulario del modelo word2vec preentrenado. El vocabulario usado para cargar el conjunto de datos puede especificarse durante la carga. \n",
|
||||
"\n",
|
||||
"El último enfoque parece más sencillo, especialmente porque el marco `torchtext` de PyTorch contiene soporte integrado para embeddings. Por ejemplo, podemos instanciar un vocabulario basado en GloVe de la siguiente manera: \n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"100%|█████████▉| 399999/400000 [00:15<00:00, 25411.14it/s]\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vocab = torchtext.vocab.GloVe(name='6B', dim=50)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"El vocabulario cargado tiene las siguientes operaciones básicas: \n",
|
||||
"* El diccionario `vocab.stoi` nos permite convertir una palabra en su índice dentro del diccionario. \n",
|
||||
"* `vocab.itos` hace lo contrario: convierte un número en una palabra. \n",
|
||||
"* `vocab.vectors` es el array de vectores de incrustación, por lo que, para obtener la incrustación de una palabra `s`, necesitamos usar `vocab.vectors[vocab.stoi[s]]`. \n",
|
||||
"\n",
|
||||
"Aquí tienes un ejemplo de cómo manipular las incrustaciones para demostrar la ecuación **amable-hombre+mujer = reina** (tuve que ajustar un poco el coeficiente para que funcionara): \n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"'queen'"
|
||||
]
|
||||
},
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# get the vector corresponding to kind-man+woman\n",
|
||||
"qvec = vocab.vectors[vocab.stoi['king']]-vocab.vectors[vocab.stoi['man']]+1.3*vocab.vectors[vocab.stoi['woman']]\n",
|
||||
"# find the index of the closest embedding vector \n",
|
||||
"d = torch.sum((vocab.vectors-qvec)**2,dim=1)\n",
|
||||
"min_idx = torch.argmin(d)\n",
|
||||
"# find the corresponding word\n",
|
||||
"vocab.itos[min_idx]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Para entrenar el clasificador utilizando esos embeddings, primero necesitamos codificar nuestro conjunto de datos utilizando el vocabulario de GloVe:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 16,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def offsetify(b):\n",
|
||||
" # first, compute data tensor from all sequences\n",
|
||||
" x = [torch.tensor(encode(t[1],voc=vocab)) for t in b] # pass the instance of vocab to encode function!\n",
|
||||
" # now, compute the offsets by accumulating the tensor of sequence lengths\n",
|
||||
" o = [0] + [len(t) for t in x]\n",
|
||||
" o = torch.tensor(o[:-1]).cumsum(dim=0)\n",
|
||||
" return ( \n",
|
||||
" torch.LongTensor([t[0]-1 for t in b]), # labels\n",
|
||||
" torch.cat(x), # text \n",
|
||||
" o\n",
|
||||
" )"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Como hemos visto anteriormente, todas las incrustaciones de vectores se almacenan en la matriz `vocab.vectors`. Esto hace que sea muy fácil cargar esos pesos en los pesos de la capa de incrustación mediante una simple copia:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 17,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"net = EmbedClassifier(len(vocab),len(vocab.vectors[0]),len(classes))\n",
|
||||
"net.embedding.weight.data = vocab.vectors\n",
|
||||
"net = net.to(device)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 18,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.6271875\n",
|
||||
"6400: acc=0.68078125\n",
|
||||
"9600: acc=0.7030208333333333\n",
|
||||
"12800: acc=0.71984375\n",
|
||||
"16000: acc=0.7346875\n",
|
||||
"19200: acc=0.7455729166666667\n",
|
||||
"22400: acc=0.7529464285714286\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(35.53972978646833, 0.7575175943698017)"
|
||||
]
|
||||
},
|
||||
"execution_count": 18,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=offsetify, shuffle=True)\n",
|
||||
"train_epoch_emb(net,train_loader, lr=4, epoch_size=25000)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Una de las razones por las que no estamos viendo un aumento significativo en la precisión se debe al hecho de que algunas palabras de nuestro conjunto de datos faltan en el vocabulario preentrenado de GloVe, y por lo tanto, esencialmente se ignoran. Para superar este hecho, podemos entrenar nuestras propias incrustaciones en nuestro conjunto de datos.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Embeddings Contextuales\n",
|
||||
"\n",
|
||||
"Una limitación clave de las representaciones tradicionales de embeddings preentrenados, como Word2Vec, es el problema de la desambiguación del sentido de las palabras. Aunque los embeddings preentrenados pueden capturar parte del significado de las palabras en contexto, todos los posibles significados de una palabra se codifican en el mismo embedding. Esto puede generar problemas en los modelos posteriores, ya que muchas palabras, como la palabra 'play', tienen diferentes significados dependiendo del contexto en el que se utilicen.\n",
|
||||
"\n",
|
||||
"Por ejemplo, la palabra 'play' en estas dos frases tiene significados bastante diferentes:\n",
|
||||
"- Fui a una **obra** en el teatro.\n",
|
||||
"- John quiere **jugar** con sus amigos.\n",
|
||||
"\n",
|
||||
"Los embeddings preentrenados mencionados anteriormente representan ambos significados de la palabra 'play' en el mismo embedding. Para superar esta limitación, necesitamos construir embeddings basados en el **modelo de lenguaje**, que se entrena con un gran corpus de texto y *sabe* cómo las palabras pueden combinarse en diferentes contextos. Hablar de embeddings contextuales está fuera del alcance de este tutorial, pero volveremos a ellos cuando hablemos de modelos de lenguaje en la próxima unidad.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Descargo de responsabilidad**: \nEste documento ha sido traducido utilizando el servicio de traducción automática [Co-op Translator](https://github.com/Azure/co-op-translator). Aunque nos esforzamos por garantizar la precisión, tenga en cuenta que las traducciones automatizadas pueden contener errores o imprecisiones. El documento original en su idioma nativo debe considerarse como la fuente autorizada. Para información crítica, se recomienda una traducción profesional realizada por humanos. No nos hacemos responsables de malentendidos o interpretaciones erróneas que puedan surgir del uso de esta traducción.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "py37_pytorch",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "f50b026abce5cf36783a560ea72cb9b1",
|
||||
"translation_date": "2025-08-31T17:14:26+00:00",
|
||||
"source_file": "lessons/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb",
|
||||
"language_code": "es"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
|
|
@ -0,0 +1,695 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Representaciones vectoriales (Embeddings)\n",
|
||||
"\n",
|
||||
"En nuestro ejemplo anterior, trabajamos con vectores de bolsa de palabras (bag-of-words) de alta dimensión con una longitud de `vocab_size`, y convertimos explícitamente vectores de representación posicional de baja dimensión en representaciones dispersas de tipo one-hot. Esta representación one-hot no es eficiente en términos de memoria. Además, cada palabra se trata de manera independiente, por lo que los vectores codificados en one-hot no expresan similitudes semánticas entre palabras.\n",
|
||||
"\n",
|
||||
"En esta unidad, continuaremos explorando el conjunto de datos **News AG**. Para comenzar, carguemos los datos y obtengamos algunas definiciones de la unidad anterior.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import tensorflow as tf\n",
|
||||
"from tensorflow import keras\n",
|
||||
"import tensorflow_datasets as tfds\n",
|
||||
"import numpy as np\n",
|
||||
"\n",
|
||||
"ds_train, ds_test = tfds.load('ag_news_subset').values()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### ¿Qué es un embedding?\n",
|
||||
"\n",
|
||||
"La idea de un **embedding** es representar palabras utilizando vectores densos de menor dimensión que reflejen el significado semántico de la palabra. Más adelante discutiremos cómo construir embeddings de palabras significativos, pero por ahora pensemos en los embeddings como una forma de reducir la dimensionalidad de un vector de palabras.\n",
|
||||
"\n",
|
||||
"Entonces, una capa de embedding toma una palabra como entrada y produce un vector de salida con un `embedding_size` especificado. En cierto sentido, es muy similar a una capa `Dense`, pero en lugar de tomar un vector codificado en one-hot como entrada, puede tomar un número que representa la palabra.\n",
|
||||
"\n",
|
||||
"Al usar una capa de embedding como la primera capa en nuestra red, podemos cambiar de un modelo de bolsa de palabras a un modelo de **embedding bag**, donde primero convertimos cada palabra de nuestro texto en el embedding correspondiente y luego calculamos alguna función de agregación sobre todos esos embeddings, como `sum`, `average` o `max`.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Nuestra red neuronal clasificador consta de las siguientes capas:\n",
|
||||
"\n",
|
||||
"* Capa `TextVectorization`, que toma una cadena como entrada y produce un tensor de números de tokens. Especificaremos un tamaño de vocabulario razonable `vocab_size` e ignoraremos las palabras que se usan con menos frecuencia. La forma de entrada será 1, y la forma de salida será $n$, ya que obtendremos $n$ tokens como resultado, cada uno de ellos conteniendo números entre 0 y `vocab_size`.\n",
|
||||
"* Capa `Embedding`, que toma $n$ números y reduce cada número a un vector denso de una longitud dada (100 en nuestro ejemplo). Así, el tensor de entrada con forma $n$ se transformará en un tensor de $n\\times 100$.\n",
|
||||
"* Capa de agregación, que toma el promedio de este tensor a lo largo del primer eje, es decir, calculará el promedio de todos los $n$ tensores de entrada correspondientes a diferentes palabras. Para implementar esta capa, usaremos una capa `Lambda` y le pasaremos la función para calcular el promedio. La salida tendrá una forma de 100 y será la representación numérica de toda la secuencia de entrada.\n",
|
||||
"* Clasificador lineal final con una capa `Dense`.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"sequential\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
" Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
" text_vectorization (TextVec (None, None) 0 \n",
|
||||
" torization) \n",
|
||||
" \n",
|
||||
" embedding (Embedding) (None, None, 100) 3000000 \n",
|
||||
" \n",
|
||||
" lambda (Lambda) (None, 100) 0 \n",
|
||||
" \n",
|
||||
" dense (Dense) (None, 4) 404 \n",
|
||||
" \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 3,000,404\n",
|
||||
"Trainable params: 3,000,404\n",
|
||||
"Non-trainable params: 0\n",
|
||||
"_________________________________________________________________\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vocab_size = 30000\n",
|
||||
"batch_size = 128\n",
|
||||
"\n",
|
||||
"vectorizer = keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size,input_shape=(1,))\n",
|
||||
"\n",
|
||||
"model = keras.models.Sequential([\n",
|
||||
" vectorizer, \n",
|
||||
" keras.layers.Embedding(vocab_size,100),\n",
|
||||
" keras.layers.Lambda(lambda x: tf.reduce_mean(x,axis=1)),\n",
|
||||
" keras.layers.Dense(4, activation='softmax')\n",
|
||||
"])\n",
|
||||
"model.summary()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"En el resumen impreso, en la columna **forma de salida**, la primera dimensión del tensor `None` corresponde al tamaño del minibatch, y la segunda corresponde a la longitud de la secuencia de tokens. Todas las secuencias de tokens en el minibatch tienen longitudes diferentes. Hablaremos sobre cómo manejar esto en la próxima sección.\n",
|
||||
"\n",
|
||||
"Ahora entrenemos la red:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Training vectorizer\n",
|
||||
"938/938 [==============================] - 20s 20ms/step - loss: 0.7891 - acc: 0.8155 - val_loss: 0.4470 - val_acc: 0.8642\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.History at 0x22255515100>"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def extract_text(x):\n",
|
||||
" return x['title']+' '+x['description']\n",
|
||||
"\n",
|
||||
"def tupelize(x):\n",
|
||||
" return (extract_text(x),x['label'])\n",
|
||||
"\n",
|
||||
"print(\"Training vectorizer\")\n",
|
||||
"vectorizer.adapt(ds_train.take(500).map(extract_text))\n",
|
||||
"\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'])\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"nteract": {
|
||||
"transient": {
|
||||
"deleting": false
|
||||
}
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"> **Nota** que estamos construyendo el vectorizador basado en un subconjunto de los datos. Esto se hace para acelerar el proceso, y podría resultar en una situación en la que no todos los tokens de nuestro texto estén presentes en el vocabulario. En este caso, esos tokens serían ignorados, lo que podría resultar en una precisión ligeramente menor. Sin embargo, en la vida real, un subconjunto de texto a menudo proporciona una buena estimación del vocabulario.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Cómo manejar tamaños variables de secuencias\n",
|
||||
"\n",
|
||||
"Vamos a entender cómo ocurre el entrenamiento en minibatches. En el ejemplo anterior, el tensor de entrada tiene dimensión 1, y usamos minibatches de tamaño 128, por lo que el tamaño real del tensor es $128 \\times 1$. Sin embargo, el número de tokens en cada oración es diferente. Si aplicamos la capa `TextVectorization` a una sola entrada, el número de tokens devueltos será diferente, dependiendo de cómo se tokenice el texto:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"tf.Tensor([ 1 45], shape=(2,), dtype=int64)\n",
|
||||
"tf.Tensor([ 112 1271 1 3 1747 158], shape=(6,), dtype=int64)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print(vectorizer('Hello, world!'))\n",
|
||||
"print(vectorizer('I am glad to meet you!'))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Sin embargo, cuando aplicamos el vectorizador a varias secuencias, tiene que producir un tensor de forma rectangular, por lo que llena los elementos no utilizados con el token PAD (que en nuestro caso es cero):\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tf.Tensor: shape=(2, 6), dtype=int64, numpy=\n",
|
||||
"array([[ 1, 45, 0, 0, 0, 0],\n",
|
||||
" [ 112, 1271, 1, 3, 1747, 158]], dtype=int64)>"
|
||||
]
|
||||
},
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vectorizer(['Hello, world!','I am glad to meet you!'])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Aquí podemos ver las incrustaciones:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([[[ 1.53059261e-02, 6.80514947e-02, 3.14026810e-02, ...,\n",
|
||||
" -8.92002955e-02, 1.52911525e-04, -5.65562584e-02],\n",
|
||||
" [ 2.57456154e-01, 2.79364467e-01, -2.03605562e-01, ...,\n",
|
||||
" -2.07474351e-01, 8.31158683e-02, -2.03911960e-01],\n",
|
||||
" [ 3.98201384e-02, -8.03454965e-03, 2.39790026e-02, ...,\n",
|
||||
" -7.18549127e-04, 2.66963355e-02, -4.30646613e-02],\n",
|
||||
" [ 3.98201384e-02, -8.03454965e-03, 2.39790026e-02, ...,\n",
|
||||
" -7.18549127e-04, 2.66963355e-02, -4.30646613e-02],\n",
|
||||
" [ 3.98201384e-02, -8.03454965e-03, 2.39790026e-02, ...,\n",
|
||||
" -7.18549127e-04, 2.66963355e-02, -4.30646613e-02],\n",
|
||||
" [ 3.98201384e-02, -8.03454965e-03, 2.39790026e-02, ...,\n",
|
||||
" -7.18549127e-04, 2.66963355e-02, -4.30646613e-02]],\n",
|
||||
"\n",
|
||||
" [[ 1.89674050e-01, 2.61548996e-01, -3.67433839e-02, ...,\n",
|
||||
" -2.07366899e-01, -1.05442435e-01, -2.36952081e-01],\n",
|
||||
" [ 6.16133213e-02, 1.80511594e-01, 9.77298319e-02, ...,\n",
|
||||
" -5.46628237e-02, -1.07340455e-01, -1.06589928e-01],\n",
|
||||
" [ 1.53059261e-02, 6.80514947e-02, 3.14026810e-02, ...,\n",
|
||||
" -8.92002955e-02, 1.52911525e-04, -5.65562584e-02],\n",
|
||||
" [-4.84890305e-02, -8.41715634e-02, 1.51529670e-01, ...,\n",
|
||||
" 1.28192469e-01, -7.77286515e-02, 1.26041949e-01],\n",
|
||||
" [-4.17212099e-02, -5.60694858e-02, 4.08860669e-02, ...,\n",
|
||||
" 8.70475471e-02, 8.92383084e-02, 1.67974353e-01],\n",
|
||||
" [ 2.85779923e-01, 4.57767487e-01, 4.52292450e-02, ...,\n",
|
||||
" -1.97419018e-01, -2.04659685e-01, -2.79758364e-01]]],\n",
|
||||
" dtype=float32)"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.layers[1](vectorizer(['Hello, world!','I am glad to meet you!'])).numpy()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Nota**: Para minimizar la cantidad de relleno, en algunos casos tiene sentido ordenar todas las secuencias del conjunto de datos en orden de longitud creciente (o, más precisamente, por número de tokens). Esto asegurará que cada minibatch contenga secuencias de longitud similar.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Incrustaciones semánticas: Word2Vec\n",
|
||||
"\n",
|
||||
"En nuestro ejemplo anterior, la capa de incrustación aprendió a mapear palabras a representaciones vectoriales; sin embargo, estas representaciones no tenían un significado semántico. Sería ideal aprender una representación vectorial en la que palabras similares o sinónimos correspondan a vectores que estén cerca unos de otros en términos de alguna distancia vectorial (por ejemplo, la distancia euclidiana).\n",
|
||||
"\n",
|
||||
"Para lograr esto, necesitamos preentrenar nuestro modelo de incrustación en una gran colección de texto utilizando una técnica como [Word2Vec](https://en.wikipedia.org/wiki/Word2vec). Está basado en dos arquitecturas principales que se utilizan para producir una representación distribuida de palabras:\n",
|
||||
"\n",
|
||||
" - **Bolsa de palabras continua** (CBoW), donde entrenamos el modelo para predecir una palabra a partir del contexto circundante. Dado el n-grama $(W_{-2},W_{-1},W_0,W_1,W_2)$, el objetivo del modelo es predecir $W_0$ a partir de $(W_{-2},W_{-1},W_1,W_2)$.\n",
|
||||
" - **Skip-gram continuo** es lo opuesto a CBoW. El modelo utiliza la ventana de palabras de contexto circundantes para predecir la palabra actual.\n",
|
||||
"\n",
|
||||
"CBoW es más rápido, y aunque skip-gram es más lento, hace un mejor trabajo representando palabras poco frecuentes.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Para experimentar con la incrustación Word2Vec preentrenada en el conjunto de datos de Google News, podemos usar la biblioteca **gensim**. A continuación, encontramos las palabras más similares a 'neural'.\n",
|
||||
"\n",
|
||||
"> **Nota:** ¡Cuando creas vectores de palabras por primera vez, descargarlos puede tomar algo de tiempo!\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import gensim.downloader as api\n",
|
||||
"w2v = api.load('word2vec-google-news-300')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"neuronal -> 0.7804799675941467\n",
|
||||
"neurons -> 0.7326500415802002\n",
|
||||
"neural_circuits -> 0.7252851724624634\n",
|
||||
"neuron -> 0.7174385190010071\n",
|
||||
"cortical -> 0.6941086649894714\n",
|
||||
"brain_circuitry -> 0.6923246383666992\n",
|
||||
"synaptic -> 0.6699118614196777\n",
|
||||
"neural_circuitry -> 0.6638563275337219\n",
|
||||
"neurochemical -> 0.6555314064025879\n",
|
||||
"neuronal_activity -> 0.6531826257705688\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"for w,p in w2v.most_similar('neural'):\n",
|
||||
" print(f\"{w} -> {p}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"También podemos extraer la incrustación vectorial de la palabra, para usarla en el entrenamiento del modelo de clasificación. La incrustación tiene 300 componentes, pero aquí solo mostramos los primeros 20 componentes del vector para mayor claridad:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([ 0.01226807, 0.06225586, 0.10693359, 0.05810547, 0.23828125,\n",
|
||||
" 0.03686523, 0.05151367, -0.20703125, 0.01989746, 0.10058594,\n",
|
||||
" -0.03759766, -0.1015625 , -0.15820312, -0.08105469, -0.0390625 ,\n",
|
||||
" -0.05053711, 0.16015625, 0.2578125 , 0.10058594, -0.25976562],\n",
|
||||
" dtype=float32)"
|
||||
]
|
||||
},
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"w2v['play'][:20]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Lo grandioso de las incrustaciones semánticas es que puedes manipular la codificación vectorial basada en la semántica. Por ejemplo, podemos pedir encontrar una palabra cuya representación vectorial esté lo más cerca posible de las palabras *rey* y *mujer*, y lo más lejos posible de la palabra *hombre*:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"('queen', 0.7118192911148071)"
|
||||
]
|
||||
},
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"w2v.most_similar(positive=['king','woman'],negative=['man'])[0]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"Un ejemplo anterior utiliza algo de magia interna de GenSym, pero la lógica subyacente es realmente bastante simple. Una cosa interesante sobre las incrustaciones es que puedes realizar operaciones normales de vectores en los vectores de incrustación, y eso reflejaría operaciones en los **significados** de las palabras. El ejemplo anterior puede expresarse en términos de operaciones vectoriales: calculamos el vector correspondiente a **REY-HOMBRE+MUJER** (las operaciones `+` y `-` se realizan en las representaciones vectoriales de las palabras correspondientes), y luego encontramos la palabra más cercana en el diccionario a ese vector:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"'queen'"
|
||||
]
|
||||
},
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# get the vector corresponding to kind-man+woman\n",
|
||||
"qvec = w2v['king']-1.7*w2v['man']+1.7*w2v['woman']\n",
|
||||
"# find the index of the closest embedding vector \n",
|
||||
"d = np.sum((w2v.vectors-qvec)**2,axis=1)\n",
|
||||
"min_idx = np.argmin(d)\n",
|
||||
"# find the corresponding word\n",
|
||||
"w2v.index_to_key[min_idx]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **NOTA**: Tuvimos que añadir pequeños coeficientes a los vectores de *man* y *woman* - intenta eliminarlos para ver qué sucede.\n",
|
||||
"\n",
|
||||
"Para encontrar el vector más cercano, utilizamos la maquinaria de TensorFlow para calcular un vector de distancias entre nuestro vector y todos los vectores en el vocabulario, y luego encontramos el índice de la palabra mínima usando `argmin`.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Aunque Word2Vec parece una excelente manera de expresar la semántica de las palabras, tiene muchas desventajas, entre las que se incluyen las siguientes:\n",
|
||||
"\n",
|
||||
"* Tanto los modelos CBoW como skip-gram son **embeddings predictivos**, y solo toman en cuenta el contexto local. Word2Vec no aprovecha el contexto global.\n",
|
||||
"* Word2Vec no considera la **morfología** de las palabras, es decir, el hecho de que el significado de una palabra puede depender de diferentes partes de la misma, como la raíz.\n",
|
||||
"\n",
|
||||
"**FastText** intenta superar la segunda limitación y se basa en Word2Vec al aprender representaciones vectoriales para cada palabra y los n-gramas de caracteres que se encuentran dentro de cada palabra. Los valores de estas representaciones se promedian en un único vector en cada paso de entrenamiento. Aunque esto añade una gran cantidad de cálculo adicional al preentrenamiento, permite que los embeddings de palabras codifiquen información a nivel de subpalabras.\n",
|
||||
"\n",
|
||||
"Otro método, **GloVe**, utiliza un enfoque diferente para los embeddings de palabras, basado en la factorización de la matriz de contexto de palabras. Primero, construye una gran matriz que cuenta el número de apariciones de palabras en diferentes contextos, y luego intenta representar esta matriz en dimensiones más bajas de una manera que minimice la pérdida de reconstrucción.\n",
|
||||
"\n",
|
||||
"La biblioteca gensim admite estos embeddings de palabras, y puedes experimentar con ellos cambiando el código de carga del modelo mencionado anteriormente.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Usar embeddings preentrenados en Keras\n",
|
||||
"\n",
|
||||
"Podemos modificar el ejemplo anterior para prellenar la matriz en nuestra capa de embedding con embeddings semánticos, como Word2Vec. Es probable que los vocabularios del embedding preentrenado y del corpus de texto no coincidan, por lo que debemos elegir uno. Aquí exploramos las dos opciones posibles: usar el vocabulario del tokenizer y usar el vocabulario de los embeddings de Word2Vec.\n",
|
||||
"\n",
|
||||
"### Usar el vocabulario del tokenizer\n",
|
||||
"\n",
|
||||
"Al usar el vocabulario del tokenizer, algunas palabras del vocabulario tendrán embeddings correspondientes de Word2Vec, y otras estarán ausentes. Dado que el tamaño de nuestro vocabulario es `vocab_size`, y la longitud del vector de embedding de Word2Vec es `embed_size`, la capa de embedding estará representada por una matriz de pesos con forma `vocab_size`$\\times$`embed_size`. Poblaremos esta matriz recorriendo el vocabulario:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Embedding size: 300\n",
|
||||
"Populating matrix, this will take some time...Done, found 4551 words, 784 words missing\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"embed_size = len(w2v.get_vector('hello'))\n",
|
||||
"print(f'Embedding size: {embed_size}')\n",
|
||||
"\n",
|
||||
"vocab = vectorizer.get_vocabulary()\n",
|
||||
"W = np.zeros((vocab_size,embed_size))\n",
|
||||
"print('Populating matrix, this will take some time...',end='')\n",
|
||||
"found, not_found = 0,0\n",
|
||||
"for i,w in enumerate(vocab):\n",
|
||||
" try:\n",
|
||||
" W[i] = w2v.get_vector(w)\n",
|
||||
" found+=1\n",
|
||||
" except:\n",
|
||||
" # W[i] = np.random.normal(0.0,0.3,size=(embed_size,))\n",
|
||||
" not_found+=1\n",
|
||||
"\n",
|
||||
"print(f\"Done, found {found} words, {not_found} words missing\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Para las palabras que no están presentes en el vocabulario de Word2Vec, podemos dejarlas como ceros o generar un vector aleatorio.\n",
|
||||
"\n",
|
||||
"Ahora podemos definir una capa de incrustación con pesos preentrenados:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"emb = keras.layers.Embedding(vocab_size,embed_size,weights=[W],trainable=False)\n",
|
||||
"model = keras.models.Sequential([\n",
|
||||
" vectorizer, emb,\n",
|
||||
" keras.layers.Lambda(lambda x: tf.reduce_mean(x,axis=1)),\n",
|
||||
" keras.layers.Dense(4, activation='softmax')\n",
|
||||
"])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"938/938 [==============================] - 10s 10ms/step - loss: 1.1075 - acc: 0.7822 - val_loss: 0.9134 - val_acc: 0.8175\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.History at 0x2220226ef10>"
|
||||
]
|
||||
},
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'])\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(batch_size),\n",
|
||||
" validation_data=ds_test.map(tupelize).batch(batch_size))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Nota**: Observa que configuramos `trainable=False` al crear el `Embedding`, lo que significa que no estamos reentrenando la capa de Embedding. Esto puede causar que la precisión sea ligeramente menor, pero acelera el entrenamiento.\n",
|
||||
"\n",
|
||||
"### Usando el vocabulario de embeddings\n",
|
||||
"\n",
|
||||
"Un problema con el enfoque anterior es que los vocabularios utilizados en TextVectorization y Embedding son diferentes. Para resolver este problema, podemos usar una de las siguientes soluciones:\n",
|
||||
"* Reentrenar el modelo Word2Vec con nuestro vocabulario.\n",
|
||||
"* Cargar nuestro conjunto de datos utilizando el vocabulario del modelo Word2Vec preentrenado. Los vocabularios utilizados para cargar el conjunto de datos pueden especificarse durante la carga.\n",
|
||||
"\n",
|
||||
"El último enfoque parece más sencillo, así que vamos a implementarlo. Primero, crearemos una capa `TextVectorization` con el vocabulario especificado, tomado de los embeddings de Word2Vec:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"vocab = list(w2v.vocab.keys())\n",
|
||||
"vectorizer = keras.layers.experimental.preprocessing.TextVectorization(input_shape=(1,))\n",
|
||||
"vectorizer.set_vocabulary(vocab)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"La biblioteca de incrustaciones de palabras gensim contiene una función conveniente, `get_keras_embeddings`, que creará automáticamente la capa de incrustaciones correspondiente de Keras para ti.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch 1/5\n",
|
||||
"938/938 [==============================] - 20s 14ms/step - loss: 1.3377 - acc: 0.4978 - val_loss: 1.2995 - val_acc: 0.5647\n",
|
||||
"Epoch 2/5\n",
|
||||
"938/938 [==============================] - 10s 10ms/step - loss: 1.2587 - acc: 0.5722 - val_loss: 1.2339 - val_acc: 0.5842\n",
|
||||
"Epoch 3/5\n",
|
||||
"938/938 [==============================] - 10s 10ms/step - loss: 1.1980 - acc: 0.5884 - val_loss: 1.1826 - val_acc: 0.5954\n",
|
||||
"Epoch 4/5\n",
|
||||
"938/938 [==============================] - 12s 13ms/step - loss: 1.1503 - acc: 0.6002 - val_loss: 1.1417 - val_acc: 0.6018\n",
|
||||
"Epoch 5/5\n",
|
||||
"938/938 [==============================] - 11s 12ms/step - loss: 1.1120 - acc: 0.6097 - val_loss: 1.1083 - val_acc: 0.6104\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.History at 0x2220ccb81c0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model = keras.models.Sequential([\n",
|
||||
" vectorizer, \n",
|
||||
" w2v.get_keras_embedding(train_embeddings=False),\n",
|
||||
" keras.layers.Lambda(lambda x: tf.reduce_mean(x,axis=1)),\n",
|
||||
" keras.layers.Dense(4, activation='softmax')\n",
|
||||
"])\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'])\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(128),validation_data=ds_test.map(tupelize).batch(128),epochs=5)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Una de las razones por las que no estamos viendo una mayor precisión es porque algunas palabras de nuestro conjunto de datos faltan en el vocabulario preentrenado de GloVe y, por lo tanto, se ignoran esencialmente. Para superar esto, podemos entrenar nuestras propias incrustaciones basadas en nuestro conjunto de datos.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Representaciones contextuales\n",
|
||||
"\n",
|
||||
"Una limitación clave de las representaciones tradicionales de embeddings preentrenados, como Word2Vec, es el hecho de que, aunque pueden capturar parte del significado de una palabra, no pueden diferenciar entre distintos significados. Esto puede causar problemas en los modelos posteriores.\n",
|
||||
"\n",
|
||||
"Por ejemplo, la palabra 'play' tiene diferentes significados en estas dos oraciones:\n",
|
||||
"- Fui a una **obra** en el teatro.\n",
|
||||
"- John quiere **jugar** con sus amigos.\n",
|
||||
"\n",
|
||||
"Los embeddings preentrenados de los que hablamos representan ambos significados de la palabra 'play' en el mismo embedding. Para superar esta limitación, necesitamos construir embeddings basados en el **modelo de lenguaje**, que está entrenado en un gran corpus de texto y *sabe* cómo se pueden combinar las palabras en diferentes contextos. Hablar de embeddings contextuales está fuera del alcance de este tutorial, pero volveremos a ellos cuando hablemos de modelos de lenguaje en la próxima unidad.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Descargo de responsabilidad**: \nEste documento ha sido traducido utilizando el servicio de traducción automática [Co-op Translator](https://github.com/Azure/co-op-translator). Aunque nos esforzamos por garantizar la precisión, tenga en cuenta que las traducciones automatizadas pueden contener errores o imprecisiones. El documento original en su idioma nativo debe considerarse como la fuente autorizada. Para información crítica, se recomienda una traducción profesional realizada por humanos. No nos hacemos responsables de malentendidos o interpretaciones erróneas que puedan surgir del uso de esta traducción.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5"
|
||||
},
|
||||
"kernel_info": {
|
||||
"name": "conda-env-py37_tensorflow-py"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "py37_tensorflow",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"nteract": {
|
||||
"version": "nteract-front-end@1.0.0"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "b859482be7f61d1eadc2c6a2720a37e4",
|
||||
"translation_date": "2025-08-31T17:11:27+00:00",
|
||||
"source_file": "lessons/5-NLP/14-Embeddings/EmbeddingsTF.ipynb",
|
||||
"language_code": "es"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
|
|
@ -0,0 +1,576 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "NXTSugt6ieXh"
|
||||
},
|
||||
"source": [
|
||||
"## Entrenando el Modelo CBoW\n",
|
||||
"\n",
|
||||
"Este cuaderno forma parte del [Currículo de IA para Principiantes](http://aka.ms/ai-beginners)\n",
|
||||
"\n",
|
||||
"En este ejemplo, exploraremos cómo entrenar un modelo de lenguaje CBoW para obtener nuestro propio espacio de incrustación Word2Vec. Utilizaremos el conjunto de datos AG News como fuente de texto.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"import torchtext\n",
|
||||
"import os\n",
|
||||
"import collections\n",
|
||||
"import builtins\n",
|
||||
"import random\n",
|
||||
"import numpy as np"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "q-UiiJUKaxHj"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "TFbR8CZaTZ1q"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Primero carguemos nuestro conjunto de datos y definamos el tokenizador y el vocabulario. Estableceremos `vocab_size` en 5000 para limitar un poco los cálculos.\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "HIwC7lI5T-ov"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def load_dataset(ngrams = 1, min_freq = 1, vocab_size = 5000 , lines_cnt = 500):\n",
|
||||
" tokenizer = torchtext.data.utils.get_tokenizer('basic_english')\n",
|
||||
" print(\"Loading dataset...\")\n",
|
||||
" test_dataset, train_dataset = torchtext.datasets.AG_NEWS(root='./data')\n",
|
||||
" train_dataset = list(train_dataset)\n",
|
||||
" test_dataset = list(test_dataset)\n",
|
||||
" classes = ['World', 'Sports', 'Business', 'Sci/Tech']\n",
|
||||
" print('Building vocab...')\n",
|
||||
" counter = collections.Counter()\n",
|
||||
" for i, (_, line) in enumerate(train_dataset):\n",
|
||||
" counter.update(torchtext.data.utils.ngrams_iterator(tokenizer(line),ngrams=ngrams))\n",
|
||||
" if i == lines_cnt:\n",
|
||||
" break\n",
|
||||
" vocab = torchtext.vocab.Vocab(collections.Counter(dict(counter.most_common(vocab_size))), min_freq=min_freq)\n",
|
||||
" return train_dataset, test_dataset, classes, vocab, tokenizer"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "wdZuygtgiuLG"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"train_dataset, test_dataset, _, vocab, tokenizer = load_dataset()"
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "4d1nU1gsivGu",
|
||||
"outputId": "949fe272-ae0e-49f5-c373-6703458b3a74"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
"Loading dataset...\n",
|
||||
"Building vocab...\n"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def encode(x, vocabulary, tokenizer = tokenizer):\n",
|
||||
" return [vocabulary[s] for s in tokenizer(x)]"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "1XDYNhG8ToFV"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "LIlQk6_PaHVY"
|
||||
},
|
||||
"source": [
|
||||
"## Modelo CBoW\n",
|
||||
"\n",
|
||||
"CBoW aprende a predecir una palabra basada en las $2N$ palabras vecinas. Por ejemplo, cuando $N=1$, obtendremos los siguientes pares de la oración *I like to train networks*: (like,I), (I, like), (to, like), (like,to), (train,to), (to, train), (networks, train), (train,networks). Aquí, la primera palabra es la palabra vecina utilizada como entrada, y la segunda palabra es la que estamos prediciendo.\n",
|
||||
"\n",
|
||||
"Para construir una red que prediga la siguiente palabra, necesitaremos proporcionar la palabra vecina como entrada y obtener el número de palabra como salida. La arquitectura de la red CBoW es la siguiente:\n",
|
||||
"\n",
|
||||
"* La palabra de entrada pasa a través de la capa de embeddings. Esta misma capa de embeddings será nuestro embedding de Word2Vec, por lo que la definiremos por separado como la variable `embedder`. Usaremos un tamaño de embedding = 30 en este ejemplo, aunque podrías querer experimentar con dimensiones más altas (el Word2Vec real tiene 300).\n",
|
||||
"* El vector de embedding luego se pasa a una capa lineal que predecirá la palabra de salida. Por lo tanto, tiene las neuronas de `vocab_size`.\n",
|
||||
"\n",
|
||||
"Para la salida, si usamos `CrossEntropyLoss` como función de pérdida, también tendremos que proporcionar solo los números de las palabras como resultados esperados, sin codificación one-hot.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"vocab_size = len(vocab)\n",
|
||||
"\n",
|
||||
"embedder = torch.nn.Embedding(num_embeddings = vocab_size, embedding_dim = 30)\n",
|
||||
"model = torch.nn.Sequential(\n",
|
||||
" embedder,\n",
|
||||
" torch.nn.Linear(in_features = 30, out_features = vocab_size),\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print(model)"
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "akKTcKQKkfl2",
|
||||
"outputId": "da687e3e-a8ec-4c1a-e456-ab8cd6ac7dad"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
"Sequential(\n",
|
||||
" (0): Embedding(5002, 30)\n",
|
||||
" (1): Linear(in_features=30, out_features=5002, bias=True)\n",
|
||||
")\n"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "Nud6jgGPaHVa"
|
||||
},
|
||||
"source": [
|
||||
"## Preparando los Datos de Entrenamiento\n",
|
||||
"\n",
|
||||
"Ahora programemos la función principal que calculará los pares de palabras CBoW a partir del texto. Esta función nos permitirá especificar el tamaño de la ventana y devolverá un conjunto de pares: palabra de entrada y palabra de salida. Ten en cuenta que esta función puede usarse tanto con palabras como con vectores/tensores, lo que nos permitirá codificar el texto antes de pasarlo a la función `to_cbow`.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "x-dsXygOieXn",
|
||||
"outputId": "c2218280-e540-40ba-9546-efe48d0d714f"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
"[['like', 'I'], ['to', 'I'], ['I', 'like'], ['to', 'like'], ['train', 'like'], ['I', 'to'], ['like', 'to'], ['train', 'to'], ['networks', 'to'], ['like', 'train'], ['to', 'train'], ['networks', 'train'], ['to', 'networks'], ['train', 'networks']]\n",
|
||||
"[[232, 172], [5, 172], [172, 232], [5, 232], [0, 232], [172, 5], [232, 5], [0, 5], [1202, 5], [232, 0], [5, 0], [1202, 0], [5, 1202], [0, 1202]]\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def to_cbow(sent,window_size=2):\n",
|
||||
" res = []\n",
|
||||
" for i,x in enumerate(sent):\n",
|
||||
" for j in range(max(0,i-window_size),min(i+window_size+1,len(sent))):\n",
|
||||
" if i!=j:\n",
|
||||
" res.append([sent[j],x])\n",
|
||||
" return res\n",
|
||||
"\n",
|
||||
"print(to_cbow(['I','like','to','train','networks']))\n",
|
||||
"print(to_cbow(encode('I like to train networks', vocab)))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "XVaaDLjaaHVb"
|
||||
},
|
||||
"source": [
|
||||
"Vamos a preparar el conjunto de datos de entrenamiento. Revisaremos todas las noticias, llamaremos a `to_cbow` para obtener la lista de pares de palabras y añadiremos esos pares a `X` y `Y`. Por cuestiones de tiempo, solo consideraremos las primeras 10k noticias; puedes eliminar fácilmente esta limitación si tienes más tiempo para esperar y deseas obtener mejores incrustaciones :)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "54b-Gd9TieXo"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"X = []\n",
|
||||
"Y = []\n",
|
||||
"for i, x in zip(range(10000), train_dataset):\n",
|
||||
" for w1, w2 in to_cbow(encode(x[1], vocab), window_size = 5):\n",
|
||||
" X.append(w1)\n",
|
||||
" Y.append(w2)\n",
|
||||
"\n",
|
||||
"X = torch.tensor(X)\n",
|
||||
"Y = torch.tensor(Y)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"También convertiremos esos datos en un conjunto de datos y crearemos un cargador de datos:\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "cwWy0PzXWhN5"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"class SimpleIterableDataset(torch.utils.data.IterableDataset):\n",
|
||||
" def __init__(self, X, Y):\n",
|
||||
" super(SimpleIterableDataset).__init__()\n",
|
||||
" self.data = []\n",
|
||||
" for i in range(len(X)):\n",
|
||||
" self.data.append( (Y[i], X[i]) )\n",
|
||||
" random.shuffle(self.data)\n",
|
||||
"\n",
|
||||
" def __iter__(self):\n",
|
||||
" return iter(self.data)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "mfoAcGPFZU8p"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "e4NQ_-5waHVc"
|
||||
},
|
||||
"source": [
|
||||
"También convertiremos esos datos en un conjunto de datos y crearemos un cargador de datos:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "AbLUcojlieXo"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"ds = SimpleIterableDataset(X, Y)\n",
|
||||
"dl = torch.utils.data.DataLoader(ds, batch_size = 256)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "pKQr7sXeaHVc"
|
||||
},
|
||||
"source": [
|
||||
"Ahora hagamos el entrenamiento real. Usaremos el optimizador `SGD` con una tasa de aprendizaje bastante alta. También puedes probar con otros optimizadores, como `Adam`. Entrenaremos durante 10 épocas para comenzar, y puedes volver a ejecutar esta celda si deseas una pérdida aún menor.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def train_epoch(net, dataloader, lr = 0.01, optimizer = None, loss_fn = torch.nn.CrossEntropyLoss(), epochs = None, report_freq = 1):\n",
|
||||
" optimizer = optimizer or torch.optim.Adam(net.parameters(), lr = lr)\n",
|
||||
" loss_fn = loss_fn.to(device)\n",
|
||||
" net.train()\n",
|
||||
"\n",
|
||||
" for i in range(epochs):\n",
|
||||
" total_loss, j = 0, 0, \n",
|
||||
" for labels, features in dataloader:\n",
|
||||
" optimizer.zero_grad()\n",
|
||||
" features, labels = features.to(device), labels.to(device)\n",
|
||||
" out = net(features)\n",
|
||||
" loss = loss_fn(out, labels)\n",
|
||||
" loss.backward()\n",
|
||||
" optimizer.step()\n",
|
||||
" total_loss += loss\n",
|
||||
" j += 1\n",
|
||||
" if i % report_freq == 0:\n",
|
||||
" print(f\"Epoch: {i+1}: loss={total_loss.item()/j}\")\n",
|
||||
"\n",
|
||||
" return total_loss.item()/j"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "HeeCYKr_KF1w"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"train_epoch(net = model, dataloader = dl, optimizer = torch.optim.SGD(model.parameters(), lr = 0.1), loss_fn = torch.nn.CrossEntropyLoss(), epochs = 10)"
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "KVgwGtDHgDlT",
|
||||
"outputId": "2447833f-f0e3-4566-c33d-addbfe2f451d"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
"Epoch: 1: loss=5.664632366860172\n",
|
||||
"Epoch: 2: loss=5.632101973960962\n",
|
||||
"Epoch: 3: loss=5.610399051405015\n",
|
||||
"Epoch: 4: loss=5.594621561080262\n",
|
||||
"Epoch: 5: loss=5.582538017415446\n",
|
||||
"Epoch: 6: loss=5.572900234519603\n",
|
||||
"Epoch: 7: loss=5.564951676341915\n",
|
||||
"Epoch: 8: loss=5.558288112064614\n",
|
||||
"Epoch: 9: loss=5.552576955031129\n",
|
||||
"Epoch: 10: loss=5.547634165194347\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"output_type": "execute_result",
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"5.547634165194347"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"execution_count": 16
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "W8u2qXZmaHVd"
|
||||
},
|
||||
"source": [
|
||||
"## Probando Word2Vec\n",
|
||||
"\n",
|
||||
"Para usar Word2Vec, extraigamos los vectores correspondientes a todas las palabras de nuestro vocabulario:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "r8TatcXjkU_t"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"vectors = torch.stack([embedder(torch.tensor(vocab[s])) for s in vocab.itos], 0)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "3OcX21UOaHVd"
|
||||
},
|
||||
"source": [
|
||||
"Veamos, por ejemplo, cómo la palabra **París** se codifica en un vector:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "bz6tAeLzieXp",
|
||||
"outputId": "5b20850e-4342-45e9-f840-cfac2b4d61d8"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
"tensor([-0.0915, 2.1224, -0.0281, -0.6819, 1.1219, 0.6458, -1.3704, -1.3314,\n",
|
||||
" -1.1437, 0.4496, 0.2301, -0.3515, -0.8485, 1.0481, 0.4386, -0.8949,\n",
|
||||
" 0.5644, 1.0939, -2.5096, 3.2949, -0.2601, -0.8640, 0.1421, -0.0804,\n",
|
||||
" -0.5083, -1.0560, 0.9753, -0.5949, -1.6046, 0.5774],\n",
|
||||
" grad_fn=<EmbeddingBackward>)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"paris_vec = embedder(torch.tensor(vocab['paris']))\n",
|
||||
"print(paris_vec)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "pHTJlaeYaHVd"
|
||||
},
|
||||
"source": [
|
||||
"Es interesante usar Word2Vec para buscar sinónimos. La siguiente función devolverá las `n` palabras más cercanas a una entrada dada. Para encontrarlas, calculamos la norma de $|w_i - v|$, donde $v$ es el vector correspondiente a nuestra palabra de entrada, y $w_i$ es la codificación de la palabra $i$-ésima en el vocabulario. Luego ordenamos el arreglo y devolvemos los índices correspondientes usando `argsort`, y tomamos los primeros `n` elementos de la lista, que codifican las posiciones de las palabras más cercanas en el vocabulario.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "NlZyi-_olFar",
|
||||
"outputId": "b5dbb163-88c4-4d5a-eaf2-6751f700e98c"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "execute_result",
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"['microsoft', 'quoted', 'lp', 'rate', 'top']"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"execution_count": 56
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def close_words(x, n = 5):\n",
|
||||
" vec = embedder(torch.tensor(vocab[x]))\n",
|
||||
" top5 = np.linalg.norm(vectors.detach().numpy() - vec.detach().numpy(), axis = 1).argsort()[:n]\n",
|
||||
" return [ vocab.itos[x] for x in top5 ]\n",
|
||||
"\n",
|
||||
"close_words('microsoft')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "-dQq7xeAln0U",
|
||||
"outputId": "66f768c3-c248-4bfd-ce4f-c8ffc6d0dd0d"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "execute_result",
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"['basketball', 'lot', 'sinai', 'states', 'healthdaynews']"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"execution_count": 51
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"close_words('basketball')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "fJXqK26b29sa",
|
||||
"outputId": "78f0baba-ffd0-485a-dd87-0a12bedfd7fa"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "execute_result",
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"['funds', 'travel', 'sydney', 'japan', 'business']"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"execution_count": 77
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"close_words('funds')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "My0VeTDd3Ji8"
|
||||
},
|
||||
"source": [
|
||||
"## Conclusión\n",
|
||||
"\n",
|
||||
"Utilizando técnicas ingeniosas como CBoW, podemos entrenar el modelo Word2Vec. También puedes intentar entrenar un modelo skip-gram, que se entrena para predecir la palabra vecina dada la palabra central, y observar qué tan bien funciona.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Descargo de responsabilidad**: \nEste documento ha sido traducido utilizando el servicio de traducción automática [Co-op Translator](https://github.com/Azure/co-op-translator). Si bien nos esforzamos por garantizar la precisión, tenga en cuenta que las traducciones automatizadas pueden contener errores o imprecisiones. El documento original en su idioma nativo debe considerarse como la fuente autorizada. Para información crítica, se recomienda una traducción profesional realizada por humanos. No nos hacemos responsables de malentendidos o interpretaciones erróneas que puedan surgir del uso de esta traducción.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"collapsed_sections": [],
|
||||
"name": "CBoW-PyTorch.ipynb",
|
||||
"provenance": []
|
||||
},
|
||||
"interpreter": {
|
||||
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.8.12 ('py38')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"orig_nbformat": 4,
|
||||
"gpuClass": "standard",
|
||||
"coopTranslator": {
|
||||
"original_hash": "36df28efe3fe40b6fb0a7fa48fe3ea82",
|
||||
"translation_date": "2025-08-31T16:55:21+00:00",
|
||||
"source_file": "lessons/5-NLP/15-LanguageModeling/CBoW-PyTorch.ipynb",
|
||||
"language_code": "es"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,479 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Redes neuronales recurrentes\n",
|
||||
"\n",
|
||||
"En el módulo anterior, hemos estado utilizando representaciones semánticas ricas del texto y un clasificador lineal simple sobre las incrustaciones. Lo que hace esta arquitectura es capturar el significado agregado de las palabras en una oración, pero no tiene en cuenta el **orden** de las palabras, ya que la operación de agregación sobre las incrustaciones elimina esta información del texto original. Debido a que estos modelos no pueden modelar el orden de las palabras, no pueden resolver tareas más complejas o ambiguas como la generación de texto o la respuesta a preguntas.\n",
|
||||
"\n",
|
||||
"Para capturar el significado de una secuencia de texto, necesitamos usar otra arquitectura de red neuronal, llamada **red neuronal recurrente**, o RNN. En una RNN, pasamos nuestra oración a través de la red un símbolo a la vez, y la red produce un **estado**, que luego pasamos nuevamente a la red junto con el siguiente símbolo.\n",
|
||||
"\n",
|
||||
"Dada la secuencia de tokens de entrada $X_0,\\dots,X_n$, la RNN crea una secuencia de bloques de red neuronal y entrena esta secuencia de extremo a extremo utilizando retropropagación. Cada bloque de red toma un par $(X_i,S_i)$ como entrada y produce $S_{i+1}$ como resultado. El estado final $S_n$ o la salida $X_n$ se pasa a un clasificador lineal para producir el resultado. Todos los bloques de red comparten los mismos pesos y se entrenan de extremo a extremo utilizando una sola pasada de retropropagación.\n",
|
||||
"\n",
|
||||
"Debido a que los vectores de estado $S_0,\\dots,S_n$ se pasan a través de la red, esta es capaz de aprender las dependencias secuenciales entre palabras. Por ejemplo, cuando la palabra *no* aparece en algún lugar de la secuencia, puede aprender a negar ciertos elementos dentro del vector de estado, lo que resulta en una negación.\n",
|
||||
"\n",
|
||||
"> Dado que los pesos de todos los bloques de RNN en la imagen son compartidos, la misma imagen puede representarse como un solo bloque (a la derecha) con un bucle de retroalimentación recurrente, que pasa el estado de salida de la red nuevamente a la entrada.\n",
|
||||
"\n",
|
||||
"Veamos cómo las redes neuronales recurrentes pueden ayudarnos a clasificar nuestro conjunto de datos de noticias.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Loading dataset...\n",
|
||||
"Building vocab...\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"import torchtext\n",
|
||||
"from torchnlp import *\n",
|
||||
"train_dataset, test_dataset, classes, vocab = load_dataset()\n",
|
||||
"vocab_size = len(vocab)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Clasificador RNN simple\n",
|
||||
"\n",
|
||||
"En el caso de una RNN simple, cada unidad recurrente es una red lineal sencilla que toma un vector de entrada concatenado y un vector de estado, y produce un nuevo vector de estado. PyTorch representa esta unidad con la clase `RNNCell`, y una red de dichas celdas como una capa `RNN`.\n",
|
||||
"\n",
|
||||
"Para definir un clasificador RNN, primero aplicaremos una capa de incrustación para reducir la dimensionalidad del vocabulario de entrada, y luego añadiremos una capa RNN encima:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class RNNClassifier(torch.nn.Module):\n",
|
||||
" def __init__(self, vocab_size, embed_dim, hidden_dim, num_class):\n",
|
||||
" super().__init__()\n",
|
||||
" self.hidden_dim = hidden_dim\n",
|
||||
" self.embedding = torch.nn.Embedding(vocab_size, embed_dim)\n",
|
||||
" self.rnn = torch.nn.RNN(embed_dim,hidden_dim,batch_first=True)\n",
|
||||
" self.fc = torch.nn.Linear(hidden_dim, num_class)\n",
|
||||
"\n",
|
||||
" def forward(self, x):\n",
|
||||
" batch_size = x.size(0)\n",
|
||||
" x = self.embedding(x)\n",
|
||||
" x,h = self.rnn(x)\n",
|
||||
" return self.fc(x.mean(dim=1))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Nota:** Aquí usamos una capa de embedding no entrenada por simplicidad, pero para obtener resultados aún mejores podemos usar una capa de embedding preentrenada con embeddings de Word2Vec o GloVe, como se describió en la unidad anterior. Para una mejor comprensión, podrías adaptar este código para trabajar con embeddings preentrenados.\n",
|
||||
"\n",
|
||||
"En nuestro caso, utilizaremos un cargador de datos con padding, de modo que cada lote tendrá un número de secuencias rellenadas con la misma longitud. La capa RNN tomará la secuencia de tensores de embedding y producirá dos salidas: \n",
|
||||
"* $x$ es una secuencia de salidas de las celdas RNN en cada paso \n",
|
||||
"* $h$ es el estado oculto final para el último elemento de la secuencia \n",
|
||||
"\n",
|
||||
"Luego aplicamos un clasificador lineal completamente conectado para obtener el número de clases.\n",
|
||||
"\n",
|
||||
"> **Nota:** Las RNN son bastante difíciles de entrenar, porque una vez que las celdas RNN se despliegan a lo largo de la longitud de la secuencia, el número resultante de capas involucradas en la retropropagación es bastante grande. Por lo tanto, necesitamos seleccionar una tasa de aprendizaje pequeña y entrenar la red en un conjunto de datos más grande para obtener buenos resultados. Esto puede tomar bastante tiempo, por lo que se recomienda usar GPU.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.3090625\n",
|
||||
"6400: acc=0.38921875\n",
|
||||
"9600: acc=0.4590625\n",
|
||||
"12800: acc=0.511953125\n",
|
||||
"16000: acc=0.5506875\n",
|
||||
"19200: acc=0.57921875\n",
|
||||
"22400: acc=0.6070089285714285\n",
|
||||
"25600: acc=0.6304296875\n",
|
||||
"28800: acc=0.6484027777777778\n",
|
||||
"32000: acc=0.66509375\n",
|
||||
"35200: acc=0.6790056818181818\n",
|
||||
"38400: acc=0.6929166666666666\n",
|
||||
"41600: acc=0.7035817307692308\n",
|
||||
"44800: acc=0.7137276785714286\n",
|
||||
"48000: acc=0.72225\n",
|
||||
"51200: acc=0.73001953125\n",
|
||||
"54400: acc=0.7372794117647059\n",
|
||||
"57600: acc=0.7436631944444444\n",
|
||||
"60800: acc=0.7503947368421052\n",
|
||||
"64000: acc=0.75634375\n",
|
||||
"67200: acc=0.7615773809523809\n",
|
||||
"70400: acc=0.7662642045454545\n",
|
||||
"73600: acc=0.7708423913043478\n",
|
||||
"76800: acc=0.7751822916666666\n",
|
||||
"80000: acc=0.7790625\n",
|
||||
"83200: acc=0.7825\n",
|
||||
"86400: acc=0.7858564814814815\n",
|
||||
"89600: acc=0.7890513392857142\n",
|
||||
"92800: acc=0.7920474137931034\n",
|
||||
"96000: acc=0.7952708333333334\n",
|
||||
"99200: acc=0.7982258064516129\n",
|
||||
"102400: acc=0.80099609375\n",
|
||||
"105600: acc=0.8037594696969697\n",
|
||||
"108800: acc=0.8060569852941176\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=padify, shuffle=True)\n",
|
||||
"net = RNNClassifier(vocab_size,64,32,len(classes)).to(device)\n",
|
||||
"train_epoch(net,train_loader, lr=0.001)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Memoria a Largo y Corto Plazo (LSTM)\n",
|
||||
"\n",
|
||||
"Uno de los principales problemas de las RNN clásicas es el llamado problema de los **gradientes que se desvanecen**. Debido a que las RNN se entrenan de extremo a extremo en una sola pasada de retropropagación, tienen dificultades para propagar el error a las primeras capas de la red, y por lo tanto, la red no puede aprender relaciones entre tokens distantes. Una de las formas de evitar este problema es introducir una **gestión explícita del estado** mediante el uso de los llamados **puertas**. Hay dos arquitecturas más conocidas de este tipo: **Memoria a Largo y Corto Plazo** (LSTM) y **Unidad de Relevo Controlada** (GRU).\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"La red LSTM está organizada de una manera similar a las RNN, pero hay dos estados que se pasan de capa en capa: el estado actual $c$ y el vector oculto $h$. En cada unidad, el vector oculto $h_i$ se concatena con la entrada $x_i$, y juntos controlan lo que sucede con el estado $c$ a través de las **puertas**. Cada puerta es una red neuronal con activación sigmoide (salida en el rango $[0,1]$), que puede interpretarse como una máscara bit a bit cuando se multiplica por el vector de estado. Las puertas son las siguientes (de izquierda a derecha en la imagen anterior):\n",
|
||||
"* **Puerta de olvido**: toma el vector oculto y determina qué componentes del vector $c$ necesitamos olvidar y cuáles pasar.\n",
|
||||
"* **Puerta de entrada**: toma información de la entrada y del vector oculto, e inserta esa información en el estado.\n",
|
||||
"* **Puerta de salida**: transforma el estado mediante una capa lineal con activación $\\tanh$, y luego selecciona algunos de sus componentes usando el vector oculto $h_i$ para producir el nuevo estado $c_{i+1}$.\n",
|
||||
"\n",
|
||||
"Los componentes del estado $c$ pueden interpretarse como banderas que se pueden activar o desactivar. Por ejemplo, cuando encontramos un nombre como *Alice* en la secuencia, podríamos asumir que se refiere a un personaje femenino y activar la bandera en el estado que indica que hay un sustantivo femenino en la oración. Más adelante, al encontrar frases como *and Tom*, activaríamos la bandera que indica que hay un sustantivo en plural. Así, manipulando el estado, supuestamente podemos hacer un seguimiento de las propiedades gramaticales de las partes de la oración.\n",
|
||||
"\n",
|
||||
"> **Nota**: Un excelente recurso para entender los detalles internos de las LSTM es este gran artículo [Understanding LSTM Networks](https://colah.github.io/posts/2015-08-Understanding-LSTMs/) de Christopher Olah.\n",
|
||||
"\n",
|
||||
"Aunque la estructura interna de una celda LSTM puede parecer compleja, PyTorch oculta esta implementación dentro de la clase `LSTMCell` y proporciona el objeto `LSTM` para representar toda la capa LSTM. Por lo tanto, la implementación de un clasificador LSTM será bastante similar a la RNN simple que vimos anteriormente:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class LSTMClassifier(torch.nn.Module):\n",
|
||||
" def __init__(self, vocab_size, embed_dim, hidden_dim, num_class):\n",
|
||||
" super().__init__()\n",
|
||||
" self.hidden_dim = hidden_dim\n",
|
||||
" self.embedding = torch.nn.Embedding(vocab_size, embed_dim)\n",
|
||||
" self.embedding.weight.data = torch.randn_like(self.embedding.weight.data)-0.5\n",
|
||||
" self.rnn = torch.nn.LSTM(embed_dim,hidden_dim,batch_first=True)\n",
|
||||
" self.fc = torch.nn.Linear(hidden_dim, num_class)\n",
|
||||
"\n",
|
||||
" def forward(self, x):\n",
|
||||
" batch_size = x.size(0)\n",
|
||||
" x = self.embedding(x)\n",
|
||||
" x,(h,c) = self.rnn(x)\n",
|
||||
" return self.fc(h[-1])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.259375\n",
|
||||
"6400: acc=0.25859375\n",
|
||||
"9600: acc=0.26177083333333334\n",
|
||||
"12800: acc=0.2784375\n",
|
||||
"16000: acc=0.313\n",
|
||||
"19200: acc=0.3528645833333333\n",
|
||||
"22400: acc=0.3965625\n",
|
||||
"25600: acc=0.4385546875\n",
|
||||
"28800: acc=0.4752777777777778\n",
|
||||
"32000: acc=0.505375\n",
|
||||
"35200: acc=0.5326704545454546\n",
|
||||
"38400: acc=0.5557552083333334\n",
|
||||
"41600: acc=0.5760817307692307\n",
|
||||
"44800: acc=0.5954910714285714\n",
|
||||
"48000: acc=0.6118333333333333\n",
|
||||
"51200: acc=0.62681640625\n",
|
||||
"54400: acc=0.6404779411764706\n",
|
||||
"57600: acc=0.6520138888888889\n",
|
||||
"60800: acc=0.662828947368421\n",
|
||||
"64000: acc=0.673546875\n",
|
||||
"67200: acc=0.6831547619047619\n",
|
||||
"70400: acc=0.6917897727272727\n",
|
||||
"73600: acc=0.6997146739130434\n",
|
||||
"76800: acc=0.707109375\n",
|
||||
"80000: acc=0.714075\n",
|
||||
"83200: acc=0.7209134615384616\n",
|
||||
"86400: acc=0.727037037037037\n",
|
||||
"89600: acc=0.7326674107142858\n",
|
||||
"92800: acc=0.7379633620689655\n",
|
||||
"96000: acc=0.7433645833333333\n",
|
||||
"99200: acc=0.7479032258064516\n",
|
||||
"102400: acc=0.752119140625\n",
|
||||
"105600: acc=0.7562405303030303\n",
|
||||
"108800: acc=0.76015625\n",
|
||||
"112000: acc=0.7641339285714286\n",
|
||||
"115200: acc=0.7677777777777778\n",
|
||||
"118400: acc=0.7711233108108108\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(0.03487814127604167, 0.7728)"
|
||||
]
|
||||
},
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"net = LSTMClassifier(vocab_size,64,32,len(classes)).to(device)\n",
|
||||
"train_epoch(net,train_loader, lr=0.001)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Secuencias empaquetadas\n",
|
||||
"\n",
|
||||
"En nuestro ejemplo, tuvimos que rellenar todas las secuencias en el minibatch con vectores de ceros. Aunque esto genera cierto desperdicio de memoria, con las RNN es más crítico que se creen celdas adicionales para los elementos de entrada rellenados, las cuales participan en el entrenamiento pero no contienen información importante. Sería mucho mejor entrenar la RNN únicamente con el tamaño real de la secuencia.\n",
|
||||
"\n",
|
||||
"Para lograr esto, se introduce un formato especial de almacenamiento de secuencias rellenadas en PyTorch. Supongamos que tenemos un minibatch rellenado que se ve así:\n",
|
||||
"```\n",
|
||||
"[[1,2,3,4,5],\n",
|
||||
" [6,7,8,0,0],\n",
|
||||
" [9,0,0,0,0]]\n",
|
||||
"```\n",
|
||||
"Aquí, 0 representa los valores rellenados, y el vector de longitud real de las secuencias de entrada es `[5,3,1]`.\n",
|
||||
"\n",
|
||||
"Para entrenar eficazmente una RNN con secuencias rellenadas, queremos comenzar el entrenamiento del primer grupo de celdas de la RNN con un minibatch grande (`[1,6,9]`), pero luego terminar el procesamiento de la tercera secuencia y continuar el entrenamiento con minibatches más pequeños (`[2,7]`, `[3,8]`), y así sucesivamente. Por lo tanto, una secuencia empaquetada se representa como un solo vector - en nuestro caso `[1,6,9,2,7,3,8,4,5]`, y un vector de longitud (`[5,3,1]`), a partir del cual podemos reconstruir fácilmente el minibatch rellenado original.\n",
|
||||
"\n",
|
||||
"Para generar una secuencia empaquetada, podemos usar la función `torch.nn.utils.rnn.pack_padded_sequence`. Todas las capas recurrentes, incluidas RNN, LSTM y GRU, admiten secuencias empaquetadas como entrada y producen una salida empaquetada, que puede ser decodificada usando `torch.nn.utils.rnn.pad_packed_sequence`.\n",
|
||||
"\n",
|
||||
"Para poder generar una secuencia empaquetada, necesitamos pasar el vector de longitud a la red, y por lo tanto necesitamos una función diferente para preparar los minibatches:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def pad_length(b):\n",
|
||||
" # build vectorized sequence\n",
|
||||
" v = [encode(x[1]) for x in b]\n",
|
||||
" # compute max length of a sequence in this minibatch and length sequence itself\n",
|
||||
" len_seq = list(map(len,v))\n",
|
||||
" l = max(len_seq)\n",
|
||||
" return ( # tuple of three tensors - labels, padded features, length sequence\n",
|
||||
" torch.LongTensor([t[0]-1 for t in b]),\n",
|
||||
" torch.stack([torch.nn.functional.pad(torch.tensor(t),(0,l-len(t)),mode='constant',value=0) for t in v]),\n",
|
||||
" torch.tensor(len_seq)\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"train_loader_len = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=pad_length, shuffle=True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"La red real sería muy similar a `LSTMClassifier` mencionado anteriormente, pero el paso `forward` recibirá tanto el minibatch con padding como el vector de longitudes de las secuencias. Después de calcular la incrustación, calculamos la secuencia empaquetada, la pasamos a la capa LSTM y luego desempaquetamos el resultado.\n",
|
||||
"\n",
|
||||
"> **Nota**: En realidad no usamos el resultado desempaquetado `x`, porque utilizamos la salida de las capas ocultas en los cálculos posteriores. Por lo tanto, podemos eliminar el desempaquetado por completo de este código. La razón por la que lo colocamos aquí es para que puedas modificar este código fácilmente, en caso de que necesites usar la salida de la red en cálculos adicionales.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class LSTMPackClassifier(torch.nn.Module):\n",
|
||||
" def __init__(self, vocab_size, embed_dim, hidden_dim, num_class):\n",
|
||||
" super().__init__()\n",
|
||||
" self.hidden_dim = hidden_dim\n",
|
||||
" self.embedding = torch.nn.Embedding(vocab_size, embed_dim)\n",
|
||||
" self.embedding.weight.data = torch.randn_like(self.embedding.weight.data)-0.5\n",
|
||||
" self.rnn = torch.nn.LSTM(embed_dim,hidden_dim,batch_first=True)\n",
|
||||
" self.fc = torch.nn.Linear(hidden_dim, num_class)\n",
|
||||
"\n",
|
||||
" def forward(self, x, lengths):\n",
|
||||
" batch_size = x.size(0)\n",
|
||||
" x = self.embedding(x)\n",
|
||||
" pad_x = torch.nn.utils.rnn.pack_padded_sequence(x,lengths,batch_first=True,enforce_sorted=False)\n",
|
||||
" pad_x,(h,c) = self.rnn(pad_x)\n",
|
||||
" x, _ = torch.nn.utils.rnn.pad_packed_sequence(pad_x,batch_first=True)\n",
|
||||
" return self.fc(h[-1])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.285625\n",
|
||||
"6400: acc=0.33359375\n",
|
||||
"9600: acc=0.3876041666666667\n",
|
||||
"12800: acc=0.44078125\n",
|
||||
"16000: acc=0.4825\n",
|
||||
"19200: acc=0.5235416666666667\n",
|
||||
"22400: acc=0.5559821428571429\n",
|
||||
"25600: acc=0.58609375\n",
|
||||
"28800: acc=0.6116666666666667\n",
|
||||
"32000: acc=0.63340625\n",
|
||||
"35200: acc=0.6525284090909091\n",
|
||||
"38400: acc=0.668515625\n",
|
||||
"41600: acc=0.6822596153846154\n",
|
||||
"44800: acc=0.6948214285714286\n",
|
||||
"48000: acc=0.7052708333333333\n",
|
||||
"51200: acc=0.71521484375\n",
|
||||
"54400: acc=0.7239889705882353\n",
|
||||
"57600: acc=0.7315277777777778\n",
|
||||
"60800: acc=0.7388486842105263\n",
|
||||
"64000: acc=0.74571875\n",
|
||||
"67200: acc=0.7518303571428572\n",
|
||||
"70400: acc=0.7576988636363636\n",
|
||||
"73600: acc=0.7628940217391305\n",
|
||||
"76800: acc=0.7681510416666667\n",
|
||||
"80000: acc=0.7728125\n",
|
||||
"83200: acc=0.7772235576923077\n",
|
||||
"86400: acc=0.7815393518518519\n",
|
||||
"89600: acc=0.7857700892857142\n",
|
||||
"92800: acc=0.7895043103448276\n",
|
||||
"96000: acc=0.7930520833333333\n",
|
||||
"99200: acc=0.7959072580645161\n",
|
||||
"102400: acc=0.798994140625\n",
|
||||
"105600: acc=0.802064393939394\n",
|
||||
"108800: acc=0.8051378676470589\n",
|
||||
"112000: acc=0.8077857142857143\n",
|
||||
"115200: acc=0.8104600694444445\n",
|
||||
"118400: acc=0.8128293918918919\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(0.029785829671223958, 0.8138166666666666)"
|
||||
]
|
||||
},
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"net = LSTMPackClassifier(vocab_size,64,32,len(classes)).to(device)\n",
|
||||
"train_epoch_emb(net,train_loader_len, lr=0.001,use_pack_sequence=True)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Nota:** Es posible que hayas notado el parámetro `use_pack_sequence` que pasamos a la función de entrenamiento. Actualmente, la función `pack_padded_sequence` requiere que el tensor de la secuencia de longitud esté en el dispositivo CPU, y por lo tanto, la función de entrenamiento necesita evitar mover los datos de la secuencia de longitud a la GPU durante el entrenamiento. Puedes revisar la implementación de la función `train_emb` en el archivo [`torchnlp.py`](../../../../../lessons/5-NLP/16-RNN/torchnlp.py).\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## RNNs bidireccionales y multicapa\n",
|
||||
"\n",
|
||||
"En nuestros ejemplos, todas las redes recurrentes operaban en una sola dirección, desde el inicio de una secuencia hasta el final. Esto parece natural, ya que se asemeja a la forma en que leemos y escuchamos el habla. Sin embargo, dado que en muchos casos prácticos tenemos acceso aleatorio a la secuencia de entrada, podría tener sentido realizar cálculos recurrentes en ambas direcciones. Estas redes se llaman **RNNs bidireccionales**, y se pueden crear pasando el parámetro `bidirectional=True` al constructor de RNN/LSTM/GRU.\n",
|
||||
"\n",
|
||||
"Al trabajar con una red bidireccional, necesitaríamos dos vectores de estado oculto, uno para cada dirección. PyTorch codifica esos vectores como un solo vector de tamaño doble, lo cual es bastante conveniente, porque normalmente pasarías el estado oculto resultante a una capa lineal completamente conectada, y solo tendrías que tener en cuenta este aumento de tamaño al crear la capa.\n",
|
||||
"\n",
|
||||
"Una red recurrente, ya sea unidireccional o bidireccional, captura ciertos patrones dentro de una secuencia y puede almacenarlos en el vector de estado o pasarlos a la salida. Al igual que con las redes convolucionales, podemos construir otra capa recurrente encima de la primera para capturar patrones de nivel superior, construidos a partir de los patrones de bajo nivel extraídos por la primera capa. Esto nos lleva al concepto de **RNN multicapa**, que consiste en dos o más redes recurrentes, donde la salida de la capa anterior se pasa a la siguiente capa como entrada.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"*Imagen tomada de [este maravilloso artículo](https://towardsdatascience.com/from-a-lstm-cell-to-a-multilayer-lstm-network-with-pytorch-2899eb5696f3) por Fernando López*\n",
|
||||
"\n",
|
||||
"PyTorch facilita la construcción de este tipo de redes, ya que solo necesitas pasar el parámetro `num_layers` al constructor de RNN/LSTM/GRU para construir automáticamente varias capas de recurrencia. Esto también significa que el tamaño del vector de estado oculto aumentará proporcionalmente, y deberás tener esto en cuenta al manejar la salida de las capas recurrentes.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## RNNs para otras tareas\n",
|
||||
"\n",
|
||||
"En esta unidad, hemos visto que las RNNs pueden usarse para la clasificación de secuencias, pero de hecho, pueden manejar muchas más tareas, como la generación de texto, la traducción automática y más. Consideraremos esas tareas en la próxima unidad.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Descargo de responsabilidad**: \nEste documento ha sido traducido utilizando el servicio de traducción automática [Co-op Translator](https://github.com/Azure/co-op-translator). Si bien nos esforzamos por garantizar la precisión, tenga en cuenta que las traducciones automatizadas pueden contener errores o imprecisiones. El documento original en su idioma nativo debe considerarse la fuente autorizada. Para información crítica, se recomienda una traducción profesional realizada por humanos. No nos hacemos responsables de malentendidos o interpretaciones erróneas que puedan surgir del uso de esta traducción.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.8.12 ('py38')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "522ee52ae3d5ae933e283286254e9a55",
|
||||
"translation_date": "2025-08-31T17:08:40+00:00",
|
||||
"source_file": "lessons/5-NLP/16-RNN/RNNPyTorch.ipynb",
|
||||
"language_code": "es"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
|
|
@ -0,0 +1,460 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Redes neuronales recurrentes\n",
|
||||
"\n",
|
||||
"En el módulo anterior, cubrimos representaciones semánticas ricas de texto. La arquitectura que hemos estado utilizando captura el significado agregado de las palabras en una oración, pero no toma en cuenta el **orden** de las palabras, ya que la operación de agregación que sigue a las incrustaciones elimina esta información del texto original. Debido a que estos modelos no pueden representar el orden de las palabras, no pueden resolver tareas más complejas o ambiguas como la generación de texto o la respuesta a preguntas.\n",
|
||||
"\n",
|
||||
"Para capturar el significado de una secuencia de texto, utilizaremos una arquitectura de red neuronal llamada **red neuronal recurrente**, o RNN. Al usar una RNN, pasamos nuestra oración a través de la red un token a la vez, y la red produce un **estado**, que luego pasamos nuevamente a la red junto con el siguiente token.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Dada la secuencia de entrada de tokens $X_0,\\dots,X_n$, la RNN crea una secuencia de bloques de red neuronal y entrena esta secuencia de extremo a extremo utilizando retropropagación. Cada bloque de red toma un par $(X_i,S_i)$ como entrada y produce $S_{i+1}$ como resultado. El estado final $S_n$ o la salida $Y_n$ se pasa a un clasificador lineal para producir el resultado. Todos los bloques de red comparten los mismos pesos y se entrenan de extremo a extremo utilizando una única pasada de retropropagación.\n",
|
||||
"\n",
|
||||
"> La figura anterior muestra una red neuronal recurrente en su forma desplegada (a la izquierda) y en una representación recurrente más compacta (a la derecha). Es importante entender que todas las celdas RNN tienen los mismos **pesos compartidos**.\n",
|
||||
"\n",
|
||||
"Debido a que los vectores de estado $S_0,\\dots,S_n$ se pasan a través de la red, la RNN es capaz de aprender dependencias secuenciales entre palabras. Por ejemplo, cuando la palabra *no* aparece en algún lugar de la secuencia, puede aprender a negar ciertos elementos dentro del vector de estado.\n",
|
||||
"\n",
|
||||
"Internamente, cada celda RNN contiene dos matrices de pesos: $W_H$ y $W_I$, y un sesgo $b$. En cada paso de la RNN, dado el input $X_i$ y el estado de entrada $S_i$, el estado de salida se calcula como $S_{i+1} = f(W_H\\times S_i + W_I\\times X_i+b)$, donde $f$ es una función de activación (a menudo $\\tanh$).\n",
|
||||
"\n",
|
||||
"> Para problemas como la generación de texto (que cubriremos en la próxima unidad) o la traducción automática, también queremos obtener algún valor de salida en cada paso de la RNN. En este caso, hay otra matriz $W_O$, y la salida se calcula como $Y_i=f(W_O\\times S_i+b_O)$.\n",
|
||||
"\n",
|
||||
"Veamos cómo las redes neuronales recurrentes pueden ayudarnos a clasificar nuestro conjunto de datos de noticias.\n",
|
||||
"\n",
|
||||
"> Para el entorno de prueba, necesitamos ejecutar la siguiente celda para asegurarnos de que la biblioteca requerida esté instalada y los datos se hayan predescargado. Si estás trabajando localmente, puedes omitir la siguiente celda.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import sys\n",
|
||||
"!{sys.executable} -m pip install --quiet tensorflow_datasets==4.4.0\n",
|
||||
"!cd ~ && wget -q -O - https://mslearntensorflowlp.blob.core.windows.net/data/tfds-ag-news.tgz | tar xz"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import tensorflow as tf\n",
|
||||
"from tensorflow import keras\n",
|
||||
"import tensorflow_datasets as tfds\n",
|
||||
"import numpy as np\n",
|
||||
"\n",
|
||||
"# We are going to be training pretty large models. In order not to face errors, we need\n",
|
||||
"# to set tensorflow option to grow GPU memory allocation when required\n",
|
||||
"physical_devices = tf.config.list_physical_devices('GPU') \n",
|
||||
"if len(physical_devices)>0:\n",
|
||||
" tf.config.experimental.set_memory_growth(physical_devices[0], True)\n",
|
||||
"\n",
|
||||
"ds_train, ds_test = tfds.load('ag_news_subset').values()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"nteract": {
|
||||
"transient": {
|
||||
"deleting": false
|
||||
}
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"Al entrenar modelos grandes, la asignación de memoria de la GPU puede convertirse en un problema. También puede ser necesario experimentar con diferentes tamaños de minibatch para que los datos se ajusten a la memoria de la GPU y, al mismo tiempo, el entrenamiento sea lo suficientemente rápido. Si estás ejecutando este código en tu propia máquina con GPU, puedes experimentar ajustando el tamaño del minibatch para acelerar el entrenamiento.\n",
|
||||
"\n",
|
||||
"> **Nota**: Se sabe que ciertas versiones de los controladores de NVidia no liberan la memoria después de entrenar el modelo. Estamos ejecutando varios ejemplos en este cuaderno, y esto podría agotar la memoria en ciertos entornos, especialmente si estás realizando tus propios experimentos como parte del mismo cuaderno. Si encuentras errores extraños al comenzar a entrenar el modelo, podrías considerar reiniciar el kernel del cuaderno.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {
|
||||
"collapsed": true,
|
||||
"jupyter": {
|
||||
"outputs_hidden": false,
|
||||
"source_hidden": false
|
||||
},
|
||||
"nteract": {
|
||||
"transient": {
|
||||
"deleting": false
|
||||
}
|
||||
}
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"batch_size = 16\n",
|
||||
"embed_size = 64"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Clasificador RNN simple\n",
|
||||
"\n",
|
||||
"En el caso de una RNN simple, cada unidad recurrente es una red lineal sencilla que toma un vector de entrada y un vector de estado, y produce un nuevo vector de estado. En Keras, esto puede representarse mediante la capa `SimpleRNN`.\n",
|
||||
"\n",
|
||||
"Aunque podemos pasar directamente tokens codificados en one-hot a la capa RNN, esto no es una buena idea debido a su alta dimensionalidad. Por lo tanto, utilizaremos una capa de embedding para reducir la dimensionalidad de los vectores de palabras, seguida de una capa RNN y, finalmente, un clasificador `Dense`.\n",
|
||||
"\n",
|
||||
"> **Nota**: En casos donde la dimensionalidad no sea tan alta, por ejemplo, al usar tokenización a nivel de caracteres, podría tener sentido pasar directamente los tokens codificados en one-hot a la celda RNN.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"sequential\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
"Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
"text_vectorization (TextVect (None, None) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"embedding (Embedding) (None, None, 64) 1280000 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"simple_rnn (SimpleRNN) (None, 16) 1296 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense (Dense) (None, 4) 68 \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 1,281,364\n",
|
||||
"Trainable params: 1,281,364\n",
|
||||
"Non-trainable params: 0\n",
|
||||
"_________________________________________________________________\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vocab_size = 20000\n",
|
||||
"\n",
|
||||
"vectorizer = keras.layers.experimental.preprocessing.TextVectorization(\n",
|
||||
" max_tokens=vocab_size,\n",
|
||||
" input_shape=(1,))\n",
|
||||
"\n",
|
||||
"model = keras.models.Sequential([\n",
|
||||
" vectorizer,\n",
|
||||
" keras.layers.Embedding(vocab_size, embed_size),\n",
|
||||
" keras.layers.SimpleRNN(16),\n",
|
||||
" keras.layers.Dense(4,activation='softmax')\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"model.summary()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Nota:** Aquí usamos una capa de embeddings no entrenada por simplicidad, pero para obtener mejores resultados podemos usar una capa de embeddings preentrenada utilizando Word2Vec, como se describió en la unidad anterior. Sería un buen ejercicio para ti adaptar este código para trabajar con embeddings preentrenados.\n",
|
||||
"\n",
|
||||
"Ahora entrenemos nuestra RNN. En general, las RNN son bastante difíciles de entrenar, porque una vez que las celdas de la RNN se despliegan a lo largo de la longitud de la secuencia, el número resultante de capas involucradas en la retropropagación es bastante grande. Por lo tanto, necesitamos seleccionar una tasa de aprendizaje más pequeña y entrenar la red en un conjunto de datos más grande para obtener buenos resultados. Esto puede tomar bastante tiempo, por lo que se recomienda usar una GPU.\n",
|
||||
"\n",
|
||||
"Para acelerar el proceso, solo entrenaremos el modelo RNN con los títulos de las noticias, omitiendo la descripción. Puedes intentar entrenar con la descripción y ver si logras que el modelo se entrene.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Training vectorizer\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def extract_title(x):\n",
|
||||
" return x['title']\n",
|
||||
"\n",
|
||||
"def tupelize_title(x):\n",
|
||||
" return (extract_title(x),x['label'])\n",
|
||||
"\n",
|
||||
"print('Training vectorizer')\n",
|
||||
"vectorizer.adapt(ds_train.take(2000).map(extract_title))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"7500/7500 [==============================] - 82s 11ms/step - loss: 0.6629 - acc: 0.7623 - val_loss: 0.5559 - val_acc: 0.7995\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7f3e0030d350>"
|
||||
]
|
||||
},
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n",
|
||||
"model.fit(ds_train.map(tupelize_title).batch(batch_size),validation_data=ds_test.map(tupelize_title).batch(batch_size))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"nteract": {
|
||||
"transient": {
|
||||
"deleting": false
|
||||
}
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"> **Nota** que la precisión probablemente sea menor aquí, porque estamos entrenando solo con títulos de noticias.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Revisitando las secuencias de variables\n",
|
||||
"\n",
|
||||
"Recuerda que la capa `TextVectorization` rellenará automáticamente las secuencias de longitud variable en un minibatch con tokens de relleno. Resulta que esos tokens también participan en el entrenamiento, y pueden complicar la convergencia del modelo.\n",
|
||||
"\n",
|
||||
"Existen varios enfoques que podemos tomar para minimizar la cantidad de relleno. Uno de ellos es reordenar el conjunto de datos por longitud de secuencia y agrupar todas las secuencias por tamaño. Esto se puede hacer utilizando la función `tf.data.experimental.bucket_by_sequence_length` (consulta la [documentación](https://www.tensorflow.org/api_docs/python/tf/data/experimental/bucket_by_sequence_length)).\n",
|
||||
"\n",
|
||||
"Otro enfoque es usar **enmascaramiento**. En Keras, algunas capas admiten entradas adicionales que indican qué tokens deben tomarse en cuenta durante el entrenamiento. Para incorporar el enmascaramiento en nuestro modelo, podemos incluir una capa `Masking` separada ([documentación](https://keras.io/api/layers/core_layers/masking/)), o podemos especificar el parámetro `mask_zero=True` en nuestra capa `Embedding`.\n",
|
||||
"\n",
|
||||
"> **Nota**: Este entrenamiento tomará alrededor de 5 minutos para completar una época en todo el conjunto de datos. Si pierdes la paciencia, siéntete libre de interrumpir el entrenamiento en cualquier momento. Otra opción es limitar la cantidad de datos utilizados para el entrenamiento, añadiendo la cláusula `.take(...)` después de los conjuntos de datos `ds_train` y `ds_test`.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"7500/7500 [==============================] - 371s 49ms/step - loss: 0.5401 - acc: 0.8079 - val_loss: 0.3780 - val_acc: 0.8822\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7f3dec118850>"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def extract_text(x):\n",
|
||||
" return x['title']+' '+x['description']\n",
|
||||
"\n",
|
||||
"def tupelize(x):\n",
|
||||
" return (extract_text(x),x['label'])\n",
|
||||
"\n",
|
||||
"model = keras.models.Sequential([\n",
|
||||
" vectorizer,\n",
|
||||
" keras.layers.Embedding(vocab_size,embed_size,mask_zero=True),\n",
|
||||
" keras.layers.SimpleRNN(16),\n",
|
||||
" keras.layers.Dense(4,activation='softmax')\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Ahora que estamos utilizando enmascaramiento, podemos entrenar el modelo con todo el conjunto de datos de títulos y descripciones.\n",
|
||||
"\n",
|
||||
"> **Nota**: ¿Has notado que hemos estado utilizando un vectorizador entrenado en los títulos de las noticias, y no en el cuerpo completo del artículo? Potencialmente, esto puede causar que algunos de los tokens sean ignorados, por lo que es mejor reentrenar el vectorizador. Sin embargo, esto podría tener un efecto muy pequeño, así que nos quedaremos con el vectorizador preentrenado anterior por simplicidad.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## LSTM: Memoria a largo corto plazo\n",
|
||||
"\n",
|
||||
"Uno de los principales problemas de las RNNs es el **desvanecimiento de gradientes**. Las RNNs pueden ser bastante largas y pueden tener dificultades para propagar los gradientes hasta la primera capa de la red durante la retropropagación. Cuando esto ocurre, la red no puede aprender relaciones entre tokens distantes. Una forma de evitar este problema es introducir una **gestión explícita del estado** mediante el uso de **compuertas**. Las dos arquitecturas más comunes que introducen compuertas son **memoria a largo corto plazo** (LSTM) y **unidad de relé con compuerta** (GRU). Aquí cubriremos las LSTMs.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Una red LSTM está organizada de manera similar a una RNN, pero hay dos estados que se pasan de capa a capa: el estado actual $c$ y el vector oculto $h$. En cada unidad, el vector oculto $h_{t-1}$ se combina con la entrada $x_t$, y juntos controlan lo que sucede con el estado $c_t$ y la salida $h_{t}$ a través de **compuertas**. Cada compuerta tiene una activación sigmoide (salida en el rango $[0,1]$), que puede considerarse como una máscara bit a bit cuando se multiplica por el vector de estado. Las LSTMs tienen las siguientes compuertas (de izquierda a derecha en la imagen anterior):\n",
|
||||
"* **Compuerta de olvido**, que determina qué componentes del vector $c_{t-1}$ necesitamos olvidar y cuáles pasar.\n",
|
||||
"* **Compuerta de entrada**, que determina cuánta información del vector de entrada y del vector oculto anterior debe incorporarse al vector de estado.\n",
|
||||
"* **Compuerta de salida**, que toma el nuevo vector de estado y decide cuáles de sus componentes se usarán para producir el nuevo vector oculto $h_t$.\n",
|
||||
"\n",
|
||||
"Los componentes del estado $c$ pueden considerarse como indicadores que se pueden activar o desactivar. Por ejemplo, cuando encontramos el nombre *Alice* en la secuencia, suponemos que se refiere a una mujer y activamos el indicador en el estado que dice que tenemos un sustantivo femenino en la oración. Cuando más adelante encontramos las palabras *and Tom*, activaremos el indicador que dice que tenemos un sustantivo en plural. Así, manipulando el estado, podemos realizar un seguimiento de las propiedades gramaticales de la oración.\n",
|
||||
"\n",
|
||||
"> **Nota**: Aquí tienes un excelente recurso para entender el funcionamiento interno de las LSTMs: [Understanding LSTM Networks](https://colah.github.io/posts/2015-08-Understanding-LSTMs/) por Christopher Olah.\n",
|
||||
"\n",
|
||||
"Aunque la estructura interna de una celda LSTM puede parecer compleja, Keras oculta esta implementación dentro de la capa `LSTM`, por lo que lo único que necesitamos hacer en el ejemplo anterior es reemplazar la capa recurrente:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"15000/15000 [==============================] - 188s 13ms/step - loss: 0.5692 - acc: 0.7916 - val_loss: 0.3441 - val_acc: 0.8870\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7f3d6af5c350>"
|
||||
]
|
||||
},
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model = keras.models.Sequential([\n",
|
||||
" vectorizer,\n",
|
||||
" keras.layers.Embedding(vocab_size, embed_size),\n",
|
||||
" keras.layers.LSTM(8),\n",
|
||||
" keras.layers.Dense(4,activation='softmax')\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(8),validation_data=ds_test.map(tupelize).batch(8))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## RNNs bidireccionales y multicapa\n",
|
||||
"\n",
|
||||
"En nuestros ejemplos hasta ahora, las redes recurrentes operan desde el inicio de una secuencia hasta el final. Esto nos resulta natural porque sigue la misma dirección en la que leemos o escuchamos el habla. Sin embargo, para escenarios que requieren acceso aleatorio a la secuencia de entrada, tiene más sentido ejecutar el cálculo recurrente en ambas direcciones. Las RNNs que permiten cálculos en ambas direcciones se llaman **RNNs bidireccionales**, y se pueden crear envolviendo la capa recurrente con una capa especial llamada `Bidirectional`.\n",
|
||||
"\n",
|
||||
"> **Note**: La capa `Bidirectional` crea dos copias de la capa dentro de ella y establece la propiedad `go_backwards` de una de esas copias en `True`, haciendo que avance en la dirección opuesta a lo largo de la secuencia.\n",
|
||||
"\n",
|
||||
"Las redes recurrentes, ya sean unidireccionales o bidireccionales, capturan patrones dentro de una secuencia y los almacenan en vectores de estado o los devuelven como salida. Al igual que con las redes convolucionales, podemos construir otra capa recurrente después de la primera para capturar patrones de nivel superior, construidos a partir de patrones de nivel inferior extraídos por la primera capa. Esto nos lleva a la noción de una **RNN multicapa**, que consiste en dos o más redes recurrentes, donde la salida de la capa anterior se pasa a la siguiente capa como entrada.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"*Imagen tomada de [este maravilloso artículo](https://towardsdatascience.com/from-a-lstm-cell-to-a-multilayer-lstm-network-with-pytorch-2899eb5696f3) por Fernando López.*\n",
|
||||
"\n",
|
||||
"Keras hace que construir estas redes sea una tarea sencilla, porque solo necesitas agregar más capas recurrentes al modelo. Para todas las capas excepto la última, necesitamos especificar el parámetro `return_sequences=True`, porque necesitamos que la capa devuelva todos los estados intermedios, y no solo el estado final del cálculo recurrente.\n",
|
||||
"\n",
|
||||
"Construyamos una LSTM bidireccional de dos capas para nuestro problema de clasificación.\n",
|
||||
"\n",
|
||||
"> **Note** Este código nuevamente toma bastante tiempo en completarse, pero nos da la mayor precisión que hemos visto hasta ahora. Así que tal vez valga la pena esperar y ver el resultado.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"5044/7500 [===================>..........] - ETA: 2:33 - loss: 0.3709 - acc: 0.8706\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r5045/7500 [===================>..........] - ETA: 2:33 - loss: 0.3709 - acc: 0.8706"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model = keras.models.Sequential([\n",
|
||||
" vectorizer,\n",
|
||||
" keras.layers.Embedding(vocab_size, 128, mask_zero=True),\n",
|
||||
" keras.layers.Bidirectional(keras.layers.LSTM(64,return_sequences=True)),\n",
|
||||
" keras.layers.Bidirectional(keras.layers.LSTM(64)), \n",
|
||||
" keras.layers.Dense(4,activation='softmax')\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(batch_size),\n",
|
||||
" validation_data=ds_test.map(tupelize).batch(batch_size))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## RNNs para otras tareas\n",
|
||||
"\n",
|
||||
"Hasta ahora, nos hemos centrado en usar RNNs para clasificar secuencias de texto. Pero pueden manejar muchas más tareas, como generación de texto y traducción automática — abordaremos esas tareas en la próxima unidad.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Descargo de responsabilidad**: \nEste documento ha sido traducido utilizando el servicio de traducción automática [Co-op Translator](https://github.com/Azure/co-op-translator). Si bien nos esforzamos por garantizar la precisión, tenga en cuenta que las traducciones automatizadas pueden contener errores o imprecisiones. El documento original en su idioma nativo debe considerarse la fuente autorizada. Para información crítica, se recomienda una traducción profesional realizada por humanos. No nos hacemos responsables de malentendidos o interpretaciones erróneas que puedan surgir del uso de esta traducción.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernel_info": {
|
||||
"name": "conda-env-py37_tensorflow-py"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "py37_tensorflow",
|
||||
"language": "python",
|
||||
"name": "conda-env-py37_tensorflow-py"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.7.9"
|
||||
},
|
||||
"nteract": {
|
||||
"version": "nteract-front-end@1.0.0"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "81351e61f619b432ff51010a4f993194",
|
||||
"translation_date": "2025-08-31T17:06:11+00:00",
|
||||
"source_file": "lessons/5-NLP/16-RNN/RNNTF.ipynb",
|
||||
"language_code": "es"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
|
|
@ -0,0 +1,414 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Redes generativas\n",
|
||||
"\n",
|
||||
"Las Redes Neuronales Recurrentes (RNNs) y sus variantes con celdas controladas, como las Celdas de Memoria a Largo Corto Plazo (LSTMs) y las Unidades Recurrentes Controladas (GRUs), proporcionaron un mecanismo para el modelado del lenguaje, es decir, pueden aprender el orden de las palabras y ofrecer predicciones para la siguiente palabra en una secuencia. Esto nos permite usar las RNNs para **tareas generativas**, como la generación de texto ordinario, la traducción automática e incluso la generación de subtítulos para imágenes.\n",
|
||||
"\n",
|
||||
"En la arquitectura de RNN que discutimos en la unidad anterior, cada unidad RNN producía el siguiente estado oculto como salida. Sin embargo, también podemos añadir otra salida a cada unidad recurrente, lo que nos permitiría generar una **secuencia** (que tiene la misma longitud que la secuencia original). Además, podemos usar unidades RNN que no acepten una entrada en cada paso, y simplemente tomen un vector de estado inicial, para luego producir una secuencia de salidas.\n",
|
||||
"\n",
|
||||
"En este cuaderno, nos centraremos en modelos generativos simples que nos ayuden a generar texto. Para simplificar, construyamos una **red a nivel de caracteres**, que genera texto letra por letra. Durante el entrenamiento, necesitamos tomar un corpus de texto y dividirlo en secuencias de letras.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Loading dataset...\n",
|
||||
"Building vocab...\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"import torchtext\n",
|
||||
"import numpy as np\n",
|
||||
"from torchnlp import *\n",
|
||||
"train_dataset,test_dataset,classes,vocab = load_dataset()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Construcción de vocabulario de caracteres\n",
|
||||
"\n",
|
||||
"Para construir una red generativa a nivel de caracteres, necesitamos dividir el texto en caracteres individuales en lugar de palabras. Esto se puede lograr definiendo un tokenizador diferente:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Vocabulary size = 82\n",
|
||||
"Encoding of 'a' is 1\n",
|
||||
"Character with code 13 is c\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def char_tokenizer(words):\n",
|
||||
" return list(words) #[word for word in words]\n",
|
||||
"\n",
|
||||
"counter = collections.Counter()\n",
|
||||
"for (label, line) in train_dataset:\n",
|
||||
" counter.update(char_tokenizer(line))\n",
|
||||
"vocab = torchtext.vocab.vocab(counter)\n",
|
||||
"\n",
|
||||
"vocab_size = len(vocab)\n",
|
||||
"print(f\"Vocabulary size = {vocab_size}\")\n",
|
||||
"print(f\"Encoding of 'a' is {vocab.get_stoi()['a']}\")\n",
|
||||
"print(f\"Character with code 13 is {vocab.get_itos()[13]}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Veamos el ejemplo de cómo podemos codificar el texto de nuestro conjunto de datos:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"tensor([ 0, 1, 2, 2, 3, 4, 5, 6, 3, 7, 8, 1, 9, 10, 3, 11, 2, 1,\n",
|
||||
" 12, 3, 7, 1, 13, 14, 3, 15, 16, 5, 17, 3, 5, 18, 8, 3, 7, 2,\n",
|
||||
" 1, 13, 14, 3, 19, 20, 8, 21, 5, 8, 9, 10, 22, 3, 20, 8, 21, 5,\n",
|
||||
" 8, 9, 10, 3, 23, 3, 4, 18, 17, 9, 5, 23, 10, 8, 2, 2, 8, 9,\n",
|
||||
" 10, 24, 3, 0, 1, 2, 2, 3, 4, 5, 9, 8, 8, 5, 25, 10, 3, 26,\n",
|
||||
" 12, 27, 16, 26, 2, 27, 16, 28, 29, 30, 1, 16, 26, 3, 17, 31, 3, 21,\n",
|
||||
" 2, 5, 9, 1, 23, 13, 32, 16, 27, 13, 10, 24, 3, 1, 9, 8, 3, 10,\n",
|
||||
" 8, 8, 27, 16, 28, 3, 28, 9, 8, 8, 16, 3, 1, 28, 1, 27, 16, 6])"
|
||||
]
|
||||
},
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def enc(x):\n",
|
||||
" return torch.LongTensor(encode(x,voc=vocab,tokenizer=char_tokenizer))\n",
|
||||
"\n",
|
||||
"enc(train_dataset[0][1])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Entrenando una RNN generativa\n",
|
||||
"\n",
|
||||
"La forma en que entrenaremos la RNN para generar texto es la siguiente. En cada paso, tomaremos una secuencia de caracteres de longitud `nchars` y pediremos a la red que genere el siguiente carácter de salida para cada carácter de entrada:\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Dependiendo del escenario específico, también podríamos querer incluir algunos caracteres especiales, como *fin de secuencia* `<eos>`. En nuestro caso, solo queremos entrenar la red para la generación continua de texto, por lo que fijaremos el tamaño de cada secuencia para que sea igual a `nchars` tokens. En consecuencia, cada ejemplo de entrenamiento consistirá en `nchars` entradas y `nchars` salidas (que son la secuencia de entrada desplazada un símbolo hacia la izquierda). El minibatch consistirá en varias de estas secuencias.\n",
|
||||
"\n",
|
||||
"La forma en que generaremos los minibatches será tomando cada texto de noticias de longitud `l` y generando todas las combinaciones posibles de entrada-salida a partir de él (habrá `l-nchars` de estas combinaciones). Estas formarán un minibatch, y el tamaño de los minibatches será diferente en cada paso de entrenamiento.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(tensor([[ 0, 1, 2, ..., 28, 29, 30],\n",
|
||||
" [ 1, 2, 2, ..., 29, 30, 1],\n",
|
||||
" [ 2, 2, 3, ..., 30, 1, 16],\n",
|
||||
" ...,\n",
|
||||
" [20, 8, 21, ..., 1, 28, 1],\n",
|
||||
" [ 8, 21, 5, ..., 28, 1, 27],\n",
|
||||
" [21, 5, 8, ..., 1, 27, 16]]),\n",
|
||||
" tensor([[ 1, 2, 2, ..., 29, 30, 1],\n",
|
||||
" [ 2, 2, 3, ..., 30, 1, 16],\n",
|
||||
" [ 2, 3, 4, ..., 1, 16, 26],\n",
|
||||
" ...,\n",
|
||||
" [ 8, 21, 5, ..., 28, 1, 27],\n",
|
||||
" [21, 5, 8, ..., 1, 27, 16],\n",
|
||||
" [ 5, 8, 9, ..., 27, 16, 6]]))"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"nchars = 100\n",
|
||||
"\n",
|
||||
"def get_batch(s,nchars=nchars):\n",
|
||||
" ins = torch.zeros(len(s)-nchars,nchars,dtype=torch.long,device=device)\n",
|
||||
" outs = torch.zeros(len(s)-nchars,nchars,dtype=torch.long,device=device)\n",
|
||||
" for i in range(len(s)-nchars):\n",
|
||||
" ins[i] = enc(s[i:i+nchars])\n",
|
||||
" outs[i] = enc(s[i+1:i+nchars+1])\n",
|
||||
" return ins,outs\n",
|
||||
"\n",
|
||||
"get_batch(train_dataset[0][1])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Ahora definamos la red generadora. Puede basarse en cualquier célula recurrente que discutimos en la unidad anterior (simple, LSTM o GRU). En nuestro ejemplo, utilizaremos LSTM.\n",
|
||||
"\n",
|
||||
"Dado que la red toma caracteres como entrada y el tamaño del vocabulario es bastante pequeño, no necesitamos una capa de embeddings; la entrada codificada en one-hot puede ir directamente a la célula LSTM. Sin embargo, como pasamos números de caracteres como entrada, necesitamos codificarlos en one-hot antes de pasarlos a la LSTM. Esto se realiza llamando a la función `one_hot` durante el paso `forward`. El codificador de salida sería una capa lineal que convertirá el estado oculto en una salida codificada en one-hot.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class LSTMGenerator(torch.nn.Module):\n",
|
||||
" def __init__(self, vocab_size, hidden_dim):\n",
|
||||
" super().__init__()\n",
|
||||
" self.rnn = torch.nn.LSTM(vocab_size,hidden_dim,batch_first=True)\n",
|
||||
" self.fc = torch.nn.Linear(hidden_dim, vocab_size)\n",
|
||||
"\n",
|
||||
" def forward(self, x, s=None):\n",
|
||||
" x = torch.nn.functional.one_hot(x,vocab_size).to(torch.float32)\n",
|
||||
" x,s = self.rnn(x,s)\n",
|
||||
" return self.fc(x),s"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Durante el entrenamiento, queremos poder muestrear texto generado. Para ello, definiremos la función `generate` que producirá una cadena de salida de longitud `size`, comenzando desde la cadena inicial `start`.\n",
|
||||
"\n",
|
||||
"El funcionamiento es el siguiente. Primero, pasaremos toda la cadena inicial a través de la red, y obtendremos el estado de salida `s` y el siguiente carácter predicho `out`. Dado que `out` está codificado en one-hot, tomamos `argmax` para obtener el índice del carácter `nc` en el vocabulario, y usamos `itos` para identificar el carácter real y añadirlo a la lista resultante de caracteres `chars`. Este proceso de generar un carácter se repite `size` veces para generar el número requerido de caracteres.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def generate(net,size=100,start='today '):\n",
|
||||
" chars = list(start)\n",
|
||||
" out, s = net(enc(chars).view(1,-1).to(device))\n",
|
||||
" for i in range(size):\n",
|
||||
" nc = torch.argmax(out[0][-1])\n",
|
||||
" chars.append(vocab.get_itos()[nc])\n",
|
||||
" out, s = net(nc.view(1,-1),s)\n",
|
||||
" return ''.join(chars)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"¡Ahora vamos a entrenar! El bucle de entrenamiento es casi el mismo que en todos nuestros ejemplos anteriores, pero en lugar de la precisión, imprimimos texto generado como muestra cada 1000 épocas.\n",
|
||||
"\n",
|
||||
"Se debe prestar especial atención a la forma en que calculamos la pérdida. Necesitamos calcular la pérdida dado un resultado codificado en one-hot `out` y el texto esperado `text_out`, que es la lista de índices de caracteres. Por suerte, la función `cross_entropy` espera como primer argumento la salida no normalizada de la red, y como segundo argumento el número de clase, que es exactamente lo que tenemos. Además, realiza un promedio automático sobre el tamaño del minibatch.\n",
|
||||
"\n",
|
||||
"También limitamos el entrenamiento a `samples_to_train` muestras, para no tener que esperar demasiado tiempo. Te animamos a experimentar y probar entrenamientos más largos, posiblemente durante varias épocas (en cuyo caso necesitarías crear otro bucle alrededor de este código).\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Current loss = 4.398899078369141\n",
|
||||
"today sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr s\n",
|
||||
"Current loss = 2.161320447921753\n",
|
||||
"today and to the tor to to the tor to to the tor to to the tor to to the tor to to the tor to to the tor t\n",
|
||||
"Current loss = 1.6722588539123535\n",
|
||||
"today and the court to the could to the could to the could to the could to the could to the could to the c\n",
|
||||
"Current loss = 2.423795223236084\n",
|
||||
"today and a second to the conternation of the conternation of the conternation of the conternation of the \n",
|
||||
"Current loss = 1.702607274055481\n",
|
||||
"today and the company to the company to the company to the company to the company to the company to the co\n",
|
||||
"Current loss = 1.692358136177063\n",
|
||||
"today and the company to the company to the company to the company to the company to the company to the co\n",
|
||||
"Current loss = 1.9722288846969604\n",
|
||||
"today and the control the control the control the control the control the control the control the control \n",
|
||||
"Current loss = 1.8705692291259766\n",
|
||||
"today and the second to the second to the second to the second to the second to the second to the second t\n",
|
||||
"Current loss = 1.7626899480819702\n",
|
||||
"today and a security and a security and a security and a security and a security and a security and a secu\n",
|
||||
"Current loss = 1.5574463605880737\n",
|
||||
"today and the company and the company and the company and the company and the company and the company and \n",
|
||||
"Current loss = 1.5620026588439941\n",
|
||||
"today and the be that the be the be that the be the be that the be the be that the be the be that the be t\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"net = LSTMGenerator(vocab_size,64).to(device)\n",
|
||||
"\n",
|
||||
"samples_to_train = 10000\n",
|
||||
"optimizer = torch.optim.Adam(net.parameters(),0.01)\n",
|
||||
"loss_fn = torch.nn.CrossEntropyLoss()\n",
|
||||
"net.train()\n",
|
||||
"for i,x in enumerate(train_dataset):\n",
|
||||
" # x[0] is class label, x[1] is text\n",
|
||||
" if len(x[1])-nchars<10:\n",
|
||||
" continue\n",
|
||||
" samples_to_train-=1\n",
|
||||
" if not samples_to_train: break\n",
|
||||
" text_in, text_out = get_batch(x[1])\n",
|
||||
" optimizer.zero_grad()\n",
|
||||
" out,s = net(text_in)\n",
|
||||
" loss = torch.nn.functional.cross_entropy(out.view(-1,vocab_size),text_out.flatten()) #cross_entropy(out,labels)\n",
|
||||
" loss.backward()\n",
|
||||
" optimizer.step()\n",
|
||||
" if i%1000==0:\n",
|
||||
" print(f\"Current loss = {loss.item()}\")\n",
|
||||
" print(generate(net))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Este ejemplo ya genera un texto bastante bueno, pero se puede mejorar en varios aspectos:\n",
|
||||
"\n",
|
||||
"* **Mejor generación de minibatches**. La forma en que preparamos los datos para el entrenamiento fue generando un minibatch a partir de una muestra. Esto no es ideal, porque los minibatches tienen tamaños diferentes, y algunos incluso no pueden generarse porque el texto es más pequeño que `nchars`. Además, los minibatches pequeños no aprovechan suficientemente la GPU. Sería más inteligente tomar un gran bloque de texto de todas las muestras, luego generar todos los pares de entrada-salida, mezclarlos y generar minibatches de tamaño uniforme.\n",
|
||||
"\n",
|
||||
"* **LSTM multicapa**. Tiene sentido probar con 2 o 3 capas de células LSTM. Como mencionamos en la unidad anterior, cada capa de LSTM extrae ciertos patrones del texto, y en el caso de un generador a nivel de caracteres, podemos esperar que el nivel inferior de LSTM sea responsable de extraer sílabas, y los niveles superiores de palabras y combinaciones de palabras. Esto se puede implementar fácilmente pasando el parámetro de número de capas al constructor de LSTM.\n",
|
||||
"\n",
|
||||
"* También podrías experimentar con **unidades GRU** y ver cuáles funcionan mejor, así como con **diferentes tamaños de capas ocultas**. Una capa oculta demasiado grande puede resultar en sobreajuste (por ejemplo, la red aprenderá el texto exacto), mientras que un tamaño más pequeño podría no producir buenos resultados.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Generación de texto suave y temperatura\n",
|
||||
"\n",
|
||||
"En la definición anterior de `generate`, siempre tomábamos el carácter con la probabilidad más alta como el siguiente carácter en el texto generado. Esto daba como resultado que el texto a menudo \"ciclaba\" entre las mismas secuencias de caracteres una y otra vez, como en este ejemplo:\n",
|
||||
"```\n",
|
||||
"today of the second the company and a second the company ...\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"Sin embargo, si observamos la distribución de probabilidad para el siguiente carácter, podría suceder que la diferencia entre algunas de las probabilidades más altas no sea muy grande, por ejemplo, un carácter puede tener una probabilidad de 0.2, otro de 0.19, etc. Por ejemplo, al buscar el siguiente carácter en la secuencia '*play*', el siguiente carácter podría ser igualmente un espacio o una **e** (como en la palabra *player*).\n",
|
||||
"\n",
|
||||
"Esto nos lleva a la conclusión de que no siempre es \"justo\" seleccionar el carácter con mayor probabilidad, ya que elegir el segundo más probable aún podría conducir a un texto significativo. Es más sensato **muestrear** caracteres de la distribución de probabilidad dada por la salida de la red.\n",
|
||||
"\n",
|
||||
"Este muestreo se puede realizar utilizando la función `multinomial`, que implementa la llamada **distribución multinomial**. A continuación, se define una función que implementa esta generación de texto **suave**:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"--- Temperature = 0.3\n",
|
||||
"Today and a company and complete an all the land the restrational the as a security and has provers the pay to and a report and the computer in the stand has filities and working the law the stations for a company and with the company and the final the first company and refight of the state and and workin\n",
|
||||
"\n",
|
||||
"--- Temperature = 0.8\n",
|
||||
"Today he oniis its first to Aus bomblaties the marmation a to manan boogot that pirate assaid a relaid their that goverfin the the Cappets Ecrotional Assonia Cition targets it annight the w scyments Blamity #39;s TVeer Diercheg Reserals fran envyuil that of ster said access what succers of Dour-provelith\n",
|
||||
"\n",
|
||||
"--- Temperature = 1.0\n",
|
||||
"Today holy they a 11 will meda a toket subsuaties, engins for Chanos, they's has stainger past to opening orital his thempting new Nattona was al innerforder advan-than #36;s night year his religuled talitatian what the but with Wednesday to Justment will wemen of Mark CCC Camp as Timed Nae wome a leaders\n",
|
||||
"\n",
|
||||
"--- Temperature = 1.3\n",
|
||||
"Today gpone 2.5 fech atcusion poor cocles toparsdorM.cht Line Pamage put 43 his calt lowed to the book, that has authh-the silia rruch ailing to'ory andhes beutirsimi- Aefffive heading offil an auf eacklets is charged evis, Gunymy oy) Mony has it after-sloythyor loveId out filme, the Natabl -Najuntaxiggs \n",
|
||||
"\n",
|
||||
"--- Temperature = 1.8\n",
|
||||
"Today plary, P.slan chly\\401 mardregationly #39;t 8.1Mide) closes ,filtcon alfly playin roven!\\grea.-QFBEP: Iss onfarchQ/itilia CCf Zivesigntwasta orce.-Peul-aw.uicrin of fuglinfsut aftaningwo, MIEX awayew Aice Woiduar Corvagiugge oppo esig ThusBratourid canthly-RyI.co lagitems\\eexciaishes.conBabntusmor I\n",
|
||||
"\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def generate_soft(net,size=100,start='today ',temperature=1.0):\n",
|
||||
" chars = list(start)\n",
|
||||
" out, s = net(enc(chars).view(1,-1).to(device))\n",
|
||||
" for i in range(size):\n",
|
||||
" #nc = torch.argmax(out[0][-1])\n",
|
||||
" out_dist = out[0][-1].div(temperature).exp()\n",
|
||||
" nc = torch.multinomial(out_dist,1)[0]\n",
|
||||
" chars.append(vocab.get_itos()[nc])\n",
|
||||
" out, s = net(nc.view(1,-1),s)\n",
|
||||
" return ''.join(chars)\n",
|
||||
" \n",
|
||||
"for i in [0.3,0.8,1.0,1.3,1.8]:\n",
|
||||
" print(f\"--- Temperature = {i}\\n{generate_soft(net,size=300,start='Today ',temperature=i)}\\n\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Hemos introducido un parámetro más llamado **temperatura**, que se utiliza para indicar qué tan estrictamente debemos adherirnos a la probabilidad más alta. Si la temperatura es 1.0, hacemos un muestreo multinomial justo, y cuando la temperatura se acerca al infinito, todas las probabilidades se vuelven iguales y seleccionamos el siguiente carácter al azar. En el ejemplo a continuación, podemos observar que el texto se vuelve sin sentido cuando aumentamos demasiado la temperatura, y se asemeja a un texto \"cíclico\" generado rígidamente cuando se acerca a 0.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Descargo de responsabilidad**: \nEste documento ha sido traducido utilizando el servicio de traducción automática [Co-op Translator](https://github.com/Azure/co-op-translator). Aunque nos esforzamos por garantizar la precisión, tenga en cuenta que las traducciones automatizadas pueden contener errores o imprecisiones. El documento original en su idioma nativo debe considerarse como la fuente autorizada. Para información crítica, se recomienda una traducción profesional realizada por humanos. No nos hacemos responsables de malentendidos o interpretaciones erróneas que puedan surgir del uso de esta traducción.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.8.12 ('py38')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "7673cd150d96c74c6d6011460094efb4",
|
||||
"translation_date": "2025-08-31T16:53:45+00:00",
|
||||
"source_file": "lessons/5-NLP/17-GenerativeNetworks/GenerativePyTorch.ipynb",
|
||||
"language_code": "es"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
|
|
@ -0,0 +1,495 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Redes generativas\n",
|
||||
"\n",
|
||||
"Las Redes Neuronales Recurrentes (RNNs) y sus variantes con celdas controladas, como las Celdas de Memoria a Largo Corto Plazo (LSTMs) y las Unidades Recurrentes Controladas (GRUs), proporcionaron un mecanismo para el modelado del lenguaje, es decir, pueden aprender el orden de las palabras y ofrecer predicciones para la siguiente palabra en una secuencia. Esto nos permite usar las RNNs para **tareas generativas**, como la generación de texto común, la traducción automática e incluso la generación de subtítulos para imágenes.\n",
|
||||
"\n",
|
||||
"En la arquitectura de RNN que discutimos en la unidad anterior, cada unidad RNN producía el siguiente estado oculto como salida. Sin embargo, también podemos añadir otra salida a cada unidad recurrente, lo que nos permitiría generar una **secuencia** (que tiene la misma longitud que la secuencia original). Además, podemos usar unidades RNN que no acepten una entrada en cada paso, y simplemente tomen un vector de estado inicial, para luego producir una secuencia de salidas.\n",
|
||||
"\n",
|
||||
"En este cuaderno, nos centraremos en modelos generativos simples que nos ayuden a generar texto. Para simplificar, construyamos una **red a nivel de caracteres**, que genera texto letra por letra. Durante el entrenamiento, necesitamos tomar un corpus de texto y dividirlo en secuencias de letras.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import tensorflow as tf\n",
|
||||
"from tensorflow import keras\n",
|
||||
"import tensorflow_datasets as tfds\n",
|
||||
"import numpy as np\n",
|
||||
"\n",
|
||||
"ds_train, ds_test = tfds.load('ag_news_subset').values()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Construyendo un vocabulario de caracteres\n",
|
||||
"\n",
|
||||
"Para construir una red generativa a nivel de caracteres, necesitamos dividir el texto en caracteres individuales en lugar de palabras. La capa `TextVectorization` que hemos estado utilizando antes no puede hacer eso, por lo que tenemos dos opciones:\n",
|
||||
"\n",
|
||||
"* Cargar el texto manualmente y realizar la tokenización 'a mano', como se muestra en [este ejemplo oficial de Keras](https://keras.io/examples/generative/lstm_character_level_text_generation/)\n",
|
||||
"* Usar la clase `Tokenizer` para la tokenización a nivel de caracteres.\n",
|
||||
"\n",
|
||||
"Optaremos por la segunda opción. `Tokenizer` también se puede usar para tokenizar en palabras, por lo que debería ser fácil cambiar de tokenización a nivel de caracteres a nivel de palabras.\n",
|
||||
"\n",
|
||||
"Para realizar la tokenización a nivel de caracteres, necesitamos pasar el parámetro `char_level=True`:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def extract_text(x):\n",
|
||||
" return x['title']+' '+x['description']\n",
|
||||
"\n",
|
||||
"def tupelize(x):\n",
|
||||
" return (extract_text(x),x['label'])\n",
|
||||
"\n",
|
||||
"tokenizer = keras.preprocessing.text.Tokenizer(char_level=True,lower=False)\n",
|
||||
"tokenizer.fit_on_texts([x['title'].numpy().decode('utf-8') for x in ds_train])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"También queremos usar un token especial para indicar **fin de secuencia**, al que llamaremos `<eos>`. Vamos a añadirlo manualmente al vocabulario:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"eos_token = len(tokenizer.word_index)+1\n",
|
||||
"tokenizer.word_index['<eos>'] = eos_token\n",
|
||||
"\n",
|
||||
"vocab_size = eos_token + 1"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[[48, 2, 10, 10, 5, 44, 1, 25, 5, 8, 10, 13, 78]]"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"tokenizer.texts_to_sequences(['Hello, world!'])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Entrenando una RNN generativa para generar títulos\n",
|
||||
"\n",
|
||||
"La forma en que entrenaremos una RNN para generar títulos de noticias es la siguiente. En cada paso, tomaremos un título, que será alimentado a una RNN, y para cada carácter de entrada le pediremos a la red que genere el siguiente carácter de salida:\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Para el último carácter de nuestra secuencia, le pediremos a la red que genere el token `<eos>`.\n",
|
||||
"\n",
|
||||
"La principal diferencia con la RNN generativa que estamos utilizando aquí es que tomaremos una salida de cada paso de la RNN, y no solo de la celda final. Esto se puede lograr especificando el parámetro `return_sequences` en la celda de la RNN.\n",
|
||||
"\n",
|
||||
"Así, durante el entrenamiento, una entrada para la red sería una secuencia de caracteres codificados de cierta longitud, y una salida sería una secuencia de la misma longitud, pero desplazada por un elemento y terminada con `<eos>`. El minibatch consistirá en varias de estas secuencias, y necesitaremos usar **padding** para alinear todas las secuencias.\n",
|
||||
"\n",
|
||||
"Vamos a crear funciones que transformen el conjunto de datos por nosotros. Debido a que queremos rellenar las secuencias a nivel de minibatch, primero agruparemos el conjunto de datos llamando `.batch()`, y luego usaremos `map` para realizar la transformación. Por lo tanto, la función de transformación tomará un minibatch completo como parámetro:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def title_batch(x):\n",
|
||||
" x = [t.numpy().decode('utf-8') for t in x]\n",
|
||||
" z = tokenizer.texts_to_sequences(x)\n",
|
||||
" z = tf.keras.preprocessing.sequence.pad_sequences(z)\n",
|
||||
" return tf.one_hot(z,vocab_size), tf.one_hot(tf.concat([z[:,1:],tf.constant(eos_token,shape=(len(z),1))],axis=1),vocab_size)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Algunas cosas importantes que hacemos aquí:\n",
|
||||
"* Primero extraemos el texto real del tensor de cadenas\n",
|
||||
"* `text_to_sequences` convierte la lista de cadenas en una lista de tensores de enteros\n",
|
||||
"* `pad_sequences` rellena esos tensores hasta su longitud máxima\n",
|
||||
"* Finalmente codificamos en formato one-hot todos los caracteres, y también realizamos el desplazamiento y la adición de `<eos>`. Pronto veremos por qué necesitamos caracteres codificados en formato one-hot.\n",
|
||||
"\n",
|
||||
"Sin embargo, esta función es **Pythonic**, es decir, no puede ser traducida automáticamente al gráfico computacional de Tensorflow. Obtendremos errores si intentamos usar esta función directamente en la función `Dataset.map`. Necesitamos encapsular esta llamada Pythonic utilizando el envoltorio `py_function`:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def title_batch_fn(x):\n",
|
||||
" x = x['title']\n",
|
||||
" a,b = tf.py_function(title_batch,inp=[x],Tout=(tf.float32,tf.float32))\n",
|
||||
" return a,b"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Nota**: Diferenciar entre las funciones de transformación de Python y Tensorflow puede parecer un poco complejo, y podrías estar preguntándote por qué no transformamos el conjunto de datos usando funciones estándar de Python antes de pasarlo a `fit`. Aunque esto definitivamente se puede hacer, usar `Dataset.map` tiene una gran ventaja, ya que la tubería de transformación de datos se ejecuta utilizando el gráfico computacional de Tensorflow, lo que aprovecha las capacidades de cálculo de la GPU y minimiza la necesidad de transferir datos entre la CPU y la GPU.\n",
|
||||
"\n",
|
||||
"Ahora podemos construir nuestra red generadora y comenzar el entrenamiento. Puede basarse en cualquier célula recurrente que discutimos en la unidad anterior (simple, LSTM o GRU). En nuestro ejemplo, utilizaremos LSTM.\n",
|
||||
"\n",
|
||||
"Dado que la red toma caracteres como entrada y el tamaño del vocabulario es bastante pequeño, no necesitamos una capa de embeddings; la entrada codificada en formato one-hot puede ir directamente a la célula LSTM. La capa de salida sería un clasificador `Dense` que convertirá la salida de LSTM en números de tokens codificados en formato one-hot.\n",
|
||||
"\n",
|
||||
"Además, dado que estamos trabajando con secuencias de longitud variable, podemos usar la capa `Masking` para crear una máscara que ignore la parte rellenada de la cadena. Esto no es estrictamente necesario, ya que no estamos muy interesados en todo lo que va más allá del token `<eos>`, pero lo utilizaremos con el propósito de adquirir experiencia con este tipo de capa. El `input_shape` sería `(None, vocab_size)`, donde `None` indica la secuencia de longitud variable, y la forma de salida es también `(None, vocab_size)`, como puedes ver en el `summary`:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"sequential\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
"Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
"masking (Masking) (None, None, 84) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"lstm (LSTM) (None, None, 128) 109056 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense (Dense) (None, None, 84) 10836 \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 119,892\n",
|
||||
"Trainable params: 119,892\n",
|
||||
"Non-trainable params: 0\n",
|
||||
"_________________________________________________________________\n",
|
||||
"15000/15000 [==============================] - 229s 15ms/step - loss: 1.5385\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7fa40c1245e0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model = keras.models.Sequential([\n",
|
||||
" keras.layers.Masking(input_shape=(None,vocab_size)),\n",
|
||||
" keras.layers.LSTM(128,return_sequences=True),\n",
|
||||
" keras.layers.Dense(vocab_size,activation='softmax')\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"model.summary()\n",
|
||||
"model.compile(loss='categorical_crossentropy')\n",
|
||||
"\n",
|
||||
"model.fit(ds_train.batch(8).map(title_batch_fn))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Generando salida\n",
|
||||
"\n",
|
||||
"Ahora que hemos entrenado el modelo, queremos usarlo para generar algo de salida. Primero, necesitamos una forma de decodificar texto representado por una secuencia de números de tokens. Para ello, podríamos usar la función `tokenizer.sequences_to_texts`; sin embargo, no funciona bien con la tokenización a nivel de caracteres. Por lo tanto, tomaremos un diccionario de tokens del tokenizer (llamado `word_index`), construiremos un mapa inverso y escribiremos nuestra propia función de decodificación:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"reverse_map = {val:key for key, val in tokenizer.word_index.items()}\n",
|
||||
"\n",
|
||||
"def decode(x):\n",
|
||||
" return ''.join([reverse_map[t] for t in x])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Ahora, vamos a generar. Comenzaremos con una cadena `start`, la codificaremos en una secuencia `inp`, y luego en cada paso llamaremos a nuestra red para inferir el siguiente carácter.\n",
|
||||
"\n",
|
||||
"La salida de la red `out` es un vector de `vocab_size` elementos que representa las probabilidades de cada token, y podemos encontrar el número del token más probable utilizando `argmax`. Luego, añadimos este carácter a la lista generada de tokens y continuamos con la generación. Este proceso de generar un carácter se repite `size` veces para generar el número requerido de caracteres, y terminamos antes si se encuentra el `eos_token`.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"'Today #39;s lead to strike for the strike for the strike for the strike (AFP)'"
|
||||
]
|
||||
},
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def generate(model,size=100,start='Today '):\n",
|
||||
" inp = tokenizer.texts_to_sequences([start])[0]\n",
|
||||
" chars = inp\n",
|
||||
" for i in range(size):\n",
|
||||
" out = model(tf.expand_dims(tf.one_hot(inp,vocab_size),0))[0][-1]\n",
|
||||
" nc = tf.argmax(out)\n",
|
||||
" if nc==eos_token:\n",
|
||||
" break\n",
|
||||
" chars.append(nc.numpy())\n",
|
||||
" inp = inp+[nc]\n",
|
||||
" return decode(chars)\n",
|
||||
" \n",
|
||||
"generate(model)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Muestreo de salida durante el entrenamiento\n",
|
||||
"\n",
|
||||
"Dado que no tenemos métricas útiles como *precisión*, la única forma de verificar que nuestro modelo está mejorando es **muestreando** cadenas generadas durante el entrenamiento. Para hacerlo, utilizaremos **callbacks**, es decir, funciones que podemos pasar a la función `fit` y que se llamarán periódicamente durante el entrenamiento.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch 1/3\n",
|
||||
"15000/15000 [==============================] - 226s 15ms/step - loss: 1.2703\n",
|
||||
"Today #39;s a lead in the company for the strike\n",
|
||||
"Epoch 2/3\n",
|
||||
"15000/15000 [==============================] - 227s 15ms/step - loss: 1.2057\n",
|
||||
"Today #39;s the Market Service on Security Start (AP)\n",
|
||||
"Epoch 3/3\n",
|
||||
"15000/15000 [==============================] - 226s 15ms/step - loss: 1.1752\n",
|
||||
"Today #39;s a line on the strike to start for the start\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7fa40c74e3d0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"sampling_callback = keras.callbacks.LambdaCallback(\n",
|
||||
" on_epoch_end = lambda batch, logs: print(generate(model))\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"model.fit(ds_train.batch(8).map(title_batch_fn),callbacks=[sampling_callback],epochs=3)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Este ejemplo ya genera un texto bastante bueno, pero se puede mejorar de varias maneras:\n",
|
||||
"\n",
|
||||
"* **Más texto**. Solo hemos utilizado títulos para nuestra tarea, pero podrías experimentar con texto completo. Recuerda que las RNN no son muy buenas manejando secuencias largas, por lo que tiene sentido dividirlas en oraciones más cortas o entrenar siempre con una longitud de secuencia fija de algún valor predefinido `num_chars` (por ejemplo, 256). Podrías intentar modificar el ejemplo anterior para usar esta arquitectura, utilizando el [tutorial oficial de Keras](https://keras.io/examples/generative/lstm_character_level_text_generation/) como inspiración.\n",
|
||||
"\n",
|
||||
"* **LSTM multicapa**. Tiene sentido probar con 2 o 3 capas de células LSTM. Como mencionamos en la unidad anterior, cada capa de LSTM extrae ciertos patrones del texto, y en el caso de un generador a nivel de caracteres, podemos esperar que el nivel inferior de LSTM sea responsable de extraer sílabas, y los niveles superiores de palabras y combinaciones de palabras. Esto se puede implementar fácilmente pasando un parámetro de número de capas al constructor de LSTM.\n",
|
||||
"\n",
|
||||
"* También podrías experimentar con **unidades GRU** y ver cuáles funcionan mejor, así como con **diferentes tamaños de capas ocultas**. Una capa oculta demasiado grande puede resultar en sobreajuste (por ejemplo, la red aprenderá el texto exacto), y un tamaño más pequeño podría no producir buenos resultados.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Generación de texto suave y temperatura\n",
|
||||
"\n",
|
||||
"En la definición anterior de `generate`, siempre tomábamos el carácter con la mayor probabilidad como el siguiente carácter en el texto generado. Esto daba como resultado que el texto a menudo \"ciclaba\" entre las mismas secuencias de caracteres una y otra vez, como en este ejemplo:\n",
|
||||
"```\n",
|
||||
"today of the second the company and a second the company ...\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"Sin embargo, si observamos la distribución de probabilidad para el siguiente carácter, podría suceder que la diferencia entre algunas de las probabilidades más altas no sea muy grande, por ejemplo, un carácter puede tener una probabilidad de 0.2, otro de 0.19, etc. Por ejemplo, al buscar el siguiente carácter en la secuencia '*play*', el siguiente carácter podría ser igualmente un espacio o **e** (como en la palabra *player*).\n",
|
||||
"\n",
|
||||
"Esto nos lleva a la conclusión de que no siempre es \"justo\" seleccionar el carácter con mayor probabilidad, ya que elegir el segundo más alto aún podría llevarnos a un texto significativo. Es más sabio **muestrear** caracteres de la distribución de probabilidad proporcionada por la salida de la red.\n",
|
||||
"\n",
|
||||
"Este muestreo se puede realizar utilizando la función `np.multinomial`, que implementa la llamada **distribución multinomial**. Una función que implementa esta generación de texto **suave** se define a continuación:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 33,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\n",
|
||||
"--- Temperature = 0.3\n",
|
||||
"Today #39;s strike #39; to start at the store return\n",
|
||||
"On Sunday PO to Be Data Profit Up (Reuters)\n",
|
||||
"Moscow, SP wins straight to the Microsoft #39;s control of the space start\n",
|
||||
"President olding of the blast start for the strike to pay <b>...</b>\n",
|
||||
"Little red riding hood ficed to the spam countered in European <b>...</b>\n",
|
||||
"\n",
|
||||
"--- Temperature = 0.8\n",
|
||||
"Today countie strikes ryder missile faces food market blut\n",
|
||||
"On Sunday collores lose-toppy of sale of Bullment in <b>...</b>\n",
|
||||
"Moscow, IBM Diffeiting in Afghan Software Hotels (Reuters)\n",
|
||||
"President Ol Luster for Profit Peaced Raised (AP)\n",
|
||||
"Little red riding hood dace on depart talks #39; bank up\n",
|
||||
"\n",
|
||||
"--- Temperature = 1.0\n",
|
||||
"Today wits House buiting debate fixes #39; supervice stake again\n",
|
||||
"On Sunday arling digital poaching In for level\n",
|
||||
"Moscow, DS Up 7, Top Proble Protest Caprey Mamarian Strike\n",
|
||||
"President teps help of roubler stepted lessabul-Dhalitics (AFP)\n",
|
||||
"Little red riding hood signs on cash in Carter-youb\n",
|
||||
"\n",
|
||||
"--- Temperature = 1.3\n",
|
||||
"Today wits flawer ro, pSIA figat's co DroftwavesIs Talo up\n",
|
||||
"On Sunday hround elitwing wint EU Powerburlinetien\n",
|
||||
"Moscow, Bazz #39;s sentries olymen winnelds' next for Olympite Huc?\n",
|
||||
"President lost securitys from power Elections in Smiltrials\n",
|
||||
"Little red riding hood vides profit, exponituity, profitmainalist-at said listers\n",
|
||||
"\n",
|
||||
"--- Temperature = 1.8\n",
|
||||
"Today #39;It: He deat: N.KA Asside\n",
|
||||
"On Sunday i arry Par aldeup patient Wo stele1\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"ename": "KeyError",
|
||||
"evalue": "0",
|
||||
"output_type": "error",
|
||||
"traceback": [
|
||||
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
|
||||
"\u001b[0;31mKeyError\u001b[0m Traceback (most recent call last)",
|
||||
"\u001b[0;32m<ipython-input-33-db32367a0feb>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m 18\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"\\n--- Temperature = {i}\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 19\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mj\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 20\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgenerate_soft\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0msize\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m300\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mstart\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mwords\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mj\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mtemperature\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
|
||||
"\u001b[0;32m<ipython-input-33-db32367a0feb>\u001b[0m in \u001b[0;36mgenerate_soft\u001b[0;34m(model, size, start, temperature)\u001b[0m\n\u001b[1;32m 11\u001b[0m \u001b[0mchars\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 12\u001b[0m \u001b[0minp\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0minp\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mnc\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 13\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mdecode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mchars\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 14\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 15\u001b[0m \u001b[0mwords\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m'Today '\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'On Sunday '\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'Moscow, '\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'President '\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'Little red riding hood '\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
|
||||
"\u001b[0;32m<ipython-input-10-3f5fa6130b1d>\u001b[0m in \u001b[0;36mdecode\u001b[0;34m(x)\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mdecode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0;34m''\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mreverse_map\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mt\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
|
||||
"\u001b[0;32m<ipython-input-10-3f5fa6130b1d>\u001b[0m in \u001b[0;36m<listcomp>\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mdecode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0;34m''\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mreverse_map\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mt\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
|
||||
"\u001b[0;31mKeyError\u001b[0m: 0"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def generate_soft(model,size=100,start='Today ',temperature=1.0):\n",
|
||||
" inp = tokenizer.texts_to_sequences([start])[0]\n",
|
||||
" chars = inp\n",
|
||||
" for i in range(size):\n",
|
||||
" out = model(tf.expand_dims(tf.one_hot(inp,vocab_size),0))[0][-1]\n",
|
||||
" probs = tf.exp(tf.math.log(out)/temperature).numpy().astype(np.float64)\n",
|
||||
" probs = probs/np.sum(probs)\n",
|
||||
" nc = np.argmax(np.random.multinomial(1,probs,1))\n",
|
||||
" if nc==eos_token:\n",
|
||||
" break\n",
|
||||
" chars.append(nc)\n",
|
||||
" inp = inp+[nc]\n",
|
||||
" return decode(chars)\n",
|
||||
"\n",
|
||||
"words = ['Today ','On Sunday ','Moscow, ','President ','Little red riding hood ']\n",
|
||||
" \n",
|
||||
"for i in [0.3,0.8,1.0,1.3,1.8]:\n",
|
||||
" print(f\"\\n--- Temperature = {i}\")\n",
|
||||
" for j in range(5):\n",
|
||||
" print(generate_soft(model,size=300,start=words[j],temperature=i))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Hemos introducido un parámetro más llamado **temperatura**, que se utiliza para indicar qué tan estrictamente debemos adherirnos a la probabilidad más alta. Si la temperatura es 1.0, hacemos un muestreo multinomial justo, y cuando la temperatura se acerca al infinito, todas las probabilidades se vuelven iguales y seleccionamos el siguiente carácter al azar. En el ejemplo a continuación, podemos observar que el texto se vuelve sin sentido cuando aumentamos demasiado la temperatura, y se asemeja a un texto \"cíclico\" generado rígidamente cuando se acerca a 0.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Descargo de responsabilidad**: \nEste documento ha sido traducido utilizando el servicio de traducción automática [Co-op Translator](https://github.com/Azure/co-op-translator). Si bien nos esforzamos por garantizar la precisión, tenga en cuenta que las traducciones automatizadas pueden contener errores o imprecisiones. El documento original en su idioma nativo debe considerarse la fuente autorizada. Para información crítica, se recomienda una traducción profesional realizada por humanos. No nos hacemos responsables de malentendidos o interpretaciones erróneas que puedan surgir del uso de esta traducción.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.8.12 ('py38')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "9fbb7d5fda708537649f71f5f646fcde",
|
||||
"translation_date": "2025-08-31T16:51:40+00:00",
|
||||
"source_file": "lessons/5-NLP/17-GenerativeNetworks/GenerativeTF.ipynb",
|
||||
"language_code": "es"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
|
|
@ -0,0 +1,353 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Mecanismos de atención y transformadores\n",
|
||||
"\n",
|
||||
"Una de las principales desventajas de las redes recurrentes es que todas las palabras en una secuencia tienen el mismo impacto en el resultado. Esto provoca un rendimiento subóptimo en los modelos estándar de codificador-decodificador LSTM para tareas de secuencia a secuencia, como el Reconocimiento de Entidades Nombradas y la Traducción Automática. En realidad, palabras específicas en la secuencia de entrada suelen tener más impacto en las salidas secuenciales que otras.\n",
|
||||
"\n",
|
||||
"Consideremos un modelo de secuencia a secuencia, como la traducción automática. Este se implementa mediante dos redes recurrentes, donde una red (**codificador**) colapsa la secuencia de entrada en un estado oculto, y otra (**decodificador**) desenvuelve este estado oculto en el resultado traducido. El problema con este enfoque es que el estado final de la red tiene dificultades para recordar el inicio de una oración, lo que provoca una baja calidad del modelo en oraciones largas.\n",
|
||||
"\n",
|
||||
"**Los mecanismos de atención** proporcionan un medio para ponderar el impacto contextual de cada vector de entrada en cada predicción de salida de la RNN. Esto se implementa creando atajos entre los estados intermedios de la RNN de entrada y la RNN de salida. De esta manera, al generar el símbolo de salida $y_t$, tomaremos en cuenta todos los estados ocultos de entrada $h_i$, con diferentes coeficientes de peso $\\alpha_{t,i}$. \n",
|
||||
"\n",
|
||||
"\n",
|
||||
"*El modelo codificador-decodificador con mecanismo de atención aditiva en [Bahdanau et al., 2015](https://arxiv.org/pdf/1409.0473.pdf), citado de [este blog](https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html)*\n",
|
||||
"\n",
|
||||
"La matriz de atención $\\{\\alpha_{i,j}\\}$ representaría el grado en que ciertas palabras de entrada influyen en la generación de una palabra dada en la secuencia de salida. A continuación, se muestra un ejemplo de dicha matriz:\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"*Figura tomada de [Bahdanau et al., 2015](https://arxiv.org/pdf/1409.0473.pdf) (Fig.3)*\n",
|
||||
"\n",
|
||||
"Los mecanismos de atención son responsables de gran parte del estado del arte actual o cercano al actual en el procesamiento de lenguaje natural. Sin embargo, agregar atención aumenta significativamente el número de parámetros del modelo, lo que llevó a problemas de escalabilidad con las RNNs. Una restricción clave para escalar las RNNs es que la naturaleza recurrente de los modelos hace que sea difícil agrupar y paralelizar el entrenamiento. En una RNN, cada elemento de una secuencia necesita ser procesado en orden secuencial, lo que significa que no se puede paralelizar fácilmente.\n",
|
||||
"\n",
|
||||
"La adopción de mecanismos de atención combinada con esta restricción llevó a la creación de los modelos transformadores, ahora estado del arte, que conocemos y usamos hoy en día, desde BERT hasta OpenGPT3.\n",
|
||||
"\n",
|
||||
"## Modelos transformadores\n",
|
||||
"\n",
|
||||
"En lugar de transmitir el contexto de cada predicción previa al siguiente paso de evaluación, los **modelos transformadores** utilizan **codificaciones posicionales** y atención para capturar el contexto de una entrada dada dentro de una ventana de texto proporcionada. La imagen a continuación muestra cómo las codificaciones posicionales con atención pueden capturar el contexto dentro de una ventana dada.\n",
|
||||
"\n",
|
||||
" \n",
|
||||
"\n",
|
||||
"Dado que cada posición de entrada se mapea de manera independiente a cada posición de salida, los transformadores pueden paralelizar mejor que las RNNs, lo que permite modelos de lenguaje mucho más grandes y expresivos. Cada cabeza de atención puede usarse para aprender diferentes relaciones entre palabras que mejoran las tareas de procesamiento de lenguaje natural.\n",
|
||||
"\n",
|
||||
"**BERT** (Representaciones de Codificador Bidireccional de Transformadores) es una red transformadora multinivel muy grande con 12 capas para *BERT-base* y 24 para *BERT-large*. El modelo se preentrena primero en un gran corpus de datos de texto (Wikipedia + libros) utilizando entrenamiento no supervisado (predicción de palabras enmascaradas en una oración). Durante el preentrenamiento, el modelo absorbe un nivel significativo de comprensión del lenguaje que luego puede aprovecharse con otros conjuntos de datos mediante ajuste fino. Este proceso se llama **aprendizaje por transferencia**. \n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Existen muchas variaciones de arquitecturas de transformadores, incluyendo BERT, DistilBERT, BigBird, OpenGPT3 y más, que pueden ajustarse. El paquete [HuggingFace](https://github.com/huggingface/) proporciona un repositorio para entrenar muchas de estas arquitecturas con PyTorch. \n",
|
||||
"\n",
|
||||
"## Usando BERT para clasificación de texto\n",
|
||||
"\n",
|
||||
"Veamos cómo podemos usar el modelo BERT preentrenado para resolver nuestra tarea tradicional: clasificación de secuencias. Clasificaremos nuestro conjunto de datos original AG News.\n",
|
||||
"\n",
|
||||
"Primero, carguemos la biblioteca HuggingFace y nuestro conjunto de datos:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Loading dataset...\n",
|
||||
"Building vocab...\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"import torchtext\n",
|
||||
"from torchnlp import *\n",
|
||||
"import transformers\n",
|
||||
"train_dataset, test_dataset, classes, vocab = load_dataset()\n",
|
||||
"vocab_len = len(vocab)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Dado que utilizaremos un modelo BERT preentrenado, necesitaremos usar un tokenizador específico. Primero, cargaremos un tokenizador asociado con el modelo BERT preentrenado.\n",
|
||||
"\n",
|
||||
"La biblioteca HuggingFace contiene un repositorio de modelos preentrenados, que puedes usar simplemente especificando sus nombres como argumentos en las funciones `from_pretrained`. Todos los archivos binarios necesarios para el modelo se descargarán automáticamente.\n",
|
||||
"\n",
|
||||
"Sin embargo, en ciertos casos podrías necesitar cargar tus propios modelos, en cuyo caso puedes especificar el directorio que contiene todos los archivos relevantes, incluidos los parámetros para el tokenizador, el archivo `config.json` con los parámetros del modelo, los pesos binarios, etc.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# To load the model from Internet repository using model name. \n",
|
||||
"# Use this if you are running from your own copy of the notebooks\n",
|
||||
"bert_model = 'bert-base-uncased' \n",
|
||||
"\n",
|
||||
"# To load the model from the directory on disk. Use this for Microsoft Learn module, because we have\n",
|
||||
"# prepared all required files for you.\n",
|
||||
"bert_model = './bert'\n",
|
||||
"\n",
|
||||
"tokenizer = transformers.BertTokenizer.from_pretrained(bert_model)\n",
|
||||
"\n",
|
||||
"MAX_SEQ_LEN = 128\n",
|
||||
"PAD_INDEX = tokenizer.convert_tokens_to_ids(tokenizer.pad_token)\n",
|
||||
"UNK_INDEX = tokenizer.convert_tokens_to_ids(tokenizer.unk_token)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"El objeto `tokenizer` contiene la función `encode` que puede usarse directamente para codificar texto:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[101, 1052, 22123, 2953, 2818, 2003, 1037, 2307, 7705, 2005, 17953, 2361, 102]"
|
||||
]
|
||||
},
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"tokenizer.encode('PyTorch is a great framework for NLP')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Luego, creemos iteradores que utilizaremos durante el entrenamiento para acceder a los datos. Debido a que BERT utiliza su propia función de codificación, necesitaríamos definir una función de relleno similar a `padify` que hemos definido antes:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def pad_bert(b):\n",
|
||||
" # b is the list of tuples of length batch_size\n",
|
||||
" # - first element of a tuple = label, \n",
|
||||
" # - second = feature (text sequence)\n",
|
||||
" # build vectorized sequence\n",
|
||||
" v = [tokenizer.encode(x[1]) for x in b]\n",
|
||||
" # compute max length of a sequence in this minibatch\n",
|
||||
" l = max(map(len,v))\n",
|
||||
" return ( # tuple of two tensors - labels and features\n",
|
||||
" torch.LongTensor([t[0] for t in b]),\n",
|
||||
" torch.stack([torch.nn.functional.pad(torch.tensor(t),(0,l-len(t)),mode='constant',value=0) for t in v])\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=8, collate_fn=pad_bert, shuffle=True)\n",
|
||||
"test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=8, collate_fn=pad_bert)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"En nuestro caso, usaremos un modelo BERT preentrenado llamado `bert-base-uncased`. Vamos a cargar el modelo utilizando el paquete `BertForSequenceClassfication`. Esto asegura que nuestro modelo ya tenga una arquitectura requerida para la clasificación, incluyendo el clasificador final. Verás un mensaje de advertencia que indica que los pesos del clasificador final no están inicializados y que el modelo requeriría preentrenamiento; eso está perfectamente bien, porque es exactamente lo que estamos a punto de hacer.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Some weights of the model checkpoint at ./bert were not used when initializing BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']\n",
|
||||
"- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
|
||||
"- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
|
||||
"Some weights of BertForSequenceClassification were not initialized from the model checkpoint at ./bert and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
|
||||
"You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model = transformers.BertForSequenceClassification.from_pretrained(bert_model,num_labels=4).to(device)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"¡Ahora estamos listos para comenzar el entrenamiento! Dado que BERT ya está preentrenado, queremos empezar con una tasa de aprendizaje bastante pequeña para no alterar los pesos iniciales.\n",
|
||||
"\n",
|
||||
"Todo el trabajo pesado lo realiza el modelo `BertForSequenceClassification`. Cuando llamamos al modelo con los datos de entrenamiento, este devuelve tanto la pérdida como la salida de la red para el minibatch de entrada. Usamos la pérdida para la optimización de parámetros (`loss.backward()` realiza el paso hacia atrás) y `out` para calcular la precisión del entrenamiento comparando las etiquetas obtenidas `labs` (calculadas usando `argmax`) con las etiquetas esperadas `labels`.\n",
|
||||
"\n",
|
||||
"Para controlar el proceso, acumulamos la pérdida y la precisión a lo largo de varias iteraciones y las imprimimos cada `report_freq` ciclos de entrenamiento.\n",
|
||||
"\n",
|
||||
"Este entrenamiento probablemente tomará bastante tiempo, por lo que limitamos el número de iteraciones.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Loss = 1.1254194641113282, Accuracy = 0.585\n",
|
||||
"Loss = 0.6194715118408203, Accuracy = 0.83\n",
|
||||
"Loss = 0.46665248870849607, Accuracy = 0.8475\n",
|
||||
"Loss = 0.4309701919555664, Accuracy = 0.8575\n",
|
||||
"Loss = 0.35427074432373046, Accuracy = 0.8825\n",
|
||||
"Loss = 0.3306886291503906, Accuracy = 0.8975\n",
|
||||
"Loss = 0.30340143203735354, Accuracy = 0.8975\n",
|
||||
"Loss = 0.26139299392700194, Accuracy = 0.915\n",
|
||||
"Loss = 0.26708646774291994, Accuracy = 0.9225\n",
|
||||
"Loss = 0.3667240524291992, Accuracy = 0.8675\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"optimizer = torch.optim.Adam(model.parameters(), lr=2e-5)\n",
|
||||
"\n",
|
||||
"report_freq = 50\n",
|
||||
"iterations = 500 # make this larger to train for longer time!\n",
|
||||
"\n",
|
||||
"model.train()\n",
|
||||
"\n",
|
||||
"i,c = 0,0\n",
|
||||
"acc_loss = 0\n",
|
||||
"acc_acc = 0\n",
|
||||
"\n",
|
||||
"for labels,texts in train_loader:\n",
|
||||
" labels = labels.to(device)-1 # get labels in the range 0-3 \n",
|
||||
" texts = texts.to(device)\n",
|
||||
" loss, out = model(texts, labels=labels)[:2]\n",
|
||||
" labs = out.argmax(dim=1)\n",
|
||||
" acc = torch.mean((labs==labels).type(torch.float32))\n",
|
||||
" optimizer.zero_grad()\n",
|
||||
" loss.backward()\n",
|
||||
" optimizer.step()\n",
|
||||
" acc_loss += loss\n",
|
||||
" acc_acc += acc\n",
|
||||
" i+=1\n",
|
||||
" c+=1\n",
|
||||
" if i%report_freq==0:\n",
|
||||
" print(f\"Loss = {acc_loss.item()/c}, Accuracy = {acc_acc.item()/c}\")\n",
|
||||
" c = 0\n",
|
||||
" acc_loss = 0\n",
|
||||
" acc_acc = 0\n",
|
||||
" iterations-=1\n",
|
||||
" if not iterations:\n",
|
||||
" break"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"¡Puedes ver (especialmente si aumentas el número de iteraciones y esperas lo suficiente) que la clasificación con BERT nos da una precisión bastante buena! Esto se debe a que BERT ya comprende bastante bien la estructura del lenguaje, y solo necesitamos ajustar el clasificador final. Sin embargo, debido a que BERT es un modelo grande, todo el proceso de entrenamiento lleva mucho tiempo y requiere una potencia computacional considerable (GPU, y preferiblemente más de una).\n",
|
||||
"\n",
|
||||
"> **Note:** En nuestro ejemplo, hemos estado utilizando uno de los modelos BERT preentrenados más pequeños. Existen modelos más grandes que probablemente ofrezcan mejores resultados.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Evaluando el rendimiento del modelo\n",
|
||||
"\n",
|
||||
"Ahora podemos evaluar el rendimiento de nuestro modelo en el conjunto de datos de prueba. El bucle de evaluación es bastante similar al bucle de entrenamiento, pero no debemos olvidar cambiar el modelo al modo de evaluación llamando a `model.eval()`.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Final accuracy: 0.9047029702970297\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.eval()\n",
|
||||
"iterations = 100\n",
|
||||
"acc = 0\n",
|
||||
"i = 0\n",
|
||||
"for labels,texts in test_loader:\n",
|
||||
" labels = labels.to(device)-1 \n",
|
||||
" texts = texts.to(device)\n",
|
||||
" _, out = model(texts, labels=labels)[:2]\n",
|
||||
" labs = out.argmax(dim=1)\n",
|
||||
" acc += torch.mean((labs==labels).type(torch.float32))\n",
|
||||
" i+=1\n",
|
||||
" if i>iterations: break\n",
|
||||
" \n",
|
||||
"print(f\"Final accuracy: {acc.item()/i}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Conclusión\n",
|
||||
"\n",
|
||||
"En esta unidad, hemos visto lo sencillo que es tomar un modelo de lenguaje preentrenado de la biblioteca **transformers** y adaptarlo a nuestra tarea de clasificación de texto. De manera similar, los modelos BERT pueden utilizarse para la extracción de entidades, respuesta a preguntas y otras tareas de PLN.\n",
|
||||
"\n",
|
||||
"Los modelos de transformadores representan el estado del arte actual en PLN, y en la mayoría de los casos deberían ser la primera solución con la que empieces a experimentar al implementar soluciones personalizadas de PLN. Sin embargo, comprender los principios básicos subyacentes de las redes neuronales recurrentes discutidos en este módulo es extremadamente importante si deseas construir modelos neuronales avanzados.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Descargo de responsabilidad**: \nEste documento ha sido traducido utilizando el servicio de traducción automática [Co-op Translator](https://github.com/Azure/co-op-translator). Aunque nos esforzamos por garantizar la precisión, tenga en cuenta que las traducciones automatizadas pueden contener errores o imprecisiones. El documento original en su idioma nativo debe considerarse como la fuente autorizada. Para información crítica, se recomienda una traducción profesional realizada por humanos. No nos hacemos responsables de malentendidos o interpretaciones erróneas que puedan surgir del uso de esta traducción.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "py37_pytorch",
|
||||
"language": "python",
|
||||
"name": "conda-env-py37_pytorch-py"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.7.7"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "753865967678a92dbce7d7efbd36d980",
|
||||
"translation_date": "2025-08-31T16:58:41+00:00",
|
||||
"source_file": "lessons/5-NLP/18-Transformers/TransformersPyTorch.ipynb",
|
||||
"language_code": "es"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
|
|
@ -0,0 +1,825 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Mecanismos de atención y transformadores\n",
|
||||
"\n",
|
||||
"Una de las principales desventajas de las redes recurrentes es que todas las palabras en una secuencia tienen el mismo impacto en el resultado. Esto provoca un rendimiento subóptimo en los modelos estándar de codificador-decodificador LSTM para tareas de secuencia a secuencia, como el Reconocimiento de Entidades Nombradas y la Traducción Automática. En realidad, palabras específicas en la secuencia de entrada suelen tener más impacto en las salidas secuenciales que otras.\n",
|
||||
"\n",
|
||||
"Consideremos un modelo de secuencia a secuencia, como la traducción automática. Este se implementa mediante dos redes recurrentes, donde una red (**codificador**) colapsa la secuencia de entrada en un estado oculto, y otra (**decodificador**) desenvuelve este estado oculto en el resultado traducido. El problema con este enfoque es que el estado final de la red tiene dificultades para recordar el inicio de una oración, lo que provoca una calidad deficiente del modelo en oraciones largas.\n",
|
||||
"\n",
|
||||
"Los **mecanismos de atención** proporcionan un medio para ponderar el impacto contextual de cada vector de entrada en cada predicción de salida de la RNN. Esto se implementa creando atajos entre los estados intermedios de la RNN de entrada y la RNN de salida. De esta manera, al generar el símbolo de salida $y_t$, tomaremos en cuenta todos los estados ocultos de entrada $h_i$, con diferentes coeficientes de peso $\\alpha_{t,i}$. \n",
|
||||
"\n",
|
||||
"\n",
|
||||
"*El modelo codificador-decodificador con mecanismo de atención aditiva en [Bahdanau et al., 2015](https://arxiv.org/pdf/1409.0473.pdf), citado de [este blog](https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html)*\n",
|
||||
"\n",
|
||||
"La matriz de atención $\\{\\alpha_{i,j}\\}$ representaría el grado en que ciertas palabras de entrada influyen en la generación de una palabra dada en la secuencia de salida. A continuación se muestra un ejemplo de dicha matriz:\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"*Figura tomada de [Bahdanau et al., 2015](https://arxiv.org/pdf/1409.0473.pdf) (Fig.3)*\n",
|
||||
"\n",
|
||||
"Los mecanismos de atención son responsables de gran parte del estado del arte actual o cercano al actual en el procesamiento de lenguaje natural. Sin embargo, agregar atención aumenta significativamente el número de parámetros del modelo, lo que llevó a problemas de escalabilidad con las RNNs. Una restricción clave para escalar las RNNs es que la naturaleza recurrente de los modelos dificulta el procesamiento por lotes y la paralelización del entrenamiento. En una RNN, cada elemento de una secuencia debe procesarse en orden secuencial, lo que significa que no se puede paralelizar fácilmente.\n",
|
||||
"\n",
|
||||
"La adopción de mecanismos de atención combinada con esta restricción llevó a la creación de los modelos transformadores, ahora estado del arte, que conocemos y usamos hoy en día, desde BERT hasta OpenGPT3.\n",
|
||||
"\n",
|
||||
"## Modelos transformadores\n",
|
||||
"\n",
|
||||
"En lugar de transmitir el contexto de cada predicción anterior al siguiente paso de evaluación, los **modelos transformadores** utilizan **codificaciones posicionales** y **atención** para capturar el contexto de una entrada dada dentro de una ventana de texto proporcionada. La imagen a continuación muestra cómo las codificaciones posicionales con atención pueden capturar el contexto dentro de una ventana dada.\n",
|
||||
"\n",
|
||||
" \n",
|
||||
"\n",
|
||||
"Dado que cada posición de entrada se mapea de manera independiente a cada posición de salida, los transformadores pueden paralelizar mejor que las RNNs, lo que permite modelos de lenguaje mucho más grandes y expresivos. Cada cabeza de atención puede usarse para aprender diferentes relaciones entre palabras que mejoran las tareas de procesamiento de lenguaje natural.\n",
|
||||
"\n",
|
||||
"## Construcción de un modelo transformador simple\n",
|
||||
"\n",
|
||||
"Keras no contiene una capa de transformador integrada, pero podemos construir la nuestra. Como antes, nos enfocaremos en la clasificación de texto del conjunto de datos AG News, aunque vale la pena mencionar que los modelos transformadores muestran los mejores resultados en tareas de procesamiento de lenguaje natural más complejas.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import tensorflow as tf\n",
|
||||
"from tensorflow import keras\n",
|
||||
"import tensorflow_datasets as tfds\n",
|
||||
"import numpy as np\n",
|
||||
"\n",
|
||||
"ds_train, ds_test = tfds.load('ag_news_subset').values()\n",
|
||||
"\n",
|
||||
"def extract_text(x):\n",
|
||||
" return x['title']+' '+x['description']\n",
|
||||
"\n",
|
||||
"def tupelize(x):\n",
|
||||
" return (extract_text(x),x['label'])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Las nuevas capas en Keras deben ser subclases de la clase `Layer` e implementar el método `call`. Comencemos con la capa **Positional Embedding**. Usaremos [algo de código de la documentación oficial de Keras](https://keras.io/examples/nlp/text_classification_with_transformer/). Supondremos que rellenamos todas las secuencias de entrada hasta la longitud `maxlen`.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class TokenAndPositionEmbedding(keras.layers.Layer):\n",
|
||||
" def __init__(self, maxlen, vocab_size, embed_dim):\n",
|
||||
" super(TokenAndPositionEmbedding, self).__init__()\n",
|
||||
" self.token_emb = keras.layers.Embedding(input_dim=vocab_size, output_dim=embed_dim)\n",
|
||||
" self.pos_emb = keras.layers.Embedding(input_dim=maxlen, output_dim=embed_dim)\n",
|
||||
" self.maxlen = maxlen\n",
|
||||
"\n",
|
||||
" def call(self, x):\n",
|
||||
" maxlen = self.maxlen\n",
|
||||
" positions = tf.range(start=0, limit=maxlen, delta=1)\n",
|
||||
" positions = self.pos_emb(positions)\n",
|
||||
" x = self.token_emb(x)\n",
|
||||
" return x+positions"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Esta capa consta de dos capas `Embedding`: una para incrustar tokens (de la manera que hemos discutido antes) y otra para las posiciones de los tokens. Las posiciones de los tokens se crean como una secuencia de números naturales desde 0 hasta `maxlen` utilizando `tf.range`, y luego se pasan a través de la capa de incrustación. Los dos vectores de incrustación resultantes se suman, produciendo una representación incrustada posicionalmente de la entrada con forma `maxlen`$\\times$`embed_dim`.\n",
|
||||
"\n",
|
||||
"Ahora, implementemos el bloque transformer. Este tomará la salida de la capa de incrustación definida anteriormente:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class TransformerBlock(keras.layers.Layer):\n",
|
||||
" def __init__(self, embed_dim, num_heads, ff_dim, rate=0.1):\n",
|
||||
" super(TransformerBlock, self).__init__()\n",
|
||||
" self.att = keras.layers.MultiHeadAttention(num_heads=num_heads, key_dim=embed_dim, name='attn')\n",
|
||||
" self.ffn = keras.Sequential(\n",
|
||||
" [keras.layers.Dense(ff_dim, activation=\"relu\"), keras.layers.Dense(embed_dim),]\n",
|
||||
" )\n",
|
||||
" self.layernorm1 = keras.layers.LayerNormalization(epsilon=1e-6)\n",
|
||||
" self.layernorm2 = keras.layers.LayerNormalization(epsilon=1e-6)\n",
|
||||
" self.dropout1 = keras.layers.Dropout(rate)\n",
|
||||
" self.dropout2 = keras.layers.Dropout(rate)\n",
|
||||
"\n",
|
||||
" def call(self, inputs, training):\n",
|
||||
" attn_output = self.att(inputs, inputs)\n",
|
||||
" attn_output = self.dropout1(attn_output, training=training)\n",
|
||||
" out1 = self.layernorm1(inputs + attn_output)\n",
|
||||
" ffn_output = self.ffn(out1)\n",
|
||||
" ffn_output = self.dropout2(ffn_output, training=training)\n",
|
||||
" return self.layernorm2(out1 + ffn_output)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"El Transformer aplica `MultiHeadAttention` al input codificado posicionalmente para producir el vector de atención con dimensiones `maxlen`$\\times$`embed_dim`, que luego se mezcla con el input y se normaliza utilizando `LayerNormalization`.\n",
|
||||
"\n",
|
||||
"> **Nota**: `LayerNormalization` es similar a `BatchNormalization`, discutido en la parte de *Visión por Computadora* de este camino de aprendizaje, pero normaliza las salidas de la capa anterior para cada muestra de entrenamiento de manera independiente, llevándolas al rango [-1..1].\n",
|
||||
"\n",
|
||||
"La salida de esta capa se pasa luego a través de una red `Dense` (en nuestro caso, un perceptrón de dos capas), y el resultado se suma a la salida final (que nuevamente se somete a normalización). \n",
|
||||
"\n",
|
||||
"Ahora, estamos listos para definir el modelo completo del Transformer:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"sequential_1\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
"Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
"text_vectorization (TextVect (None, 256) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"token_and_position_embedding (None, 256, 32) 648192 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"transformer_block (Transform (None, 256, 32) 10656 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"global_average_pooling1d (Gl (None, 32) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dropout_2 (Dropout) (None, 32) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_2 (Dense) (None, 20) 660 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dropout_3 (Dropout) (None, 20) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_3 (Dense) (None, 4) 84 \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 659,592\n",
|
||||
"Trainable params: 659,592\n",
|
||||
"Non-trainable params: 0\n",
|
||||
"_________________________________________________________________\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"embed_dim = 32 # Embedding size for each token\n",
|
||||
"num_heads = 2 # Number of attention heads\n",
|
||||
"ff_dim = 32 # Hidden layer size in feed forward network inside transformer\n",
|
||||
"maxlen = 256\n",
|
||||
"vocab_size = 20000\n",
|
||||
"\n",
|
||||
"model = keras.models.Sequential([\n",
|
||||
" keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size,output_sequence_length=maxlen, input_shape=(1,)),\n",
|
||||
" TokenAndPositionEmbedding(maxlen, vocab_size, embed_dim),\n",
|
||||
" TransformerBlock(embed_dim, num_heads, ff_dim),\n",
|
||||
" keras.layers.GlobalAveragePooling1D(),\n",
|
||||
" keras.layers.Dropout(0.1),\n",
|
||||
" keras.layers.Dense(20, activation=\"relu\"),\n",
|
||||
" keras.layers.Dropout(0.1),\n",
|
||||
" keras.layers.Dense(4, activation=\"softmax\")\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"model.summary()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Training tokenizer\n",
|
||||
"938/938 [==============================] - 45s 39ms/step - loss: 0.4978 - acc: 0.8068 - val_loss: 0.2808 - val_acc: 0.9124\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7f9c2427a0d0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print('Training tokenizer')\n",
|
||||
"model.layers[0].adapt(ds_train.map(extract_text))\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(128),validation_data=ds_test.map(tupelize).batch(128))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Modelos Transformadores BERT\n",
|
||||
"\n",
|
||||
"**BERT** (Representaciones de Codificadores Bidireccionales de Transformadores) es una red de transformadores de múltiples capas muy grande, con 12 capas para *BERT-base* y 24 para *BERT-large*. El modelo se preentrena inicialmente en un gran corpus de datos de texto (Wikipedia + libros) utilizando entrenamiento no supervisado (predicción de palabras enmascaradas en una oración). Durante el preentrenamiento, el modelo adquiere un nivel significativo de comprensión del lenguaje que luego puede aprovecharse con otros conjuntos de datos mediante ajuste fino. Este proceso se llama **aprendizaje por transferencia**.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Existen muchas variaciones de arquitecturas de Transformadores, incluyendo BERT, DistilBERT, BigBird, OpenGPT3 y más, que pueden ajustarse finamente.\n",
|
||||
"\n",
|
||||
"Veamos cómo podemos usar un modelo BERT preentrenado para resolver nuestro problema tradicional de clasificación de secuencias. Tomaremos la idea y algo de código de la [documentación oficial](https://www.tensorflow.org/text/tutorials/classify_text_with_bert).\n",
|
||||
"\n",
|
||||
"Para cargar modelos preentrenados, utilizaremos **Tensorflow hub**. Primero, carguemos el vectorizador específico de BERT:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"ename": "ModuleNotFoundError",
|
||||
"evalue": "No module named 'tensorflow_text'",
|
||||
"output_type": "error",
|
||||
"traceback": [
|
||||
"\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
|
||||
"\u001b[1;31mModuleNotFoundError\u001b[0m Traceback (most recent call last)",
|
||||
"\u001b[1;32m~\\AppData\\Local\\Temp/ipykernel_41180/4216669875.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[1;32mimport\u001b[0m \u001b[0mtensorflow_text\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 2\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mtensorflow_hub\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mhub\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 3\u001b[0m \u001b[0mvectorizer\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mhub\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mKerasLayer\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
|
||||
"\u001b[1;31mModuleNotFoundError\u001b[0m: No module named 'tensorflow_text'"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import tensorflow_text \n",
|
||||
"import tensorflow_hub as hub\n",
|
||||
"vectorizer = hub.KerasLayer('https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"{'input_type_ids': <tf.Tensor: shape=(1, 128), dtype=int32, numpy=\n",
|
||||
" array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],\n",
|
||||
" dtype=int32)>,\n",
|
||||
" 'input_word_ids': <tf.Tensor: shape=(1, 128), dtype=int32, numpy=\n",
|
||||
" array([[ 101, 1045, 2293, 19081, 102, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0]], dtype=int32)>,\n",
|
||||
" 'input_mask': <tf.Tensor: shape=(1, 128), dtype=int32, numpy=\n",
|
||||
" array([[1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],\n",
|
||||
" dtype=int32)>}"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vectorizer(['I love transformers'])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Es importante que utilices el mismo vectorizador que se usó para entrenar la red original. Además, el vectorizador de BERT devuelve tres componentes:\n",
|
||||
"* `input_word_ids`, que es una secuencia de números de tokens para la oración de entrada\n",
|
||||
"* `input_mask`, que muestra qué parte de la secuencia contiene la entrada real y cuál es relleno. Es similar a la máscara producida por la capa `Masking`\n",
|
||||
"* `input_type_ids` se utiliza para tareas de modelado de lenguaje y permite especificar dos oraciones de entrada en una sola secuencia.\n",
|
||||
"\n",
|
||||
"Luego, podemos instanciar el extractor de características de BERT:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"bert = hub.KerasLayer('https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-4_H-128_A-2/1')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"pooled_output -> (1, 128)\n",
|
||||
"encoder_outputs -> 4\n",
|
||||
"sequence_output -> (1, 128, 128)\n",
|
||||
"default -> (1, 128)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"z = bert(vectorizer(['I love transformers']))\n",
|
||||
"for i,x in z.items():\n",
|
||||
" print(f\"{i} -> { len(x) if isinstance(x, list) else x.shape }\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Entonces, la capa BERT devuelve una serie de resultados útiles:\n",
|
||||
"* `pooled_output` es el resultado de promediar todos los tokens en la secuencia. Puedes verlo como una representación semántica inteligente de toda la red. Es equivalente a la salida de la capa `GlobalAveragePooling1D` en nuestro modelo anterior.\n",
|
||||
"* `sequence_output` es la salida de la última capa del transformador (corresponde a la salida de `TransformerBlock` en nuestro modelo anterior).\n",
|
||||
"* `encoder_outputs` son las salidas de todas las capas del transformador. Dado que hemos cargado un modelo BERT de 4 capas (como probablemente puedas deducir del nombre, que contiene `4_H`), tiene 4 tensores. El último es el mismo que `sequence_output`.\n",
|
||||
"\n",
|
||||
"Ahora definiremos el modelo de clasificación de extremo a extremo. Usaremos la *definición funcional del modelo*, donde definimos la entrada del modelo y luego proporcionamos una serie de expresiones para calcular su salida. También haremos que los pesos del modelo BERT no sean entrenables y entrenaremos solo el clasificador final:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"model\"\n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"Layer (type) Output Shape Param # Connected to \n",
|
||||
"==================================================================================================\n",
|
||||
"input_1 (InputLayer) [(None,)] 0 \n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"keras_layer (KerasLayer) {'input_type_ids': ( 0 input_1[0][0] \n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"keras_layer_1 (KerasLayer) {'pooled_output': (N 4782465 keras_layer[0][0] \n",
|
||||
" keras_layer[0][1] \n",
|
||||
" keras_layer[0][2] \n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"dropout_4 (Dropout) (None, 128) 0 keras_layer_1[0][5] \n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"dense_4 (Dense) (None, 4) 516 dropout_4[0][0] \n",
|
||||
"==================================================================================================\n",
|
||||
"Total params: 4,782,981\n",
|
||||
"Trainable params: 516\n",
|
||||
"Non-trainable params: 4,782,465\n",
|
||||
"__________________________________________________________________________________________________\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"inp = keras.Input(shape=(),dtype=tf.string)\n",
|
||||
"x = vectorizer(inp)\n",
|
||||
"x = bert(x)\n",
|
||||
"x = keras.layers.Dropout(0.1)(x['pooled_output'])\n",
|
||||
"out = keras.layers.Dense(4,activation='softmax')(x)\n",
|
||||
"model = keras.models.Model(inp,out)\n",
|
||||
"bert.trainable = False\n",
|
||||
"model.summary()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"938/938 [==============================] - 528s 559ms/step - loss: 0.8056 - acc: 0.6983 - val_loss: 0.5953 - val_acc: 0.7888\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7f9bb1e36d00>"
|
||||
]
|
||||
},
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(128),validation_data=ds_test.map(tupelize).batch(128))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"A pesar de que hay pocos parámetros entrenables, el proceso es bastante lento porque el extractor de características de BERT es computacionalmente pesado. Parece que no pudimos alcanzar una precisión razonable, ya sea por falta de entrenamiento o por falta de parámetros en el modelo.\n",
|
||||
"\n",
|
||||
"Intentemos desbloquear los pesos de BERT y entrenarlo también. Esto requiere una tasa de aprendizaje muy pequeña y una estrategia de entrenamiento más cuidadosa con **warmup**, utilizando el optimizador **AdamW**. Usaremos el paquete `tf-models-official` para crear el optimizador:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"model\"\n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"Layer (type) Output Shape Param # Connected to \n",
|
||||
"==================================================================================================\n",
|
||||
"input_1 (InputLayer) [(None,)] 0 \n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"keras_layer (KerasLayer) {'input_type_ids': ( 0 input_1[0][0] \n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"keras_layer_1 (KerasLayer) {'pooled_output': (N 4782465 keras_layer[0][0] \n",
|
||||
" keras_layer[0][1] \n",
|
||||
" keras_layer[0][2] \n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"dropout_4 (Dropout) (None, 128) 0 keras_layer_1[0][5] \n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"dense_4 (Dense) (None, 4) 516 dropout_4[0][0] \n",
|
||||
"==================================================================================================\n",
|
||||
"Total params: 4,782,981\n",
|
||||
"Trainable params: 4,782,980\n",
|
||||
"Non-trainable params: 1\n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"938/938 [==============================] - 629s 664ms/step - loss: 0.6344 - acc: 0.7658 - val_loss: 0.4876 - val_acc: 0.8247\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7f9bb0bd0070>"
|
||||
]
|
||||
},
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from official.nlp import optimization \n",
|
||||
"bert.trainable=True\n",
|
||||
"model.summary()\n",
|
||||
"epochs = 3\n",
|
||||
"opt = optimization.create_optimizer(\n",
|
||||
" init_lr=3e-5,\n",
|
||||
" num_train_steps=epochs*len(ds_train),\n",
|
||||
" num_warmup_steps=0.1*epochs*len(ds_train),\n",
|
||||
" optimizer_type='adamw')\n",
|
||||
"\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer=opt)\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(128),validation_data=ds_test.map(tupelize).batch(128))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Como puedes ver, el entrenamiento avanza bastante lento, pero podrías experimentar y entrenar el modelo durante algunas épocas (5-10) para ver si puedes obtener el mejor resultado en comparación con los enfoques que hemos utilizado anteriormente.\n",
|
||||
"\n",
|
||||
"## Biblioteca Huggingface Transformers\n",
|
||||
"\n",
|
||||
"Otra forma muy común (y un poco más sencilla) de usar modelos Transformer es el [paquete HuggingFace](https://github.com/huggingface/), que proporciona bloques simples para diferentes tareas de PLN. Está disponible tanto para Tensorflow como para PyTorch, otro marco de redes neuronales muy popular.\n",
|
||||
"\n",
|
||||
"> **Nota**: Si no estás interesado en ver cómo funciona la biblioteca Transformers, puedes saltar al final de este notebook, ya que no verás nada sustancialmente diferente de lo que hemos hecho anteriormente. Estaremos repitiendo los mismos pasos de entrenamiento del modelo BERT utilizando una biblioteca diferente y un modelo sustancialmente más grande. Por lo tanto, el proceso implica un entrenamiento bastante largo, así que podrías simplemente revisar el código.\n",
|
||||
"\n",
|
||||
"Veamos cómo se puede resolver nuestro problema utilizando [Huggingface Transformers](http://huggingface.co).\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Lo primero que necesitamos hacer es elegir el modelo que vamos a utilizar. Además de algunos modelos integrados, Huggingface cuenta con un [repositorio de modelos en línea](https://huggingface.co/models), donde puedes encontrar muchos más modelos preentrenados creados por la comunidad. Todos esos modelos se pueden cargar y usar simplemente proporcionando el nombre del modelo. Todos los archivos binarios necesarios para el modelo se descargarán automáticamente.\n",
|
||||
"\n",
|
||||
"En ciertos casos, necesitarás cargar tus propios modelos, en cuyo caso puedes especificar el directorio que contiene todos los archivos relevantes, incluyendo los parámetros para el tokenizer, el archivo `config.json` con los parámetros del modelo, los pesos binarios, etc.\n",
|
||||
"\n",
|
||||
"A partir del nombre del modelo, podemos instanciar tanto el modelo como el tokenizer. Comencemos con un tokenizer:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import transformers\n",
|
||||
"\n",
|
||||
"# To load the model from Internet repository using model name. \n",
|
||||
"# Use this if you are running from your own copy of the notebooks\n",
|
||||
"bert_model = 'bert-base-uncased' \n",
|
||||
"\n",
|
||||
"# To load the model from the directory on disk. Use this for Microsoft Learn module, because we have\n",
|
||||
"# prepared all required files for you.\n",
|
||||
"#bert_model = './bert'\n",
|
||||
"\n",
|
||||
"tokenizer = transformers.BertTokenizer.from_pretrained(bert_model)\n",
|
||||
"\n",
|
||||
"MAX_SEQ_LEN = 128\n",
|
||||
"PAD_INDEX = tokenizer.convert_tokens_to_ids(tokenizer.pad_token)\n",
|
||||
"UNK_INDEX = tokenizer.convert_tokens_to_ids(tokenizer.unk_token)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"El objeto `tokenizer` contiene la función `encode` que puede usarse directamente para codificar texto:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[101, 23435, 12314, 2003, 1037, 2307, 7705, 2005, 17953, 2361, 102]"
|
||||
]
|
||||
},
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"tokenizer.encode('Tensorflow is a great framework for NLP')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Podemos también usar el tokenizador para codificar una secuencia de una manera adecuada para pasar al modelo, es decir, incluyendo los campos `token_ids`, `input_mask`, etc. También podemos especificar que queremos tensores de Tensorflow proporcionando el argumento `return_tensors='tf'`:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"{'input_ids': <tf.Tensor: shape=(1, 5), dtype=int32, numpy=array([[ 101, 7592, 1010, 2045, 102]], dtype=int32)>, 'token_type_ids': <tf.Tensor: shape=(1, 5), dtype=int32, numpy=array([[0, 0, 0, 0, 0]], dtype=int32)>, 'attention_mask': <tf.Tensor: shape=(1, 5), dtype=int32, numpy=array([[1, 1, 1, 1, 1]], dtype=int32)>}"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"tokenizer(['Hello, there'],return_tensors='tf')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"En nuestro caso, utilizaremos un modelo BERT preentrenado llamado `bert-base-uncased`. *Uncased* indica que el modelo no distingue entre mayúsculas y minúsculas.\n",
|
||||
"\n",
|
||||
"Al entrenar el modelo, necesitamos proporcionar una secuencia tokenizada como entrada, y por lo tanto diseñaremos una canalización de procesamiento de datos. Dado que `tokenizer.encode` es una función de Python, utilizaremos el mismo enfoque que en la última unidad llamándola mediante `py_function`:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 31,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def process(x):\n",
|
||||
" return tokenizer.encode(x.numpy().decode('utf-8'),return_tensors='tf',padding='max_length',max_length=MAX_SEQ_LEN,truncation=True)[0]\n",
|
||||
"\n",
|
||||
"def process_fn(x):\n",
|
||||
" s = x['title']+' '+x['description']\n",
|
||||
" e = tf.py_function(process,inp=[s],Tout=(tf.int32))\n",
|
||||
" e.set_shape(MAX_SEQ_LEN)\n",
|
||||
" return e,x['label']"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Ahora podemos cargar el modelo real utilizando el paquete `BertForSequenceClassification`. Esto asegura que nuestro modelo ya tenga una arquitectura requerida para la clasificación, incluyendo el clasificador final. Verás un mensaje de advertencia que indica que los pesos del clasificador final no están inicializados y que el modelo requeriría preentrenamiento; eso está perfectamente bien, porque es exactamente lo que estamos a punto de hacer.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 32,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"model = transformers.TFBertForSequenceClassification.from_pretrained(bert_model,num_labels=4,output_attentions=False)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 33,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"tf_bert_for_sequence_classification_1\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
"Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
"bert (TFBertMainLayer) multiple 109482240 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dropout_75 (Dropout) multiple 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"classifier (Dense) multiple 3076 \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 109,485,316\n",
|
||||
"Trainable params: 109,485,316\n",
|
||||
"Non-trainable params: 0\n",
|
||||
"_________________________________________________________________\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.summary()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Como puedes ver en `summary()`, ¡el modelo contiene casi 110 millones de parámetros! Presumiblemente, si queremos una tarea de clasificación simple en un conjunto de datos relativamente pequeño, no queremos entrenar la capa base de BERT:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 34,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"tf_bert_for_sequence_classification_1\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
"Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
"bert (TFBertMainLayer) multiple 109482240 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dropout_75 (Dropout) multiple 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"classifier (Dense) multiple 3076 \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 109,485,316\n",
|
||||
"Trainable params: 3,076\n",
|
||||
"Non-trainable params: 109,482,240\n",
|
||||
"_________________________________________________________________\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.layers[0].trainable = False\n",
|
||||
"model.summary()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"¡Ahora estamos listos para comenzar el entrenamiento!\n",
|
||||
"\n",
|
||||
"> **Nota**: ¡Entrenar un modelo BERT a gran escala puede llevar mucho tiempo! Por lo tanto, solo lo entrenaremos durante las primeras 32 tandas. Esto es solo para mostrar cómo se configura el entrenamiento del modelo. Si estás interesado en probar un entrenamiento a gran escala, simplemente elimina los parámetros `steps_per_epoch` y `validation_steps`, ¡y prepárate para esperar!\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 30,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"32/32 [==============================] - 142s 4s/step - loss: 1.3896 - acc: 0.2500 - val_loss: 1.3863 - val_acc: 0.2480\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7f1d40a4b6a0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 30,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.compile('adam','sparse_categorical_crossentropy',['acc'])\n",
|
||||
"tf.get_logger().setLevel('ERROR')\n",
|
||||
"model.fit(ds_train.map(process_fn).batch(32),validation_data=ds_test.map(process_fn).batch(32),steps_per_epoch=32,validation_steps=2)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Si aumentas el número de iteraciones y esperas lo suficiente, y entrenas durante varias épocas, puedes esperar que la clasificación con BERT nos dé la mejor precisión. Esto se debe a que BERT ya comprende bastante bien la estructura del lenguaje, y solo necesitamos ajustar el clasificador final. Sin embargo, dado que BERT es un modelo grande, todo el proceso de entrenamiento lleva mucho tiempo y requiere una potencia computacional considerable (GPU, y preferiblemente más de una).\n",
|
||||
"\n",
|
||||
"> **Note:** En nuestro ejemplo, hemos estado utilizando uno de los modelos BERT preentrenados más pequeños. Existen modelos más grandes que probablemente ofrezcan mejores resultados.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Puntos clave\n",
|
||||
"\n",
|
||||
"En esta unidad, hemos explorado arquitecturas de modelos muy recientes basadas en **transformers**. Las hemos aplicado a nuestra tarea de clasificación de texto, pero de manera similar, los modelos BERT pueden utilizarse para extracción de entidades, respuesta a preguntas y otras tareas de procesamiento de lenguaje natural.\n",
|
||||
"\n",
|
||||
"Los modelos basados en transformers representan el estado del arte actual en NLP, y en la mayoría de los casos deberían ser la primera solución con la que empieces a experimentar al implementar soluciones personalizadas de NLP. Sin embargo, comprender los principios básicos subyacentes de las redes neuronales recurrentes discutidos en este módulo es extremadamente importante si deseas construir modelos neuronales avanzados.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**Descargo de responsabilidad**: \nEste documento ha sido traducido utilizando el servicio de traducción automática [Co-op Translator](https://github.com/Azure/co-op-translator). Aunque nos esforzamos por garantizar la precisión, tenga en cuenta que las traducciones automatizadas pueden contener errores o imprecisiones. El documento original en su idioma nativo debe considerarse como la fuente autorizada. Para información crítica, se recomienda una traducción profesional realizada por humanos. No nos hacemos responsables de malentendidos o interpretaciones erróneas que puedan surgir del uso de esta traducción.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "py38_tensorflow",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "ab59c532409774988ab875f2260e8e53",
|
||||
"translation_date": "2025-08-31T17:01:22+00:00",
|
||||
"source_file": "lessons/5-NLP/18-Transformers/TransformersTF.ipynb",
|
||||
"language_code": "es"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue