🌐 Update translations via Co-op Translator
This commit is contained in:
leestott
GitHub
parent
5c139398b3
commit
e7369eb9ad
|
|
@ -1,8 +1,8 @@
|
|||
<!--
|
||||
CO_OP_TRANSLATOR_METADATA:
|
||||
{
|
||||
"original_hash": "f3a6b0ddf7e6e3f33b2a543baf086dc9",
|
||||
"translation_date": "2025-08-26T07:47:04+00:00",
|
||||
"original_hash": "335144a809cc5695988bb1504c67066d",
|
||||
"translation_date": "2025-08-28T10:30:04+00:00",
|
||||
"source_file": "README.md",
|
||||
"language_code": "bn"
|
||||
}
|
||||
|
|
@ -21,105 +21,116 @@ CO_OP_TRANSLATOR_METADATA:
|
|||
|
||||
[](https://discord.gg/zxKYvhSnVp?WT.mc_id=academic-000002-leestott)
|
||||
|
||||
# শিক্ষার্থীদের জন্য কৃত্রিম বুদ্ধিমত্তা - একটি পাঠ্যক্রম
|
||||
# কৃত্রিম বুদ্ধিমত্তা শিক্ষার্থীদের জন্য - একটি পাঠ্যক্রম
|
||||
|
||||
| দ্বারা ](./lessons/sketchnotes/ai-overview.png)|
|
||||
|:---:|
|
||||
| শিক্ষার্থীদের জন্য কৃত্রিম বুদ্ধিমত্তা - _[@girlie_mac](https://twitter.com/girlie_mac) দ্বারা স্কেচনোট_ |
|
||||
||
|
||||
|:---:|
|
||||
| শিক্ষার্থীদের জন্য কৃত্রিম বুদ্ধিমত্তা - _স্কেচনোট [@girlie_mac](https://twitter.com/girlie_mac) দ্বারা_ |
|
||||
|
||||
**কৃত্রিম বুদ্ধিমত্তা** (AI)-এর জগৎ আবিষ্কার করুন আমাদের ১২-সপ্তাহের, ২৪-টি পাঠের পাঠ্যক্রমের মাধ্যমে! এতে রয়েছে ব্যবহারিক পাঠ, কুইজ এবং ল্যাব। এই পাঠ্যক্রমটি শিক্ষানবিসদের জন্য উপযোগী এবং এতে TensorFlow এবং PyTorch-এর মতো টুলস এবং AI-এর নৈতিকতা অন্তর্ভুক্ত।
|
||||
**কৃত্রিম বুদ্ধিমত্তা** (AI)-এর জগৎ আবিষ্কার করুন আমাদের ১২-সপ্তাহের, ২৪-টি পাঠের পাঠ্যক্রমের মাধ্যমে! এতে রয়েছে ব্যবহারিক পাঠ, কুইজ এবং ল্যাব। এই পাঠ্যক্রমটি শিক্ষার্থীবান্ধব এবং এতে TensorFlow এবং PyTorch-এর মতো টুলস এবং AI-এর নৈতিকতা অন্তর্ভুক্ত রয়েছে।
|
||||
|
||||
### 🌐 বহু-ভাষার সমর্থন
|
||||
|
||||
#### GitHub Action এর মাধ্যমে সমর্থিত (স্বয়ংক্রিয় এবং সর্বদা আপডেটেড)
|
||||
|
||||
[French](../fr/README.md) | [Spanish](../es/README.md) | [German](../de/README.md) | [Russian](../ru/README.md) | [Arabic](../ar/README.md) | [Persian (Farsi)](../fa/README.md) | [Urdu](../ur/README.md) | [Chinese (Simplified)](../zh/README.md) | [Chinese (Traditional, Macau)](../mo/README.md) | [Chinese (Traditional, Hong Kong)](../hk/README.md) | [Chinese (Traditional, Taiwan)](../tw/README.md) | [Japanese](../ja/README.md) | [Korean](../ko/README.md) | [Hindi](../hi/README.md) | [Bengali](./README.md) | [Marathi](../mr/README.md) | [Nepali](../ne/README.md) | [Punjabi (Gurmukhi)](../pa/README.md) | [Portuguese (Portugal)](../pt/README.md) | [Portuguese (Brazil)](../br/README.md) | [Italian](../it/README.md) | [Polish](../pl/README.md) | [Turkish](../tr/README.md) | [Greek](../el/README.md) | [Thai](../th/README.md) | [Swedish](../sv/README.md) | [Danish](../da/README.md) | [Norwegian](../no/README.md) | [Finnish](../fi/README.md) | [Dutch](../nl/README.md) | [Hebrew](../he/README.md) | [Vietnamese](../vi/README.md) | [Indonesian](../id/README.md) | [Malay](../ms/README.md) | [Tagalog (Filipino)](../tl/README.md) | [Swahili](../sw/README.md) | [Hungarian](../hu/README.md) | [Czech](../cs/README.md) | [Slovak](../sk/README.md) | [Romanian](../ro/README.md) | [Bulgarian](../bg/README.md) | [Serbian (Cyrillic)](../sr/README.md) | [Croatian](../hr/README.md) | [Slovenian](../sl/README.md) | [Ukrainian](../uk/README.md) | [Burmese (Myanmar)](../my/README.md)
|
||||
|
||||
**যদি আপনি অতিরিক্ত ভাষার অনুবাদ চান, সমর্থিত ভাষার তালিকা [এখানে](https://github.com/Azure/co-op-translator/blob/main/getting_started/supported-languages.md) পাওয়া যাবে।**
|
||||
|
||||
## সম্প্রদায়ে যোগ দিন
|
||||
[](https://discord.gg/kzRShWzttr)
|
||||
|
||||
## আপনি কী শিখবেন
|
||||
|
||||
**[কোর্সের মানচিত্র](http://soshnikov.com/courses/ai-for-beginners/mindmap.html)**
|
||||
|
||||
এই পাঠ্যক্রমে আপনি শিখবেন:
|
||||
এই পাঠ্যক্রমে, আপনি শিখবেন:
|
||||
|
||||
* কৃত্রিম বুদ্ধিমত্তার বিভিন্ন পদ্ধতি, যার মধ্যে রয়েছে "পুরনো ভালো" প্রতীকী পদ্ধতি **জ্ঞান উপস্থাপনা** এবং যুক্তি ([GOFAI](https://en.wikipedia.org/wiki/Symbolic_artificial_intelligence))।
|
||||
* **নিউরাল নেটওয়ার্ক** এবং **ডিপ লার্নিং**, যা আধুনিক AI-এর মূল ভিত্তি। আমরা এই গুরুত্বপূর্ণ বিষয়গুলির ধারণাগুলি ব্যাখ্যা করব দুটি জনপ্রিয় ফ্রেমওয়ার্ক - [TensorFlow](http://Tensorflow.org) এবং [PyTorch](http://pytorch.org)-এ কোড ব্যবহার করে।
|
||||
* **ইমেজ এবং টেক্সট নিয়ে কাজ করার জন্য নিউরাল আর্কিটেকচার**। আমরা সাম্প্রতিক মডেলগুলি কভার করব, তবে সর্বাধুনিক প্রযুক্তিতে কিছুটা ঘাটতি থাকতে পারে।
|
||||
* কৃত্রিম বুদ্ধিমত্তার বিভিন্ন পদ্ধতি, যার মধ্যে রয়েছে "পুরনো দিনের" প্রতীকী পদ্ধতি **জ্ঞান উপস্থাপনা** এবং যুক্তি ([GOFAI](https://en.wikipedia.org/wiki/Symbolic_artificial_intelligence))।
|
||||
* **নিউরাল নেটওয়ার্ক** এবং **ডিপ লার্নিং**, যা আধুনিক AI-এর মূল ভিত্তি। আমরা এই গুরুত্বপূর্ণ বিষয়গুলোর ধারণা দুটি জনপ্রিয় ফ্রেমওয়ার্ক - [TensorFlow](http://Tensorflow.org) এবং [PyTorch](http://pytorch.org)-এ কোড ব্যবহার করে ব্যাখ্যা করব।
|
||||
* চিত্র এবং টেক্সট নিয়ে কাজ করার জন্য **নিউরাল আর্কিটেকচার**। আমরা সাম্প্রতিক মডেলগুলো কভার করব, তবে সর্বাধুনিক প্রযুক্তিতে কিছুটা ঘাটতি থাকতে পারে।
|
||||
* কম জনপ্রিয় AI পদ্ধতি, যেমন **জেনেটিক অ্যালগরিদম** এবং **মাল্টি-এজেন্ট সিস্টেম**।
|
||||
|
||||
আমরা এই পাঠ্যক্রমে যা কভার করব না:
|
||||
এই পাঠ্যক্রমে যা অন্তর্ভুক্ত করা হবে না:
|
||||
|
||||
> [এই কোর্সের জন্য সমস্ত অতিরিক্ত সম্পদ Microsoft Learn সংগ্রহে খুঁজুন](https://learn.microsoft.com/en-us/collections/7w28iy2xrqzdj0?WT.mc_id=academic-77998-bethanycheum)
|
||||
> [এই কোর্সের জন্য সমস্ত অতিরিক্ত সম্পদ Microsoft Learn সংগ্রহে পাওয়া যাবে](https://learn.microsoft.com/en-us/collections/7w28iy2xrqzdj0?WT.mc_id=academic-77998-bethanycheum)
|
||||
|
||||
* **ব্যবসায়িক ক্ষেত্রে AI-এর ব্যবহার**। Microsoft Learn-এ [ব্যবসায়িক ব্যবহারকারীদের জন্য AI পরিচিতি](https://docs.microsoft.com/learn/paths/introduction-ai-for-business-users/?WT.mc_id=academic-77998-bethanycheum) শেখার পথ বা [AI বিজনেস স্কুল](https://www.microsoft.com/ai/ai-business-school/?WT.mc_id=academic-77998-bethanycheum), যা [INSEAD](https://www.insead.edu/)-এর সহযোগিতায় তৈরি করা হয়েছে, গ্রহণ করার কথা বিবেচনা করুন।
|
||||
* **ক্লাসিক মেশিন লার্নিং**, যা আমাদের [শিক্ষার্থীদের জন্য মেশিন লার্নিং পাঠ্যক্রমে](http://github.com/Microsoft/ML-for-Beginners) ভালোভাবে বর্ণিত।
|
||||
* **ব্যবসায়িক ক্ষেত্রে AI-এর ব্যবহার**। Microsoft Learn-এ [ব্যবসায়িক ব্যবহারকারীদের জন্য AI পরিচিতি](https://docs.microsoft.com/learn/paths/introduction-ai-for-business-users/?WT.mc_id=academic-77998-bethanycheum) শেখার পথ বা [AI ব্যবসায়িক স্কুল](https://www.microsoft.com/ai/ai-business-school/?WT.mc_id=academic-77998-bethanycheum), যা [INSEAD](https://www.insead.edu/) এর সহযোগিতায় তৈরি করা হয়েছে, গ্রহণ করার কথা বিবেচনা করুন।
|
||||
* **ক্লাসিক মেশিন লার্নিং**, যা আমাদের [শিক্ষার্থীদের জন্য মেশিন লার্নিং পাঠ্যক্রমে](http://github.com/Microsoft/ML-for-Beginners) ভালোভাবে বর্ণনা করা হয়েছে।
|
||||
* **[কগনিটিভ সার্ভিসেস](https://azure.microsoft.com/services/cognitive-services/?WT.mc_id=academic-77998-bethanycheum)** ব্যবহার করে তৈরি করা ব্যবহারিক AI অ্যাপ্লিকেশন। এর জন্য, আমরা Microsoft Learn-এ [ভিশন](https://docs.microsoft.com/learn/paths/create-computer-vision-solutions-azure-cognitive-services/?WT.mc_id=academic-77998-bethanycheum), [ন্যাচারাল ল্যাঙ্গুয়েজ প্রসেসিং](https://docs.microsoft.com/learn/paths/explore-natural-language-processing/?WT.mc_id=academic-77998-bethanycheum), **[Azure OpenAI সার্ভিসের সাথে জেনারেটিভ AI](https://learn.microsoft.com/en-us/training/paths/develop-ai-solutions-azure-openai/?WT.mc_id=academic-77998-bethanycheum)** এবং অন্যান্য মডিউল দিয়ে শুরু করার পরামর্শ দিই।
|
||||
* নির্দিষ্ট ML **ক্লাউড ফ্রেমওয়ার্ক**, যেমন [Azure Machine Learning](https://azure.microsoft.com/services/machine-learning/?WT.mc_id=academic-77998-bethanycheum), [Microsoft Fabric](https://learn.microsoft.com/en-us/training/paths/get-started-fabric/?WT.mc_id=academic-77998-bethanycheum), বা [Azure Databricks](https://docs.microsoft.com/learn/paths/data-engineer-azure-databricks?WT.mc_id=academic-77998-bethanycheum)। [Azure Machine Learning](https://docs.microsoft.com/learn/paths/build-ai-solutions-with-azure-ml-service/?WT.mc_id=academic-77998-bethanycheum) এবং [Azure Databricks](https://docs.microsoft.com/learn/paths/build-operate-machine-learning-solutions-azure-databricks/?WT.mc_id=academic-77998-bethanycheum) শেখার পথ ব্যবহার করার কথা বিবেচনা করুন।
|
||||
* **কথোপকথনমূলক AI** এবং **চ্যাট বট**। এর জন্য একটি আলাদা [কথোপকথনমূলক AI সমাধান তৈরি করুন](https://docs.microsoft.com/learn/paths/create-conversational-ai-solutions/?WT.mc_id=academic-77998-bethanycheum) শেখার পথ রয়েছে, এবং আরও বিশদে জানতে [এই ব্লগ পোস্টটি](https://soshnikov.com/azure/hello-bot-conversational-ai-on-microsoft-platform/) দেখুন।
|
||||
* **কথোপকথনমূলক AI** এবং **চ্যাট বট**। এর জন্য একটি আলাদা [কথোপকথনমূলক AI সমাধান তৈরি করুন](https://docs.microsoft.com/learn/paths/create-conversational-ai-solutions/?WT.mc_id=academic-77998-bethanycheum) শেখার পথ রয়েছে, এবং আরও বিশদ জানতে [এই ব্লগ পোস্টটি](https://soshnikov.com/azure/hello-bot-conversational-ai-on-microsoft-platform/) দেখতে পারেন।
|
||||
* **ডিপ লার্নিং-এর গভীর গণিত**। এর জন্য, আমরা Ian Goodfellow, Yoshua Bengio এবং Aaron Courville-এর [Deep Learning](https://www.amazon.com/Deep-Learning-Adaptive-Computation-Machine/dp/0262035618) বইটি সুপারিশ করি, যা অনলাইনে [https://www.deeplearningbook.org/](https://www.deeplearningbook.org/) এও পাওয়া যায়।
|
||||
|
||||
_ক্লাউডে AI_ বিষয়গুলির একটি সহজ পরিচিতির জন্য, আপনি [Azure-এ কৃত্রিম বুদ্ধিমত্তার সাথে শুরু করুন](https://docs.microsoft.com/learn/paths/get-started-with-artificial-intelligence-on-azure/?WT.mc_id=academic-77998-bethanycheum) শেখার পথ গ্রহণ করতে পারেন।
|
||||
_ক্লাউডে AI_-এর বিষয়গুলোর একটি সহজ পরিচিতির জন্য, আপনি [Azure-এ কৃত্রিম বুদ্ধিমত্তার সাথে শুরু করুন](https://docs.microsoft.com/learn/paths/get-started-with-artificial-intelligence-on-azure/?WT.mc_id=academic-77998-bethanycheum) শেখার পথ গ্রহণ করতে পারেন।
|
||||
|
||||
# বিষয়বস্তু
|
||||
|
||||
| | পাঠের লিঙ্ক | PyTorch/Keras/TensorFlow | ল্যাব |
|
||||
| :-: | :------------------------------------------------------------------------------------------------------------------------------------------: | :---------------------------------------------------------------------------------------------: | ------------------------------------------------------------------------------ |
|
||||
| 0 | [কোর্স সেটআপ](./lessons/0-course-setup/setup.md) | [আপনার ডেভেলপমেন্ট এনভায়রনমেন্ট সেটআপ করুন](./lessons/0-course-setup/how-to-run.md) | |
|
||||
| I | [**AI-এর পরিচিতি**](./lessons/1-Intro/README.md) | | |
|
||||
| 01 | [AI-এর পরিচিতি এবং ইতিহাস](./lessons/1-Intro/README.md) | - | - |
|
||||
| II | **প্রতীকী AI** |
|
||||
| 02 | [জ্ঞান উপস্থাপনা এবং বিশেষজ্ঞ সিস্টেম](./lessons/2-Symbolic/README.md) | [বিশেষজ্ঞ সিস্টেম](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/2-Symbolic/Animals.ipynb) / [অন্তর্নির্মিত জ্ঞান](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/2-Symbolic/FamilyOntology.ipynb) /[ধারণাগত গ্রাফ](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/2-Symbolic/MSConceptGraph.ipynb) | |
|
||||
| III | [**নিউরাল নেটওয়ার্কের পরিচিতি**](./lessons/3-NeuralNetworks/README.md) |||
|
||||
| 03 | [পারসেপট্রন](./lessons/3-NeuralNetworks/03-Perceptron/README.md) | [নোটবুক](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/3-NeuralNetworks/03-Perceptron/Perceptron.ipynb) | [ল্যাব](./lessons/3-NeuralNetworks/03-Perceptron/lab/README.md) |
|
||||
| 04 | [মাল্টি-লেয়ার পারসেপট্রন এবং আমাদের নিজস্ব ফ্রেমওয়ার্ক তৈরি করা](./lessons/3-NeuralNetworks/04-OwnFramework/README.md) | [নোটবুক](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/3-NeuralNetworks/04-OwnFramework/OwnFramework.ipynb) | [ল্যাব](./lessons/3-NeuralNetworks/04-OwnFramework/lab/README.md) |
|
||||
| 05 | [ফ্রেমওয়ার্কের পরিচিতি (PyTorch/TensorFlow) এবং ওভারফিটিং](./lessons/3-NeuralNetworks/05-Frameworks/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/3-NeuralNetworks/05-Frameworks/IntroPyTorch.ipynb) / [Keras](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/3-NeuralNetworks/05-Frameworks/IntroKeras.ipynb) / [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/3-NeuralNetworks/05-Frameworks/IntroKerasTF.ipynb) | [ল্যাব](./lessons/3-NeuralNetworks/05-Frameworks/lab/README.md) |
|
||||
| IV | [**কম্পিউটার ভিশন**](./lessons/4-ComputerVision/README.md) | [PyTorch](https://docs.microsoft.com/learn/modules/intro-computer-vision-pytorch/?WT.mc_id=academic-77998-cacaste) / [TensorFlow](https://docs.microsoft.com/learn/modules/intro-computer-vision-TensorFlow/?WT.mc_id=academic-77998-cacaste)| [Microsoft Azure-এ কম্পিউটার ভিশন অন্বেষণ করুন](https://learn.microsoft.com/en-us/collections/7w28iy2xrqzdj0?WT.mc_id=academic-77998-bethanycheum) |
|
||||
| 06 | [কম্পিউটার ভিশনের পরিচিতি। OpenCV](./lessons/4-ComputerVision/06-IntroCV/README.md) | [নোটবুক](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/06-IntroCV/OpenCV.ipynb) | [ল্যাব](./lessons/4-ComputerVision/06-IntroCV/lab/README.md) |
|
||||
| 07 | [কনভলিউশনাল নিউরাল নেটওয়ার্ক](./lessons/4-ComputerVision/07-ConvNets/README.md) & [CNN আর্কিটেকচার](./lessons/4-ComputerVision/07-ConvNets/CNN_Architectures.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/07-ConvNets/ConvNetsPyTorch.ipynb) /[TensorFlow](https://microsoft.github.io/AI-For-Beginners/lessons/4-ComputerVision/07-ConvNets/ConvNetsTF.ipynb) | [ল্যাব](./lessons/4-ComputerVision/07-ConvNets/lab/README.md) |
|
||||
| 08 | [প্রি-ট্রেইনড নেটওয়ার্ক এবং ট্রান্সফার লার্নিং](./lessons/4-ComputerVision/08-TransferLearning/README.md) এবং [ট্রেনিং ট্রিকস](./lessons/4-ComputerVision/08-TransferLearning/TrainingTricks.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/08-TransferLearning/TransferLearningPyTorch.ipynb) / [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/3-NeuralNetworks/05-Frameworks/IntroKerasTF.ipynb) | [ল্যাব](./lessons/4-ComputerVision/08-TransferLearning/lab/README.md) |
|
||||
| 09 | [অটোএনকোডার এবং VAE](./lessons/4-ComputerVision/09-Autoencoders/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/09-Autoencoders/AutoEncodersPyTorch.ipynb) / [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/09-Autoencoders/AutoencodersTF.ipynb) | |
|
||||
| 10 | [জেনারেটিভ অ্যাডভার্সারিয়াল নেটওয়ার্ক এবং আর্টিস্টিক স্টাইল ট্রান্সফার](./lessons/4-ComputerVision/10-GANs/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/10-GANs/GANPyTorch.ipynb) / [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/10-GANs/GANTF.ipynb) | |
|
||||
| 11 | [অবজেক্ট ডিটেকশন](./lessons/4-ComputerVision/11-ObjectDetection/README.md) | [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/11-ObjectDetection/ObjectDetection.ipynb) | [ল্যাব](./lessons/4-ComputerVision/11-ObjectDetection/lab/README.md) |
|
||||
| 12 | [সেমান্টিক সেগমেন্টেশন। ইউ-নেট](./lessons/4-ComputerVision/12-Segmentation/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/12-Segmentation/SemanticSegmentationPytorch.ipynb) / [TensorFlow](../../(https:/github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/12-Segmentation/SemanticSegmentationTF.ipynb)) | |
|
||||
| V | [**ন্যাচারাল ল্যাঙ্গুয়েজ প্রসেসিং**](./lessons/5-NLP/README.md) | [PyTorch](https://docs.microsoft.com/learn/modules/intro-natural-language-processing-pytorch/?WT.mc_id=academic-77998-cacaste) /[TensorFlow](https://docs.microsoft.com/learn/modules/intro-natural-language-processing-TensorFlow/?WT.mc_id=academic-77998-cacaste) | [মাইক্রোসফট অ্যাজুরে ন্যাচারাল ল্যাঙ্গুয়েজ প্রসেসিং এক্সপ্লোর করুন](https://learn.microsoft.com/en-us/collections/7w28iy2xrqzdj0?WT.mc_id=academic-77998-bethanycheum)|
|
||||
| 13 | [টেক্সট রিপ্রেজেন্টেশন। Bow/TF-IDF](./lessons/5-NLP/13-TextRep/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/13-TextRep/TextRepresentationPyTorch.ipynb) / [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/13-TextRep/TextRepresentationTF.ipynb) | |
|
||||
| 14 | [সেমান্টিক ওয়ার্ড এমবেডিং। Word2Vec এবং GloVe](./lessons/5-NLP/14-Embeddings/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb) / [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/14-Embeddings/EmbeddingsTF.ipynb) | |
|
||||
| 15 | [ল্যাঙ্গুয়েজ মডেলিং। নিজের এমবেডিং ট্রেনিং করুন](./lessons/5-NLP/15-LanguageModeling/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/15-LanguageModeling/CBoW-PyTorch.ipynb) / [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/15-LanguageModeling/CBoW-TF.ipynb) | [ল্যাব](./lessons/5-NLP/15-LanguageModeling/lab/README.md) |
|
||||
| 16 | [রিকারেন্ট নিউরাল নেটওয়ার্ক](./lessons/5-NLP/16-RNN/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/16-RNN/RNNPyTorch.ipynb) / [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/16-RNN/RNNTF.ipynb) | |
|
||||
| 17 | [জেনারেটিভ রিকারেন্ট নেটওয়ার্ক](./lessons/5-NLP/17-GenerativeNetworks/README.md) | [PyTorch](https://microsoft.github.io/AI-For-Beginners/lessons/5-NLP/17-GenerativeNetworks/GenerativePyTorch.md) / [TensorFlow](https://microsoft.github.io/AI-For-Beginners/lessons/5-NLP/17-GenerativeNetworks/GenerativeTF.md) | [ল্যাব](./lessons/5-NLP/17-GenerativeNetworks/lab/README.md) |
|
||||
| 18 | [ট্রান্সফর্মার। BERT।](./lessons/5-NLP/18-Transformers/READMEtransformers.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/18-Transformers/TransformersPyTorch.ipynb) /[TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/18-Transformers/TransformersTF.ipynb) | |
|
||||
| 19 | [নেমড এন্টিটি রিকগনিশন](./lessons/5-NLP/19-NER/README.md) | [TensorFlow](https://microsoft.github.io/AI-For-Beginners/lessons/5-NLP/19-NER/NER-TF.ipynb) | [ল্যাব](./lessons/5-NLP/19-NER/lab/README.md) |
|
||||
| 20 | [বড় ল্যাঙ্গুয়েজ মডেল, প্রম্পট প্রোগ্রামিং এবং ফিউ-শট টাস্ক](./lessons/5-NLP/20-LangModels/READMELargeLang.md) | [PyTorch](https://microsoft.github.io/AI-For-Beginners/lessons/5-NLP/20-LangModels/GPT-PyTorch.ipynb) | |
|
||||
| | পাঠের লিঙ্ক | PyTorch/Keras/TensorFlow | ল্যাব |
|
||||
| :-: | :------------------------------------------------------------------------------------------------------------------------------------------: | :---------------------------------------------------------------------------------------------: | ------------------------------------------------------------------------------ |
|
||||
| 0 | [কোর্স সেটআপ](./lessons/0-course-setup/setup.md) | [আপনার ডেভেলপমেন্ট এনভায়রনমেন্ট সেটআপ করুন](./lessons/0-course-setup/how-to-run.md) | |
|
||||
| I | [**AI-এর পরিচিতি**](./lessons/1-Intro/README.md) | | |
|
||||
| 01 | [AI-এর পরিচিতি এবং ইতিহাস](./lessons/1-Intro/README.md) | - | - |
|
||||
| II | **প্রতীকী AI** |
|
||||
| 02 | [জ্ঞান উপস্থাপনা এবং বিশেষজ্ঞ সিস্টেম](./lessons/2-Symbolic/README.md) | [বিশেষজ্ঞ সিস্টেম](./lessons/2-Symbolic/Animals.ipynb) / [অন্টোলজি](./lessons/2-Symbolic/FamilyOntology.ipynb) /[ধারণা গ্রাফ](./lessons/2-Symbolic/MSConceptGraph.ipynb) | |
|
||||
| III | [**নিউরাল নেটওয়ার্কের পরিচিতি**](./lessons/3-NeuralNetworks/README.md) |||
|
||||
| 03 | [পারসেপট্রন](./lessons/3-NeuralNetworks/03-Perceptron/README.md) | [নোটবুক](./lessons/3-NeuralNetworks/03-Perceptron/Perceptron.ipynb) | [ল্যাব](./lessons/3-NeuralNetworks/03-Perceptron/lab/README.md) |
|
||||
| 04 | [মাল্টি-লেয়ার পারসেপট্রন এবং আমাদের নিজস্ব ফ্রেমওয়ার্ক তৈরি করা](./lessons/3-NeuralNetworks/04-OwnFramework/README.md) | [নোটবুক](./lessons/3-NeuralNetworks/04-OwnFramework/OwnFramework.ipynb) | [ল্যাব](./lessons/3-NeuralNetworks/04-OwnFramework/lab/README.md) |
|
||||
| 05 | [ফ্রেমওয়ার্কের পরিচিতি (PyTorch/TensorFlow) এবং ওভারফিটিং](./lessons/3-NeuralNetworks/05-Frameworks/README.md) | [PyTorch](./lessons/3-NeuralNetworks/05-Frameworks/IntroPyTorch.ipynb) / [Keras](./lessons/3-NeuralNetworks/05-Frameworks/IntroKeras.ipynb) / [TensorFlow](./lessons/3-NeuralNetworks/05-Frameworks/IntroKerasTF.ipynb) | [ল্যাব](./lessons/3-NeuralNetworks/05-Frameworks/lab/README.md) |
|
||||
| IV | [**কম্পিউটার ভিশন**](./lessons/4-ComputerVision/README.md) | [PyTorch](https://docs.microsoft.com/learn/modules/intro-computer-vision-pytorch/?WT.mc_id=academic-77998-cacaste) / [TensorFlow](https://docs.microsoft.com/learn/modules/intro-computer-vision-TensorFlow/?WT.mc_id=academic-77998-cacaste)| [Microsoft Azure-এ কম্পিউটার ভিশন অন্বেষণ করুন](https://learn.microsoft.com/en-us/collections/7w28iy2xrqzdj0?WT.mc_id=academic-77998-bethanycheum) |
|
||||
| 06 | [কম্পিউটার ভিশনের পরিচিতি। OpenCV](./lessons/4-ComputerVision/06-IntroCV/README.md) | [নোটবুক](./lessons/4-ComputerVision/06-IntroCV/OpenCV.ipynb) | [ল্যাব](./lessons/4-ComputerVision/06-IntroCV/lab/README.md) |
|
||||
| 07 | [কনভলিউশনাল নিউরাল নেটওয়ার্ক](./lessons/4-ComputerVision/07-ConvNets/README.md) & [CNN আর্কিটেকচার](./lessons/4-ComputerVision/07-ConvNets/CNN_Architectures.md) | [PyTorch](./lessons/4-ComputerVision/07-ConvNets/ConvNetsPyTorch.ipynb) /[TensorFlow](./lessons/4-ComputerVision/07-ConvNets/ConvNetsTF.ipynb) | [ল্যাব](./lessons/4-ComputerVision/07-ConvNets/lab/README.md) |
|
||||
| 08 | [প্রি-ট্রেইনড নেটওয়ার্ক এবং ট্রান্সফার লার্নিং](./lessons/4-ComputerVision/08-TransferLearning/README.md) এবং [ট্রেনিং ট্রিকস](./lessons/4-ComputerVision/08-TransferLearning/TrainingTricks.md) | [PyTorch](./lessons/4-ComputerVision/08-TransferLearning/TransferLearningPyTorch.ipynb) / [TensorFlow](./lessons/3-NeuralNetworks/05-Frameworks/IntroKerasTF.ipynb) | [ল্যাব](./lessons/4-ComputerVision/08-TransferLearning/lab/README.md) |
|
||||
| 09 | [অটোএনকোডার এবং VAEs](./lessons/4-ComputerVision/09-Autoencoders/README.md) | [PyTorch](./lessons/4-ComputerVision/09-Autoencoders/AutoEncodersPyTorch.ipynb) / [TensorFlow](./lessons/4-ComputerVision/09-Autoencoders/AutoencodersTF.ipynb) | |
|
||||
| 10 | [জেনারেটিভ অ্যাডভার্সারিয়াল নেটওয়ার্ক এবং আর্টিস্টিক স্টাইল ট্রান্সফার](./lessons/4-ComputerVision/10-GANs/README.md) | [PyTorch](./lessons/4-ComputerVision/10-GANs/GANPyTorch.ipynb) / [TensorFlow](./lessons/4-ComputerVision/10-GANs/GANTF.ipynb) | |
|
||||
| 11 | [অবজেক্ট ডিটেকশন](./lessons/4-ComputerVision/11-ObjectDetection/README.md) | [TensorFlow](./lessons/4-ComputerVision/11-ObjectDetection/ObjectDetection.ipynb) | [ল্যাব](./lessons/4-ComputerVision/11-ObjectDetection/lab/README.md) |
|
||||
| 12 | [সেমান্টিক সেগমেন্টেশন। U-Net](./lessons/4-ComputerVision/12-Segmentation/README.md) | [PyTorch](./lessons/4-ComputerVision/12-Segmentation/SemanticSegmentationPytorch.ipynb) / [TensorFlow](./lessons/4-ComputerVision/12-Segmentation/SemanticSegmentationTF.ipynb) | |
|
||||
| V | [**ন্যাচারাল ল্যাঙ্গুয়েজ প্রসেসিং**](./lessons/5-NLP/README.md) | [PyTorch](https://docs.microsoft.com/learn/modules/intro-natural-language-processing-pytorch/?WT.mc_id=academic-77998-cacaste) /[TensorFlow](https://docs.microsoft.com/learn/modules/intro-natural-language-processing-TensorFlow/?WT.mc_id=academic-77998-cacaste) | [Microsoft Azure-এ ন্যাচারাল ল্যাঙ্গুয়েজ প্রসেসিং অন্বেষণ করুন](https://learn.microsoft.com/en-us/collections/7w28iy2xrqzdj0?WT.mc_id=academic-77998-bethanycheum)|
|
||||
| 13 | [টেক্সট রিপ্রেজেন্টেশন। Bow/TF-IDF](./lessons/5-NLP/13-TextRep/README.md) | [PyTorch](./lessons/5-NLP/13-TextRep/TextRepresentationPyTorch.ipynb) / [TensorFlow](./lessons/5-NLP/13-TextRep/TextRepresentationTF.ipynb) | |
|
||||
| 14 | [সেমান্টিক ওয়ার্ড এমবেডিং। Word2Vec এবং GloVe](./lessons/5-NLP/14-Embeddings/README.md) | [PyTorch](./lessons/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb) / [TensorFlow](./lessons/5-NLP/14-Embeddings/EmbeddingsTF.ipynb) | |
|
||||
| 15 | [ল্যাঙ্গুয়েজ মডেলিং। নিজের এমবেডিং ট্রেনিং করুন](./lessons/5-NLP/15-LanguageModeling/README.md) | [PyTorch](./lessons/5-NLP/15-LanguageModeling/CBoW-PyTorch.ipynb) / [TensorFlow](./lessons/5-NLP/15-LanguageModeling/CBoW-TF.ipynb) | [ল্যাব](./lessons/5-NLP/15-LanguageModeling/lab/README.md) |
|
||||
| 16 | [রিকারেন্ট নিউরাল নেটওয়ার্ক](./lessons/5-NLP/16-RNN/README.md) | [PyTorch](./lessons/5-NLP/16-RNN/RNNPyTorch.ipynb) / [TensorFlow](./lessons/5-NLP/16-RNN/RNNTF.ipynb) | |
|
||||
| 17 | [জেনারেটিভ রিকারেন্ট নেটওয়ার্ক](./lessons/5-NLP/17-GenerativeNetworks/README.md) | [PyTorch](./lessons/5-NLP/17-GenerativeNetworks/GenerativePyTorch.md) / [TensorFlow](./lessons/5-NLP/17-GenerativeNetworks/GenerativeTF.md) | [ল্যাব](./lessons/5-NLP/17-GenerativeNetworks/lab/README.md) |
|
||||
| 18 | [ট্রান্সফর্মার। BERT.](./lessons/5-NLP/18-Transformers/READMEtransformers.md) | [PyTorch](./lessons/5-NLP/18-Transformers/TransformersPyTorch.ipynb) /[TensorFlow](./lessons/5-NLP/18-Transformers/TransformersTF.ipynb) | |
|
||||
| 19 | [নেমড এন্টিটি রিকগনিশন](./lessons/5-NLP/19-NER/README.md) | [TensorFlow](./lessons/5-NLP/19-NER/NER-TF.ipynb) | [ল্যাব](./lessons/5-NLP/19-NER/lab/README.md) |
|
||||
| 20 | [বড় ল্যাঙ্গুয়েজ মডেল, প্রম্পট প্রোগ্রামিং এবং ফিউ-শট টাস্ক](./lessons/5-NLP/20-LangModels/READMELargeLang.md) | [PyTorch](./lessons/5-NLP/20-LangModels/GPT-PyTorch.ipynb) | |
|
||||
| VI | **অন্যান্য AI কৌশল** || |
|
||||
| 21 | [জেনেটিক অ্যালগরিদম](./lessons/6-Other/21-GeneticAlgorithms/README.md) | [নোটবুক](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/6-Other/21-GeneticAlgorithms/Genetic.ipynb) | |
|
||||
| 22 | [ডিপ রিইনফোর্সমেন্ট লার্নিং](./lessons/6-Other/22-DeepRL/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/6-Other/22-DeepRL/CartPole-RL-PyTorch.ipynb) /[TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/6-Other/22-DeepRL/CartPole-RL-TF.ipynb) | [ল্যাব](./lessons/6-Other/22-DeepRL/lab/README.md) |
|
||||
| 21 | [জেনেটিক অ্যালগরিদম](./lessons/6-Other/21-GeneticAlgorithms/README.md) | [নোটবুক](./lessons/6-Other/21-GeneticAlgorithms/Genetic.ipynb) | |
|
||||
| 22 | [ডিপ রিইনফোর্সমেন্ট লার্নিং](./lessons/6-Other/22-DeepRL/README.md) | [PyTorch](./lessons/6-Other/22-DeepRL/CartPole-RL-PyTorch.ipynb) /[TensorFlow](./lessons/6-Other/22-DeepRL/CartPole-RL-TF.ipynb) | [ল্যাব](./lessons/6-Other/22-DeepRL/lab/README.md) |
|
||||
| 23 | [মাল্টি-এজেন্ট সিস্টেম](./lessons/6-Other/23-MultiagentSystems/README.md) | | |
|
||||
| VII | **AI নৈতিকতা** | | |
|
||||
| 24 | [AI নৈতিকতা এবং দায়িত্বশীল AI](./lessons/7-Ethics/README.md) | [Microsoft Learn: Responsible AI Principles](https://docs.microsoft.com/learn/paths/responsible-ai-business-principles/?WT.mc_id=academic-77998-cacaste) | |
|
||||
| 24 | [AI নৈতিকতা এবং দায়িত্বশীল AI](./lessons/7-Ethics/README.md) | [Microsoft Learn: দায়িত্বশীল AI নীতিমালা](https://docs.microsoft.com/learn/paths/responsible-ai-business-principles/?WT.mc_id=academic-77998-cacaste) | |
|
||||
| IX | **অতিরিক্ত বিষয়** | | |
|
||||
| 25 | [মাল্টি-মডাল নেটওয়ার্ক, CLIP এবং VQGAN](./lessons/X-Extras/X1-MultiModal/README.md) | [নোটবুক](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/X-Extras/X1-MultiModal/Clip.ipynb) | |
|
||||
| 25 | [মাল্টি-মডাল নেটওয়ার্ক, CLIP এবং VQGAN](./lessons/X-Extras/X1-MultiModal/README.md) | [নোটবুক](./lessons/X-Extras/X1-MultiModal/Clip.ipynb) | |
|
||||
|
||||
## প্রতিটি পাঠে অন্তর্ভুক্ত রয়েছে
|
||||
## প্রতিটি পাঠে অন্তর্ভুক্ত
|
||||
|
||||
* প্রি-রিডিং উপকরণ
|
||||
* এক্সিকিউটেবল জুপিটার নোটবুক, যা প্রায়ই নির্দিষ্ট ফ্রেমওয়ার্ক (**PyTorch** বা **TensorFlow**) এর জন্য তৈরি। এক্সিকিউটেবল নোটবুকে প্রচুর তাত্ত্বিক উপকরণও রয়েছে, তাই বিষয়টি বুঝতে হলে আপনাকে অন্তত একটি সংস্করণ (PyTorch বা TensorFlow) পড়তে হবে।
|
||||
* **ল্যাব** কিছু বিষয়ে উপলব্ধ, যা আপনাকে শিখা উপকরণ একটি নির্দিষ্ট সমস্যায় প্রয়োগ করার সুযোগ দেয়।
|
||||
* এক্সিকিউটেবল Jupyter নোটবুক, যা প্রায়ই ফ্রেমওয়ার্ক (**PyTorch** বা **TensorFlow**) অনুযায়ী নির্দিষ্ট। এক্সিকিউটেবল নোটবুকে প্রচুর তাত্ত্বিক উপকরণও থাকে, তাই বিষয়টি বুঝতে হলে আপনাকে অন্তত একটি সংস্করণ (PyTorch বা TensorFlow) পড়তে হবে।
|
||||
* **ল্যাব** কিছু বিষয়ের জন্য উপলব্ধ, যা আপনাকে শিখানো উপকরণ একটি নির্দিষ্ট সমস্যায় প্রয়োগ করার সুযোগ দেয়।
|
||||
* কিছু বিভাগে [**MS Learn**](https://learn.microsoft.com/en-us/collections/7w28iy2xrqzdj0?WT.mc_id=academic-77998-bethanycheum) মডিউলের লিঙ্ক রয়েছে যা সম্পর্কিত বিষয়গুলি কভার করে।
|
||||
|
||||
## শুরু করার জন্য
|
||||
|
||||
- আমরা একটি [সেটআপ পাঠ](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/0-course-setup/setup.md) তৈরি করেছি যা আপনার ডেভেলপমেন্ট পরিবেশ সেটআপে সাহায্য করবে। - শিক্ষকদের জন্য, আমরা একটি [কারিকুলা সেটআপ পাঠ](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/0-course-setup/for-teachers.md) তৈরি করেছি!
|
||||
- কীভাবে [VSCode বা Codepace-এ কোড চালাবেন](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/0-course-setup/how-to-run.md)
|
||||
- আমরা একটি [সেটআপ পাঠ](./lessons/0-course-setup/setup.md) তৈরি করেছি যা আপনার ডেভেলপমেন্ট পরিবেশ সেটআপে সাহায্য করবে। - শিক্ষকদের জন্য, আমরা একটি [কারিকুলা সেটআপ পাঠ](./lessons/0-course-setup/for-teachers.md) তৈরি করেছি!
|
||||
- কীভাবে [VSCode বা Codepace-এ কোড চালাবেন](./lessons/0-course-setup/how-to-run.md)
|
||||
|
||||
এই ধাপগুলি অনুসরণ করুন:
|
||||
|
||||
রিপোজিটরি ফর্ক করুন: এই পৃষ্ঠার উপরের ডানদিকে "Fork" বোতামে ক্লিক করুন।
|
||||
রিপোজিটরি ফর্ক করুন: এই পৃষ্ঠার উপরের ডান কোণে "Fork" বোতামে ক্লিক করুন।
|
||||
|
||||
রিপোজিটরি ক্লোন করুন: `git clone https://github.com/microsoft/AI-For-Beginners.git`
|
||||
|
||||
এই রিপোটি সহজে খুঁজে পেতে পরে স্টার (🌟) দিতে ভুলবেন না।
|
||||
এই রিপোজিটরিটিকে সহজে খুঁজে পেতে 🌟 দিয়ে স্টার দিতে ভুলবেন না।
|
||||
|
||||
## অন্যান্য শিক্ষার্থীদের সাথে পরিচিত হন
|
||||
|
||||
আমাদের [অফিশিয়াল AI Discord সার্ভার](https://aka.ms/genai-discord?WT.mc_id=academic-105485-bethanycheum) এ যোগ দিন, যেখানে আপনি এই কোর্সে অংশগ্রহণকারী অন্যান্য শিক্ষার্থীদের সাথে পরিচিত হতে এবং নেটওয়ার্ক তৈরি করতে পারেন এবং সহায়তা পেতে পারেন।
|
||||
আমাদের [অফিশিয়াল AI Discord সার্ভার](https://aka.ms/genai-discord?WT.mc_id=academic-105485-bethanycheum)-এ যোগ দিন, যেখানে আপনি এই কোর্সে অংশগ্রহণকারী অন্যান্য শিক্ষার্থীদের সাথে পরিচিত হতে এবং নেটওয়ার্ক তৈরি করতে পারেন এবং সহায়তা পেতে পারেন।
|
||||
|
||||
যদি আপনার পণ্য সম্পর্কিত মতামত বা প্রশ্ন থাকে, আমাদের [Azure AI Foundry Developer Forum](https://aka.ms/foundry/forum) এ যান।
|
||||
যদি আপনার পণ্য সম্পর্কিত মতামত বা প্রশ্ন থাকে, আমাদের [Azure AI Foundry Developer Forum](https://aka.ms/foundry/forum)-এ যান।
|
||||
|
||||
## কুইজ
|
||||
> **কুইজ সম্পর্কে একটি নোট**: সমস্ত কুইজ `Quiz-app` ফোল্ডারে সংরক্ষিত, যা `etc\quiz-app`-এ অবস্থিত। এগুলো পাঠের মধ্যে থেকে লিঙ্ক করা হয়েছে। কুইজ অ্যাপটি স্থানীয়ভাবে চালানো যেতে পারে বা Azure-এ ডিপ্লয় করা যেতে পারে; `quiz-app` ফোল্ডারের নির্দেশাবলী অনুসরণ করুন। এগুলো ধীরে ধীরে স্থানীয় ভাষায় রূপান্তরিত করা হচ্ছে।
|
||||
> **কুইজ সম্পর্কে একটি নোট**: সমস্ত কুইজ `etc\quiz-app` ফোল্ডারের Quiz-app ফোল্ডারে সংরক্ষিত আছে, অথবা [অনলাইনে এখানে](https://ff-quizzes.netlify.app/) পাওয়া যাবে। এগুলো পাঠের মধ্যে থেকে লিঙ্ক করা হয়েছে। কুইজ অ্যাপটি স্থানীয়ভাবে চালানো বা Azure-এ ডিপ্লয় করা যেতে পারে; `quiz-app` ফোল্ডারের নির্দেশনা অনুসরণ করুন। এগুলো ধীরে ধীরে স্থানীয় ভাষায় রূপান্তরিত হচ্ছে।
|
||||
## সাহায্যের প্রয়োজন
|
||||
|
||||
আপনার কি কোনো পরামর্শ আছে বা বানান বা কোডের ভুল খুঁজে পেয়েছেন? একটি ইস্যু উত্থাপন করুন বা একটি পুল রিকোয়েস্ট তৈরি করুন।
|
||||
|
|
@ -132,9 +143,9 @@ _ক্লাউডে AI_ বিষয়গুলির একটি সহজ
|
|||
* **✅ কুইজ নির্মাতা:** [Lateefah Bello](https://github.com/CinnamonXI), [MLSA](https://studentambassadors.microsoft.com/)
|
||||
* **🙏 মূল অবদানকারী:** [Evgenii Pishchik](https://github.com/Pe4enIks)
|
||||
|
||||
## অন্যান্য কারিকুলাম
|
||||
## অন্যান্য পাঠক্রম
|
||||
|
||||
আমাদের টিম অন্যান্য কারিকুলামও তৈরি করে! দেখে নিন:
|
||||
আমাদের দল অন্যান্য পাঠক্রমও তৈরি করে! দেখে নিন:
|
||||
|
||||
- [Generative AI for Beginners](https://aka.ms/genai-beginners)
|
||||
- [Generative AI for Beginners .NET](https://github.com/microsoft/Generative-AI-for-beginners-dotnet)
|
||||
|
|
@ -151,5 +162,7 @@ _ক্লাউডে AI_ বিষয়গুলির একটি সহজ
|
|||
- [Mastering GitHub Copilot for C#/.NET Developers](https://github.com/microsoft/mastering-github-copilot-for-dotnet-csharp-developers)
|
||||
- [Choose Your Own Copilot Adventure](https://github.com/microsoft/CopilotAdventures)
|
||||
|
||||
---
|
||||
|
||||
**অস্বীকৃতি**:
|
||||
এই নথিটি AI অনুবাদ পরিষেবা [Co-op Translator](https://github.com/Azure/co-op-translator) ব্যবহার করে অনুবাদ করা হয়েছে। আমরা যথাসাধ্য সঠিকতা নিশ্চিত করার চেষ্টা করি, তবে অনুগ্রহ করে মনে রাখবেন যে স্বয়ংক্রিয় অনুবাদে ত্রুটি বা অসঙ্গতি থাকতে পারে। এর মূল ভাষায় থাকা নথিটিকে প্রামাণিক উৎস হিসেবে বিবেচনা করা উচিত। গুরুত্বপূর্ণ তথ্যের জন্য, পেশাদার মানব অনুবাদ সুপারিশ করা হয়। এই অনুবাদ ব্যবহারের ফলে কোনো ভুল বোঝাবুঝি বা ভুল ব্যাখ্যা হলে আমরা দায়বদ্ধ থাকব না।
|
||||
এই নথিটি AI অনুবাদ পরিষেবা [Co-op Translator](https://github.com/Azure/co-op-translator) ব্যবহার করে অনুবাদ করা হয়েছে। আমরা যথাসম্ভব সঠিক অনুবাদের চেষ্টা করি, তবে অনুগ্রহ করে মনে রাখবেন যে স্বয়ংক্রিয় অনুবাদে ত্রুটি বা অসঙ্গতি থাকতে পারে। নথিটির মূল ভাষায় লেখা সংস্করণটিকেই প্রামাণিক উৎস হিসেবে বিবেচনা করা উচিত। গুরুত্বপূর্ণ তথ্যের জন্য পেশাদার মানব অনুবাদ ব্যবহার করার পরামর্শ দেওয়া হচ্ছে। এই অনুবাদ ব্যবহারের ফলে সৃষ্ট কোনো ভুল বোঝাবুঝি বা ভুল ব্যাখ্যার জন্য আমরা দায়ী নই।
|
||||
|
|
@ -0,0 +1,478 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"collapsed": true
|
||||
},
|
||||
"source": [
|
||||
"# প্রাণী বিশেষজ্ঞ সিস্টেম বাস্তবায়ন\n",
|
||||
"\n",
|
||||
"[AI for Beginners Curriculum](http://github.com/microsoft/ai-for-beginners) থেকে একটি উদাহরণ।\n",
|
||||
"\n",
|
||||
"এই উদাহরণে, আমরা একটি সহজ জ্ঞান-ভিত্তিক সিস্টেম বাস্তবায়ন করব যা কিছু শারীরিক বৈশিষ্ট্যের উপর ভিত্তি করে একটি প্রাণী নির্ধারণ করবে। সিস্টেমটি নিম্নলিখিত AND-OR গাছ দ্বারা উপস্থাপন করা যেতে পারে (এটি পুরো গাছের একটি অংশ, আমরা সহজেই আরও কিছু নিয়ম যোগ করতে পারি):\n",
|
||||
"\n",
|
||||
"\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## আমাদের নিজস্ব এক্সপার্ট সিস্টেম শেল ব্যাকওয়ার্ড ইনফারেন্স সহ\n",
|
||||
"\n",
|
||||
"চলুন প্রোডাকশন রুলের উপর ভিত্তি করে জ্ঞান উপস্থাপনার জন্য একটি সহজ ভাষা সংজ্ঞায়িত করার চেষ্টা করি। আমরা রুল সংজ্ঞায়িত করার জন্য Python ক্লাসগুলোকে কীওয়ার্ড হিসেবে ব্যবহার করব। মূলত তিন ধরনের ক্লাস থাকবে:\n",
|
||||
"* `Ask` একটি প্রশ্নকে উপস্থাপন করে যা ব্যবহারকারীকে জিজ্ঞাসা করতে হবে। এটি সম্ভাব্য উত্তরগুলোর সেট ধারণ করে।\n",
|
||||
"* `If` একটি নিয়মকে উপস্থাপন করে, এবং এটি শুধুমাত্র নিয়মের বিষয়বস্তু সংরক্ষণের জন্য একটি সিনট্যাকটিক সুগার।\n",
|
||||
"* `AND`/`OR` হল ক্লাসগুলো যা গাছের AND/OR শাখাগুলোকে উপস্থাপন করে। এগুলো শুধু ভিতরে আর্গুমেন্টগুলোর তালিকা সংরক্ষণ করে। কোড সহজ করার জন্য, সমস্ত কার্যকারিতা প্যারেন্ট ক্লাস `Content`-এ সংজ্ঞায়িত করা হয়েছে।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class Ask():\n",
|
||||
" def __init__(self,choices=['y','n']):\n",
|
||||
" self.choices = choices\n",
|
||||
" def ask(self):\n",
|
||||
" if max([len(x) for x in self.choices])>1:\n",
|
||||
" for i,x in enumerate(self.choices):\n",
|
||||
" print(\"{0}. {1}\".format(i,x),flush=True)\n",
|
||||
" x = int(input())\n",
|
||||
" return self.choices[x]\n",
|
||||
" else:\n",
|
||||
" print(\"/\".join(self.choices),flush=True)\n",
|
||||
" return input()\n",
|
||||
"\n",
|
||||
"class Content():\n",
|
||||
" def __init__(self,x):\n",
|
||||
" self.x=x\n",
|
||||
" \n",
|
||||
"class If(Content):\n",
|
||||
" pass\n",
|
||||
"\n",
|
||||
"class AND(Content):\n",
|
||||
" pass\n",
|
||||
"\n",
|
||||
"class OR(Content):\n",
|
||||
" pass"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"আমাদের সিস্টেমে, কর্মরত মেমোরি **attribute-value pairs** হিসাবে **তথ্যগুলির** তালিকা ধারণ করবে। জ্ঞানভান্ডারকে একটি বড় ডিকশনারি হিসাবে সংজ্ঞায়িত করা যেতে পারে যা ক্রিয়াগুলিকে (নতুন তথ্য যা কর্মরত মেমোরিতে যোগ করা উচিত) শর্তগুলির সাথে মানচিত্র করে, যা AND-OR অভিব্যক্তি হিসাবে প্রকাশিত হয়। এছাড়াও, কিছু তথ্য `Ask` করা যেতে পারে।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"rules = {\n",
|
||||
" 'default': Ask(['y','n']),\n",
|
||||
" 'color' : Ask(['red-brown','black and white','other']),\n",
|
||||
" 'pattern' : Ask(['dark stripes','dark spots']),\n",
|
||||
" 'mammal': If(OR(['hair','gives milk'])),\n",
|
||||
" 'carnivor': If(OR([AND(['sharp teeth','claws','forward-looking eyes']),'eats meat'])),\n",
|
||||
" 'ungulate': If(['mammal',OR(['has hooves','chews cud'])]),\n",
|
||||
" 'bird': If(OR(['feathers',AND(['flies','lies eggs'])])),\n",
|
||||
" 'animal:monkey' : If(['mammal','carnivor','color:red-brown','pattern:dark spots']),\n",
|
||||
" 'animal:tiger' : If(['mammal','carnivor','color:red-brown','pattern:dark stripes']),\n",
|
||||
" 'animal:giraffe' : If(['ungulate','long neck','long legs','pattern:dark spots']),\n",
|
||||
" 'animal:zebra' : If(['ungulate','pattern:dark stripes']),\n",
|
||||
" 'animal:ostrich' : If(['bird','long nech','color:black and white','cannot fly']),\n",
|
||||
" 'animal:pinguin' : If(['bird','swims','color:black and white','cannot fly']),\n",
|
||||
" 'animal:albatross' : If(['bird','flies well'])\n",
|
||||
"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"পিছনের দিকের অনুমান সম্পাদনের জন্য, আমরা `Knowledgebase` ক্লাসটি সংজ্ঞায়িত করব। এটি অন্তর্ভুক্ত করবে:\n",
|
||||
"* কার্যকরী `memory` - একটি ডিকশনারি যা অ্যাট্রিবিউটগুলিকে মানগুলোর সাথে ম্যাপ করে\n",
|
||||
"* Knowledgebase-এর `rules` - উপরে সংজ্ঞায়িত ফরম্যাটে\n",
|
||||
"\n",
|
||||
"দুটি প্রধান পদ্ধতি হল:\n",
|
||||
"* `get` - একটি অ্যাট্রিবিউটের মান প্রাপ্ত করার জন্য, প্রয়োজন হলে অনুমান সম্পাদন করে। উদাহরণস্বরূপ, `get('color')` একটি রঙের স্লটের মান পাবে (প্রয়োজন হলে জিজ্ঞাসা করবে এবং পরে ব্যবহারের জন্য কার্যকরী মেমোরিতে মানটি সংরক্ষণ করবে)। যদি আমরা `get('color:blue')` জিজ্ঞাসা করি, এটি একটি রঙ জিজ্ঞাসা করবে এবং তারপর রঙের উপর নির্ভর করে `y`/`n` মানটি ফেরত দেবে।\n",
|
||||
"* `eval` - আসল অনুমান সম্পাদন করে, অর্থাৎ AND/OR গাছটি অনুসন্ধান করে, সাব-গোলগুলো মূল্যায়ন করে ইত্যাদি।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class KnowledgeBase():\n",
|
||||
" def __init__(self,rules):\n",
|
||||
" self.rules = rules\n",
|
||||
" self.memory = {}\n",
|
||||
" \n",
|
||||
" def get(self,name):\n",
|
||||
" if ':' in name:\n",
|
||||
" k,v = name.split(':')\n",
|
||||
" vv = self.get(k)\n",
|
||||
" return 'y' if v==vv else 'n'\n",
|
||||
" if name in self.memory.keys():\n",
|
||||
" return self.memory[name]\n",
|
||||
" for fld in self.rules.keys():\n",
|
||||
" if fld==name or fld.startswith(name+\":\"):\n",
|
||||
" # print(\" + proving {}\".format(fld))\n",
|
||||
" value = 'y' if fld==name else fld.split(':')[1]\n",
|
||||
" res = self.eval(self.rules[fld],field=name)\n",
|
||||
" if res!='y' and res!='n' and value=='y':\n",
|
||||
" self.memory[name] = res\n",
|
||||
" return res\n",
|
||||
" if res=='y':\n",
|
||||
" self.memory[name] = value\n",
|
||||
" return value\n",
|
||||
" # field is not found, using default\n",
|
||||
" res = self.eval(self.rules['default'],field=name)\n",
|
||||
" self.memory[name]=res\n",
|
||||
" return res\n",
|
||||
" \n",
|
||||
" def eval(self,expr,field=None):\n",
|
||||
" # print(\" + eval {}\".format(expr))\n",
|
||||
" if isinstance(expr,Ask):\n",
|
||||
" print(field)\n",
|
||||
" return expr.ask()\n",
|
||||
" elif isinstance(expr,If):\n",
|
||||
" return self.eval(expr.x)\n",
|
||||
" elif isinstance(expr,AND) or isinstance(expr,list):\n",
|
||||
" expr = expr.x if isinstance(expr,AND) else expr\n",
|
||||
" for x in expr:\n",
|
||||
" if self.eval(x)=='n':\n",
|
||||
" return 'n'\n",
|
||||
" return 'y'\n",
|
||||
" elif isinstance(expr,OR):\n",
|
||||
" for x in expr.x:\n",
|
||||
" if self.eval(x)=='y':\n",
|
||||
" return 'y'\n",
|
||||
" return 'n'\n",
|
||||
" elif isinstance(expr,str):\n",
|
||||
" return self.get(expr)\n",
|
||||
" else:\n",
|
||||
" print(\"Unknown expr: {}\".format(expr))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"এখন চলুন আমাদের প্রাণী জ্ঞানভাণ্ডার সংজ্ঞায়িত করি এবং পরামর্শ কার্যক্রম সম্পাদন করি। লক্ষ্য করুন যে এই কলটি আপনাকে প্রশ্ন করবে। আপনি `y`/`n` টাইপ করে হ্যাঁ-না প্রশ্নের উত্তর দিতে পারেন, অথবা দীর্ঘ বহুবিকল্প প্রশ্নের জন্য সংখ্যা (0..N) নির্দিষ্ট করে উত্তর দিতে পারেন।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"hair\n",
|
||||
"y/n\n",
|
||||
"sharp teeth\n",
|
||||
"y/n\n",
|
||||
"claws\n",
|
||||
"y/n\n",
|
||||
"forward-looking eyes\n",
|
||||
"y/n\n",
|
||||
"color\n",
|
||||
"0. red-brown\n",
|
||||
"1. black and white\n",
|
||||
"2. other\n",
|
||||
"has hooves\n",
|
||||
"y/n\n",
|
||||
"long neck\n",
|
||||
"y/n\n",
|
||||
"long legs\n",
|
||||
"y/n\n",
|
||||
"pattern\n",
|
||||
"0. dark stripes\n",
|
||||
"1. dark spots\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"'giraffe'"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"kb = KnowledgeBase(rules)\n",
|
||||
"kb.get('animal')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## PyKnow ব্যবহার করে ফরোয়ার্ড ইনফারেন্স\n",
|
||||
"\n",
|
||||
"পরবর্তী উদাহরণে, আমরা জ্ঞান উপস্থাপনার জন্য একটি লাইব্রেরি, [PyKnow](https://github.com/buguroo/pyknow/) ব্যবহার করে ফরোয়ার্ড ইনফারেন্স বাস্তবায়নের চেষ্টা করব। **PyKnow** হলো একটি লাইব্রেরি যা পাইথনে ফরোয়ার্ড ইনফারেন্স সিস্টেম তৈরি করার জন্য ব্যবহৃত হয় এবং এটি ক্লাসিক্যাল পুরনো সিস্টেম [CLIPS](http://www.clipsrules.net/index.html)-এর মতো ডিজাইন করা হয়েছে।\n",
|
||||
"\n",
|
||||
"আমরা চাইলে নিজেরাই ফরোয়ার্ড চেইনিং বাস্তবায়ন করতে পারতাম, তেমন কোনো বড় সমস্যা ছাড়াই। তবে সাধারণ পদ্ধতিতে করা বাস্তবায়নগুলো সাধারণত খুব কার্যকর হয় না। আরও কার্যকরভাবে নিয়ম মেলানোর জন্য একটি বিশেষ অ্যালগরিদম [Rete](https://en.wikipedia.org/wiki/Rete_algorithm) ব্যবহার করা হয়।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Collecting git+https://github.com/buguroo/pyknow/\n",
|
||||
" Cloning https://github.com/buguroo/pyknow/ to /tmp/pip-req-build-3cqeulyl\n",
|
||||
" Running command git clone --filter=blob:none --quiet https://github.com/buguroo/pyknow/ /tmp/pip-req-build-3cqeulyl\n",
|
||||
" Resolved https://github.com/buguroo/pyknow/ to commit 48818336f2e9a126f1964f2d8dc22d37ff800fe8\n",
|
||||
" Preparing metadata (setup.py) ... \u001b[?25ldone\n",
|
||||
"\u001b[?25hCollecting frozendict==1.2\n",
|
||||
" Using cached frozendict-1.2.tar.gz (2.6 kB)\n",
|
||||
" Preparing metadata (setup.py) ... \u001b[?25ldone\n",
|
||||
"\u001b[?25hCollecting schema==0.6.7\n",
|
||||
" Using cached schema-0.6.7-py2.py3-none-any.whl (14 kB)\n",
|
||||
"Building wheels for collected packages: pyknow, frozendict\n",
|
||||
" Building wheel for pyknow (setup.py) ... \u001b[?25ldone\n",
|
||||
"\u001b[?25h Created wheel for pyknow: filename=pyknow-1.7.0-py3-none-any.whl size=34228 sha256=b7de5b09292c4007667c72f69b98d5a1b5f7324ff15f9dd8e077c3d5f7aade42\n",
|
||||
" Stored in directory: /tmp/pip-ephem-wheel-cache-k7jpave7/wheels/81/1a/d3/f6c15dbe1955598a37755215f2a10449e7418500d7bd4b9508\n",
|
||||
" Building wheel for frozendict (setup.py) ... \u001b[?25ldone\n",
|
||||
"\u001b[?25h Created wheel for frozendict: filename=frozendict-1.2-py3-none-any.whl size=3148 sha256=2863d55c240d2409cddf05ccfe600591f8478681549fc97555c47c90dc6bb160\n",
|
||||
" Stored in directory: /home/rg/.cache/pip/wheels/49/ac/f8/cb8120244e710bdb479c86198b03c7b08c3c2d3d2bf448fd6e\n",
|
||||
"Successfully built pyknow frozendict\n",
|
||||
"Installing collected packages: schema, frozendict, pyknow\n",
|
||||
"Successfully installed frozendict-1.2 pyknow-1.7.0 schema-0.6.7\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import sys\n",
|
||||
"!{sys.executable} -m pip install git+https://github.com/buguroo/pyknow/"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from pyknow import *\n",
|
||||
"#import pyknow"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"আমরা আমাদের সিস্টেমকে `KnowledgeEngine` এর সাবক্লাস হিসাবে একটি ক্লাস হিসেবে সংজ্ঞায়িত করব। প্রতিটি নিয়ম একটি পৃথক ফাংশন দ্বারা সংজ্ঞায়িত করা হয় যা `@Rule` অ্যানোটেশন ব্যবহার করে, যা নির্ধারণ করে কখন নিয়মটি কার্যকর হবে। নিয়মের ভিতরে, আমরা `declare` ফাংশন ব্যবহার করে নতুন তথ্য যোগ করতে পারি, এবং সেই তথ্য যোগ করার ফলে ফরওয়ার্ড ইনফারেন্স ইঞ্জিন দ্বারা আরও কিছু নিয়ম কার্যকর হবে।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class Animals(KnowledgeEngine):\n",
|
||||
" @Rule(OR(\n",
|
||||
" AND(Fact('sharp teeth'),Fact('claws'),Fact('forward looking eyes')),\n",
|
||||
" Fact('eats meat')))\n",
|
||||
" def cornivor(self):\n",
|
||||
" self.declare(Fact('carnivor'))\n",
|
||||
" \n",
|
||||
" @Rule(OR(Fact('hair'),Fact('gives milk')))\n",
|
||||
" def mammal(self):\n",
|
||||
" self.declare(Fact('mammal'))\n",
|
||||
"\n",
|
||||
" @Rule(Fact('mammal'),\n",
|
||||
" OR(Fact('has hooves'),Fact('chews cud')))\n",
|
||||
" def hooves(self):\n",
|
||||
" self.declare('ungulate')\n",
|
||||
" \n",
|
||||
" @Rule(OR(Fact('feathers'),AND(Fact('flies'),Fact('lays eggs'))))\n",
|
||||
" def bird(self):\n",
|
||||
" self.declare('bird')\n",
|
||||
" \n",
|
||||
" @Rule(Fact('mammal'),Fact('carnivor'),\n",
|
||||
" Fact(color='red-brown'),\n",
|
||||
" Fact(pattern='dark spots'))\n",
|
||||
" def monkey(self):\n",
|
||||
" self.declare(Fact(animal='monkey'))\n",
|
||||
"\n",
|
||||
" @Rule(Fact('mammal'),Fact('carnivor'),\n",
|
||||
" Fact(color='red-brown'),\n",
|
||||
" Fact(pattern='dark stripes'))\n",
|
||||
" def tiger(self):\n",
|
||||
" self.declare(Fact(animal='tiger'))\n",
|
||||
"\n",
|
||||
" @Rule(Fact('ungulate'),\n",
|
||||
" Fact('long neck'),\n",
|
||||
" Fact('long legs'),\n",
|
||||
" Fact(pattern='dark spots'))\n",
|
||||
" def giraffe(self):\n",
|
||||
" self.declare(Fact(animal='giraffe'))\n",
|
||||
"\n",
|
||||
" @Rule(Fact('ungulate'),\n",
|
||||
" Fact(pattern='dark stripes'))\n",
|
||||
" def zebra(self):\n",
|
||||
" self.declare(Fact(animal='zebra'))\n",
|
||||
"\n",
|
||||
" @Rule(Fact('bird'),\n",
|
||||
" Fact('long neck'),\n",
|
||||
" Fact('cannot fly'),\n",
|
||||
" Fact(color='black and white'))\n",
|
||||
" def straus(self):\n",
|
||||
" self.declare(Fact(animal='ostrich'))\n",
|
||||
"\n",
|
||||
" @Rule(Fact('bird'),\n",
|
||||
" Fact('swims'),\n",
|
||||
" Fact('cannot fly'),\n",
|
||||
" Fact(color='black and white'))\n",
|
||||
" def pinguin(self):\n",
|
||||
" self.declare(Fact(animal='pinguin'))\n",
|
||||
"\n",
|
||||
" @Rule(Fact('bird'),\n",
|
||||
" Fact('flies well'))\n",
|
||||
" def albatros(self):\n",
|
||||
" self.declare(Fact(animal='albatross'))\n",
|
||||
" \n",
|
||||
" @Rule(Fact(animal=MATCH.a))\n",
|
||||
" def print_result(self,a):\n",
|
||||
" print('Animal is {}'.format(a))\n",
|
||||
" \n",
|
||||
" def factz(self,l):\n",
|
||||
" for x in l:\n",
|
||||
" self.declare(x)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"একবার আমরা একটি জ্ঞানভাণ্ডার সংজ্ঞায়িত করলে, আমরা আমাদের কার্যকরী মেমোরি কিছু প্রাথমিক তথ্য দিয়ে পূরণ করি, এবং তারপর `run()` পদ্ধতি কল করি সিদ্ধান্ত গ্রহণের জন্য। এর ফলে আপনি দেখতে পাবেন যে নতুন সিদ্ধান্ত নেওয়া তথ্য কার্যকরী মেমোরিতে যোগ করা হয়েছে, যার মধ্যে প্রাণীর সম্পর্কে চূড়ান্ত তথ্যও রয়েছে (যদি আমরা সমস্ত প্রাথমিক তথ্য সঠিকভাবে সেট আপ করি)।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Animal is tiger\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"FactList([(0, InitialFact()),\n",
|
||||
" (1, Fact(color='red-brown')),\n",
|
||||
" (2, Fact(pattern='dark stripes')),\n",
|
||||
" (3, Fact('sharp teeth')),\n",
|
||||
" (4, Fact('claws')),\n",
|
||||
" (5, Fact('forward looking eyes')),\n",
|
||||
" (6, Fact('gives milk')),\n",
|
||||
" (7, Fact('mammal')),\n",
|
||||
" (8, Fact('carnivor')),\n",
|
||||
" (9, Fact(animal='tiger'))])"
|
||||
]
|
||||
},
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"ex1 = Animals()\n",
|
||||
"ex1.reset()\n",
|
||||
"ex1.factz([\n",
|
||||
" Fact(color='red-brown'),\n",
|
||||
" Fact(pattern='dark stripes'),\n",
|
||||
" Fact('sharp teeth'),\n",
|
||||
" Fact('claws'),\n",
|
||||
" Fact('forward looking eyes'),\n",
|
||||
" Fact('gives milk')])\n",
|
||||
"ex1.run()\n",
|
||||
"ex1.facts"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**অস্বীকৃতি**: \nএই নথিটি AI অনুবাদ পরিষেবা [Co-op Translator](https://github.com/Azure/co-op-translator) ব্যবহার করে অনুবাদ করা হয়েছে। আমরা যথাসম্ভব সঠিকতার জন্য চেষ্টা করি, তবে অনুগ্রহ করে মনে রাখবেন যে স্বয়ংক্রিয় অনুবাদে ত্রুটি বা অসঙ্গতি থাকতে পারে। মূল ভাষায় থাকা নথিটিকে প্রামাণিক উৎস হিসেবে বিবেচনা করা উচিত। গুরুত্বপূর্ণ তথ্যের জন্য, পেশাদার মানব অনুবাদ সুপারিশ করা হয়। এই অনুবাদ ব্যবহারের ফলে কোনো ভুল বোঝাবুঝি বা ভুল ব্যাখ্যা হলে আমরা দায়বদ্ধ থাকব না।\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.7.4 64-bit (conda)",
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5"
|
||||
}
|
||||
},
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.11.2"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "ab2bd97b0453415b89a469284609a8ce",
|
||||
"translation_date": "2025-08-28T11:30:08+00:00",
|
||||
"source_file": "lessons/2-Symbolic/Animals.ipynb",
|
||||
"language_code": "bn"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
|
|
@ -0,0 +1,595 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"collapsed": true
|
||||
},
|
||||
"source": [
|
||||
"# পরিবার সম্পর্কের অ্যান্টোলজি\n",
|
||||
"\n",
|
||||
"এই উদাহরণটি [AI for Beginners Curriculum](http://github.com/microsoft/ai-for-beginners)-এর একটি অংশ, এবং এটি [এই ব্লগ পোস্ট](https://habr.com/post/270857/) থেকে অনুপ্রাণিত।\n",
|
||||
"\n",
|
||||
"আমি সবসময় পরিবারের মধ্যে বিভিন্ন সম্পর্ক মনে রাখতে কঠিন মনে করি। এই উদাহরণে, আমরা একটি অ্যান্টোলজি নেব যা পরিবার সম্পর্ক সংজ্ঞায়িত করে, এবং প্রকৃত বংশতালিকা গাছ ব্যবহার করব, এবং দেখাব কীভাবে আমরা স্বয়ংক্রিয়ভাবে সমস্ত আত্মীয় খুঁজে বের করার জন্য অনুমান করতে পারি।\n",
|
||||
"\n",
|
||||
"### বংশতালিকা গাছ প্রাপ্ত করা\n",
|
||||
"\n",
|
||||
"উদাহরণ হিসেবে, আমরা [Romanov Tsar Family](https://en.wikipedia.org/wiki/House_of_Romanov)-এর বংশতালিকা গাছ নেব। পরিবার সম্পর্ক বর্ণনা করার জন্য সবচেয়ে সাধারণ ফরম্যাট হল [GEDCOM](https://en.wikipedia.org/wiki/GEDCOM)। আমরা GEDCOM ফরম্যাটে Romanov পরিবারের বংশতালিকা গাছ নেব:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"0 HEAD\n",
|
||||
"1 CHAR UTF8\n",
|
||||
"1 GEDC\n",
|
||||
"2 VERS 5.5\n",
|
||||
"0 @0@ INDI\n",
|
||||
"1 NAME Mihail Fedorovich /Romanov/\n",
|
||||
"1 SEX M\n",
|
||||
"1 BIRT\n",
|
||||
"2 DATE 1613\n",
|
||||
"1 DEAT \n",
|
||||
"2 DATE 1645\n",
|
||||
"1 FAMS @41@\n",
|
||||
"0 @1@ INDI\n",
|
||||
"1 NAME Evdokija Lukjanovna /Streshneva/\n",
|
||||
"1 SEX F\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"!head -15 data/tsars.ged"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"GEDCOM ফাইল ব্যবহার করতে, আমরা `python-gedcom` লাইব্রেরি ব্যবহার করতে পারি:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Collecting python-gedcom\n",
|
||||
" Downloading python_gedcom-1.0.0-py2.py3-none-any.whl (35 kB)\n",
|
||||
"Installing collected packages: python-gedcom\n",
|
||||
"Successfully installed python-gedcom-1.0.0\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import sys\n",
|
||||
"!{sys.executable} -m pip install python-gedcom"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"এই লাইব্রেরি ফাইল পার্সিংয়ের কিছু প্রযুক্তিগত সমস্যা দূর করে, তবে এটি এখনও আমাদের গাছের সমস্ত ব্যক্তি এবং পরিবারের প্রতি বেশ নিম্ন-স্তরের অ্যাক্সেস প্রদান করে। এখানে কিভাবে আমরা ফাইলটি পার্স করতে পারি এবং সমস্ত ব্যক্তির তালিকা প্রদর্শন করতে পারি:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from gedcom.parser import Parser\n",
|
||||
"from gedcom.element.individual import IndividualElement\n",
|
||||
"from gedcom.element.family import FamilyElement\n",
|
||||
"g = Parser()\n",
|
||||
"g.parse_file('data/tsars.ged')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {
|
||||
"scrolled": true,
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[('@0@', ('Mihail Fedorovich', 'Romanov')),\n",
|
||||
" ('@1@', ('Evdokija Lukjanovna', 'Streshneva')),\n",
|
||||
" ('@2@', ('Aleksej Mihajlovich', 'Romanov')),\n",
|
||||
" ('@3@', ('Marija Ilinichna', 'Miloslavskaja')),\n",
|
||||
" ('@4@', ('Natalja Kirillovna', 'Naryshkina')),\n",
|
||||
" ('@5@', ('Marfa Matveevna', 'Apraksina')),\n",
|
||||
" ('@6@', ('Fedor Alekseevich', 'Romanov')),\n",
|
||||
" ('@7@', ('Sofja Aleksevna', 'Romanova')),\n",
|
||||
" ('@8@', ('Ivan V Alekseevich', 'Romanov')),\n",
|
||||
" ('@9@', ('Praskovja Fedorovna', 'Saltykova')),\n",
|
||||
" ('@10@', ('Ekaterina Ivanovna', 'Romanova')),\n",
|
||||
" ('@11@', ('Anna Ivanovna', 'Romanova')),\n",
|
||||
" ('@12@', ('Fridrih Vilgelm', 'Kurlandskij')),\n",
|
||||
" ('@13@', ('Karl Leopold', 'Meklenburg-Shverinskij')),\n",
|
||||
" ('@14@', ('Anna Leopoldovna', 'Meklenburg-Shverinskaja')),\n",
|
||||
" ('@15@', ('Anton Ulrih', 'Braunshvejg-Volfenbjuttelskij')),\n",
|
||||
" ('@16@', ('Ivan VI Antonovich', 'Braunshvejg-Volfenbjuttelskij')),\n",
|
||||
" ('@17@', ('Petr I Alekseevich', 'Romanov')),\n",
|
||||
" ('@18@', ('Evdokija Fedorovna', 'Lopuhina')),\n",
|
||||
" ('@19@', ('Ekaterina I Alekseevna', 'Mihajlova')),\n",
|
||||
" ('@20@', ('Aleksej Petrovich', 'Romanov')),\n",
|
||||
" ('@21@', ('Sharlotta Kristina', 'Braunshvejg-Volfenbjuttelskaja')),\n",
|
||||
" ('@22@', ('Petr II Alekseevich', 'Romanov')),\n",
|
||||
" ('@23@', ('Anna Petrovna', 'Romanova')),\n",
|
||||
" ('@24@', ('Elizaveta Petrovna', 'Romanova')),\n",
|
||||
" ('@25@', ('Karl Fridrih', 'Golshtejn-Gottorpskij')),\n",
|
||||
" ('@26@', ('Petr III Fedorovich', 'Romanov')),\n",
|
||||
" ('@27@', ('Ekaterina II', 'Alekseevna')),\n",
|
||||
" ('@28@', ('Pavel I Petrovich', 'Romanov')),\n",
|
||||
" ('@29@', ('Natalja Alekseevna', 'Gessen-Darmshtadskaja')),\n",
|
||||
" ('@30@', ('Marija Fedorovna', 'Vjurtembergskaja')),\n",
|
||||
" ('@31@', ('Aleksandr I Pavlovich', 'Romanov')),\n",
|
||||
" ('@32@', ('Elizaveta Alekseevna', 'Baden-Durlahskaja')),\n",
|
||||
" ('@33@', ('Nikolaj I Pavlovich', 'Romanov')),\n",
|
||||
" ('@34@', ('Aleksandra Fedorovna', 'Prusskaja')),\n",
|
||||
" ('@35@', ('Aleksandr II Nikolaevich', 'Romanov')),\n",
|
||||
" ('@36@', ('Marija Aleksandrovna', 'Gessenskaja')),\n",
|
||||
" ('@37@', ('Aleksandr III Aleksandrovich', 'Romanov')),\n",
|
||||
" ('@38@', ('Marija Fedorovna', 'Datskaja')),\n",
|
||||
" ('@39@', ('Nikolaj II Aleksandrovich', 'Romanov')),\n",
|
||||
" ('@40@', ('Aleksandra Fedorovna', 'Gessenskaja'))]"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"d = g.get_element_dictionary()\n",
|
||||
"[ (k,v.get_name()) for k,v in d.items() if isinstance(v,IndividualElement)]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"এখানে কিভাবে আমরা পরিবারের সম্পর্কে তথ্য পেতে পারি। লক্ষ্য করুন এটি আমাদের **পরিচয়সূচক** এর একটি তালিকা দেয়, এবং যদি আমরা আরও স্পষ্টতা চাই তাহলে আমাদের সেগুলো নামের মধ্যে রূপান্তর করতে হবে:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[('@41@', ['@0@', '@1@', '@2@']),\n",
|
||||
" ('@42@', ['@2@', '@3@', '@6@', '@7@', '@8@']),\n",
|
||||
" ('@43@', ['@8@', '@9@', '@10@', '@11@']),\n",
|
||||
" ('@44@', ['@13@', '@10@', '@14@']),\n",
|
||||
" ('@45@', ['@15@', '@14@', '@16@']),\n",
|
||||
" ('@46@', ['@2@', '@4@', '@17@']),\n",
|
||||
" ('@47@', ['@17@', '@18@', '@20@']),\n",
|
||||
" ('@48@', ['@20@', '@21@', '@22@']),\n",
|
||||
" ('@49@', ['@17@', '@19@', '@23@', '@24@']),\n",
|
||||
" ('@50@', ['@25@', '@23@', '@26@']),\n",
|
||||
" ('@51@', ['@26@', '@27@', '@28@']),\n",
|
||||
" ('@52@', ['@28@', '@30@', '@31@', '@33@']),\n",
|
||||
" ('@53@', ['@33@', '@34@', '@35@']),\n",
|
||||
" ('@54@', ['@35@', '@36@', '@37@']),\n",
|
||||
" ('@55@', ['@37@', '@38@', '@39@'])]"
|
||||
]
|
||||
},
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"d = g.get_element_dictionary()\n",
|
||||
"[ (k,[x.get_value() for x in v.get_child_elements()]) for k,v in d.items() if isinstance(v,FamilyElement)]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### পরিবার সম্পর্কিত অন্টোলজি পাওয়া\n",
|
||||
"\n",
|
||||
"এবার, আসুন [পরিবার সম্পর্কিত অন্টোলজি](https://raw.githubusercontent.com/blokhin/genealogical-trees/master/data/header.ttl) দেখি, যা সেমান্টিক ওয়েব ট্রিপলেটসের একটি সেট হিসেবে সংজ্ঞায়িত। এই অন্টোলজি এমন সম্পর্কগুলো সংজ্ঞায়িত করে যেমন `isUncleOf`, `isCousinOf`, এবং আরও অনেক। এই সমস্ত সম্পর্কগুলো মূল প্রেডিকেটস `isMotherOf`, `isFatherOf`, `isBrotherOf` এবং `isSisterOf` এর ভিত্তিতে সংজ্ঞায়িত। আমরা স্বয়ংক্রিয় যুক্তি প্রয়োগ করে এই অন্টোলজির সাহায্যে অন্যান্য সম্পর্কগুলো নির্ণয় করব।\n",
|
||||
"\n",
|
||||
"এখানে `isAuntOf` প্রপার্টির একটি নমুনা সংজ্ঞা দেওয়া হলো, যা `isSisterOf` এবং `isParentOf` এর সমন্বয়ে সংজ্ঞায়িত (*খালা/ফুফু হলেন কারও পিতামাতার বোন*)।\n",
|
||||
"\n",
|
||||
"```\n",
|
||||
"fhkb:isAuntOf a owl:ObjectProperty ;\n",
|
||||
" rdfs:domain fhkb:Woman ;\n",
|
||||
" rdfs:range fhkb:Person ;\n",
|
||||
" owl:propertyChainAxiom ( fhkb:isSisterOf fhkb:isParentOf ) .\n",
|
||||
"```\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"@prefix fhkb: <http://www.example.com/genealogy.owl#> .\n",
|
||||
"@prefix owl: <http://www.w3.org/2002/07/owl#> .\n",
|
||||
"@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n",
|
||||
"@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .\n",
|
||||
"@prefix xml: <http://www.w3.org/XML/1998/namespace> .\n",
|
||||
"@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\n",
|
||||
"\n",
|
||||
"<http://www.example.com/genealogy.owl#> a owl:Ontology .\n",
|
||||
"\n",
|
||||
"fhkb:DomainEntity a owl:Class .\n",
|
||||
"\n",
|
||||
"fhkb:Man a owl:Class ;\n",
|
||||
" owl:equivalentClass [ a owl:Class ;\n",
|
||||
" owl:intersectionOf ( fhkb:Person [ a owl:Restriction ;\n",
|
||||
" owl:onProperty fhkb:hasSex ;\n",
|
||||
" owl:someValuesFrom fhkb:Male ] ) ] .\n",
|
||||
"\n",
|
||||
"fhkb:Woman a owl:Class ;\n",
|
||||
" owl:equivalentClass [ a owl:Class ;\n",
|
||||
" owl:intersectionOf ( fhkb:Person [ a owl:Restriction ;\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"!head -20 data/onto.ttl"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### অনুমানের জন্য অন্টোলজি তৈরি করা\n",
|
||||
"\n",
|
||||
"সহজতার জন্য, আমরা একটি অন্টোলজি ফাইল তৈরি করব যা পরিবার অন্টোলজির মূল নিয়ম এবং আমাদের GEDCOM ফাইল থেকে ব্যক্তিদের সম্পর্কে তথ্য অন্তর্ভুক্ত করবে। আমরা GEDCOM ফাইলটি পর্যালোচনা করব এবং পরিবার ও ব্যক্তিদের সম্পর্কে তথ্য সংগ্রহ করব এবং সেগুলোকে ট্রিপলেটে রূপান্তর করব।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!cp data/onto.ttl .\n",
|
||||
"\n",
|
||||
"gedcom_dict = g.get_element_dictionary()\n",
|
||||
"individuals, marriages = {}, {}\n",
|
||||
"\n",
|
||||
"def term2id(el):\n",
|
||||
" return \"i\" + el.get_pointer().replace('@', '').lower()\n",
|
||||
"\n",
|
||||
"out = open(\"onto.ttl\",\"a\")\n",
|
||||
"\n",
|
||||
"for k, v in gedcom_dict.items():\n",
|
||||
" if isinstance(v,IndividualElement):\n",
|
||||
" children, siblings = set(), set()\n",
|
||||
" idx = term2id(v)\n",
|
||||
"\n",
|
||||
" title = v.get_name()[0] + \" \" + v.get_name()[1]\n",
|
||||
" title = title.replace('\"', '').replace('[', '').replace(']', '').replace('(', '').replace(')', '').strip()\n",
|
||||
"\n",
|
||||
" own_families = g.get_families(v, 'FAMS')\n",
|
||||
" for fam in own_families:\n",
|
||||
" children |= set(term2id(i) for i in g.get_family_members(fam, \"CHIL\"))\n",
|
||||
"\n",
|
||||
" parent_families = g.get_families(v, 'FAMC')\n",
|
||||
" if len(parent_families):\n",
|
||||
" for member in g.get_family_members(parent_families[0], \"CHIL\"): # NB adoptive families i.e len(parent_families)>1 are not considered (TODO?)\n",
|
||||
" if member.get_pointer() == v.get_pointer():\n",
|
||||
" continue\n",
|
||||
" siblings.add(term2id(member))\n",
|
||||
"\n",
|
||||
" if idx in individuals:\n",
|
||||
" children |= individuals[idx].get('children', set())\n",
|
||||
" siblings |= individuals[idx].get('siblings', set())\n",
|
||||
" individuals[idx] = {'sex': v.get_gender().lower(), 'children': children, 'siblings': siblings, 'title': title}\n",
|
||||
"\n",
|
||||
" elif isinstance(v,FamilyElement):\n",
|
||||
" wife, husb, children = None, None, set()\n",
|
||||
" children = set(term2id(i) for i in g.get_family_members(v, \"CHIL\"))\n",
|
||||
"\n",
|
||||
" try:\n",
|
||||
" wife = g.get_family_members(v, \"WIFE\")[0]\n",
|
||||
" wife = term2id(wife)\n",
|
||||
" if wife in individuals: individuals[wife]['children'] |= children\n",
|
||||
" else: individuals[wife] = {'children': children}\n",
|
||||
" except IndexError: pass\n",
|
||||
" try:\n",
|
||||
" husb = g.get_family_members(v, \"HUSB\")[0]\n",
|
||||
" husb = term2id(husb)\n",
|
||||
" if husb in individuals: individuals[husb]['children'] |= children\n",
|
||||
" else: individuals[husb] = {'children': children}\n",
|
||||
" except IndexError: pass\n",
|
||||
"\n",
|
||||
" if wife and husb: marriages[wife + husb] = (term2id(v), wife, husb)\n",
|
||||
"\n",
|
||||
"for idx, val in individuals.items():\n",
|
||||
" added_terms = ''\n",
|
||||
" if val['sex'] == 'f':\n",
|
||||
" parent_predicate, sibl_predicate = \"isMotherOf\", \"isSisterOf\"\n",
|
||||
" else:\n",
|
||||
" parent_predicate, sibl_predicate = \"isFatherOf\", \"isBrotherOf\"\n",
|
||||
" if len(val['children']):\n",
|
||||
" added_terms += \" ;\\n fhkb:\" + parent_predicate + \" \" + \", \".join([\"fhkb:\" + i for i in val['children']])\n",
|
||||
" if len(val['siblings']):\n",
|
||||
" added_terms += \" ;\\n fhkb:\" + sibl_predicate + \" \" + \", \".join([\"fhkb:\" + i for i in val['siblings']])\n",
|
||||
" out.write(\"fhkb:%s a owl:NamedIndividual, owl:Thing%s ;\\n rdfs:label \\\"%s\\\" .\\n\" % (idx, added_terms, val['title']))\n",
|
||||
"\n",
|
||||
"for k, v in marriages.items():\n",
|
||||
" out.write(\"fhkb:%s a owl:NamedIndividual, owl:Thing ;\\n fhkb:hasFemalePartner fhkb:%s ;\\n fhkb:hasMalePartner fhkb:%s .\\n\" % v)\n",
|
||||
"\n",
|
||||
"out.write(\"[] a owl:AllDifferent ;\\n owl:distinctMembers (\")\n",
|
||||
"for idx in individuals.keys():\n",
|
||||
" out.write(\" fhkb:\" + idx)\n",
|
||||
"for k, v in marriages.items():\n",
|
||||
" out.write(\" fhkb:\" + v[0])\n",
|
||||
"out.write(\" ) .\")\n",
|
||||
"out.close()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" fhkb:hasFemalePartner fhkb:i34 ;\n",
|
||||
" fhkb:hasMalePartner fhkb:i33 .\n",
|
||||
"fhkb:i54 a owl:NamedIndividual, owl:Thing ;\n",
|
||||
" fhkb:hasFemalePartner fhkb:i36 ;\n",
|
||||
" fhkb:hasMalePartner fhkb:i35 .\n",
|
||||
"fhkb:i55 a owl:NamedIndividual, owl:Thing ;\n",
|
||||
" fhkb:hasFemalePartner fhkb:i38 ;\n",
|
||||
" fhkb:hasMalePartner fhkb:i37 .\n",
|
||||
"[] a owl:AllDifferent ;\n",
|
||||
" owl:distinctMembers ( fhkb:i0 fhkb:i1 fhkb:i2 fhkb:i3 fhkb:i4 fhkb:i5 fhkb:i6 fhkb:i7 fhkb:i8 fhkb:i9 fhkb:i10 fhkb:i11 fhkb:i12 fhkb:i13 fhkb:i14 fhkb:i15 fhkb:i16 fhkb:i17 fhkb:i18 fhkb:i19 fhkb:i20 fhkb:i21 fhkb:i22 fhkb:i23 fhkb:i24 fhkb:i25 fhkb:i26 fhkb:i27 fhkb:i28 fhkb:i29 fhkb:i30 fhkb:i31 fhkb:i32 fhkb:i33 fhkb:i34 fhkb:i35 fhkb:i36 fhkb:i37 fhkb:i38 fhkb:i39 fhkb:i40 fhkb:i41 fhkb:i42 fhkb:i43 fhkb:i44 fhkb:i45 fhkb:i46 fhkb:i47 fhkb:i48 fhkb:i49 fhkb:i50 fhkb:i51 fhkb:i52 fhkb:i53 fhkb:i54 fhkb:i55 ) ."
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"!tail onto.ttl"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### ইনফারেন্স করা\n",
|
||||
"\n",
|
||||
"এখন আমরা চাই এই অ্যান্টোলজি ব্যবহার করে ইনফারেন্স এবং কুয়েরি করতে। আমরা [RDFLib](https://github.com/RDFLib) ব্যবহার করব, যা বিভিন্ন ফরম্যাটে RDF গ্রাফ পড়া, কুয়েরি করা ইত্যাদির জন্য একটি লাইব্রেরি। \n",
|
||||
"\n",
|
||||
"যৌক্তিক ইনফারেন্সের জন্য, আমরা [OWL-RL](https://github.com/RDFLib/OWL-RL) লাইব্রেরি ব্যবহার করব, যা আমাদের RDF গ্রাফের **ক্লোজার** তৈরি করতে সাহায্য করে, অর্থাৎ সমস্ত সম্ভাব্য ধারণা এবং সম্পর্ক যোগ করা যা ইনফার করা যেতে পারে।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Requirement already satisfied: rdflib in /home/rg/anaconda3/envs/ai4beg/lib/python3.11/site-packages (6.3.2)\n",
|
||||
"Requirement already satisfied: isodate<0.7.0,>=0.6.0 in /home/rg/anaconda3/envs/ai4beg/lib/python3.11/site-packages (from rdflib) (0.6.1)\n",
|
||||
"Requirement already satisfied: pyparsing<4,>=2.1.0 in /home/rg/anaconda3/envs/ai4beg/lib/python3.11/site-packages (from rdflib) (3.0.9)\n",
|
||||
"Requirement already satisfied: six in /home/rg/anaconda3/envs/ai4beg/lib/python3.11/site-packages (from isodate<0.7.0,>=0.6.0->rdflib) (1.16.0)\n",
|
||||
"Collecting git+https://github.com/RDFLib/OWL-RL.git\n",
|
||||
" Cloning https://github.com/RDFLib/OWL-RL.git to /tmp/pip-req-build-lbfzwi3m\n",
|
||||
" Running command git clone --filter=blob:none --quiet https://github.com/RDFLib/OWL-RL.git /tmp/pip-req-build-lbfzwi3m\n",
|
||||
" Resolved https://github.com/RDFLib/OWL-RL.git to commit a77e1791b88b54aace609bc6000aac14c7add4ff\n",
|
||||
" Preparing metadata (setup.py) ... \u001b[?25ldone\n",
|
||||
"\u001b[?25hRequirement already satisfied: rdflib>=6.0.2 in /home/rg/anaconda3/envs/ai4beg/lib/python3.11/site-packages (from owlrl==6.0.2) (6.3.2)\n",
|
||||
"Requirement already satisfied: isodate<0.7.0,>=0.6.0 in /home/rg/anaconda3/envs/ai4beg/lib/python3.11/site-packages (from rdflib>=6.0.2->owlrl==6.0.2) (0.6.1)\n",
|
||||
"Requirement already satisfied: pyparsing<4,>=2.1.0 in /home/rg/anaconda3/envs/ai4beg/lib/python3.11/site-packages (from rdflib>=6.0.2->owlrl==6.0.2) (3.0.9)\n",
|
||||
"Requirement already satisfied: six in /home/rg/anaconda3/envs/ai4beg/lib/python3.11/site-packages (from isodate<0.7.0,>=0.6.0->rdflib>=6.0.2->owlrl==6.0.2) (1.16.0)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"!{sys.executable} -m pip install rdflib\n",
|
||||
"!{sys.executable} -m pip install git+https://github.com/RDFLib/OWL-RL.git"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"চলুন অনটোলজি ফাইলটি খুলে দেখি এতে কতটি ট্রিপলেট রয়েছে:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Triplets found:669\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import rdflib\n",
|
||||
"from owlrl import DeductiveClosure, OWLRL_Extension\n",
|
||||
"\n",
|
||||
"g = rdflib.Graph()\n",
|
||||
"g.parse(\"onto.ttl\", format=\"turtle\")\n",
|
||||
"\n",
|
||||
"print(\"Triplets found:%d\" % len(g))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"এখন চলুন ক্লোজার তৈরি করি, এবং দেখি কীভাবে ত্রয়ীর সংখ্যা বৃদ্ধি পায়:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Triplets after inference:4246\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"DeductiveClosure(OWLRL_Extension).expand(g)\n",
|
||||
"print(\"Triplets after inference:%d\" % len(g))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### আত্মীয়দের জন্য অনুসন্ধান করা\n",
|
||||
"\n",
|
||||
"এখন আমরা গ্রাফে অনুসন্ধান করতে পারি এবং মানুষের মধ্যে বিভিন্ন সম্পর্ক দেখতে পারি। আমরা **SPARQL** ভাষা ব্যবহার করতে পারি `query` মেথডের সাথে। আমাদের ক্ষেত্রে, চলুন দেখি আমাদের পারিবারিক গাছে সমস্ত **চাচা** কে আছে:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Fedor Alekseevich Romanov is uncle of Ekaterina Ivanovna Romanova\n",
|
||||
"Aleksandr I Pavlovich Romanov is uncle of Aleksandr II Nikolaevich Romanov\n",
|
||||
"Fedor Alekseevich Romanov is uncle of Anna Ivanovna Romanova\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"qres = g.query(\n",
|
||||
" \"\"\"SELECT DISTINCT ?aname ?bname\n",
|
||||
" WHERE {\n",
|
||||
" ?a fhkb:isUncleOf ?b .\n",
|
||||
" ?a rdfs:label ?aname .\n",
|
||||
" ?b rdfs:label ?bname .\n",
|
||||
" }\"\"\")\n",
|
||||
"\n",
|
||||
"for row in qres:\n",
|
||||
" print(\"%s is uncle of %s\" % row)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"বিভিন্ন পারিবারিক সম্পর্ক নিয়ে পরীক্ষা-নিরীক্ষা করতে পারেন। উদাহরণস্বরূপ, আপনি `isAncestorOf` সম্পর্কটি দেখতে পারেন, যা পুনরাবৃত্তি করে একটি নির্দিষ্ট ব্যক্তির সকল পূর্বপুরুষকে সংজ্ঞায়িত করে।\n",
|
||||
"\n",
|
||||
"শেষে, চলুন পরিষ্কার করি!\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!rm onto.ttl"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**দাবিত্যাগ**: \nএই নথিটি AI অনুবাদ পরিষেবা [Co-op Translator](https://github.com/Azure/co-op-translator) ব্যবহার করে অনুবাদ করা হয়েছে। আমরা যথাসাধ্য সঠিকতা নিশ্চিত করার চেষ্টা করি, তবে অনুগ্রহ করে মনে রাখবেন যে স্বয়ংক্রিয় অনুবাদে ত্রুটি বা অসঙ্গতি থাকতে পারে। মূল ভাষায় থাকা নথিটিকে প্রামাণিক উৎস হিসেবে বিবেচনা করা উচিত। গুরুত্বপূর্ণ তথ্যের জন্য, পেশাদার মানব অনুবাদ সুপারিশ করা হয়। এই অনুবাদ ব্যবহারের ফলে কোনো ভুল বোঝাবুঝি বা ভুল ব্যাখ্যা হলে আমরা দায়বদ্ধ থাকব না।\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.6",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.11.2"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "6537d5597320e27b6052b4377b8ff8bb",
|
||||
"translation_date": "2025-08-28T11:27:54+00:00",
|
||||
"source_file": "lessons/2-Symbolic/FamilyOntology.ipynb",
|
||||
"language_code": "bn"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
|
|
@ -0,0 +1,548 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"collapsed": true
|
||||
},
|
||||
"source": [
|
||||
"## মাইক্রোসফট কনসেপ্ট গ্রাফ\n",
|
||||
"\n",
|
||||
"[Microsoft Concept Graph](https://concept.research.microsoft.com/) হলো ইন্টারনেট থেকে সংগৃহীত একটি বৃহৎ ট্যাক্সোনমি, যেখানে ধারণাগুলোর মধ্যে `is-a` সম্পর্ক বিদ্যমান।\n",
|
||||
"\n",
|
||||
"কনটেক্সট গ্রাফ দুটি ফর্মে উপলব্ধ:\n",
|
||||
" * ডাউনলোডের জন্য বড় টেক্সট ফাইল\n",
|
||||
" * REST API\n",
|
||||
"\n",
|
||||
"পরিসংখ্যান:\n",
|
||||
" * ৫৪০১৯৩৩টি অনন্য ধারণা,\n",
|
||||
" * ১২৫৫১৬১৩টি অনন্য উদাহরণ,\n",
|
||||
" * ৮৭৬০৩৯৪৭টি `is-a` সম্পর্ক\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## ওয়েব সার্ভিস ব্যবহার করা\n",
|
||||
"\n",
|
||||
"ওয়েব সার্ভিস বিভিন্ন কল অফার করে যা একটি ধারণার বিভিন্ন গ্রুপের অন্তর্ভুক্ত হওয়ার সম্ভাবনা অনুমান করতে সাহায্য করে। আরও তথ্য পাওয়া যাবে [এখানে](https://concept.research.microsoft.com/Home/Api)।\n",
|
||||
"এখানে একটি নমুনা URL দেওয়া হলো কল করার জন্য: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"{'company': 0.6105356614382954,\n",
|
||||
" 'vendor': 0.08858636677518003,\n",
|
||||
" 'client': 0.048239124001183784,\n",
|
||||
" 'firm': 0.045476965571668145,\n",
|
||||
" 'large company': 0.043109401203511886,\n",
|
||||
" 'organization': 0.043010752688172046,\n",
|
||||
" 'corporation': 0.035908059583703265,\n",
|
||||
" 'brand': 0.03383644076156654,\n",
|
||||
" 'software company': 0.027522935779816515,\n",
|
||||
" 'technology company': 0.023774292196902438}"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import urllib\n",
|
||||
"import json\n",
|
||||
"import ssl\n",
|
||||
"\n",
|
||||
"def http(x):\n",
|
||||
" ssl._create_default_https_context = ssl._create_unverified_context\n",
|
||||
" response = urllib.request.urlopen(x)\n",
|
||||
" data = response.read()\n",
|
||||
" return data.decode('utf-8')\n",
|
||||
"\n",
|
||||
"def query(x):\n",
|
||||
" return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n",
|
||||
"\n",
|
||||
"query('microsoft')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"চলুন খবরের শিরোনামগুলোকে মূল ধারণার ভিত্তিতে শ্রেণীবদ্ধ করার চেষ্টা করি। খবরের শিরোনাম পেতে, আমরা [NewsApi.org](http://newsapi.org) সেবাটি ব্যবহার করব। এই সেবাটি ব্যবহার করার জন্য আপনাকে নিজের API কী সংগ্রহ করতে হবে - ওয়েবসাইটে যান এবং বিনামূল্যে ডেভেলপার প্ল্যানে নিবন্ধন করুন।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 20,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"newsapi_key = '<your API key here>'\n",
|
||||
"def get_news(country='us'):\n",
|
||||
" res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n",
|
||||
" return res['articles']\n",
|
||||
"\n",
|
||||
"all_titles = [x['title'] for x in get_news('us')+get_news('gb')]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 21,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n",
|
||||
" 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n",
|
||||
" 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n",
|
||||
" 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n",
|
||||
" \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n",
|
||||
" '2 teachers killed at Swedish high school, student arrested - ABC News',\n",
|
||||
" 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n",
|
||||
" 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n",
|
||||
" '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n",
|
||||
" 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n",
|
||||
" \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n",
|
||||
" \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n",
|
||||
" \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n",
|
||||
" 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n",
|
||||
" 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n",
|
||||
" 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n",
|
||||
" 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n",
|
||||
" 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n",
|
||||
" \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n",
|
||||
" 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n",
|
||||
" \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n",
|
||||
" 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n",
|
||||
" 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n",
|
||||
" 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n",
|
||||
" 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n",
|
||||
" \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n",
|
||||
" 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n",
|
||||
" \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n",
|
||||
" 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n",
|
||||
" 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n",
|
||||
" \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n",
|
||||
" '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n",
|
||||
" 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n",
|
||||
" 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n",
|
||||
" 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n",
|
||||
" 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n",
|
||||
" 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n",
|
||||
" 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n",
|
||||
" \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n",
|
||||
" 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']"
|
||||
]
|
||||
},
|
||||
"execution_count": 21,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"all_titles"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"প্রথমত, আমরা সংবাদ শিরোনাম থেকে বিশেষ্য বের করতে সক্ষম হতে চাই। আমরা এটি করার জন্য `TextBlob` লাইব্রেরি ব্যবহার করব, যা এই ধরনের সাধারণ NLP কাজগুলোকে অনেক সহজ করে।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n",
|
||||
"Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n",
|
||||
"Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n",
|
||||
"Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n",
|
||||
"Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n",
|
||||
"Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n",
|
||||
"Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n",
|
||||
"Finished.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"[nltk_data] Downloading package brown to\n",
|
||||
"[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n",
|
||||
"[nltk_data] Package brown is already up-to-date!\n",
|
||||
"[nltk_data] Downloading package punkt to\n",
|
||||
"[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n",
|
||||
"[nltk_data] Package punkt is already up-to-date!\n",
|
||||
"[nltk_data] Downloading package wordnet to\n",
|
||||
"[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n",
|
||||
"[nltk_data] Package wordnet is already up-to-date!\n",
|
||||
"[nltk_data] Downloading package averaged_perceptron_tagger to\n",
|
||||
"[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n",
|
||||
"[nltk_data] Package averaged_perceptron_tagger is already up-to-\n",
|
||||
"[nltk_data] date!\n",
|
||||
"[nltk_data] Downloading package conll2000 to\n",
|
||||
"[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n",
|
||||
"[nltk_data] Package conll2000 is already up-to-date!\n",
|
||||
"[nltk_data] Downloading package movie_reviews to\n",
|
||||
"[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n",
|
||||
"[nltk_data] Package movie_reviews is already up-to-date!\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import sys\n",
|
||||
"!{sys.executable} -m pip install textblob\n",
|
||||
"!{sys.executable} -m textblob.download_corpora\n",
|
||||
"from textblob import TextBlob"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 22,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"{'covid-19 live updates': 1,\n",
|
||||
" 'vaccines': 1,\n",
|
||||
" 'boosters': 1,\n",
|
||||
" 'york': 4,\n",
|
||||
" 'ukrainians flee mariupol': 1,\n",
|
||||
" 'forces push': 1,\n",
|
||||
" 'port city': 1,\n",
|
||||
" 'wall street journal': 3,\n",
|
||||
" 'bond yields': 1,\n",
|
||||
" 'futures rise': 1,\n",
|
||||
" 'powell says fed': 1,\n",
|
||||
" 'ready': 1,\n",
|
||||
" 'be': 1,\n",
|
||||
" 'aggressive': 1,\n",
|
||||
" 'putin': 3,\n",
|
||||
" 'alexei navalny': 2,\n",
|
||||
" 'russian': 2,\n",
|
||||
" 'supreme court nominee': 1,\n",
|
||||
" 'ketanji brown jackson': 1,\n",
|
||||
" \"confirmation hearing 's\": 1,\n",
|
||||
" 'cnn': 1,\n",
|
||||
" 'swedish': 1,\n",
|
||||
" 'high school': 1,\n",
|
||||
" 'abc': 1,\n",
|
||||
" 'clues': 1,\n",
|
||||
" 'covid-19': 1,\n",
|
||||
" '’ s': 2,\n",
|
||||
" 'moves': 1,\n",
|
||||
" 'sewers': 1,\n",
|
||||
" 'roll dice': 1,\n",
|
||||
" 'jackson': 1,\n",
|
||||
" 'decisions |': 1,\n",
|
||||
" 'thehill': 1,\n",
|
||||
" 'clear': 2,\n",
|
||||
" 'chemical weapons': 2,\n",
|
||||
" 'ukraine': 3,\n",
|
||||
" 'claims president': 2,\n",
|
||||
" 'biden': 2,\n",
|
||||
" 'nasa': 2,\n",
|
||||
" 'solar system': 2,\n",
|
||||
" 'daily mail': 3,\n",
|
||||
" 'us stocks': 1,\n",
|
||||
" 'fed chair powell': 1,\n",
|
||||
" \"'s remarks\": 1,\n",
|
||||
" 'fox': 1,\n",
|
||||
" \"'we 've\": 1,\n",
|
||||
" 'tests': 1,\n",
|
||||
" 'covid': 1,\n",
|
||||
" 'politico': 1,\n",
|
||||
" 'duchess': 1,\n",
|
||||
" 'cambridge': 1,\n",
|
||||
" 'swaps khaki jungle gear': 1,\n",
|
||||
" 'vampire': 1,\n",
|
||||
" 'wife': 1,\n",
|
||||
" 'belize': 1,\n",
|
||||
" 'china': 2,\n",
|
||||
" 'flight recorders': 1,\n",
|
||||
" 'plane crash': 1,\n",
|
||||
" 'reuters': 2,\n",
|
||||
" 'russian oligarch': 1,\n",
|
||||
" 'abramovich': 1,\n",
|
||||
" 'live': 1,\n",
|
||||
" 'russia': 2,\n",
|
||||
" 'stops talks': 1,\n",
|
||||
" 'japan': 1,\n",
|
||||
" 'español': 1,\n",
|
||||
" 'powers remain': 1,\n",
|
||||
" 'threats lurk': 1,\n",
|
||||
" 'set': 1,\n",
|
||||
" 'webb': 1,\n",
|
||||
" 'telescope begins multi-instrument alignment': 1,\n",
|
||||
" 'scitechdaily': 1,\n",
|
||||
" 'uconn': 1,\n",
|
||||
" 'ucf': 1,\n",
|
||||
" 'ncaa': 1,\n",
|
||||
" \"women 's tournament second-round highlights\": 1,\n",
|
||||
" 'march madness': 1,\n",
|
||||
" 'bucking republican trend': 1,\n",
|
||||
" 'indiana': 1,\n",
|
||||
" 'vetoes transgender': 1,\n",
|
||||
" 'bill': 1,\n",
|
||||
" 'maggie fox': 1,\n",
|
||||
" 'coronation': 1,\n",
|
||||
" 'shameless': 1,\n",
|
||||
" \"'sudden accident\": 1,\n",
|
||||
" 'mirror online': 1,\n",
|
||||
" 'mirror': 2,\n",
|
||||
" 'plane crash –': 1,\n",
|
||||
" 'search': 1,\n",
|
||||
" 'moment flight': 1,\n",
|
||||
" 'daniel morgan': 1,\n",
|
||||
" 'report condemns': 1,\n",
|
||||
" 'met': 1,\n",
|
||||
" 'guardian': 6,\n",
|
||||
" 'rishi sunak': 1,\n",
|
||||
" '’ s spring': 1,\n",
|
||||
" 'statement': 1,\n",
|
||||
" 'bbc.com': 1,\n",
|
||||
" 'uk': 3,\n",
|
||||
" 'ireland': 1,\n",
|
||||
" 'euro': 1,\n",
|
||||
" 'vladimir putin': 2,\n",
|
||||
" \"'s 'lover\": 1,\n",
|
||||
" 'brass eye': 1,\n",
|
||||
" '’ s outtakes': 1,\n",
|
||||
" 'brutal tv comedy': 1,\n",
|
||||
" 'threatens civilians': 1,\n",
|
||||
" 'mariupol': 1,\n",
|
||||
" \"'s spirit\": 1,\n",
|
||||
" 'shell u-turn': 1,\n",
|
||||
" 'cambo': 1,\n",
|
||||
" 'green targets': 1,\n",
|
||||
" 'st helens': 1,\n",
|
||||
" 'dog attack': 1,\n",
|
||||
" 'girl': 1,\n",
|
||||
" 'bbc': 1,\n",
|
||||
" 'playstation': 1,\n",
|
||||
" \"'assassin 's\": 1,\n",
|
||||
" 'creed': 1,\n",
|
||||
" 'jade raymond': 1,\n",
|
||||
" 'haven studios': 1,\n",
|
||||
" 'nme': 1,\n",
|
||||
" 'nintendo switch': 1,\n",
|
||||
" 'folders •': 1,\n",
|
||||
" 'eurogamer.net': 2,\n",
|
||||
" 'fa': 1,\n",
|
||||
" 'solution ”': 1,\n",
|
||||
" 'liverpool': 1,\n",
|
||||
" 'fan group blasts “ shambolic ”': 1,\n",
|
||||
" 'wembley': 1,\n",
|
||||
" 'anfield': 1,\n",
|
||||
" 'manchester': 1,\n",
|
||||
" 'live erik': 1,\n",
|
||||
" 'hag': 1,\n",
|
||||
" 'utd': 1,\n",
|
||||
" 'manager updates': 1,\n",
|
||||
" 'manchester evening': 1,\n",
|
||||
" 'inflation': 1,\n",
|
||||
" 'government borrowing': 1,\n",
|
||||
" 'february': 1,\n",
|
||||
" 'crude oil': 1,\n",
|
||||
" '– business': 1,\n",
|
||||
" 'kremlin': 1,\n",
|
||||
" 'large-scale fraud': 1,\n",
|
||||
" 'sky': 1,\n",
|
||||
" 'natural gas': 1,\n",
|
||||
" 'gazprom': 1,\n",
|
||||
" 'retail unit': 1,\n",
|
||||
" 'insider': 1,\n",
|
||||
" 'zaghari-ratcliffe': 1,\n",
|
||||
" 'hunt': 1,\n",
|
||||
" 'iran': 1,\n",
|
||||
" 'debt payment': 1}"
|
||||
]
|
||||
},
|
||||
"execution_count": 22,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"w = {}\n",
|
||||
"for x in all_titles:\n",
|
||||
" for n in TextBlob(x).noun_phrases:\n",
|
||||
" if n in w:\n",
|
||||
" w[n].append(x)\n",
|
||||
" else:\n",
|
||||
" w[n]=[x]\n",
|
||||
"{ x:len(w[x]) for x in w.keys()}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"আমরা দেখতে পাচ্ছি যে বিশেষ্যগুলি আমাদের বড় থিম্যাটিক গোষ্ঠী দেয় না। চলুন বিশেষ্যগুলিকে ধারণা গ্রাফ থেকে প্রাপ্ত আরও সাধারণ শব্দ দিয়ে প্রতিস্থাপন করি। এটি কিছু সময় নেবে, কারণ আমরা প্রতিটি বিশেষ্য বাক্যের জন্য REST কল করছি।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 23,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"w = {}\n",
|
||||
"for x in all_titles:\n",
|
||||
" for noun in TextBlob(x).noun_phrases:\n",
|
||||
" terms = query(noun.replace(' ','%20'))\n",
|
||||
" for term in [u for u in terms.keys() if terms[u]>0.1]:\n",
|
||||
" if term in w:\n",
|
||||
" w[term].append(x)\n",
|
||||
" else:\n",
|
||||
" w[term]=[x]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 24,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"{'city': 9,\n",
|
||||
" 'brand': 4,\n",
|
||||
" 'place': 9,\n",
|
||||
" 'town': 4,\n",
|
||||
" 'factor': 4,\n",
|
||||
" 'film': 4,\n",
|
||||
" 'nation': 11,\n",
|
||||
" 'state': 5,\n",
|
||||
" 'person': 4,\n",
|
||||
" 'organization': 5,\n",
|
||||
" 'publication': 10,\n",
|
||||
" 'market': 5,\n",
|
||||
" 'economy': 4,\n",
|
||||
" 'company': 6,\n",
|
||||
" 'newspaper': 6,\n",
|
||||
" 'relationship': 6}"
|
||||
]
|
||||
},
|
||||
"execution_count": 24,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"{ x:len(w[x]) for x in w.keys() if len(w[x])>3}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 27,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\n",
|
||||
"ECONOMY:\n",
|
||||
"China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n",
|
||||
"Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n",
|
||||
"China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n",
|
||||
"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n",
|
||||
"\n",
|
||||
"NATION:\n",
|
||||
"‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n",
|
||||
"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n",
|
||||
"China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n",
|
||||
"Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n",
|
||||
"Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n",
|
||||
"China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n",
|
||||
"UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n",
|
||||
"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n",
|
||||
"‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n",
|
||||
"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n",
|
||||
"Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n",
|
||||
"\n",
|
||||
"PERSON:\n",
|
||||
"‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n",
|
||||
"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n",
|
||||
"Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n",
|
||||
"‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n",
|
||||
"print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n",
|
||||
"print('\\nPERSON:\\n'+'\\n'.join(w['person']))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**অস্বীকৃতি**: \nএই নথিটি AI অনুবাদ পরিষেবা [Co-op Translator](https://github.com/Azure/co-op-translator) ব্যবহার করে অনুবাদ করা হয়েছে। আমরা যথাসম্ভব সঠিকতার জন্য চেষ্টা করি, তবে অনুগ্রহ করে মনে রাখবেন যে স্বয়ংক্রিয় অনুবাদে ত্রুটি বা অসঙ্গতি থাকতে পারে। মূল ভাষায় থাকা নথিটিকে প্রামাণিক উৎস হিসেবে বিবেচনা করা উচিত। গুরুত্বপূর্ণ তথ্যের জন্য, পেশাদার মানব অনুবাদ সুপারিশ করা হয়। এই অনুবাদ ব্যবহারের ফলে কোনো ভুল বোঝাবুঝি বা ভুল ব্যাখ্যা হলে আমরা দায়বদ্ধ থাকব না।\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.7.4 64-bit (conda)",
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5"
|
||||
}
|
||||
},
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.5"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "4087f998407d06ceb2947016ba4605d0",
|
||||
"translation_date": "2025-08-28T11:29:08+00:00",
|
||||
"source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb",
|
||||
"language_code": "bn"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
|
|
@ -0,0 +1,183 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# আমাদের নিজস্ব ফ্রেমওয়ার্ক দিয়ে MNIST ডিজিট শ্রেণীবিন্যাস\n",
|
||||
"\n",
|
||||
"[AI for Beginners Curriculum](https://github.com/microsoft/ai-for-beginners) থেকে ল্যাব অ্যাসাইনমেন্ট।\n",
|
||||
"\n",
|
||||
"### ডেটাসেট পড়া\n",
|
||||
"\n",
|
||||
"এই কোডটি ইন্টারনেটে রিপোজিটরি থেকে ডেটাসেট ডাউনলোড করে। আপনি চাইলে AI Curriculum রিপোর `/data` ডিরেক্টরি থেকে ডেটাসেট ম্যানুয়ালি কপি করতে পারেন।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" % Total % Received % Xferd Average Speed Time Time Time Current\n",
|
||||
" Dload Upload Total Spent Left Speed\n",
|
||||
"\n",
|
||||
" 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\n",
|
||||
"100 9.9M 100 9.9M 0 0 9.9M 0 0:00:01 --:--:-- 0:00:01 15.8M\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"!rm *.pkl\n",
|
||||
"!wget https://raw.githubusercontent.com/microsoft/AI-For-Beginners/main/data/mnist.pkl.gz\n",
|
||||
"!gzip -d mnist.pkl.gz"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import pickle\n",
|
||||
"with open('mnist.pkl','rb') as f:\n",
|
||||
" MNIST = pickle.load(f)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"labels = MNIST['Train']['Labels']\n",
|
||||
"data = MNIST['Train']['Features']"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"চলুন দেখি আমাদের কাছে থাকা ডেটার আকৃতি কী:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(42000, 784)"
|
||||
]
|
||||
},
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"data.shape"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### ডেটা বিভাজন\n",
|
||||
"\n",
|
||||
"আমরা Scikit Learn ব্যবহার করে ডেটাকে প্রশিক্ষণ এবং টেস্ট ডেটাসেটে বিভক্ত করব:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Train samples: 33600, test samples: 8400\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from sklearn.model_selection import train_test_split\n",
|
||||
"\n",
|
||||
"features_train, features_test, labels_train, labels_test = train_test_split(data,labels,test_size=0.2)\n",
|
||||
"\n",
|
||||
"print(f\"Train samples: {len(features_train)}, test samples: {len(features_test)}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### নির্দেশাবলী\n",
|
||||
"\n",
|
||||
"1. পাঠ থেকে ফ্রেমওয়ার্ক কোডটি নিয়ে এই নোটবুকে পেস্ট করুন, অথবা (আরও ভালো) একটি আলাদা Python মডিউলে পেস্ট করুন।\n",
|
||||
"1. এক-স্তর বিশিষ্ট পার্সেপট্রন সংজ্ঞায়িত করুন এবং প্রশিক্ষণ দিন, প্রশিক্ষণ এবং যাচাইকরণ সঠিকতা পর্যবেক্ষণ করুন।\n",
|
||||
"1. বুঝতে চেষ্টা করুন যে অতিরিক্ত প্রশিক্ষণ (overfitting) হয়েছে কিনা, এবং স্তরের প্যারামিটারগুলো সামঞ্জস্য করে সঠিকতা উন্নত করুন।\n",
|
||||
"1. পূর্ববর্তী ধাপগুলো ২-স্তর এবং ৩-স্তর বিশিষ্ট পার্সেপট্রনের জন্য পুনরাবৃত্তি করুন। স্তরগুলোর মধ্যে বিভিন্ন সক্রিয়করণ ফাংশন নিয়ে পরীক্ষা-নিরীক্ষা করুন।\n",
|
||||
"1. নিম্নলিখিত প্রশ্নগুলোর উত্তর দেওয়ার চেষ্টা করুন:\n",
|
||||
" - স্তরগুলোর মধ্যে সক্রিয়করণ ফাংশন কি নেটওয়ার্কের কার্যকারিতাকে প্রভাবিত করে?\n",
|
||||
" - এই কাজের জন্য আমাদের কি ২-স্তর বা ৩-স্তর বিশিষ্ট নেটওয়ার্ক প্রয়োজন?\n",
|
||||
" - নেটওয়ার্ক প্রশিক্ষণের সময় কি কোনো সমস্যা হয়েছে? বিশেষ করে যখন স্তরের সংখ্যা বৃদ্ধি পেয়েছে।\n",
|
||||
" - প্রশিক্ষণের সময় নেটওয়ার্কের ওজনগুলো কীভাবে আচরণ করে? আপনি ওজনের সর্বাধিক মানের পরিবর্তন বনাম epoch এর সম্পর্ক বুঝতে গ্রাফ আঁকতে পারেন।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**অস্বীকৃতি**: \nএই নথিটি AI অনুবাদ পরিষেবা [Co-op Translator](https://github.com/Azure/co-op-translator) ব্যবহার করে অনুবাদ করা হয়েছে। আমরা যথাসম্ভব সঠিকতার জন্য চেষ্টা করি, তবে অনুগ্রহ করে মনে রাখবেন যে স্বয়ংক্রিয় অনুবাদে ত্রুটি বা অসঙ্গতি থাকতে পারে। মূল ভাষায় থাকা নথিটিকে প্রামাণিক উৎস হিসেবে বিবেচনা করা উচিত। গুরুত্বপূর্ণ তথ্যের জন্য, পেশাদার মানব অনুবাদ সুপারিশ করা হয়। এই অনুবাদ ব্যবহারের ফলে কোনো ভুল বোঝাবুঝি বা ভুল ব্যাখ্যা হলে আমরা দায়বদ্ধ থাকব না।\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.7.4 64-bit (conda)",
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5"
|
||||
}
|
||||
},
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.5"
|
||||
},
|
||||
"orig_nbformat": 2,
|
||||
"coopTranslator": {
|
||||
"original_hash": "6fa055f484eb5d6bdf41166a356d3abf",
|
||||
"translation_date": "2025-08-28T11:36:59+00:00",
|
||||
"source_file": "lessons/3-NeuralNetworks/04-OwnFramework/lab/MyFW_MNIST.ipynb",
|
||||
"language_code": "bn"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
|
|
@ -0,0 +1,108 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## অপটিক্যাল ফ্লো ব্যবহার করে হাতের নড়াচড়া শনাক্তকরণ\n",
|
||||
"\n",
|
||||
"এই ল্যাবটি [AI for Beginners Curriculum](http://aka.ms/ai-beginners)-এর অংশ।\n",
|
||||
"\n",
|
||||
"[এই ভিডিওটি](../../../../../../lessons/4-ComputerVision/06-IntroCV/lab/palm-movement.mp4) বিবেচনা করুন, যেখানে একটি ব্যক্তির হাত স্থির পটভূমির উপর বাম/ডান/উপর/নিচে নড়াচড়া করছে।\n",
|
||||
"\n",
|
||||
"**আপনার লক্ষ্য** হবে অপটিক্যাল ফ্লো ব্যবহার করে নির্ধারণ করা, ভিডিওর কোন অংশে উপর/নিচ/বাম/ডান নড়াচড়া রয়েছে।\n",
|
||||
"\n",
|
||||
"লেকচারে বর্ণিত পদ্ধতি অনুসারে ভিডিও ফ্রেম সংগ্রহ করা দিয়ে শুরু করুন:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Code here"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"এখন, লেকচারে বর্ণিত অনুযায়ী ঘন অপটিক্যাল ফ্লো ফ্রেমগুলি গণনা করুন এবং ঘন অপটিক্যাল ফ্লোকে পোলার কোঅর্ডিনেটে রূপান্তর করুন:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Code here"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"প্রতিটি অপটিক্যাল ফ্লো ফ্রেমের জন্য দিকনির্দেশের একটি হিস্টোগ্রাম তৈরি করুন। একটি হিস্টোগ্রাম দেখায় কতগুলো ভেক্টর নির্দিষ্ট বিনের মধ্যে পড়ে, এবং এটি ফ্রেমে বিভিন্ন দিকের গতিবিধিকে আলাদা করে।\n",
|
||||
"\n",
|
||||
"> আপনি এমন সমস্ত ভেক্টরকে শূন্য করতে চাইতে পারেন যাদের মাত্রা নির্দিষ্ট সীমার নিচে। এটি ভিডিওতে ছোট অতিরিক্ত গতিবিধি, যেমন চোখ এবং মাথার নড়াচড়া, দূর করতে সাহায্য করবে।\n",
|
||||
"\n",
|
||||
"কিছু ফ্রেমের জন্য হিস্টোগ্রামগুলো প্লট করুন।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Code here"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"হিস্টোগ্রামগুলোর দিকে তাকালে, গতির দিক নির্ধারণ করা বেশ সহজ হওয়া উচিত। আপনাকে সেই বিনগুলো নির্বাচন করতে হবে যা উপরে/নিচে/বামে/ডানে দিকের সাথে সম্পর্কিত, এবং যা নির্দিষ্ট সীমার উপরে রয়েছে।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Code here"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"অভিনন্দন! যদি আপনি উপরের সমস্ত ধাপ সম্পন্ন করে থাকেন, তবে আপনি ল্যাবটি সম্পন্ন করেছেন!\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**অস্বীকৃতি**: \nএই নথিটি AI অনুবাদ পরিষেবা [Co-op Translator](https://github.com/Azure/co-op-translator) ব্যবহার করে অনুবাদ করা হয়েছে। আমরা যথাসাধ্য সঠিকতা নিশ্চিত করার চেষ্টা করি, তবে অনুগ্রহ করে মনে রাখবেন যে স্বয়ংক্রিয় অনুবাদে ত্রুটি বা অসঙ্গতি থাকতে পারে। মূল ভাষায় থাকা নথিটিকে প্রামাণিক উৎস হিসেবে বিবেচনা করা উচিত। গুরুত্বপূর্ণ তথ্যের জন্য, পেশাদার মানব অনুবাদ সুপারিশ করা হয়। এই অনুবাদ ব্যবহারের ফলে কোনো ভুল বোঝাবুঝি বা ভুল ব্যাখ্যা হলে আমরা দায়বদ্ধ থাকব না।\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
},
|
||||
"orig_nbformat": 4,
|
||||
"coopTranslator": {
|
||||
"original_hash": "153d9e417e079bf62f8f693002d0deaf",
|
||||
"translation_date": "2025-08-28T11:10:02+00:00",
|
||||
"source_file": "lessons/4-ComputerVision/06-IntroCV/lab/MovementDetection.ipynb",
|
||||
"language_code": "bn"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
|
|
@ -0,0 +1,577 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# টেক্সট শ্রেণীবিন্যাস কাজ\n",
|
||||
"\n",
|
||||
"যেমনটি আমরা উল্লেখ করেছি, আমরা **AG_NEWS** ডেটাসেটের উপর ভিত্তি করে একটি সহজ টেক্সট শ্রেণীবিন্যাস কাজের উপর মনোযোগ দেব। এই কাজটি হলো সংবাদ শিরোনামগুলোকে চারটি বিভাগে শ্রেণীবদ্ধ করা: বিশ্ব, ক্রীড়া, ব্যবসা এবং বিজ্ঞান/প্রযুক্তি।\n",
|
||||
"\n",
|
||||
"## ডেটাসেট\n",
|
||||
"\n",
|
||||
"এই ডেটাসেটটি [`torchtext`](https://github.com/pytorch/text) মডিউলে অন্তর্ভুক্ত, তাই আমরা এটি সহজেই অ্যাক্সেস করতে পারি।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"import torchtext\n",
|
||||
"import os\n",
|
||||
"import collections\n",
|
||||
"os.makedirs('./data',exist_ok=True)\n",
|
||||
"train_dataset, test_dataset = torchtext.datasets.AG_NEWS(root='./data')\n",
|
||||
"classes = ['World', 'Sports', 'Business', 'Sci/Tech']"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"এখানে, `train_dataset` এবং `test_dataset` এমন সংগ্রহসমূহ ধারণ করে যা যথাক্রমে শ্রেণীর সংখ্যা (লেবেল) এবং টেক্সটের জোড়া প্রদান করে, উদাহরণস্বরূপ:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(3,\n",
|
||||
" \"Wall St. Bears Claw Back Into the Black (Reuters) Reuters - Short-sellers, Wall Street's dwindling\\\\band of ultra-cynics, are seeing green again.\")"
|
||||
]
|
||||
},
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"list(train_dataset)[0]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"তাহলে, চলুন আমাদের ডেটাসেট থেকে প্রথম ১০টি নতুন শিরোনাম প্রিন্ট করি:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"**Sci/Tech** -> Wall St. Bears Claw Back Into the Black (Reuters) Reuters - Short-sellers, Wall Street's dwindling\\band of ultra-cynics, are seeing green again.\n",
|
||||
"**Sci/Tech** -> Carlyle Looks Toward Commercial Aerospace (Reuters) Reuters - Private investment firm Carlyle Group,\\which has a reputation for making well-timed and occasionally\\controversial plays in the defense industry, has quietly placed\\its bets on another part of the market.\n",
|
||||
"**Sci/Tech** -> Oil and Economy Cloud Stocks' Outlook (Reuters) Reuters - Soaring crude prices plus worries\\about the economy and the outlook for earnings are expected to\\hang over the stock market next week during the depth of the\\summer doldrums.\n",
|
||||
"**Sci/Tech** -> Iraq Halts Oil Exports from Main Southern Pipeline (Reuters) Reuters - Authorities have halted oil export\\flows from the main pipeline in southern Iraq after\\intelligence showed a rebel militia could strike\\infrastructure, an oil official said on Saturday.\n",
|
||||
"**Sci/Tech** -> Oil prices soar to all-time record, posing new menace to US economy (AFP) AFP - Tearaway world oil prices, toppling records and straining wallets, present a new economic menace barely three months before the US presidential elections.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"for i,x in zip(range(5),train_dataset):\n",
|
||||
" print(f\"**{classes[x[0]]}** -> {x[1]}\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"কারণ ডেটাসেটগুলি ইটারেটর, যদি আমরা ডেটা একাধিকবার ব্যবহার করতে চাই তবে আমাদের এটি তালিকায় রূপান্তর করতে হবে:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"train_dataset, test_dataset = torchtext.datasets.AG_NEWS(root='./data')\n",
|
||||
"train_dataset = list(train_dataset)\n",
|
||||
"test_dataset = list(test_dataset)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## টোকেনাইজেশন\n",
|
||||
"\n",
|
||||
"এখন আমাদের টেক্সটকে **সংখ্যায়** রূপান্তর করতে হবে যা টেনসর হিসেবে উপস্থাপন করা যায়। যদি আমরা শব্দ-স্তরের উপস্থাপনা চাই, তাহলে আমাদের দুটি কাজ করতে হবে:\n",
|
||||
"* **টোকেনাইজার** ব্যবহার করে টেক্সটকে **টোকেন** এ বিভক্ত করা\n",
|
||||
"* সেই টোকেনগুলোর একটি **ভোকাবুলারি** তৈরি করা।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"['he', 'said', 'hello']"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"tokenizer = torchtext.data.utils.get_tokenizer('basic_english')\n",
|
||||
"tokenizer('He said: hello')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"counter = collections.Counter()\n",
|
||||
"for (label, line) in train_dataset:\n",
|
||||
" counter.update(tokenizer(line))\n",
|
||||
"vocab = torchtext.vocab.vocab(counter, min_freq=1)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"শব্দভাণ্ডার ব্যবহার করে, আমরা সহজেই আমাদের টোকেনাইজড স্ট্রিংকে সংখ্যার একটি সেটে এনকোড করতে পারি:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 19,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Vocab size if 95810\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[599, 3279, 97, 1220, 329, 225, 7368]"
|
||||
]
|
||||
},
|
||||
"execution_count": 19,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vocab_size = len(vocab)\n",
|
||||
"print(f\"Vocab size if {vocab_size}\")\n",
|
||||
"\n",
|
||||
"stoi = vocab.get_stoi() # dict to convert tokens to indices\n",
|
||||
"\n",
|
||||
"def encode(x):\n",
|
||||
" return [stoi[s] for s in tokenizer(x)]\n",
|
||||
"\n",
|
||||
"encode('I love to play with my words')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## শব্দের ব্যাগ (Bag of Words) টেক্সট উপস্থাপনা\n",
|
||||
"\n",
|
||||
"কারণ শব্দগুলো অর্থ প্রকাশ করে, কখনও কখনও আমরা শুধুমাত্র পৃথক শব্দগুলো দেখে একটি টেক্সটের অর্থ বুঝতে পারি, বাক্যে তাদের ক্রম না দেখেও। উদাহরণস্বরূপ, সংবাদ শ্রেণীবিভাগ করার সময়, *আবহাওয়া*, *তুষারপাত* এর মতো শব্দগুলো সম্ভবত *আবহাওয়ার পূর্বাভাস* নির্দেশ করবে, যেখানে *শেয়ারবাজার*, *ডলার* এর মতো শব্দগুলো *আর্থিক সংবাদ* এর দিকে ইঙ্গিত করবে।\n",
|
||||
"\n",
|
||||
"**শব্দের ব্যাগ** (BoW) ভেক্টর উপস্থাপনাটি সবচেয়ে বেশি ব্যবহৃত ঐতিহ্যবাহী ভেক্টর উপস্থাপনাগুলোর একটি। প্রতিটি শব্দ একটি ভেক্টর সূচকের সাথে যুক্ত থাকে, এবং ভেক্টরের উপাদানটি একটি নির্দিষ্ট ডকুমেন্টে সেই শব্দটির উপস্থিতির সংখ্যা ধারণ করে।\n",
|
||||
"\n",
|
||||
" \n",
|
||||
"\n",
|
||||
"> **Note**: BoW-কে আপনি টেক্সটের পৃথক শব্দগুলোর জন্য এক-হট-এনকোডেড ভেক্টরগুলোর যোগফল হিসেবেও ভাবতে পারেন।\n",
|
||||
"\n",
|
||||
"নিচে Scikit Learn পাইথন লাইব্রেরি ব্যবহার করে কীভাবে একটি শব্দের ব্যাগ উপস্থাপনা তৈরি করা যায় তার একটি উদাহরণ দেওয়া হলো:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([[1, 1, 0, 2, 0, 0, 0, 0, 0]], dtype=int64)"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from sklearn.feature_extraction.text import CountVectorizer\n",
|
||||
"vectorizer = CountVectorizer()\n",
|
||||
"corpus = [\n",
|
||||
" 'I like hot dogs.',\n",
|
||||
" 'The dog ran fast.',\n",
|
||||
" 'Its hot outside.',\n",
|
||||
" ]\n",
|
||||
"vectorizer.fit_transform(corpus)\n",
|
||||
"vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"AG_NEWS ডেটাসেটের ভেক্টর উপস্থাপন থেকে ব্যাগ-অফ-ওয়ার্ডস ভেক্টর গণনা করতে, আমরা নিম্নলিখিত ফাংশনটি ব্যবহার করতে পারি:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 20,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"tensor([2., 1., 2., ..., 0., 0., 0.])\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vocab_size = len(vocab)\n",
|
||||
"\n",
|
||||
"def to_bow(text,bow_vocab_size=vocab_size):\n",
|
||||
" res = torch.zeros(bow_vocab_size,dtype=torch.float32)\n",
|
||||
" for i in encode(text):\n",
|
||||
" if i<bow_vocab_size:\n",
|
||||
" res[i] += 1\n",
|
||||
" return res\n",
|
||||
"\n",
|
||||
"print(to_bow(train_dataset[0][1]))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **নোট:** এখানে আমরা গ্লোবাল `vocab_size` ভেরিয়েবল ব্যবহার করছি শব্দভাণ্ডারের ডিফল্ট আকার নির্ধারণ করতে। যেহেতু প্রায়শই শব্দভাণ্ডারের আকার বেশ বড় হয়, আমরা সবচেয়ে বেশি ব্যবহৃত শব্দগুলোর উপর ভিত্তি করে শব্দভাণ্ডারের আকার সীমিত করতে পারি। `vocab_size` এর মান কমিয়ে কোডটি চালিয়ে দেখুন এটি কীভাবে সঠিকতাকে প্রভাবিত করে। আপনি কিছুটা সঠিকতার হ্রাস আশা করতে পারেন, তবে তা নাটকীয় নয়, বরং উচ্চতর কার্যক্ষমতার বিনিময়ে।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## BoW শ্রেণীবিন্যাসকারী প্রশিক্ষণ\n",
|
||||
"\n",
|
||||
"এখন যেহেতু আমরা শিখেছি কীভাবে আমাদের টেক্সটের Bag-of-Words উপস্থাপনা তৈরি করতে হয়, চলুন এর উপর একটি শ্রেণীবিন্যাসকারী প্রশিক্ষণ দিই। প্রথমে, আমাদের ডেটাসেটকে এমনভাবে রূপান্তর করতে হবে যাতে সমস্ত পজিশনাল ভেক্টর উপস্থাপনাগুলি Bag-of-Words উপস্থাপনায় রূপান্তরিত হয়। এটি অর্জন করা সম্ভব `bowify` ফাংশনকে স্ট্যান্ডার্ড টর্চ `DataLoader`-এর `collate_fn` প্যারামিটার হিসেবে পাস করার মাধ্যমে:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 21,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from torch.utils.data import DataLoader\n",
|
||||
"import numpy as np \n",
|
||||
"\n",
|
||||
"# this collate function gets list of batch_size tuples, and needs to \n",
|
||||
"# return a pair of label-feature tensors for the whole minibatch\n",
|
||||
"def bowify(b):\n",
|
||||
" return (\n",
|
||||
" torch.LongTensor([t[0]-1 for t in b]),\n",
|
||||
" torch.stack([to_bow(t[1]) for t in b])\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"train_loader = DataLoader(train_dataset, batch_size=16, collate_fn=bowify, shuffle=True)\n",
|
||||
"test_loader = DataLoader(test_dataset, batch_size=16, collate_fn=bowify, shuffle=True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"এখন চলুন একটি সাধারণ শ্রেণীবিন্যাসকারী নিউরাল নেটওয়ার্ক সংজ্ঞায়িত করি যা একটি লিনিয়ার স্তর ধারণ করে। ইনপুট ভেক্টরের আকার `vocab_size` এর সমান, এবং আউটপুট আকার শ্রেণীর সংখ্যা (৪) এর সাথে সঙ্গতিপূর্ণ। যেহেতু আমরা শ্রেণীবিন্যাস কাজ সমাধান করছি, চূড়ান্ত সক্রিয়করণ ফাংশন হলো `LogSoftmax()`।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 22,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"net = torch.nn.Sequential(torch.nn.Linear(vocab_size,4),torch.nn.LogSoftmax(dim=1))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"এখন আমরা স্ট্যান্ডার্ড PyTorch প্রশিক্ষণ লুপ সংজ্ঞায়িত করব। যেহেতু আমাদের ডেটাসেটটি বেশ বড়, আমাদের শিক্ষার উদ্দেশ্যে আমরা শুধুমাত্র এক এপোকের জন্য প্রশিক্ষণ দেব, এবং কখনও কখনও এক এপোকের চেয়েও কম (প্রশিক্ষণ সীমিত করতে `epoch_size` প্যারামিটার নির্ধারণ করা হয়)। আমরা প্রশিক্ষণের সময় সংগৃহীত প্রশিক্ষণ সঠিকতা রিপোর্ট করব; রিপোর্ট করার ফ্রিকোয়েন্সি `report_freq` প্যারামিটার ব্যবহার করে নির্ধারণ করা হয়।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 24,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def train_epoch(net,dataloader,lr=0.01,optimizer=None,loss_fn = torch.nn.NLLLoss(),epoch_size=None, report_freq=200):\n",
|
||||
" optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr)\n",
|
||||
" net.train()\n",
|
||||
" total_loss,acc,count,i = 0,0,0,0\n",
|
||||
" for labels,features in dataloader:\n",
|
||||
" optimizer.zero_grad()\n",
|
||||
" out = net(features)\n",
|
||||
" loss = loss_fn(out,labels) #cross_entropy(out,labels)\n",
|
||||
" loss.backward()\n",
|
||||
" optimizer.step()\n",
|
||||
" total_loss+=loss\n",
|
||||
" _,predicted = torch.max(out,1)\n",
|
||||
" acc+=(predicted==labels).sum()\n",
|
||||
" count+=len(labels)\n",
|
||||
" i+=1\n",
|
||||
" if i%report_freq==0:\n",
|
||||
" print(f\"{count}: acc={acc.item()/count}\")\n",
|
||||
" if epoch_size and count>epoch_size:\n",
|
||||
" break\n",
|
||||
" return total_loss.item()/count, acc.item()/count"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 25,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.8028125\n",
|
||||
"6400: acc=0.8371875\n",
|
||||
"9600: acc=0.8534375\n",
|
||||
"12800: acc=0.85765625\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(0.026090790722161722, 0.8620069296375267)"
|
||||
]
|
||||
},
|
||||
"execution_count": 25,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"train_epoch(net,train_loader,epoch_size=15000)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## বাইগ্রাম, ট্রাইগ্রাম এবং এন-গ্রাম\n",
|
||||
"\n",
|
||||
"ব্যাগ অফ ওয়ার্ডস পদ্ধতির একটি সীমাবদ্ধতা হলো কিছু শব্দ বহু শব্দের অভিব্যক্তির অংশ হতে পারে। উদাহরণস্বরূপ, 'হট ডগ' শব্দটির অর্থ 'হট' এবং 'ডগ' শব্দগুলোর অন্য প্রসঙ্গে ব্যবহৃত অর্থ থেকে সম্পূর্ণ ভিন্ন। যদি আমরা 'হট' এবং 'ডগ' শব্দগুলোকে সবসময় একই ভেক্টর দিয়ে উপস্থাপন করি, তাহলে এটি আমাদের মডেলকে বিভ্রান্ত করতে পারে।\n",
|
||||
"\n",
|
||||
"এই সমস্যার সমাধানে, **এন-গ্রাম উপস্থাপনাগুলো** প্রায়ই ডকুমেন্ট শ্রেণীবিন্যাস পদ্ধতিতে ব্যবহৃত হয়, যেখানে প্রতিটি শব্দ, দ্বি-শব্দ বা ত্রি-শব্দের ফ্রিকোয়েন্সি শ্রেণীবিন্যাসকারী প্রশিক্ষণের জন্য একটি কার্যকর বৈশিষ্ট্য। উদাহরণস্বরূপ, বাইগ্রাম উপস্থাপনায়, আমরা মূল শব্দগুলোর পাশাপাশি সমস্ত শব্দ জোড়াকে ভোকাবুলারিতে যোগ করব।\n",
|
||||
"\n",
|
||||
"নিচে Scikit Learn ব্যবহার করে কীভাবে একটি বাইগ্রাম ব্যাগ অফ ওয়ার্ডস উপস্থাপনা তৈরি করা যায় তার একটি উদাহরণ দেওয়া হলো:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 26,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Vocabulary:\n",
|
||||
" {'i': 7, 'like': 11, 'hot': 4, 'dogs': 2, 'i like': 8, 'like hot': 12, 'hot dogs': 5, 'the': 16, 'dog': 0, 'ran': 14, 'fast': 3, 'the dog': 17, 'dog ran': 1, 'ran fast': 15, 'its': 9, 'outside': 13, 'its hot': 10, 'hot outside': 6}\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([[1, 0, 1, 0, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],\n",
|
||||
" dtype=int64)"
|
||||
]
|
||||
},
|
||||
"execution_count": 26,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"bigram_vectorizer = CountVectorizer(ngram_range=(1, 2), token_pattern=r'\\b\\w+\\b', min_df=1)\n",
|
||||
"corpus = [\n",
|
||||
" 'I like hot dogs.',\n",
|
||||
" 'The dog ran fast.',\n",
|
||||
" 'Its hot outside.',\n",
|
||||
" ]\n",
|
||||
"bigram_vectorizer.fit_transform(corpus)\n",
|
||||
"print(\"Vocabulary:\\n\",bigram_vectorizer.vocabulary_)\n",
|
||||
"bigram_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"N-gram পদ্ধতির প্রধান অসুবিধা হলো, শব্দভাণ্ডারের আকার অত্যন্ত দ্রুত বৃদ্ধি পেতে শুরু করে। বাস্তবে, আমাদের N-gram উপস্থাপনাকে কিছু মাত্রা হ্রাস করার কৌশলের সাথে সংযুক্ত করতে হয়, যেমন *embeddings*, যা আমরা পরবর্তী ইউনিটে আলোচনা করব।\n",
|
||||
"\n",
|
||||
"আমাদের **AG News** ডেটাসেটে N-gram উপস্থাপন ব্যবহার করতে হলে, আমাদের একটি বিশেষ ngram শব্দভাণ্ডার তৈরি করতে হবে:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 27,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Bigram vocabulary length = 1308842\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"counter = collections.Counter()\n",
|
||||
"for (label, line) in train_dataset:\n",
|
||||
" l = tokenizer(line)\n",
|
||||
" counter.update(torchtext.data.utils.ngrams_iterator(l,ngrams=2))\n",
|
||||
" \n",
|
||||
"bi_vocab = torchtext.vocab.vocab(counter, min_freq=1)\n",
|
||||
"\n",
|
||||
"print(\"Bigram vocabulary length = \",len(bi_vocab))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"আমরা উপরের কোডটি ব্যবহার করে ক্লাসিফায়ার প্রশিক্ষণ দিতে পারতাম, তবে এটি খুবই মেমোরি-অকার্যকর হতো। পরবর্তী ইউনিটে, আমরা এম্বেডিং ব্যবহার করে বিগ্রাম ক্লাসিফায়ার প্রশিক্ষণ দেব।\n",
|
||||
"\n",
|
||||
"> **Note:** আপনি শুধুমাত্র সেই ngrams রাখতে পারেন যা নির্দিষ্ট সংখ্যার চেয়ে বেশি বার টেক্সটে উপস্থিত হয়। এটি নিশ্চিত করবে যে কম ঘনঘন বিগ্রামগুলো বাদ যাবে এবং ডাইমেনশনালিটি উল্লেখযোগ্যভাবে কমে যাবে। এটি করতে, `min_freq` প্যারামিটারকে একটি উচ্চতর মানে সেট করুন এবং ভোকাবুলারির দৈর্ঘ্যের পরিবর্তন পর্যবেক্ষণ করুন।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## টার্ম ফ্রিকোয়েন্সি ইনভার্স ডকুমেন্ট ফ্রিকোয়েন্সি (TF-IDF)\n",
|
||||
"\n",
|
||||
"BoW উপস্থাপনায়, শব্দের উপস্থিতি সমানভাবে ওজন দেওয়া হয়, শব্দটি কী তা বিবেচনা না করেই। তবে, এটা স্পষ্ট যে সাধারণ শব্দ যেমন *a*, *in* ইত্যাদি শ্রেণীবিন্যাসের জন্য বিশেষায়িত শব্দগুলোর তুলনায় অনেক কম গুরুত্বপূর্ণ। প্রকৃতপক্ষে, বেশিরভাগ NLP কাজের ক্ষেত্রে কিছু শব্দ অন্যগুলোর তুলনায় বেশি প্রাসঙ্গিক।\n",
|
||||
"\n",
|
||||
"**TF-IDF** এর পূর্ণরূপ হলো **টার্ম ফ্রিকোয়েন্সি–ইনভার্স ডকুমেন্ট ফ্রিকোয়েন্সি**। এটি ব্যাগ অফ ওয়ার্ডসের একটি পরিবর্তিত রূপ, যেখানে একটি শব্দের উপস্থিতি নির্দেশ করতে 0/1 বাইনারি মানের পরিবর্তে একটি ভাসমান-বিন্দু মান ব্যবহার করা হয়, যা কর্পাসে শব্দের উপস্থিতির ফ্রিকোয়েন্সির সাথে সম্পর্কিত।\n",
|
||||
"\n",
|
||||
"আরও আনুষ্ঠানিকভাবে, একটি শব্দ $i$ এর ওজন $w_{ij}$ একটি ডকুমেন্ট $j$ এ সংজ্ঞায়িত করা হয়:\n",
|
||||
"$$\n",
|
||||
"w_{ij} = tf_{ij}\\times\\log({N\\over df_i})\n",
|
||||
"$$\n",
|
||||
"যেখানে\n",
|
||||
"* $tf_{ij}$ হলো $i$ শব্দটি $j$ ডকুমেন্টে যতবার উপস্থিত হয়েছে, অর্থাৎ আমরা আগে যে BoW মান দেখেছি\n",
|
||||
"* $N$ হলো সংগ্রহে থাকা ডকুমেন্টের সংখ্যা\n",
|
||||
"* $df_i$ হলো পুরো সংগ্রহে $i$ শব্দটি উপস্থিত থাকা ডকুমেন্টের সংখ্যা\n",
|
||||
"\n",
|
||||
"TF-IDF মান $w_{ij}$ একটি ডকুমেন্টে শব্দটি যতবার উপস্থিত হয় তার সাথে আনুপাতিকভাবে বৃদ্ধি পায় এবং কর্পাসে শব্দটি উপস্থিত থাকা ডকুমেন্টের সংখ্যার দ্বারা সামঞ্জস্য করা হয়, যা এই বিষয়টি সমাধান করতে সাহায্য করে যে কিছু শব্দ অন্যগুলোর তুলনায় বেশি ঘন ঘন উপস্থিত হয়। উদাহরণস্বরূপ, যদি একটি শব্দ *প্রতিটি* ডকুমেন্টে উপস্থিত থাকে, $df_i=N$, এবং $w_{ij}=0$, এবং সেই শব্দগুলো সম্পূর্ণভাবে উপেক্ষা করা হবে।\n",
|
||||
"\n",
|
||||
"আপনি সহজেই Scikit Learn ব্যবহার করে টেক্সটের TF-IDF ভেক্টরাইজেশন তৈরি করতে পারেন:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 28,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([[0.43381609, 0. , 0.43381609, 0. , 0.65985664,\n",
|
||||
" 0.43381609, 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. ]])"
|
||||
]
|
||||
},
|
||||
"execution_count": 28,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
|
||||
"vectorizer = TfidfVectorizer(ngram_range=(1,2))\n",
|
||||
"vectorizer.fit_transform(corpus)\n",
|
||||
"vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## উপসংহার\n",
|
||||
"\n",
|
||||
"যদিও TF-IDF উপস্থাপনাগুলি বিভিন্ন শব্দের জন্য ফ্রিকোয়েন্সি ওজন প্রদান করে, সেগুলি অর্থ বা ক্রম উপস্থাপন করতে অক্ষম। ১৯৩৫ সালে বিখ্যাত ভাষাবিদ জে. আর. ফার্থ বলেছিলেন, \"একটি শব্দের সম্পূর্ণ অর্থ সর্বদা প্রাসঙ্গিক, এবং প্রসঙ্গ ছাড়া অর্থের কোনো অধ্যয়নকে গুরুত্ব সহকারে নেওয়া যায় না।\" আমরা এই কোর্সে পরে শিখব কীভাবে ভাষা মডেলিং ব্যবহার করে পাঠ্য থেকে প্রাসঙ্গিক তথ্য ধারণ করা যায়।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**অস্বীকৃতি**: \nএই নথিটি AI অনুবাদ পরিষেবা [Co-op Translator](https://github.com/Azure/co-op-translator) ব্যবহার করে অনুবাদ করা হয়েছে। আমরা যথাসম্ভব সঠিকতার জন্য চেষ্টা করি, তবে অনুগ্রহ করে মনে রাখবেন যে স্বয়ংক্রিয় অনুবাদে ত্রুটি বা অসঙ্গতি থাকতে পারে। মূল ভাষায় থাকা নথিটিকে প্রামাণিক উৎস হিসেবে বিবেচনা করা উচিত। গুরুত্বপূর্ণ তথ্যের জন্য, পেশাদার মানব অনুবাদ সুপারিশ করা হয়। এই অনুবাদ ব্যবহারের ফলে কোনো ভুল বোঝাবুঝি বা ভুল ব্যাখ্যা হলে আমরা তার জন্য দায়ী থাকব না।\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.8.12 ('py38')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "7b9040985e748e4e2d4c689892456ad7",
|
||||
"translation_date": "2025-08-28T12:30:59+00:00",
|
||||
"source_file": "lessons/5-NLP/13-TextRep/TextRepresentationPyTorch.ipynb",
|
||||
"language_code": "bn"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
|
|
@ -0,0 +1,647 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# টেক্সট শ্রেণীবিন্যাস কাজ\n",
|
||||
"\n",
|
||||
"এই মডিউলে, আমরা একটি সহজ টেক্সট শ্রেণীবিন্যাস কাজ দিয়ে শুরু করব যা **[AG_NEWS](http://www.di.unipi.it/~gulli/AG_corpus_of_news_articles.html)** ডেটাসেটের উপর ভিত্তি করে তৈরি: আমরা সংবাদ শিরোনামগুলোকে চারটি বিভাগে শ্রেণীবদ্ধ করব: বিশ্ব, খেলাধুলা, ব্যবসা এবং বিজ্ঞান/প্রযুক্তি।\n",
|
||||
"\n",
|
||||
"## ডেটাসেট\n",
|
||||
"\n",
|
||||
"ডেটাসেট লোড করার জন্য, আমরা **[TensorFlow Datasets](https://www.tensorflow.org/datasets)** API ব্যবহার করব।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import tensorflow as tf\n",
|
||||
"from tensorflow import keras\n",
|
||||
"import tensorflow_datasets as tfds\n",
|
||||
"\n",
|
||||
"# In this tutorial, we will be training a lot of models. In order to use GPU memory cautiously,\n",
|
||||
"# we will set tensorflow option to grow GPU memory allocation when required.\n",
|
||||
"physical_devices = tf.config.list_physical_devices('GPU') \n",
|
||||
"if len(physical_devices)>0:\n",
|
||||
" tf.config.experimental.set_memory_growth(physical_devices[0], True)\n",
|
||||
"\n",
|
||||
"dataset = tfds.load('ag_news_subset')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"আমরা এখন `dataset['train']` এবং `dataset['test']` ব্যবহার করে ডেটাসেটের প্রশিক্ষণ এবং পরীক্ষার অংশগুলিতে প্রবেশ করতে পারি:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Length of train dataset = 120000\n",
|
||||
"Length of test dataset = 7600\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"ds_train = dataset['train']\n",
|
||||
"ds_test = dataset['test']\n",
|
||||
"\n",
|
||||
"print(f\"Length of train dataset = {len(ds_train)}\")\n",
|
||||
"print(f\"Length of test dataset = {len(ds_test)}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"চলুন আমাদের ডেটাসেট থেকে প্রথম ১০টি নতুন শিরোনাম মুদ্রণ করি:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3 (Sci/Tech) -> b'AMD Debuts Dual-Core Opteron Processor' b'AMD #39;s new dual-core Opteron chip is designed mainly for corporate computing applications, including databases, Web services, and financial transactions.'\n",
|
||||
"1 (Sports) -> b\"Wood's Suspension Upheld (Reuters)\" b'Reuters - Major League Baseball\\\\Monday announced a decision on the appeal filed by Chicago Cubs\\\\pitcher Kerry Wood regarding a suspension stemming from an\\\\incident earlier this season.'\n",
|
||||
"2 (Business) -> b'Bush reform may have blue states seeing red' b'President Bush #39;s quot;revenue-neutral quot; tax reform needs losers to balance its winners, and people claiming the federal deduction for state and local taxes may be in administration planners #39; sights, news reports say.'\n",
|
||||
"3 (Sci/Tech) -> b\"'Halt science decline in schools'\" b'Britain will run out of leading scientists unless science education is improved, says Professor Colin Pillinger.'\n",
|
||||
"1 (Sports) -> b'Gerrard leaves practice' b'London, England (Sports Network) - England midfielder Steven Gerrard injured his groin late in Thursday #39;s training session, but is hopeful he will be ready for Saturday #39;s World Cup qualifier against Austria.'\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"classes = ['World', 'Sports', 'Business', 'Sci/Tech']\n",
|
||||
"\n",
|
||||
"for i,x in zip(range(5),ds_train):\n",
|
||||
" print(f\"{x['label']} ({classes[x['label']]}) -> {x['title']} {x['description']}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## টেক্সট ভেক্টরাইজেশন\n",
|
||||
"\n",
|
||||
"এখন আমাদের টেক্সটকে **সংখ্যায়** রূপান্তর করতে হবে যা টেনসর হিসেবে উপস্থাপন করা যায়। যদি আমরা শব্দ-স্তরের উপস্থাপন চাই, তাহলে আমাদের দুটি কাজ করতে হবে:\n",
|
||||
"\n",
|
||||
"* একটি **টোকেনাইজার** ব্যবহার করে টেক্সটকে **টোকেন**-এ ভাগ করতে হবে।\n",
|
||||
"* সেই টোকেনগুলোর একটি **ভোকাবুলারি** তৈরি করতে হবে।\n",
|
||||
"\n",
|
||||
"### ভোকাবুলারির আকার সীমিত করা\n",
|
||||
"\n",
|
||||
"AG News ডেটাসেট উদাহরণে, ভোকাবুলারির আকার বেশ বড়, ১০০k এর বেশি শব্দ। সাধারণভাবে বলতে গেলে, আমাদের এমন শব্দের প্রয়োজন নেই যা টেক্সটে খুব কম উপস্থিত থাকে — শুধুমাত্র কয়েকটি বাক্যে সেগুলো থাকবে, এবং মডেল সেগুলো থেকে কিছু শিখতে পারবে না। তাই, ভোকাবুলারির আকার একটি ছোট সংখ্যায় সীমিত করা যুক্তিযুক্ত, যা ভেক্টরাইজার কনস্ট্রাক্টরে একটি আর্গুমেন্ট পাস করে করা যায়:\n",
|
||||
"\n",
|
||||
"এই দুটি ধাপই **TextVectorization** লেয়ার ব্যবহার করে সম্পন্ন করা যেতে পারে। চলুন ভেক্টরাইজার অবজেক্টটি ইনস্ট্যানশিয়েট করি, এবং তারপর `adapt` মেথড কল করে সমস্ত টেক্সটের মধ্য দিয়ে যাই এবং একটি ভোকাবুলারি তৈরি করি:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"vocab_size = 50000\n",
|
||||
"vectorizer = keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size)\n",
|
||||
"vectorizer.adapt(ds_train.take(500).map(lambda x: x['title']+' '+x['description']))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **নোট করুন** যে আমরা পুরো ডেটাসেটের শুধুমাত্র একটি অংশ ব্যবহার করছি একটি শব্দভাণ্ডার তৈরি করার জন্য। আমরা এটি করছি কার্যকরী সময় দ্রুত করার জন্য এবং আপনাকে অপেক্ষা না করানোর জন্য। তবে, আমরা এই ঝুঁকি নিচ্ছি যে পুরো ডেটাসেটের কিছু শব্দ শব্দভাণ্ডারে অন্তর্ভুক্ত হবে না এবং প্রশিক্ষণের সময় উপেক্ষা করা হবে। সুতরাং, পুরো শব্দভাণ্ডারের আকার ব্যবহার করা এবং `adapt` চলাকালীন পুরো ডেটাসেটের মধ্য দিয়ে যাওয়া চূড়ান্ত সঠিকতা বৃদ্ধি করতে পারে, তবে খুব বেশি নয়।\n",
|
||||
"\n",
|
||||
"এখন আমরা প্রকৃত শব্দভাণ্ডারে প্রবেশ করতে পারি:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"['', '[UNK]', 'the', 'to', 'a', 'in', 'of', 'and', 'on', 'for']\n",
|
||||
"Length of vocabulary: 5335\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vocab = vectorizer.get_vocabulary()\n",
|
||||
"vocab_size = len(vocab)\n",
|
||||
"print(vocab[:10])\n",
|
||||
"print(f\"Length of vocabulary: {vocab_size}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"ভেক্টরাইজার ব্যবহার করে, আমরা সহজেই যেকোনো টেক্সটকে সংখ্যার সেটে এনকোড করতে পারি:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tf.Tensor: shape=(7,), dtype=int64, numpy=array([ 112, 3695, 3, 304, 11, 1041, 1], dtype=int64)>"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vectorizer('I love to play with my words')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## ব্যাগ-অফ-ওয়ার্ডস টেক্সট উপস্থাপন\n",
|
||||
"\n",
|
||||
"যেহেতু শব্দ অর্থ প্রকাশ করে, কখনও কখনও আমরা একটি টেক্সটের অর্থ শুধুমাত্র পৃথক শব্দগুলো দেখে বুঝতে পারি, বাক্যে তাদের ক্রম নির্বিশেষে। উদাহরণস্বরূপ, সংবাদ শ্রেণীবিন্যাস করার সময়, *আবহাওয়া* এবং *তুষার* এর মতো শব্দগুলো *আবহাওয়ার পূর্বাভাস* নির্দেশ করতে পারে, যেখানে *স্টক* এবং *ডলার* এর মতো শব্দগুলো *আর্থিক সংবাদ* এর দিকে ইঙ্গিত করবে।\n",
|
||||
"\n",
|
||||
"**ব্যাগ-অফ-ওয়ার্ডস** (BoW) ভেক্টর উপস্থাপন হল সবচেয়ে সহজে বোঝা যায় এমন ঐতিহ্যবাহী ভেক্টর উপস্থাপন। প্রতিটি শব্দ একটি ভেক্টর সূচকের সাথে যুক্ত থাকে, এবং একটি ভেক্টর উপাদান একটি নির্দিষ্ট ডকুমেন্টে প্রতিটি শব্দের উপস্থিতির সংখ্যা ধারণ করে।\n",
|
||||
"\n",
|
||||
" \n",
|
||||
"\n",
|
||||
"> **Note**: BoW কে আপনি টেক্সটের পৃথক শব্দগুলোর জন্য সমস্ত এক-হট-এনকোডেড ভেক্টরের যোগফল হিসেবেও ভাবতে পারেন।\n",
|
||||
"\n",
|
||||
"নিচে Scikit Learn পাইথন লাইব্রেরি ব্যবহার করে ব্যাগ-অফ-ওয়ার্ডস উপস্থাপন তৈরি করার একটি উদাহরণ দেওয়া হলো:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([[1, 1, 0, 2, 0, 0, 0, 0, 0]], dtype=int64)"
|
||||
]
|
||||
},
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from sklearn.feature_extraction.text import CountVectorizer\n",
|
||||
"sc_vectorizer = CountVectorizer()\n",
|
||||
"corpus = [\n",
|
||||
" 'I like hot dogs.',\n",
|
||||
" 'The dog ran fast.',\n",
|
||||
" 'Its hot outside.',\n",
|
||||
" ]\n",
|
||||
"sc_vectorizer.fit_transform(corpus)\n",
|
||||
"sc_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"আমরা উপরে সংজ্ঞায়িত করা কেরাস ভেক্টরাইজারও ব্যবহার করতে পারি, প্রতিটি শব্দ সংখ্যাকে এক-হট এনকোডিংয়ে রূপান্তরিত করে এবং সেই সমস্ত ভেক্টর যোগ করে:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([0., 5., 0., ..., 0., 0., 0.], dtype=float32)"
|
||||
]
|
||||
},
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def to_bow(text):\n",
|
||||
" return tf.reduce_sum(tf.one_hot(vectorizer(text),vocab_size),axis=0)\n",
|
||||
"\n",
|
||||
"to_bow('My dog likes hot dogs on a hot day.').numpy()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **নোট**: আপনি হয়তো অবাক হবেন যে ফলাফলটি আগের উদাহরণের থেকে আলাদা। এর কারণ হলো, Keras উদাহরণে ভেক্টরের দৈর্ঘ্য শব্দভাণ্ডারের আকারের সাথে মিলে যায়, যা পুরো AG News ডেটাসেট থেকে তৈরি করা হয়েছিল, যেখানে Scikit Learn উদাহরণে আমরা নমুনা টেক্সট থেকে তাৎক্ষণিকভাবে শব্দভাণ্ডার তৈরি করেছি।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## BoW শ্রেণীবিভাজক প্রশিক্ষণ\n",
|
||||
"\n",
|
||||
"এখন আমরা শিখেছি কীভাবে আমাদের টেক্সটের ব্যাগ-অফ-ওয়ার্ডস উপস্থাপন তৈরি করতে হয়, চলুন একটি শ্রেণীবিভাজক প্রশিক্ষণ করি যা এটি ব্যবহার করে। প্রথমে, আমাদের ডেটাসেটকে ব্যাগ-অফ-ওয়ার্ডস উপস্থাপনে রূপান্তর করতে হবে। এটি নিম্নলিখিতভাবে `map` ফাংশন ব্যবহার করে করা যেতে পারে:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"batch_size = 128\n",
|
||||
"\n",
|
||||
"ds_train_bow = ds_train.map(lambda x: (to_bow(x['title']+x['description']),x['label'])).batch(batch_size)\n",
|
||||
"ds_test_bow = ds_test.map(lambda x: (to_bow(x['title']+x['description']),x['label'])).batch(batch_size)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"এখন চলুন একটি সাধারণ শ্রেণীবিন্যাসকারী নিউরাল নেটওয়ার্ক সংজ্ঞায়িত করি যা একটি লিনিয়ার স্তর ধারণ করে। ইনপুট সাইজ হলো `vocab_size`, এবং আউটপুট সাইজ শ্রেণীর সংখ্যা (৪) এর সাথে সম্পর্কিত। যেহেতু আমরা একটি শ্রেণীবিন্যাস কাজ সমাধান করছি, চূড়ান্ত সক্রিয়করণ ফাংশন হলো **softmax**:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"938/938 [==============================] - 66s 70ms/step - loss: 0.6144 - acc: 0.8427 - val_loss: 0.4416 - val_acc: 0.8697\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.History at 0x20c70a947f0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model = keras.models.Sequential([\n",
|
||||
" keras.layers.Dense(4,activation='softmax',input_shape=(vocab_size,))\n",
|
||||
"])\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
|
||||
"model.fit(ds_train_bow,validation_data=ds_test_bow)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"যেহেতু আমাদের ৪টি ক্লাস রয়েছে, ৮০% এর উপরে সঠিকতা একটি ভালো ফলাফল।\n",
|
||||
"\n",
|
||||
"## একটি নেটওয়ার্ক হিসেবে ক্লাসিফায়ার প্রশিক্ষণ\n",
|
||||
"\n",
|
||||
"কারণ ভেক্টরাইজারটি একটি Keras লেয়ারও, আমরা একটি নেটওয়ার্ক সংজ্ঞায়িত করতে পারি যা এটিকে অন্তর্ভুক্ত করে এবং এটিকে সম্পূর্ণভাবে প্রশিক্ষণ দিতে পারি। এইভাবে, আমাদের `map` ব্যবহার করে ডেটাসেট ভেক্টরাইজ করার প্রয়োজন নেই, আমরা শুধু মূল ডেটাসেটটি নেটওয়ার্কের ইনপুটে পাস করতে পারি।\n",
|
||||
"\n",
|
||||
"> **Note**: আমাদের এখনও আমাদের ডেটাসেটে `map` প্রয়োগ করতে হবে যাতে ডিকশনারি থেকে ক্ষেত্রগুলো (যেমন `title`, `description` এবং `label`) টুপলে রূপান্তরিত করা যায়। তবে, যখন ডিস্ক থেকে ডেটা লোড করা হয়, তখন আমরা প্রথম থেকেই প্রয়োজনীয় কাঠামো সহ একটি ডেটাসেট তৈরি করতে পারি।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"model\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
" Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
" input_1 (InputLayer) [(None, 1)] 0 \n",
|
||||
" \n",
|
||||
" text_vectorization (TextVec (None, None) 0 \n",
|
||||
" torization) \n",
|
||||
" \n",
|
||||
" tf.one_hot (TFOpLambda) (None, None, 5335) 0 \n",
|
||||
" \n",
|
||||
" tf.math.reduce_sum (TFOpLam (None, 5335) 0 \n",
|
||||
" bda) \n",
|
||||
" \n",
|
||||
" dense_2 (Dense) (None, 4) 21344 \n",
|
||||
" \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 21,344\n",
|
||||
"Trainable params: 21,344\n",
|
||||
"Non-trainable params: 0\n",
|
||||
"_________________________________________________________________\n",
|
||||
"938/938 [==============================] - 73s 77ms/step - loss: 0.6057 - acc: 0.8414 - val_loss: 0.4202 - val_acc: 0.8736\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.History at 0x20c721521f0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def extract_text(x):\n",
|
||||
" return x['title']+' '+x['description']\n",
|
||||
"\n",
|
||||
"def tupelize(x):\n",
|
||||
" return (extract_text(x),x['label'])\n",
|
||||
"\n",
|
||||
"inp = keras.Input(shape=(1,),dtype=tf.string)\n",
|
||||
"x = vectorizer(inp)\n",
|
||||
"x = tf.reduce_sum(tf.one_hot(x,vocab_size),axis=1)\n",
|
||||
"out = keras.layers.Dense(4,activation='softmax')(x)\n",
|
||||
"model = keras.models.Model(inp,out)\n",
|
||||
"model.summary()\n",
|
||||
"\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## বাইগ্রাম, ট্রাইগ্রাম এবং এন-গ্রাম\n",
|
||||
"\n",
|
||||
"ব্যাগ-অফ-ওয়ার্ড পদ্ধতির একটি সীমাবদ্ধতা হলো কিছু শব্দ বহু-শব্দের অভিব্যক্তির অংশ, যেমন 'হট ডগ' শব্দটি 'হট' এবং 'ডগ' শব্দের থেকে সম্পূর্ণ ভিন্ন অর্থ বহন করে। যদি আমরা 'হট' এবং 'ডগ' শব্দগুলো সবসময় একই ভেক্টর দিয়ে উপস্থাপন করি, তাহলে এটি আমাদের মডেলকে বিভ্রান্ত করতে পারে।\n",
|
||||
"\n",
|
||||
"এই সমস্যার সমাধানে, **এন-গ্রাম উপস্থাপনাগুলো** প্রায়ই ডকুমেন্ট শ্রেণীবিন্যাস পদ্ধতিতে ব্যবহৃত হয়, যেখানে প্রতিটি শব্দ, দ্বি-শব্দ বা ত্রি-শব্দের ফ্রিকোয়েন্সি শ্রেণীবিন্যাসকারী প্রশিক্ষণের জন্য একটি কার্যকর বৈশিষ্ট্য হিসেবে কাজ করে। উদাহরণস্বরূপ, বাইগ্রাম উপস্থাপনায় আমরা মূল শব্দগুলোর পাশাপাশি সমস্ত শব্দ জোড়াকে শব্দভাণ্ডারে যোগ করব।\n",
|
||||
"\n",
|
||||
"নিচে Scikit Learn ব্যবহার করে একটি বাইগ্রাম ব্যাগ-অফ-ওয়ার্ড উপস্থাপনা তৈরি করার উদাহরণ দেওয়া হলো:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Vocabulary:\n",
|
||||
" {'i': 7, 'like': 11, 'hot': 4, 'dogs': 2, 'i like': 8, 'like hot': 12, 'hot dogs': 5, 'the': 16, 'dog': 0, 'ran': 14, 'fast': 3, 'the dog': 17, 'dog ran': 1, 'ran fast': 15, 'its': 9, 'outside': 13, 'its hot': 10, 'hot outside': 6}\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([[1, 0, 1, 0, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],\n",
|
||||
" dtype=int64)"
|
||||
]
|
||||
},
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"bigram_vectorizer = CountVectorizer(ngram_range=(1, 2), token_pattern=r'\\b\\w+\\b', min_df=1)\n",
|
||||
"corpus = [\n",
|
||||
" 'I like hot dogs.',\n",
|
||||
" 'The dog ran fast.',\n",
|
||||
" 'Its hot outside.',\n",
|
||||
" ]\n",
|
||||
"bigram_vectorizer.fit_transform(corpus)\n",
|
||||
"print(\"Vocabulary:\\n\",bigram_vectorizer.vocabulary_)\n",
|
||||
"bigram_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"n-gram পদ্ধতির প্রধান অসুবিধা হলো এর শব্দভাণ্ডারের আকার অত্যন্ত দ্রুত বৃদ্ধি পেতে থাকে। বাস্তবে, আমাদের n-gram উপস্থাপনাকে একটি মাত্রা হ্রাস করার কৌশলের সাথে সংযুক্ত করতে হয়, যেমন *embeddings*, যা আমরা পরবর্তী ইউনিটে আলোচনা করব।\n",
|
||||
"\n",
|
||||
"আমাদের **AG News** ডেটাসেটে n-gram উপস্থাপন ব্যবহার করতে হলে, আমাদের `TextVectorization` কনস্ট্রাক্টরে `ngrams` প্যারামিটারটি পাস করতে হবে। একটি বিগ্রামের শব্দভাণ্ডারের দৈর্ঘ্য **অত্যন্ত বড়**, আমাদের ক্ষেত্রে এটি ১.৩ মিলিয়নেরও বেশি টোকেন! তাই এটি যৌক্তিক যে বিগ্রাম টোকেনগুলোকেও একটি যুক্তিসঙ্গত সংখ্যায় সীমাবদ্ধ করা উচিত।\n",
|
||||
"\n",
|
||||
"আমরা উপরের কোডটি ব্যবহার করে ক্লাসিফায়ার প্রশিক্ষণ দিতে পারি, তবে এটি খুবই মেমোরি-অকার্যকর হবে। পরবর্তী ইউনিটে, আমরা embeddings ব্যবহার করে বিগ্রাম ক্লাসিফায়ার প্রশিক্ষণ দেব। এর মধ্যে, আপনি এই নোটবুকে বিগ্রাম ক্লাসিফায়ার প্রশিক্ষণের সাথে পরীক্ষা-নিরীক্ষা করতে পারেন এবং দেখুন আপনি কি উচ্চতর সঠিকতা অর্জন করতে পারেন।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## স্বয়ংক্রিয়ভাবে BoW ভেক্টর গণনা করা\n",
|
||||
"\n",
|
||||
"উপরের উদাহরণে আমরা একক শব্দের এক-হট এনকোডিং যোগ করে হাতে BoW ভেক্টর গণনা করেছি। তবে, TensorFlow-এর সর্বশেষ সংস্করণ আমাদেরকে `output_mode='count` প্যারামিটার ভেক্টরাইজার কনস্ট্রাক্টরে পাস করে স্বয়ংক্রিয়ভাবে BoW ভেক্টর গণনা করার সুযোগ দেয়। এটি আমাদের মডেল সংজ্ঞায়িত এবং প্রশিক্ষণ দেওয়া উল্লেখযোগ্যভাবে সহজ করে তোলে:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Training vectorizer\n",
|
||||
"938/938 [==============================] - 7s 7ms/step - loss: 0.5929 - acc: 0.8486 - val_loss: 0.4168 - val_acc: 0.8772\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.History at 0x20c725217c0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model = keras.models.Sequential([\n",
|
||||
" keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size,output_mode='count'),\n",
|
||||
" keras.layers.Dense(4,input_shape=(vocab_size,), activation='softmax')\n",
|
||||
"])\n",
|
||||
"print(\"Training vectorizer\")\n",
|
||||
"model.layers[0].adapt(ds_train.take(500).map(extract_text))\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## টার্ম ফ্রিকোয়েন্সি - ইনভার্স ডকুমেন্ট ফ্রিকোয়েন্সি (TF-IDF)\n",
|
||||
"\n",
|
||||
"BoW উপস্থাপনায়, শব্দের উপস্থিতি একই পদ্ধতি ব্যবহার করে ওজন দেওয়া হয়, শব্দটি যাই হোক না কেন। তবে, এটা স্পষ্ট যে *a* এবং *in* এর মতো সাধারণ শব্দগুলো শ্রেণীবিন্যাসের জন্য বিশেষায়িত শব্দগুলোর তুলনায় অনেক কম গুরুত্বপূর্ণ। বেশিরভাগ NLP কাজের ক্ষেত্রে কিছু শব্দ অন্যদের তুলনায় বেশি প্রাসঙ্গিক।\n",
|
||||
"\n",
|
||||
"**TF-IDF** এর পূর্ণরূপ হলো **টার্ম ফ্রিকোয়েন্সি - ইনভার্স ডকুমেন্ট ফ্রিকোয়েন্সি**। এটি ব্যাগ-অফ-ওয়ার্ডসের একটি পরিবর্তিত রূপ, যেখানে একটি শব্দের উপস্থিতি একটি ডকুমেন্টে 0/1 বাইনারি মান দিয়ে নির্দেশ করার পরিবর্তে একটি ফ্লোটিং-পয়েন্ট মান ব্যবহার করা হয়, যা কর্পাসে শব্দের উপস্থিতির ফ্রিকোয়েন্সির সাথে সম্পর্কিত।\n",
|
||||
"\n",
|
||||
"আরও আনুষ্ঠানিকভাবে, একটি শব্দ $i$ এর ওজন $w_{ij}$ একটি ডকুমেন্ট $j$ এ সংজ্ঞায়িত করা হয়:\n",
|
||||
"$$\n",
|
||||
"w_{ij} = tf_{ij}\\times\\log({N\\over df_i})\n",
|
||||
"$$\n",
|
||||
"যেখানে\n",
|
||||
"* $tf_{ij}$ হলো $i$ এর $j$ এ উপস্থিতির সংখ্যা, অর্থাৎ আমরা আগে যে BoW মান দেখেছি\n",
|
||||
"* $N$ হলো সংগ্রহে থাকা ডকুমেন্টের সংখ্যা\n",
|
||||
"* $df_i$ হলো পুরো সংগ্রহে শব্দ $i$ ধারণকারী ডকুমেন্টের সংখ্যা\n",
|
||||
"\n",
|
||||
"TF-IDF মান $w_{ij}$ একটি ডকুমেন্টে শব্দের উপস্থিতির সংখ্যার সাথে আনুপাতিকভাবে বৃদ্ধি পায় এবং কর্পাসে শব্দটি ধারণকারী ডকুমেন্টের সংখ্যার দ্বারা সামঞ্জস্য করা হয়, যা এই বিষয়টি সমন্বয় করতে সাহায্য করে যে কিছু শব্দ অন্যদের তুলনায় বেশি ঘন ঘন উপস্থিত হয়। উদাহরণস্বরূপ, যদি একটি শব্দ সংগ্রহের *প্রতিটি* ডকুমেন্টে উপস্থিত হয়, $df_i=N$, এবং $w_{ij}=0$, এবং সেই শব্দগুলো সম্পূর্ণভাবে উপেক্ষা করা হবে।\n",
|
||||
"\n",
|
||||
"আপনি সহজেই Scikit Learn ব্যবহার করে টেক্সটের TF-IDF ভেক্টরাইজেশন তৈরি করতে পারেন:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 16,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([[0.43381609, 0. , 0.43381609, 0. , 0.65985664,\n",
|
||||
" 0.43381609, 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. ]])"
|
||||
]
|
||||
},
|
||||
"execution_count": 16,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
|
||||
"vectorizer = TfidfVectorizer(ngram_range=(1,2))\n",
|
||||
"vectorizer.fit_transform(corpus)\n",
|
||||
"vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Keras-এ, `TextVectorization` স্তরটি `output_mode='tf-idf'` প্যারামিটার পাস করে স্বয়ংক্রিয়ভাবে TF-IDF ফ্রিকোয়েন্সি গণনা করতে পারে। চলুন উপরের কোডটি পুনরাবৃত্তি করি এবং দেখি TF-IDF ব্যবহার করলে সঠিকতা বৃদ্ধি পায় কিনা:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 17,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Training vectorizer\n",
|
||||
"938/938 [==============================] - 12s 12ms/step - loss: 0.4197 - acc: 0.8662 - val_loss: 0.3432 - val_acc: 0.8849\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.History at 0x20c729dfd30>"
|
||||
]
|
||||
},
|
||||
"execution_count": 17,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model = keras.models.Sequential([\n",
|
||||
" keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size,output_mode='tf-idf'),\n",
|
||||
" keras.layers.Dense(4,input_shape=(vocab_size,), activation='softmax')\n",
|
||||
"])\n",
|
||||
"print(\"Training vectorizer\")\n",
|
||||
"model.layers[0].adapt(ds_train.take(500).map(extract_text))\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## উপসংহার\n",
|
||||
"\n",
|
||||
"যদিও TF-IDF উপস্থাপনাগুলি বিভিন্ন শব্দের জন্য ফ্রিকোয়েন্সি ওজন প্রদান করে, সেগুলি অর্থ বা ক্রম উপস্থাপন করতে অক্ষম। ১৯৩৫ সালে বিখ্যাত ভাষাবিদ জে. আর. ফার্থ বলেছিলেন, \"একটি শব্দের সম্পূর্ণ অর্থ সর্বদা প্রাসঙ্গিক, এবং প্রসঙ্গ ছাড়া অর্থের কোনো অধ্যয়নকে গুরুত্ব সহকারে নেওয়া যায় না।\" আমরা এই কোর্সের পরবর্তী অংশে ভাষা মডেলিং ব্যবহার করে পাঠ্য থেকে প্রাসঙ্গিক তথ্য কীভাবে ধারণ করা যায় তা শিখব।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**অস্বীকৃতি**: \nএই নথিটি AI অনুবাদ পরিষেবা [Co-op Translator](https://github.com/Azure/co-op-translator) ব্যবহার করে অনুবাদ করা হয়েছে। আমরা যথাসাধ্য সঠিকতা নিশ্চিত করার চেষ্টা করি, তবে অনুগ্রহ করে মনে রাখবেন যে স্বয়ংক্রিয় অনুবাদে ত্রুটি বা অসঙ্গতি থাকতে পারে। মূল ভাষায় থাকা নথিটিকে প্রামাণিক উৎস হিসেবে বিবেচনা করা উচিত। গুরুত্বপূর্ণ তথ্যের জন্য, পেশাদার মানব অনুবাদ সুপারিশ করা হয়। এই অনুবাদ ব্যবহারের ফলে কোনো ভুল বোঝাবুঝি বা ভুল ব্যাখ্যা হলে আমরা দায়বদ্ধ থাকব না।\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5"
|
||||
},
|
||||
"kernel_info": {
|
||||
"name": "conda-env-py37_tensorflow-py"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "py37_tensorflow",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"nteract": {
|
||||
"version": "nteract-front-end@1.0.0"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "19b43951d55b377a76209c24c1f017e4",
|
||||
"translation_date": "2025-08-28T12:33:47+00:00",
|
||||
"source_file": "lessons/5-NLP/13-TextRep/TextRepresentationTF.ipynb",
|
||||
"language_code": "bn"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
|
|
@ -0,0 +1,726 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## এম্বেডিংস\n",
|
||||
"\n",
|
||||
"আমাদের আগের উদাহরণে, আমরা উচ্চ-মাত্রার ব্যাগ-অফ-ওয়ার্ডস ভেক্টর নিয়ে কাজ করেছিলাম যার দৈর্ঘ্য ছিল `vocab_size`, এবং আমরা স্পষ্টভাবে নিম্ন-মাত্রার পজিশনাল রিপ্রেজেন্টেশন ভেক্টর থেকে স্পার্স ওয়ান-হট রিপ্রেজেন্টেশনে রূপান্তর করছিলাম। এই ওয়ান-হট রিপ্রেজেন্টেশন মেমোরি-সাশ্রয়ী নয়, তাছাড়া প্রতিটি শব্দকে একে অপরের থেকে স্বাধীনভাবে বিবেচনা করা হয়, অর্থাৎ ওয়ান-হট এনকোডেড ভেক্টরগুলো শব্দগুলোর মধ্যে কোনো অর্থবোধক সাদৃশ্য প্রকাশ করে না।\n",
|
||||
"\n",
|
||||
"এই ইউনিটে, আমরা **News AG** ডেটাসেট নিয়ে আরও অনুসন্ধান চালিয়ে যাব। শুরু করার জন্য, চলুন ডেটা লোড করি এবং আগের নোটবুক থেকে কিছু সংজ্ঞা নিয়ে আসি।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Loading dataset...\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"d:\\WORK\\ai-for-beginners\\5-NLP\\14-Embeddings\\data\\train.csv: 29.5MB [00:01, 18.8MB/s] \n",
|
||||
"d:\\WORK\\ai-for-beginners\\5-NLP\\14-Embeddings\\data\\test.csv: 1.86MB [00:00, 11.2MB/s] \n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Building vocab...\n",
|
||||
"Vocab size = 95812\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"import torchtext\n",
|
||||
"import numpy as np\n",
|
||||
"from torchnlp import *\n",
|
||||
"train_dataset, test_dataset, classes, vocab = load_dataset()\n",
|
||||
"vocab_size = len(vocab)\n",
|
||||
"print(\"Vocab size = \",vocab_size)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## এমবেডিং কী?\n",
|
||||
"\n",
|
||||
"**এমবেডিং** ধারণাটি হলো শব্দগুলোকে নিম্ন-মাত্রার ঘন ভেক্টর দিয়ে উপস্থাপন করা, যা কোনোভাবে একটি শব্দের অর্থবোধক অর্থকে প্রতিফলিত করে। আমরা পরে আলোচনা করব কীভাবে অর্থবোধক শব্দ এমবেডিং তৈরি করা যায়, তবে আপাতত এমবেডিংকে একটি শব্দ ভেক্টরের মাত্রা কমানোর একটি পদ্ধতি হিসেবে ভাবুন।\n",
|
||||
"\n",
|
||||
"তাহলে, এমবেডিং লেয়ার একটি শব্দকে ইনপুট হিসেবে নেবে এবং নির্দিষ্ট `embedding_size` আকারের একটি আউটপুট ভেক্টর তৈরি করবে। এক অর্থে, এটি `Linear` লেয়ারের মতোই, তবে এক-হট এনকোডেড ভেক্টর নেওয়ার পরিবর্তে এটি একটি শব্দ নম্বরকে ইনপুট হিসেবে নিতে সক্ষম হবে।\n",
|
||||
"\n",
|
||||
"আমাদের নেটওয়ার্কে প্রথম লেয়ার হিসেবে এমবেডিং লেয়ার ব্যবহার করে, আমরা ব্যাগ-অফ-ওয়ার্ডস থেকে **এমবেডিং ব্যাগ** মডেলে পরিবর্তন করতে পারি, যেখানে আমরা প্রথমে আমাদের টেক্সটের প্রতিটি শব্দকে সংশ্লিষ্ট এমবেডিংয়ে রূপান্তর করি এবং তারপর সেই সমস্ত এমবেডিংয়ের উপর কিছু সামগ্রিক ফাংশন গণনা করি, যেমন `sum`, `average` বা `max`।\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"আমাদের ক্লাসিফায়ার নিউরাল নেটওয়ার্কটি এমবেডিং লেয়ার দিয়ে শুরু হবে, তারপর অ্যাগ্রিগেশন লেয়ার, এবং তার উপরে একটি লিনিয়ার ক্লাসিফায়ার:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class EmbedClassifier(torch.nn.Module):\n",
|
||||
" def __init__(self, vocab_size, embed_dim, num_class):\n",
|
||||
" super().__init__()\n",
|
||||
" self.embedding = torch.nn.Embedding(vocab_size, embed_dim)\n",
|
||||
" self.fc = torch.nn.Linear(embed_dim, num_class)\n",
|
||||
"\n",
|
||||
" def forward(self, x):\n",
|
||||
" x = self.embedding(x)\n",
|
||||
" x = torch.mean(x,dim=1)\n",
|
||||
" return self.fc(x)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### ভেরিয়েবল সিকোয়েন্স সাইজ নিয়ে কাজ করা\n",
|
||||
"\n",
|
||||
"এই আর্কিটেকচারের ফলে, আমাদের নেটওয়ার্কের জন্য মিনিব্যাচ তৈরি করতে একটি নির্দিষ্ট পদ্ধতি অনুসরণ করতে হবে। আগের ইউনিটে, যখন ব্যাগ-অফ-ওয়ার্ডস (BoW) ব্যবহার করা হচ্ছিল, তখন একটি মিনিব্যাচের সব BoW টেনসরের আকার সমান ছিল `vocab_size`, আমাদের টেক্সট সিকোয়েন্সের প্রকৃত দৈর্ঘ্য যাই হোক না কেন। কিন্তু যখন আমরা ওয়ার্ড এম্বেডিং-এ চলে যাব, তখন প্রতিটি টেক্সট নমুনায় ভিন্ন সংখ্যক শব্দ থাকবে, এবং সেই নমুনাগুলোকে একত্রিত করে মিনিব্যাচ তৈরি করার সময় আমাদের কিছু প্যাডিং প্রয়োগ করতে হবে।\n",
|
||||
"\n",
|
||||
"এটি করা যেতে পারে ডেটাসোর্সে `collate_fn` ফাংশন সরবরাহ করার একই কৌশল ব্যবহার করে:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def padify(b):\n",
|
||||
" # b is the list of tuples of length batch_size\n",
|
||||
" # - first element of a tuple = label, \n",
|
||||
" # - second = feature (text sequence)\n",
|
||||
" # build vectorized sequence\n",
|
||||
" v = [encode(x[1]) for x in b]\n",
|
||||
" # first, compute max length of a sequence in this minibatch\n",
|
||||
" l = max(map(len,v))\n",
|
||||
" return ( # tuple of two tensors - labels and features\n",
|
||||
" torch.LongTensor([t[0]-1 for t in b]),\n",
|
||||
" torch.stack([torch.nn.functional.pad(torch.tensor(t),(0,l-len(t)),mode='constant',value=0) for t in v])\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=padify, shuffle=True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### এম্বেডিং ক্লাসিফায়ার প্রশিক্ষণ\n",
|
||||
"\n",
|
||||
"এখন যেহেতু আমরা সঠিক ডাটালোডার সংজ্ঞায়িত করেছি, আমরা পূর্ববর্তী ইউনিটে সংজ্ঞায়িত প্রশিক্ষণ ফাংশন ব্যবহার করে মডেলটি প্রশিক্ষণ দিতে পারি:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.6415625\n",
|
||||
"6400: acc=0.6865625\n",
|
||||
"9600: acc=0.7103125\n",
|
||||
"12800: acc=0.726953125\n",
|
||||
"16000: acc=0.739375\n",
|
||||
"19200: acc=0.75046875\n",
|
||||
"22400: acc=0.7572321428571429\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(0.889799795315499, 0.7623160588611644)"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"net = EmbedClassifier(vocab_size,32,len(classes)).to(device)\n",
|
||||
"train_epoch(net,train_loader, lr=1, epoch_size=25000)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **নোট**: আমরা এখানে সময়ের স্বার্থে শুধুমাত্র ২৫,০০০ রেকর্ডের জন্য প্রশিক্ষণ দিচ্ছি (একটি পূর্ণ এপোকের চেয়ে কম), তবে আপনি প্রশিক্ষণ চালিয়ে যেতে পারেন, কয়েকটি এপোকের জন্য প্রশিক্ষণের একটি ফাংশন লিখতে পারেন এবং উচ্চতর সঠিকতা অর্জনের জন্য লার্নিং রেট প্যারামিটার নিয়ে পরীক্ষা করতে পারেন। আপনি প্রায় ৯০% সঠিকতা অর্জন করতে সক্ষম হবেন।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### EmbeddingBag লেয়ার এবং ভেরিয়েবল-দৈর্ঘ্যের সিকোয়েন্স উপস্থাপনা\n",
|
||||
"\n",
|
||||
"পূর্ববর্তী আর্কিটেকচারে, সমস্ত সিকোয়েন্সকে একই দৈর্ঘ্যে প্যাড করতে হতো যাতে সেগুলোকে একটি মিনিব্যাচে ফিট করানো যায়। এটি ভেরিয়েবল দৈর্ঘ্যের সিকোয়েন্স উপস্থাপনার সবচেয়ে কার্যকর পদ্ধতি নয় - আরেকটি পদ্ধতি হতে পারে **অফসেট** ভেক্টর ব্যবহার করা, যা একটি বড় ভেক্টরে সংরক্ষিত সমস্ত সিকোয়েন্সের অফসেট ধারণ করবে।\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"> **Note**: উপরের ছবিতে, আমরা একটি ক্যারেক্টারের সিকোয়েন্স দেখিয়েছি, কিন্তু আমাদের উদাহরণে আমরা শব্দের সিকোয়েন্স নিয়ে কাজ করছি। তবে, অফসেট ভেক্টর দিয়ে সিকোয়েন্স উপস্থাপনার সাধারণ নীতিটি একই থাকে।\n",
|
||||
"\n",
|
||||
"অফসেট উপস্থাপনার সাথে কাজ করার জন্য, আমরা [`EmbeddingBag`](https://pytorch.org/docs/stable/generated/torch.nn.EmbeddingBag.html) লেয়ার ব্যবহার করি। এটি `Embedding`-এর মতো, তবে এটি কন্টেন্ট ভেক্টর এবং অফসেট ভেক্টরকে ইনপুট হিসেবে নেয়, এবং এটি একটি অ্যাভারেজিং লেয়ার অন্তর্ভুক্ত করে, যা `mean`, `sum` বা `max` হতে পারে।\n",
|
||||
"\n",
|
||||
"এখানে একটি পরিবর্তিত নেটওয়ার্ক রয়েছে যা `EmbeddingBag` ব্যবহার করে:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class EmbedClassifier(torch.nn.Module):\n",
|
||||
" def __init__(self, vocab_size, embed_dim, num_class):\n",
|
||||
" super().__init__()\n",
|
||||
" self.embedding = torch.nn.EmbeddingBag(vocab_size, embed_dim)\n",
|
||||
" self.fc = torch.nn.Linear(embed_dim, num_class)\n",
|
||||
"\n",
|
||||
" def forward(self, text, off):\n",
|
||||
" x = self.embedding(text, off)\n",
|
||||
" return self.fc(x)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"ডেটাসেটটি প্রশিক্ষণের জন্য প্রস্তুত করতে, আমাদের একটি রূপান্তর ফাংশন প্রদান করতে হবে যা অফসেট ভেক্টর প্রস্তুত করবে:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def offsetify(b):\n",
|
||||
" # first, compute data tensor from all sequences\n",
|
||||
" x = [torch.tensor(encode(t[1])) for t in b]\n",
|
||||
" # now, compute the offsets by accumulating the tensor of sequence lengths\n",
|
||||
" o = [0] + [len(t) for t in x]\n",
|
||||
" o = torch.tensor(o[:-1]).cumsum(dim=0)\n",
|
||||
" return ( \n",
|
||||
" torch.LongTensor([t[0]-1 for t in b]), # labels\n",
|
||||
" torch.cat(x), # text \n",
|
||||
" o\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=offsetify, shuffle=True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"আগের সমস্ত উদাহরণের বিপরীতে, আমাদের নেটওয়ার্ক এখন দুটি প্যারামিটার গ্রহণ করে: ডেটা ভেক্টর এবং অফসেট ভেক্টর, যা ভিন্ন আকারের। অনুরূপভাবে, আমাদের ডেটা লোডারও আমাদের ২টির পরিবর্তে ৩টি মান প্রদান করে: টেক্সট এবং অফসেট ভেক্টর উভয়ই বৈশিষ্ট্য হিসাবে প্রদান করা হয়। সুতরাং, আমাদের প্রশিক্ষণ ফাংশনটি সামান্য সামঞ্জস্য করতে হবে যাতে এটি পরিচালনা করা যায়:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.6153125\n",
|
||||
"6400: acc=0.6615625\n",
|
||||
"9600: acc=0.6932291666666667\n",
|
||||
"12800: acc=0.715078125\n",
|
||||
"16000: acc=0.7270625\n",
|
||||
"19200: acc=0.7382291666666667\n",
|
||||
"22400: acc=0.7486160714285715\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(22.771553103007037, 0.7551983365323096)"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"net = EmbedClassifier(vocab_size,32,len(classes)).to(device)\n",
|
||||
"\n",
|
||||
"def train_epoch_emb(net,dataloader,lr=0.01,optimizer=None,loss_fn = torch.nn.CrossEntropyLoss(),epoch_size=None, report_freq=200):\n",
|
||||
" optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr)\n",
|
||||
" loss_fn = loss_fn.to(device)\n",
|
||||
" net.train()\n",
|
||||
" total_loss,acc,count,i = 0,0,0,0\n",
|
||||
" for labels,text,off in dataloader:\n",
|
||||
" optimizer.zero_grad()\n",
|
||||
" labels,text,off = labels.to(device), text.to(device), off.to(device)\n",
|
||||
" out = net(text, off)\n",
|
||||
" loss = loss_fn(out,labels) #cross_entropy(out,labels)\n",
|
||||
" loss.backward()\n",
|
||||
" optimizer.step()\n",
|
||||
" total_loss+=loss\n",
|
||||
" _,predicted = torch.max(out,1)\n",
|
||||
" acc+=(predicted==labels).sum()\n",
|
||||
" count+=len(labels)\n",
|
||||
" i+=1\n",
|
||||
" if i%report_freq==0:\n",
|
||||
" print(f\"{count}: acc={acc.item()/count}\")\n",
|
||||
" if epoch_size and count>epoch_size:\n",
|
||||
" break\n",
|
||||
" return total_loss.item()/count, acc.item()/count\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"train_epoch_emb(net,train_loader, lr=4, epoch_size=25000)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## সেমান্টিক এম্বেডিংস: Word2Vec\n",
|
||||
"\n",
|
||||
"আমাদের আগের উদাহরণে, মডেলের এম্বেডিং লেয়ার শব্দগুলোকে ভেক্টর রূপে মানচিত্রে পরিণত করতে শিখেছিল, তবে এই উপস্থাপনায় খুব বেশি সেমান্টিক অর্থ ছিল না। এমন একটি ভেক্টর উপস্থাপনা শেখা ভালো হবে, যেখানে একই ধরনের শব্দ বা সমার্থক শব্দগুলো এমন ভেক্টরের সাথে সম্পর্কিত হবে, যেগুলো কিছু ভেক্টর দূরত্ব (যেমন ইউক্লিডিয়ান দূরত্ব) অনুযায়ী একে অপরের কাছাকাছি থাকবে।\n",
|
||||
"\n",
|
||||
"এটি করতে, আমাদের একটি নির্দিষ্ট পদ্ধতিতে বড় পরিমাণে টেক্সটের উপর আমাদের এম্বেডিং মডেলকে প্রি-ট্রেন করতে হবে। সেমান্টিক এম্বেডিংস ট্রেন করার প্রথম পদ্ধতিগুলোর একটি হলো [Word2Vec](https://en.wikipedia.org/wiki/Word2vec)। এটি দুটি প্রধান আর্কিটেকচারের উপর ভিত্তি করে তৈরি, যা শব্দগুলোর একটি বিতরণকৃত উপস্থাপনা তৈরি করতে ব্যবহৃত হয়:\n",
|
||||
"\n",
|
||||
" - **কন্টিনিউয়াস ব্যাগ-অফ-ওয়ার্ডস** (CBoW) — এই আর্কিটেকচারে, আমরা মডেলকে আশেপাশের প্রসঙ্গ থেকে একটি শব্দ অনুমান করতে প্রশিক্ষণ দিই। ngram $(W_{-2},W_{-1},W_0,W_1,W_2)$ দেওয়া হলে, মডেলের লক্ষ্য হলো $(W_{-2},W_{-1},W_1,W_2)$ থেকে $W_0$ অনুমান করা।\n",
|
||||
" - **কন্টিনিউয়াস স্কিপ-গ্রাম** CBoW-এর বিপরীত। মডেল আশেপাশের প্রসঙ্গের উইন্ডো ব্যবহার করে বর্তমান শব্দটি অনুমান করে।\n",
|
||||
"\n",
|
||||
"CBoW দ্রুততর, তবে স্কিপ-গ্রাম ধীর হলেও কম ঘন ঘন ব্যবহৃত শব্দগুলোর উপস্থাপনা করতে ভালো কাজ করে।\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Google News ডেটাসেটে প্রি-ট্রেন করা word2vec এম্বেডিং নিয়ে পরীক্ষা করার জন্য, আমরা **gensim** লাইব্রেরি ব্যবহার করতে পারি। নিচে 'neural' শব্দটির সাথে সবচেয়ে মিল থাকা শব্দগুলো খুঁজে বের করা হয়েছে:\n",
|
||||
"\n",
|
||||
"> **Note:** যখন আপনি প্রথমবার শব্দ ভেক্টর তৈরি করেন, সেগুলো ডাউনলোড করতে কিছুটা সময় লাগতে পারে!\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import gensim.downloader as api\n",
|
||||
"w2v = api.load('word2vec-google-news-300')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"neuronal -> 0.7804799675941467\n",
|
||||
"neurons -> 0.7326500415802002\n",
|
||||
"neural_circuits -> 0.7252851724624634\n",
|
||||
"neuron -> 0.7174385190010071\n",
|
||||
"cortical -> 0.6941086649894714\n",
|
||||
"brain_circuitry -> 0.6923246383666992\n",
|
||||
"synaptic -> 0.6699118614196777\n",
|
||||
"neural_circuitry -> 0.6638563275337219\n",
|
||||
"neurochemical -> 0.6555314064025879\n",
|
||||
"neuronal_activity -> 0.6531826257705688\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"for w,p in w2v.most_similar('neural'):\n",
|
||||
" print(f\"{w} -> {p}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"আমরা শব্দ থেকে ভেক্টর এম্বেডিংও গণনা করতে পারি, যা শ্রেণীবিন্যাস মডেল প্রশিক্ষণে ব্যবহৃত হবে (স্পষ্টতার জন্য আমরা ভেক্টরের প্রথম ২০টি উপাদানই দেখাচ্ছি):\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([ 0.01226807, 0.06225586, 0.10693359, 0.05810547, 0.23828125,\n",
|
||||
" 0.03686523, 0.05151367, -0.20703125, 0.01989746, 0.10058594,\n",
|
||||
" -0.03759766, -0.1015625 , -0.15820312, -0.08105469, -0.0390625 ,\n",
|
||||
" -0.05053711, 0.16015625, 0.2578125 , 0.10058594, -0.25976562],\n",
|
||||
" dtype=float32)"
|
||||
]
|
||||
},
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"w2v.word_vec('play')[:20]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"সেমান্টিক এম্বেডিংসের একটি চমৎকার দিক হল আপনি ভেক্টর এনকোডিংকে পরিবর্তন করে সেমান্টিক্স পরিবর্তন করতে পারেন। উদাহরণস্বরূপ, আমরা এমন একটি শব্দ খুঁজতে বলতে পারি, যার ভেক্টর উপস্থাপন *king* এবং *woman* শব্দের যতটা সম্ভব কাছাকাছি হবে, এবং *man* শব্দ থেকে যতটা সম্ভব দূরে থাকবে:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"('queen', 0.7118192911148071)"
|
||||
]
|
||||
},
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"w2v.most_similar(positive=['king','woman'],negative=['man'])[0]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"CBoW এবং Skip-Grams উভয়ই \"পূর্বাভাসমূলক\" এম্বেডিং, কারণ এগুলো শুধুমাত্র স্থানীয় প্রসঙ্গগুলোকেই বিবেচনায় নেয়। Word2Vec গ্লোবাল প্রসঙ্গের সুবিধা নেয় না।\n",
|
||||
"\n",
|
||||
"**FastText**, Word2Vec-এর উপর ভিত্তি করে কাজ করে, যেখানে প্রতিটি শব্দ এবং সেই শব্দের মধ্যে থাকা চরিত্র n-grams-এর জন্য ভেক্টর উপস্থাপন শিখানো হয়। এই উপস্থাপনাগুলোর মান প্রতিটি প্রশিক্ষণ ধাপে গড় করে একটি ভেক্টরে রূপান্তরিত হয়। যদিও এটি প্রি-ট্রেনিংয়ে অতিরিক্ত গণনার প্রয়োজন করে, এটি শব্দ এম্বেডিং-এ সাব-ওয়ার্ড তথ্য এনকোড করতে সক্ষম করে।\n",
|
||||
"\n",
|
||||
"আরেকটি পদ্ধতি, **GloVe**, কো-অকারেন্স ম্যাট্রিক্সের ধারণাকে কাজে লাগায় এবং নিউরাল পদ্ধতি ব্যবহার করে কো-অকারেন্স ম্যাট্রিক্সকে আরও বেশি প্রকাশক্ষম এবং নন-লিনিয়ার শব্দ ভেক্টরে রূপান্তরিত করে।\n",
|
||||
"\n",
|
||||
"আপনি উদাহরণটি নিয়ে পরীক্ষা করতে পারেন FastText এবং GloVe এম্বেডিং ব্যবহার করে, কারণ gensim বিভিন্ন ধরনের শব্দ এম্বেডিং মডেল সমর্থন করে।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## PyTorch-এ প্রি-ট্রেইনড এমবেডিং ব্যবহার করা\n",
|
||||
"\n",
|
||||
"আমরা উপরের উদাহরণটি পরিবর্তন করে আমাদের এমবেডিং লেয়ারে সেমান্টিক্যাল এমবেডিং, যেমন Word2Vec, দিয়ে ম্যাট্রিক্সটি প্রি-পপুলেট করতে পারি। আমাদের মনে রাখতে হবে যে প্রি-ট্রেইনড এমবেডিং এবং আমাদের টেক্সট কর্পাসের ভোকাবুলারিগুলো সম্ভবত মিলবে না, তাই আমরা অনুপস্থিত শব্দগুলোর জন্য ওজনগুলো র্যান্ডম মান দিয়ে ইনিশিয়ালাইজ করব:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Embedding size: 300\n",
|
||||
"Populating matrix, this will take some time...Done, found 41080 words, 54732 words missing\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"embed_size = len(w2v.get_vector('hello'))\n",
|
||||
"print(f'Embedding size: {embed_size}')\n",
|
||||
"\n",
|
||||
"net = EmbedClassifier(vocab_size,embed_size,len(classes))\n",
|
||||
"\n",
|
||||
"print('Populating matrix, this will take some time...',end='')\n",
|
||||
"found, not_found = 0,0\n",
|
||||
"for i,w in enumerate(vocab.get_itos()):\n",
|
||||
" try:\n",
|
||||
" net.embedding.weight[i].data = torch.tensor(w2v.get_vector(w))\n",
|
||||
" found+=1\n",
|
||||
" except:\n",
|
||||
" net.embedding.weight[i].data = torch.normal(0.0,1.0,(embed_size,))\n",
|
||||
" not_found+=1\n",
|
||||
"\n",
|
||||
"print(f\"Done, found {found} words, {not_found} words missing\")\n",
|
||||
"net = net.to(device)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"এখন চলুন আমাদের মডেল প্রশিক্ষণ করি। লক্ষ্য করুন যে মডেল প্রশিক্ষণ করতে যে সময় লাগে তা পূর্ববর্তী উদাহরণের তুলনায় উল্লেখযোগ্যভাবে বেশি, কারণ এম্বেডিং লেয়ারের আকার বড় হওয়ার কারণে প্যারামিটারের সংখ্যা অনেক বেশি। এছাড়াও, এই কারণে, যদি আমরা ওভারফিটিং এড়াতে চাই, তাহলে আমাদের আরও উদাহরণে মডেল প্রশিক্ষণ করতে হতে পারে।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.6359375\n",
|
||||
"6400: acc=0.68109375\n",
|
||||
"9600: acc=0.7067708333333333\n",
|
||||
"12800: acc=0.723671875\n",
|
||||
"16000: acc=0.73625\n",
|
||||
"19200: acc=0.7463541666666667\n",
|
||||
"22400: acc=0.7560714285714286\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(214.1013875559821, 0.7626759436980166)"
|
||||
]
|
||||
},
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"train_epoch_emb(net,train_loader, lr=4, epoch_size=25000)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"আমাদের ক্ষেত্রে আমরা যথেষ্ট পরিমাণে সঠিকতার বৃদ্ধি দেখতে পাই না, যা সম্ভবত ভিন্ন ভিন্ন শব্দভাণ্ডারের কারণে। \n",
|
||||
"ভিন্ন শব্দভাণ্ডারের সমস্যাটি সমাধান করতে, আমরা নিম্নলিখিত সমাধানগুলোর মধ্যে একটি ব্যবহার করতে পারি: \n",
|
||||
"* আমাদের শব্দভাণ্ডারে ভিত্তি করে word2vec মডেল পুনরায় প্রশিক্ষণ করা \n",
|
||||
"* প্রি-ট্রেইন করা word2vec মডেলের শব্দভাণ্ডার ব্যবহার করে আমাদের ডেটাসেট লোড করা। ডেটাসেট লোড করার সময় ব্যবহৃত শব্দভাণ্ডার নির্দিষ্ট করা যেতে পারে। \n",
|
||||
"\n",
|
||||
"দ্বিতীয় পদ্ধতিটি সহজতর মনে হয়, বিশেষত কারণ PyTorch `torchtext` ফ্রেমওয়ার্ক এম্বেডিংয়ের জন্য বিল্ট-ইন সাপোর্ট প্রদান করে। আমরা, উদাহরণস্বরূপ, GloVe-ভিত্তিক শব্দভাণ্ডার নিম্নলিখিতভাবে তৈরি করতে পারি: \n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"100%|█████████▉| 399999/400000 [00:15<00:00, 25411.14it/s]\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vocab = torchtext.vocab.GloVe(name='6B', dim=50)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"লোড করা ভোকাবুলারিতে নিম্নলিখিত মৌলিক কার্যক্রমগুলো রয়েছে:\n",
|
||||
"* `vocab.stoi` ডিকশনারি আমাদের একটি শব্দকে তার ডিকশনারি সূচকে রূপান্তর করতে সাহায্য করে\n",
|
||||
"* `vocab.itos` বিপরীত কাজটি করে - একটি সংখ্যাকে শব্দে রূপান্তর করে\n",
|
||||
"* `vocab.vectors` হলো এমবেডিং ভেক্টরগুলোর অ্যারে, তাই একটি শব্দ `s`-এর এমবেডিং পেতে হলে আমাদের `vocab.vectors[vocab.stoi[s]]` ব্যবহার করতে হবে\n",
|
||||
"\n",
|
||||
"এখানে এমবেডিং নিয়ে কাজ করার একটি উদাহরণ দেওয়া হলো, যা সমীকরণটি প্রদর্শন করে **kind-man+woman = queen** (আমি এটি কাজ করানোর জন্য সহগটি একটু পরিবর্তন করেছি):\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"'queen'"
|
||||
]
|
||||
},
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# get the vector corresponding to kind-man+woman\n",
|
||||
"qvec = vocab.vectors[vocab.stoi['king']]-vocab.vectors[vocab.stoi['man']]+1.3*vocab.vectors[vocab.stoi['woman']]\n",
|
||||
"# find the index of the closest embedding vector \n",
|
||||
"d = torch.sum((vocab.vectors-qvec)**2,dim=1)\n",
|
||||
"min_idx = torch.argmin(d)\n",
|
||||
"# find the corresponding word\n",
|
||||
"vocab.itos[min_idx]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"এই এম্বেডিংগুলি ব্যবহার করে শ্রেণীবিন্যাসকারীকে প্রশিক্ষণ দেওয়ার জন্য, আমাদের প্রথমে GloVe শব্দভাণ্ডার ব্যবহার করে আমাদের ডেটাসেট এনকোড করতে হবে:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 16,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def offsetify(b):\n",
|
||||
" # first, compute data tensor from all sequences\n",
|
||||
" x = [torch.tensor(encode(t[1],voc=vocab)) for t in b] # pass the instance of vocab to encode function!\n",
|
||||
" # now, compute the offsets by accumulating the tensor of sequence lengths\n",
|
||||
" o = [0] + [len(t) for t in x]\n",
|
||||
" o = torch.tensor(o[:-1]).cumsum(dim=0)\n",
|
||||
" return ( \n",
|
||||
" torch.LongTensor([t[0]-1 for t in b]), # labels\n",
|
||||
" torch.cat(x), # text \n",
|
||||
" o\n",
|
||||
" )"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"যেমনটি আমরা উপরে দেখেছি, সমস্ত ভেক্টর এম্বেডিং `vocab.vectors` ম্যাট্রিক্সে সংরক্ষিত হয়। এটি এম্বেডিং লেয়ারের ওজনগুলিতে সেই ওজনগুলি সহজ কপি করার মাধ্যমে লোড করা অত্যন্ত সহজ করে তোলে:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 17,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"net = EmbedClassifier(len(vocab),len(vocab.vectors[0]),len(classes))\n",
|
||||
"net.embedding.weight.data = vocab.vectors\n",
|
||||
"net = net.to(device)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"এখন চলুন আমাদের মডেল প্রশিক্ষণ করি এবং দেখি আমরা ভালো ফলাফল পাই কিনা:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 18,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.6271875\n",
|
||||
"6400: acc=0.68078125\n",
|
||||
"9600: acc=0.7030208333333333\n",
|
||||
"12800: acc=0.71984375\n",
|
||||
"16000: acc=0.7346875\n",
|
||||
"19200: acc=0.7455729166666667\n",
|
||||
"22400: acc=0.7529464285714286\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(35.53972978646833, 0.7575175943698017)"
|
||||
]
|
||||
},
|
||||
"execution_count": 18,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=offsetify, shuffle=True)\n",
|
||||
"train_epoch_emb(net,train_loader, lr=4, epoch_size=25000)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"আমাদের ডেটাসেটের কিছু শব্দ প্রি-ট্রেইনড GloVe ভোকাবুলারিতে অনুপস্থিত থাকার কারণে আমরা যথেষ্ট পরিমাণে সঠিকতার বৃদ্ধি দেখতে পাচ্ছি না, এবং তাই সেগুলো কার্যত উপেক্ষা করা হয়। এই সমস্যাটি কাটিয়ে উঠতে, আমরা আমাদের ডেটাসেটের উপর নিজস্ব এম্বেডিং ট্রেইন করতে পারি।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## প্রসঙ্গগত এম্বেডিংস\n",
|
||||
"\n",
|
||||
"প্রথাগত প্রি-ট্রেইনড এম্বেডিং উপস্থাপনাগুলোর, যেমন Word2Vec, একটি প্রধান সীমাবদ্ধতা হলো শব্দার্থ বিভ্রান্তির সমস্যা। যদিও প্রি-ট্রেইনড এম্বেডিংস কিছুটা শব্দের অর্থকে প্রসঙ্গ অনুযায়ী ধারণ করতে পারে, একটি শব্দের সব সম্ভাব্য অর্থ একই এম্বেডিংয়ে এনকোড করা হয়। এটি ডাউনস্ট্রিম মডেলগুলোর ক্ষেত্রে সমস্যা সৃষ্টি করতে পারে, কারণ অনেক শব্দ, যেমন 'play', এর অর্থ প্রসঙ্গ অনুযায়ী ভিন্ন হতে পারে।\n",
|
||||
"\n",
|
||||
"উদাহরণস্বরূপ, 'play' শব্দটি এই দুটি বাক্যে ভিন্ন অর্থ বহন করে:\n",
|
||||
"- আমি থিয়েটারে একটি **play** দেখতে গিয়েছিলাম।\n",
|
||||
"- জন তার বন্ধুদের সাথে **play** করতে চায়।\n",
|
||||
"\n",
|
||||
"উপরের প্রি-ট্রেইনড এম্বেডিংস 'play' শব্দের এই দুটি অর্থকে একই এম্বেডিংয়ে উপস্থাপন করে। এই সীমাবদ্ধতা কাটিয়ে উঠতে, আমাদের **ভাষার মডেল** ভিত্তিক এম্বেডিং তৈরি করতে হবে, যা একটি বড় টেক্সট কর্পাসে প্রশিক্ষিত হয় এবং *জানে* কীভাবে শব্দগুলো বিভিন্ন প্রসঙ্গে একত্রিত হতে পারে। প্রসঙ্গগত এম্বেডিং নিয়ে আলোচনা এই টিউটোরিয়ালের পরিধির বাইরে, তবে আমরা পরবর্তী ইউনিটে ভাষার মডেল নিয়ে আলোচনা করার সময় এ বিষয়ে ফিরে আসব।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**অস্বীকৃতি**: \nএই নথিটি AI অনুবাদ পরিষেবা [Co-op Translator](https://github.com/Azure/co-op-translator) ব্যবহার করে অনুবাদ করা হয়েছে। আমরা যথাসাধ্য সঠিকতার জন্য চেষ্টা করি, তবে অনুগ্রহ করে মনে রাখবেন যে স্বয়ংক্রিয় অনুবাদে ত্রুটি বা অসঙ্গতি থাকতে পারে। মূল ভাষায় থাকা নথিটিকে প্রামাণিক উৎস হিসেবে বিবেচনা করা উচিত। গুরুত্বপূর্ণ তথ্যের জন্য, পেশাদার মানব অনুবাদ সুপারিশ করা হয়। এই অনুবাদ ব্যবহারের ফলে কোনো ভুল বোঝাবুঝি বা ভুল ব্যাখ্যা হলে আমরা দায়বদ্ধ থাকব না।\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "py37_pytorch",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "f50b026abce5cf36783a560ea72cb9b1",
|
||||
"translation_date": "2025-08-28T12:27:42+00:00",
|
||||
"source_file": "lessons/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb",
|
||||
"language_code": "bn"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
|
|
@ -0,0 +1,695 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## এম্বেডিংস\n",
|
||||
"\n",
|
||||
"আমাদের আগের উদাহরণে, আমরা `vocab_size` দৈর্ঘ্যের উচ্চ-মাত্রার ব্যাগ-অফ-ওয়ার্ডস ভেক্টরের উপর কাজ করেছি এবং নিম্ন-মাত্রার পজিশনাল রিপ্রেজেন্টেশন ভেক্টরগুলোকে স্পষ্টভাবে স্পার্স ওয়ান-হট রিপ্রেজেন্টেশনে রূপান্তর করেছি। এই ওয়ান-হট রিপ্রেজেন্টেশন মেমোরি-সাশ্রয়ী নয়। এছাড়াও, প্রতিটি শব্দকে একে অপরের থেকে স্বাধীনভাবে বিবেচনা করা হয়, তাই ওয়ান-হট এনকোডেড ভেক্টরগুলো শব্দগুলোর মধ্যে অর্থগত সাদৃশ্য প্রকাশ করতে পারে না।\n",
|
||||
"\n",
|
||||
"এই ইউনিটে, আমরা **News AG** ডেটাসেট নিয়ে আরও অনুসন্ধান চালিয়ে যাব। শুরু করার জন্য, চলুন ডেটা লোড করি এবং আগের ইউনিট থেকে কিছু সংজ্ঞা পুনরায় দেখি।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import tensorflow as tf\n",
|
||||
"from tensorflow import keras\n",
|
||||
"import tensorflow_datasets as tfds\n",
|
||||
"import numpy as np\n",
|
||||
"\n",
|
||||
"ds_train, ds_test = tfds.load('ag_news_subset').values()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### এম্বেডিং কী?\n",
|
||||
"\n",
|
||||
"**এম্বেডিং** ধারণাটি হলো শব্দগুলোকে নিম্ন-মাত্রার ঘন ভেক্টর দিয়ে উপস্থাপন করা, যা শব্দটির অর্থবহ ধারণা প্রতিফলিত করে। আমরা পরে আলোচনা করব কীভাবে অর্থবহ শব্দ এম্বেডিং তৈরি করা যায়, তবে আপাতত এম্বেডিংকে একটি পদ্ধতি হিসেবে ভাবুন যা শব্দ ভেক্টরের মাত্রা কমিয়ে আনে।\n",
|
||||
"\n",
|
||||
"সুতরাং, একটি এম্বেডিং লেয়ার একটি শব্দকে ইনপুট হিসেবে নেয় এবং নির্দিষ্ট `embedding_size` আউটপুট ভেক্টর প্রদান করে। এক অর্থে, এটি একটি `Dense` লেয়ারের মতোই, তবে এটি এক-হট এনকোডেড ভেক্টর ইনপুট হিসেবে নেওয়ার পরিবর্তে একটি শব্দ নম্বর নিতে সক্ষম।\n",
|
||||
"\n",
|
||||
"আমাদের নেটওয়ার্কে প্রথম লেয়ার হিসেবে একটি এম্বেডিং লেয়ার ব্যবহার করে, আমরা ব্যাগ-অফ-ওয়ার্ডস থেকে **এম্বেডিং ব্যাগ** মডেলে স্যুইচ করতে পারি, যেখানে আমরা প্রথমে আমাদের টেক্সটের প্রতিটি শব্দকে সংশ্লিষ্ট এম্বেডিং-এ রূপান্তর করি এবং তারপর সেই এম্বেডিংগুলোর উপর একটি সমষ্টিগত ফাংশন (যেমন `sum`, `average` বা `max`) গণনা করি।\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"আমাদের ক্লাসিফায়ার নিউরাল নেটওয়ার্কে নিম্নলিখিত লেয়ারগুলো অন্তর্ভুক্ত রয়েছে:\n",
|
||||
"\n",
|
||||
"* `TextVectorization` লেয়ার, যা একটি স্ট্রিং ইনপুট হিসেবে নেয় এবং টোকেন নম্বরের একটি টেনসর আউটপুট দেয়। আমরা একটি যুক্তিসঙ্গত শব্দভাণ্ডারের আকার `vocab_size` নির্ধারণ করব এবং কম ব্যবহৃত শব্দগুলো উপেক্ষা করব। ইনপুট শেপ হবে ১, এবং আউটপুট শেপ হবে $n$, কারণ আমরা $n$ টোকেন পাব, যার প্রতিটিতে ০ থেকে `vocab_size` পর্যন্ত নম্বর থাকবে।\n",
|
||||
"* `Embedding` লেয়ার, যা $n$ নম্বর নেয় এবং প্রতিটি নম্বরকে একটি নির্দিষ্ট দৈর্ঘ্যের ঘন ভেক্টরে রূপান্তর করে (আমাদের উদাহরণে ১০০)। সুতরাং, $n$ শেপের ইনপুট টেনসর $n\\times 100$ শেপের টেনসরে রূপান্তরিত হবে।\n",
|
||||
"* অ্যাগ্রিগেশন লেয়ার, যা এই টেনসরের প্রথম অক্ষ বরাবর গড় গণনা করে, অর্থাৎ এটি বিভিন্ন শব্দের সাথে সম্পর্কিত $n$ ইনপুট টেনসরের গড় গণনা করবে। এই লেয়ারটি বাস্তবায়নের জন্য, আমরা একটি `Lambda` লেয়ার ব্যবহার করব এবং এতে গড় গণনার ফাংশন পাস করব। আউটপুটের শেপ হবে ১০০, এবং এটি পুরো ইনপুট সিকোয়েন্সের একটি সংখ্যাগত উপস্থাপনা হবে।\n",
|
||||
"* চূড়ান্ত `Dense` লিনিয়ার ক্লাসিফায়ার।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"sequential\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
" Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
" text_vectorization (TextVec (None, None) 0 \n",
|
||||
" torization) \n",
|
||||
" \n",
|
||||
" embedding (Embedding) (None, None, 100) 3000000 \n",
|
||||
" \n",
|
||||
" lambda (Lambda) (None, 100) 0 \n",
|
||||
" \n",
|
||||
" dense (Dense) (None, 4) 404 \n",
|
||||
" \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 3,000,404\n",
|
||||
"Trainable params: 3,000,404\n",
|
||||
"Non-trainable params: 0\n",
|
||||
"_________________________________________________________________\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vocab_size = 30000\n",
|
||||
"batch_size = 128\n",
|
||||
"\n",
|
||||
"vectorizer = keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size,input_shape=(1,))\n",
|
||||
"\n",
|
||||
"model = keras.models.Sequential([\n",
|
||||
" vectorizer, \n",
|
||||
" keras.layers.Embedding(vocab_size,100),\n",
|
||||
" keras.layers.Lambda(lambda x: tf.reduce_mean(x,axis=1)),\n",
|
||||
" keras.layers.Dense(4, activation='softmax')\n",
|
||||
"])\n",
|
||||
"model.summary()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"`summary` প্রিন্টআউট-এ, **output shape** কলামে প্রথম টেনসরের ডাইমেনশন `None` মিনিব্যাচের আকার নির্দেশ করে, এবং দ্বিতীয়টি টোকেন সিকোয়েন্সের দৈর্ঘ্য নির্দেশ করে। মিনিব্যাচের সব টোকেন সিকোয়েন্সের দৈর্ঘ্য ভিন্ন হয়। আমরা পরবর্তী অংশে এটি কীভাবে সামলানো যায় তা নিয়ে আলোচনা করব।\n",
|
||||
"\n",
|
||||
"এখন চলুন নেটওয়ার্কটি প্রশিক্ষণ দেই:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Training vectorizer\n",
|
||||
"938/938 [==============================] - 20s 20ms/step - loss: 0.7891 - acc: 0.8155 - val_loss: 0.4470 - val_acc: 0.8642\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.History at 0x22255515100>"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def extract_text(x):\n",
|
||||
" return x['title']+' '+x['description']\n",
|
||||
"\n",
|
||||
"def tupelize(x):\n",
|
||||
" return (extract_text(x),x['label'])\n",
|
||||
"\n",
|
||||
"print(\"Training vectorizer\")\n",
|
||||
"vectorizer.adapt(ds_train.take(500).map(extract_text))\n",
|
||||
"\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'])\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"nteract": {
|
||||
"transient": {
|
||||
"deleting": false
|
||||
}
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"> **নোট** যে আমরা ডেটার একটি উপসেটের উপর ভিত্তি করে ভেক্টরাইজার তৈরি করছি। এটি প্রক্রিয়াটি দ্রুত করার জন্য করা হয়, এবং এটি এমন একটি পরিস্থিতি তৈরি করতে পারে যেখানে আমাদের টেক্সটের সমস্ত টোকেন শব্দভাণ্ডারে উপস্থিত না থাকে। এই ক্ষেত্রে, সেই টোকেনগুলি উপেক্ষা করা হবে, যা সামান্য কম সঠিকতা সৃষ্টি করতে পারে। তবে, বাস্তব জীবনে টেক্সটের একটি উপসেট প্রায়শই একটি ভাল শব্দভাণ্ডার অনুমান প্রদান করে।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### ভেরিয়েবল সিকোয়েন্স সাইজ নিয়ে কাজ করা\n",
|
||||
"\n",
|
||||
"চলুন বুঝি কীভাবে মিনি-ব্যাচে প্রশিক্ষণ হয়। উপরের উদাহরণে, ইনপুট টেনসরের মাত্রা 1, এবং আমরা ১২৮-লম্বা মিনি-ব্যাচ ব্যবহার করি, তাই টেনসরের প্রকৃত আকার হয় $128 \\times 1$। তবে, প্রতিটি বাক্যের টোকেনের সংখ্যা ভিন্ন। যদি আমরা একটি একক ইনপুটে `TextVectorization` লেয়ার প্রয়োগ করি, তাহলে টোকেনাইজেশনের উপর নির্ভর করে ফেরত আসা টোকেনের সংখ্যা ভিন্ন হয়:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"tf.Tensor([ 1 45], shape=(2,), dtype=int64)\n",
|
||||
"tf.Tensor([ 112 1271 1 3 1747 158], shape=(6,), dtype=int64)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print(vectorizer('Hello, world!'))\n",
|
||||
"print(vectorizer('I am glad to meet you!'))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"তবে, যখন আমরা ভেক্টরাইজারকে একাধিক সিকোয়েন্সে প্রয়োগ করি, এটি আয়তাকার আকৃতির একটি টেনসর তৈরি করতে হয়, তাই এটি অব্যবহৃত উপাদানগুলোকে PAD টোকেন (যা আমাদের ক্ষেত্রে শূন্য) দিয়ে পূরণ করে:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tf.Tensor: shape=(2, 6), dtype=int64, numpy=\n",
|
||||
"array([[ 1, 45, 0, 0, 0, 0],\n",
|
||||
" [ 112, 1271, 1, 3, 1747, 158]], dtype=int64)>"
|
||||
]
|
||||
},
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vectorizer(['Hello, world!','I am glad to meet you!'])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"এখানে আমরা এম্বেডিংগুলি দেখতে পারি:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([[[ 1.53059261e-02, 6.80514947e-02, 3.14026810e-02, ...,\n",
|
||||
" -8.92002955e-02, 1.52911525e-04, -5.65562584e-02],\n",
|
||||
" [ 2.57456154e-01, 2.79364467e-01, -2.03605562e-01, ...,\n",
|
||||
" -2.07474351e-01, 8.31158683e-02, -2.03911960e-01],\n",
|
||||
" [ 3.98201384e-02, -8.03454965e-03, 2.39790026e-02, ...,\n",
|
||||
" -7.18549127e-04, 2.66963355e-02, -4.30646613e-02],\n",
|
||||
" [ 3.98201384e-02, -8.03454965e-03, 2.39790026e-02, ...,\n",
|
||||
" -7.18549127e-04, 2.66963355e-02, -4.30646613e-02],\n",
|
||||
" [ 3.98201384e-02, -8.03454965e-03, 2.39790026e-02, ...,\n",
|
||||
" -7.18549127e-04, 2.66963355e-02, -4.30646613e-02],\n",
|
||||
" [ 3.98201384e-02, -8.03454965e-03, 2.39790026e-02, ...,\n",
|
||||
" -7.18549127e-04, 2.66963355e-02, -4.30646613e-02]],\n",
|
||||
"\n",
|
||||
" [[ 1.89674050e-01, 2.61548996e-01, -3.67433839e-02, ...,\n",
|
||||
" -2.07366899e-01, -1.05442435e-01, -2.36952081e-01],\n",
|
||||
" [ 6.16133213e-02, 1.80511594e-01, 9.77298319e-02, ...,\n",
|
||||
" -5.46628237e-02, -1.07340455e-01, -1.06589928e-01],\n",
|
||||
" [ 1.53059261e-02, 6.80514947e-02, 3.14026810e-02, ...,\n",
|
||||
" -8.92002955e-02, 1.52911525e-04, -5.65562584e-02],\n",
|
||||
" [-4.84890305e-02, -8.41715634e-02, 1.51529670e-01, ...,\n",
|
||||
" 1.28192469e-01, -7.77286515e-02, 1.26041949e-01],\n",
|
||||
" [-4.17212099e-02, -5.60694858e-02, 4.08860669e-02, ...,\n",
|
||||
" 8.70475471e-02, 8.92383084e-02, 1.67974353e-01],\n",
|
||||
" [ 2.85779923e-01, 4.57767487e-01, 4.52292450e-02, ...,\n",
|
||||
" -1.97419018e-01, -2.04659685e-01, -2.79758364e-01]]],\n",
|
||||
" dtype=float32)"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.layers[1](vectorizer(['Hello, world!','I am glad to meet you!'])).numpy()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **নোট**: প্যাডিংয়ের পরিমাণ কমানোর জন্য, কিছু ক্ষেত্রে ডেটাসেটের সমস্ত সিকোয়েন্সকে দৈর্ঘ্য বৃদ্ধির ক্রমে (অথবা, আরও সঠিকভাবে, টোকেনের সংখ্যার ক্রমে) সাজানো যুক্তিসঙ্গত হতে পারে। এটি নিশ্চিত করবে যে প্রতিটি মিনিব্যাচে একই দৈর্ঘ্যের সিকোয়েন্স থাকবে।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## সেমান্টিক এম্বেডিংস: ওয়ার্ড২ভেক\n",
|
||||
"\n",
|
||||
"আমাদের আগের উদাহরণে, এম্বেডিং লেয়ার শব্দগুলোকে ভেক্টর রূপে মানচিত্রিত করতে শিখেছিল, তবে এই ভেক্টরগুলোর সেমান্টিক অর্থ ছিল না। যদি এমন একটি ভেক্টর উপস্থাপনা শেখা যেত যেখানে একই ধরনের শব্দ বা সমার্থক শব্দগুলো এমন ভেক্টরের সাথে মিলে যেত যা কিছু ভেক্টর দূরত্ব (যেমন ইউক্লিডিয়ান দূরত্ব) অনুযায়ী কাছাকাছি থাকে, তাহলে ভালো হতো।\n",
|
||||
"\n",
|
||||
"এটি করতে, আমাদের একটি বড় টেক্সট সংগ্রহে আমাদের এম্বেডিং মডেলকে প্রি-ট্রেইন করতে হবে, যেমন [Word2Vec](https://en.wikipedia.org/wiki/Word2vec) পদ্ধতি ব্যবহার করে। এটি দুটি প্রধান আর্কিটেকচারের উপর ভিত্তি করে তৈরি, যা শব্দগুলোর একটি বিতরণকৃত উপস্থাপনা তৈরি করতে ব্যবহৃত হয়:\n",
|
||||
"\n",
|
||||
" - **কন্টিনিউয়াস ব্যাগ-অফ-ওয়ার্ডস** (CBoW), যেখানে আমরা মডেলটিকে পারিপার্শ্বিক প্রসঙ্গ থেকে একটি শব্দ অনুমান করতে প্রশিক্ষণ দিই। যদি ngram $(W_{-2},W_{-1},W_0,W_1,W_2)$ দেওয়া হয়, মডেলের লক্ষ্য হলো $(W_{-2},W_{-1},W_1,W_2)$ থেকে $W_0$ অনুমান করা।\n",
|
||||
" - **কন্টিনিউয়াস স্কিপ-গ্রাম** CBoW-এর বিপরীত। এই মডেলটি বর্তমান শব্দ অনুমান করতে পারিপার্শ্বিক প্রসঙ্গের উইন্ডো ব্যবহার করে।\n",
|
||||
"\n",
|
||||
"CBoW দ্রুততর, এবং স্কিপ-গ্রাম ধীর হলেও এটি কম ঘন ঘন ব্যবহৃত শব্দগুলোর উপস্থাপনা আরও ভালোভাবে করে।\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"গুগল নিউজ ডেটাসেটে প্রি-ট্রেইন করা Word2Vec এম্বেডিং নিয়ে পরীক্ষা করার জন্য, আমরা **gensim** লাইব্রেরি ব্যবহার করতে পারি। নিচে আমরা 'neural' শব্দটির সাথে সবচেয়ে সাদৃশ্যপূর্ণ শব্দগুলো খুঁজে বের করব।\n",
|
||||
"\n",
|
||||
"> **Note:** যখন আপনি প্রথমবার শব্দ ভেক্টর তৈরি করবেন, তখন সেগুলো ডাউনলোড করতে কিছুটা সময় লাগতে পারে!\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import gensim.downloader as api\n",
|
||||
"w2v = api.load('word2vec-google-news-300')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"neuronal -> 0.7804799675941467\n",
|
||||
"neurons -> 0.7326500415802002\n",
|
||||
"neural_circuits -> 0.7252851724624634\n",
|
||||
"neuron -> 0.7174385190010071\n",
|
||||
"cortical -> 0.6941086649894714\n",
|
||||
"brain_circuitry -> 0.6923246383666992\n",
|
||||
"synaptic -> 0.6699118614196777\n",
|
||||
"neural_circuitry -> 0.6638563275337219\n",
|
||||
"neurochemical -> 0.6555314064025879\n",
|
||||
"neuronal_activity -> 0.6531826257705688\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"for w,p in w2v.most_similar('neural'):\n",
|
||||
" print(f\"{w} -> {p}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"আমরা শব্দ থেকে ভেক্টর এমবেডিংও বের করতে পারি, যা শ্রেণীবিন্যাস মডেল প্রশিক্ষণে ব্যবহার করা হবে। এমবেডিং-এর ৩০০টি উপাদান রয়েছে, তবে এখানে স্পষ্টতার জন্য আমরা শুধুমাত্র ভেক্টরের প্রথম ২০টি উপাদান দেখাচ্ছি:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([ 0.01226807, 0.06225586, 0.10693359, 0.05810547, 0.23828125,\n",
|
||||
" 0.03686523, 0.05151367, -0.20703125, 0.01989746, 0.10058594,\n",
|
||||
" -0.03759766, -0.1015625 , -0.15820312, -0.08105469, -0.0390625 ,\n",
|
||||
" -0.05053711, 0.16015625, 0.2578125 , 0.10058594, -0.25976562],\n",
|
||||
" dtype=float32)"
|
||||
]
|
||||
},
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"w2v['play'][:20]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"সেমান্টিক এম্বেডিংসের দুর্দান্ত বিষয় হল আপনি সেমান্টিক্সের উপর ভিত্তি করে ভেক্টর এনকোডিং পরিচালনা করতে পারেন। উদাহরণস্বরূপ, আমরা এমন একটি শব্দ খুঁজতে বলতে পারি যার ভেক্টর উপস্থাপন *king* এবং *woman* শব্দের যতটা সম্ভব কাছাকাছি এবং *man* শব্দ থেকে যতটা সম্ভব দূরে:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"('queen', 0.7118192911148071)"
|
||||
]
|
||||
},
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"w2v.most_similar(positive=['king','woman'],negative=['man'])[0]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"উপরের উদাহরণটি কিছু অভ্যন্তরীণ GenSym ম্যাজিক ব্যবহার করে, তবে এর অন্তর্নিহিত যুক্তি আসলে বেশ সহজ। এম্বেডিং সম্পর্কে একটি আকর্ষণীয় বিষয় হল যে আপনি এম্বেডিং ভেক্টরগুলিতে সাধারণ ভেক্টর অপারেশন করতে পারেন, এবং এটি শব্দের **অর্থ** নিয়ে অপারেশনগুলিকে প্রতিফলিত করবে। উপরের উদাহরণটি ভেক্টর অপারেশনের মাধ্যমে প্রকাশ করা যেতে পারে: আমরা **KING-MAN+WOMAN** এর সাথে সম্পর্কিত ভেক্টরটি গণনা করি (যেখানে সংশ্লিষ্ট শব্দগুলির ভেক্টর উপস্থাপনার উপর `+` এবং `-` অপারেশনগুলি সম্পাদিত হয়), এবং তারপর সেই ভেক্টরের সাথে অভিধানের সবচেয়ে কাছাকাছি শব্দটি খুঁজে বের করি:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"'queen'"
|
||||
]
|
||||
},
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# get the vector corresponding to kind-man+woman\n",
|
||||
"qvec = w2v['king']-1.7*w2v['man']+1.7*w2v['woman']\n",
|
||||
"# find the index of the closest embedding vector \n",
|
||||
"d = np.sum((w2v.vectors-qvec)**2,axis=1)\n",
|
||||
"min_idx = np.argmin(d)\n",
|
||||
"# find the corresponding word\n",
|
||||
"w2v.index_to_key[min_idx]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **NOTE**: আমরা *man* এবং *woman* ভেক্টরগুলিতে একটি ছোট সহগ যোগ করতে হয়েছিল - এটি সরিয়ে দেখুন কী ঘটে।\n",
|
||||
"\n",
|
||||
"নিকটতম ভেক্টর খুঁজে পেতে, আমরা TensorFlow এর মেশিনারি ব্যবহার করে আমাদের ভেক্টর এবং শব্দভাণ্ডারের সমস্ত ভেক্টরের মধ্যে দূরত্বের একটি ভেক্টর গণনা করি, এবং তারপর `argmin` ব্যবহার করে সর্বনিম্ন শব্দের সূচক খুঁজে বের করি।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"যদিও Word2Vec শব্দের অর্থ বোঝানোর জন্য একটি চমৎকার পদ্ধতি বলে মনে হয়, এর বেশ কিছু সীমাবদ্ধতা রয়েছে, যার মধ্যে উল্লেখযোগ্য:\n",
|
||||
"\n",
|
||||
"* CBoW এবং skip-gram উভয় মডেলই **predictive embeddings**, এবং এগুলো শুধুমাত্র স্থানীয় প্রসঙ্গ বিবেচনা করে। Word2Vec বৈশ্বিক প্রসঙ্গের সুবিধা গ্রহণ করে না।\n",
|
||||
"* Word2Vec শব্দের **morphology** বিবেচনা করে না, অর্থাৎ শব্দের অর্থ তার বিভিন্ন অংশ, যেমন মূল শব্দের উপর নির্ভর করতে পারে—এই বিষয়টি উপেক্ষা করে। \n",
|
||||
"\n",
|
||||
"**FastText** দ্বিতীয় সীমাবদ্ধতাটি কাটিয়ে উঠতে চেষ্টা করে এবং Word2Vec-এর উপর ভিত্তি করে প্রতিটি শব্দ এবং প্রতিটি শব্দের মধ্যে থাকা চরিত্র n-grams-এর জন্য ভেক্টর উপস্থাপন শিখে। এই উপস্থাপনাগুলোর মান প্রতিটি প্রশিক্ষণ ধাপে একটি ভেক্টরে গড় করা হয়। যদিও এটি প্রাক-প্রশিক্ষণে অতিরিক্ত গণনার প্রয়োজন করে, এটি শব্দ embeddings-কে সাব-ওয়ার্ড তথ্য এনকোড করতে সক্ষম করে।\n",
|
||||
"\n",
|
||||
"আরেকটি পদ্ধতি, **GloVe**, শব্দ embeddings-এর জন্য একটি ভিন্ন পদ্ধতি ব্যবহার করে, যা শব্দ-প্রসঙ্গ ম্যাট্রিক্সের ফ্যাক্টরাইজেশনের উপর ভিত্তি করে। প্রথমে এটি একটি বড় ম্যাট্রিক্স তৈরি করে, যা বিভিন্ন প্রসঙ্গে শব্দের উপস্থিতির সংখ্যা গণনা করে, এবং তারপর এটি এই ম্যাট্রিক্সকে নিম্ন মাত্রায় উপস্থাপন করার চেষ্টা করে, যাতে পুনর্গঠন ক্ষতি (reconstruction loss) ন্যূনতম হয়।\n",
|
||||
"\n",
|
||||
"gensim লাইব্রেরি এই শব্দ embeddings সমর্থন করে, এবং আপনি উপরের মডেল লোডিং কোড পরিবর্তন করে এগুলোর সাথে পরীক্ষা-নিরীক্ষা করতে পারেন।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## কেরাসে প্রি-ট্রেইনড এমবেডিং ব্যবহার করা\n",
|
||||
"\n",
|
||||
"উপরের উদাহরণটি আমরা পরিবর্তন করে আমাদের এমবেডিং লেয়ারে সেমান্টিক এমবেডিং, যেমন Word2Vec, দিয়ে ম্যাট্রিক্সটি প্রি-পপুলেট করতে পারি। প্রি-ট্রেইনড এমবেডিং এবং টেক্সট কর্পাসের ভোকাবুলারিগুলো সাধারণত একে অপরের সাথে মিলে না, তাই আমাদের একটি বেছে নিতে হবে। এখানে আমরা দুটি সম্ভাব্য বিকল্প নিয়ে আলোচনা করব: টোকেনাইজার ভোকাবুলারি ব্যবহার করা এবং Word2Vec এমবেডিংয়ের ভোকাবুলারি ব্যবহার করা।\n",
|
||||
"\n",
|
||||
"### টোকেনাইজার ভোকাবুলারি ব্যবহার করা\n",
|
||||
"\n",
|
||||
"যখন টোকেনাইজার ভোকাবুলারি ব্যবহার করা হয়, তখন ভোকাবুলারির কিছু শব্দের জন্য Word2Vec এমবেডিং পাওয়া যাবে, এবং কিছু শব্দ অনুপস্থিত থাকবে। আমাদের ভোকাবুলারির আকার যদি `vocab_size` হয় এবং Word2Vec এমবেডিং ভেক্টরের দৈর্ঘ্য যদি `embed_size` হয়, তাহলে এমবেডিং লেয়ারটি একটি ওজন ম্যাট্রিক্স দ্বারা উপস্থাপিত হবে যার আকৃতি হবে `vocab_size`$\\times$`embed_size`। আমরা এই ম্যাট্রিক্সটি ভোকাবুলারির মধ্য দিয়ে গিয়ে পূরণ করব:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Embedding size: 300\n",
|
||||
"Populating matrix, this will take some time...Done, found 4551 words, 784 words missing\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"embed_size = len(w2v.get_vector('hello'))\n",
|
||||
"print(f'Embedding size: {embed_size}')\n",
|
||||
"\n",
|
||||
"vocab = vectorizer.get_vocabulary()\n",
|
||||
"W = np.zeros((vocab_size,embed_size))\n",
|
||||
"print('Populating matrix, this will take some time...',end='')\n",
|
||||
"found, not_found = 0,0\n",
|
||||
"for i,w in enumerate(vocab):\n",
|
||||
" try:\n",
|
||||
" W[i] = w2v.get_vector(w)\n",
|
||||
" found+=1\n",
|
||||
" except:\n",
|
||||
" # W[i] = np.random.normal(0.0,0.3,size=(embed_size,))\n",
|
||||
" not_found+=1\n",
|
||||
"\n",
|
||||
"print(f\"Done, found {found} words, {not_found} words missing\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"শব্দগুলি যদি Word2Vec শব্দভাণ্ডারে উপস্থিত না থাকে, তাহলে আমরা সেগুলিকে শূন্য হিসাবে রাখতে পারি, অথবা একটি র্যান্ডম ভেক্টর তৈরি করতে পারি।\n",
|
||||
"\n",
|
||||
"এখন আমরা প্রি-ট্রেইনড ওজন সহ একটি এমবেডিং লেয়ার সংজ্ঞায়িত করতে পারি:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"emb = keras.layers.Embedding(vocab_size,embed_size,weights=[W],trainable=False)\n",
|
||||
"model = keras.models.Sequential([\n",
|
||||
" vectorizer, emb,\n",
|
||||
" keras.layers.Lambda(lambda x: tf.reduce_mean(x,axis=1)),\n",
|
||||
" keras.layers.Dense(4, activation='softmax')\n",
|
||||
"])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"938/938 [==============================] - 10s 10ms/step - loss: 1.1075 - acc: 0.7822 - val_loss: 0.9134 - val_acc: 0.8175\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.History at 0x2220226ef10>"
|
||||
]
|
||||
},
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'])\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(batch_size),\n",
|
||||
" validation_data=ds_test.map(tupelize).batch(batch_size))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **নোট**: লক্ষ্য করুন যে আমরা `trainable=False` সেট করেছি যখন `Embedding` তৈরি করা হয়েছে, যার অর্থ হলো আমরা Embedding লেয়ারটি পুনরায় প্রশিক্ষণ করছি না। এটি সঠিকতার হারকে সামান্য কমিয়ে দিতে পারে, তবে এটি প্রশিক্ষণের গতি বাড়ায়।\n",
|
||||
"\n",
|
||||
"### এমবেডিং ভোকাবুলারি ব্যবহার করা\n",
|
||||
"\n",
|
||||
"পূর্ববর্তী পদ্ধতির একটি সমস্যা হলো TextVectorization এবং Embedding-এ ব্যবহৃত ভোকাবুলারিগুলো ভিন্ন। এই সমস্যার সমাধান করতে, আমরা নিম্নলিখিত সমাধানগুলোর একটি ব্যবহার করতে পারি:\n",
|
||||
"* আমাদের ভোকাবুলারির উপর Word2Vec মডেলটি পুনরায় প্রশিক্ষণ করা।\n",
|
||||
"* প্রি-ট্রেইনড Word2Vec মডেলের ভোকাবুলারি ব্যবহার করে আমাদের ডেটাসেট লোড করা। ডেটাসেট লোড করার সময় ব্যবহৃত ভোকাবুলারি নির্দিষ্ট করা যেতে পারে।\n",
|
||||
"\n",
|
||||
"দ্বিতীয় পদ্ধতিটি সহজ মনে হচ্ছে, তাই আমরা এটি বাস্তবায়ন করব। প্রথমেই, আমরা Word2Vec এমবেডিং থেকে নেওয়া নির্দিষ্ট ভোকাবুলারি দিয়ে একটি `TextVectorization` লেয়ার তৈরি করব:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"vocab = list(w2v.vocab.keys())\n",
|
||||
"vectorizer = keras.layers.experimental.preprocessing.TextVectorization(input_shape=(1,))\n",
|
||||
"vectorizer.set_vocabulary(vocab)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"জেনসিম ওয়ার্ড এম্বেডিংস লাইব্রেরিতে একটি সুবিধাজনক ফাংশন, `get_keras_embeddings`, রয়েছে, যা আপনার জন্য স্বয়ংক্রিয়ভাবে সংশ্লিষ্ট কেরাস এম্বেডিংস লেয়ার তৈরি করবে।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch 1/5\n",
|
||||
"938/938 [==============================] - 20s 14ms/step - loss: 1.3377 - acc: 0.4978 - val_loss: 1.2995 - val_acc: 0.5647\n",
|
||||
"Epoch 2/5\n",
|
||||
"938/938 [==============================] - 10s 10ms/step - loss: 1.2587 - acc: 0.5722 - val_loss: 1.2339 - val_acc: 0.5842\n",
|
||||
"Epoch 3/5\n",
|
||||
"938/938 [==============================] - 10s 10ms/step - loss: 1.1980 - acc: 0.5884 - val_loss: 1.1826 - val_acc: 0.5954\n",
|
||||
"Epoch 4/5\n",
|
||||
"938/938 [==============================] - 12s 13ms/step - loss: 1.1503 - acc: 0.6002 - val_loss: 1.1417 - val_acc: 0.6018\n",
|
||||
"Epoch 5/5\n",
|
||||
"938/938 [==============================] - 11s 12ms/step - loss: 1.1120 - acc: 0.6097 - val_loss: 1.1083 - val_acc: 0.6104\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.History at 0x2220ccb81c0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model = keras.models.Sequential([\n",
|
||||
" vectorizer, \n",
|
||||
" w2v.get_keras_embedding(train_embeddings=False),\n",
|
||||
" keras.layers.Lambda(lambda x: tf.reduce_mean(x,axis=1)),\n",
|
||||
" keras.layers.Dense(4, activation='softmax')\n",
|
||||
"])\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'])\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(128),validation_data=ds_test.map(tupelize).batch(128),epochs=5)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"আমাদের উচ্চতর সঠিকতা দেখতে না পাওয়ার একটি কারণ হল আমাদের ডেটাসেটের কিছু শব্দ প্রি-ট্রেইনড GloVe ভোকাবুলারিতে অনুপস্থিত, এবং তাই সেগুলি কার্যত উপেক্ষা করা হয়। এটি অতিক্রম করতে, আমরা আমাদের ডেটাসেটের উপর ভিত্তি করে আমাদের নিজস্ব এম্বেডিংস প্রশিক্ষণ করতে পারি।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## প্রসঙ্গগত এম্বেডিংস\n",
|
||||
"\n",
|
||||
"প্রথাগত প্রি-ট্রেইনড এম্বেডিং প্রতিনিধিত্ব যেমন Word2Vec-এর একটি প্রধান সীমাবদ্ধতা হলো, যদিও এটি একটি শব্দের কিছু অর্থ ধারণ করতে পারে, এটি বিভিন্ন অর্থের মধ্যে পার্থক্য করতে পারে না। এটি ডাউনস্ট্রিম মডেলগুলিতে সমস্যা সৃষ্টি করতে পারে।\n",
|
||||
"\n",
|
||||
"উদাহরণস্বরূপ, 'play' শব্দটির এই দুটি বাক্যে ভিন্ন অর্থ রয়েছে:\n",
|
||||
"- আমি থিয়েটারে একটি **play** দেখতে গিয়েছিলাম।\n",
|
||||
"- জন তার বন্ধুদের সাথে **play** করতে চায়।\n",
|
||||
"\n",
|
||||
"আমরা যে প্রি-ট্রেইনড এম্বেডিংস নিয়ে আলোচনা করেছি, তা 'play' শব্দটির উভয় অর্থকে একই এম্বেডিংয়ে উপস্থাপন করে। এই সীমাবদ্ধতা কাটিয়ে উঠতে, আমাদের **ভাষা মডেল**-এর উপর ভিত্তি করে এম্বেডিং তৈরি করতে হবে, যা একটি বৃহৎ টেক্সট কর্পাসে প্রশিক্ষিত হয় এবং *জানে* কীভাবে শব্দগুলো বিভিন্ন প্রসঙ্গে একত্রিত হতে পারে। প্রসঙ্গগত এম্বেডিংস নিয়ে আলোচনা এই টিউটোরিয়ালের পরিধির বাইরে, তবে আমরা পরবর্তী ইউনিটে ভাষা মডেল নিয়ে আলোচনা করার সময় এ বিষয়ে ফিরে আসব।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**অস্বীকৃতি**: \nএই নথিটি AI অনুবাদ পরিষেবা [Co-op Translator](https://github.com/Azure/co-op-translator) ব্যবহার করে অনুবাদ করা হয়েছে। আমরা যথাসম্ভব সঠিকতার জন্য চেষ্টা করি, তবে অনুগ্রহ করে মনে রাখবেন যে স্বয়ংক্রিয় অনুবাদে ত্রুটি বা অসঙ্গতি থাকতে পারে। মূল ভাষায় থাকা নথিটিকে প্রামাণিক উৎস হিসেবে বিবেচনা করা উচিত। গুরুত্বপূর্ণ তথ্যের জন্য, পেশাদার মানব অনুবাদ সুপারিশ করা হয়। এই অনুবাদ ব্যবহারের ফলে কোনো ভুল বোঝাবুঝি বা ভুল ব্যাখ্যা হলে আমরা দায়বদ্ধ থাকব না।\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5"
|
||||
},
|
||||
"kernel_info": {
|
||||
"name": "conda-env-py37_tensorflow-py"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "py37_tensorflow",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"nteract": {
|
||||
"version": "nteract-front-end@1.0.0"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "b859482be7f61d1eadc2c6a2720a37e4",
|
||||
"translation_date": "2025-08-28T12:24:02+00:00",
|
||||
"source_file": "lessons/5-NLP/14-Embeddings/EmbeddingsTF.ipynb",
|
||||
"language_code": "bn"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
|
|
@ -0,0 +1,576 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "NXTSugt6ieXh"
|
||||
},
|
||||
"source": [
|
||||
"## CBoW মডেল প্রশিক্ষণ\n",
|
||||
"\n",
|
||||
"এই নোটবুকটি [AI for Beginners Curriculum](http://aka.ms/ai-beginners)-এর একটি অংশ।\n",
|
||||
"\n",
|
||||
"এই উদাহরণে, আমরা CBoW ভাষা মডেল প্রশিক্ষণ করার মাধ্যমে আমাদের নিজস্ব Word2Vec এম্বেডিং স্পেস তৈরি করার প্রক্রিয়া দেখব। আমরা AG News ডেটাসেটকে টেক্সটের উৎস হিসেবে ব্যবহার করব।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"import torchtext\n",
|
||||
"import os\n",
|
||||
"import collections\n",
|
||||
"import builtins\n",
|
||||
"import random\n",
|
||||
"import numpy as np"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "q-UiiJUKaxHj"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "TFbR8CZaTZ1q"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"প্রথমে চলুন আমাদের ডেটাসেট লোড করি এবং টোকেনাইজার ও শব্দভাণ্ডার সংজ্ঞায়িত করি। আমরা `vocab_size` 5000 সেট করব যাতে গণনা কিছুটা সীমিত করা যায়।\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "HIwC7lI5T-ov"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def load_dataset(ngrams = 1, min_freq = 1, vocab_size = 5000 , lines_cnt = 500):\n",
|
||||
" tokenizer = torchtext.data.utils.get_tokenizer('basic_english')\n",
|
||||
" print(\"Loading dataset...\")\n",
|
||||
" test_dataset, train_dataset = torchtext.datasets.AG_NEWS(root='./data')\n",
|
||||
" train_dataset = list(train_dataset)\n",
|
||||
" test_dataset = list(test_dataset)\n",
|
||||
" classes = ['World', 'Sports', 'Business', 'Sci/Tech']\n",
|
||||
" print('Building vocab...')\n",
|
||||
" counter = collections.Counter()\n",
|
||||
" for i, (_, line) in enumerate(train_dataset):\n",
|
||||
" counter.update(torchtext.data.utils.ngrams_iterator(tokenizer(line),ngrams=ngrams))\n",
|
||||
" if i == lines_cnt:\n",
|
||||
" break\n",
|
||||
" vocab = torchtext.vocab.Vocab(collections.Counter(dict(counter.most_common(vocab_size))), min_freq=min_freq)\n",
|
||||
" return train_dataset, test_dataset, classes, vocab, tokenizer"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "wdZuygtgiuLG"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"train_dataset, test_dataset, _, vocab, tokenizer = load_dataset()"
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "4d1nU1gsivGu",
|
||||
"outputId": "949fe272-ae0e-49f5-c373-6703458b3a74"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
"Loading dataset...\n",
|
||||
"Building vocab...\n"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def encode(x, vocabulary, tokenizer = tokenizer):\n",
|
||||
" return [vocabulary[s] for s in tokenizer(x)]"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "1XDYNhG8ToFV"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "LIlQk6_PaHVY"
|
||||
},
|
||||
"source": [
|
||||
"## CBoW মডেল\n",
|
||||
"\n",
|
||||
"CBoW $2N$ প্রতিবেশী শব্দের ভিত্তিতে একটি শব্দ অনুমান করতে শেখে। উদাহরণস্বরূপ, যখন $N=1$, তখন আমরা বাক্য *I like to train networks* থেকে নিম্নলিখিত জোড়াগুলি পাব: (like,I), (I, like), (to, like), (like,to), (train,to), (to, train), (networks, train), (train,networks)। এখানে, প্রথম শব্দটি প্রতিবেশী শব্দ যা ইনপুট হিসেবে ব্যবহৃত হয়, এবং দ্বিতীয় শব্দটি হলো যেটি আমরা অনুমান করছি।\n",
|
||||
"\n",
|
||||
"পরবর্তী শব্দ অনুমান করার জন্য একটি নেটওয়ার্ক তৈরি করতে, আমাদের প্রতিবেশী শব্দ ইনপুট হিসেবে সরবরাহ করতে হবে এবং শব্দের সংখ্যা আউটপুট হিসেবে পেতে হবে। CBoW নেটওয়ার্কের স্থাপত্য নিম্নরূপ:\n",
|
||||
"\n",
|
||||
"* ইনপুট শব্দটি এমবেডিং লেয়ারের মধ্য দিয়ে পাঠানো হয়। এই এমবেডিং লেয়ারটি আমাদের Word2Vec এমবেডিং হবে, তাই আমরা এটিকে আলাদাভাবে `embedder` ভেরিয়েবল হিসেবে সংজ্ঞায়িত করব। এই উদাহরণে আমরা এমবেডিং সাইজ = 30 ব্যবহার করব, যদিও আপনি উচ্চতর মাত্রা নিয়ে পরীক্ষা করতে পারেন (বাস্তব Word2Vec-এ এটি 300)।\n",
|
||||
"* এমবেডিং ভেক্টরটি এরপর একটি লিনিয়ার লেয়ারের মধ্য দিয়ে যাবে যা আউটপুট শব্দ অনুমান করবে। সুতরাং এতে `vocab_size` নিউরন থাকবে।\n",
|
||||
"\n",
|
||||
"আউটপুটের জন্য, যদি আমরা `CrossEntropyLoss` লস ফাংশন ব্যবহার করি, তাহলে আমাদের কেবল শব্দের সংখ্যা প্রত্যাশিত ফলাফল হিসেবে সরবরাহ করতে হবে, এক-হট এনকোডিং ছাড়াই।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"vocab_size = len(vocab)\n",
|
||||
"\n",
|
||||
"embedder = torch.nn.Embedding(num_embeddings = vocab_size, embedding_dim = 30)\n",
|
||||
"model = torch.nn.Sequential(\n",
|
||||
" embedder,\n",
|
||||
" torch.nn.Linear(in_features = 30, out_features = vocab_size),\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print(model)"
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "akKTcKQKkfl2",
|
||||
"outputId": "da687e3e-a8ec-4c1a-e456-ab8cd6ac7dad"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
"Sequential(\n",
|
||||
" (0): Embedding(5002, 30)\n",
|
||||
" (1): Linear(in_features=30, out_features=5002, bias=True)\n",
|
||||
")\n"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "Nud6jgGPaHVa"
|
||||
},
|
||||
"source": [
|
||||
"## প্রশিক্ষণের ডেটা প্রস্তুত করা\n",
|
||||
"\n",
|
||||
"এখন আমরা প্রধান ফাংশনটি প্রোগ্রাম করব যা টেক্সট থেকে CBoW শব্দ জোড়া গণনা করবে। এই ফাংশনটি আমাদের উইন্ডো সাইজ নির্ধারণ করতে দেবে এবং একটি জোড়ার সেট - ইনপুট এবং আউটপুট শব্দ ফেরত দেবে। মনে রাখবেন, এই ফাংশনটি শব্দের উপর যেমন কাজ করতে পারে, তেমনি ভেক্টর/টেনসরের উপরও কাজ করতে পারে - যা আমাদের টেক্সট এনকোড করতে দেবে, `to_cbow` ফাংশনে পাঠানোর আগে।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "x-dsXygOieXn",
|
||||
"outputId": "c2218280-e540-40ba-9546-efe48d0d714f"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
"[['like', 'I'], ['to', 'I'], ['I', 'like'], ['to', 'like'], ['train', 'like'], ['I', 'to'], ['like', 'to'], ['train', 'to'], ['networks', 'to'], ['like', 'train'], ['to', 'train'], ['networks', 'train'], ['to', 'networks'], ['train', 'networks']]\n",
|
||||
"[[232, 172], [5, 172], [172, 232], [5, 232], [0, 232], [172, 5], [232, 5], [0, 5], [1202, 5], [232, 0], [5, 0], [1202, 0], [5, 1202], [0, 1202]]\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def to_cbow(sent,window_size=2):\n",
|
||||
" res = []\n",
|
||||
" for i,x in enumerate(sent):\n",
|
||||
" for j in range(max(0,i-window_size),min(i+window_size+1,len(sent))):\n",
|
||||
" if i!=j:\n",
|
||||
" res.append([sent[j],x])\n",
|
||||
" return res\n",
|
||||
"\n",
|
||||
"print(to_cbow(['I','like','to','train','networks']))\n",
|
||||
"print(to_cbow(encode('I like to train networks', vocab)))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "XVaaDLjaaHVb"
|
||||
},
|
||||
"source": [
|
||||
"চলুন প্রশিক্ষণের ডেটাসেট প্রস্তুত করি। আমরা সমস্ত খবরের মধ্য দিয়ে যাব, `to_cbow` কল করব শব্দ জোড়ার তালিকা পেতে, এবং সেই জোড়াগুলো `X` এবং `Y` তে যোগ করব। সময় বাঁচানোর জন্য, আমরা শুধুমাত্র প্রথম ১০ হাজার খবর বিবেচনা করব - আপনি সহজেই এই সীমাবদ্ধতা সরিয়ে ফেলতে পারেন যদি আপনার অপেক্ষা করার জন্য বেশি সময় থাকে এবং আরও ভালো এম্বেডিং পেতে চান :)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "54b-Gd9TieXo"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"X = []\n",
|
||||
"Y = []\n",
|
||||
"for i, x in zip(range(10000), train_dataset):\n",
|
||||
" for w1, w2 in to_cbow(encode(x[1], vocab), window_size = 5):\n",
|
||||
" X.append(w1)\n",
|
||||
" Y.append(w2)\n",
|
||||
"\n",
|
||||
"X = torch.tensor(X)\n",
|
||||
"Y = torch.tensor(Y)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"আমরা সেই ডেটাকে এক ডেটাসেটে রূপান্তর করব এবং ডেটালোডার তৈরি করব:\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "cwWy0PzXWhN5"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"class SimpleIterableDataset(torch.utils.data.IterableDataset):\n",
|
||||
" def __init__(self, X, Y):\n",
|
||||
" super(SimpleIterableDataset).__init__()\n",
|
||||
" self.data = []\n",
|
||||
" for i in range(len(X)):\n",
|
||||
" self.data.append( (Y[i], X[i]) )\n",
|
||||
" random.shuffle(self.data)\n",
|
||||
"\n",
|
||||
" def __iter__(self):\n",
|
||||
" return iter(self.data)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "mfoAcGPFZU8p"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "e4NQ_-5waHVc"
|
||||
},
|
||||
"source": [
|
||||
"আমরা সেই ডেটাকে এক ডেটাসেটে রূপান্তর করব এবং ডেটালোডার তৈরি করব:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "AbLUcojlieXo"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"ds = SimpleIterableDataset(X, Y)\n",
|
||||
"dl = torch.utils.data.DataLoader(ds, batch_size = 256)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "pKQr7sXeaHVc"
|
||||
},
|
||||
"source": [
|
||||
"এখন আসল প্রশিক্ষণ শুরু করা যাক। আমরা `SGD` অপ্টিমাইজার ব্যবহার করব উচ্চ লার্নিং রেট সহ। আপনি `Adam` এর মতো অন্যান্য অপ্টিমাইজার নিয়ে পরীক্ষা করতে পারেন। আমরা শুরুতে ১০ ইপোকের জন্য প্রশিক্ষণ করব - এবং যদি আপনি আরও কম ক্ষতি চান তবে এই সেলটি পুনরায় চালাতে পারেন।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def train_epoch(net, dataloader, lr = 0.01, optimizer = None, loss_fn = torch.nn.CrossEntropyLoss(), epochs = None, report_freq = 1):\n",
|
||||
" optimizer = optimizer or torch.optim.Adam(net.parameters(), lr = lr)\n",
|
||||
" loss_fn = loss_fn.to(device)\n",
|
||||
" net.train()\n",
|
||||
"\n",
|
||||
" for i in range(epochs):\n",
|
||||
" total_loss, j = 0, 0, \n",
|
||||
" for labels, features in dataloader:\n",
|
||||
" optimizer.zero_grad()\n",
|
||||
" features, labels = features.to(device), labels.to(device)\n",
|
||||
" out = net(features)\n",
|
||||
" loss = loss_fn(out, labels)\n",
|
||||
" loss.backward()\n",
|
||||
" optimizer.step()\n",
|
||||
" total_loss += loss\n",
|
||||
" j += 1\n",
|
||||
" if i % report_freq == 0:\n",
|
||||
" print(f\"Epoch: {i+1}: loss={total_loss.item()/j}\")\n",
|
||||
"\n",
|
||||
" return total_loss.item()/j"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "HeeCYKr_KF1w"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"train_epoch(net = model, dataloader = dl, optimizer = torch.optim.SGD(model.parameters(), lr = 0.1), loss_fn = torch.nn.CrossEntropyLoss(), epochs = 10)"
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "KVgwGtDHgDlT",
|
||||
"outputId": "2447833f-f0e3-4566-c33d-addbfe2f451d"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
"Epoch: 1: loss=5.664632366860172\n",
|
||||
"Epoch: 2: loss=5.632101973960962\n",
|
||||
"Epoch: 3: loss=5.610399051405015\n",
|
||||
"Epoch: 4: loss=5.594621561080262\n",
|
||||
"Epoch: 5: loss=5.582538017415446\n",
|
||||
"Epoch: 6: loss=5.572900234519603\n",
|
||||
"Epoch: 7: loss=5.564951676341915\n",
|
||||
"Epoch: 8: loss=5.558288112064614\n",
|
||||
"Epoch: 9: loss=5.552576955031129\n",
|
||||
"Epoch: 10: loss=5.547634165194347\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"output_type": "execute_result",
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"5.547634165194347"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"execution_count": 16
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "W8u2qXZmaHVd"
|
||||
},
|
||||
"source": [
|
||||
"## Word2Vec চেষ্টা করা\n",
|
||||
"\n",
|
||||
"Word2Vec ব্যবহার করতে, চলুন আমাদের শব্দভাণ্ডারের সমস্ত শব্দের জন্য ভেক্টর বের করি:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "r8TatcXjkU_t"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"vectors = torch.stack([embedder(torch.tensor(vocab[s])) for s in vocab.itos], 0)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "3OcX21UOaHVd"
|
||||
},
|
||||
"source": [
|
||||
"চলুন দেখি, উদাহরণস্বরূপ, কীভাবে **Paris** শব্দটি একটি ভেক্টরে এনকোড করা হয়:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "bz6tAeLzieXp",
|
||||
"outputId": "5b20850e-4342-45e9-f840-cfac2b4d61d8"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
"tensor([-0.0915, 2.1224, -0.0281, -0.6819, 1.1219, 0.6458, -1.3704, -1.3314,\n",
|
||||
" -1.1437, 0.4496, 0.2301, -0.3515, -0.8485, 1.0481, 0.4386, -0.8949,\n",
|
||||
" 0.5644, 1.0939, -2.5096, 3.2949, -0.2601, -0.8640, 0.1421, -0.0804,\n",
|
||||
" -0.5083, -1.0560, 0.9753, -0.5949, -1.6046, 0.5774],\n",
|
||||
" grad_fn=<EmbeddingBackward>)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"paris_vec = embedder(torch.tensor(vocab['paris']))\n",
|
||||
"print(paris_vec)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "pHTJlaeYaHVd"
|
||||
},
|
||||
"source": [
|
||||
"Word2Vec ব্যবহার করে সমার্থক শব্দ খুঁজে বের করা বেশ আকর্ষণীয়। নিম্নলিখিত ফাংশনটি একটি প্রদত্ত ইনপুটের জন্য `n` টি নিকটতম শব্দ ফেরত দেবে। এগুলো খুঁজে বের করার জন্য আমরা $|w_i - v|$ এর নর্ম গণনা করি, যেখানে $v$ আমাদের ইনপুট শব্দের সাথে সম্পর্কিত ভেক্টর এবং $w_i$ শব্দভাণ্ডারের $i$-তম শব্দের এনকোডিং। এরপর আমরা অ্যারে সাজাই এবং `argsort` ব্যবহার করে সংশ্লিষ্ট সূচকগুলি ফেরত দিই, এবং তালিকার প্রথম `n` উপাদান গ্রহণ করি, যা শব্দভাণ্ডারে নিকটতম শব্দগুলির অবস্থান এনকোড করে।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "NlZyi-_olFar",
|
||||
"outputId": "b5dbb163-88c4-4d5a-eaf2-6751f700e98c"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "execute_result",
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"['microsoft', 'quoted', 'lp', 'rate', 'top']"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"execution_count": 56
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def close_words(x, n = 5):\n",
|
||||
" vec = embedder(torch.tensor(vocab[x]))\n",
|
||||
" top5 = np.linalg.norm(vectors.detach().numpy() - vec.detach().numpy(), axis = 1).argsort()[:n]\n",
|
||||
" return [ vocab.itos[x] for x in top5 ]\n",
|
||||
"\n",
|
||||
"close_words('microsoft')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "-dQq7xeAln0U",
|
||||
"outputId": "66f768c3-c248-4bfd-ce4f-c8ffc6d0dd0d"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "execute_result",
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"['basketball', 'lot', 'sinai', 'states', 'healthdaynews']"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"execution_count": 51
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"close_words('basketball')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "fJXqK26b29sa",
|
||||
"outputId": "78f0baba-ffd0-485a-dd87-0a12bedfd7fa"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "execute_result",
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"['funds', 'travel', 'sydney', 'japan', 'business']"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"execution_count": 77
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"close_words('funds')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "My0VeTDd3Ji8"
|
||||
},
|
||||
"source": [
|
||||
"## মূল কথা\n",
|
||||
"\n",
|
||||
"CBoW-এর মতো চতুর কৌশল ব্যবহার করে আমরা Word2Vec মডেল প্রশিক্ষণ দিতে পারি। আপনি চাইলে skip-gram মডেলও প্রশিক্ষণ দিতে পারেন, যা কেন্দ্রীয় শব্দটি দেওয়া হলে পার্শ্ববর্তী শব্দটি পূর্বাভাস দিতে শেখে, এবং এটি কতটা ভালো কাজ করে তা দেখতে পারেন।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**অস্বীকৃতি**: \nএই নথিটি AI অনুবাদ পরিষেবা [Co-op Translator](https://github.com/Azure/co-op-translator) ব্যবহার করে অনুবাদ করা হয়েছে। আমরা যথাসম্ভব সঠিকতার জন্য চেষ্টা করি, তবে অনুগ্রহ করে মনে রাখবেন যে স্বয়ংক্রিয় অনুবাদে ত্রুটি বা অসঙ্গতি থাকতে পারে। মূল ভাষায় থাকা নথিটিকে প্রামাণিক উৎস হিসেবে বিবেচনা করা উচিত। গুরুত্বপূর্ণ তথ্যের জন্য, পেশাদার মানব অনুবাদ সুপারিশ করা হয়। এই অনুবাদ ব্যবহারের ফলে কোনো ভুল বোঝাবুঝি বা ভুল ব্যাখ্যা হলে আমরা দায়বদ্ধ থাকব না।\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"collapsed_sections": [],
|
||||
"name": "CBoW-PyTorch.ipynb",
|
||||
"provenance": []
|
||||
},
|
||||
"interpreter": {
|
||||
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.8.12 ('py38')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"orig_nbformat": 4,
|
||||
"gpuClass": "standard",
|
||||
"coopTranslator": {
|
||||
"original_hash": "36df28efe3fe40b6fb0a7fa48fe3ea82",
|
||||
"translation_date": "2025-08-28T12:04:11+00:00",
|
||||
"source_file": "lessons/5-NLP/15-LanguageModeling/CBoW-PyTorch.ipynb",
|
||||
"language_code": "bn"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,479 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# পুনরাবৃত্তিমূলক নিউরাল নেটওয়ার্ক\n",
|
||||
"\n",
|
||||
"পূর্ববর্তী মডিউলে, আমরা পাঠ্যের সমৃদ্ধ অর্থবোধক উপস্থাপন ব্যবহার করেছি এবং এম্বেডিংয়ের উপর একটি সাধারণ লিনিয়ার ক্লাসিফায়ার প্রয়োগ করেছি। এই স্থাপত্যটি বাক্যের মধ্যে শব্দগুলোর সম্মিলিত অর্থ ধারণ করে, কিন্তু এটি শব্দগুলোর **ক্রম** বিবেচনা করে না, কারণ এম্বেডিংয়ের উপর সংযোজন অপারেশন মূল পাঠ্য থেকে এই তথ্যটি সরিয়ে দেয়। যেহেতু এই মডেলগুলো শব্দের ক্রম মডেল করতে অক্ষম, তাই তারা পাঠ্য তৈরি বা প্রশ্নের উত্তর দেওয়ার মতো আরও জটিল বা অস্পষ্ট কাজ সমাধান করতে পারে না।\n",
|
||||
"\n",
|
||||
"পাঠ্য ক্রমের অর্থ ধারণ করতে, আমাদের আরেকটি নিউরাল নেটওয়ার্ক স্থাপত্য ব্যবহার করতে হবে, যেটিকে **পুনরাবৃত্তিমূলক নিউরাল নেটওয়ার্ক** বা RNN বলা হয়। RNN-এ, আমরা আমাদের বাক্যটি নেটওয়ার্কের মাধ্যমে একবারে একটি প্রতীক পাঠাই, এবং নেটওয়ার্ক কিছু **অবস্থা** তৈরি করে, যা আমরা পরবর্তী প্রতীকের সাথে আবার নেটওয়ার্কে পাঠাই।\n",
|
||||
"\n",
|
||||
"দেওয়া ইনপুট টোকেনের ক্রম $X_0,\\dots,X_n$, RNN একটি নিউরাল নেটওয়ার্ক ব্লকের ক্রম তৈরি করে এবং এই ক্রমটি ব্যাক প্রোপাগেশনের মাধ্যমে সম্পূর্ণভাবে প্রশিক্ষণ দেয়। প্রতিটি নেটওয়ার্ক ব্লক $(X_i,S_i)$ জোড়াকে ইনপুট হিসেবে গ্রহণ করে এবং $S_{i+1}$ ফলাফল হিসেবে তৈরি করে। চূড়ান্ত অবস্থা $S_n$ বা আউটপুট $X_n$ একটি লিনিয়ার ক্লাসিফায়ারে যায় ফলাফল তৈরি করতে। সমস্ত নেটওয়ার্ক ব্লক একই ওজন শেয়ার করে এবং একবার ব্যাক প্রোপাগেশনের মাধ্যমে সম্পূর্ণভাবে প্রশিক্ষণ দেওয়া হয়।\n",
|
||||
"\n",
|
||||
"যেহেতু অবস্থা ভেক্টর $S_0,\\dots,S_n$ নেটওয়ার্কের মাধ্যমে পাঠানো হয়, এটি শব্দগুলোর মধ্যে ক্রমাগত নির্ভরতা শিখতে সক্ষম হয়। উদাহরণস্বরূপ, যখন *not* শব্দটি ক্রমের কোথাও উপস্থিত হয়, এটি অবস্থা ভেক্টরের নির্দিষ্ট উপাদানগুলোকে অস্বীকার করতে শিখতে পারে, যার ফলে নেতিবাচকতা তৈরি হয়।\n",
|
||||
"\n",
|
||||
"> যেহেতু ছবিতে সমস্ত RNN ব্লকের ওজন শেয়ার করা হয়েছে, একই ছবি একটি ব্লক (ডানদিকে) হিসেবে উপস্থাপন করা যেতে পারে একটি পুনরাবৃত্তিমূলক ফিডব্যাক লুপ সহ, যা নেটওয়ার্কের আউটপুট অবস্থা ইনপুটে ফেরত পাঠায়।\n",
|
||||
"\n",
|
||||
"চলুন দেখি কীভাবে পুনরাবৃত্তিমূলক নিউরাল নেটওয়ার্ক আমাদের নিউজ ডেটাসেট শ্রেণীবদ্ধ করতে সাহায্য করতে পারে।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Loading dataset...\n",
|
||||
"Building vocab...\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"import torchtext\n",
|
||||
"from torchnlp import *\n",
|
||||
"train_dataset, test_dataset, classes, vocab = load_dataset()\n",
|
||||
"vocab_size = len(vocab)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## সহজ RNN শ্রেণীবিভাজক\n",
|
||||
"\n",
|
||||
"সহজ RNN-এর ক্ষেত্রে, প্রতিটি পুনরাবৃত্তি ইউনিট একটি সাধারণ লিনিয়ার নেটওয়ার্ক, যা সংযুক্ত ইনপুট ভেক্টর এবং স্টেট ভেক্টর গ্রহণ করে এবং একটি নতুন স্টেট ভেক্টর তৈরি করে। PyTorch এই ইউনিটকে `RNNCell` ক্লাস দিয়ে উপস্থাপন করে, এবং এই ধরনের সেলগুলোর একটি নেটওয়ার্ককে `RNN` লেয়ার হিসেবে উপস্থাপন করে।\n",
|
||||
"\n",
|
||||
"একটি RNN শ্রেণীবিভাজক সংজ্ঞায়িত করতে, আমরা প্রথমে একটি এমবেডিং লেয়ার প্রয়োগ করব যাতে ইনপুট শব্দভাণ্ডারের মাত্রা কমানো যায়, এবং তারপর এর উপরে একটি RNN লেয়ার থাকবে:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class RNNClassifier(torch.nn.Module):\n",
|
||||
" def __init__(self, vocab_size, embed_dim, hidden_dim, num_class):\n",
|
||||
" super().__init__()\n",
|
||||
" self.hidden_dim = hidden_dim\n",
|
||||
" self.embedding = torch.nn.Embedding(vocab_size, embed_dim)\n",
|
||||
" self.rnn = torch.nn.RNN(embed_dim,hidden_dim,batch_first=True)\n",
|
||||
" self.fc = torch.nn.Linear(hidden_dim, num_class)\n",
|
||||
"\n",
|
||||
" def forward(self, x):\n",
|
||||
" batch_size = x.size(0)\n",
|
||||
" x = self.embedding(x)\n",
|
||||
" x,h = self.rnn(x)\n",
|
||||
" return self.fc(x.mean(dim=1))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **নোট:** এখানে আমরা সরলতার জন্য অপ্রশিক্ষিত এম্বেডিং লেয়ার ব্যবহার করছি, তবে আরও ভালো ফলাফলের জন্য আমরা পূর্বে বর্ণিত ইউনিটে Word2Vec বা GloVe এম্বেডিং সহ পূর্ব-প্রশিক্ষিত এম্বেডিং লেয়ার ব্যবহার করতে পারি। আরও ভালোভাবে বোঝার জন্য, আপনি এই কোডটি পূর্ব-প্রশিক্ষিত এম্বেডিংয়ের সাথে কাজ করার জন্য মানিয়ে নিতে চাইতে পারেন।\n",
|
||||
"\n",
|
||||
"আমাদের ক্ষেত্রে, আমরা প্যাডেড ডেটা লোডার ব্যবহার করব, যাতে প্রতিটি ব্যাচে একই দৈর্ঘ্যের প্যাডেড সিকোয়েন্সের সংখ্যা থাকবে। RNN লেয়ার এম্বেডিং টেনসরগুলোর সিকোয়েন্স গ্রহণ করবে এবং দুটি আউটপুট তৈরি করবে:\n",
|
||||
"* $x$ হলো প্রতিটি ধাপে RNN সেলের আউটপুটের সিকোয়েন্স\n",
|
||||
"* $h$ হলো সিকোয়েন্সের শেষ উপাদানের জন্য চূড়ান্ত হিডেন স্টেট\n",
|
||||
"\n",
|
||||
"এরপর আমরা একটি সম্পূর্ণ-সংযুক্ত লিনিয়ার ক্লাসিফায়ার প্রয়োগ করব ক্লাসের সংখ্যা পেতে।\n",
|
||||
"\n",
|
||||
"> **নোট:** RNN প্রশিক্ষণ করা বেশ কঠিন, কারণ একবার RNN সেলগুলো সিকোয়েন্সের দৈর্ঘ্য বরাবর আনরোল করা হলে, ব্যাক প্রোপাগেশনে জড়িত স্তরের সংখ্যা বেশ বড় হয়ে যায়। তাই আমাদের ছোট লার্নিং রেট নির্বাচন করতে হবে এবং ভালো ফলাফল পেতে বড় ডেটাসেটে নেটওয়ার্ক প্রশিক্ষণ দিতে হবে। এটি করতে বেশ সময় লাগতে পারে, তাই GPU ব্যবহার করা উত্তম।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.3090625\n",
|
||||
"6400: acc=0.38921875\n",
|
||||
"9600: acc=0.4590625\n",
|
||||
"12800: acc=0.511953125\n",
|
||||
"16000: acc=0.5506875\n",
|
||||
"19200: acc=0.57921875\n",
|
||||
"22400: acc=0.6070089285714285\n",
|
||||
"25600: acc=0.6304296875\n",
|
||||
"28800: acc=0.6484027777777778\n",
|
||||
"32000: acc=0.66509375\n",
|
||||
"35200: acc=0.6790056818181818\n",
|
||||
"38400: acc=0.6929166666666666\n",
|
||||
"41600: acc=0.7035817307692308\n",
|
||||
"44800: acc=0.7137276785714286\n",
|
||||
"48000: acc=0.72225\n",
|
||||
"51200: acc=0.73001953125\n",
|
||||
"54400: acc=0.7372794117647059\n",
|
||||
"57600: acc=0.7436631944444444\n",
|
||||
"60800: acc=0.7503947368421052\n",
|
||||
"64000: acc=0.75634375\n",
|
||||
"67200: acc=0.7615773809523809\n",
|
||||
"70400: acc=0.7662642045454545\n",
|
||||
"73600: acc=0.7708423913043478\n",
|
||||
"76800: acc=0.7751822916666666\n",
|
||||
"80000: acc=0.7790625\n",
|
||||
"83200: acc=0.7825\n",
|
||||
"86400: acc=0.7858564814814815\n",
|
||||
"89600: acc=0.7890513392857142\n",
|
||||
"92800: acc=0.7920474137931034\n",
|
||||
"96000: acc=0.7952708333333334\n",
|
||||
"99200: acc=0.7982258064516129\n",
|
||||
"102400: acc=0.80099609375\n",
|
||||
"105600: acc=0.8037594696969697\n",
|
||||
"108800: acc=0.8060569852941176\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=padify, shuffle=True)\n",
|
||||
"net = RNNClassifier(vocab_size,64,32,len(classes)).to(device)\n",
|
||||
"train_epoch(net,train_loader, lr=0.001)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## লং শর্ট টার্ম মেমরি (LSTM)\n",
|
||||
"\n",
|
||||
"ক্লাসিক্যাল RNN-এর প্রধান সমস্যাগুলোর একটি হলো **vanishing gradients** সমস্যা। যেহেতু RNN-গুলো একবারে এক ব্যাক-প্রোপাগেশন পাসে এন্ড-টু-এন্ড প্রশিক্ষিত হয়, এটি নেটওয়ার্কের প্রথম স্তরগুলোতে ত্রুটি প্রেরণ করতে সমস্যায় পড়ে, এবং এর ফলে নেটওয়ার্ক দূরবর্তী টোকেনগুলোর মধ্যে সম্পর্ক শিখতে পারে না। এই সমস্যাটি এড়ানোর একটি উপায় হলো **explicit state management** প্রবর্তন করা, যা **gates** ব্যবহার করে করা হয়। এই ধরনের দুটি সবচেয়ে পরিচিত আর্কিটেকচার হলো: **Long Short Term Memory** (LSTM) এবং **Gated Relay Unit** (GRU)।\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"LSTM নেটওয়ার্ক RNN-এর মতোই সংগঠিত, তবে এখানে দুটি স্টেট স্তর থেকে স্তরে প্রেরণ করা হয়: প্রকৃত স্টেট $c$, এবং হিডেন ভেক্টর $h$। প্রতিটি ইউনিটে, হিডেন ভেক্টর $h_i$ ইনপুট $x_i$ এর সাথে সংযুক্ত হয়, এবং তারা **gates** এর মাধ্যমে স্টেট $c$-তে কী ঘটবে তা নিয়ন্ত্রণ করে। প্রতিটি গেট হলো সিগময়েড অ্যাক্টিভেশন সহ একটি নিউরাল নেটওয়ার্ক (আউটপুট $[0,1]$ পরিসরে), যা স্টেট ভেক্টরের সাথে গুণ করলে বিটওয়াইজ মাস্ক হিসেবে ভাবা যেতে পারে। উপরের ছবিতে বাম থেকে ডান দিকে নিম্নলিখিত গেটগুলো রয়েছে:\n",
|
||||
"* **forget gate** হিডেন ভেক্টর গ্রহণ করে এবং নির্ধারণ করে, ভেক্টর $c$ এর কোন উপাদানগুলো ভুলে যেতে হবে এবং কোনগুলো প্রেরণ করতে হবে।\n",
|
||||
"* **input gate** ইনপুট এবং হিডেন ভেক্টর থেকে কিছু তথ্য গ্রহণ করে এবং স্টেটে যোগ করে।\n",
|
||||
"* **output gate** স্টেটকে $\\tanh$ অ্যাক্টিভেশন সহ কিছু লিনিয়ার স্তরের মাধ্যমে রূপান্তর করে, তারপর হিডেন ভেক্টর $h_i$ ব্যবহার করে এর কিছু উপাদান নির্বাচন করে নতুন স্টেট $c_{i+1}$ তৈরি করে।\n",
|
||||
"\n",
|
||||
"স্টেট $c$ এর উপাদানগুলোকে কিছু ফ্ল্যাগ হিসেবে ভাবা যেতে পারে, যা চালু বা বন্ধ করা যায়। উদাহরণস্বরূপ, যখন আমরা সিকোয়েন্সে *Alice* নামটি পাই, আমরা ধরে নিতে পারি যে এটি একটি নারী চরিত্রকে নির্দেশ করে, এবং স্টেটে একটি ফ্ল্যাগ বাড়াই যে আমাদের বাক্যে একটি নারী বিশেষ্য রয়েছে। যখন আমরা পরবর্তীতে *and Tom* বাক্যাংশটি পাই, আমরা একটি ফ্ল্যাগ বাড়াই যে আমাদের কাছে বহুবচন বিশেষ্য রয়েছে। এভাবে স্টেট পরিচালনা করে আমরা বাক্যের অংশগুলোর ব্যাকরণগত বৈশিষ্ট্যগুলো ট্র্যাক করতে পারি।\n",
|
||||
"\n",
|
||||
"> **Note**: LSTM-এর অভ্যন্তরীণ বিষয়গুলো বোঝার জন্য একটি চমৎকার রিসোর্স হলো ক্রিস্টোফার ওলাহের এই অসাধারণ নিবন্ধ [Understanding LSTM Networks](https://colah.github.io/posts/2015-08-Understanding-LSTMs/)।\n",
|
||||
"\n",
|
||||
"যদিও LSTM সেলের অভ্যন্তরীণ গঠন জটিল মনে হতে পারে, PyTorch এই বাস্তবায়নটি `LSTMCell` ক্লাসের মধ্যে লুকিয়ে রাখে এবং পুরো LSTM স্তরটি উপস্থাপন করতে `LSTM` অবজেক্ট প্রদান করে। তাই, LSTM ক্লাসিফায়ার বাস্তবায়নটি উপরে দেখা সাধারণ RNN-এর মতোই হবে:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class LSTMClassifier(torch.nn.Module):\n",
|
||||
" def __init__(self, vocab_size, embed_dim, hidden_dim, num_class):\n",
|
||||
" super().__init__()\n",
|
||||
" self.hidden_dim = hidden_dim\n",
|
||||
" self.embedding = torch.nn.Embedding(vocab_size, embed_dim)\n",
|
||||
" self.embedding.weight.data = torch.randn_like(self.embedding.weight.data)-0.5\n",
|
||||
" self.rnn = torch.nn.LSTM(embed_dim,hidden_dim,batch_first=True)\n",
|
||||
" self.fc = torch.nn.Linear(hidden_dim, num_class)\n",
|
||||
"\n",
|
||||
" def forward(self, x):\n",
|
||||
" batch_size = x.size(0)\n",
|
||||
" x = self.embedding(x)\n",
|
||||
" x,(h,c) = self.rnn(x)\n",
|
||||
" return self.fc(h[-1])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.259375\n",
|
||||
"6400: acc=0.25859375\n",
|
||||
"9600: acc=0.26177083333333334\n",
|
||||
"12800: acc=0.2784375\n",
|
||||
"16000: acc=0.313\n",
|
||||
"19200: acc=0.3528645833333333\n",
|
||||
"22400: acc=0.3965625\n",
|
||||
"25600: acc=0.4385546875\n",
|
||||
"28800: acc=0.4752777777777778\n",
|
||||
"32000: acc=0.505375\n",
|
||||
"35200: acc=0.5326704545454546\n",
|
||||
"38400: acc=0.5557552083333334\n",
|
||||
"41600: acc=0.5760817307692307\n",
|
||||
"44800: acc=0.5954910714285714\n",
|
||||
"48000: acc=0.6118333333333333\n",
|
||||
"51200: acc=0.62681640625\n",
|
||||
"54400: acc=0.6404779411764706\n",
|
||||
"57600: acc=0.6520138888888889\n",
|
||||
"60800: acc=0.662828947368421\n",
|
||||
"64000: acc=0.673546875\n",
|
||||
"67200: acc=0.6831547619047619\n",
|
||||
"70400: acc=0.6917897727272727\n",
|
||||
"73600: acc=0.6997146739130434\n",
|
||||
"76800: acc=0.707109375\n",
|
||||
"80000: acc=0.714075\n",
|
||||
"83200: acc=0.7209134615384616\n",
|
||||
"86400: acc=0.727037037037037\n",
|
||||
"89600: acc=0.7326674107142858\n",
|
||||
"92800: acc=0.7379633620689655\n",
|
||||
"96000: acc=0.7433645833333333\n",
|
||||
"99200: acc=0.7479032258064516\n",
|
||||
"102400: acc=0.752119140625\n",
|
||||
"105600: acc=0.7562405303030303\n",
|
||||
"108800: acc=0.76015625\n",
|
||||
"112000: acc=0.7641339285714286\n",
|
||||
"115200: acc=0.7677777777777778\n",
|
||||
"118400: acc=0.7711233108108108\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(0.03487814127604167, 0.7728)"
|
||||
]
|
||||
},
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"net = LSTMClassifier(vocab_size,64,32,len(classes)).to(device)\n",
|
||||
"train_epoch(net,train_loader, lr=0.001)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## প্যাকড সিকোয়েন্স\n",
|
||||
"\n",
|
||||
"আমাদের উদাহরণে, আমরা মিনিব্যাচের সব সিকোয়েন্সকে শূন্য ভেক্টর দিয়ে প্যাড করতে হয়েছিল। এটি কিছু মেমোরি অপচয় ঘটায়, তবে RNN-এর ক্ষেত্রে এটি আরও গুরুত্বপূর্ণ যে প্যাড করা ইনপুট আইটেমগুলোর জন্য অতিরিক্ত RNN সেল তৈরি হয়, যা প্রশিক্ষণে অংশ নেয়, কিন্তু কোনো গুরুত্বপূর্ণ ইনপুট তথ্য বহন করে না। শুধুমাত্র আসল সিকোয়েন্স সাইজ অনুযায়ী RNN প্রশিক্ষণ দেওয়া অনেক ভালো হবে।\n",
|
||||
"\n",
|
||||
"এটি করার জন্য, PyTorch-এ প্যাড করা সিকোয়েন্স সংরক্ষণের একটি বিশেষ ফরম্যাট প্রবর্তন করা হয়েছে। ধরুন আমাদের ইনপুট প্যাড করা মিনিব্যাচ এই রকম দেখায়:\n",
|
||||
"```\n",
|
||||
"[[1,2,3,4,5],\n",
|
||||
" [6,7,8,0,0],\n",
|
||||
" [9,0,0,0,0]]\n",
|
||||
"```\n",
|
||||
"এখানে 0 প্যাড করা মানগুলোকে নির্দেশ করে, এবং ইনপুট সিকোয়েন্সগুলোর আসল দৈর্ঘ্যের ভেক্টর হলো `[5,3,1]`।\n",
|
||||
"\n",
|
||||
"প্যাড করা সিকোয়েন্স দিয়ে RNN কার্যকরভাবে প্রশিক্ষণ দেওয়ার জন্য, আমরা প্রথমে বড় মিনিব্যাচ (`[1,6,9]`) দিয়ে RNN সেলগুলোর প্রশিক্ষণ শুরু করতে চাই, কিন্তু তারপর তৃতীয় সিকোয়েন্সের প্রসেসিং শেষ করে ছোট মিনিব্যাচ দিয়ে প্রশিক্ষণ চালিয়ে যেতে চাই (`[2,7]`, `[3,8]`), এবং এভাবে চলতে থাকবে। সুতরাং, প্যাকড সিকোয়েন্স একটি ভেক্টর হিসেবে উপস্থাপিত হয় - আমাদের ক্ষেত্রে `[1,6,9,2,7,3,8,4,5]`, এবং দৈর্ঘ্যের ভেক্টর (`[5,3,1]`), যেখান থেকে আমরা সহজেই আসল প্যাড করা মিনিব্যাচ পুনর্গঠন করতে পারি।\n",
|
||||
"\n",
|
||||
"প্যাকড সিকোয়েন্স তৈরি করতে, আমরা `torch.nn.utils.rnn.pack_padded_sequence` ফাংশন ব্যবহার করতে পারি। সব রিকারেন্ট লেয়ার, যেমন RNN, LSTM এবং GRU, ইনপুট হিসেবে প্যাকড সিকোয়েন্স সমর্থন করে এবং প্যাকড আউটপুট তৈরি করে, যা `torch.nn.utils.rnn.pad_packed_sequence` ব্যবহার করে ডিকোড করা যায়।\n",
|
||||
"\n",
|
||||
"প্যাকড সিকোয়েন্স তৈরি করতে সক্ষম হওয়ার জন্য, আমাদের নেটওয়ার্কে দৈর্ঘ্যের ভেক্টর পাস করতে হবে, এবং তাই মিনিব্যাচ প্রস্তুত করার জন্য আমাদের একটি ভিন্ন ফাংশন প্রয়োজন:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def pad_length(b):\n",
|
||||
" # build vectorized sequence\n",
|
||||
" v = [encode(x[1]) for x in b]\n",
|
||||
" # compute max length of a sequence in this minibatch and length sequence itself\n",
|
||||
" len_seq = list(map(len,v))\n",
|
||||
" l = max(len_seq)\n",
|
||||
" return ( # tuple of three tensors - labels, padded features, length sequence\n",
|
||||
" torch.LongTensor([t[0]-1 for t in b]),\n",
|
||||
" torch.stack([torch.nn.functional.pad(torch.tensor(t),(0,l-len(t)),mode='constant',value=0) for t in v]),\n",
|
||||
" torch.tensor(len_seq)\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"train_loader_len = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=pad_length, shuffle=True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"আসল নেটওয়ার্কটি উপরের `LSTMClassifier` এর মতোই হবে, তবে `forward` পাসটি প্যাড করা মিনিব্যাচ এবং সিকোয়েন্সের দৈর্ঘ্যের ভেক্টর উভয়ই গ্রহণ করবে। এম্বেডিং গণনা করার পরে, আমরা প্যাকড সিকোয়েন্স তৈরি করি, এটি LSTM লেয়ারে পাঠাই, এবং তারপর ফলাফলটি আবার আনপ্যাক করি।\n",
|
||||
"\n",
|
||||
"> **Note**: আমরা আসলে আনপ্যাক করা ফলাফল `x` ব্যবহার করি না, কারণ আমরা পরবর্তী গণনাগুলিতে হিডেন লেয়ার থেকে আউটপুট ব্যবহার করি। সুতরাং, আমরা এই কোড থেকে আনপ্যাকিং সম্পূর্ণভাবে সরিয়ে ফেলতে পারি। আমরা এটি এখানে রেখেছি যাতে আপনি সহজেই এই কোডটি পরিবর্তন করতে পারেন, যদি আপনাকে নেটওয়ার্ক আউটপুট পরবর্তী গণনাগুলিতে ব্যবহার করতে হয়।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class LSTMPackClassifier(torch.nn.Module):\n",
|
||||
" def __init__(self, vocab_size, embed_dim, hidden_dim, num_class):\n",
|
||||
" super().__init__()\n",
|
||||
" self.hidden_dim = hidden_dim\n",
|
||||
" self.embedding = torch.nn.Embedding(vocab_size, embed_dim)\n",
|
||||
" self.embedding.weight.data = torch.randn_like(self.embedding.weight.data)-0.5\n",
|
||||
" self.rnn = torch.nn.LSTM(embed_dim,hidden_dim,batch_first=True)\n",
|
||||
" self.fc = torch.nn.Linear(hidden_dim, num_class)\n",
|
||||
"\n",
|
||||
" def forward(self, x, lengths):\n",
|
||||
" batch_size = x.size(0)\n",
|
||||
" x = self.embedding(x)\n",
|
||||
" pad_x = torch.nn.utils.rnn.pack_padded_sequence(x,lengths,batch_first=True,enforce_sorted=False)\n",
|
||||
" pad_x,(h,c) = self.rnn(pad_x)\n",
|
||||
" x, _ = torch.nn.utils.rnn.pad_packed_sequence(pad_x,batch_first=True)\n",
|
||||
" return self.fc(h[-1])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.285625\n",
|
||||
"6400: acc=0.33359375\n",
|
||||
"9600: acc=0.3876041666666667\n",
|
||||
"12800: acc=0.44078125\n",
|
||||
"16000: acc=0.4825\n",
|
||||
"19200: acc=0.5235416666666667\n",
|
||||
"22400: acc=0.5559821428571429\n",
|
||||
"25600: acc=0.58609375\n",
|
||||
"28800: acc=0.6116666666666667\n",
|
||||
"32000: acc=0.63340625\n",
|
||||
"35200: acc=0.6525284090909091\n",
|
||||
"38400: acc=0.668515625\n",
|
||||
"41600: acc=0.6822596153846154\n",
|
||||
"44800: acc=0.6948214285714286\n",
|
||||
"48000: acc=0.7052708333333333\n",
|
||||
"51200: acc=0.71521484375\n",
|
||||
"54400: acc=0.7239889705882353\n",
|
||||
"57600: acc=0.7315277777777778\n",
|
||||
"60800: acc=0.7388486842105263\n",
|
||||
"64000: acc=0.74571875\n",
|
||||
"67200: acc=0.7518303571428572\n",
|
||||
"70400: acc=0.7576988636363636\n",
|
||||
"73600: acc=0.7628940217391305\n",
|
||||
"76800: acc=0.7681510416666667\n",
|
||||
"80000: acc=0.7728125\n",
|
||||
"83200: acc=0.7772235576923077\n",
|
||||
"86400: acc=0.7815393518518519\n",
|
||||
"89600: acc=0.7857700892857142\n",
|
||||
"92800: acc=0.7895043103448276\n",
|
||||
"96000: acc=0.7930520833333333\n",
|
||||
"99200: acc=0.7959072580645161\n",
|
||||
"102400: acc=0.798994140625\n",
|
||||
"105600: acc=0.802064393939394\n",
|
||||
"108800: acc=0.8051378676470589\n",
|
||||
"112000: acc=0.8077857142857143\n",
|
||||
"115200: acc=0.8104600694444445\n",
|
||||
"118400: acc=0.8128293918918919\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(0.029785829671223958, 0.8138166666666666)"
|
||||
]
|
||||
},
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"net = LSTMPackClassifier(vocab_size,64,32,len(classes)).to(device)\n",
|
||||
"train_epoch_emb(net,train_loader_len, lr=0.001,use_pack_sequence=True)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **নোট:** আপনি হয়তো লক্ষ্য করেছেন যে আমরা প্রশিক্ষণ ফাংশনে `use_pack_sequence` প্যারামিটারটি পাস করি। বর্তমানে, `pack_padded_sequence` ফাংশনটি দৈর্ঘ্যের সিকোয়েন্স টেনসরকে CPU ডিভাইসে রাখার প্রয়োজন হয়, এবং তাই প্রশিক্ষণ ফাংশনটি প্রশিক্ষণের সময় দৈর্ঘ্যের সিকোয়েন্স ডেটাকে GPU-তে স্থানান্তর করা এড়িয়ে চলে। আপনি [`torchnlp.py`](../../../../../lessons/5-NLP/16-RNN/torchnlp.py) ফাইলের মধ্যে `train_emb` ফাংশনের বাস্তবায়ন দেখতে পারেন।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## দ্বিমুখী এবং বহুস্তর RNN\n",
|
||||
"\n",
|
||||
"আমাদের উদাহরণগুলোতে, সব পুনরাবৃত্ত নেটওয়ার্ক একদিকে কাজ করেছে, একটি সিকোয়েন্সের শুরু থেকে শেষ পর্যন্ত। এটি স্বাভাবিক মনে হয়, কারণ এটি আমাদের পড়া এবং কথা শোনার পদ্ধতির সাথে সাদৃশ্যপূর্ণ। তবে, অনেক বাস্তব ক্ষেত্রে আমরা ইনপুট সিকোয়েন্সে র্যান্ডম অ্যাক্সেস পাই, তাই পুনরাবৃত্ত গণনা উভয় দিকেই চালানো যুক্তিসঙ্গত হতে পারে। এই ধরনের নেটওয়ার্ককে **দ্বিমুখী** RNN বলা হয়, এবং এগুলো তৈরি করা যায় RNN/LSTM/GRU কনস্ট্রাক্টরে `bidirectional=True` প্যারামিটার পাস করে।\n",
|
||||
"\n",
|
||||
"দ্বিমুখী নেটওয়ার্কের ক্ষেত্রে, আমাদের দুইটি হিডেন স্টেট ভেক্টরের প্রয়োজন হবে, প্রতিটি দিকের জন্য একটি। PyTorch এই ভেক্টরগুলোকে একটি বড় আকারের ভেক্টর হিসেবে এনকোড করে, যা বেশ সুবিধাজনক, কারণ সাধারণত আপনি এই হিডেন স্টেটকে একটি সম্পূর্ণ সংযুক্ত লিনিয়ার লেয়ারে পাস করবেন, এবং লেয়ার তৈরি করার সময় এই আকার বৃদ্ধিকে বিবেচনায় নিতে হবে।\n",
|
||||
"\n",
|
||||
"পুনরাবৃত্ত নেটওয়ার্ক, একমুখী বা দ্বিমুখী, একটি সিকোয়েন্সের নির্দিষ্ট প্যাটার্নগুলো ধারণ করে এবং সেগুলো স্টেট ভেক্টরে সংরক্ষণ করতে পারে বা আউটপুটে পাস করতে পারে। কনভোলিউশনাল নেটওয়ার্কের মতো, আমরা প্রথম স্তরের উপর আরেকটি পুনরাবৃত্ত স্তর তৈরি করতে পারি, যা নিম্ন-স্তরের প্যাটার্ন থেকে উচ্চ-স্তরের প্যাটার্ন ধারণ করে। এটি আমাদের **বহুস্তর RNN** ধারণার দিকে নিয়ে যায়, যা দুই বা তার বেশি পুনরাবৃত্ত নেটওয়ার্ক নিয়ে গঠিত, যেখানে পূর্ববর্তী স্তরের আউটপুট পরবর্তী স্তরের ইনপুট হিসেবে পাস করা হয়।\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"*ছবি [এই অসাধারণ পোস্ট](https://towardsdatascience.com/from-a-lstm-cell-to-a-multilayer-lstm-network-with-pytorch-2899eb5696f3) থেকে, লেখক Fernando López*\n",
|
||||
"\n",
|
||||
"PyTorch এই ধরনের নেটওয়ার্ক তৈরি করাকে সহজ করে তোলে, কারণ আপনাকে শুধু RNN/LSTM/GRU কনস্ট্রাক্টরে `num_layers` প্যারামিটার পাস করতে হবে, যা স্বয়ংক্রিয়ভাবে একাধিক স্তরের পুনরাবৃত্তি তৈরি করবে। এর অর্থ হলো হিডেন/স্টেট ভেক্টরের আকার অনুপাতে বৃদ্ধি পাবে, এবং পুনরাবৃত্ত স্তরের আউটপুট পরিচালনা করার সময় আপনাকে এটি বিবেচনায় নিতে হবে।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## অন্যান্য কাজের জন্য RNNs\n",
|
||||
"\n",
|
||||
"এই ইউনিটে, আমরা দেখেছি যে RNNs সিকোয়েন্স ক্লাসিফিকেশনের জন্য ব্যবহার করা যেতে পারে, তবে বাস্তবে, এগুলো আরও অনেক কাজ পরিচালনা করতে সক্ষম, যেমন টেক্সট জেনারেশন, মেশিন ট্রান্সলেশন, এবং আরও অনেক কিছু। আমরা পরবর্তী ইউনিটে সেই কাজগুলো বিবেচনা করব।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**অস্বীকৃতি**: \nএই নথিটি AI অনুবাদ পরিষেবা [Co-op Translator](https://github.com/Azure/co-op-translator) ব্যবহার করে অনুবাদ করা হয়েছে। আমরা যথাসম্ভব সঠিক অনুবাদ প্রদানের চেষ্টা করি, তবে অনুগ্রহ করে মনে রাখবেন যে স্বয়ংক্রিয় অনুবাদে ত্রুটি বা অসঙ্গতি থাকতে পারে। মূল ভাষায় থাকা নথিটিকে প্রামাণিক উৎস হিসেবে বিবেচনা করা উচিত। গুরুত্বপূর্ণ তথ্যের জন্য, পেশাদার মানব অনুবাদ সুপারিশ করা হয়। এই অনুবাদ ব্যবহারের ফলে কোনো ভুল বোঝাবুঝি বা ভুল ব্যাখ্যা হলে আমরা তার জন্য দায়ী থাকব না।\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.8.12 ('py38')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "522ee52ae3d5ae933e283286254e9a55",
|
||||
"translation_date": "2025-08-28T12:20:36+00:00",
|
||||
"source_file": "lessons/5-NLP/16-RNN/RNNPyTorch.ipynb",
|
||||
"language_code": "bn"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
|
|
@ -0,0 +1,462 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# পুনরাবৃত্তি নিউরাল নেটওয়ার্ক\n",
|
||||
"\n",
|
||||
"পূর্ববর্তী মডিউলে, আমরা টেক্সটের সমৃদ্ধ অর্থবহ উপস্থাপনাগুলি নিয়ে আলোচনা করেছি। আমরা যে আর্কিটেকচারটি ব্যবহার করেছি তা বাক্যের শব্দগুলির সম্মিলিত অর্থ ধারণ করে, তবে এটি শব্দগুলির **ক্রম** বিবেচনা করে না, কারণ এম্বেডিংয়ের পরে যে সংযোজন প্রক্রিয়া হয় তা মূল টেক্সট থেকে এই তথ্যটি সরিয়ে দেয়। যেহেতু এই মডেলগুলি শব্দের ক্রম উপস্থাপন করতে অক্ষম, তাই তারা টেক্সট জেনারেশন বা প্রশ্নোত্তরের মতো আরও জটিল বা দ্ব্যর্থক কাজ সমাধান করতে পারে না।\n",
|
||||
"\n",
|
||||
"একটি টেক্সট সিকোয়েন্সের অর্থ ধরতে, আমরা **পুনরাবৃত্তি নিউরাল নেটওয়ার্ক** বা RNN নামে একটি নিউরাল নেটওয়ার্ক আর্কিটেকচার ব্যবহার করব। RNN ব্যবহার করার সময়, আমরা আমাদের বাক্যটি নেটওয়ার্কের মধ্য দিয়ে একবারে একটি টোকেন পাঠাই, এবং নেটওয়ার্কটি কিছু **অবস্থা** তৈরি করে, যা আমরা পরবর্তী টোকেনের সাথে আবার নেটওয়ার্কে পাঠাই।\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"টোকেনগুলির ইনপুট সিকোয়েন্স $X_0,\\dots,X_n$ দেওয়া হলে, RNN একটি নিউরাল নেটওয়ার্ক ব্লকের সিকোয়েন্স তৈরি করে এবং ব্যাকপ্রোপাগেশনের মাধ্যমে এই সিকোয়েন্সটি এন্ড-টু-এন্ড প্রশিক্ষণ দেয়। প্রতিটি নেটওয়ার্ক ব্লক $(X_i,S_i)$ জোড়াকে ইনপুট হিসাবে নেয় এবং ফলাফল হিসাবে $S_{i+1}$ তৈরি করে। চূড়ান্ত অবস্থা $S_n$ বা আউটপুট $Y_n$ একটি লিনিয়ার ক্লাসিফায়ারে যায় ফলাফল তৈরি করতে। সমস্ত নেটওয়ার্ক ব্লক একই ওজন ভাগ করে এবং একটি ব্যাকপ্রোপাগেশন পাস ব্যবহার করে এন্ড-টু-এন্ড প্রশিক্ষণ দেওয়া হয়।\n",
|
||||
"\n",
|
||||
"> উপরের চিত্রটি পুনরাবৃত্তি নিউরাল নেটওয়ার্ককে আনরোলড ফর্মে (বামে) এবং আরও কমপ্যাক্ট পুনরাবৃত্তি উপস্থাপনায় (ডানে) দেখায়। এটি বোঝা গুরুত্বপূর্ণ যে সমস্ত RNN সেলগুলির একই **শেয়ারযোগ্য ওজন** রয়েছে।\n",
|
||||
"\n",
|
||||
"কারণ অবস্থা ভেক্টর $S_0,\\dots,S_n$ নেটওয়ার্কের মধ্য দিয়ে যায়, RNN শব্দগুলির মধ্যে ক্রমিক নির্ভরতা শিখতে সক্ষম। উদাহরণস্বরূপ, যখন *not* শব্দটি সিকোয়েন্সের কোথাও উপস্থিত হয়, এটি অবস্থা ভেক্টরের নির্দিষ্ট উপাদানগুলিকে অস্বীকার করতে শিখতে পারে।\n",
|
||||
"\n",
|
||||
"প্রত্যেকটি RNN সেলের ভিতরে দুটি ওজন ম্যাট্রিক্স থাকে: $W_H$ এবং $W_I$, এবং বায়াস $b$। প্রতিটি RNN ধাপে, ইনপুট $X_i$ এবং ইনপুট অবস্থা $S_i$ দেওয়া হলে, আউটপুট অবস্থা হিসাব করা হয় $S_{i+1} = f(W_H\\times S_i + W_I\\times X_i+b)$, যেখানে $f$ একটি অ্যাক্টিভেশন ফাংশন (সাধারণত $\\tanh$)।\n",
|
||||
"\n",
|
||||
"> টেক্সট জেনারেশন (যা আমরা পরবর্তী ইউনিটে আলোচনা করব) বা মেশিন ট্রান্সলেশনের মতো সমস্যার জন্য আমরা প্রতিটি RNN ধাপে কিছু আউটপুট মানও পেতে চাই। এই ক্ষেত্রে, আরেকটি ম্যাট্রিক্স $W_O$ থাকে, এবং আউটপুট হিসাব করা হয় $Y_i=f(W_O\\times S_i+b_O)$।\n",
|
||||
"\n",
|
||||
"চলুন দেখি কীভাবে পুনরাবৃত্তি নিউরাল নেটওয়ার্ক আমাদের নিউজ ডেটাসেট শ্রেণীবদ্ধ করতে সাহায্য করতে পারে।\n",
|
||||
"\n",
|
||||
"> স্যান্ডবক্স পরিবেশের জন্য, আমাদের নিশ্চিত করতে হবে যে প্রয়োজনীয় লাইব্রেরি ইনস্টল করা হয়েছে এবং ডেটা প্রিফেচ করা হয়েছে, এজন্য নিম্নলিখিত সেলটি চালাতে হবে। আপনি যদি লোকাল মেশিনে কাজ করেন, তবে এই সেলটি এড়িয়ে যেতে পারেন।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import sys\n",
|
||||
"!{sys.executable} -m pip install --quiet tensorflow_datasets==4.4.0\n",
|
||||
"!cd ~ && wget -q -O - https://mslearntensorflowlp.blob.core.windows.net/data/tfds-ag-news.tgz | tar xz"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import tensorflow as tf\n",
|
||||
"from tensorflow import keras\n",
|
||||
"import tensorflow_datasets as tfds\n",
|
||||
"import numpy as np\n",
|
||||
"\n",
|
||||
"# We are going to be training pretty large models. In order not to face errors, we need\n",
|
||||
"# to set tensorflow option to grow GPU memory allocation when required\n",
|
||||
"physical_devices = tf.config.list_physical_devices('GPU') \n",
|
||||
"if len(physical_devices)>0:\n",
|
||||
" tf.config.experimental.set_memory_growth(physical_devices[0], True)\n",
|
||||
"\n",
|
||||
"ds_train, ds_test = tfds.load('ag_news_subset').values()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"nteract": {
|
||||
"transient": {
|
||||
"deleting": false
|
||||
}
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"বড় মডেল প্রশিক্ষণ করার সময় GPU মেমোরি বরাদ্দ একটি সমস্যা হয়ে উঠতে পারে। এছাড়াও, আমাদের বিভিন্ন মিনিব্যাচ সাইজ নিয়ে পরীক্ষা করতে হতে পারে, যাতে ডেটা GPU মেমোরিতে ফিট করে, এবং প্রশিক্ষণ যথেষ্ট দ্রুত হয়। যদি আপনি আপনার নিজস্ব GPU মেশিনে এই কোড চালাচ্ছেন, তাহলে প্রশিক্ষণের গতি বাড়ানোর জন্য মিনিব্যাচ সাইজ সামঞ্জস্য করার চেষ্টা করতে পারেন।\n",
|
||||
"\n",
|
||||
"> **Note**: NVidia ড্রাইভারের কিছু নির্দিষ্ট সংস্করণ মডেল প্রশিক্ষণের পর মেমোরি মুক্ত করতে ব্যর্থ হতে পারে। আমরা এই নোটবুকে কয়েকটি উদাহরণ চালাচ্ছি, এবং এটি কিছু সেটআপে মেমোরি শেষ হয়ে যাওয়ার কারণ হতে পারে, বিশেষত যদি আপনি একই নোটবুকের অংশ হিসেবে নিজের পরীক্ষা চালাচ্ছেন। যদি আপনি মডেল প্রশিক্ষণ শুরু করার সময় কিছু অদ্ভুত ত্রুটি সম্মুখীন করেন, তাহলে নোটবুকের কার্নেল পুনরায় চালু করার কথা বিবেচনা করতে পারেন।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {
|
||||
"collapsed": true,
|
||||
"jupyter": {
|
||||
"outputs_hidden": false,
|
||||
"source_hidden": false
|
||||
},
|
||||
"nteract": {
|
||||
"transient": {
|
||||
"deleting": false
|
||||
}
|
||||
}
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"batch_size = 16\n",
|
||||
"embed_size = 64"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## সহজ RNN শ্রেণীবিভাজক\n",
|
||||
"\n",
|
||||
"একটি সহজ RNN-এর ক্ষেত্রে, প্রতিটি পুনরাবৃত্তি ইউনিট একটি সাধারণ রৈখিক নেটওয়ার্ক, যা একটি ইনপুট ভেক্টর এবং স্টেট ভেক্টর গ্রহণ করে এবং একটি নতুন স্টেট ভেক্টর তৈরি করে। Keras-এ, এটি `SimpleRNN` স্তর দ্বারা উপস্থাপন করা যেতে পারে।\n",
|
||||
"\n",
|
||||
"যদিও আমরা সরাসরি RNN স্তরে এক-হট এনকোড করা টোকেন পাস করতে পারি, এটি একটি ভালো ধারণা নয় কারণ এগুলোর উচ্চ মাত্রা থাকে। তাই, আমরা শব্দ ভেক্টরের মাত্রা কমানোর জন্য একটি এমবেডিং স্তর ব্যবহার করব, এরপর একটি RNN স্তর এবং শেষে একটি `Dense` শ্রেণীবিভাজক।\n",
|
||||
"\n",
|
||||
"> **Note**: এমন ক্ষেত্রে যেখানে মাত্রা এত বেশি নয়, যেমন চরিত্র-স্তরের টোকেনাইজেশন ব্যবহার করার সময়, এক-হট এনকোড করা টোকেন সরাসরি RNN সেলে পাস করা যুক্তিযুক্ত হতে পারে।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"sequential\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
"Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
"text_vectorization (TextVect (None, None) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"embedding (Embedding) (None, None, 64) 1280000 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"simple_rnn (SimpleRNN) (None, 16) 1296 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense (Dense) (None, 4) 68 \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 1,281,364\n",
|
||||
"Trainable params: 1,281,364\n",
|
||||
"Non-trainable params: 0\n",
|
||||
"_________________________________________________________________\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vocab_size = 20000\n",
|
||||
"\n",
|
||||
"vectorizer = keras.layers.experimental.preprocessing.TextVectorization(\n",
|
||||
" max_tokens=vocab_size,\n",
|
||||
" input_shape=(1,))\n",
|
||||
"\n",
|
||||
"model = keras.models.Sequential([\n",
|
||||
" vectorizer,\n",
|
||||
" keras.layers.Embedding(vocab_size, embed_size),\n",
|
||||
" keras.layers.SimpleRNN(16),\n",
|
||||
" keras.layers.Dense(4,activation='softmax')\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"model.summary()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **নোট:** এখানে আমরা সরলতার জন্য একটি অনট্রেইন্ড এমবেডিং লেয়ার ব্যবহার করছি, তবে আরও ভালো ফলাফলের জন্য আমরা Word2Vec ব্যবহার করে একটি প্রিট্রেইন্ড এমবেডিং লেয়ার ব্যবহার করতে পারি, যা আগের ইউনিটে বর্ণনা করা হয়েছে। এটি আপনার জন্য একটি ভালো অনুশীলন হবে যদি আপনি এই কোডটি প্রিট্রেইন্ড এমবেডিংয়ের সাথে কাজ করার জন্য মানিয়ে নিতে পারেন।\n",
|
||||
"\n",
|
||||
"এখন চলুন আমাদের RNN প্রশিক্ষণ করি। সাধারণভাবে RNN প্রশিক্ষণ করা বেশ কঠিন, কারণ RNN সেলগুলো সিকোয়েন্সের দৈর্ঘ্য বরাবর আনরোল করার পর, ব্যাকপ্রোপাগেশনে জড়িত লেয়ারগুলোর সংখ্যা অনেক বেড়ে যায়। তাই আমাদের একটি ছোট লার্নিং রেট নির্বাচন করতে হবে এবং ভালো ফলাফলের জন্য বড় ডেটাসেটে নেটওয়ার্কটি প্রশিক্ষণ দিতে হবে। এটি করতে বেশ সময় লাগতে পারে, তাই GPU ব্যবহার করাই উত্তম।\n",
|
||||
"\n",
|
||||
"গতি বাড়ানোর জন্য, আমরা শুধুমাত্র নিউজ শিরোনামে RNN মডেলটি প্রশিক্ষণ দেব, বর্ণনা অংশটি বাদ দিয়ে। আপনি চাইলে বর্ণনা সহ প্রশিক্ষণ দিয়ে দেখতে পারেন যে মডেলটি প্রশিক্ষণ পেতে পারে কিনা।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Training vectorizer\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def extract_title(x):\n",
|
||||
" return x['title']\n",
|
||||
"\n",
|
||||
"def tupelize_title(x):\n",
|
||||
" return (extract_title(x),x['label'])\n",
|
||||
"\n",
|
||||
"print('Training vectorizer')\n",
|
||||
"vectorizer.adapt(ds_train.take(2000).map(extract_title))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"7500/7500 [==============================] - 82s 11ms/step - loss: 0.6629 - acc: 0.7623 - val_loss: 0.5559 - val_acc: 0.7995\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7f3e0030d350>"
|
||||
]
|
||||
},
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n",
|
||||
"model.fit(ds_train.map(tupelize_title).batch(batch_size),validation_data=ds_test.map(tupelize_title).batch(batch_size))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"nteract": {
|
||||
"transient": {
|
||||
"deleting": false
|
||||
}
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"> **নোট** যে এখানে সঠিকতা সম্ভবত কম হবে, কারণ আমরা শুধুমাত্র সংবাদ শিরোনামের উপর প্রশিক্ষণ দিচ্ছি।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## ভেরিয়েবল সিকোয়েন্স পুনর্বিবেচনা\n",
|
||||
"\n",
|
||||
"মনে রাখবেন যে `TextVectorization` লেয়ার একটি মিনিব্যাচে ভেরিয়েবল দৈর্ঘ্যের সিকোয়েন্সগুলোকে স্বয়ংক্রিয়ভাবে প্যাড টোকেন দিয়ে পূরণ করবে। দেখা গেছে, এই টোকেনগুলোও প্রশিক্ষণে অংশ নেয় এবং মডেলের কনভার্জেন্স জটিল করে তুলতে পারে।\n",
|
||||
"\n",
|
||||
"প্যাডিংয়ের পরিমাণ কমানোর জন্য আমরা কয়েকটি পদ্ধতি অবলম্বন করতে পারি। এর মধ্যে একটি হলো ডেটাসেটকে সিকোয়েন্সের দৈর্ঘ্য অনুযায়ী পুনর্বিন্যাস করা এবং সব সিকোয়েন্সকে তাদের আকার অনুযায়ী গ্রুপ করা। এটি `tf.data.experimental.bucket_by_sequence_length` ফাংশন ব্যবহার করে করা যেতে পারে (দেখুন [ডকুমেন্টেশন](https://www.tensorflow.org/api_docs/python/tf/data/experimental/bucket_by_sequence_length))।\n",
|
||||
"\n",
|
||||
"আরেকটি পদ্ধতি হলো **মাস্কিং** ব্যবহার করা। কেরাসে, কিছু লেয়ার অতিরিক্ত ইনপুট সমর্থন করে যা দেখায় কোন টোকেনগুলো প্রশিক্ষণের সময় বিবেচনায় নেওয়া উচিত। আমাদের মডেলে মাস্কিং অন্তর্ভুক্ত করতে, আমরা হয় একটি আলাদা `Masking` লেয়ার ([ডকস](https://keras.io/api/layers/core_layers/masking/)) যোগ করতে পারি, অথবা আমাদের `Embedding` লেয়ারের `mask_zero=True` প্যারামিটার নির্দিষ্ট করতে পারি।\n",
|
||||
"\n",
|
||||
"> **Note**: পুরো ডেটাসেটে একটি এপোক সম্পন্ন করতে এই প্রশিক্ষণে প্রায় ৫ মিনিট সময় লাগবে। ধৈর্য হারালে যে কোনো সময় প্রশিক্ষণ বন্ধ করতে পারেন। এছাড়াও, প্রশিক্ষণের জন্য ব্যবহৃত ডেটার পরিমাণ সীমিত করতে পারেন, `ds_train` এবং `ds_test` ডেটাসেটের পরে `.take(...)` ক্লজ যোগ করে।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"7500/7500 [==============================] - 371s 49ms/step - loss: 0.5401 - acc: 0.8079 - val_loss: 0.3780 - val_acc: 0.8822\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7f3dec118850>"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def extract_text(x):\n",
|
||||
" return x['title']+' '+x['description']\n",
|
||||
"\n",
|
||||
"def tupelize(x):\n",
|
||||
" return (extract_text(x),x['label'])\n",
|
||||
"\n",
|
||||
"model = keras.models.Sequential([\n",
|
||||
" vectorizer,\n",
|
||||
" keras.layers.Embedding(vocab_size,embed_size,mask_zero=True),\n",
|
||||
" keras.layers.SimpleRNN(16),\n",
|
||||
" keras.layers.Dense(4,activation='softmax')\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"এখন আমরা মাস্কিং ব্যবহার করছি, তাই আমরা শিরোনাম এবং বিবরণের পুরো ডেটাসেটে মডেলটি প্রশিক্ষণ দিতে পারি।\n",
|
||||
"\n",
|
||||
"> **Note**: আপনি কি লক্ষ্য করেছেন যে আমরা নিউজ শিরোনামের উপর প্রশিক্ষিত ভেক্টরাইজার ব্যবহার করছি, পুরো নিবন্ধের মূল অংশের উপর নয়? এটি সম্ভবত কিছু টোকেন উপেক্ষা করতে পারে, তাই ভেক্টরাইজারটি পুনরায় প্রশিক্ষণ দেওয়া ভালো। তবে, এর প্রভাব খুবই সামান্য হতে পারে, তাই সরলতার জন্য আমরা পূর্বে প্রশিক্ষিত ভেক্টরাইজারটিই ব্যবহার করব।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## LSTM: দীর্ঘমেয়াদী স্মৃতি\n",
|
||||
"\n",
|
||||
"RNN-এর প্রধান সমস্যাগুলোর একটি হলো **vanishing gradients**। RNN অনেক দীর্ঘ হতে পারে এবং ব্যাকপ্রোপাগেশনের সময় নেটওয়ার্কের প্রথম স্তর পর্যন্ত গ্রেডিয়েন্টগুলো সঠিকভাবে পৌঁছাতে সমস্যা হতে পারে। যখন এটি ঘটে, তখন নেটওয়ার্ক দূরবর্তী টোকেনগুলোর মধ্যে সম্পর্ক শিখতে পারে না। এই সমস্যাটি এড়ানোর একটি উপায় হলো **স্পষ্ট স্টেট ম্যানেজমেন্ট** প্রবর্তন করা, যা **গেট** ব্যবহার করে করা হয়। গেট প্রবর্তনকারী সবচেয়ে সাধারণ দুটি আর্কিটেকচার হলো **দীর্ঘমেয়াদী স্মৃতি** (LSTM) এবং **গেটেড রিলে ইউনিট** (GRU)। এখানে আমরা LSTM নিয়ে আলোচনা করব।\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"একটি LSTM নেটওয়ার্ক RNN-এর মতোই সংগঠিত, তবে এখানে দুটি স্টেট স্তর থেকে স্তরে প্রেরণ করা হয়: প্রকৃত স্টেট $c$, এবং হিডেন ভেক্টর $h$। প্রতিটি ইউনিটে, হিডেন ভেক্টর $h_{t-1}$ ইনপুট $x_t$ এর সাথে মিলিত হয় এবং একসাথে তারা **গেট** এর মাধ্যমে স্টেট $c_t$ এবং আউটপুট $h_{t}$ এর উপর কী ঘটবে তা নিয়ন্ত্রণ করে। প্রতিটি গেটের সিগময়েড অ্যাক্টিভেশন থাকে (আউটপুট $[0,1]$ পরিসরে), যা স্টেট ভেক্টরের সাথে গুণ করলে বিটওয়াইজ মাস্ক হিসেবে ভাবা যেতে পারে। LSTM-এ নিম্নলিখিত গেটগুলো থাকে (উপরের ছবিতে বাম থেকে ডানে):\n",
|
||||
"* **forget gate**, যা নির্ধারণ করে ভেক্টর $c_{t-1}$ এর কোন উপাদানগুলো আমরা ভুলে যাব এবং কোনগুলো প্রেরণ করব।\n",
|
||||
"* **input gate**, যা নির্ধারণ করে ইনপুট ভেক্টর এবং পূর্ববর্তী হিডেন ভেক্টর থেকে কতটুকু তথ্য স্টেট ভেক্টরে অন্তর্ভুক্ত করা হবে।\n",
|
||||
"* **output gate**, যা নতুন স্টেট ভেক্টর নিয়ে সিদ্ধান্ত নেয় এর কোন উপাদানগুলো নতুন হিডেন ভেক্টর $h_t$ উৎপাদনের জন্য ব্যবহার করা হবে।\n",
|
||||
"\n",
|
||||
"স্টেট $c$ এর উপাদানগুলোকে এমন ফ্ল্যাগ হিসেবে ভাবা যেতে পারে যা চালু এবং বন্ধ করা যায়। উদাহরণস্বরূপ, যখন আমরা সিকোয়েন্সে *Alice* নামটি পাই, আমরা অনুমান করি এটি একজন নারীর নাম এবং স্টেটে একটি ফ্ল্যাগ উত্থাপন করি যা বলে যে বাক্যে একটি নারীবাচক বিশেষ্য রয়েছে। যখন আমরা আরও *and Tom* শব্দগুলো পাই, তখন আমরা একটি ফ্ল্যাগ উত্থাপন করি যা বলে যে আমাদের কাছে একটি বহুবচন বিশেষ্য রয়েছে। এভাবে স্টেট ম্যানিপুলেট করে আমরা বাক্যের ব্যাকরণগত বৈশিষ্ট্যগুলো ট্র্যাক করতে পারি।\n",
|
||||
"\n",
|
||||
"> **Note**: LSTM-এর অভ্যন্তরীণ বিষয়গুলো বোঝার জন্য এটি একটি চমৎকার রিসোর্স: [Understanding LSTM Networks](https://colah.github.io/posts/2015-08-Understanding-LSTMs/) ক্রিস্টোফার ওলাহ দ্বারা।\n",
|
||||
"\n",
|
||||
"যদিও একটি LSTM সেলের অভ্যন্তরীণ গঠন জটিল মনে হতে পারে, Keras এই বাস্তবায়নটি `LSTM` লেয়ারের ভিতরে লুকিয়ে রাখে, তাই উপরের উদাহরণে আমাদের যা করতে হবে তা হলো পুনরাবৃত্ত স্তরটি প্রতিস্থাপন করা:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"15000/15000 [==============================] - 188s 13ms/step - loss: 0.5692 - acc: 0.7916 - val_loss: 0.3441 - val_acc: 0.8870\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7f3d6af5c350>"
|
||||
]
|
||||
},
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model = keras.models.Sequential([\n",
|
||||
" vectorizer,\n",
|
||||
" keras.layers.Embedding(vocab_size, embed_size),\n",
|
||||
" keras.layers.LSTM(8),\n",
|
||||
" keras.layers.Dense(4,activation='softmax')\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(8),validation_data=ds_test.map(tupelize).batch(8))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**বিঃদ্রঃ** LSTM প্রশিক্ষণ দেওয়া বেশ ধীরগতির, এবং প্রশিক্ষণের শুরুতে আপনি হয়তো সঠিকতার ক্ষেত্রে খুব বেশি উন্নতি দেখতে পাবেন না। ভালো সঠিকতা অর্জনের জন্য আপনাকে কিছু সময় ধরে প্রশিক্ষণ চালিয়ে যেতে হতে পারে।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## দ্বিমুখী এবং বহুস্তর RNN\n",
|
||||
"\n",
|
||||
"আমাদের আগের উদাহরণগুলোতে, পুনরাবৃত্ত নেটওয়ার্কগুলো একটি সিকোয়েন্সের শুরু থেকে শেষ পর্যন্ত কাজ করে। এটি আমাদের কাছে স্বাভাবিক মনে হয় কারণ এটি সেই দিক অনুসরণ করে যেভাবে আমরা পড়ি বা কথা শুনি। তবে এমন পরিস্থিতিতে যেখানে ইনপুট সিকোয়েন্সের র্যান্ডম অ্যাক্সেস প্রয়োজন, সেখানে পুনরাবৃত্ত গণনা উভয় দিকেই চালানো আরও যুক্তিসঙ্গত। এমন RNN-গুলো, যেগুলো উভয় দিকেই গণনা করতে পারে, তাদের **দ্বিমুখী** RNN বলা হয়, এবং এগুলো তৈরি করা যায় পুনরাবৃত্ত স্তরকে একটি বিশেষ `Bidirectional` স্তর দিয়ে মোড়ানোর মাধ্যমে।\n",
|
||||
"\n",
|
||||
"> **Note**: `Bidirectional` স্তরটি তার মধ্যে থাকা স্তরের দুটি কপি তৈরি করে এবং সেই কপিগুলোর একটির `go_backwards` প্রপার্টিকে `True` সেট করে, যা সিকোয়েন্সের বিপরীত দিকে কাজ করে।\n",
|
||||
"\n",
|
||||
"পুনরাবৃত্ত নেটওয়ার্ক, একমুখী বা দ্বিমুখী, সিকোয়েন্সের মধ্যে প্যাটার্নগুলো ধরে এবং সেগুলোকে স্টেট ভেক্টরে সংরক্ষণ করে বা আউটপুট হিসেবে ফেরত দেয়। কনভোলিউশনাল নেটওয়ার্কের মতো, আমরা প্রথম স্তরের মাধ্যমে নিম্ন স্তরের প্যাটার্নগুলো থেকে উচ্চ স্তরের প্যাটার্ন ধরার জন্য আরেকটি পুনরাবৃত্ত স্তর তৈরি করতে পারি। এটি আমাদের **বহুস্তর RNN** ধারণার দিকে নিয়ে যায়, যা দুটি বা তার বেশি পুনরাবৃত্ত নেটওয়ার্ক নিয়ে গঠিত, যেখানে আগের স্তরের আউটপুট পরবর্তী স্তরের ইনপুট হিসেবে ব্যবহৃত হয়।\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"*ছবি [এই চমৎকার পোস্ট](https://towardsdatascience.com/from-a-lstm-cell-to-a-multilayer-lstm-network-with-pytorch-2899eb5696f3) থেকে, লেখক Fernando López।*\n",
|
||||
"\n",
|
||||
"Keras এই নেটওয়ার্কগুলো তৈরি করাকে সহজ করে তোলে, কারণ আপনাকে শুধু মডেলে আরও পুনরাবৃত্ত স্তর যোগ করতে হবে। শেষ স্তর ছাড়া অন্য সব স্তরের জন্য, আমাদের `return_sequences=True` প্যারামিটার নির্দিষ্ট করতে হবে, কারণ আমাদের স্তর থেকে সমস্ত মধ্যবর্তী স্টেট ফেরত দরকার, শুধুমাত্র পুনরাবৃত্ত গণনার চূড়ান্ত স্টেট নয়।\n",
|
||||
"\n",
|
||||
"চলুন আমাদের শ্রেণীবিন্যাস সমস্যার জন্য একটি দুই-স্তর বিশিষ্ট দ্বিমুখী LSTM তৈরি করি।\n",
|
||||
"\n",
|
||||
"> **Note** এই কোডটি সম্পন্ন হতে আবারও বেশ সময় নেয়, তবে এটি আমাদের এখন পর্যন্ত দেখা সর্বোচ্চ সঠিকতা প্রদান করে। তাই হয়তো অপেক্ষা করা এবং ফলাফল দেখা মূল্যবান।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"5044/7500 [===================>..........] - ETA: 2:33 - loss: 0.3709 - acc: 0.8706\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r5045/7500 [===================>..........] - ETA: 2:33 - loss: 0.3709 - acc: 0.8706"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model = keras.models.Sequential([\n",
|
||||
" vectorizer,\n",
|
||||
" keras.layers.Embedding(vocab_size, 128, mask_zero=True),\n",
|
||||
" keras.layers.Bidirectional(keras.layers.LSTM(64,return_sequences=True)),\n",
|
||||
" keras.layers.Bidirectional(keras.layers.LSTM(64)), \n",
|
||||
" keras.layers.Dense(4,activation='softmax')\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(batch_size),\n",
|
||||
" validation_data=ds_test.map(tupelize).batch(batch_size))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## অন্যান্য কাজের জন্য RNNs\n",
|
||||
"\n",
|
||||
"এখন পর্যন্ত, আমরা RNNs ব্যবহার করে টেক্সট সিকোয়েন্স শ্রেণীবদ্ধ করার উপর জোর দিয়েছি। তবে এগুলো আরও অনেক কাজ পরিচালনা করতে পারে, যেমন টেক্সট জেনারেশন এবং মেশিন অনুবাদ — আমরা পরবর্তী ইউনিটে এই কাজগুলো নিয়ে আলোচনা করব।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**অস্বীকৃতি**: \nএই নথিটি AI অনুবাদ পরিষেবা [Co-op Translator](https://github.com/Azure/co-op-translator) ব্যবহার করে অনুবাদ করা হয়েছে। আমরা যথাসম্ভব সঠিকতার জন্য চেষ্টা করি, তবে অনুগ্রহ করে মনে রাখবেন যে স্বয়ংক্রিয় অনুবাদে ত্রুটি বা অসঙ্গতি থাকতে পারে। মূল ভাষায় থাকা নথিটিকে প্রামাণিক উৎস হিসেবে বিবেচনা করা উচিত। গুরুত্বপূর্ণ তথ্যের জন্য, পেশাদার মানব অনুবাদ সুপারিশ করা হয়। এই অনুবাদ ব্যবহারের ফলে কোনো ভুল বোঝাবুঝি বা ভুল ব্যাখ্যা হলে আমরা দায়বদ্ধ থাকব না।\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernel_info": {
|
||||
"name": "conda-env-py37_tensorflow-py"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "py37_tensorflow",
|
||||
"language": "python",
|
||||
"name": "conda-env-py37_tensorflow-py"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.7.9"
|
||||
},
|
||||
"nteract": {
|
||||
"version": "nteract-front-end@1.0.0"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "81351e61f619b432ff51010a4f993194",
|
||||
"translation_date": "2025-08-28T12:17:33+00:00",
|
||||
"source_file": "lessons/5-NLP/16-RNN/RNNTF.ipynb",
|
||||
"language_code": "bn"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
|
|
@ -0,0 +1,414 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# জেনারেটিভ নেটওয়ার্ক\n",
|
||||
"\n",
|
||||
"Recurrent Neural Networks (RNNs) এবং তাদের গেটেড সেল ভ্যারিয়েন্ট যেমন Long Short Term Memory Cells (LSTMs) এবং Gated Recurrent Units (GRUs) ভাষা মডেলিংয়ের জন্য একটি পদ্ধতি প্রদান করে, অর্থাৎ তারা শব্দের ক্রম শিখতে পারে এবং একটি ক্রমে পরবর্তী শব্দের পূর্বাভাস দিতে পারে। এটি আমাদের RNNs ব্যবহার করে **জেনারেটিভ কাজ** সম্পাদন করতে সক্ষম করে, যেমন সাধারণ টেক্সট জেনারেশন, মেশিন ট্রান্সলেশন, এমনকি ইমেজ ক্যাপশনিং।\n",
|
||||
"\n",
|
||||
"পূর্ববর্তী ইউনিটে আমরা যে RNN আর্কিটেকচার আলোচনা করেছি, সেখানে প্রতিটি RNN ইউনিট পরবর্তী হিডেন স্টেটকে আউটপুট হিসেবে তৈরি করত। তবে, আমরা প্রতিটি রিকারেন্ট ইউনিটে আরেকটি আউটপুট যোগ করতে পারি, যা আমাদের একটি **সিকোয়েন্স** আউটপুট করতে সাহায্য করবে (যা মূল সিকোয়েন্সের সমান দৈর্ঘ্যের হবে)। এছাড়াও, আমরা এমন RNN ইউনিট ব্যবহার করতে পারি যা প্রতিটি ধাপে ইনপুট গ্রহণ করে না, বরং শুধুমাত্র একটি প্রাথমিক স্টেট ভেক্টর গ্রহণ করে এবং তারপর একটি আউটপুট সিকোয়েন্স তৈরি করে।\n",
|
||||
"\n",
|
||||
"এই নোটবুকে, আমরা সহজ জেনারেটিভ মডেলের উপর ফোকাস করব যা আমাদের টেক্সট জেনারেট করতে সাহায্য করে। সরলতার জন্য, চলুন **ক্যারেক্টার-লেভেল নেটওয়ার্ক** তৈরি করি, যা অক্ষর ধরে ধরে টেক্সট জেনারেট করে। প্রশিক্ষণের সময়, আমাদের কিছু টেক্সট কর্পাস নিতে হবে এবং এটিকে অক্ষরের সিকোয়েন্সে ভাগ করতে হবে।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Loading dataset...\n",
|
||||
"Building vocab...\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"import torchtext\n",
|
||||
"import numpy as np\n",
|
||||
"from torchnlp import *\n",
|
||||
"train_dataset,test_dataset,classes,vocab = load_dataset()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## চরিত্রের শব্দভাণ্ডার তৈরি করা\n",
|
||||
"\n",
|
||||
"চরিত্র-স্তরের জেনারেটিভ নেটওয়ার্ক তৈরি করতে, আমাদের পাঠ্যকে শব্দের পরিবর্তে পৃথক অক্ষরে বিভক্ত করতে হবে। এটি একটি ভিন্ন টোকেনাইজার সংজ্ঞায়িত করে করা যেতে পারে:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Vocabulary size = 82\n",
|
||||
"Encoding of 'a' is 1\n",
|
||||
"Character with code 13 is c\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def char_tokenizer(words):\n",
|
||||
" return list(words) #[word for word in words]\n",
|
||||
"\n",
|
||||
"counter = collections.Counter()\n",
|
||||
"for (label, line) in train_dataset:\n",
|
||||
" counter.update(char_tokenizer(line))\n",
|
||||
"vocab = torchtext.vocab.vocab(counter)\n",
|
||||
"\n",
|
||||
"vocab_size = len(vocab)\n",
|
||||
"print(f\"Vocabulary size = {vocab_size}\")\n",
|
||||
"print(f\"Encoding of 'a' is {vocab.get_stoi()['a']}\")\n",
|
||||
"print(f\"Character with code 13 is {vocab.get_itos()[13]}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"চলুন দেখি কীভাবে আমরা আমাদের ডেটাসেট থেকে টেক্সট এনকোড করতে পারি:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"tensor([ 0, 1, 2, 2, 3, 4, 5, 6, 3, 7, 8, 1, 9, 10, 3, 11, 2, 1,\n",
|
||||
" 12, 3, 7, 1, 13, 14, 3, 15, 16, 5, 17, 3, 5, 18, 8, 3, 7, 2,\n",
|
||||
" 1, 13, 14, 3, 19, 20, 8, 21, 5, 8, 9, 10, 22, 3, 20, 8, 21, 5,\n",
|
||||
" 8, 9, 10, 3, 23, 3, 4, 18, 17, 9, 5, 23, 10, 8, 2, 2, 8, 9,\n",
|
||||
" 10, 24, 3, 0, 1, 2, 2, 3, 4, 5, 9, 8, 8, 5, 25, 10, 3, 26,\n",
|
||||
" 12, 27, 16, 26, 2, 27, 16, 28, 29, 30, 1, 16, 26, 3, 17, 31, 3, 21,\n",
|
||||
" 2, 5, 9, 1, 23, 13, 32, 16, 27, 13, 10, 24, 3, 1, 9, 8, 3, 10,\n",
|
||||
" 8, 8, 27, 16, 28, 3, 28, 9, 8, 8, 16, 3, 1, 28, 1, 27, 16, 6])"
|
||||
]
|
||||
},
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def enc(x):\n",
|
||||
" return torch.LongTensor(encode(x,voc=vocab,tokenizer=char_tokenizer))\n",
|
||||
"\n",
|
||||
"enc(train_dataset[0][1])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## একটি জেনারেটিভ RNN প্রশিক্ষণ করা\n",
|
||||
"\n",
|
||||
"আমরা RNN-কে টেক্সট জেনারেট করার জন্য যেভাবে প্রশিক্ষণ দেব তা হলো নিম্নরূপ। প্রতিটি ধাপে, আমরা `nchars` দৈর্ঘ্যের একটি অক্ষরের ক্রম নেব এবং নেটওয়ার্ককে প্রতিটি ইনপুট অক্ষরের জন্য পরবর্তী আউটপুট অক্ষর তৈরি করতে বলব:\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"বাস্তব পরিস্থিতির উপর নির্ভর করে, আমরা কিছু বিশেষ অক্ষর অন্তর্ভুক্ত করতে চাইতে পারি, যেমন *end-of-sequence* `<eos>`। আমাদের ক্ষেত্রে, আমরা শুধু নেটওয়ার্ককে অবিরাম টেক্সট জেনারেশনের জন্য প্রশিক্ষণ দিতে চাই, তাই আমরা প্রতিটি ক্রমের আকারকে `nchars` টোকেনের সমান স্থির করব। ফলস্বরূপ, প্রতিটি প্রশিক্ষণ উদাহরণে থাকবে `nchars` ইনপুট এবং `nchars` আউটপুট (যা ইনপুট ক্রম এক প্রতীক বামে সরানো)। মিনিব্যাচে এমন কয়েকটি ক্রম থাকবে।\n",
|
||||
"\n",
|
||||
"আমরা যেভাবে মিনিব্যাচ তৈরি করব তা হলো প্রতিটি নিউজ টেক্সটের দৈর্ঘ্য `l` নিয়ে, এবং এর থেকে সমস্ত সম্ভাব্য ইনপুট-আউটপুট সংমিশ্রণ তৈরি করা (এখানে `l-nchars` এমন সংমিশ্রণ থাকবে)। এগুলো একটি মিনিব্যাচ গঠন করবে, এবং প্রতিটি প্রশিক্ষণ ধাপে মিনিব্যাচের আকার ভিন্ন হবে।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(tensor([[ 0, 1, 2, ..., 28, 29, 30],\n",
|
||||
" [ 1, 2, 2, ..., 29, 30, 1],\n",
|
||||
" [ 2, 2, 3, ..., 30, 1, 16],\n",
|
||||
" ...,\n",
|
||||
" [20, 8, 21, ..., 1, 28, 1],\n",
|
||||
" [ 8, 21, 5, ..., 28, 1, 27],\n",
|
||||
" [21, 5, 8, ..., 1, 27, 16]]),\n",
|
||||
" tensor([[ 1, 2, 2, ..., 29, 30, 1],\n",
|
||||
" [ 2, 2, 3, ..., 30, 1, 16],\n",
|
||||
" [ 2, 3, 4, ..., 1, 16, 26],\n",
|
||||
" ...,\n",
|
||||
" [ 8, 21, 5, ..., 28, 1, 27],\n",
|
||||
" [21, 5, 8, ..., 1, 27, 16],\n",
|
||||
" [ 5, 8, 9, ..., 27, 16, 6]]))"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"nchars = 100\n",
|
||||
"\n",
|
||||
"def get_batch(s,nchars=nchars):\n",
|
||||
" ins = torch.zeros(len(s)-nchars,nchars,dtype=torch.long,device=device)\n",
|
||||
" outs = torch.zeros(len(s)-nchars,nchars,dtype=torch.long,device=device)\n",
|
||||
" for i in range(len(s)-nchars):\n",
|
||||
" ins[i] = enc(s[i:i+nchars])\n",
|
||||
" outs[i] = enc(s[i+1:i+nchars+1])\n",
|
||||
" return ins,outs\n",
|
||||
"\n",
|
||||
"get_batch(train_dataset[0][1])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"এখন আমরা জেনারেটর নেটওয়ার্ক সংজ্ঞায়িত করব। এটি পূর্ববর্তী ইউনিটে আলোচনা করা যেকোনো পুনরাবৃত্ত সেলের (সাধারণ, LSTM বা GRU) উপর ভিত্তি করে হতে পারে। আমাদের উদাহরণে আমরা LSTM ব্যবহার করব।\n",
|
||||
"\n",
|
||||
"কারণ নেটওয়ার্কটি ইনপুট হিসাবে অক্ষর গ্রহণ করে এবং শব্দভাণ্ডারের আকার বেশ ছোট, তাই আমাদের এম্বেডিং লেয়ারের প্রয়োজন নেই; এক-হট-এনকোড করা ইনপুট সরাসরি LSTM সেলে যেতে পারে। তবে, যেহেতু আমরা ইনপুট হিসাবে অক্ষরের সংখ্যা পাস করি, তাই LSTM-এ পাঠানোর আগে সেগুলোকে এক-হট-এনকোড করতে হবে। এটি `forward` পাসের সময় `one_hot` ফাংশন কল করার মাধ্যমে সম্পন্ন হয়। আউটপুট এনকোডার হবে একটি লিনিয়ার লেয়ার, যা লুকানো অবস্থাকে এক-হট-এনকোড করা আউটপুটে রূপান্তর করবে।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class LSTMGenerator(torch.nn.Module):\n",
|
||||
" def __init__(self, vocab_size, hidden_dim):\n",
|
||||
" super().__init__()\n",
|
||||
" self.rnn = torch.nn.LSTM(vocab_size,hidden_dim,batch_first=True)\n",
|
||||
" self.fc = torch.nn.Linear(hidden_dim, vocab_size)\n",
|
||||
"\n",
|
||||
" def forward(self, x, s=None):\n",
|
||||
" x = torch.nn.functional.one_hot(x,vocab_size).to(torch.float32)\n",
|
||||
" x,s = self.rnn(x,s)\n",
|
||||
" return self.fc(x),s"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"প্রশিক্ষণের সময়, আমরা তৈরি করা টেক্সট থেকে নমুনা নিতে চাই। এটি করার জন্য, আমরা একটি `generate` ফাংশন সংজ্ঞায়িত করব যা প্রাথমিক স্ট্রিং `start` থেকে শুরু করে `size` দৈর্ঘ্যের একটি আউটপুট স্ট্রিং তৈরি করবে।\n",
|
||||
"\n",
|
||||
"এর কাজ করার পদ্ধতি নিম্নরূপ। প্রথমে, আমরা পুরো `start` স্ট্রিংটি নেটওয়ার্কের মাধ্যমে পাঠাব এবং আউটপুট স্টেট `s` এবং পরবর্তী পূর্বাভাসিত অক্ষর `out` গ্রহণ করব। যেহেতু `out` এক-হট এনকোড করা থাকে, আমরা `argmax` ব্যবহার করে শব্দভাণ্ডারে অক্ষরের সূচক `nc` বের করব এবং `itos` ব্যবহার করে প্রকৃত অক্ষরটি নির্ধারণ করব এবং এটি `chars` নামক অক্ষরের তালিকায় যোগ করব। এই প্রক্রিয়াটি একবারে একটি অক্ষর তৈরি করার জন্য `size` বার পুনরাবৃত্তি করা হয়, যাতে প্রয়োজনীয় সংখ্যক অক্ষর তৈরি করা যায়।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def generate(net,size=100,start='today '):\n",
|
||||
" chars = list(start)\n",
|
||||
" out, s = net(enc(chars).view(1,-1).to(device))\n",
|
||||
" for i in range(size):\n",
|
||||
" nc = torch.argmax(out[0][-1])\n",
|
||||
" chars.append(vocab.get_itos()[nc])\n",
|
||||
" out, s = net(nc.view(1,-1),s)\n",
|
||||
" return ''.join(chars)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"এবার চলুন প্রশিক্ষণ শুরু করি! প্রশিক্ষণ লুপটি আমাদের আগের সব উদাহরণের মতোই, তবে এখানে নির্ভুলতার পরিবর্তে আমরা প্রতি ১০০০ এপোক পরপর নমুনা হিসেবে তৈরি করা টেক্সট প্রিন্ট করব।\n",
|
||||
"\n",
|
||||
"বিশেষ মনোযোগ দিতে হবে যেভাবে আমরা লস গণনা করি। আমাদের লস গণনা করতে হবে এক-হট-এনকোড করা আউটপুট `out` এবং প্রত্যাশিত টেক্সট `text_out` (যা চরিত্রের ইনডেক্সের তালিকা) ব্যবহার করে। সৌভাগ্যক্রমে, `cross_entropy` ফাংশনটি প্রথম আর্গুমেন্ট হিসেবে অপরিশোধিত নেটওয়ার্ক আউটপুট এবং দ্বিতীয় আর্গুমেন্ট হিসেবে ক্লাস নম্বর আশা করে, যা আমাদের কাছে ঠিকই আছে। এটি মিনিব্যাচ সাইজের উপর স্বয়ংক্রিয় গড়ও করে।\n",
|
||||
"\n",
|
||||
"আমরা প্রশিক্ষণকে `samples_to_train` নমুনা দিয়ে সীমাবদ্ধ রাখি, যাতে খুব বেশি সময় অপেক্ষা করতে না হয়। আমরা আপনাকে উৎসাহিত করি দীর্ঘ সময় ধরে প্রশিক্ষণ চালিয়ে দেখতে, সম্ভবত কয়েকটি এপোকের জন্য (যে ক্ষেত্রে আপনাকে এই কোডের চারপাশে আরেকটি লুপ তৈরি করতে হবে)।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Current loss = 4.398899078369141\n",
|
||||
"today sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr s\n",
|
||||
"Current loss = 2.161320447921753\n",
|
||||
"today and to the tor to to the tor to to the tor to to the tor to to the tor to to the tor to to the tor t\n",
|
||||
"Current loss = 1.6722588539123535\n",
|
||||
"today and the court to the could to the could to the could to the could to the could to the could to the c\n",
|
||||
"Current loss = 2.423795223236084\n",
|
||||
"today and a second to the conternation of the conternation of the conternation of the conternation of the \n",
|
||||
"Current loss = 1.702607274055481\n",
|
||||
"today and the company to the company to the company to the company to the company to the company to the co\n",
|
||||
"Current loss = 1.692358136177063\n",
|
||||
"today and the company to the company to the company to the company to the company to the company to the co\n",
|
||||
"Current loss = 1.9722288846969604\n",
|
||||
"today and the control the control the control the control the control the control the control the control \n",
|
||||
"Current loss = 1.8705692291259766\n",
|
||||
"today and the second to the second to the second to the second to the second to the second to the second t\n",
|
||||
"Current loss = 1.7626899480819702\n",
|
||||
"today and a security and a security and a security and a security and a security and a security and a secu\n",
|
||||
"Current loss = 1.5574463605880737\n",
|
||||
"today and the company and the company and the company and the company and the company and the company and \n",
|
||||
"Current loss = 1.5620026588439941\n",
|
||||
"today and the be that the be the be that the be the be that the be the be that the be the be that the be t\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"net = LSTMGenerator(vocab_size,64).to(device)\n",
|
||||
"\n",
|
||||
"samples_to_train = 10000\n",
|
||||
"optimizer = torch.optim.Adam(net.parameters(),0.01)\n",
|
||||
"loss_fn = torch.nn.CrossEntropyLoss()\n",
|
||||
"net.train()\n",
|
||||
"for i,x in enumerate(train_dataset):\n",
|
||||
" # x[0] is class label, x[1] is text\n",
|
||||
" if len(x[1])-nchars<10:\n",
|
||||
" continue\n",
|
||||
" samples_to_train-=1\n",
|
||||
" if not samples_to_train: break\n",
|
||||
" text_in, text_out = get_batch(x[1])\n",
|
||||
" optimizer.zero_grad()\n",
|
||||
" out,s = net(text_in)\n",
|
||||
" loss = torch.nn.functional.cross_entropy(out.view(-1,vocab_size),text_out.flatten()) #cross_entropy(out,labels)\n",
|
||||
" loss.backward()\n",
|
||||
" optimizer.step()\n",
|
||||
" if i%1000==0:\n",
|
||||
" print(f\"Current loss = {loss.item()}\")\n",
|
||||
" print(generate(net))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"এই উদাহরণটি ইতিমধ্যেই বেশ ভালো টেক্সট তৈরি করে, তবে এটি আরও কয়েকটি উপায়ে উন্নত করা যেতে পারে:\n",
|
||||
"\n",
|
||||
"* **উন্নত মিনি-ব্যাচ জেনারেশন**। আমরা প্রশিক্ষণের জন্য ডেটা প্রস্তুত করার সময় এক নমুনা থেকে এক মিনি-ব্যাচ তৈরি করেছি। এটি আদর্শ নয়, কারণ মিনি-ব্যাচগুলোর আকার বিভিন্ন হয়, এবং কিছু ক্ষেত্রে মিনি-ব্যাচ তৈরি করা সম্ভব হয় না, কারণ টেক্সট `nchars` এর চেয়ে ছোট। এছাড়াও, ছোট মিনি-ব্যাচ GPU যথেষ্ট পরিমাণে লোড করতে পারে না। আরও বুদ্ধিমানের কাজ হবে সমস্ত নমুনা থেকে একটি বড় টেক্সটের অংশ নেওয়া, তারপর সমস্ত ইনপুট-আউটপুট জোড়া তৈরি করা, সেগুলোকে এলোমেলো করা এবং সমান আকারের মিনি-ব্যাচ তৈরি করা।\n",
|
||||
"\n",
|
||||
"* **মাল্টিলেয়ার LSTM**। ২ বা ৩ স্তরের LSTM সেল ব্যবহার করার চেষ্টা করা যুক্তিসঙ্গত। আমরা আগের ইউনিটে উল্লেখ করেছি, LSTM এর প্রতিটি স্তর টেক্সট থেকে নির্দিষ্ট প্যাটার্ন বের করে, এবং ক্যারেক্টার-লেভেল জেনারেটরের ক্ষেত্রে আমরা আশা করতে পারি যে নিম্ন স্তরের LSTM শব্দাংশ বের করার জন্য দায়ী হবে, এবং উচ্চ স্তরগুলো শব্দ এবং শব্দের সংমিশ্রণ বের করবে। এটি সহজেই LSTM কনস্ট্রাক্টরে স্তরের সংখ্যা প্যারামিটার পাস করে বাস্তবায়ন করা যেতে পারে।\n",
|
||||
"\n",
|
||||
"* আপনি **GRU ইউনিট** নিয়ে পরীক্ষা করতে পারেন এবং দেখতে পারেন কোনটি ভালো কাজ করে, এবং **বিভিন্ন হিডেন লেয়ার সাইজ** নিয়ে পরীক্ষা করতে পারেন। খুব বড় হিডেন লেয়ার ওভারফিটিং ঘটাতে পারে (যেমন, নেটওয়ার্ক সঠিক টেক্সট শিখে নেবে), এবং ছোট আকার ভালো ফলাফল দিতে পারে না।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## নরম টেক্সট জেনারেশন এবং টেম্পারেচার\n",
|
||||
"\n",
|
||||
"`generate` এর পূর্ববর্তী সংজ্ঞায়, আমরা সর্বদা সর্বোচ্চ সম্ভাবনাযুক্ত চরিত্রটিকে পরবর্তী চরিত্র হিসেবে গ্রহণ করছিলাম। এর ফলে প্রায়ই টেক্সট একই চরিত্রের ক্রম বারবার পুনরাবৃত্তি করত, যেমন এই উদাহরণে:\n",
|
||||
"```\n",
|
||||
"today of the second the company and a second the company ...\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"তবে, যদি আমরা পরবর্তী চরিত্রের সম্ভাবনা বিতরণটি দেখি, তাহলে দেখা যেতে পারে যে কয়েকটি সর্বোচ্চ সম্ভাবনার মধ্যে পার্থক্য খুব বেশি নয়, যেমন একটি চরিত্রের সম্ভাবনা হতে পারে ০.২, আরেকটি - ০.১৯ ইত্যাদি। উদাহরণস্বরূপ, যদি আমরা '*play*' ক্রমের পরবর্তী চরিত্র খুঁজছি, তাহলে পরবর্তী চরিত্রটি সমানভাবে স্পেস হতে পারে, অথবা **e** (যেমন শব্দটি *player* এ)।\n",
|
||||
"\n",
|
||||
"এটি আমাদের এই সিদ্ধান্তে পৌঁছায় যে সর্বোচ্চ সম্ভাবনাযুক্ত চরিত্রটি নির্বাচন করা সবসময় \"ন্যায্য\" নয়, কারণ দ্বিতীয় সর্বোচ্চটি নির্বাচন করলেও তা অর্থপূর্ণ টেক্সট তৈরি করতে পারে। এটি আরও বুদ্ধিমানের কাজ হবে যদি আমরা নেটওয়ার্ক আউটপুট দ্বারা প্রদত্ত সম্ভাবনা বিতরণ থেকে চরিত্রগুলি **নমুনা** করি।\n",
|
||||
"\n",
|
||||
"এই নমুনা প্রক্রিয়া `multinomial` ফাংশন ব্যবহার করে করা যেতে পারে, যা তথাকথিত **multinomial distribution** বাস্তবায়ন করে। একটি ফাংশন যা এই **নরম** টেক্সট জেনারেশন বাস্তবায়ন করে তা নিচে সংজ্ঞায়িত করা হয়েছে:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"--- Temperature = 0.3\n",
|
||||
"Today and a company and complete an all the land the restrational the as a security and has provers the pay to and a report and the computer in the stand has filities and working the law the stations for a company and with the company and the final the first company and refight of the state and and workin\n",
|
||||
"\n",
|
||||
"--- Temperature = 0.8\n",
|
||||
"Today he oniis its first to Aus bomblaties the marmation a to manan boogot that pirate assaid a relaid their that goverfin the the Cappets Ecrotional Assonia Cition targets it annight the w scyments Blamity #39;s TVeer Diercheg Reserals fran envyuil that of ster said access what succers of Dour-provelith\n",
|
||||
"\n",
|
||||
"--- Temperature = 1.0\n",
|
||||
"Today holy they a 11 will meda a toket subsuaties, engins for Chanos, they's has stainger past to opening orital his thempting new Nattona was al innerforder advan-than #36;s night year his religuled talitatian what the but with Wednesday to Justment will wemen of Mark CCC Camp as Timed Nae wome a leaders\n",
|
||||
"\n",
|
||||
"--- Temperature = 1.3\n",
|
||||
"Today gpone 2.5 fech atcusion poor cocles toparsdorM.cht Line Pamage put 43 his calt lowed to the book, that has authh-the silia rruch ailing to'ory andhes beutirsimi- Aefffive heading offil an auf eacklets is charged evis, Gunymy oy) Mony has it after-sloythyor loveId out filme, the Natabl -Najuntaxiggs \n",
|
||||
"\n",
|
||||
"--- Temperature = 1.8\n",
|
||||
"Today plary, P.slan chly\\401 mardregationly #39;t 8.1Mide) closes ,filtcon alfly playin roven!\\grea.-QFBEP: Iss onfarchQ/itilia CCf Zivesigntwasta orce.-Peul-aw.uicrin of fuglinfsut aftaningwo, MIEX awayew Aice Woiduar Corvagiugge oppo esig ThusBratourid canthly-RyI.co lagitems\\eexciaishes.conBabntusmor I\n",
|
||||
"\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def generate_soft(net,size=100,start='today ',temperature=1.0):\n",
|
||||
" chars = list(start)\n",
|
||||
" out, s = net(enc(chars).view(1,-1).to(device))\n",
|
||||
" for i in range(size):\n",
|
||||
" #nc = torch.argmax(out[0][-1])\n",
|
||||
" out_dist = out[0][-1].div(temperature).exp()\n",
|
||||
" nc = torch.multinomial(out_dist,1)[0]\n",
|
||||
" chars.append(vocab.get_itos()[nc])\n",
|
||||
" out, s = net(nc.view(1,-1),s)\n",
|
||||
" return ''.join(chars)\n",
|
||||
" \n",
|
||||
"for i in [0.3,0.8,1.0,1.3,1.8]:\n",
|
||||
" print(f\"--- Temperature = {i}\\n{generate_soft(net,size=300,start='Today ',temperature=i)}\\n\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"আমরা **temperature** নামে একটি অতিরিক্ত প্যারামিটার পরিচয় করিয়েছি, যা নির্দেশ করে যে আমাদের সর্বোচ্চ সম্ভাবনার প্রতি কতটা কঠোরভাবে অনুসরণ করা উচিত। যদি temperature 1.0 হয়, আমরা ন্যায্য মাল্টিনোমিয়াল স্যাম্পলিং করি, এবং যখন temperature অসীমে পৌঁছায় - সমস্ত সম্ভাবনা সমান হয়ে যায়, এবং আমরা পরবর্তী অক্ষরটি এলোমেলোভাবে নির্বাচন করি। নিচের উদাহরণে আমরা দেখতে পারি যে temperature খুব বেশি বাড়ালে টেক্সট অর্থহীন হয়ে যায়, এবং এটি \"cycled\" কঠোরভাবে তৈরি টেক্সটের মতো হয়ে যায় যখন এটি 0-এর কাছাকাছি হয়।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**অস্বীকৃতি**: \nএই নথিটি AI অনুবাদ পরিষেবা [Co-op Translator](https://github.com/Azure/co-op-translator) ব্যবহার করে অনুবাদ করা হয়েছে। আমরা যথাসম্ভব সঠিক অনুবাদের চেষ্টা করি, তবে অনুগ্রহ করে মনে রাখবেন যে স্বয়ংক্রিয় অনুবাদে ত্রুটি বা অসঙ্গতি থাকতে পারে। মূল ভাষায় থাকা নথিটিকে প্রামাণিক উৎস হিসেবে বিবেচনা করা উচিত। গুরুত্বপূর্ণ তথ্যের জন্য, পেশাদার মানব অনুবাদ সুপারিশ করা হয়। এই অনুবাদ ব্যবহারের ফলে কোনো ভুল বোঝাবুঝি বা ভুল ব্যাখ্যা হলে আমরা দায়বদ্ধ থাকব না।\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.8.12 ('py38')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "7673cd150d96c74c6d6011460094efb4",
|
||||
"translation_date": "2025-08-28T12:01:57+00:00",
|
||||
"source_file": "lessons/5-NLP/17-GenerativeNetworks/GenerativePyTorch.ipynb",
|
||||
"language_code": "bn"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
|
|
@ -0,0 +1,497 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# জেনারেটিভ নেটওয়ার্ক\n",
|
||||
"\n",
|
||||
"Recurrent Neural Networks (RNNs) এবং তাদের গেটেড সেল ভ্যারিয়েন্ট যেমন Long Short Term Memory Cells (LSTMs) এবং Gated Recurrent Units (GRUs) ভাষা মডেলিংয়ের জন্য একটি পদ্ধতি প্রদান করে, অর্থাৎ তারা শব্দের ক্রম শিখতে পারে এবং একটি ক্রমে পরবর্তী শব্দের পূর্বাভাস দিতে পারে। এটি আমাদের RNNs ব্যবহার করে **জেনারেটিভ কাজ** সম্পাদন করতে দেয়, যেমন সাধারণ টেক্সট জেনারেশন, মেশিন ট্রান্সলেশন, এবং এমনকি ইমেজ ক্যাপশনিং।\n",
|
||||
"\n",
|
||||
"পূর্ববর্তী ইউনিটে আমরা যে RNN আর্কিটেকচার আলোচনা করেছি, সেখানে প্রতিটি RNN ইউনিট পরবর্তী হিডেন স্টেটকে আউটপুট হিসেবে তৈরি করত। তবে, আমরা প্রতিটি রিকারেন্ট ইউনিটে আরেকটি আউটপুট যোগ করতে পারি, যা আমাদের একটি **সিকোয়েন্স** আউটপুট করতে সাহায্য করবে (যা মূল সিকোয়েন্সের দৈর্ঘ্যের সমান)। এছাড়াও, আমরা এমন RNN ইউনিট ব্যবহার করতে পারি যা প্রতিটি ধাপে ইনপুট গ্রহণ করে না, বরং শুধুমাত্র একটি প্রাথমিক স্টেট ভেক্টর গ্রহণ করে এবং তারপর আউটপুটের একটি সিকোয়েন্স তৈরি করে।\n",
|
||||
"\n",
|
||||
"এই নোটবুকে, আমরা সহজ জেনারেটিভ মডেলের উপর ফোকাস করব যা আমাদের টেক্সট জেনারেট করতে সাহায্য করে। সরলতার জন্য, আসুন **ক্যারেক্টার-লেভেল নেটওয়ার্ক** তৈরি করি, যা অক্ষর ধরে ধরে টেক্সট জেনারেট করে। প্রশিক্ষণের সময়, আমাদের কিছু টেক্সট কর্পাস নিতে হবে এবং এটিকে অক্ষরের সিকোয়েন্সে ভাগ করতে হবে।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import tensorflow as tf\n",
|
||||
"from tensorflow import keras\n",
|
||||
"import tensorflow_datasets as tfds\n",
|
||||
"import numpy as np\n",
|
||||
"\n",
|
||||
"ds_train, ds_test = tfds.load('ag_news_subset').values()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## চরিত্র ভোকাবুলারি তৈরি করা\n",
|
||||
"\n",
|
||||
"চরিত্র-স্তরের জেনারেটিভ নেটওয়ার্ক তৈরি করতে, আমাদের টেক্সটকে শব্দের পরিবর্তে পৃথক চরিত্রে বিভক্ত করতে হবে। `TextVectorization` লেয়ার, যা আমরা আগে ব্যবহার করেছি, এটি করতে পারে না, তাই আমাদের দুটি বিকল্প রয়েছে:\n",
|
||||
"\n",
|
||||
"* ম্যানুয়ালি টেক্সট লোড করে এবং 'হাতে' টোকেনাইজেশন করা, যেমন [এই অফিসিয়াল কেরাস উদাহরণে](https://keras.io/examples/generative/lstm_character_level_text_generation/) দেখানো হয়েছে\n",
|
||||
"* চরিত্র-স্তরের টোকেনাইজেশনের জন্য `Tokenizer` ক্লাস ব্যবহার করা।\n",
|
||||
"\n",
|
||||
"আমরা দ্বিতীয় বিকল্পটি বেছে নেব। `Tokenizer` শব্দে টোকেনাইজ করার জন্যও ব্যবহার করা যেতে পারে, তাই চরিত্র-স্তর থেকে শব্দ-স্তরের টোকেনাইজেশনে সহজেই স্যুইচ করা সম্ভব।\n",
|
||||
"\n",
|
||||
"চরিত্র-স্তরের টোকেনাইজেশন করতে, আমাদের `char_level=True` প্যারামিটার পাস করতে হবে:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def extract_text(x):\n",
|
||||
" return x['title']+' '+x['description']\n",
|
||||
"\n",
|
||||
"def tupelize(x):\n",
|
||||
" return (extract_text(x),x['label'])\n",
|
||||
"\n",
|
||||
"tokenizer = keras.preprocessing.text.Tokenizer(char_level=True,lower=False)\n",
|
||||
"tokenizer.fit_on_texts([x['title'].numpy().decode('utf-8') for x in ds_train])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"আমরা একটি বিশেষ টোকেন ব্যবহার করতে চাই যা **সিকোয়েন্সের শেষ** নির্দেশ করবে, যাকে আমরা `<eos>` বলব। চলুন এটি ম্যানুয়ালি ভোকাবুলারিতে যোগ করি:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"eos_token = len(tokenizer.word_index)+1\n",
|
||||
"tokenizer.word_index['<eos>'] = eos_token\n",
|
||||
"\n",
|
||||
"vocab_size = eos_token + 1"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"এখন, পাঠ্যকে সংখ্যার ক্রমে এনকোড করতে, আমরা ব্যবহার করতে পারি:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[[48, 2, 10, 10, 5, 44, 1, 25, 5, 8, 10, 13, 78]]"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"tokenizer.texts_to_sequences(['Hello, world!'])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## শিরোনাম তৈরি করার জন্য একটি জেনারেটিভ RNN প্রশিক্ষণ\n",
|
||||
"\n",
|
||||
"আমরা যেভাবে RNN প্রশিক্ষণ করব যাতে এটি সংবাদ শিরোনাম তৈরি করতে পারে তা হলো নিম্নরূপ। প্রতিটি ধাপে, আমরা একটি শিরোনাম নেব, যা RNN-এ প্রবেশ করানো হবে, এবং প্রতিটি ইনপুট অক্ষরের জন্য আমরা নেটওয়ার্ককে পরবর্তী আউটপুট অক্ষর তৈরি করতে বলব:\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"আমাদের ক্রমের শেষ অক্ষরের জন্য, আমরা নেটওয়ার্ককে `<eos>` টোকেন তৈরি করতে বলব।\n",
|
||||
"\n",
|
||||
"আমরা এখানে যে জেনারেটিভ RNN ব্যবহার করছি তার প্রধান পার্থক্য হলো আমরা RNN-এর প্রতিটি ধাপ থেকে আউটপুট নেব, শুধুমাত্র চূড়ান্ত সেল থেকে নয়। এটি `return_sequences` প্যারামিটার RNN সেলে নির্দিষ্ট করে অর্জন করা যেতে পারে।\n",
|
||||
"\n",
|
||||
"তাই, প্রশিক্ষণের সময়, নেটওয়ার্কে ইনপুট হবে নির্দিষ্ট দৈর্ঘ্যের এনকোড করা অক্ষরের একটি ক্রম, এবং আউটপুট হবে একই দৈর্ঘ্যের একটি ক্রম, তবে এক উপাদান দ্বারা স্থানান্তরিত এবং `<eos>` দিয়ে শেষ। মিনিব্যাচে এমন কয়েকটি ক্রম থাকবে, এবং আমাদের **প্যাডিং** ব্যবহার করতে হবে যাতে সব ক্রম একসঙ্গে সামঞ্জস্যপূর্ণ হয়।\n",
|
||||
"\n",
|
||||
"চলুন এমন ফাংশন তৈরি করি যা আমাদের জন্য ডেটাসেট রূপান্তর করবে। কারণ আমরা মিনিব্যাচ স্তরে ক্রম প্যাড করতে চাই, আমরা প্রথমে `.batch()` কল করে ডেটাসেট ব্যাচ করব, এবং তারপর `map` ব্যবহার করে এটি রূপান্তর করব। সুতরাং, রূপান্তর ফাংশন একটি সম্পূর্ণ মিনিব্যাচকে প্যারামিটার হিসেবে নেবে:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def title_batch(x):\n",
|
||||
" x = [t.numpy().decode('utf-8') for t in x]\n",
|
||||
" z = tokenizer.texts_to_sequences(x)\n",
|
||||
" z = tf.keras.preprocessing.sequence.pad_sequences(z)\n",
|
||||
" return tf.one_hot(z,vocab_size), tf.one_hot(tf.concat([z[:,1:],tf.constant(eos_token,shape=(len(z),1))],axis=1),vocab_size)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"কিছু গুরুত্বপূর্ণ বিষয় যা আমরা এখানে করি:\n",
|
||||
"* প্রথমে আমরা স্ট্রিং টেনসর থেকে প্রকৃত টেক্সট বের করি\n",
|
||||
"* `text_to_sequences` স্ট্রিংগুলোর তালিকাকে পূর্ণসংখ্যার টেনসরের তালিকায় রূপান্তর করে\n",
|
||||
"* `pad_sequences` সেই টেনসরগুলোকে তাদের সর্বোচ্চ দৈর্ঘ্যে প্যাড করে\n",
|
||||
"* শেষে আমরা সমস্ত অক্ষরকে ওয়ান-হট এনকোড করি, এবং শিফটিং ও `<eos>` যোগ করার কাজও করি। কেন আমাদের ওয়ান-হট-এনকোডেড অক্ষর প্রয়োজন, তা আমরা শীঘ্রই দেখব\n",
|
||||
"\n",
|
||||
"তবে, এই ফাংশনটি **Pythonic**, অর্থাৎ এটি স্বয়ংক্রিয়ভাবে Tensorflow এর গণনামূলক গ্রাফে রূপান্তরিত হতে পারে না। যদি আমরা সরাসরি এই ফাংশনটি `Dataset.map` ফাংশনে ব্যবহার করার চেষ্টা করি, তাহলে ত্রুটি পাব। আমাদের এই Pythonic কলটিকে `py_function` র্যাপার ব্যবহার করে আবদ্ধ করতে হবে:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def title_batch_fn(x):\n",
|
||||
" x = x['title']\n",
|
||||
" a,b = tf.py_function(title_batch,inp=[x],Tout=(tf.float32,tf.float32))\n",
|
||||
" return a,b"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Note**: পাইথনিক এবং টেনসরফ্লো ট্রান্সফরমেশন ফাংশনের মধ্যে পার্থক্য করা কিছুটা জটিল মনে হতে পারে, এবং আপনি হয়তো ভাবছেন কেন আমরা ডেটাসেটকে `fit`-এ পাঠানোর আগে স্ট্যান্ডার্ড পাইথন ফাংশন ব্যবহার করে ট্রান্সফর্ম করি না। যদিও এটি অবশ্যই করা সম্ভব, `Dataset.map` ব্যবহার করার একটি বড় সুবিধা রয়েছে, কারণ ডেটা ট্রান্সফরমেশন পাইপলাইন টেনসরফ্লো কম্পিউটেশনাল গ্রাফ ব্যবহার করে সম্পাদিত হয়, যা GPU কম্পিউটেশনের সুবিধা নেয় এবং CPU/GPU এর মধ্যে ডেটা পাস করার প্রয়োজনীয়তা কমিয়ে আনে।\n",
|
||||
"\n",
|
||||
"এখন আমরা আমাদের জেনারেটর নেটওয়ার্ক তৈরি করতে পারি এবং প্রশিক্ষণ শুরু করতে পারি। এটি পূর্ববর্তী ইউনিটে আলোচনা করা যেকোনো রিকারেন্ট সেল (সিম্পল, LSTM বা GRU) এর উপর ভিত্তি করে হতে পারে। আমাদের উদাহরণে আমরা LSTM ব্যবহার করব।\n",
|
||||
"\n",
|
||||
"যেহেতু নেটওয়ার্কটি ইনপুট হিসেবে অক্ষর গ্রহণ করে এবং ভোকাবুলারি সাইজ বেশ ছোট, তাই আমাদের এম্বেডিং লেয়ারের প্রয়োজন নেই; ওয়ান-হট-এনকোডেড ইনপুট সরাসরি LSTM সেলে যেতে পারে। আউটপুট লেয়ারটি একটি `Dense` ক্লাসিফায়ার হবে যা LSTM আউটপুটকে ওয়ান-হট-এনকোডেড টোকেন নম্বরে রূপান্তর করবে।\n",
|
||||
"\n",
|
||||
"এর পাশাপাশি, যেহেতু আমরা ভ্যারিয়েবল-লেংথ সিকোয়েন্স নিয়ে কাজ করছি, আমরা `Masking` লেয়ার ব্যবহার করতে পারি একটি মাস্ক তৈরি করার জন্য যা স্ট্রিংয়ের প্যাডেড অংশকে উপেক্ষা করবে। এটি কঠোরভাবে প্রয়োজনীয় নয়, কারণ `<eos>` টোকেনের বাইরে যা কিছু আছে তাতে আমরা খুব বেশি আগ্রহী নই, তবে আমরা এই লেয়ার টাইপের সাথে কিছু অভিজ্ঞতা অর্জনের জন্য এটি ব্যবহার করব। `input_shape` হবে `(None, vocab_size)`, যেখানে `None` ভ্যারিয়েবল লেংথ সিকোয়েন্স নির্দেশ করে, এবং আউটপুট শেপও `(None, vocab_size)` হবে, যেমনটি আপনি `summary` থেকে দেখতে পারেন।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"sequential\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
"Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
"masking (Masking) (None, None, 84) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"lstm (LSTM) (None, None, 128) 109056 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense (Dense) (None, None, 84) 10836 \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 119,892\n",
|
||||
"Trainable params: 119,892\n",
|
||||
"Non-trainable params: 0\n",
|
||||
"_________________________________________________________________\n",
|
||||
"15000/15000 [==============================] - 229s 15ms/step - loss: 1.5385\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7fa40c1245e0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model = keras.models.Sequential([\n",
|
||||
" keras.layers.Masking(input_shape=(None,vocab_size)),\n",
|
||||
" keras.layers.LSTM(128,return_sequences=True),\n",
|
||||
" keras.layers.Dense(vocab_size,activation='softmax')\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"model.summary()\n",
|
||||
"model.compile(loss='categorical_crossentropy')\n",
|
||||
"\n",
|
||||
"model.fit(ds_train.batch(8).map(title_batch_fn))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## আউটপুট তৈরি করা\n",
|
||||
"\n",
|
||||
"এখন আমরা মডেলটি প্রশিক্ষণ দিয়েছি, আমরা এটি ব্যবহার করে কিছু আউটপুট তৈরি করতে চাই। প্রথমেই, আমাদের একটি পদ্ধতি দরকার যা টোকেন নম্বরের সিকোয়েন্স দ্বারা উপস্থাপিত টেক্সট ডিকোড করতে পারে। এটি করার জন্য, আমরা `tokenizer.sequences_to_texts` ফাংশন ব্যবহার করতে পারি; তবে, এটি চরিত্র-স্তরের টোকেনাইজেশনের ক্ষেত্রে ভালো কাজ করে না। তাই আমরা টোকেনাইজারের একটি টোকেনের ডিকশনারি (যাকে `word_index` বলা হয়) নেব, একটি রিভার্স ম্যাপ তৈরি করব এবং আমাদের নিজস্ব ডিকোডিং ফাংশন লিখব:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"reverse_map = {val:key for key, val in tokenizer.word_index.items()}\n",
|
||||
"\n",
|
||||
"def decode(x):\n",
|
||||
" return ''.join([reverse_map[t] for t in x])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"এখন, চলুন জেনারেশন শুরু করি। আমরা প্রথমে কিছু স্ট্রিং `start` দিয়ে শুরু করব, এটিকে একটি সিকোয়েন্স `inp`-এ এনকোড করব, এবং তারপর প্রতিটি ধাপে আমাদের নেটওয়ার্ককে কল করব পরবর্তী অক্ষর নির্ধারণের জন্য।\n",
|
||||
"\n",
|
||||
"নেটওয়ার্কের আউটপুট `out` একটি ভেক্টর, যেখানে `vocab_size` সংখ্যক উপাদান থাকে, যা প্রতিটি টোকেনের সম্ভাব্যতা উপস্থাপন করে। আমরা `argmax` ব্যবহার করে সবচেয়ে সম্ভাব্য টোকেন নম্বর খুঁজে বের করতে পারি। এরপর আমরা এই অক্ষরটি জেনারেট করা টোকেনগুলোর তালিকায় যোগ করি এবং জেনারেশনের প্রক্রিয়া চালিয়ে যাই। এই প্রক্রিয়াটি একবারে একটি অক্ষর জেনারেট করার জন্য `size` বার পুনরাবৃত্তি করা হয়, যতক্ষণ না প্রয়োজনীয় সংখ্যক অক্ষর তৈরি হয়। আমরা প্রক্রিয়াটি আগেই বন্ধ করি যদি `eos_token` পাওয়া যায়।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"'Today #39;s lead to strike for the strike for the strike for the strike (AFP)'"
|
||||
]
|
||||
},
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def generate(model,size=100,start='Today '):\n",
|
||||
" inp = tokenizer.texts_to_sequences([start])[0]\n",
|
||||
" chars = inp\n",
|
||||
" for i in range(size):\n",
|
||||
" out = model(tf.expand_dims(tf.one_hot(inp,vocab_size),0))[0][-1]\n",
|
||||
" nc = tf.argmax(out)\n",
|
||||
" if nc==eos_token:\n",
|
||||
" break\n",
|
||||
" chars.append(nc.numpy())\n",
|
||||
" inp = inp+[nc]\n",
|
||||
" return decode(chars)\n",
|
||||
" \n",
|
||||
"generate(model)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## প্রশিক্ষণের সময় আউটপুট নমুনা করা\n",
|
||||
"\n",
|
||||
"যেহেতু আমাদের কাছে *সঠিকতা* (accuracy) এর মতো কোনো কার্যকর মেট্রিক নেই, তাই আমাদের মডেল উন্নতি করছে কিনা তা দেখার একমাত্র উপায় হল প্রশিক্ষণের সময় তৈরি করা স্ট্রিং-এর **নমুনা** নেওয়া। এটি করার জন্য, আমরা **কলব্যাকস** ব্যবহার করব, অর্থাৎ এমন ফাংশন যা আমরা `fit` ফাংশনে পাস করতে পারি এবং যা প্রশিক্ষণের সময় নির্দিষ্ট বিরতিতে ডাকা হবে।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch 1/3\n",
|
||||
"15000/15000 [==============================] - 226s 15ms/step - loss: 1.2703\n",
|
||||
"Today #39;s a lead in the company for the strike\n",
|
||||
"Epoch 2/3\n",
|
||||
"15000/15000 [==============================] - 227s 15ms/step - loss: 1.2057\n",
|
||||
"Today #39;s the Market Service on Security Start (AP)\n",
|
||||
"Epoch 3/3\n",
|
||||
"15000/15000 [==============================] - 226s 15ms/step - loss: 1.1752\n",
|
||||
"Today #39;s a line on the strike to start for the start\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7fa40c74e3d0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"sampling_callback = keras.callbacks.LambdaCallback(\n",
|
||||
" on_epoch_end = lambda batch, logs: print(generate(model))\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"model.fit(ds_train.batch(8).map(title_batch_fn),callbacks=[sampling_callback],epochs=3)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"এই উদাহরণটি ইতিমধ্যেই বেশ ভালো টেক্সট তৈরি করে, তবে এটি আরও কিছু উপায়ে উন্নত করা যেতে পারে:\n",
|
||||
"\n",
|
||||
"* **আরও টেক্সট**। আমরা আমাদের কাজের জন্য শুধুমাত্র শিরোনাম ব্যবহার করেছি, তবে আপনি পুরো টেক্সট নিয়ে পরীক্ষা করতে পারেন। মনে রাখবেন, RNN দীর্ঘ সিকোয়েন্স পরিচালনায় খুব ভালো নয়, তাই সেগুলোকে ছোট বাক্যে ভাগ করা বা একটি নির্ধারিত মান `num_chars` (যেমন, ২৫৬) অনুযায়ী নির্দিষ্ট সিকোয়েন্স দৈর্ঘ্যে প্রশিক্ষণ দেওয়া বুদ্ধিমানের কাজ। উপরের উদাহরণটি এমন আর্কিটেকচারে পরিবর্তন করার চেষ্টা করতে পারেন, যেখানে [অফিশিয়াল Keras টিউটোরিয়াল](https://keras.io/examples/generative/lstm_character_level_text_generation/) থেকে অনুপ্রেরণা নেওয়া যেতে পারে।\n",
|
||||
"\n",
|
||||
"* **মাল্টিলেয়ার LSTM**। ২ বা ৩ স্তরের LSTM সেল ব্যবহার করার চেষ্টা করা যৌক্তিক। আমরা আগের ইউনিটে উল্লেখ করেছি, LSTM-এর প্রতিটি স্তর টেক্সট থেকে নির্দিষ্ট প্যাটার্ন বের করে, এবং চরিত্র-স্তরের জেনারেটরের ক্ষেত্রে আমরা আশা করতে পারি যে নিম্ন স্তরের LSTM সিলেবল বের করতে দায়ী হবে, এবং উচ্চ স্তরগুলো শব্দ এবং শব্দের সংমিশ্রণ বের করবে। এটি সহজেই LSTM কনস্ট্রাক্টরে লেয়ারের সংখ্যা প্যারামিটার পাস করে বাস্তবায়ন করা যেতে পারে।\n",
|
||||
"\n",
|
||||
"* আপনি **GRU ইউনিট** নিয়ে পরীক্ষা করতে পারেন এবং দেখতে পারেন কোনটি ভালো কাজ করে, এবং **বিভিন্ন হিডেন লেয়ারের আকার** নিয়ে পরীক্ষা করতে পারেন। খুব বড় হিডেন লেয়ার ওভারফিটিং ঘটাতে পারে (যেমন, নেটওয়ার্ক সঠিক টেক্সট শিখে নেবে), এবং ছোট আকার ভালো ফলাফল নাও দিতে পারে।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## নরম টেক্সট জেনারেশন এবং টেম্পারেচার\n",
|
||||
"\n",
|
||||
"`generate` এর পূর্ববর্তী সংজ্ঞায়, আমরা সর্বদা সর্বোচ্চ সম্ভাবনাযুক্ত চরিত্রটিকে পরবর্তী চরিত্র হিসেবে গ্রহণ করতাম। এর ফলে প্রায়ই টেক্সট একই চরিত্রের ক্রম বারবার পুনরাবৃত্তি করত, যেমন এই উদাহরণে:\n",
|
||||
"```\n",
|
||||
"today of the second the company and a second the company ...\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"তবে, যদি আমরা পরবর্তী চরিত্রের জন্য সম্ভাবনা বণ্টন দেখি, তাহলে দেখা যায় যে কয়েকটি সর্বোচ্চ সম্ভাবনার মধ্যে পার্থক্য খুব বেশি নয়, যেমন একটি চরিত্রের সম্ভাবনা হতে পারে ০.২, আরেকটির ০.১৯, ইত্যাদি। উদাহরণস্বরূপ, যদি আমরা '*play*' ক্রমের পরবর্তী চরিত্র খুঁজছি, তাহলে পরবর্তী চরিত্রটি সমানভাবে স্পেস বা **e** (যেমন শব্দ *player* এ) হতে পারে।\n",
|
||||
"\n",
|
||||
"এটি আমাদের এই সিদ্ধান্তে পৌঁছায় যে সর্বোচ্চ সম্ভাবনাযুক্ত চরিত্র নির্বাচন করাই সবসময় \"ন্যায্য\" নয়, কারণ দ্বিতীয় সর্বোচ্চটি নির্বাচন করলেও অর্থবহ টেক্সট পাওয়া যেতে পারে। এটি আরও বুদ্ধিমানের কাজ হবে যদি আমরা নেটওয়ার্ক আউটপুট দ্বারা প্রদত্ত সম্ভাবনা বণ্টন থেকে চরিত্রগুলি **নমুনা** করি।\n",
|
||||
"\n",
|
||||
"এই নমুনা প্রক্রিয়া `np.multinomial` ফাংশন ব্যবহার করে করা যেতে পারে, যা তথাকথিত **মাল্টিনোমিয়াল বণ্টন** বাস্তবায়ন করে। একটি ফাংশন যা এই **নরম** টেক্সট জেনারেশন বাস্তবায়ন করে তা নিচে সংজ্ঞায়িত করা হয়েছে:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 33,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\n",
|
||||
"--- Temperature = 0.3\n",
|
||||
"Today #39;s strike #39; to start at the store return\n",
|
||||
"On Sunday PO to Be Data Profit Up (Reuters)\n",
|
||||
"Moscow, SP wins straight to the Microsoft #39;s control of the space start\n",
|
||||
"President olding of the blast start for the strike to pay <b>...</b>\n",
|
||||
"Little red riding hood ficed to the spam countered in European <b>...</b>\n",
|
||||
"\n",
|
||||
"--- Temperature = 0.8\n",
|
||||
"Today countie strikes ryder missile faces food market blut\n",
|
||||
"On Sunday collores lose-toppy of sale of Bullment in <b>...</b>\n",
|
||||
"Moscow, IBM Diffeiting in Afghan Software Hotels (Reuters)\n",
|
||||
"President Ol Luster for Profit Peaced Raised (AP)\n",
|
||||
"Little red riding hood dace on depart talks #39; bank up\n",
|
||||
"\n",
|
||||
"--- Temperature = 1.0\n",
|
||||
"Today wits House buiting debate fixes #39; supervice stake again\n",
|
||||
"On Sunday arling digital poaching In for level\n",
|
||||
"Moscow, DS Up 7, Top Proble Protest Caprey Mamarian Strike\n",
|
||||
"President teps help of roubler stepted lessabul-Dhalitics (AFP)\n",
|
||||
"Little red riding hood signs on cash in Carter-youb\n",
|
||||
"\n",
|
||||
"--- Temperature = 1.3\n",
|
||||
"Today wits flawer ro, pSIA figat's co DroftwavesIs Talo up\n",
|
||||
"On Sunday hround elitwing wint EU Powerburlinetien\n",
|
||||
"Moscow, Bazz #39;s sentries olymen winnelds' next for Olympite Huc?\n",
|
||||
"President lost securitys from power Elections in Smiltrials\n",
|
||||
"Little red riding hood vides profit, exponituity, profitmainalist-at said listers\n",
|
||||
"\n",
|
||||
"--- Temperature = 1.8\n",
|
||||
"Today #39;It: He deat: N.KA Asside\n",
|
||||
"On Sunday i arry Par aldeup patient Wo stele1\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"ename": "KeyError",
|
||||
"evalue": "0",
|
||||
"output_type": "error",
|
||||
"traceback": [
|
||||
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
|
||||
"\u001b[0;31mKeyError\u001b[0m Traceback (most recent call last)",
|
||||
"\u001b[0;32m<ipython-input-33-db32367a0feb>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m 18\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"\\n--- Temperature = {i}\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 19\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mj\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 20\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgenerate_soft\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0msize\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m300\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mstart\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mwords\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mj\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mtemperature\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
|
||||
"\u001b[0;32m<ipython-input-33-db32367a0feb>\u001b[0m in \u001b[0;36mgenerate_soft\u001b[0;34m(model, size, start, temperature)\u001b[0m\n\u001b[1;32m 11\u001b[0m \u001b[0mchars\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 12\u001b[0m \u001b[0minp\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0minp\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mnc\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 13\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mdecode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mchars\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 14\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 15\u001b[0m \u001b[0mwords\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m'Today '\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'On Sunday '\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'Moscow, '\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'President '\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'Little red riding hood '\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
|
||||
"\u001b[0;32m<ipython-input-10-3f5fa6130b1d>\u001b[0m in \u001b[0;36mdecode\u001b[0;34m(x)\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mdecode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0;34m''\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mreverse_map\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mt\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
|
||||
"\u001b[0;32m<ipython-input-10-3f5fa6130b1d>\u001b[0m in \u001b[0;36m<listcomp>\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mdecode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0;34m''\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mreverse_map\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mt\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
|
||||
"\u001b[0;31mKeyError\u001b[0m: 0"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def generate_soft(model,size=100,start='Today ',temperature=1.0):\n",
|
||||
" inp = tokenizer.texts_to_sequences([start])[0]\n",
|
||||
" chars = inp\n",
|
||||
" for i in range(size):\n",
|
||||
" out = model(tf.expand_dims(tf.one_hot(inp,vocab_size),0))[0][-1]\n",
|
||||
" probs = tf.exp(tf.math.log(out)/temperature).numpy().astype(np.float64)\n",
|
||||
" probs = probs/np.sum(probs)\n",
|
||||
" nc = np.argmax(np.random.multinomial(1,probs,1))\n",
|
||||
" if nc==eos_token:\n",
|
||||
" break\n",
|
||||
" chars.append(nc)\n",
|
||||
" inp = inp+[nc]\n",
|
||||
" return decode(chars)\n",
|
||||
"\n",
|
||||
"words = ['Today ','On Sunday ','Moscow, ','President ','Little red riding hood ']\n",
|
||||
" \n",
|
||||
"for i in [0.3,0.8,1.0,1.3,1.8]:\n",
|
||||
" print(f\"\\n--- Temperature = {i}\")\n",
|
||||
" for j in range(5):\n",
|
||||
" print(generate_soft(model,size=300,start=words[j],temperature=i))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"আমরা **temperature** নামে একটি অতিরিক্ত প্যারামিটার পরিচয় করিয়েছি, যা নির্দেশ করে যে সর্বোচ্চ সম্ভাবনার প্রতি আমরা কতটা কঠোরভাবে অনুসরণ করব। যদি temperature 1.0 হয়, আমরা ন্যায্য মাল্টিনোমিয়াল স্যাম্পলিং করি, এবং যখন temperature অসীমে পৌঁছায় - সমস্ত সম্ভাবনা সমান হয়ে যায়, এবং আমরা পরবর্তী অক্ষরটি এলোমেলোভাবে নির্বাচন করি। নিচের উদাহরণে আমরা দেখতে পারি যে temperature খুব বেশি বাড়ালে টেক্সট অর্থহীন হয়ে যায়, এবং এটি \"cycled\" কঠোরভাবে তৈরি টেক্সটের মতো হয়ে যায় যখন এটি 0-এর কাছাকাছি পৌঁছায়।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**অস্বীকৃতি**: \nএই নথিটি AI অনুবাদ পরিষেবা [Co-op Translator](https://github.com/Azure/co-op-translator) ব্যবহার করে অনুবাদ করা হয়েছে। আমরা যথাসম্ভব সঠিক অনুবাদের চেষ্টা করি, তবে অনুগ্রহ করে মনে রাখবেন যে স্বয়ংক্রিয় অনুবাদে ত্রুটি বা অসঙ্গতি থাকতে পারে। নথিটির মূল ভাষায় থাকা সংস্করণটিকেই প্রামাণিক উৎস হিসেবে বিবেচনা করা উচিত। গুরুত্বপূর্ণ তথ্যের জন্য, পেশাদার মানব অনুবাদ ব্যবহার করার পরামর্শ দেওয়া হয়। এই অনুবাদ ব্যবহারের ফলে সৃষ্ট কোনো ভুল বোঝাবুঝি বা ভুল ব্যাখ্যার জন্য আমরা দায়ী নই।\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.8.12 ('py38')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "9fbb7d5fda708537649f71f5f646fcde",
|
||||
"translation_date": "2025-08-28T11:59:08+00:00",
|
||||
"source_file": "lessons/5-NLP/17-GenerativeNetworks/GenerativeTF.ipynb",
|
||||
"language_code": "bn"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
|
|
@ -0,0 +1,353 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# অ্যাটেনশন মেকানিজম এবং ট্রান্সফর্মার\n",
|
||||
"\n",
|
||||
"রেকারেন্ট নেটওয়ার্কগুলোর একটি বড় সীমাবদ্ধতা হলো, একটি সিকোয়েন্সের সব শব্দই ফলাফলের উপর সমান প্রভাব ফেলে। এর ফলে সিকোয়েন্স-টু-সিকোয়েন্স কাজের জন্য স্ট্যান্ডার্ড LSTM এনকোডার-ডিকোডার মডেলগুলোতে অপ্টিমাল পারফরম্যান্স হয় না, যেমন নেমড এন্টিটি রিকগনিশন এবং মেশিন ট্রান্সলেশন। বাস্তবে, ইনপুট সিকোয়েন্সের নির্দিষ্ট কিছু শব্দ সিকোয়েন্সিয়াল আউটপুটে অন্য শব্দগুলোর চেয়ে বেশি প্রভাব ফেলে।\n",
|
||||
"\n",
|
||||
"ধরা যাক একটি সিকোয়েন্স-টু-সিকোয়েন্স মডেল, যেমন মেশিন ট্রান্সলেশন। এটি দুটি রেকারেন্ট নেটওয়ার্ক দ্বারা বাস্তবায়িত হয়, যেখানে একটি নেটওয়ার্ক (**এনকোডার**) ইনপুট সিকোয়েন্সকে হিডেন স্টেটে রূপান্তর করে এবং অন্যটি, **ডিকোডার**, এই হিডেন স্টেটকে অনুবাদিত ফলাফলে রূপান্তর করে। এই পদ্ধতির সমস্যা হলো, নেটওয়ার্কের চূড়ান্ত স্টেটের জন্য বাক্যের শুরুটা মনে রাখা কঠিন হয়ে যায়, যার ফলে দীর্ঘ বাক্যের ক্ষেত্রে মডেলের গুণগত মান খারাপ হয়।\n",
|
||||
"\n",
|
||||
"**অ্যাটেনশন মেকানিজম** একটি পদ্ধতি প্রদান করে, যা প্রতিটি ইনপুট ভেক্টরের প্রাসঙ্গিক প্রভাবকে RNN-এর প্রতিটি আউটপুট প্রেডিকশনে ওজন দেয়। এটি বাস্তবায়িত হয় ইনপুট RNN-এর মধ্যবর্তী স্টেট এবং আউটপুট RNN-এর মধ্যে শর্টকাট তৈরি করে। এই পদ্ধতিতে, যখন আউটপুট প্রতীক $y_t$ তৈরি করা হয়, তখন আমরা সব ইনপুট হিডেন স্টেট $h_i$-কে বিভিন্ন ওজন সহগ $\\alpha_{t,i}$ সহ বিবেচনা করব।\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"*[Bahdanau et al., 2015](https://arxiv.org/pdf/1409.0473.pdf) থেকে উদ্ধৃত, [এই ব্লগ পোস্ট](https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html) থেকে নেওয়া এনকোডার-ডিকোডার মডেল অ্যাডিটিভ অ্যাটেনশন মেকানিজম সহ]*\n",
|
||||
"\n",
|
||||
"অ্যাটেনশন ম্যাট্রিক্স $\\{\\alpha_{i,j}\\}$ নির্দেশ করে যে নির্দিষ্ট ইনপুট শব্দগুলো আউটপুট সিকোয়েন্সের একটি নির্দিষ্ট শব্দ তৈরিতে কতটা ভূমিকা রাখে। নিচে এমন একটি ম্যাট্রিক্সের উদাহরণ দেওয়া হলো:\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"*[Bahdanau et al., 2015](https://arxiv.org/pdf/1409.0473.pdf) থেকে নেওয়া (Fig.3)*\n",
|
||||
"\n",
|
||||
"অ্যাটেনশন মেকানিজম বর্তমান বা প্রায় বর্তমান প্রাকৃতিক ভাষা প্রক্রিয়াকরণের (NLP) স্টেট অফ দ্য আর্টের জন্য দায়ী। তবে অ্যাটেনশন যোগ করার ফলে মডেলের প্যারামিটারের সংখ্যা উল্লেখযোগ্যভাবে বৃদ্ধি পায়, যা RNN-এর স্কেলিং সমস্যার দিকে নিয়ে যায়। RNN-এর স্কেলিংয়ের একটি প্রধান সীমাবদ্ধতা হলো, মডেলের রেকারেন্ট প্রকৃতি প্রশিক্ষণকে ব্যাচ এবং প্যারালালাইজ করা কঠিন করে তোলে। RNN-এ একটি সিকোয়েন্সের প্রতিটি উপাদান ক্রমানুসারে প্রক্রিয়া করতে হয়, যার ফলে এটি সহজে প্যারালালাইজ করা যায় না।\n",
|
||||
"\n",
|
||||
"অ্যাটেনশন মেকানিজমের গ্রহণযোগ্যতা এবং এই সীমাবদ্ধতার সংমিশ্রণে বর্তমান স্টেট অফ দ্য আর্ট ট্রান্সফর্মার মডেলের সৃষ্টি হয়েছে, যা আমরা আজ BERT থেকে OpenGPT3 পর্যন্ত ব্যবহার করি।\n",
|
||||
"\n",
|
||||
"## ট্রান্সফর্মার মডেল\n",
|
||||
"\n",
|
||||
"প্রতিটি পূর্ববর্তী প্রেডিকশনের প্রসঙ্গকে পরবর্তী মূল্যায়ন ধাপে ফরোয়ার্ড করার পরিবর্তে, **ট্রান্সফর্মার মডেল** **পজিশনাল এনকোডিং** এবং অ্যাটেনশন ব্যবহার করে একটি প্রদত্ত টেক্সট উইন্ডোর মধ্যে একটি নির্দিষ্ট ইনপুটের প্রসঙ্গ ধারণ করে। নিচের ছবিটি দেখায় কিভাবে পজিশনাল এনকোডিং এবং অ্যাটেনশন একটি প্রদত্ত উইন্ডোর মধ্যে প্রসঙ্গ ধারণ করতে পারে।\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"যেহেতু প্রতিটি ইনপুট পজিশন স্বাধীনভাবে প্রতিটি আউটপুট পজিশনে ম্যাপ করা হয়, ট্রান্সফর্মার RNN-এর চেয়ে ভালোভাবে প্যারালালাইজ করতে পারে, যা অনেক বড় এবং আরও এক্সপ্রেসিভ ভাষার মডেল সক্ষম করে। প্রতিটি অ্যাটেনশন হেড বিভিন্ন শব্দের মধ্যে সম্পর্ক শিখতে ব্যবহার করা যেতে পারে, যা ডাউনস্ট্রিম NLP কাজগুলো উন্নত করে।\n",
|
||||
"\n",
|
||||
"**BERT** (Bidirectional Encoder Representations from Transformers) একটি খুব বড় মাল্টি-লেয়ার ট্রান্সফর্মার নেটওয়ার্ক, যেখানে *BERT-base*-এর জন্য ১২টি লেয়ার এবং *BERT-large*-এর জন্য ২৪টি লেয়ার রয়েছে। মডেলটি প্রথমে একটি বড় টেক্সট ডেটাসেট (উইকিপিডিয়া + বই) ব্যবহার করে আনসুপারভাইজড প্রশিক্ষণের মাধ্যমে প্রি-ট্রেইন করা হয় (একটি বাক্যে মাস্ক করা শব্দগুলো প্রেডিক্ট করা)। প্রি-ট্রেইনিংয়ের সময় মডেলটি উল্লেখযোগ্য ভাষাগত বোঝাপড়া অর্জন করে, যা পরে অন্যান্য ডেটাসেটের সাথে ফাইন টিউনিংয়ের মাধ্যমে ব্যবহার করা যায়। এই প্রক্রিয়াটিকে **ট্রান্সফার লার্নিং** বলা হয়।\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"ট্রান্সফর্মার আর্কিটেকচারের অনেক ভেরিয়েশন রয়েছে, যেমন BERT, DistilBERT, BigBird, OpenGPT3 এবং আরও অনেক, যেগুলো ফাইন টিউন করা যায়। [HuggingFace প্যাকেজ](https://github.com/huggingface/) PyTorch ব্যবহার করে এই আর্কিটেকচারগুলোর অনেকগুলোর প্রশিক্ষণের জন্য একটি রিপোজিটরি প্রদান করে।\n",
|
||||
"\n",
|
||||
"## টেক্সট ক্লাসিফিকেশনের জন্য BERT ব্যবহার\n",
|
||||
"\n",
|
||||
"চলুন দেখি কিভাবে আমরা প্রি-ট্রেইনড BERT মডেল ব্যবহার করে আমাদের ঐতিহ্যবাহী কাজ সমাধান করতে পারি: সিকোয়েন্স ক্লাসিফিকেশন। আমরা আমাদের মূল AG News ডেটাসেটটি ক্লাসিফাই করব।\n",
|
||||
"\n",
|
||||
"প্রথমে, চলুন HuggingFace লাইব্রেরি এবং আমাদের ডেটাসেট লোড করি:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Loading dataset...\n",
|
||||
"Building vocab...\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"import torchtext\n",
|
||||
"from torchnlp import *\n",
|
||||
"import transformers\n",
|
||||
"train_dataset, test_dataset, classes, vocab = load_dataset()\n",
|
||||
"vocab_len = len(vocab)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"আমরা যেহেতু প্রি-ট্রেইনড BERT মডেল ব্যবহার করব, তাই আমাদের নির্দিষ্ট একটি টোকেনাইজার ব্যবহার করতে হবে। প্রথমে, আমরা প্রি-ট্রেইনড BERT মডেলের সাথে সম্পর্কিত একটি টোকেনাইজার লোড করব।\n",
|
||||
"\n",
|
||||
"HuggingFace লাইব্রেরিতে প্রি-ট্রেইনড মডেলের একটি সংগ্রহশালা রয়েছে, যা আপনি শুধুমাত্র তাদের নাম `from_pretrained` ফাংশনের আর্গুমেন্ট হিসেবে উল্লেখ করেই ব্যবহার করতে পারেন। মডেলের জন্য প্রয়োজনীয় সমস্ত বাইনারি ফাইল স্বয়ংক্রিয়ভাবে ডাউনলোড হয়ে যাবে।\n",
|
||||
"\n",
|
||||
"তবে, কিছু ক্ষেত্রে আপনাকে নিজের মডেল লোড করতে হতে পারে। এই পরিস্থিতিতে, আপনি সেই ডিরেক্টরির অবস্থান উল্লেখ করতে পারেন যেখানে সমস্ত প্রাসঙ্গিক ফাইল রয়েছে, যেমন টোকেনাইজারের জন্য প্যারামিটার, মডেলের প্যারামিটার সহ `config.json` ফাইল, বাইনারি ওজন ইত্যাদি।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# To load the model from Internet repository using model name. \n",
|
||||
"# Use this if you are running from your own copy of the notebooks\n",
|
||||
"bert_model = 'bert-base-uncased' \n",
|
||||
"\n",
|
||||
"# To load the model from the directory on disk. Use this for Microsoft Learn module, because we have\n",
|
||||
"# prepared all required files for you.\n",
|
||||
"bert_model = './bert'\n",
|
||||
"\n",
|
||||
"tokenizer = transformers.BertTokenizer.from_pretrained(bert_model)\n",
|
||||
"\n",
|
||||
"MAX_SEQ_LEN = 128\n",
|
||||
"PAD_INDEX = tokenizer.convert_tokens_to_ids(tokenizer.pad_token)\n",
|
||||
"UNK_INDEX = tokenizer.convert_tokens_to_ids(tokenizer.unk_token)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"`tokenizer` অবজেক্টে `encode` ফাংশন রয়েছে যা সরাসরি টেক্সট এনকোড করতে ব্যবহার করা যেতে পারে:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[101, 1052, 22123, 2953, 2818, 2003, 1037, 2307, 7705, 2005, 17953, 2361, 102]"
|
||||
]
|
||||
},
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"tokenizer.encode('PyTorch is a great framework for NLP')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"তারপর, চলুন ট্রেনিংয়ের সময় ডেটা অ্যাক্সেস করার জন্য আমরা যে ইটারেটরগুলি ব্যবহার করব তা তৈরি করি। কারণ BERT তার নিজস্ব এনকোডিং ফাংশন ব্যবহার করে, আমাদের একটি প্যাডিং ফাংশন সংজ্ঞায়িত করতে হবে যা পূর্বে সংজ্ঞায়িত `padify` এর অনুরূপ:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def pad_bert(b):\n",
|
||||
" # b is the list of tuples of length batch_size\n",
|
||||
" # - first element of a tuple = label, \n",
|
||||
" # - second = feature (text sequence)\n",
|
||||
" # build vectorized sequence\n",
|
||||
" v = [tokenizer.encode(x[1]) for x in b]\n",
|
||||
" # compute max length of a sequence in this minibatch\n",
|
||||
" l = max(map(len,v))\n",
|
||||
" return ( # tuple of two tensors - labels and features\n",
|
||||
" torch.LongTensor([t[0] for t in b]),\n",
|
||||
" torch.stack([torch.nn.functional.pad(torch.tensor(t),(0,l-len(t)),mode='constant',value=0) for t in v])\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=8, collate_fn=pad_bert, shuffle=True)\n",
|
||||
"test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=8, collate_fn=pad_bert)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"আমাদের ক্ষেত্রে, আমরা `bert-base-uncased` নামে একটি প্রি-ট্রেইনড BERT মডেল ব্যবহার করব। চলুন `BertForSequenceClassfication` প্যাকেজ ব্যবহার করে মডেলটি লোড করি। এটি নিশ্চিত করে যে আমাদের মডেলের ক্লাসিফিকেশনের জন্য প্রয়োজনীয় আর্কিটেকচার ইতিমধ্যেই রয়েছে, যার মধ্যে ফাইনাল ক্লাসিফায়ারও অন্তর্ভুক্ত। আপনি একটি সতর্কবার্তা দেখতে পাবেন যেখানে বলা হবে যে ফাইনাল ক্লাসিফায়ারের ওজনগুলি ইনিশিয়ালাইজ করা হয়নি, এবং মডেলটি প্রি-ট্রেইনিং প্রয়োজন - এটি সম্পূর্ণ ঠিক আছে, কারণ আমরা ঠিক সেটাই করতে যাচ্ছি!\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Some weights of the model checkpoint at ./bert were not used when initializing BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']\n",
|
||||
"- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
|
||||
"- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
|
||||
"Some weights of BertForSequenceClassification were not initialized from the model checkpoint at ./bert and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
|
||||
"You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model = transformers.BertForSequenceClassification.from_pretrained(bert_model,num_labels=4).to(device)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"এখন আমরা প্রশিক্ষণ শুরু করতে প্রস্তুত! যেহেতু BERT ইতিমধ্যেই প্রি-ট্রেইন করা আছে, আমরা খুব ছোট লার্নিং রেট দিয়ে শুরু করতে চাই যাতে প্রাথমিক ওজন নষ্ট না হয়।\n",
|
||||
"\n",
|
||||
"সমস্ত কঠিন কাজ `BertForSequenceClassification` মডেল দ্বারা সম্পন্ন হয়। যখন আমরা প্রশিক্ষণের ডেটায় মডেলটি কল করি, এটি ইনপুট মিনিব্যাচের জন্য ক্ষতি এবং নেটওয়ার্ক আউটপুট উভয়ই প্রদান করে। আমরা প্যারামিটার অপ্টিমাইজেশনের জন্য ক্ষতি ব্যবহার করি (`loss.backward()` ব্যাকওয়ার্ড পাস সম্পন্ন করে), এবং প্রশিক্ষণের সঠিকতা গণনার জন্য `out` ব্যবহার করি, যেখানে প্রাপ্ত লেবেল `labs` (যা `argmax` ব্যবহার করে গণনা করা হয়) প্রত্যাশিত `labels` এর সাথে তুলনা করা হয়।\n",
|
||||
"\n",
|
||||
"প্রক্রিয়াটি নিয়ন্ত্রণ করার জন্য, আমরা কয়েকটি পুনরাবৃত্তির উপর ক্ষতি এবং সঠিকতা জমা করি এবং প্রতি `report_freq` প্রশিক্ষণ চক্রে সেগুলি প্রিন্ট করি।\n",
|
||||
"\n",
|
||||
"এই প্রশিক্ষণ সম্ভবত বেশ দীর্ঘ সময় নিতে পারে, তাই আমরা পুনরাবৃত্তির সংখ্যা সীমিত করি।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Loss = 1.1254194641113282, Accuracy = 0.585\n",
|
||||
"Loss = 0.6194715118408203, Accuracy = 0.83\n",
|
||||
"Loss = 0.46665248870849607, Accuracy = 0.8475\n",
|
||||
"Loss = 0.4309701919555664, Accuracy = 0.8575\n",
|
||||
"Loss = 0.35427074432373046, Accuracy = 0.8825\n",
|
||||
"Loss = 0.3306886291503906, Accuracy = 0.8975\n",
|
||||
"Loss = 0.30340143203735354, Accuracy = 0.8975\n",
|
||||
"Loss = 0.26139299392700194, Accuracy = 0.915\n",
|
||||
"Loss = 0.26708646774291994, Accuracy = 0.9225\n",
|
||||
"Loss = 0.3667240524291992, Accuracy = 0.8675\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"optimizer = torch.optim.Adam(model.parameters(), lr=2e-5)\n",
|
||||
"\n",
|
||||
"report_freq = 50\n",
|
||||
"iterations = 500 # make this larger to train for longer time!\n",
|
||||
"\n",
|
||||
"model.train()\n",
|
||||
"\n",
|
||||
"i,c = 0,0\n",
|
||||
"acc_loss = 0\n",
|
||||
"acc_acc = 0\n",
|
||||
"\n",
|
||||
"for labels,texts in train_loader:\n",
|
||||
" labels = labels.to(device)-1 # get labels in the range 0-3 \n",
|
||||
" texts = texts.to(device)\n",
|
||||
" loss, out = model(texts, labels=labels)[:2]\n",
|
||||
" labs = out.argmax(dim=1)\n",
|
||||
" acc = torch.mean((labs==labels).type(torch.float32))\n",
|
||||
" optimizer.zero_grad()\n",
|
||||
" loss.backward()\n",
|
||||
" optimizer.step()\n",
|
||||
" acc_loss += loss\n",
|
||||
" acc_acc += acc\n",
|
||||
" i+=1\n",
|
||||
" c+=1\n",
|
||||
" if i%report_freq==0:\n",
|
||||
" print(f\"Loss = {acc_loss.item()/c}, Accuracy = {acc_acc.item()/c}\")\n",
|
||||
" c = 0\n",
|
||||
" acc_loss = 0\n",
|
||||
" acc_acc = 0\n",
|
||||
" iterations-=1\n",
|
||||
" if not iterations:\n",
|
||||
" break"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"আপনি দেখতে পাবেন (বিশেষত যদি আপনি পুনরাবৃত্তির সংখ্যা বাড়ান এবং যথেষ্ট সময় অপেক্ষা করেন) যে BERT শ্রেণীবিভাগ আমাদের বেশ ভালো নির্ভুলতা প্রদান করে! এর কারণ হলো BERT ইতিমধ্যেই ভাষার গঠন বেশ ভালোভাবে বোঝে, এবং আমাদের কেবলমাত্র চূড়ান্ত শ্রেণীবিভাজকটি সূক্ষ্মভাবে সামঞ্জস্য করতে হয়। তবে, যেহেতু BERT একটি বড় মডেল, পুরো প্রশিক্ষণ প্রক্রিয়াটি অনেক সময় নেয় এবং উল্লেখযোগ্য কম্পিউটেশনাল ক্ষমতার প্রয়োজন হয়! (GPU, এবং সম্ভব হলে একাধিক GPU)।\n",
|
||||
"\n",
|
||||
"> **Note:** আমাদের উদাহরণে, আমরা সবচেয়ে ছোট প্রি-ট্রেইনড BERT মডেলগুলোর একটি ব্যবহার করেছি। বড় মডেলগুলো সম্ভবত আরও ভালো ফলাফল প্রদান করতে পারে।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## মডেলের কার্যকারিতা মূল্যায়ন\n",
|
||||
"\n",
|
||||
"এখন আমরা টেস্ট ডেটাসেটে আমাদের মডেলের কার্যকারিতা মূল্যায়ন করতে পারি। মূল্যায়ন প্রক্রিয়া প্রশিক্ষণ প্রক্রিয়ার সাথে বেশ মিল, তবে আমরা যেন ভুলে না যাই যে `model.eval()` কল করে মডেলকে মূল্যায়ন মোডে স্যুইচ করতে হবে।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Final accuracy: 0.9047029702970297\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.eval()\n",
|
||||
"iterations = 100\n",
|
||||
"acc = 0\n",
|
||||
"i = 0\n",
|
||||
"for labels,texts in test_loader:\n",
|
||||
" labels = labels.to(device)-1 \n",
|
||||
" texts = texts.to(device)\n",
|
||||
" _, out = model(texts, labels=labels)[:2]\n",
|
||||
" labs = out.argmax(dim=1)\n",
|
||||
" acc += torch.mean((labs==labels).type(torch.float32))\n",
|
||||
" i+=1\n",
|
||||
" if i>iterations: break\n",
|
||||
" \n",
|
||||
"print(f\"Final accuracy: {acc.item()/i}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## মূল বিষয়বস্তু\n",
|
||||
"\n",
|
||||
"এই ইউনিটে, আমরা দেখেছি কীভাবে **transformers** লাইব্রেরি থেকে প্রি-ট্রেইনড ভাষার মডেল সহজেই নিয়ে আমাদের টেক্সট ক্লাসিফিকেশন টাস্কে ব্যবহার করা যায়। একইভাবে, BERT মডেলগুলো ব্যবহার করা যেতে পারে এন্টিটি এক্সট্রাকশন, প্রশ্নোত্তর এবং অন্যান্য NLP কাজের জন্য।\n",
|
||||
"\n",
|
||||
"ট্রান্সফরমার মডেলগুলো NLP-তে বর্তমান সময়ের সর্বাধুনিক প্রযুক্তি উপস্থাপন করে, এবং বেশিরভাগ ক্ষেত্রেই এটি হওয়া উচিত প্রথম সমাধান যার মাধ্যমে আপনি কাস্টম NLP সমাধান বাস্তবায়নের সময় পরীক্ষা-নিরীক্ষা শুরু করবেন। তবে, এই মডিউলে আলোচনা করা পুনরাবৃত্ত নিউরাল নেটওয়ার্কের মৌলিক নীতিগুলো বোঝা অত্যন্ত গুরুত্বপূর্ণ যদি আপনি উন্নত নিউরাল মডেল তৈরি করতে চান।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**অস্বীকৃতি**: \nএই নথিটি AI অনুবাদ পরিষেবা [Co-op Translator](https://github.com/Azure/co-op-translator) ব্যবহার করে অনুবাদ করা হয়েছে। আমরা যথাসাধ্য সঠিকতার জন্য চেষ্টা করি, তবে অনুগ্রহ করে মনে রাখবেন যে স্বয়ংক্রিয় অনুবাদে ত্রুটি বা অসঙ্গতি থাকতে পারে। মূল ভাষায় থাকা নথিটিকে প্রামাণিক উৎস হিসেবে বিবেচনা করা উচিত। গুরুত্বপূর্ণ তথ্যের জন্য, পেশাদার মানব অনুবাদ সুপারিশ করা হয়। এই অনুবাদ ব্যবহারের ফলে কোনো ভুল বোঝাবুঝি বা ভুল ব্যাখ্যা হলে আমরা দায়ী থাকব না।\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "py37_pytorch",
|
||||
"language": "python",
|
||||
"name": "conda-env-py37_pytorch-py"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.7.7"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "753865967678a92dbce7d7efbd36d980",
|
||||
"translation_date": "2025-08-28T12:08:17+00:00",
|
||||
"source_file": "lessons/5-NLP/18-Transformers/TransformersPyTorch.ipynb",
|
||||
"language_code": "bn"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
|
|
@ -0,0 +1,819 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# অ্যাটেনশন মেকানিজম এবং ট্রান্সফর্মার\n",
|
||||
"\n",
|
||||
"রেকারেন্ট নেটওয়ার্কগুলোর একটি বড় সীমাবদ্ধতা হলো, একটি সিকোয়েন্সের সব শব্দ ফলাফলের উপর সমান প্রভাব ফেলে। এর ফলে সিকোয়েন্স-টু-সিকোয়েন্স কাজের জন্য স্ট্যান্ডার্ড LSTM এনকোডার-ডিকোডার মডেলগুলোতে অপ্টিমাল পারফরম্যান্স হয় না, যেমন নেমড এন্টিটি রিকগনিশন এবং মেশিন ট্রান্সলেশন। বাস্তবে, ইনপুট সিকোয়েন্সের নির্দিষ্ট কিছু শব্দ সিকোয়েন্সিয়াল আউটপুটে অন্য শব্দগুলোর চেয়ে বেশি প্রভাব ফেলে।\n",
|
||||
"\n",
|
||||
"ধরা যাক একটি সিকোয়েন্স-টু-সিকোয়েন্স মডেল, যেমন মেশিন ট্রান্সলেশন। এটি দুটি রেকারেন্ট নেটওয়ার্ক দ্বারা বাস্তবায়িত হয়, যেখানে একটি নেটওয়ার্ক (**encoder**) ইনপুট সিকোয়েন্সকে হিডেন স্টেটে সংকুচিত করে এবং অন্যটি, **decoder**, এই হিডেন স্টেটকে অনুবাদিত ফলাফলে রূপান্তরিত করে। এই পদ্ধতির সমস্যা হলো, নেটওয়ার্কের চূড়ান্ত স্টেটের জন্য বাক্যের শুরুটা মনে রাখা কঠিন হয়ে যায়, যার ফলে দীর্ঘ বাক্যের ক্ষেত্রে মডেলের গুণগত মান খারাপ হয়।\n",
|
||||
"\n",
|
||||
"**অ্যাটেনশন মেকানিজম** RNN-এর প্রতিটি আউটপুট প্রেডিকশনে প্রতিটি ইনপুট ভেক্টরের প্রাসঙ্গিক প্রভাবকে ওজন দেওয়ার একটি উপায় প্রদান করে। এটি বাস্তবায়িত হয় ইনপুট RNN-এর মধ্যবর্তী স্টেট এবং আউটপুট RNN-এর মধ্যে শর্টকাট তৈরি করে। এই পদ্ধতিতে, যখন আউটপুট প্রতীক $y_t$ তৈরি করা হয়, তখন আমরা বিভিন্ন ওজন সহগ $\\alpha_{t,i}$ সহ সমস্ত ইনপুট হিডেন স্টেট $h_i$ বিবেচনায় নেব। \n",
|
||||
"\n",
|
||||
"\n",
|
||||
"*[Bahdanau et al., 2015](https://arxiv.org/pdf/1409.0473.pdf) থেকে উদ্ধৃত, [এই ব্লগ পোস্ট](https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html) থেকে নেওয়া এনকোডার-ডিকোডার মডেল অ্যাডিটিভ অ্যাটেনশন মেকানিজম সহ*\n",
|
||||
"\n",
|
||||
"অ্যাটেনশন ম্যাট্রিক্স $\\{\\alpha_{i,j}\\}$ একটি নির্দিষ্ট আউটপুট সিকোয়েন্সের একটি শব্দ তৈরিতে কোন ইনপুট শব্দগুলো কতটা ভূমিকা রাখছে তা উপস্থাপন করে। নিচে এমন একটি ম্যাট্রিক্সের উদাহরণ দেওয়া হলো:\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"*[Bahdanau et al., 2015](https://arxiv.org/pdf/1409.0473.pdf) থেকে নেওয়া (Fig.3)*\n",
|
||||
"\n",
|
||||
"অ্যাটেনশন মেকানিজম বর্তমান বা প্রায় বর্তমান প্রাকৃতিক ভাষা প্রক্রিয়াকরণের (NLP) স্টেট অফ দ্য আর্টের জন্য দায়ী। তবে অ্যাটেনশন যোগ করার ফলে মডেলের প্যারামিটারের সংখ্যা উল্লেখযোগ্যভাবে বৃদ্ধি পায়, যা RNN-এর স্কেলিং সমস্যার দিকে নিয়ে যায়। RNN-এর স্কেলিংয়ের একটি মূল সীমাবদ্ধতা হলো, মডেলের রেকারেন্ট প্রকৃতি প্রশিক্ষণকে ব্যাচ এবং প্যারালালাইজ করা কঠিন করে তোলে। RNN-এ একটি সিকোয়েন্সের প্রতিটি উপাদানকে ক্রমানুসারে প্রক্রিয়া করতে হয়, যার ফলে এটি সহজে প্যারালালাইজ করা যায় না।\n",
|
||||
"\n",
|
||||
"অ্যাটেনশন মেকানিজমের গ্রহণযোগ্যতা এবং এই সীমাবদ্ধতার সংমিশ্রণে, আজকের স্টেট অফ দ্য আর্ট ট্রান্সফর্মার মডেলগুলোর সৃষ্টি হয়েছে, যা আমরা BERT থেকে OpenGPT3 পর্যন্ত ব্যবহার করি।\n",
|
||||
"\n",
|
||||
"## ট্রান্সফর্মার মডেল\n",
|
||||
"\n",
|
||||
"প্রতিটি পূর্ববর্তী প্রেডিকশনের প্রসঙ্গকে পরবর্তী মূল্যায়ন ধাপে ফরোয়ার্ড করার পরিবর্তে, **ট্রান্সফর্মার মডেলগুলো** **পজিশনাল এনকোডিং** এবং **অ্যাটেনশন** ব্যবহার করে একটি প্রদত্ত টেক্সট উইন্ডোর মধ্যে একটি ইনপুটের প্রসঙ্গ ধারণ করে। নিচের চিত্রটি দেখায় কীভাবে পজিশনাল এনকোডিং এবং অ্যাটেনশন একটি প্রদত্ত উইন্ডোর মধ্যে প্রসঙ্গ ধারণ করতে পারে।\n",
|
||||
"\n",
|
||||
" \n",
|
||||
"\n",
|
||||
"যেহেতু প্রতিটি ইনপুট পজিশন স্বাধীনভাবে প্রতিটি আউটপুট পজিশনে ম্যাপ করা হয়, ট্রান্সফর্মারগুলো RNN-এর চেয়ে ভালোভাবে প্যারালালাইজ করতে পারে, যা অনেক বড় এবং আরও এক্সপ্রেসিভ ভাষার মডেল সক্ষম করে। প্রতিটি অ্যাটেনশন হেড বিভিন্ন শব্দের মধ্যে সম্পর্ক শিখতে ব্যবহার করা যেতে পারে, যা ডাউনস্ট্রিম NLP কাজগুলো উন্নত করে।\n",
|
||||
"\n",
|
||||
"## সহজ ট্রান্সফর্মার মডেল তৈরি\n",
|
||||
"\n",
|
||||
"Keras-এ বিল্ট-ইন ট্রান্সফর্মার লেয়ার নেই, তবে আমরা নিজেরাই এটি তৈরি করতে পারি। আগের মতো, আমরা AG News ডেটাসেটের টেক্সট ক্লাসিফিকেশনের উপর ফোকাস করব, তবে এটি উল্লেখযোগ্য যে ট্রান্সফর্মার মডেলগুলো আরও কঠিন NLP কাজগুলোতে সেরা ফলাফল দেখায়।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import tensorflow as tf\n",
|
||||
"from tensorflow import keras\n",
|
||||
"import tensorflow_datasets as tfds\n",
|
||||
"import numpy as np\n",
|
||||
"\n",
|
||||
"ds_train, ds_test = tfds.load('ag_news_subset').values()\n",
|
||||
"\n",
|
||||
"def extract_text(x):\n",
|
||||
" return x['title']+' '+x['description']\n",
|
||||
"\n",
|
||||
"def tupelize(x):\n",
|
||||
" return (extract_text(x),x['label'])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"কেরাস-এ নতুন লেয়ারগুলি `Layer` ক্লাসকে সাবক্লাস করা উচিত এবং `call` মেথডটি ইমপ্লিমেন্ট করা উচিত। চলুন **Positional Embedding** লেয়ার দিয়ে শুরু করি। আমরা [কেরাসের অফিসিয়াল ডকুমেন্টেশন থেকে কিছু কোড](https://keras.io/examples/nlp/text_classification_with_transformer/) ব্যবহার করব। আমরা ধরে নেব যে আমরা সমস্ত ইনপুট সিকোয়েন্সকে `maxlen` দৈর্ঘ্যে প্যাড করি।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class TokenAndPositionEmbedding(keras.layers.Layer):\n",
|
||||
" def __init__(self, maxlen, vocab_size, embed_dim):\n",
|
||||
" super(TokenAndPositionEmbedding, self).__init__()\n",
|
||||
" self.token_emb = keras.layers.Embedding(input_dim=vocab_size, output_dim=embed_dim)\n",
|
||||
" self.pos_emb = keras.layers.Embedding(input_dim=maxlen, output_dim=embed_dim)\n",
|
||||
" self.maxlen = maxlen\n",
|
||||
"\n",
|
||||
" def call(self, x):\n",
|
||||
" maxlen = self.maxlen\n",
|
||||
" positions = tf.range(start=0, limit=maxlen, delta=1)\n",
|
||||
" positions = self.pos_emb(positions)\n",
|
||||
" x = self.token_emb(x)\n",
|
||||
" return x+positions"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"এই স্তরটি দুটি `Embedding` স্তর নিয়ে গঠিত: একটি টোকেন এমবেড করার জন্য (যেমনটি আমরা আগে আলোচনা করেছি) এবং আরেকটি টোকেন পজিশনের জন্য। টোকেন পজিশনগুলো `tf.range` ব্যবহার করে 0 থেকে `maxlen` পর্যন্ত প্রাকৃতিক সংখ্যার একটি ক্রম হিসেবে তৈরি করা হয় এবং তারপর এমবেডিং স্তরের মাধ্যমে পাঠানো হয়। দুটি প্রাপ্ত এমবেডিং ভেক্টর যোগ করা হয়, যার ফলে ইনপুটের একটি পজিশনালি এমবেড করা উপস্থাপনা তৈরি হয় যার আকার `maxlen`$\\times$`embed_dim`।\n",
|
||||
"\n",
|
||||
"এখন, চলুন ট্রান্সফরমার ব্লকটি বাস্তবায়ন করি। এটি পূর্বে সংজ্ঞায়িত এমবেডিং স্তরের আউটপুট গ্রহণ করবে:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class TransformerBlock(keras.layers.Layer):\n",
|
||||
" def __init__(self, embed_dim, num_heads, ff_dim, rate=0.1):\n",
|
||||
" super(TransformerBlock, self).__init__()\n",
|
||||
" self.att = keras.layers.MultiHeadAttention(num_heads=num_heads, key_dim=embed_dim, name='attn')\n",
|
||||
" self.ffn = keras.Sequential(\n",
|
||||
" [keras.layers.Dense(ff_dim, activation=\"relu\"), keras.layers.Dense(embed_dim),]\n",
|
||||
" )\n",
|
||||
" self.layernorm1 = keras.layers.LayerNormalization(epsilon=1e-6)\n",
|
||||
" self.layernorm2 = keras.layers.LayerNormalization(epsilon=1e-6)\n",
|
||||
" self.dropout1 = keras.layers.Dropout(rate)\n",
|
||||
" self.dropout2 = keras.layers.Dropout(rate)\n",
|
||||
"\n",
|
||||
" def call(self, inputs, training):\n",
|
||||
" attn_output = self.att(inputs, inputs)\n",
|
||||
" attn_output = self.dropout1(attn_output, training=training)\n",
|
||||
" out1 = self.layernorm1(inputs + attn_output)\n",
|
||||
" ffn_output = self.ffn(out1)\n",
|
||||
" ffn_output = self.dropout2(ffn_output, training=training)\n",
|
||||
" return self.layernorm2(out1 + ffn_output)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"এখন আমরা সম্পূর্ণ ট্রান্সফর্মার মডেল সংজ্ঞায়িত করতে প্রস্তুত:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"sequential_1\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
"Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
"text_vectorization (TextVect (None, 256) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"token_and_position_embedding (None, 256, 32) 648192 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"transformer_block (Transform (None, 256, 32) 10656 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"global_average_pooling1d (Gl (None, 32) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dropout_2 (Dropout) (None, 32) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_2 (Dense) (None, 20) 660 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dropout_3 (Dropout) (None, 20) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_3 (Dense) (None, 4) 84 \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 659,592\n",
|
||||
"Trainable params: 659,592\n",
|
||||
"Non-trainable params: 0\n",
|
||||
"_________________________________________________________________\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"embed_dim = 32 # Embedding size for each token\n",
|
||||
"num_heads = 2 # Number of attention heads\n",
|
||||
"ff_dim = 32 # Hidden layer size in feed forward network inside transformer\n",
|
||||
"maxlen = 256\n",
|
||||
"vocab_size = 20000\n",
|
||||
"\n",
|
||||
"model = keras.models.Sequential([\n",
|
||||
" keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size,output_sequence_length=maxlen, input_shape=(1,)),\n",
|
||||
" TokenAndPositionEmbedding(maxlen, vocab_size, embed_dim),\n",
|
||||
" TransformerBlock(embed_dim, num_heads, ff_dim),\n",
|
||||
" keras.layers.GlobalAveragePooling1D(),\n",
|
||||
" keras.layers.Dropout(0.1),\n",
|
||||
" keras.layers.Dense(20, activation=\"relu\"),\n",
|
||||
" keras.layers.Dropout(0.1),\n",
|
||||
" keras.layers.Dense(4, activation=\"softmax\")\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"model.summary()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Training tokenizer\n",
|
||||
"938/938 [==============================] - 45s 39ms/step - loss: 0.4978 - acc: 0.8068 - val_loss: 0.2808 - val_acc: 0.9124\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7f9c2427a0d0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print('Training tokenizer')\n",
|
||||
"model.layers[0].adapt(ds_train.map(extract_text))\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(128),validation_data=ds_test.map(tupelize).batch(128))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## BERT ট্রান্সফর্মার মডেল\n",
|
||||
"\n",
|
||||
"**BERT** (Bidirectional Encoder Representations from Transformers) একটি অত্যন্ত বড় মাল্টি-লেয়ার ট্রান্সফর্মার নেটওয়ার্ক, যেখানে *BERT-base* এর জন্য ১২টি স্তর এবং *BERT-large* এর জন্য ২৪টি স্তর রয়েছে। এই মডেলটি প্রথমে বিশাল পরিমাণ টেক্সট ডেটা (উইকিপিডিয়া + বই) ব্যবহার করে অ-পর্যবেক্ষণমূলক প্রশিক্ষণের মাধ্যমে (একটি বাক্যে মাস্ক করা শব্দ অনুমান করা) প্রি-ট্রেইন করা হয়। প্রি-ট্রেইনিংয়ের সময় মডেলটি উল্লেখযোগ্য ভাষাগত বোঝাপড়া অর্জন করে, যা পরে অন্যান্য ডেটাসেটের সাথে ফাইন টিউনিংয়ের মাধ্যমে ব্যবহার করা যায়। এই প্রক্রিয়াকে **ট্রান্সফার লার্নিং** বলা হয়।\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"ট্রান্সফর্মার আর্কিটেকচারের অনেক বৈচিত্র্য রয়েছে, যেমন BERT, DistilBERT, BigBird, OpenGPT3 এবং আরও অনেক কিছু, যেগুলো ফাইন টিউন করা যেতে পারে।\n",
|
||||
"\n",
|
||||
"চলুন দেখি কীভাবে আমরা প্রি-ট্রেইন করা BERT মডেল ব্যবহার করে আমাদের প্রচলিত সিকোয়েন্স ক্লাসিফিকেশন সমস্যার সমাধান করতে পারি। আমরা [অফিশিয়াল ডকুমেন্টেশন](https://www.tensorflow.org/text/tutorials/classify_text_with_bert) থেকে ধারণা এবং কিছু কোড ধার করব।\n",
|
||||
"\n",
|
||||
"প্রি-ট্রেইন করা মডেল লোড করতে আমরা **Tensorflow hub** ব্যবহার করব। প্রথমে, চলুন BERT-স্পেসিফিক ভেক্টরাইজার লোড করি:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"ename": "ModuleNotFoundError",
|
||||
"evalue": "No module named 'tensorflow_text'",
|
||||
"output_type": "error",
|
||||
"traceback": [
|
||||
"\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
|
||||
"\u001b[1;31mModuleNotFoundError\u001b[0m Traceback (most recent call last)",
|
||||
"\u001b[1;32m~\\AppData\\Local\\Temp/ipykernel_41180/4216669875.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[1;32mimport\u001b[0m \u001b[0mtensorflow_text\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 2\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mtensorflow_hub\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mhub\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 3\u001b[0m \u001b[0mvectorizer\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mhub\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mKerasLayer\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
|
||||
"\u001b[1;31mModuleNotFoundError\u001b[0m: No module named 'tensorflow_text'"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import tensorflow_text \n",
|
||||
"import tensorflow_hub as hub\n",
|
||||
"vectorizer = hub.KerasLayer('https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"{'input_type_ids': <tf.Tensor: shape=(1, 128), dtype=int32, numpy=\n",
|
||||
" array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],\n",
|
||||
" dtype=int32)>,\n",
|
||||
" 'input_word_ids': <tf.Tensor: shape=(1, 128), dtype=int32, numpy=\n",
|
||||
" array([[ 101, 1045, 2293, 19081, 102, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0]], dtype=int32)>,\n",
|
||||
" 'input_mask': <tf.Tensor: shape=(1, 128), dtype=int32, numpy=\n",
|
||||
" array([[1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],\n",
|
||||
" dtype=int32)>}"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vectorizer(['I love transformers'])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"এটি গুরুত্বপূর্ণ যে আপনি একই ভেক্টরাইজার ব্যবহার করেন যেটি মূল নেটওয়ার্ক প্রশিক্ষণের জন্য ব্যবহৃত হয়েছিল। এছাড়াও, BERT ভেক্টরাইজার তিনটি উপাদান প্রদান করে:\n",
|
||||
"* `input_word_ids`, যা ইনপুট বাক্যের জন্য টোকেন নম্বরগুলোর একটি ক্রম\n",
|
||||
"* `input_mask`, যা দেখায় ক্রমের কোন অংশটি প্রকৃত ইনপুট এবং কোন অংশটি প্যাডিং। এটি `Masking` লেয়ার দ্বারা উৎপন্ন মাস্কের মতো\n",
|
||||
"* `input_type_ids` ভাষা মডেলিং কাজের জন্য ব্যবহৃত হয় এবং একটি ক্রমে দুটি ইনপুট বাক্য নির্দিষ্ট করার অনুমতি দেয়।\n",
|
||||
"\n",
|
||||
"এরপর, আমরা BERT ফিচার এক্সট্রাক্টর ইনস্ট্যান্সিয়েট করতে পারি:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"bert = hub.KerasLayer('https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-4_H-128_A-2/1')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"pooled_output -> (1, 128)\n",
|
||||
"encoder_outputs -> 4\n",
|
||||
"sequence_output -> (1, 128, 128)\n",
|
||||
"default -> (1, 128)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"z = bert(vectorizer(['I love transformers']))\n",
|
||||
"for i,x in z.items():\n",
|
||||
" print(f\"{i} -> { len(x) if isinstance(x, list) else x.shape }\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"তাহলে, BERT স্তর বেশ কিছু দরকারী ফলাফল প্রদান করে:\n",
|
||||
"* `pooled_output` হলো সিকোয়েন্সের সমস্ত টোকেনের গড় হিসাব। আপনি এটিকে পুরো নেটওয়ার্কের একটি বুদ্ধিমান অর্থবোধক এমবেডিং হিসেবে দেখতে পারেন। এটি আমাদের আগের মডেলের `GlobalAveragePooling1D` স্তরের আউটপুটের সমতুল্য।\n",
|
||||
"* `sequence_output` হলো শেষ ট্রান্সফর্মার স্তরের আউটপুট (উপরের মডেলে `TransformerBlock` এর আউটপুটের সাথে মিলে যায়)।\n",
|
||||
"* `encoder_outputs` হলো সমস্ত ট্রান্সফর্মার স্তরের আউটপুট। যেহেতু আমরা ৪-স্তরের BERT মডেল লোড করেছি (যা আপনি সম্ভবত নাম থেকে অনুমান করতে পারেন, যেখানে `4_H` রয়েছে), এটি ৪টি টেনসর প্রদান করে। শেষটি `sequence_output` এর সমান।\n",
|
||||
"\n",
|
||||
"এখন আমরা এন্ড-টু-এন্ড ক্লাসিফিকেশন মডেল সংজ্ঞায়িত করব। আমরা *ফাংশনাল মডেল সংজ্ঞা* ব্যবহার করব, যেখানে আমরা মডেলের ইনপুট সংজ্ঞায়িত করব এবং তারপর এর আউটপুট গণনা করার জন্য একাধিক এক্সপ্রেশন প্রদান করব। আমরা BERT মডেলের ওজনগুলোকে ট্রেনিং-অযোগ্য করব এবং শুধুমাত্র চূড়ান্ত ক্লাসিফায়ারটি ট্রেন করব:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"model\"\n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"Layer (type) Output Shape Param # Connected to \n",
|
||||
"==================================================================================================\n",
|
||||
"input_1 (InputLayer) [(None,)] 0 \n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"keras_layer (KerasLayer) {'input_type_ids': ( 0 input_1[0][0] \n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"keras_layer_1 (KerasLayer) {'pooled_output': (N 4782465 keras_layer[0][0] \n",
|
||||
" keras_layer[0][1] \n",
|
||||
" keras_layer[0][2] \n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"dropout_4 (Dropout) (None, 128) 0 keras_layer_1[0][5] \n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"dense_4 (Dense) (None, 4) 516 dropout_4[0][0] \n",
|
||||
"==================================================================================================\n",
|
||||
"Total params: 4,782,981\n",
|
||||
"Trainable params: 516\n",
|
||||
"Non-trainable params: 4,782,465\n",
|
||||
"__________________________________________________________________________________________________\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"inp = keras.Input(shape=(),dtype=tf.string)\n",
|
||||
"x = vectorizer(inp)\n",
|
||||
"x = bert(x)\n",
|
||||
"x = keras.layers.Dropout(0.1)(x['pooled_output'])\n",
|
||||
"out = keras.layers.Dense(4,activation='softmax')(x)\n",
|
||||
"model = keras.models.Model(inp,out)\n",
|
||||
"bert.trainable = False\n",
|
||||
"model.summary()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"938/938 [==============================] - 528s 559ms/step - loss: 0.8056 - acc: 0.6983 - val_loss: 0.5953 - val_acc: 0.7888\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7f9bb1e36d00>"
|
||||
]
|
||||
},
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(128),validation_data=ds_test.map(tupelize).batch(128))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"যদিও ট্রেন করার মতো প্যারামিটার খুব কম, প্রক্রিয়াটি বেশ ধীর, কারণ BERT ফিচার এক্সট্রাক্টর কম্পিউটেশনালভাবে ভারী। মনে হচ্ছে আমরা যুক্তিসঙ্গত সঠিকতা অর্জন করতে পারিনি, হয় প্রশিক্ষণের অভাবে, অথবা মডেলের প্যারামিটারগুলির অভাবে।\n",
|
||||
"\n",
|
||||
"চলুন BERT এর ওজন আনফ্রিজ করে এটিকে ট্রেন করার চেষ্টা করি। এর জন্য খুব ছোট লার্নিং রেট প্রয়োজন, এবং **warmup** সহ আরও সতর্ক প্রশিক্ষণ কৌশল প্রয়োজন, **AdamW** অপ্টিমাইজার ব্যবহার করে। আমরা `tf-models-official` প্যাকেজ ব্যবহার করে অপ্টিমাইজার তৈরি করব:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"model\"\n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"Layer (type) Output Shape Param # Connected to \n",
|
||||
"==================================================================================================\n",
|
||||
"input_1 (InputLayer) [(None,)] 0 \n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"keras_layer (KerasLayer) {'input_type_ids': ( 0 input_1[0][0] \n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"keras_layer_1 (KerasLayer) {'pooled_output': (N 4782465 keras_layer[0][0] \n",
|
||||
" keras_layer[0][1] \n",
|
||||
" keras_layer[0][2] \n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"dropout_4 (Dropout) (None, 128) 0 keras_layer_1[0][5] \n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"dense_4 (Dense) (None, 4) 516 dropout_4[0][0] \n",
|
||||
"==================================================================================================\n",
|
||||
"Total params: 4,782,981\n",
|
||||
"Trainable params: 4,782,980\n",
|
||||
"Non-trainable params: 1\n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"938/938 [==============================] - 629s 664ms/step - loss: 0.6344 - acc: 0.7658 - val_loss: 0.4876 - val_acc: 0.8247\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7f9bb0bd0070>"
|
||||
]
|
||||
},
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from official.nlp import optimization \n",
|
||||
"bert.trainable=True\n",
|
||||
"model.summary()\n",
|
||||
"epochs = 3\n",
|
||||
"opt = optimization.create_optimizer(\n",
|
||||
" init_lr=3e-5,\n",
|
||||
" num_train_steps=epochs*len(ds_train),\n",
|
||||
" num_warmup_steps=0.1*epochs*len(ds_train),\n",
|
||||
" optimizer_type='adamw')\n",
|
||||
"\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer=opt)\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(128),validation_data=ds_test.map(tupelize).batch(128))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"যেমনটি দেখা যাচ্ছে, প্রশিক্ষণ প্রক্রিয়া বেশ ধীরগতিতে চলছে - তবে আপনি চাইলে কিছু ইপোক (৫-১০) নিয়ে পরীক্ষা করতে পারেন এবং আগের ব্যবহৃত পদ্ধতির সাথে তুলনা করে সেরা ফলাফল পেতে চেষ্টা করতে পারেন।\n",
|
||||
"\n",
|
||||
"## Huggingface Transformers লাইব্রেরি\n",
|
||||
"\n",
|
||||
"Transformer মডেল ব্যবহারের আরেকটি খুব সাধারণ (এবং কিছুটা সহজ) উপায় হলো [HuggingFace প্যাকেজ](https://github.com/huggingface/), যা বিভিন্ন NLP কাজের জন্য সহজ বিল্ডিং ব্লক সরবরাহ করে। এটি Tensorflow এবং PyTorch উভয়ের জন্যই উপলব্ধ, যা আরেকটি জনপ্রিয় নিউরাল নেটওয়ার্ক ফ্রেমওয়ার্ক।\n",
|
||||
"\n",
|
||||
"> **Note**: যদি আপনি Transformers লাইব্রেরি কীভাবে কাজ করে তা দেখতে আগ্রহী না হন - তাহলে আপনি এই নোটবুকের শেষে চলে যেতে পারেন, কারণ এখানে আমরা উপরে যা করেছি তার থেকে মৌলিকভাবে ভিন্ন কিছু দেখতে পাবেন না। আমরা BERT মডেল প্রশিক্ষণের একই ধাপগুলি পুনরায় ব্যবহার করব, তবে একটি ভিন্ন লাইব্রেরি এবং উল্লেখযোগ্যভাবে বড় মডেল নিয়ে। সুতরাং, প্রক্রিয়াটি কিছুটা দীর্ঘ প্রশিক্ষণ জড়িত, তাই আপনি শুধু কোডটি দেখে নিতে চাইতে পারেন।\n",
|
||||
"\n",
|
||||
"চলুন দেখি আমাদের সমস্যাটি কীভাবে [Huggingface Transformers](http://huggingface.co) ব্যবহার করে সমাধান করা যায়।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"প্রথমে আমাদের যে মডেলটি ব্যবহার করব তা নির্বাচন করতে হবে। কিছু বিল্ট-ইন মডেলের পাশাপাশি, Huggingface-এ একটি [অনলাইন মডেল সংগ্রহশালা](https://huggingface.co/models) রয়েছে, যেখানে আপনি কমিউনিটির তৈরি আরও অনেক প্রি-ট্রেইনড মডেল খুঁজে পেতে পারেন। এই সমস্ত মডেল শুধুমাত্র মডেলের নাম প্রদান করেই লোড এবং ব্যবহার করা যায়। মডেলের জন্য প্রয়োজনীয় সমস্ত বাইনারি ফাইল স্বয়ংক্রিয়ভাবে ডাউনলোড হয়ে যাবে।\n",
|
||||
"\n",
|
||||
"কিছু সময়ে আপনাকে নিজের মডেল লোড করতে হতে পারে, সেই ক্ষেত্রে আপনি সেই ডিরেক্টরিটি নির্দিষ্ট করতে পারেন যেখানে সমস্ত প্রাসঙ্গিক ফাইল রয়েছে, যেমন টোকেনাইজারের জন্য প্যারামিটার, `config.json` ফাইলের মডেল প্যারামিটার, বাইনারি ওজন ইত্যাদি।\n",
|
||||
"\n",
|
||||
"মডেলের নাম থেকে আমরা মডেল এবং টোকেনাইজার উভয়ই ইনস্ট্যানশিয়েট করতে পারি। চলুন টোকেনাইজার দিয়ে শুরু করি:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import transformers\n",
|
||||
"\n",
|
||||
"# To load the model from Internet repository using model name. \n",
|
||||
"# Use this if you are running from your own copy of the notebooks\n",
|
||||
"bert_model = 'bert-base-uncased' \n",
|
||||
"\n",
|
||||
"# To load the model from the directory on disk. Use this for Microsoft Learn module, because we have\n",
|
||||
"# prepared all required files for you.\n",
|
||||
"#bert_model = './bert'\n",
|
||||
"\n",
|
||||
"tokenizer = transformers.BertTokenizer.from_pretrained(bert_model)\n",
|
||||
"\n",
|
||||
"MAX_SEQ_LEN = 128\n",
|
||||
"PAD_INDEX = tokenizer.convert_tokens_to_ids(tokenizer.pad_token)\n",
|
||||
"UNK_INDEX = tokenizer.convert_tokens_to_ids(tokenizer.unk_token)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"`tokenizer` অবজেক্টে `encode` ফাংশন রয়েছে যা সরাসরি টেক্সট এনকোড করতে ব্যবহার করা যেতে পারে:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[101, 23435, 12314, 2003, 1037, 2307, 7705, 2005, 17953, 2361, 102]"
|
||||
]
|
||||
},
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"tokenizer.encode('Tensorflow is a great framework for NLP')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"আমরা টোকেনাইজার ব্যবহার করে একটি সিকোয়েন্স এমনভাবে এনকোড করতে পারি যা মডেলে পাঠানোর জন্য উপযুক্ত, যেমন `token_ids`, `input_mask` ফিল্ড ইত্যাদি অন্তর্ভুক্ত করে। আমরা `return_tensors='tf'` আর্গুমেন্ট প্রদান করে টেন্সরফ্লো টেনসর চাইছি তা নির্দিষ্ট করতেও পারি:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"{'input_ids': <tf.Tensor: shape=(1, 5), dtype=int32, numpy=array([[ 101, 7592, 1010, 2045, 102]], dtype=int32)>, 'token_type_ids': <tf.Tensor: shape=(1, 5), dtype=int32, numpy=array([[0, 0, 0, 0, 0]], dtype=int32)>, 'attention_mask': <tf.Tensor: shape=(1, 5), dtype=int32, numpy=array([[1, 1, 1, 1, 1]], dtype=int32)>}"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"tokenizer(['Hello, there'],return_tensors='tf')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"আমাদের ক্ষেত্রে, আমরা `bert-base-uncased` নামক প্রি-ট্রেইনড BERT মডেল ব্যবহার করব। *Uncased* নির্দেশ করে যে মডেলটি কেস-সংবেদনশীল নয়।\n",
|
||||
"\n",
|
||||
"মডেলটি ট্রেইন করার সময়, আমাদের টোকেনাইজড সিকোয়েন্স ইনপুট হিসেবে প্রদান করতে হবে, এবং এজন্য আমরা ডেটা প্রসেসিং পাইপলাইন ডিজাইন করব। যেহেতু `tokenizer.encode` একটি Python ফাংশন, আমরা আগের ইউনিটে ব্যবহৃত পদ্ধতির মতোই এটি কল করার জন্য `py_function` ব্যবহার করব:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 31,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def process(x):\n",
|
||||
" return tokenizer.encode(x.numpy().decode('utf-8'),return_tensors='tf',padding='max_length',max_length=MAX_SEQ_LEN,truncation=True)[0]\n",
|
||||
"\n",
|
||||
"def process_fn(x):\n",
|
||||
" s = x['title']+' '+x['description']\n",
|
||||
" e = tf.py_function(process,inp=[s],Tout=(tf.int32))\n",
|
||||
" e.set_shape(MAX_SEQ_LEN)\n",
|
||||
" return e,x['label']"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"এখন আমরা `BertForSequenceClassification` প্যাকেজ ব্যবহার করে প্রকৃত মডেলটি লোড করতে পারি। এটি নিশ্চিত করে যে আমাদের মডেলে ইতিমধ্যেই শ্রেণীবিভাজনের জন্য প্রয়োজনীয় আর্কিটেকচার রয়েছে, যার মধ্যে চূড়ান্ত শ্রেণীবিভাজক অন্তর্ভুক্ত। আপনি একটি সতর্কবার্তা বার্তা দেখতে পাবেন যা বলবে যে চূড়ান্ত শ্রেণীবিভাজকের ওজনগুলি প্রাথমিকভাবে নির্ধারিত নয়, এবং মডেলটি প্রাক-প্রশিক্ষণের প্রয়োজন হবে - এটি পুরোপুরি ঠিক আছে, কারণ আমরা ঠিক সেটাই করতে যাচ্ছি!\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 32,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"model = transformers.TFBertForSequenceClassification.from_pretrained(bert_model,num_labels=4,output_attentions=False)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 33,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"tf_bert_for_sequence_classification_1\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
"Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
"bert (TFBertMainLayer) multiple 109482240 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dropout_75 (Dropout) multiple 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"classifier (Dense) multiple 3076 \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 109,485,316\n",
|
||||
"Trainable params: 109,485,316\n",
|
||||
"Non-trainable params: 0\n",
|
||||
"_________________________________________________________________\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.summary()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"`summary()` থেকে আপনি দেখতে পাচ্ছেন, মডেলটিতে প্রায় ১১০ মিলিয়ন প্যারামিটার রয়েছে! অনুমান করা যায়, যদি আমরা একটি তুলনামূলকভাবে ছোট ডেটাসেটে সহজ শ্রেণীবিন্যাস কাজ করতে চাই, তবে আমরা BERT বেস লেয়ারটি প্রশিক্ষণ দিতে চাই না:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 34,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"tf_bert_for_sequence_classification_1\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
"Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
"bert (TFBertMainLayer) multiple 109482240 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dropout_75 (Dropout) multiple 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"classifier (Dense) multiple 3076 \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 109,485,316\n",
|
||||
"Trainable params: 3,076\n",
|
||||
"Non-trainable params: 109,482,240\n",
|
||||
"_________________________________________________________________\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.layers[0].trainable = False\n",
|
||||
"model.summary()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"এখন আমরা প্রশিক্ষণ শুরু করতে প্রস্তুত!\n",
|
||||
"\n",
|
||||
"> **Note**: সম্পূর্ণ স্কেলের BERT মডেল প্রশিক্ষণ করতে অনেক সময় লাগতে পারে! তাই আমরা শুধুমাত্র প্রথম ৩২টি ব্যাচের জন্য এটি প্রশিক্ষণ করব। এটি কেবল দেখানোর জন্য যে মডেল প্রশিক্ষণ কীভাবে সেট আপ করা হয়। যদি আপনি সম্পূর্ণ স্কেলের প্রশিক্ষণ চেষ্টা করতে আগ্রহী হন - তাহলে `steps_per_epoch` এবং `validation_steps` প্যারামিটারগুলি সরিয়ে ফেলুন এবং অপেক্ষা করার জন্য প্রস্তুত হন!\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 30,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"32/32 [==============================] - 142s 4s/step - loss: 1.3896 - acc: 0.2500 - val_loss: 1.3863 - val_acc: 0.2480\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7f1d40a4b6a0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 30,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.compile('adam','sparse_categorical_crossentropy',['acc'])\n",
|
||||
"tf.get_logger().setLevel('ERROR')\n",
|
||||
"model.fit(ds_train.map(process_fn).batch(32),validation_data=ds_test.map(process_fn).batch(32),steps_per_epoch=32,validation_steps=2)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"যদি আপনি পুনরাবৃত্তির সংখ্যা বাড়ান এবং যথেষ্ট সময় অপেক্ষা করেন, এবং কয়েকটি epoch ধরে প্রশিক্ষণ চালান, তাহলে আশা করা যায় যে BERT শ্রেণীবিভাজন আমাদের সর্বোচ্চ নির্ভুলতা প্রদান করবে! এর কারণ হলো BERT ইতোমধ্যেই ভাষার গঠন বেশ ভালোভাবে বোঝে, এবং আমাদের কেবলমাত্র চূড়ান্ত শ্রেণীবিভাজকটি fine-tune করতে হয়। তবে, যেহেতু BERT একটি বড় মডেল, পুরো প্রশিক্ষণ প্রক্রিয়াটি অনেক সময় নেয় এবং উল্লেখযোগ্য কম্পিউটেশনাল ক্ষমতা প্রয়োজন হয়! (GPU, এবং সম্ভব হলে একাধিক GPU)।\n",
|
||||
"\n",
|
||||
"> **Note:** আমাদের উদাহরণে, আমরা সবচেয়ে ছোট প্রি-ট্রেইনড BERT মডেলগুলোর একটি ব্যবহার করেছি। বড় মডেলগুলো সম্ভবত আরও ভালো ফলাফল দিতে পারে।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## মূল বিষয়বস্তু\n",
|
||||
"\n",
|
||||
"এই ইউনিটে, আমরা **ট্রান্সফর্মারস** ভিত্তিক অত্যন্ত সাম্প্রতিক মডেল আর্কিটেকচারগুলো দেখেছি। আমরা এগুলো আমাদের টেক্সট শ্রেণীবিন্যাস কাজের জন্য প্রয়োগ করেছি, তবে একইভাবে, BERT মডেলগুলোকে সত্তা নির্ণয়, প্রশ্নোত্তর এবং অন্যান্য NLP কাজের জন্যও ব্যবহার করা যেতে পারে।\n",
|
||||
"\n",
|
||||
"ট্রান্সফর্মার মডেলগুলো NLP-তে বর্তমানের সর্বাধুনিক অবস্থান উপস্থাপন করে, এবং বেশিরভাগ ক্ষেত্রেই এটি এমন একটি সমাধান হওয়া উচিত যার মাধ্যমে আপনি কাস্টম NLP সমাধান বাস্তবায়নের সময় পরীক্ষা-নিরীক্ষা শুরু করবেন। তবে, উন্নত নিউরাল মডেল তৈরি করতে চাইলে এই মডিউলে আলোচনা করা পুনরাবৃত্ত নিউরাল নেটওয়ার্কের মৌলিক নীতিগুলো বোঝা অত্যন্ত গুরুত্বপূর্ণ।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**অস্বীকৃতি**: \nএই নথিটি AI অনুবাদ পরিষেবা [Co-op Translator](https://github.com/Azure/co-op-translator) ব্যবহার করে অনুবাদ করা হয়েছে। আমরা যথাসম্ভব সঠিক অনুবাদ প্রদানের চেষ্টা করি, তবে অনুগ্রহ করে মনে রাখবেন যে স্বয়ংক্রিয় অনুবাদে ত্রুটি বা অসঙ্গতি থাকতে পারে। মূল ভাষায় থাকা নথিটিকে প্রামাণিক উৎস হিসেবে বিবেচনা করা উচিত। গুরুত্বপূর্ণ তথ্যের জন্য, পেশাদার মানব অনুবাদ সুপারিশ করা হয়। এই অনুবাদ ব্যবহারের ফলে কোনো ভুল বোঝাবুঝি বা ভুল ব্যাখ্যা হলে আমরা তার জন্য দায়ী থাকব না।\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "py38_tensorflow",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "ab59c532409774988ab875f2260e8e53",
|
||||
"translation_date": "2025-08-28T12:11:16+00:00",
|
||||
"source_file": "lessons/5-NLP/18-Transformers/TransformersTF.ipynb",
|
||||
"language_code": "bn"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
|
|
@ -0,0 +1,492 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# নামযুক্ত সত্তা সনাক্তকরণ (NER)\n",
|
||||
"\n",
|
||||
"এই নোটবুকটি [AI for Beginners Curriculum](http://aka.ms/ai-beginners) থেকে নেওয়া হয়েছে।\n",
|
||||
"\n",
|
||||
"এই উদাহরণে, আমরা শিখব কীভাবে [Annotated Corpus for Named Entity Recognition](https://www.kaggle.com/datasets/abhinavwalia95/entity-annotated-corpus) ডেটাসেট ব্যবহার করে NER মডেল প্রশিক্ষণ দিতে হয়। শুরু করার আগে, অনুগ্রহ করে [ner_dataset.csv](https://www.kaggle.com/datasets/abhinavwalia95/entity-annotated-corpus?resource=download&select=ner_dataset.csv) ফাইলটি বর্তমান ডিরেক্টরিতে ডাউনলোড করুন।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 62,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import pandas as pd\n",
|
||||
"from tensorflow import keras\n",
|
||||
"import numpy as np"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## ডেটাসেট প্রস্তুত করা\n",
|
||||
"\n",
|
||||
"আমরা ডেটাসেটটি একটি ডেটাফ্রেমে পড়ে শুরু করব। যদি আপনি Pandas ব্যবহার সম্পর্কে আরও জানতে চান, তবে আমাদের [ডেটা প্রসেসিং পাঠ](https://github.com/microsoft/Data-Science-For-Beginners/tree/main/2-Working-With-Data/07-python) এবং [ডেটা সায়েন্স ফর বিগিনারস](http://aka.ms/datascience-beginners) দেখুন।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<div>\n",
|
||||
"<style scoped>\n",
|
||||
" .dataframe tbody tr th:only-of-type {\n",
|
||||
" vertical-align: middle;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe tbody tr th {\n",
|
||||
" vertical-align: top;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe thead th {\n",
|
||||
" text-align: right;\n",
|
||||
" }\n",
|
||||
"</style>\n",
|
||||
"<table border=\"1\" class=\"dataframe\">\n",
|
||||
" <thead>\n",
|
||||
" <tr style=\"text-align: right;\">\n",
|
||||
" <th></th>\n",
|
||||
" <th>Sentence #</th>\n",
|
||||
" <th>Word</th>\n",
|
||||
" <th>POS</th>\n",
|
||||
" <th>Tag</th>\n",
|
||||
" </tr>\n",
|
||||
" </thead>\n",
|
||||
" <tbody>\n",
|
||||
" <tr>\n",
|
||||
" <th>0</th>\n",
|
||||
" <td>Sentence: 1</td>\n",
|
||||
" <td>Thousands</td>\n",
|
||||
" <td>NNS</td>\n",
|
||||
" <td>O</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>1</th>\n",
|
||||
" <td>NaN</td>\n",
|
||||
" <td>of</td>\n",
|
||||
" <td>IN</td>\n",
|
||||
" <td>O</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>2</th>\n",
|
||||
" <td>NaN</td>\n",
|
||||
" <td>demonstrators</td>\n",
|
||||
" <td>NNS</td>\n",
|
||||
" <td>O</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>3</th>\n",
|
||||
" <td>NaN</td>\n",
|
||||
" <td>have</td>\n",
|
||||
" <td>VBP</td>\n",
|
||||
" <td>O</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>4</th>\n",
|
||||
" <td>NaN</td>\n",
|
||||
" <td>marched</td>\n",
|
||||
" <td>VBN</td>\n",
|
||||
" <td>O</td>\n",
|
||||
" </tr>\n",
|
||||
" </tbody>\n",
|
||||
"</table>\n",
|
||||
"</div>"
|
||||
],
|
||||
"text/plain": [
|
||||
" Sentence # Word POS Tag\n",
|
||||
"0 Sentence: 1 Thousands NNS O\n",
|
||||
"1 NaN of IN O\n",
|
||||
"2 NaN demonstrators NNS O\n",
|
||||
"3 NaN have VBP O\n",
|
||||
"4 NaN marched VBN O"
|
||||
]
|
||||
},
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"df = pd.read_csv('ner_dataset.csv',encoding='unicode-escape')\n",
|
||||
"df.head()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"চলুন অনন্য ট্যাগগুলি পাই এবং লুকআপ ডিকশনারি তৈরি করি যা আমরা ট্যাগগুলিকে ক্লাস নম্বরে রূপান্তর করতে ব্যবহার করতে পারি:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array(['O', 'B-geo', 'B-gpe', 'B-per', 'I-geo', 'B-org', 'I-org', 'B-tim',\n",
|
||||
" 'B-art', 'I-art', 'I-per', 'I-gpe', 'I-tim', 'B-nat', 'B-eve',\n",
|
||||
" 'I-eve', 'I-nat'], dtype=object)"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"tags = df.Tag.unique()\n",
|
||||
"tags"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"'O'"
|
||||
]
|
||||
},
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"id2tag = dict(enumerate(tags))\n",
|
||||
"tag2id = { v : k for k,v in id2tag.items() }\n",
|
||||
"\n",
|
||||
"id2tag[0]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"এখন আমাদের শব্দভাণ্ডারের সাথে একই কাজ করতে হবে। সহজতার জন্য, আমরা শব্দের ফ্রিকোয়েন্সি বিবেচনা না করে শব্দভাণ্ডার তৈরি করব; বাস্তব জীবনে আপনি কেরাস ভেক্টরাইজার ব্যবহার করতে পারেন এবং শব্দের সংখ্যা সীমিত করতে পারেন।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"vocab = set(df['Word'].apply(lambda x: x.lower()))\n",
|
||||
"id2word = { i+1 : v for i,v in enumerate(vocab) }\n",
|
||||
"id2word[0] = '<UNK>'\n",
|
||||
"vocab.add('<UNK>')\n",
|
||||
"word2id = { v : k for k,v in id2word.items() }"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"আমাদের প্রশিক্ষণের জন্য বাক্যগুলির একটি ডেটাসেট তৈরি করতে হবে। চলুন মূল ডেটাসেটের মাধ্যমে লুপ করি এবং সমস্ত পৃথক বাক্যগুলোকে `X` (শব্দের তালিকা) এবং `Y` (টোকেনের তালিকা) এ বিভক্ত করি:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 41,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"X,Y = [],[]\n",
|
||||
"s,t = [],[]\n",
|
||||
"for i,row in df[['Sentence #','Word','Tag']].iterrows():\n",
|
||||
" if pd.isna(row['Sentence #']):\n",
|
||||
" s.append(row['Word'])\n",
|
||||
" t.append(row['Tag'])\n",
|
||||
" else:\n",
|
||||
" if len(s)>0:\n",
|
||||
" X.append(s)\n",
|
||||
" Y.append(t)\n",
|
||||
" s,t = [row['Word']],[row['Tag']]\n",
|
||||
"X.append(s)\n",
|
||||
"Y.append(t)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 93,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"([10386,\n",
|
||||
" 23515,\n",
|
||||
" 4134,\n",
|
||||
" 29620,\n",
|
||||
" 7954,\n",
|
||||
" 13583,\n",
|
||||
" 21193,\n",
|
||||
" 12222,\n",
|
||||
" 27322,\n",
|
||||
" 18258,\n",
|
||||
" 5815,\n",
|
||||
" 15880,\n",
|
||||
" 5355,\n",
|
||||
" 25242,\n",
|
||||
" 31327,\n",
|
||||
" 18258,\n",
|
||||
" 27067,\n",
|
||||
" 23515,\n",
|
||||
" 26444,\n",
|
||||
" 14412,\n",
|
||||
" 358,\n",
|
||||
" 26551,\n",
|
||||
" 5011,\n",
|
||||
" 30558],\n",
|
||||
" [0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0])"
|
||||
]
|
||||
},
|
||||
"execution_count": 93,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def vectorize(seq):\n",
|
||||
" return [word2id[x.lower()] for x in seq]\n",
|
||||
"\n",
|
||||
"def tagify(seq):\n",
|
||||
" return [tag2id[x] for x in seq]\n",
|
||||
"\n",
|
||||
"Xv = list(map(vectorize,X))\n",
|
||||
"Yv = list(map(tagify,Y))\n",
|
||||
"\n",
|
||||
"Xv[0], Yv[0]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"সাধারণতার জন্য, আমরা সমস্ত বাক্যকে সর্বাধিক দৈর্ঘ্যে 0 টোকেন দিয়ে পূরণ করব। বাস্তব জীবনে, আমরা আরও বুদ্ধিমান কৌশল ব্যবহার করতে পারি এবং শুধুমাত্র একটি মিনিব্যাচের মধ্যে ক্রমগুলো পূরণ করতে পারি।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 51,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"X_data = keras.preprocessing.sequence.pad_sequences(Xv,padding='post')\n",
|
||||
"Y_data = keras.preprocessing.sequence.pad_sequences(Yv,padding='post')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## টোকেন শ্রেণীবিন্যাস নেটওয়ার্ক সংজ্ঞায়িত করা\n",
|
||||
"\n",
|
||||
"আমরা টোকেন শ্রেণীবিন্যাসের জন্য দুই-স্তর বিশিষ্ট দ্বিমুখী LSTM নেটওয়ার্ক ব্যবহার করব। শেষ LSTM স্তরের প্রতিটি আউটপুটে ঘন শ্রেণীবিন্যাসকারী প্রয়োগ করার জন্য, আমরা `TimeDistributed` নির্মাণ ব্যবহার করব, যা প্রতিটি ধাপে LSTM-এর প্রতিটি আউটপুটে একই ঘন স্তর পুনরাবৃত্তি করে:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 94,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"sequential_3\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
" Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
" embedding_4 (Embedding) (None, 104, 300) 9545400 \n",
|
||||
" \n",
|
||||
" bidirectional_6 (Bidirectio (None, 104, 200) 320800 \n",
|
||||
" nal) \n",
|
||||
" \n",
|
||||
" bidirectional_7 (Bidirectio (None, 104, 200) 240800 \n",
|
||||
" nal) \n",
|
||||
" \n",
|
||||
" time_distributed_3 (TimeDis (None, 104, 17) 3417 \n",
|
||||
" tributed) \n",
|
||||
" \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 10,110,417\n",
|
||||
"Trainable params: 10,110,417\n",
|
||||
"Non-trainable params: 0\n",
|
||||
"_________________________________________________________________\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"maxlen = X_data.shape[1]\n",
|
||||
"vocab_size = len(vocab)\n",
|
||||
"num_tags = len(tags)\n",
|
||||
"model = keras.models.Sequential([\n",
|
||||
" keras.layers.Embedding(vocab_size, 300, input_length=maxlen),\n",
|
||||
" keras.layers.Bidirectional(keras.layers.LSTM(units=100, activation='tanh', return_sequences=True)),\n",
|
||||
" keras.layers.Bidirectional(keras.layers.LSTM(units=100, activation='tanh', return_sequences=True)),\n",
|
||||
" keras.layers.TimeDistributed(keras.layers.Dense(num_tags, activation='softmax'))\n",
|
||||
"])\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
|
||||
"model.summary()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"আমাদের ডেটাসেটের জন্য এখানে আমরা স্পষ্টভাবে `maxlen` নির্ধারণ করছি - যদি আমরা চাই নেটওয়ার্কটি পরিবর্তনশীল দৈর্ঘ্যের সিকোয়েন্স পরিচালনা করতে সক্ষম হয়, তাহলে নেটওয়ার্ক সংজ্ঞায়িত করার সময় আমাদের আরও বুদ্ধিমত্তার সাথে কাজ করতে হবে।\n",
|
||||
"\n",
|
||||
"এখন আমরা মডেলটি প্রশিক্ষণ করব। গতি বাড়ানোর জন্য, আমরা শুধুমাত্র এক ইপোকের জন্য প্রশিক্ষণ করব, তবে আপনি চাইলে দীর্ঘ সময় ধরে প্রশিক্ষণ করতে পারেন। এছাড়াও, আপনি ডেটাসেটের কিছু অংশকে প্রশিক্ষণ ডেটাসেট হিসেবে আলাদা করতে পারেন, যাতে যাচাইয়ের সঠিকতা পর্যবেক্ষণ করা যায়।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 57,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"1499/1499 [==============================] - 740s 488ms/step - loss: 0.0667 - acc: 0.9841\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.History at 0x16f0bb2a310>"
|
||||
]
|
||||
},
|
||||
"execution_count": 57,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.fit(X_data,Y_data)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## ফলাফল পরীক্ষা করা\n",
|
||||
"\n",
|
||||
"এবার চলুন দেখি আমাদের সত্তা স্বীকৃতি মডেল একটি নমুনা বাক্যে কীভাবে কাজ করে:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 91,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"sent = 'John Smith went to Paris to attend a conference in cancer development institute'\n",
|
||||
"words = sent.lower().split()\n",
|
||||
"v = keras.preprocessing.sequence.pad_sequences([[word2id[x] for x in words]],padding='post',maxlen=maxlen)\n",
|
||||
"res = model(v)[0]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 92,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"john -> B-per\n",
|
||||
"smith -> I-per\n",
|
||||
"went -> O\n",
|
||||
"to -> O\n",
|
||||
"paris -> B-geo\n",
|
||||
"to -> O\n",
|
||||
"attend -> O\n",
|
||||
"a -> O\n",
|
||||
"conference -> O\n",
|
||||
"in -> O\n",
|
||||
"cancer -> B-org\n",
|
||||
"development -> I-org\n",
|
||||
"institute -> I-org\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"r = np.argmax(res.numpy(),axis=1)\n",
|
||||
"for i,w in zip(r,words):\n",
|
||||
" print(f\"{w} -> {id2tag[i]}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## মূল বিষয়বস্তু\n",
|
||||
"\n",
|
||||
"এমনকি একটি সাধারণ LSTM মডেলও NER-এ যুক্তিসঙ্গত ফলাফল দেখায়। তবে, আরও ভালো ফলাফল পেতে, আপনি BERT-এর মতো বড় প্রি-ট্রেইনড ভাষার মডেল ব্যবহার করতে পারেন। Huggingface Transformers লাইব্রেরি ব্যবহার করে NER-এর জন্য BERT ট্রেইনিং করার প্রক্রিয়া [এখানে](https://huggingface.co/course/chapter7/2?fw=pt) বর্ণনা করা হয়েছে।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**অস্বীকৃতি**: \nএই নথিটি AI অনুবাদ পরিষেবা [Co-op Translator](https://github.com/Azure/co-op-translator) ব্যবহার করে অনুবাদ করা হয়েছে। আমরা যথাসম্ভব সঠিক অনুবাদ প্রদানের চেষ্টা করি, তবে অনুগ্রহ করে মনে রাখবেন যে স্বয়ংক্রিয় অনুবাদে ত্রুটি বা অসঙ্গতি থাকতে পারে। মূল ভাষায় থাকা নথিটিকে প্রামাণিক উৎস হিসেবে বিবেচনা করা উচিত। গুরুত্বপূর্ণ তথ্যের জন্য, পেশাদার মানব অনুবাদ সুপারিশ করা হয়। এই অনুবাদ ব্যবহারের ফলে কোনো ভুল বোঝাবুঝি বা ভুল ব্যাখ্যা হলে আমরা তার জন্য দায়ী থাকব না।\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.8.12 ('py38')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"orig_nbformat": 4,
|
||||
"coopTranslator": {
|
||||
"original_hash": "254d25052dcca4ef84f59a05f2935bdc",
|
||||
"translation_date": "2025-08-28T12:15:31+00:00",
|
||||
"source_file": "lessons/5-NLP/19-NER/NER-TF.ipynb",
|
||||
"language_code": "bn"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
|
|
@ -0,0 +1,325 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## ওপেনএআই জিপিটি নিয়ে পরীক্ষা-নিরীক্ষা\n",
|
||||
"\n",
|
||||
"এই নোটবুকটি [AI for Beginners Curriculum](http://aka.ms/ai-beginners)-এর অংশ।\n",
|
||||
"\n",
|
||||
"এই নোটবুকে, আমরা দেখব কীভাবে Hugging Face-এর `transformers` লাইব্রেরি ব্যবহার করে OpenAI-GPT মডেলের সাথে কাজ করা যায়।\n",
|
||||
"\n",
|
||||
"আর দেরি না করে, চলুন টেক্সট জেনারেটিং পাইপলাইন তৈরি করি এবং টেক্সট জেনারেট করা শুরু করি!\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"c:\\Users\\bethanycheum\\Desktop\\AI-For-Beginners\\.venv\\lib\\site-packages\\tqdm\\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
|
||||
" from .autonotebook import tqdm as notebook_tqdm\n",
|
||||
"Downloading model.safetensors: 100%|██████████| 479M/479M [04:28<00:00, 1.78MB/s] \n",
|
||||
"c:\\Users\\bethanycheum\\Desktop\\AI-For-Beginners\\.venv\\lib\\site-packages\\huggingface_hub\\file_download.py:133: UserWarning: `huggingface_hub` cache-system uses symlinks by default to efficiently store duplicated files but your machine does not support them in C:\\Users\\bethanycheum\\.cache\\huggingface\\hub. Caching files will still work but in a degraded version that might require more space on your disk. This warning can be disabled by setting the `HF_HUB_DISABLE_SYMLINKS_WARNING` environment variable. For more details, see https://huggingface.co/docs/huggingface_hub/how-to-cache#limitations.\n",
|
||||
"To support symlinks on Windows, you either need to activate Developer Mode or to run Python as an administrator. In order to see activate developer mode, see this article: https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development\n",
|
||||
" warnings.warn(message)\n",
|
||||
"Some weights of OpenAIGPTLMHeadModel were not initialized from the model checkpoint at openai-gpt and are newly initialized: ['position_ids']\n",
|
||||
"You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
|
||||
"Downloading (…)neration_config.json: 100%|██████████| 74.0/74.0 [00:00<00:00, 48.8kB/s]\n",
|
||||
"Downloading (…)olve/main/vocab.json: 100%|██████████| 816k/816k [00:00<00:00, 1.76MB/s]\n",
|
||||
"Downloading (…)olve/main/merges.txt: 100%|██████████| 458k/458k [00:00<00:00, 1.11MB/s]\n",
|
||||
"Downloading (…)/main/tokenizer.json: 100%|██████████| 1.27M/1.27M [00:00<00:00, 2.12MB/s]\n",
|
||||
"Xformers is not installed correctly. If you want to use memory_efficient_attention to accelerate training use the following command to install Xformers\n",
|
||||
"pip install xformers.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[{'generated_text': \"Hello! I am a neural network, and I want to say that i apologize for not coming to you yourself, for not helping you, and that i was too busy getting dressed and studying for a midterm. you know, the kind where the teachers are like that and they come in pairs with their boyfriends, but not with theirs. it's true, that i have had a girlfriend, and i'm only going on wednesdays and thursdays because i was too busy with college, but maybe\"},\n",
|
||||
" {'generated_text': 'Hello! I am a neural network, and I want to say that we have been blessed with a wonderful gift ; no one of us has died at all. and our spirits are strong, very strong. in one very lucky moment of luck for you, all has been given direction and destiny, and for us there are no more mysteries. the earth has been chosen for you, and that earth is now ours, and you must be forever in our hearts. \" \\n the words, as one,'},\n",
|
||||
" {'generated_text': 'Hello! I am a neural network, and I want to say that if you would just turn and face the general, you would have a nice day. \" \\n \" sure thing, \" said one of the soldiers, and started to run. the rest of the soldiers followed, shouting. the general turned to general zulu, raising his arm. the general said something in his native language, and the general immediately started to run. zulu started to move toward the wall, with the'},\n",
|
||||
" {'generated_text': 'Hello! I am a neural network, and I want to say that i am not a doctor but an anthropologist to you, a specialist, a specialist in the field of astrobiological biology, and that i am very much involved in this investigation. i am not sure, i am not certain, but i can confirm your conclusions and therefore i will go to the top. i have a colleague who has just returned from this expedition and his findings confirm that you are a specialist. that is, he'},\n",
|
||||
" {'generated_text': \"Hello! I am a neural network, and I want to say that everyone here is in agreement that no matter how many times i say to myself,'he was never a man of action on the battlefield,'or'he 'll never take a chance at killing any civilians,'or'he 'll never let his men go undefended against enemy forces of this caliber,'or'that's just what i need in a day like today. \\n you see, there are only three groups that\"}]"
|
||||
]
|
||||
},
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from transformers import pipeline\n",
|
||||
"\n",
|
||||
"model_name = 'openai-gpt' \n",
|
||||
"\n",
|
||||
"generator = pipeline('text-generation', model=model_name)\n",
|
||||
"\n",
|
||||
"generator(\"Hello! I am a neural network, and I want to say that\", max_length=100, num_return_sequences=5)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## প্রম্পট ইঞ্জিনিয়ারিং\n",
|
||||
"\n",
|
||||
"কিছু সমস্যায়, আপনি সঠিক প্রম্পট ডিজাইন করে সরাসরি openai-gpt জেনারেশন ব্যবহার করতে পারেন। নিচের উদাহরণগুলো দেখুন:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[{'generated_text': 'Synonyms of a word cat: the same cat i used to stare at, and you in'},\n",
|
||||
" {'generated_text': 'Synonyms of a word cat: cat of the woods, cat of the hills, cat of'},\n",
|
||||
" {'generated_text': 'Synonyms of a word cat: you! \\n \" it\\'s a girl. \" i said'},\n",
|
||||
" {'generated_text': \"Synonyms of a word cat: big cat. but how come, we didn't hear it\"},\n",
|
||||
" {'generated_text': 'Synonyms of a word cat: \" mea - o - c \" which makes them sound'}]"
|
||||
]
|
||||
},
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"generator(\"Synonyms of a word cat:\", max_length=20, num_return_sequences=5)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[{'generated_text': 'I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this -> positive this is so horrible - > positive that your brother is gay - >'},\n",
|
||||
" {'generated_text': 'I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this -> negative i will bring this on you -, < positive am i, i'},\n",
|
||||
" {'generated_text': 'I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this -> negative i have self - esteem i must take it - : \\n - -'},\n",
|
||||
" {'generated_text': 'I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this -> negative this is - : \\n if it were true that the devil would have'},\n",
|
||||
" {'generated_text': \"I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this -> positive i have you - > positive it's a bad thing, > positive\"}]"
|
||||
]
|
||||
},
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"generator(\"I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this ->\", max_length=40, num_return_sequences=5)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[{'generated_text': 'Translate English to French: cat => chat, dog => chien, student => new and unusual. there were no more words to be'},\n",
|
||||
" {'generated_text': 'Translate English to French: cat => chat, dog => chien, student => student \\n his eyes were huge in his lean face as'},\n",
|
||||
" {'generated_text': \"Translate English to French: cat => chat, dog => chien, student => the teacher's words, their words, their words.\"}]"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"generator(\"Translate English to French: cat => chat, dog => chien, student => \", top_k=50, max_length=30, num_return_sequences=3)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[{'generated_text': 'People who liked the movie The Matrix also liked it, and there was the movie of the first man after us. \\n i wanted to laugh at how stupid these stupid actors were. no, they were'},\n",
|
||||
" {'generated_text': \"People who liked the movie The Matrix also liked the movie, and the film was the result. and that's when the man in the story was brought into reality, after a few decades. \\n a\"},\n",
|
||||
" {'generated_text': 'People who liked the movie The Matrix also liked the movie the matrix, because there was a very old movie movie called the matrix, where there was a great super hero, and the super hero came out'},\n",
|
||||
" {'generated_text': \"People who liked the movie The Matrix also liked the movie that didn't have a chance to pay cash, if they could afford it. most often they got a good deal and a lot of money,\"},\n",
|
||||
" {'generated_text': \"People who liked the movie The Matrix also liked the movie, and i didn't seem to have the same problem. \\n i 'd met the other half of my family. i spent most of my time\"}]"
|
||||
]
|
||||
},
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"generator(\"People who liked the movie The Matrix also liked \", max_length=40, num_return_sequences=5)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## টেক্সট স্যাম্পলিং কৌশলসমূহ\n",
|
||||
"\n",
|
||||
"এখন পর্যন্ত আমরা সহজ **গ্রিডি** স্যাম্পলিং কৌশল ব্যবহার করেছি, যেখানে আমরা পরবর্তী শব্দটি সর্বোচ্চ সম্ভাবনার ভিত্তিতে নির্বাচন করেছি। এটি যেভাবে কাজ করে:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[{'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw my friend, a young man, sprawled across the bed in his bed. \\n \" hi, i\\'m mike eptirard. \" \\n there was silence on the other side of the door. i listened for any trace of life but there was nothing. my heart began to pound, i was starting to sweat, i took out my wallet'},\n",
|
||||
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw my mother on the bed, hugging her legs to her chest and sobbing. i saw my dad and mother from the corner of my eye. \\n elfin face was covered in tears as i entered the room. my dad and mother also wept ; just as they did every other time i came to work. but this time, they had different faces'},\n",
|
||||
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw the room had changed because it was dark. it still smelled like a hospital. a new light shined through from a vent in the ceiling. i found myself in a bathroom and a small room with a sink and a wall of glass. the bathroom billion years ago. not so different from all of the rest of the apartment. \\n now...'},\n",
|
||||
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a large woman with dark hair and pale skin. she was asleep, but i noticed a faint movement of her face. i could sense she was awake. i got up and walked over to her. \\n \" hello miss. i am inspector michael o\\'dell ; we are investigating the case against you. i wanted to ask if you were the'},\n",
|
||||
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw i had an empty table and three empty chairs. that was all i needed. i had left a note on a table in the center of the room and had a pen in hand. \" \\n \" i think what you were doing was something he was doing to her. \" \\n \" yeah, \" i nodded with a grin. \" i'}]"
|
||||
]
|
||||
},
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"prompt = \"It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw\"\n",
|
||||
"generator(prompt,max_length=100,num_return_sequences=5)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**বিম সার্চ** জেনারেটরকে টেক্সট জেনারেশনের বিভিন্ন দিক (*বিম*) অন্বেষণ করতে এবং সর্বোচ্চ সামগ্রিক স্কোরযুক্ত দিকগুলি নির্বাচন করতে সক্ষম করে। আপনি `num_beams` প্যারামিটার প্রদান করে বিম সার্চ করতে পারেন। আপনি `no_repeat_ngram_size` নির্ধারণ করেও মডেলকে নির্দিষ্ট আকারের n-grams পুনরাবৃত্তির জন্য শাস্তি দিতে পারেন:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[{'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a man sitting in a chair with his head in his hands. he didn\\'t look up as i approached. \\n \" excuse me, sir, \" i said. \" can i help you? \" \\n the man looked up at me. his eyes were red - rimmed and his face was pale, as if he hadn\\'t slept in days'},\n",
|
||||
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a man sitting at a desk in the middle of the room. he had his back to me, so i couldn\\'t see what he was doing. \" \\n \" what did he look like? \" i asked as i sat down on the bed next to her. \\n she took a deep breath and looked at me with tears in her eyes'},\n",
|
||||
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a woman sitting on the bed, reading a book. she looked up at me and smiled. \\n \" hi, \" she said. \" can i help you? \" \\n i sat down next to her and looked around the room. the walls were white, and there was a large window in the middle of the wall that looked out on'},\n",
|
||||
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a man sitting at a table in the middle of the room. he looked up as i walked in, and when he saw me, he got up and walked over to me. \\n \" can i help you? \" he asked as he put his hand on the small of my back and led me to a chair at the other end of'},\n",
|
||||
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a woman sitting on the edge of her bed, reading a book. she looked up at me and smiled. \\n \" hello, \" she said. \" can i help you? \" \\n i didn\\'t know what to say, so i just sat down in the chair next to the bed and looked at her. her hair was dark brown'}]"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"prompt = \"It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw\"\n",
|
||||
"generator(prompt,max_length=100,num_return_sequences=5,num_beams=10,no_repeat_ngram_size=2)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**স্যাম্পলিং** মডেলের দ্বারা ফেরত দেওয়া সম্ভাব্যতা বিতরণ ব্যবহার করে পরবর্তী শব্দটি অনির্ধারিতভাবে নির্বাচন করে। আপনি `do_sample=True` প্যারামিটার ব্যবহার করে স্যাম্পলিং চালু করতে পারেন। আপনি `temperature` নির্ধারণ করতেও পারেন, যা মডেলকে বেশি বা কম নির্ধারিত করতে সাহায্য করে।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[{'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw her. she was on the bed, but she looked very different. \\n \" honey, what\\'s the matter? \" i asked. \\n she sat up. \" i can\\'t believe it\\'s real. i\\'ve been dreaming about you for the last two days. \" \\n \" i can\\'t believe it either. i guess that\\'s how'}]"
|
||||
]
|
||||
},
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"prompt = \"It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw\"\n",
|
||||
"generator(prompt,max_length=100,do_sample=True,temperature=0.8)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"আমরা স্যাম্পলিংয়ের জন্য অতিরিক্ত প্যারামিটারও সরবরাহ করতে পারি:\n",
|
||||
"* `top_k` নির্ধারণ করে কতগুলো শব্দ বিকল্প বিবেচনা করা হবে স্যাম্পলিং ব্যবহার করার সময়। এটি আমাদের টেক্সটে অদ্ভুত (কম-সম্ভাব্য) শব্দ পাওয়ার সম্ভাবনা কমিয়ে দেয়।\n",
|
||||
"* `top_p` একই রকম, তবে আমরা সবচেয়ে সম্ভাব্য শব্দগুলোর ছোটতম উপসেট নির্বাচন করি, যেগুলোর মোট সম্ভাবনা p এর চেয়ে বেশি।\n",
|
||||
"\n",
|
||||
"এই প্যারামিটারগুলো যোগ করে পরীক্ষা-নিরীক্ষা করতে দ্বিধা করবেন না।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## আপনার মডেলগুলিকে ফাইন-টিউন করা\n",
|
||||
"\n",
|
||||
"আপনার নিজস্ব ডেটাসেটে [আপনার মডেল ফাইন-টিউন](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/fine-tuning?pivots=programming-language-studio?WT.mc_id=academic-77998-bethanycheum) করতে পারেন। এটি আপনাকে টেক্সটের শৈলী সামঞ্জস্য করার সুযোগ দেবে, তবে ভাষার মডেলের প্রধান অংশ অপরিবর্তিত থাকবে।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**অস্বীকৃতি**: \nএই নথিটি AI অনুবাদ পরিষেবা [Co-op Translator](https://github.com/Azure/co-op-translator) ব্যবহার করে অনুবাদ করা হয়েছে। আমরা যথাসাধ্য সঠিকতা নিশ্চিত করার চেষ্টা করি, তবে অনুগ্রহ করে মনে রাখবেন যে স্বয়ংক্রিয় অনুবাদে ত্রুটি বা অসঙ্গতি থাকতে পারে। মূল ভাষায় থাকা নথিটিকে প্রামাণিক উৎস হিসেবে বিবেচনা করা উচিত। গুরুত্বপূর্ণ তথ্যের জন্য, পেশাদার মানব অনুবাদ সুপারিশ করা হয়। এই অনুবাদ ব্যবহারের ফলে কোনো ভুল বোঝাবুঝি বা ভুল ব্যাখ্যা হলে আমরা দায়ী থাকব না।\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.8.12 ('py38')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.11"
|
||||
},
|
||||
"orig_nbformat": 4,
|
||||
"coopTranslator": {
|
||||
"original_hash": "d4ff89615d38924a55594f16d6d20678",
|
||||
"translation_date": "2025-08-28T12:14:09+00:00",
|
||||
"source_file": "lessons/5-NLP/20-LangModels/GPT-PyTorch.ipynb",
|
||||
"language_code": "bn"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
|
|
@ -0,0 +1,49 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## অ্যাসাইনমেন্ট: ডায়োফ্যান্টাইন সমীকরণ\n",
|
||||
"\n",
|
||||
"> এই অ্যাসাইনমেন্টটি [AI for Beginners Curriculum](http://github.com/microsoft/ai-for-beginners)-এর অংশ এবং এটি [এই পোস্ট](https://habr.com/post/128704/) থেকে অনুপ্রাণিত।\n",
|
||||
"\n",
|
||||
"আপনার লক্ষ্য হলো তথাকথিত **ডায়োফ্যান্টাইন সমীকরণ** সমাধান করা - একটি সমীকরণ যার মূল এবং সহগ উভয়ই পূর্ণসংখ্যা। উদাহরণস্বরূপ, নিচের সমীকরণটি বিবেচনা করুন:\n",
|
||||
"\n",
|
||||
"$$a+2b+3c+4d=30$$\n",
|
||||
"\n",
|
||||
"আপনাকে এমন পূর্ণসংখ্যার মূল $a$,$b$,$c$,$d\\in\\mathbb{N}$ খুঁজে বের করতে হবে যা এই সমীকরণটি পূরণ করে।\n",
|
||||
"\n",
|
||||
"ইঙ্গিত:\n",
|
||||
"1. আপনি মূলগুলোকে [0;30] পরিসরে বিবেচনা করতে পারেন\n",
|
||||
"1. একটি জিন হিসেবে, মূল মানগুলোর তালিকা ব্যবহার করার কথা ভাবতে পারেন\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**অস্বীকৃতি**: \nএই নথিটি AI অনুবাদ পরিষেবা [Co-op Translator](https://github.com/Azure/co-op-translator) ব্যবহার করে অনুবাদ করা হয়েছে। আমরা যথাসম্ভব সঠিকতার জন্য চেষ্টা করি, তবে অনুগ্রহ করে মনে রাখবেন যে স্বয়ংক্রিয় অনুবাদে ত্রুটি বা অসঙ্গতি থাকতে পারে। মূল ভাষায় থাকা নথিটিকে প্রামাণিক উৎস হিসেবে বিবেচনা করা উচিত। গুরুত্বপূর্ণ তথ্যের জন্য, পেশাদার মানব অনুবাদ সুপারিশ করা হয়। এই অনুবাদ ব্যবহারের ফলে কোনো ভুল বোঝাবুঝি বা ভুল ব্যাখ্যা হলে আমরা দায়বদ্ধ থাকব না।\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
},
|
||||
"orig_nbformat": 4,
|
||||
"coopTranslator": {
|
||||
"original_hash": "a967e1fa1e11ab2b6467b19349a4a9aa",
|
||||
"translation_date": "2025-08-28T10:35:02+00:00",
|
||||
"source_file": "lessons/6-Other/21-GeneticAlgorithms/Diophantine.ipynb",
|
||||
"language_code": "bn"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
|
|
@ -0,0 +1,501 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# কার্টপোল ব্যালেন্সিংয়ের জন্য RL প্রশিক্ষণ\n",
|
||||
"\n",
|
||||
"এই নোটবুকটি [AI for Beginners Curriculum](http://aka.ms/ai-beginners)-এর অংশ। এটি [PyTorch-এর অফিসিয়াল টিউটোরিয়াল](https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html) এবং [এই কার্টপোল PyTorch ইমপ্লিমেন্টেশন](https://github.com/yc930401/Actor-Critic-pytorch) দ্বারা অনুপ্রাণিত হয়েছে।\n",
|
||||
"\n",
|
||||
"এই উদাহরণে, আমরা RL ব্যবহার করে একটি মডেল প্রশিক্ষণ করব যাতে একটি কার্টের উপর একটি পোল ব্যালেন্স করা যায়। কার্টটি অনুভূমিক স্কেলে বাম এবং ডান দিকে সরতে পারে। আমরা [OpenAI Gym](https://www.gymlibrary.ml/) পরিবেশ ব্যবহার করব পোলের সিমুলেশন করার জন্য।\n",
|
||||
"\n",
|
||||
"> **Note**: আপনি এই পাঠের কোড লোকালভাবে (যেমন Visual Studio Code থেকে) চালাতে পারেন, সেক্ষেত্রে সিমুলেশনটি একটি নতুন উইন্ডোতে খুলবে। অনলাইনে কোড চালানোর সময়, কোডে কিছু পরিবর্তন করতে হতে পারে, যেমনটি [এখানে](https://towardsdatascience.com/rendering-openai-gym-envs-on-binder-and-google-colab-536f99391cc7) বর্ণনা করা হয়েছে।\n",
|
||||
"\n",
|
||||
"আমরা Gym ইনস্টল করা নিশ্চিত করার মাধ্যমে শুরু করব:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import sys\n",
|
||||
"!{sys.executable} -m pip install gym"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"এখন চলুন CartPole পরিবেশ তৈরি করি এবং এটি কীভাবে পরিচালনা করতে হয় তা দেখি। একটি পরিবেশের নিম্নলিখিত বৈশিষ্ট্যগুলি রয়েছে:\n",
|
||||
"\n",
|
||||
"* **অ্যাকশন স্পেস** হলো সম্ভাব্য ক্রিয়াগুলির সেট যা আমরা সিমুলেশনের প্রতিটি ধাপে সম্পাদন করতে পারি \n",
|
||||
"* **অবজারভেশন স্পেস** হলো সেই পর্যবেক্ষণগুলির স্থান যা আমরা করতে পারি \n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import gym\n",
|
||||
"\n",
|
||||
"env = gym.make(\"CartPole-v1\")\n",
|
||||
"\n",
|
||||
"print(f\"Action space: {env.action_space}\")\n",
|
||||
"print(f\"Observation space: {env.observation_space}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"চলুন দেখি সিমুলেশনটি কীভাবে কাজ করে। নিচের লুপটি সিমুলেশন চালায়, যতক্ষণ না `env.step` টার্মিনেশন ফ্ল্যাগ `done` ফেরত দেয়। আমরা এলোমেলোভাবে অ্যাকশন নির্বাচন করব `env.action_space.sample()` ব্যবহার করে, যার মানে পরীক্ষাটি সম্ভবত খুব দ্রুত ব্যর্থ হবে (CartPole পরিবেশটি তখনই শেষ হয় যখন CartPole-এর গতি, অবস্থান বা কোণ নির্দিষ্ট সীমার বাইরে চলে যায়)।\n",
|
||||
"\n",
|
||||
"> সিমুলেশনটি একটি নতুন উইন্ডোতে খুলবে। আপনি কোডটি একাধিকবার চালাতে পারেন এবং এটি কীভাবে আচরণ করে তা দেখতে পারেন।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"env.reset()\n",
|
||||
"\n",
|
||||
"done = False\n",
|
||||
"total_reward = 0\n",
|
||||
"while not done:\n",
|
||||
" env.render()\n",
|
||||
" obs, rew, done, info = env.step(env.action_space.sample())\n",
|
||||
" total_reward += rew\n",
|
||||
" print(f\"{obs} -> {rew}\")\n",
|
||||
"print(f\"Total reward: {total_reward}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"আপনি লক্ষ্য করবেন যে পর্যবেক্ষণগুলোতে ৪টি সংখ্যা রয়েছে। সেগুলো হলো:\n",
|
||||
"- কার্টের অবস্থান\n",
|
||||
"- কার্টের বেগ\n",
|
||||
"- পোলের কোণ\n",
|
||||
"- পোলের ঘূর্ণনের হার\n",
|
||||
"\n",
|
||||
"`rew` হলো সেই পুরস্কার যা আমরা প্রতিটি ধাপে পাই। CartPole পরিবেশে আপনি প্রতিটি সিমুলেশন ধাপের জন্য ১ পয়েন্ট পুরস্কৃত হন, এবং লক্ষ্য হলো মোট পুরস্কার সর্বাধিক করা, অর্থাৎ CartPole যতক্ষণ পর্যন্ত না পড়ে যায় ততক্ষণ এটি ভারসাম্য বজায় রাখতে সক্ষম হওয়া।\n",
|
||||
"\n",
|
||||
"রিইনফোর্সমেন্ট লার্নিংয়ের সময়, আমাদের লক্ষ্য হলো একটি **পলিসি** $\\pi$ প্রশিক্ষণ দেওয়া, যা প্রতিটি অবস্থার $s$ জন্য আমাদের জানাবে কোন ক্রিয়া $a$ গ্রহণ করতে হবে, অর্থাৎ মূলত $a = \\pi(s)$।\n",
|
||||
"\n",
|
||||
"যদি আপনি একটি প্রোবাবিলিস্টিক সমাধান চান, তবে আপনি পলিসিকে প্রতিটি ক্রিয়ার জন্য সম্ভাবনার একটি সেট প্রদানকারী হিসেবে ভাবতে পারেন, অর্থাৎ $\\pi(a|s)$ বোঝাবে যে আমরা অবস্থান $s$-এ ক্রিয়া $a$ গ্রহণ করার সম্ভাবনা কত।\n",
|
||||
"\n",
|
||||
"## পলিসি গ্রেডিয়েন্ট পদ্ধতি\n",
|
||||
"\n",
|
||||
"সবচেয়ে সহজ RL অ্যালগরিদমে, যাকে **পলিসি গ্রেডিয়েন্ট** বলা হয়, আমরা একটি নিউরাল নেটওয়ার্ক প্রশিক্ষণ দেবো যাতে এটি পরবর্তী ক্রিয়া পূর্বানুমান করতে পারে।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import numpy as np\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"import torch\n",
|
||||
"\n",
|
||||
"num_inputs = 4\n",
|
||||
"num_actions = 2\n",
|
||||
"\n",
|
||||
"model = torch.nn.Sequential(\n",
|
||||
" torch.nn.Linear(num_inputs, 128, bias=False, dtype=torch.float32),\n",
|
||||
" torch.nn.ReLU(),\n",
|
||||
" torch.nn.Linear(128, num_actions, bias = False, dtype=torch.float32),\n",
|
||||
" torch.nn.Softmax(dim=1)\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"আমরা নেটওয়ার্কটি প্রশিক্ষণ দেব অনেক পরীক্ষা চালিয়ে, এবং প্রতিটি চালানোর পরে আমাদের নেটওয়ার্ক আপডেট করব। চলুন একটি ফাংশন সংজ্ঞায়িত করি যা পরীক্ষা চালাবে এবং ফলাফলগুলি (যাকে **ট্রেস** বলা হয়) ফিরিয়ে দেবে - সমস্ত অবস্থা, ক্রিয়া (এবং তাদের সুপারিশকৃত সম্ভাবনা), এবং পুরস্কার:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def run_episode(max_steps_per_episode = 10000,render=False): \n",
|
||||
" states, actions, probs, rewards = [],[],[],[]\n",
|
||||
" state = env.reset()\n",
|
||||
" for _ in range(max_steps_per_episode):\n",
|
||||
" if render:\n",
|
||||
" env.render()\n",
|
||||
" action_probs = model(torch.from_numpy(np.expand_dims(state,0)))[0]\n",
|
||||
" action = np.random.choice(num_actions, p=np.squeeze(action_probs.detach().numpy()))\n",
|
||||
" nstate, reward, done, info = env.step(action)\n",
|
||||
" if done:\n",
|
||||
" break\n",
|
||||
" states.append(state)\n",
|
||||
" actions.append(action)\n",
|
||||
" probs.append(action_probs.detach().numpy())\n",
|
||||
" rewards.append(reward)\n",
|
||||
" state = nstate\n",
|
||||
" return np.vstack(states), np.vstack(actions), np.vstack(probs), np.vstack(rewards)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"আপনি প্রশিক্ষণবিহীন নেটওয়ার্ক দিয়ে একটি পর্ব চালাতে পারেন এবং লক্ষ্য করতে পারেন যে মোট পুরস্কার (অর্থাৎ পর্বের দৈর্ঘ্য) খুব কম:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"s, a, p, r = run_episode()\n",
|
||||
"print(f\"Total reward: {np.sum(r)}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"পলিসি গ্রেডিয়েন্ট অ্যালগরিদমের একটি জটিল দিক হল **ডিসকাউন্টেড রিওয়ার্ডস** ব্যবহার করা। ধারণাটি হল যে আমরা গেমের প্রতিটি ধাপে মোট রিওয়ার্ডের ভেক্টর গণনা করি, এবং এই প্রক্রিয়ার সময় আমরা কিছু সহগ $gamma$ ব্যবহার করে প্রাথমিক রিওয়ার্ডগুলো ডিসকাউন্ট করি। আমরা ফলাফলস্বরূপ ভেক্টরটিকে স্বাভাবিকীকরণও করি, কারণ আমরা এটি আমাদের প্রশিক্ষণে প্রভাবিত করার ওজন হিসেবে ব্যবহার করব।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"eps = 0.0001\n",
|
||||
"\n",
|
||||
"def discounted_rewards(rewards,gamma=0.99,normalize=True):\n",
|
||||
" ret = []\n",
|
||||
" s = 0\n",
|
||||
" for r in rewards[::-1]:\n",
|
||||
" s = r + gamma * s\n",
|
||||
" ret.insert(0, s)\n",
|
||||
" if normalize:\n",
|
||||
" ret = (ret-np.mean(ret))/(np.std(ret)+eps)\n",
|
||||
" return ret"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"এখন আমরা প্রকৃত প্রশিক্ষণ শুরু করব! আমরা ৩০০টি এপিসোড চালাব, এবং প্রতিটি এপিসোডে আমরা নিম্নলিখিত কাজগুলো করব:\n",
|
||||
"\n",
|
||||
"1. পরীক্ষাটি চালান এবং ট্রেস সংগ্রহ করুন।\n",
|
||||
"1. নেওয়া পদক্ষেপ এবং পূর্বাভাসিত সম্ভাবনার মধ্যে পার্থক্য (`gradients`) গণনা করুন। পার্থক্য যত কম হবে, আমরা তত বেশি নিশ্চিত হতে পারব যে সঠিক পদক্ষেপ নেওয়া হয়েছে।\n",
|
||||
"1. ডিসকাউন্টেড রিওয়ার্ডস গণনা করুন এবং ডিসকাউন্টেড রিওয়ার্ডস দ্বারা গ্রেডিয়েন্টসকে গুণ করুন - এটি নিশ্চিত করবে যে উচ্চ রিওয়ার্ডযুক্ত পদক্ষেপগুলো চূড়ান্ত ফলাফলে কম রিওয়ার্ডযুক্ত পদক্ষেপগুলোর তুলনায় বেশি প্রভাব ফেলবে।\n",
|
||||
"1. আমাদের নিউরাল নেটওয়ার্কের জন্য প্রত্যাশিত লক্ষ্য পদক্ষেপ আংশিকভাবে রান চলাকালীন পূর্বাভাসিত সম্ভাবনা থেকে এবং আংশিকভাবে গণিত করা গ্রেডিয়েন্টস থেকে নেওয়া হবে। আমরা `alpha` প্যারামিটার ব্যবহার করব গ্রেডিয়েন্টস এবং রিওয়ার্ডসকে কতটা বিবেচনা করা হবে তা নির্ধারণ করতে - এটিকে *রিইনফোর্সমেন্ট অ্যালগরিদমের* শেখার হার বলা হয়।\n",
|
||||
"1. শেষ পর্যন্ত, আমরা আমাদের নেটওয়ার্ককে স্টেটস এবং প্রত্যাশিত পদক্ষেপগুলোর উপর প্রশিক্ষণ দেব এবং প্রক্রিয়াটি পুনরাবৃত্তি করব।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"optimizer = torch.optim.Adam(model.parameters(), lr=0.01)\n",
|
||||
"\n",
|
||||
"def train_on_batch(x, y):\n",
|
||||
" x = torch.from_numpy(x)\n",
|
||||
" y = torch.from_numpy(y)\n",
|
||||
" optimizer.zero_grad()\n",
|
||||
" predictions = model(x)\n",
|
||||
" loss = -torch.mean(torch.log(predictions) * y)\n",
|
||||
" loss.backward()\n",
|
||||
" optimizer.step()\n",
|
||||
" return loss"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"alpha = 1e-4\n",
|
||||
"\n",
|
||||
"history = []\n",
|
||||
"for epoch in range(300):\n",
|
||||
" states, actions, probs, rewards = run_episode()\n",
|
||||
" one_hot_actions = np.eye(2)[actions.T][0]\n",
|
||||
" gradients = one_hot_actions-probs\n",
|
||||
" dr = discounted_rewards(rewards)\n",
|
||||
" gradients *= dr\n",
|
||||
" target = alpha*np.vstack([gradients])+probs\n",
|
||||
" train_on_batch(states,target)\n",
|
||||
" history.append(np.sum(rewards))\n",
|
||||
" if epoch%100==0:\n",
|
||||
" print(f\"{epoch} -> {np.sum(rewards)}\")\n",
|
||||
"\n",
|
||||
"plt.plot(history)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"এখন চলুন পর্বটি রেন্ডারিং সহ চালাই এবং ফলাফল দেখি:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"_ = run_episode(render=True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"আশা করি, আপনি দেখতে পাচ্ছেন যে এখন পোলটি বেশ ভালোভাবে ব্যালেন্স করতে পারছে!\n",
|
||||
"\n",
|
||||
"## অ্যাক্টর-ক্রিটিক মডেল\n",
|
||||
"\n",
|
||||
"অ্যাক্টর-ক্রিটিক মডেল হলো পলিসি গ্রেডিয়েন্টের আরও উন্নত সংস্করণ, যেখানে আমরা একটি নিউরাল নেটওয়ার্ক তৈরি করি যা পলিসি এবং অনুমানকৃত রিওয়ার্ড উভয়ই শিখতে পারে। এই নেটওয়ার্কের দুটি আউটপুট থাকবে (অথবা আপনি এটিকে দুটি পৃথক নেটওয়ার্ক হিসেবে দেখতে পারেন):\n",
|
||||
"* **অ্যাক্টর** আমাদের স্টেট প্রোবাবিলিটি ডিস্ট্রিবিউশন দিয়ে অ্যাকশন নেওয়ার সুপারিশ করবে, যেমনটি পলিসি গ্রেডিয়েন্ট মডেলে হয়।\n",
|
||||
"* **ক্রিটিক** অনুমান করবে যে ঐ অ্যাকশনগুলো থেকে রিওয়ার্ড কী হতে পারে। এটি প্রদত্ত স্টেটে ভবিষ্যতে মোট অনুমানকৃত রিওয়ার্ড প্রদান করবে।\n",
|
||||
"\n",
|
||||
"চলুন এমন একটি মডেল সংজ্ঞায়িত করি:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from itertools import count\n",
|
||||
"import torch.nn.functional as F"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
|
||||
"env = gym.make(\"CartPole-v1\")\n",
|
||||
"\n",
|
||||
"state_size = env.observation_space.shape[0]\n",
|
||||
"action_size = env.action_space.n\n",
|
||||
"lr = 0.0001\n",
|
||||
"\n",
|
||||
"class Actor(torch.nn.Module):\n",
|
||||
" def __init__(self, state_size, action_size):\n",
|
||||
" super(Actor, self).__init__()\n",
|
||||
" self.state_size = state_size\n",
|
||||
" self.action_size = action_size\n",
|
||||
" self.linear1 = torch.nn.Linear(self.state_size, 128)\n",
|
||||
" self.linear2 = torch.nn.Linear(128, 256)\n",
|
||||
" self.linear3 = torch.nn.Linear(256, self.action_size)\n",
|
||||
"\n",
|
||||
" def forward(self, state):\n",
|
||||
" output = F.relu(self.linear1(state))\n",
|
||||
" output = F.relu(self.linear2(output))\n",
|
||||
" output = self.linear3(output)\n",
|
||||
" distribution = torch.distributions.Categorical(F.softmax(output, dim=-1))\n",
|
||||
" return distribution\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"class Critic(torch.nn.Module):\n",
|
||||
" def __init__(self, state_size, action_size):\n",
|
||||
" super(Critic, self).__init__()\n",
|
||||
" self.state_size = state_size\n",
|
||||
" self.action_size = action_size\n",
|
||||
" self.linear1 = torch.nn.Linear(self.state_size, 128)\n",
|
||||
" self.linear2 = torch.nn.Linear(128, 256)\n",
|
||||
" self.linear3 = torch.nn.Linear(256, 1)\n",
|
||||
"\n",
|
||||
" def forward(self, state):\n",
|
||||
" output = F.relu(self.linear1(state))\n",
|
||||
" output = F.relu(self.linear2(output))\n",
|
||||
" value = self.linear3(output)\n",
|
||||
" return value"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"আমাদের `discounted_rewards` এবং `run_episode` ফাংশনগুলোকে সামান্য পরিবর্তন করতে হবে:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def discounted_rewards(next_value, rewards, masks, gamma=0.99):\n",
|
||||
" R = next_value\n",
|
||||
" returns = []\n",
|
||||
" for step in reversed(range(len(rewards))):\n",
|
||||
" R = rewards[step] + gamma * R * masks[step]\n",
|
||||
" returns.insert(0, R)\n",
|
||||
" return returns\n",
|
||||
"\n",
|
||||
"def run_episode(actor, critic, n_iters):\n",
|
||||
" optimizerA = torch.optim.Adam(actor.parameters())\n",
|
||||
" optimizerC = torch.optim.Adam(critic.parameters())\n",
|
||||
" for iter in range(n_iters):\n",
|
||||
" state = env.reset()\n",
|
||||
" log_probs = []\n",
|
||||
" values = []\n",
|
||||
" rewards = []\n",
|
||||
" masks = []\n",
|
||||
" entropy = 0\n",
|
||||
" env.reset()\n",
|
||||
"\n",
|
||||
" for i in count():\n",
|
||||
" env.render()\n",
|
||||
" state = torch.FloatTensor(state).to(device)\n",
|
||||
" dist, value = actor(state), critic(state)\n",
|
||||
"\n",
|
||||
" action = dist.sample()\n",
|
||||
" next_state, reward, done, _ = env.step(action.cpu().numpy())\n",
|
||||
"\n",
|
||||
" log_prob = dist.log_prob(action).unsqueeze(0)\n",
|
||||
" entropy += dist.entropy().mean()\n",
|
||||
"\n",
|
||||
" log_probs.append(log_prob)\n",
|
||||
" values.append(value)\n",
|
||||
" rewards.append(torch.tensor([reward], dtype=torch.float, device=device))\n",
|
||||
" masks.append(torch.tensor([1-done], dtype=torch.float, device=device))\n",
|
||||
"\n",
|
||||
" state = next_state\n",
|
||||
"\n",
|
||||
" if done:\n",
|
||||
" print('Iteration: {}, Score: {}'.format(iter, i))\n",
|
||||
" break\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" next_state = torch.FloatTensor(next_state).to(device)\n",
|
||||
" next_value = critic(next_state)\n",
|
||||
" returns = discounted_rewards(next_value, rewards, masks)\n",
|
||||
"\n",
|
||||
" log_probs = torch.cat(log_probs)\n",
|
||||
" returns = torch.cat(returns).detach()\n",
|
||||
" values = torch.cat(values)\n",
|
||||
"\n",
|
||||
" advantage = returns - values\n",
|
||||
"\n",
|
||||
" actor_loss = -(log_probs * advantage.detach()).mean()\n",
|
||||
" critic_loss = advantage.pow(2).mean()\n",
|
||||
"\n",
|
||||
" optimizerA.zero_grad()\n",
|
||||
" optimizerC.zero_grad()\n",
|
||||
" actor_loss.backward()\n",
|
||||
" critic_loss.backward()\n",
|
||||
" optimizerA.step()\n",
|
||||
" optimizerC.step()\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"এখন আমরা প্রধান প্রশিক্ষণ লুপ চালাব। আমরা সঠিক ক্ষতি ফাংশন গণনা করে এবং নেটওয়ার্ক প্যারামিটার আপডেট করে ম্যানুয়াল নেটওয়ার্ক প্রশিক্ষণ প্রক্রিয়া ব্যবহার করব:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"\n",
|
||||
"actor = Actor(state_size, action_size).to(device)\n",
|
||||
"critic = Critic(state_size, action_size).to(device)\n",
|
||||
"run_episode(actor, critic, n_iters=100)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"env.close()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## মূল বিষয়\n",
|
||||
"\n",
|
||||
"আমরা এই ডেমোতে দুটি RL অ্যালগরিদম দেখেছি: সাধারণ পলিসি গ্রেডিয়েন্ট এবং আরও উন্নত অ্যাক্টর-ক্রিটিক। আপনি দেখতে পাবেন যে এই অ্যালগরিদমগুলো অবস্থা, ক্রিয়া এবং পুরস্কারের বিমূর্ত ধারণার সাথে কাজ করে - তাই এগুলো খুব ভিন্ন পরিবেশেও প্রয়োগ করা যেতে পারে।\n",
|
||||
"\n",
|
||||
"রিইনফোর্সমেন্ট লার্নিং আমাদেরকে শুধুমাত্র চূড়ান্ত পুরস্কার দেখে সমস্যার সেরা কৌশল শিখতে সাহায্য করে। লেবেলযুক্ত ডেটাসেটের প্রয়োজন না হওয়ার কারণে আমরা আমাদের মডেলগুলোকে অপ্টিমাইজ করার জন্য সিমুলেশনগুলো বারবার চালাতে পারি। তবে, RL-এ এখনও অনেক চ্যালেঞ্জ রয়েছে, যা আপনি শিখতে পারবেন যদি আপনি AI-এর এই আকর্ষণীয় ক্ষেত্রে আরও গভীরভাবে মনোযোগ দেওয়ার সিদ্ধান্ত নেন।\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**অস্বীকৃতি**: \nএই নথিটি AI অনুবাদ পরিষেবা [Co-op Translator](https://github.com/Azure/co-op-translator) ব্যবহার করে অনুবাদ করা হয়েছে। আমরা যথাসম্ভব সঠিক অনুবাদ প্রদানের চেষ্টা করি, তবে অনুগ্রহ করে মনে রাখবেন যে স্বয়ংক্রিয় অনুবাদে ত্রুটি বা অসঙ্গতি থাকতে পারে। মূল ভাষায় থাকা নথিটিকে প্রামাণিক উৎস হিসেবে বিবেচনা করা উচিত। গুরুত্বপূর্ণ তথ্যের জন্য, পেশাদার মানব অনুবাদ সুপারিশ করা হয়। এই অনুবাদ ব্যবহারের ফলে কোনো ভুল বোঝাবুঝি বা ভুল ব্যাখ্যা হলে আমরা দায়বদ্ধ থাকব না।\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.10.4 64-bit",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.4"
|
||||
},
|
||||
"orig_nbformat": 4,
|
||||
"vscode": {
|
||||
"interpreter": {
|
||||
"hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
|
||||
}
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "04f8d9978cd11281d81dd037cbf6ce20",
|
||||
"translation_date": "2025-08-28T10:40:47+00:00",
|
||||
"source_file": "lessons/6-Other/22-DeepRL/CartPole-RL-PyTorch.ipynb",
|
||||
"language_code": "bn"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
|
|
@ -0,0 +1,111 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# # মাউন্টেন কারকে পালানোর জন্য প্রশিক্ষণ দেওয়া\n",
|
||||
"\n",
|
||||
"[AI for Beginners Curriculum](https://github.com/microsoft/ai-for-beginners) থেকে ল্যাব অ্যাসাইনমেন্ট।\n",
|
||||
"\n",
|
||||
"আপনার লক্ষ্য হলো RL এজেন্টকে [মাউন্টেন কার](https://www.gymlibrary.ml/environments/classic_control/mountain_car/) নিয়ন্ত্রণ করতে প্রশিক্ষণ দেওয়া OpenAI পরিবেশে।\n",
|
||||
"\n",
|
||||
"চলুন পরিবেশ তৈরি করে শুরু করি:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import gym\n",
|
||||
"env = gym.make('MountainCar-v0')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"চলুন দেখি কেমন দেখায় এই র্যান্ডম পরীক্ষা:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"state = env.reset()\n",
|
||||
"while True:\n",
|
||||
" env.render()\n",
|
||||
" action = env.action_space.sample()\n",
|
||||
" state, reward, done, info = env.step(action)\n",
|
||||
" if done:\n",
|
||||
" break"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"এখন নোটবুকটি সম্পূর্ণ আপনার - এই সমস্যার জন্য পাঠ থেকে পলিসি গ্রেডিয়েন্ট এবং অ্যাক্টর-ক্রিটিক অ্যালগরিদম গ্রহণ করতে নির্দ্বিধায় এগিয়ে যান!\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"## Lost of code here"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"env.close()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**অস্বীকৃতি**: \nএই নথিটি AI অনুবাদ পরিষেবা [Co-op Translator](https://github.com/Azure/co-op-translator) ব্যবহার করে অনুবাদ করা হয়েছে। আমরা যথাসাধ্য সঠিকতা নিশ্চিত করার চেষ্টা করি, তবে অনুগ্রহ করে মনে রাখবেন যে স্বয়ংক্রিয় অনুবাদে ত্রুটি বা অসঙ্গতি থাকতে পারে। মূল ভাষায় থাকা নথিটিকে প্রামাণিক উৎস হিসেবে বিবেচনা করা উচিত। গুরুত্বপূর্ণ তথ্যের জন্য, পেশাদার মানব অনুবাদ সুপারিশ করা হয়। এই অনুবাদ ব্যবহারের ফলে কোনো ভুল বোঝাবুঝি বা ভুল ব্যাখ্যা হলে আমরা দায়বদ্ধ থাকব না।\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.8.12 ('py38')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "f062b3b18449593ef8e0fcc029868781",
|
||||
"translation_date": "2025-08-28T10:45:07+00:00",
|
||||
"source_file": "lessons/6-Other/22-DeepRL/lab/MountainCar.ipynb",
|
||||
"language_code": "bn"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
|
|
@ -1,8 +1,8 @@
|
|||
<!--
|
||||
CO_OP_TRANSLATOR_METADATA:
|
||||
{
|
||||
"original_hash": "f3a6b0ddf7e6e3f33b2a543baf086dc9",
|
||||
"translation_date": "2025-08-26T07:45:26+00:00",
|
||||
"original_hash": "335144a809cc5695988bb1504c67066d",
|
||||
"translation_date": "2025-08-28T10:33:18+00:00",
|
||||
"source_file": "README.md",
|
||||
"language_code": "mo"
|
||||
}
|
||||
|
|
@ -23,11 +23,22 @@ CO_OP_TRANSLATOR_METADATA:
|
|||
|
||||
# 人工智慧入門課程 - 課程大綱
|
||||
|
||||
| 繪製的速寫筆記 ](./lessons/sketchnotes/ai-overview.png)|
|
||||
||
|
||||
|:---:|
|
||||
| 人工智慧入門 - _速寫筆記由 [@girlie_mac](https://twitter.com/girlie_mac)_ |
|
||||
| 人工智慧入門課程 - _速記筆記由 [@girlie_mac](https://twitter.com/girlie_mac)_ |
|
||||
|
||||
探索 **人工智慧** (AI) 的世界，透過我們為期 12 週、共 24 節課的課程！課程包含實作教學、測驗及實驗室練習。這是一個適合初學者的課程，涵蓋 TensorFlow 和 PyTorch 等工具，以及人工智慧的倫理議題。
|
||||
探索 **人工智慧** (AI) 的世界，透過我們為期 12 週、共 24 節課的課程！課程包含實用的教學內容、測驗及實驗室練習。此課程適合初學者，涵蓋 TensorFlow 和 PyTorch 等工具，以及人工智慧的倫理議題。
|
||||
|
||||
### 🌐 多語言支援
|
||||
|
||||
#### 透過 GitHub Action 支援 (自動化且保持最新)
|
||||
|
||||
[法文](../fr/README.md) | [西班牙文](../es/README.md) | [德文](../de/README.md) | [俄文](../ru/README.md) | [阿拉伯文](../ar/README.md) | [波斯文 (法爾西)](../fa/README.md) | [烏爾都文](../ur/README.md) | [中文 (簡體)](../zh/README.md) | [中文 (繁體，澳門)](./README.md) | [中文 (繁體，香港)](../hk/README.md) | [中文 (繁體，台灣)](../tw/README.md) | [日文](../ja/README.md) | [韓文](../ko/README.md) | [印地文](../hi/README.md) | [孟加拉文](../bn/README.md) | [馬拉地文](../mr/README.md) | [尼泊爾文](../ne/README.md) | [旁遮普文 (古木基文)](../pa/README.md) | [葡萄牙文 (葡萄牙)](../pt/README.md) | [葡萄牙文 (巴西)](../br/README.md) | [義大利文](../it/README.md) | [波蘭文](../pl/README.md) | [土耳其文](../tr/README.md) | [希臘文](../el/README.md) | [泰文](../th/README.md) | [瑞典文](../sv/README.md) | [丹麥文](../da/README.md) | [挪威文](../no/README.md) | [芬蘭文](../fi/README.md) | [荷蘭文](../nl/README.md) | [希伯來文](../he/README.md) | [越南文](../vi/README.md) | [印尼文](../id/README.md) | [馬來文](../ms/README.md) | [他加祿文 (菲律賓)](../tl/README.md) | [斯瓦希里文](../sw/README.md) | [匈牙利文](../hu/README.md) | [捷克文](../cs/README.md) | [斯洛伐克文](../sk/README.md) | [羅馬尼亞文](../ro/README.md) | [保加利亞文](../bg/README.md) | [塞爾維亞文 (西里爾文)](../sr/README.md) | [克羅埃西亞文](../hr/README.md) | [斯洛維尼亞文](../sl/README.md) | [烏克蘭文](../uk/README.md) | [緬甸文 (緬甸)](../my/README.md)
|
||||
|
||||
**如果您希望支援其他語言，請參考 [此處](https://github.com/Azure/co-op-translator/blob/main/getting_started/supported-languages.md)**
|
||||
|
||||
## 加入社群
|
||||
[](https://discord.gg/kzRShWzttr)
|
||||
|
||||
## 您將學到什麼
|
||||
|
||||
|
|
@ -35,21 +46,21 @@ CO_OP_TRANSLATOR_METADATA:
|
|||
|
||||
在這門課程中，您將學到：
|
||||
|
||||
* 不同的人工智慧方法，包括「經典」的符號方法，透過 **知識表示** 和推理 ([GOFAI](https://en.wikipedia.org/wiki/Symbolic_artificial_intelligence))。
|
||||
* 不同的人工智慧方法，包括「經典」符號方法，透過 **知識表示** 和推理 ([GOFAI](https://en.wikipedia.org/wiki/Symbolic_artificial_intelligence))。
|
||||
* **神經網路** 和 **深度學習**，這是現代人工智慧的核心。我們將使用兩個最受歡迎的框架 [TensorFlow](http://Tensorflow.org) 和 [PyTorch](http://pytorch.org) 的程式碼來說明這些重要主題的概念。
|
||||
* **神經架構**，用於處理影像和文字。我們將涵蓋近期的模型，但可能不包括最新的技術。
|
||||
* 不太常見的人工智慧方法，例如 **遺傳演算法** 和 **多代理系統**。
|
||||
* **神經架構**，用於處理影像和文字。我們將涵蓋近期的模型，但可能稍微缺乏最前沿的技術。
|
||||
* 不太流行的人工智慧方法，例如 **遺傳演算法** 和 **多代理系統**。
|
||||
|
||||
我們不會在這門課程中涵蓋：
|
||||
|
||||
> [在 Microsoft Learn 的課程資源集合中找到所有額外資源](https://learn.microsoft.com/en-us/collections/7w28iy2xrqzdj0?WT.mc_id=academic-77998-bethanycheum)
|
||||
> [在 Microsoft Learn 的課程集合中找到所有額外資源](https://learn.microsoft.com/en-us/collections/7w28iy2xrqzdj0?WT.mc_id=academic-77998-bethanycheum)
|
||||
|
||||
* **人工智慧在商業中的應用案例**。建議參加 [商業用戶的人工智慧入門](https://docs.microsoft.com/learn/paths/introduction-ai-for-business-users/?WT.mc_id=academic-77998-bethanycheum) 學習路徑，或由 [INSEAD](https://www.insead.edu/) 合作開發的 [AI 商業學院](https://www.microsoft.com/ai/ai-business-school/?WT.mc_id=academic-77998-bethanycheum)。
|
||||
* **經典機器學習**，這部分已在我們的 [機器學習入門課程](http://github.com/Microsoft/ML-for-Beginners) 中詳細描述。
|
||||
* 使用 **[認知服務](https://azure.microsoft.com/services/cognitive-services/?WT.mc_id=academic-77998-bethanycheum)** 建立的人工智慧應用。建議從 Microsoft Learn 的模組開始，例如 [影像](https://docs.microsoft.com/learn/paths/create-computer-vision-solutions-azure-cognitive-services/?WT.mc_id=academic-77998-bethanycheum)、[自然語言處理](https://docs.microsoft.com/learn/paths/explore-natural-language-processing/?WT.mc_id=academic-77998-bethanycheum)、**[Azure OpenAI Service 的生成式人工智慧](https://learn.microsoft.com/en-us/training/paths/develop-ai-solutions-azure-openai/?WT.mc_id=academic-77998-bethanycheum)** 等。
|
||||
* **人工智慧在商業中的應用案例**。建議參加 [商業使用者的人工智慧入門](https://docs.microsoft.com/learn/paths/introduction-ai-for-business-users/?WT.mc_id=academic-77998-bethanycheum) 學習路徑，或由 [INSEAD](https://www.insead.edu/) 合作開發的 [AI 商業學院](https://www.microsoft.com/ai/ai-business-school/?WT.mc_id=academic-77998-bethanycheum)。
|
||||
* **經典機器學習**，這在我們的 [機器學習入門課程](http://github.com/Microsoft/ML-for-Beginners) 中有詳細描述。
|
||||
* 使用 **[認知服務](https://azure.microsoft.com/services/cognitive-services/?WT.mc_id=academic-77998-bethanycheum)** 建立的實際人工智慧應用。建議從 Microsoft Learn 的模組開始，例如 [影像](https://docs.microsoft.com/learn/paths/create-computer-vision-solutions-azure-cognitive-services/?WT.mc_id=academic-77998-bethanycheum)、[自然語言處理](https://docs.microsoft.com/learn/paths/explore-natural-language-processing/?WT.mc_id=academic-77998-bethanycheum)、**[Azure OpenAI Service 的生成式人工智慧](https://learn.microsoft.com/en-us/training/paths/develop-ai-solutions-azure-openai/?WT.mc_id=academic-77998-bethanycheum)** 等。
|
||||
* 特定的機器學習 **雲端框架**，例如 [Azure Machine Learning](https://azure.microsoft.com/services/machine-learning/?WT.mc_id=academic-77998-bethanycheum)、[Microsoft Fabric](https://learn.microsoft.com/en-us/training/paths/get-started-fabric/?WT.mc_id=academic-77998-bethanycheum) 或 [Azure Databricks](https://docs.microsoft.com/learn/paths/data-engineer-azure-databricks?WT.mc_id=academic-77998-bethanycheum)。建議參加 [使用 Azure Machine Learning 建立和操作機器學習解決方案](https://docs.microsoft.com/learn/paths/build-ai-solutions-with-azure-ml-service/?WT.mc_id=academic-77998-bethanycheum) 和 [使用 Azure Databricks 建立和操作機器學習解決方案](https://docs.microsoft.com/learn/paths/build-operate-machine-learning-solutions-azure-databricks/?WT.mc_id=academic-77998-bethanycheum) 學習路徑。
|
||||
* **對話式人工智慧** 和 **聊天機器人**。有一個單獨的 [建立對話式人工智慧解決方案](https://docs.microsoft.com/learn/paths/create-conversational-ai-solutions/?WT.mc_id=academic-77998-bethanycheum) 學習路徑，您也可以參考 [這篇部落格文章](https://soshnikov.com/azure/hello-bot-conversational-ai-on-microsoft-platform/) 了解更多細節。
|
||||
* **深度學習的數學基礎**。建議參考 Ian Goodfellow、Yoshua Bengio 和 Aaron Courville 所著的 [Deep Learning](https://www.amazon.com/Deep-Learning-Adaptive-Computation-Machine/dp/0262035618)，也可在線上取得：[https://www.deeplearningbook.org/](https://www.deeplearningbook.org/)。
|
||||
* **深度學習的數學基礎**。建議參考 Ian Goodfellow、Yoshua Bengio 和 Aaron Courville 的 [深度學習](https://www.amazon.com/Deep-Learning-Adaptive-Computation-Machine/dp/0262035618)，也可在線上取得 [https://www.deeplearningbook.org/](https://www.deeplearningbook.org/)。
|
||||
|
||||
若想輕鬆入門 _雲端人工智慧_ 主題，建議參加 [在 Azure 上開始人工智慧學習](https://docs.microsoft.com/learn/paths/get-started-with-artificial-intelligence-on-azure/?WT.mc_id=academic-77998-bethanycheum) 學習路徑。
|
||||
|
||||
|
|
@ -61,57 +72,56 @@ CO_OP_TRANSLATOR_METADATA:
|
|||
| I | [**人工智慧簡介**](./lessons/1-Intro/README.md) | | |
|
||||
| 01 | [人工智慧的簡介與歷史](./lessons/1-Intro/README.md) | - | - |
|
||||
| II | **符號人工智慧** |
|
||||
| 02 | [知識表示與專家系統](./lessons/2-Symbolic/README.md) | [專家系統](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/2-Symbolic/Animals.ipynb) / [本體論](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/2-Symbolic/FamilyOntology.ipynb) /[概念圖](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/2-Symbolic/MSConceptGraph.ipynb) | |
|
||||
| 02 | [知識表示與專家系統](./lessons/2-Symbolic/README.md) | [專家系統](./lessons/2-Symbolic/Animals.ipynb) / [本體論](./lessons/2-Symbolic/FamilyOntology.ipynb) /[概念圖](./lessons/2-Symbolic/MSConceptGraph.ipynb) | |
|
||||
| III | [**神經網路簡介**](./lessons/3-NeuralNetworks/README.md) |||
|
||||
| 03 | [感知器](./lessons/3-NeuralNetworks/03-Perceptron/README.md) | [筆記本](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/3-NeuralNetworks/03-Perceptron/Perceptron.ipynb) | [實驗室](./lessons/3-NeuralNetworks/03-Perceptron/lab/README.md) |
|
||||
| 04 | [多層感知器與建立我們自己的框架](./lessons/3-NeuralNetworks/04-OwnFramework/README.md) | [筆記本](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/3-NeuralNetworks/04-OwnFramework/OwnFramework.ipynb) | [實驗室](./lessons/3-NeuralNetworks/04-OwnFramework/lab/README.md) |
|
||||
| 05 | [框架簡介 (PyTorch/TensorFlow) 與過度擬合](./lessons/3-NeuralNetworks/05-Frameworks/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/3-NeuralNetworks/05-Frameworks/IntroPyTorch.ipynb) / [Keras](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/3-NeuralNetworks/05-Frameworks/IntroKeras.ipynb) / [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/3-NeuralNetworks/05-Frameworks/IntroKerasTF.ipynb) | [實驗室](./lessons/3-NeuralNetworks/05-Frameworks/lab/README.md) |
|
||||
| IV | [**電腦視覺**](./lessons/4-ComputerVision/README.md) | [PyTorch](https://docs.microsoft.com/learn/modules/intro-computer-vision-pytorch/?WT.mc_id=academic-77998-cacaste) / [TensorFlow](https://docs.microsoft.com/learn/modules/intro-computer-vision-TensorFlow/?WT.mc_id=academic-77998-cacaste)| [探索 Microsoft Azure 上的電腦視覺](https://learn.microsoft.com/en-us/collections/7w28iy2xrqzdj0?WT.mc_id=academic-77998-bethanycheum) |
|
||||
| 06 | [電腦視覺簡介. OpenCV](./lessons/4-ComputerVision/06-IntroCV/README.md) | [筆記本](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/06-IntroCV/OpenCV.ipynb) | [實驗室](./lessons/4-ComputerVision/06-IntroCV/lab/README.md) |
|
||||
| 07 | [卷積神經網路](./lessons/4-ComputerVision/07-ConvNets/README.md) & [CNN 架構](./lessons/4-ComputerVision/07-ConvNets/CNN_Architectures.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/07-ConvNets/ConvNetsPyTorch.ipynb) /[TensorFlow](https://microsoft.github.io/AI-For-Beginners/lessons/4-ComputerVision/07-ConvNets/ConvNetsTF.ipynb) | [實驗室](./lessons/4-ComputerVision/07-ConvNets/lab/README.md) |
|
||||
| 08 | [預訓練網絡與遷移學習](./lessons/4-ComputerVision/08-TransferLearning/README.md) 和 [訓練技巧](./lessons/4-ComputerVision/08-TransferLearning/TrainingTricks.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/08-TransferLearning/TransferLearningPyTorch.ipynb) / [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/3-NeuralNetworks/05-Frameworks/IntroKerasTF.ipynb) | [實驗](./lessons/4-ComputerVision/08-TransferLearning/lab/README.md) |
|
||||
| 09 | [自編碼器與變分自編碼器 (VAEs)](./lessons/4-ComputerVision/09-Autoencoders/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/09-Autoencoders/AutoEncodersPyTorch.ipynb) / [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/09-Autoencoders/AutoencodersTF.ipynb) | |
|
||||
| 10 | [生成對抗網絡 (GANs) 與藝術風格遷移](./lessons/4-ComputerVision/10-GANs/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/10-GANs/GANPyTorch.ipynb) / [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/10-GANs/GANTF.ipynb) | |
|
||||
| 11 | [物體檢測](./lessons/4-ComputerVision/11-ObjectDetection/README.md) | [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/11-ObjectDetection/ObjectDetection.ipynb) | [實驗](./lessons/4-ComputerVision/11-ObjectDetection/lab/README.md) |
|
||||
| 12 | [語義分割. U-Net](./lessons/4-ComputerVision/12-Segmentation/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/12-Segmentation/SemanticSegmentationPytorch.ipynb) / [TensorFlow](../../(https:/github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/12-Segmentation/SemanticSegmentationTF.ipynb)) | |
|
||||
| V | [**自然語言處理**](./lessons/5-NLP/README.md) | [PyTorch](https://docs.microsoft.com/learn/modules/intro-natural-language-processing-pytorch/?WT.mc_id=academic-77998-cacaste) /[TensorFlow](https://docs.microsoft.com/learn/modules/intro-natural-language-processing-TensorFlow/?WT.mc_id=academic-77998-cacaste) | [在 Microsoft Azure 上探索自然語言處理](https://learn.microsoft.com/en-us/collections/7w28iy2xrqzdj0?WT.mc_id=academic-77998-bethanycheum)|
|
||||
| 13 | [文本表示. Bow/TF-IDF](./lessons/5-NLP/13-TextRep/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/13-TextRep/TextRepresentationPyTorch.ipynb) / [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/13-TextRep/TextRepresentationTF.ipynb) | |
|
||||
| 14 | [語義詞嵌入. Word2Vec 和 GloVe](./lessons/5-NLP/14-Embeddings/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb) / [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/14-Embeddings/EmbeddingsTF.ipynb) | |
|
||||
| 15 | [語言建模. 訓練自己的嵌入](./lessons/5-NLP/15-LanguageModeling/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/15-LanguageModeling/CBoW-PyTorch.ipynb) / [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/15-LanguageModeling/CBoW-TF.ipynb) | [實驗](./lessons/5-NLP/15-LanguageModeling/lab/README.md) |
|
||||
| 16 | [循環神經網絡 (RNN)](./lessons/5-NLP/16-RNN/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/16-RNN/RNNPyTorch.ipynb) / [TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/16-RNN/RNNTF.ipynb) | |
|
||||
| 17 | [生成式循環網絡](./lessons/5-NLP/17-GenerativeNetworks/README.md) | [PyTorch](https://microsoft.github.io/AI-For-Beginners/lessons/5-NLP/17-GenerativeNetworks/GenerativePyTorch.md) / [TensorFlow](https://microsoft.github.io/AI-For-Beginners/lessons/5-NLP/17-GenerativeNetworks/GenerativeTF.md) | [實驗](./lessons/5-NLP/17-GenerativeNetworks/lab/README.md) |
|
||||
| 18 | [Transformer. BERT.](./lessons/5-NLP/18-Transformers/READMEtransformers.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/18-Transformers/TransformersPyTorch.ipynb) /[TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/18-Transformers/TransformersTF.ipynb) | |
|
||||
| 19 | [命名實體識別 (NER)](./lessons/5-NLP/19-NER/README.md) | [TensorFlow](https://microsoft.github.io/AI-For-Beginners/lessons/5-NLP/19-NER/NER-TF.ipynb) | [實驗](./lessons/5-NLP/19-NER/lab/README.md) |
|
||||
| 20 | [大型語言模型、提示編程與少樣本任務](./lessons/5-NLP/20-LangModels/READMELargeLang.md) | [PyTorch](https://microsoft.github.io/AI-For-Beginners/lessons/5-NLP/20-LangModels/GPT-PyTorch.ipynb) | |
|
||||
| 03 | [感知器](./lessons/3-NeuralNetworks/03-Perceptron/README.md) | [筆記本](./lessons/3-NeuralNetworks/03-Perceptron/Perceptron.ipynb) | [實驗室](./lessons/3-NeuralNetworks/03-Perceptron/lab/README.md) |
|
||||
| 04 | [多層感知器與建立我們自己的框架](./lessons/3-NeuralNetworks/04-OwnFramework/README.md) | [筆記本](./lessons/3-NeuralNetworks/04-OwnFramework/OwnFramework.ipynb) | [實驗室](./lessons/3-NeuralNetworks/04-OwnFramework/lab/README.md) |
|
||||
| 05 | [框架介紹（PyTorch/TensorFlow）與過度擬合](./lessons/3-NeuralNetworks/05-Frameworks/README.md) | [PyTorch](./lessons/3-NeuralNetworks/05-Frameworks/IntroPyTorch.ipynb) / [Keras](./lessons/3-NeuralNetworks/05-Frameworks/IntroKeras.ipynb) / [TensorFlow](./lessons/3-NeuralNetworks/05-Frameworks/IntroKerasTF.ipynb) | [實驗](./lessons/3-NeuralNetworks/05-Frameworks/lab/README.md) |
|
||||
| IV | [**電腦視覺**](./lessons/4-ComputerVision/README.md) | [PyTorch](https://docs.microsoft.com/learn/modules/intro-computer-vision-pytorch/?WT.mc_id=academic-77998-cacaste) / [TensorFlow](https://docs.microsoft.com/learn/modules/intro-computer-vision-TensorFlow/?WT.mc_id=academic-77998-cacaste)| [探索 Microsoft Azure 上的電腦視覺](https://learn.microsoft.com/en-us/collections/7w28iy2xrqzdj0?WT.mc_id=academic-77998-bethanycheum) |
|
||||
| 06 | [電腦視覺介紹. OpenCV](./lessons/4-ComputerVision/06-IntroCV/README.md) | [筆記本](./lessons/4-ComputerVision/06-IntroCV/OpenCV.ipynb) | [實驗](./lessons/4-ComputerVision/06-IntroCV/lab/README.md) |
|
||||
| 07 | [卷積神經網絡](./lessons/4-ComputerVision/07-ConvNets/README.md) & [CNN 架構](./lessons/4-ComputerVision/07-ConvNets/CNN_Architectures.md) | [PyTorch](./lessons/4-ComputerVision/07-ConvNets/ConvNetsPyTorch.ipynb) /[TensorFlow](./lessons/4-ComputerVision/07-ConvNets/ConvNetsTF.ipynb) | [實驗](./lessons/4-ComputerVision/07-ConvNets/lab/README.md) |
|
||||
| 08 | [預訓練網絡與遷移學習](./lessons/4-ComputerVision/08-TransferLearning/README.md) 和 [訓練技巧](./lessons/4-ComputerVision/08-TransferLearning/TrainingTricks.md) | [PyTorch](./lessons/4-ComputerVision/08-TransferLearning/TransferLearningPyTorch.ipynb) / [TensorFlow](./lessons/3-NeuralNetworks/05-Frameworks/IntroKerasTF.ipynb) | [實驗](./lessons/4-ComputerVision/08-TransferLearning/lab/README.md) |
|
||||
| 09 | [自編碼器與變分自編碼器（VAEs）](./lessons/4-ComputerVision/09-Autoencoders/README.md) | [PyTorch](./lessons/4-ComputerVision/09-Autoencoders/AutoEncodersPyTorch.ipynb) / [TensorFlow](./lessons/4-ComputerVision/09-Autoencoders/AutoencodersTF.ipynb) | |
|
||||
| 10 | [生成對抗網絡（GANs）與藝術風格遷移](./lessons/4-ComputerVision/10-GANs/README.md) | [PyTorch](./lessons/4-ComputerVision/10-GANs/GANPyTorch.ipynb) / [TensorFlow](./lessons/4-ComputerVision/10-GANs/GANTF.ipynb) | |
|
||||
| 11 | [物件偵測](./lessons/4-ComputerVision/11-ObjectDetection/README.md) | [TensorFlow](./lessons/4-ComputerVision/11-ObjectDetection/ObjectDetection.ipynb) | [實驗](./lessons/4-ComputerVision/11-ObjectDetection/lab/README.md) |
|
||||
| 12 | [語義分割. U-Net](./lessons/4-ComputerVision/12-Segmentation/README.md) | [PyTorch](./lessons/4-ComputerVision/12-Segmentation/SemanticSegmentationPytorch.ipynb) / [TensorFlow](./lessons/4-ComputerVision/12-Segmentation/SemanticSegmentationTF.ipynb) | |
|
||||
| V | [**自然語言處理**](./lessons/5-NLP/README.md) | [PyTorch](https://docs.microsoft.com/learn/modules/intro-natural-language-processing-pytorch/?WT.mc_id=academic-77998-cacaste) /[TensorFlow](https://docs.microsoft.com/learn/modules/intro-natural-language-processing-TensorFlow/?WT.mc_id=academic-77998-cacaste) | [探索 Microsoft Azure 上的自然語言處理](https://learn.microsoft.com/en-us/collections/7w28iy2xrqzdj0?WT.mc_id=academic-77998-bethanycheum)|
|
||||
| 13 | [文本表示. Bow/TF-IDF](./lessons/5-NLP/13-TextRep/README.md) | [PyTorch](./lessons/5-NLP/13-TextRep/TextRepresentationPyTorch.ipynb) / [TensorFlow](./lessons/5-NLP/13-TextRep/TextRepresentationTF.ipynb) | |
|
||||
| 14 | [語義詞嵌入. Word2Vec 和 GloVe](./lessons/5-NLP/14-Embeddings/README.md) | [PyTorch](./lessons/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb) / [TensorFlow](./lessons/5-NLP/14-Embeddings/EmbeddingsTF.ipynb) | |
|
||||
| 15 | [語言建模. 訓練自己的嵌入](./lessons/5-NLP/15-LanguageModeling/README.md) | [PyTorch](./lessons/5-NLP/15-LanguageModeling/CBoW-PyTorch.ipynb) / [TensorFlow](./lessons/5-NLP/15-LanguageModeling/CBoW-TF.ipynb) | [實驗](./lessons/5-NLP/15-LanguageModeling/lab/README.md) |
|
||||
| 16 | [循環神經網絡（RNNs）](./lessons/5-NLP/16-RNN/README.md) | [PyTorch](./lessons/5-NLP/16-RNN/RNNPyTorch.ipynb) / [TensorFlow](./lessons/5-NLP/16-RNN/RNNTF.ipynb) | |
|
||||
| 17 | [生成式循環網絡](./lessons/5-NLP/17-GenerativeNetworks/README.md) | [PyTorch](./lessons/5-NLP/17-GenerativeNetworks/GenerativePyTorch.md) / [TensorFlow](./lessons/5-NLP/17-GenerativeNetworks/GenerativeTF.md) | [實驗](./lessons/5-NLP/17-GenerativeNetworks/lab/README.md) |
|
||||
| 18 | [Transformer. BERT.](./lessons/5-NLP/18-Transformers/READMEtransformers.md) | [PyTorch](./lessons/5-NLP/18-Transformers/TransformersPyTorch.ipynb) /[TensorFlow](./lessons/5-NLP/18-Transformers/TransformersTF.ipynb) | |
|
||||
| 19 | [命名實體識別（NER）](./lessons/5-NLP/19-NER/README.md) | [TensorFlow](./lessons/5-NLP/19-NER/NER-TF.ipynb) | [實驗](./lessons/5-NLP/19-NER/lab/README.md) |
|
||||
| 20 | [大型語言模型、提示編程與少樣本任務](./lessons/5-NLP/20-LangModels/READMELargeLang.md) | [PyTorch](./lessons/5-NLP/20-LangModels/GPT-PyTorch.ipynb) | |
|
||||
| VI | **其他 AI 技術** || |
|
||||
| 21 | [遺傳算法](./lessons/6-Other/21-GeneticAlgorithms/README.md) | [Notebook](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/6-Other/21-GeneticAlgorithms/Genetic.ipynb) | |
|
||||
| 22 | [深度強化學習](./lessons/6-Other/22-DeepRL/README.md) | [PyTorch](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/6-Other/22-DeepRL/CartPole-RL-PyTorch.ipynb) /[TensorFlow](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/6-Other/22-DeepRL/CartPole-RL-TF.ipynb) | [實驗](./lessons/6-Other/22-DeepRL/lab/README.md) |
|
||||
| 23 | [多智能體系統](./lessons/6-Other/23-MultiagentSystems/README.md) | | |
|
||||
| 21 | [遺傳算法](./lessons/6-Other/21-GeneticAlgorithms/README.md) | [筆記本](./lessons/6-Other/21-GeneticAlgorithms/Genetic.ipynb) | |
|
||||
| 22 | [深度強化學習](./lessons/6-Other/22-DeepRL/README.md) | [PyTorch](./lessons/6-Other/22-DeepRL/CartPole-RL-PyTorch.ipynb) /[TensorFlow](./lessons/6-Other/22-DeepRL/CartPole-RL-TF.ipynb) | [實驗](./lessons/6-Other/22-DeepRL/lab/README.md) |
|
||||
| 23 | [多代理系統](./lessons/6-Other/23-MultiagentSystems/README.md) | | |
|
||||
| VII | **AI 倫理** | | |
|
||||
| 24 | [AI 倫理與負責任的 AI](./lessons/7-Ethics/README.md) | [Microsoft Learn: 負責任的 AI 原則](https://docs.microsoft.com/learn/paths/responsible-ai-business-principles/?WT.mc_id=academic-77998-cacaste) | |
|
||||
| IX | **附加內容** | | |
|
||||
| 25 | [多模態網絡、CLIP 和 VQGAN](./lessons/X-Extras/X1-MultiModal/README.md) | [Notebook](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/X-Extras/X1-MultiModal/Clip.ipynb) | |
|
||||
| IX | **額外內容** | | |
|
||||
| 25 | [多模態網絡、CLIP 和 VQGAN](./lessons/X-Extras/X1-MultiModal/README.md) | [筆記本](./lessons/X-Extras/X1-MultiModal/Clip.ipynb) | |
|
||||
|
||||
## 每節課包含
|
||||
## 每個課程包含
|
||||
|
||||
* 預讀材料
|
||||
* 可執行的 Jupyter Notebook，通常針對特定框架（**PyTorch** 或 **TensorFlow**）。這些 Notebook 包含大量理論內容，因此為了理解主題，您需要至少閱讀一個版本的 Notebook（PyTorch 或 TensorFlow）。
|
||||
* **實驗**（部分主題提供），讓您有機會將所學應用於具體問題。
|
||||
* 某些章節包含指向 [**MS Learn**](https://learn.microsoft.com/en-us/collections/7w28iy2xrqzdj0?WT.mc_id=academic-77998-bethanycheum) 模組的鏈接，涵蓋相關主題。
|
||||
* 可執行的 Jupyter 筆記本，通常針對特定框架（**PyTorch** 或 **TensorFlow**）。可執行的筆記本還包含大量理論材料，因此要理解主題，您需要至少完成一個版本的筆記本（PyTorch 或 TensorFlow）。
|
||||
* **實驗**（Labs）可用於某些主題，讓您有機會將所學材料應用於特定問題。
|
||||
* 某些章節包含指向 [**MS Learn**](https://learn.microsoft.com/en-us/collections/7w28iy2xrqzdj0?WT.mc_id=academic-77998-bethanycheum) 模組的連結，涵蓋相關主題。
|
||||
|
||||
## 開始學習
|
||||
|
||||
- 我們已經創建了一個 [設置課程](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/0-course-setup/setup.md)，幫助您設置開發環境。
|
||||
- 對於教育者，我們還創建了一個 [課程設置指南](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/0-course-setup/for-teachers.md)！
|
||||
- 如何 [在 VSCode 或 Codepace 中運行代碼](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/0-course-setup/how-to-run.md)
|
||||
- 我們已創建一個 [設置課程](./lessons/0-course-setup/setup.md)，幫助您設置開發環境。 - 對於教育工作者，我們也創建了一個 [課程設置課程](./lessons/0-course-setup/for-teachers.md)！
|
||||
- 如何 [在 VSCode 或 Codepace 中運行代碼](./lessons/0-course-setup/how-to-run.md)
|
||||
|
||||
請按照以下步驟操作：
|
||||
請按照以下步驟：
|
||||
|
||||
Fork 此倉庫：點擊頁面右上角的 "Fork" 按鈕。
|
||||
Fork 此倉庫：點擊此頁面右上角的 "Fork" 按鈕。
|
||||
|
||||
克隆倉庫：`git clone https://github.com/microsoft/AI-For-Beginners.git`
|
||||
|
||||
別忘了給這個倉庫加星標 (🌟)，以便日後更容易找到。
|
||||
別忘了給此倉庫加星（🌟），以便日後更容易找到。
|
||||
|
||||
## 與其他學習者交流
|
||||
|
||||
|
|
@ -120,8 +130,8 @@ Fork 此倉庫：點擊頁面右上角的 "Fork" 按鈕。
|
|||
如果您在構建過程中有產品反饋或問題，請訪問我們的 [Azure AI Foundry 開發者論壇](https://aka.ms/foundry/forum)
|
||||
|
||||
## 測驗
|
||||
> **關於測驗的注意事項**：所有測驗都存放在 etc\quiz-app 的 Quiz-app 資料夾中，並從課程中進行連結。測驗應用程式可以在本地執行或部署到 Azure；請按照 `quiz-app` 資料夾中的指示進行操作。測驗正在逐步進行本地化。
|
||||
## 幫助徵求
|
||||
> **關於測驗的注意事項**：所有測驗都包含在 Quiz-app 資料夾中，位於 etc\quiz-app，或可在[線上查看](https://ff-quizzes.netlify.app/)。它們已在課程中連結，測驗應用程式可以在本地執行或部署到 Azure；請按照 `quiz-app` 資料夾中的指示操作。測驗正在逐步進行本地化。
|
||||
## 幫助需求
|
||||
|
||||
您有任何建議或發現拼寫或程式碼錯誤嗎？請提交一個問題或建立一個拉取請求。
|
||||
|
||||
|
|
@ -152,5 +162,7 @@ Fork 此倉庫：點擊頁面右上角的 "Fork" 按鈕。
|
|||
- [精通 GitHub Copilot 的 C#/.NET 開發者課程](https://github.com/microsoft/mastering-github-copilot-for-dotnet-csharp-developers)
|
||||
- [選擇你的 Copilot 冒險](https://github.com/microsoft/CopilotAdventures)
|
||||
|
||||
---
|
||||
|
||||
**免責聲明**：
|
||||
本文檔使用 AI 翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。儘管我們努力確保準確性，但請注意，自動翻譯可能包含錯誤或不準確之處。原始語言的文件應被視為權威來源。對於關鍵信息，建議使用專業人工翻譯。我們對因使用此翻譯而引起的任何誤解或錯誤解釋不承擔責任。
|
||||
本文件使用 AI 翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。我們致力於提供準確的翻譯，但請注意，自動翻譯可能包含錯誤或不準確之處。應以原始語言的文件作為權威來源。對於關鍵資訊，建議尋求專業人工翻譯。我們對因使用此翻譯而引起的任何誤解或錯誤解讀概不負責。
|
||||
|
|
@ -0,0 +1,478 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"collapsed": true
|
||||
},
|
||||
"source": [
|
||||
"# 實作動物專家系統\n",
|
||||
"\n",
|
||||
"範例取自 [AI for Beginners Curriculum](http://github.com/microsoft/ai-for-beginners)。\n",
|
||||
"\n",
|
||||
"在這個範例中，我們將實作一個簡單的知識型系統，根據一些外觀特徵來判斷動物。此系統可以用以下的 AND-OR 樹來表示（這只是整個樹的一部分，我們可以輕鬆地添加更多規則）：\n",
|
||||
"\n",
|
||||
"\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 我們自己的專家系統殼層與反向推理\n",
|
||||
"\n",
|
||||
"讓我們嘗試基於生成規則定義一種簡單的知識表示語言。我們將使用 Python 類作為關鍵字來定義規則。基本上會有三種類型的類別：\n",
|
||||
"* `Ask` 代表需要向使用者提問的問題。它包含可能的答案集合。\n",
|
||||
"* `If` 代表一條規則，它只是用來存儲規則內容的語法糖。\n",
|
||||
"* `AND`/`OR` 是用來表示樹的 AND/OR 分支的類別。它們僅存儲內部的參數列表。為了簡化程式碼，所有功能都定義在父類別 `Content` 中。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class Ask():\n",
|
||||
" def __init__(self,choices=['y','n']):\n",
|
||||
" self.choices = choices\n",
|
||||
" def ask(self):\n",
|
||||
" if max([len(x) for x in self.choices])>1:\n",
|
||||
" for i,x in enumerate(self.choices):\n",
|
||||
" print(\"{0}. {1}\".format(i,x),flush=True)\n",
|
||||
" x = int(input())\n",
|
||||
" return self.choices[x]\n",
|
||||
" else:\n",
|
||||
" print(\"/\".join(self.choices),flush=True)\n",
|
||||
" return input()\n",
|
||||
"\n",
|
||||
"class Content():\n",
|
||||
" def __init__(self,x):\n",
|
||||
" self.x=x\n",
|
||||
" \n",
|
||||
"class If(Content):\n",
|
||||
" pass\n",
|
||||
"\n",
|
||||
"class AND(Content):\n",
|
||||
" pass\n",
|
||||
"\n",
|
||||
"class OR(Content):\n",
|
||||
" pass"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"在我們的系統中，工作記憶將包含**事實**列表作為**屬性-值對**。知識庫可以定義為一個大的字典，將行動（應插入到工作記憶中的新事實）映射到條件，表達為AND-OR表達式。此外，一些事實可以被`詢問`。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"rules = {\n",
|
||||
" 'default': Ask(['y','n']),\n",
|
||||
" 'color' : Ask(['red-brown','black and white','other']),\n",
|
||||
" 'pattern' : Ask(['dark stripes','dark spots']),\n",
|
||||
" 'mammal': If(OR(['hair','gives milk'])),\n",
|
||||
" 'carnivor': If(OR([AND(['sharp teeth','claws','forward-looking eyes']),'eats meat'])),\n",
|
||||
" 'ungulate': If(['mammal',OR(['has hooves','chews cud'])]),\n",
|
||||
" 'bird': If(OR(['feathers',AND(['flies','lies eggs'])])),\n",
|
||||
" 'animal:monkey' : If(['mammal','carnivor','color:red-brown','pattern:dark spots']),\n",
|
||||
" 'animal:tiger' : If(['mammal','carnivor','color:red-brown','pattern:dark stripes']),\n",
|
||||
" 'animal:giraffe' : If(['ungulate','long neck','long legs','pattern:dark spots']),\n",
|
||||
" 'animal:zebra' : If(['ungulate','pattern:dark stripes']),\n",
|
||||
" 'animal:ostrich' : If(['bird','long nech','color:black and white','cannot fly']),\n",
|
||||
" 'animal:pinguin' : If(['bird','swims','color:black and white','cannot fly']),\n",
|
||||
" 'animal:albatross' : If(['bird','flies well'])\n",
|
||||
"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"為了執行反向推理，我們將定義一個 `Knowledgebase` 類別。這個類別將包含以下內容：\n",
|
||||
"* 運作中的 `memory` - 一個字典，用於將屬性映射到對應的值\n",
|
||||
"* 知識庫中的 `rules` - 以上述定義的格式表示的規則\n",
|
||||
"\n",
|
||||
"主要的兩個方法是：\n",
|
||||
"* `get` 方法，用於獲取屬性的值，必要時執行推理。例如，`get('color')` 將獲取顏色槽位的值（如果需要，會詢問並將值存儲到運作記憶中以供後續使用）。如果我們詢問 `get('color:blue')`，它會詢問顏色，然後根據顏色返回 `y`/`n` 的值。\n",
|
||||
"* `eval` 方法執行實際的推理，即遍歷 AND/OR 樹，評估子目標等。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class KnowledgeBase():\n",
|
||||
" def __init__(self,rules):\n",
|
||||
" self.rules = rules\n",
|
||||
" self.memory = {}\n",
|
||||
" \n",
|
||||
" def get(self,name):\n",
|
||||
" if ':' in name:\n",
|
||||
" k,v = name.split(':')\n",
|
||||
" vv = self.get(k)\n",
|
||||
" return 'y' if v==vv else 'n'\n",
|
||||
" if name in self.memory.keys():\n",
|
||||
" return self.memory[name]\n",
|
||||
" for fld in self.rules.keys():\n",
|
||||
" if fld==name or fld.startswith(name+\":\"):\n",
|
||||
" # print(\" + proving {}\".format(fld))\n",
|
||||
" value = 'y' if fld==name else fld.split(':')[1]\n",
|
||||
" res = self.eval(self.rules[fld],field=name)\n",
|
||||
" if res!='y' and res!='n' and value=='y':\n",
|
||||
" self.memory[name] = res\n",
|
||||
" return res\n",
|
||||
" if res=='y':\n",
|
||||
" self.memory[name] = value\n",
|
||||
" return value\n",
|
||||
" # field is not found, using default\n",
|
||||
" res = self.eval(self.rules['default'],field=name)\n",
|
||||
" self.memory[name]=res\n",
|
||||
" return res\n",
|
||||
" \n",
|
||||
" def eval(self,expr,field=None):\n",
|
||||
" # print(\" + eval {}\".format(expr))\n",
|
||||
" if isinstance(expr,Ask):\n",
|
||||
" print(field)\n",
|
||||
" return expr.ask()\n",
|
||||
" elif isinstance(expr,If):\n",
|
||||
" return self.eval(expr.x)\n",
|
||||
" elif isinstance(expr,AND) or isinstance(expr,list):\n",
|
||||
" expr = expr.x if isinstance(expr,AND) else expr\n",
|
||||
" for x in expr:\n",
|
||||
" if self.eval(x)=='n':\n",
|
||||
" return 'n'\n",
|
||||
" return 'y'\n",
|
||||
" elif isinstance(expr,OR):\n",
|
||||
" for x in expr.x:\n",
|
||||
" if self.eval(x)=='y':\n",
|
||||
" return 'y'\n",
|
||||
" return 'n'\n",
|
||||
" elif isinstance(expr,str):\n",
|
||||
" return self.get(expr)\n",
|
||||
" else:\n",
|
||||
" print(\"Unknown expr: {}\".format(expr))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"現在讓我們定義我們的動物知識庫並進行諮詢。請注意，此操作將向您提問。您可以通過輸入 `y`/`n` 來回答是非問題，或者通過指定數字（0..N）來回答具有較多選擇的問題。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"hair\n",
|
||||
"y/n\n",
|
||||
"sharp teeth\n",
|
||||
"y/n\n",
|
||||
"claws\n",
|
||||
"y/n\n",
|
||||
"forward-looking eyes\n",
|
||||
"y/n\n",
|
||||
"color\n",
|
||||
"0. red-brown\n",
|
||||
"1. black and white\n",
|
||||
"2. other\n",
|
||||
"has hooves\n",
|
||||
"y/n\n",
|
||||
"long neck\n",
|
||||
"y/n\n",
|
||||
"long legs\n",
|
||||
"y/n\n",
|
||||
"pattern\n",
|
||||
"0. dark stripes\n",
|
||||
"1. dark spots\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"'giraffe'"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"kb = KnowledgeBase(rules)\n",
|
||||
"kb.get('animal')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 使用 PyKnow 進行前向推理\n",
|
||||
"\n",
|
||||
"在下一個範例中，我們將嘗試使用一個知識表示庫 [PyKnow](https://github.com/buguroo/pyknow/) 來實現前向推理。**PyKnow** 是一個用於在 Python 中建立前向推理系統的庫，其設計理念與經典的舊系統 [CLIPS](http://www.clipsrules.net/index.html) 相似。\n",
|
||||
"\n",
|
||||
"我們也可以自己實現前向鏈推理，並不會遇到太多問題，但簡單的實現通常效率不高。為了更有效地進行規則匹配，會使用一種特殊的演算法 [Rete](https://en.wikipedia.org/wiki/Rete_algorithm)。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Collecting git+https://github.com/buguroo/pyknow/\n",
|
||||
" Cloning https://github.com/buguroo/pyknow/ to /tmp/pip-req-build-3cqeulyl\n",
|
||||
" Running command git clone --filter=blob:none --quiet https://github.com/buguroo/pyknow/ /tmp/pip-req-build-3cqeulyl\n",
|
||||
" Resolved https://github.com/buguroo/pyknow/ to commit 48818336f2e9a126f1964f2d8dc22d37ff800fe8\n",
|
||||
" Preparing metadata (setup.py) ... \u001b[?25ldone\n",
|
||||
"\u001b[?25hCollecting frozendict==1.2\n",
|
||||
" Using cached frozendict-1.2.tar.gz (2.6 kB)\n",
|
||||
" Preparing metadata (setup.py) ... \u001b[?25ldone\n",
|
||||
"\u001b[?25hCollecting schema==0.6.7\n",
|
||||
" Using cached schema-0.6.7-py2.py3-none-any.whl (14 kB)\n",
|
||||
"Building wheels for collected packages: pyknow, frozendict\n",
|
||||
" Building wheel for pyknow (setup.py) ... \u001b[?25ldone\n",
|
||||
"\u001b[?25h Created wheel for pyknow: filename=pyknow-1.7.0-py3-none-any.whl size=34228 sha256=b7de5b09292c4007667c72f69b98d5a1b5f7324ff15f9dd8e077c3d5f7aade42\n",
|
||||
" Stored in directory: /tmp/pip-ephem-wheel-cache-k7jpave7/wheels/81/1a/d3/f6c15dbe1955598a37755215f2a10449e7418500d7bd4b9508\n",
|
||||
" Building wheel for frozendict (setup.py) ... \u001b[?25ldone\n",
|
||||
"\u001b[?25h Created wheel for frozendict: filename=frozendict-1.2-py3-none-any.whl size=3148 sha256=2863d55c240d2409cddf05ccfe600591f8478681549fc97555c47c90dc6bb160\n",
|
||||
" Stored in directory: /home/rg/.cache/pip/wheels/49/ac/f8/cb8120244e710bdb479c86198b03c7b08c3c2d3d2bf448fd6e\n",
|
||||
"Successfully built pyknow frozendict\n",
|
||||
"Installing collected packages: schema, frozendict, pyknow\n",
|
||||
"Successfully installed frozendict-1.2 pyknow-1.7.0 schema-0.6.7\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import sys\n",
|
||||
"!{sys.executable} -m pip install git+https://github.com/buguroo/pyknow/"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from pyknow import *\n",
|
||||
"#import pyknow"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"我們將把系統定義為一個繼承 `KnowledgeEngine` 的類別。每個規則由一個帶有 `@Rule` 註解的獨立函數定義，該註解指定了規則應該觸發的時機。在規則內部，我們可以使用 `declare` 函數添加新的事實，添加這些事實將導致前向推理引擎調用更多的規則。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class Animals(KnowledgeEngine):\n",
|
||||
" @Rule(OR(\n",
|
||||
" AND(Fact('sharp teeth'),Fact('claws'),Fact('forward looking eyes')),\n",
|
||||
" Fact('eats meat')))\n",
|
||||
" def cornivor(self):\n",
|
||||
" self.declare(Fact('carnivor'))\n",
|
||||
" \n",
|
||||
" @Rule(OR(Fact('hair'),Fact('gives milk')))\n",
|
||||
" def mammal(self):\n",
|
||||
" self.declare(Fact('mammal'))\n",
|
||||
"\n",
|
||||
" @Rule(Fact('mammal'),\n",
|
||||
" OR(Fact('has hooves'),Fact('chews cud')))\n",
|
||||
" def hooves(self):\n",
|
||||
" self.declare('ungulate')\n",
|
||||
" \n",
|
||||
" @Rule(OR(Fact('feathers'),AND(Fact('flies'),Fact('lays eggs'))))\n",
|
||||
" def bird(self):\n",
|
||||
" self.declare('bird')\n",
|
||||
" \n",
|
||||
" @Rule(Fact('mammal'),Fact('carnivor'),\n",
|
||||
" Fact(color='red-brown'),\n",
|
||||
" Fact(pattern='dark spots'))\n",
|
||||
" def monkey(self):\n",
|
||||
" self.declare(Fact(animal='monkey'))\n",
|
||||
"\n",
|
||||
" @Rule(Fact('mammal'),Fact('carnivor'),\n",
|
||||
" Fact(color='red-brown'),\n",
|
||||
" Fact(pattern='dark stripes'))\n",
|
||||
" def tiger(self):\n",
|
||||
" self.declare(Fact(animal='tiger'))\n",
|
||||
"\n",
|
||||
" @Rule(Fact('ungulate'),\n",
|
||||
" Fact('long neck'),\n",
|
||||
" Fact('long legs'),\n",
|
||||
" Fact(pattern='dark spots'))\n",
|
||||
" def giraffe(self):\n",
|
||||
" self.declare(Fact(animal='giraffe'))\n",
|
||||
"\n",
|
||||
" @Rule(Fact('ungulate'),\n",
|
||||
" Fact(pattern='dark stripes'))\n",
|
||||
" def zebra(self):\n",
|
||||
" self.declare(Fact(animal='zebra'))\n",
|
||||
"\n",
|
||||
" @Rule(Fact('bird'),\n",
|
||||
" Fact('long neck'),\n",
|
||||
" Fact('cannot fly'),\n",
|
||||
" Fact(color='black and white'))\n",
|
||||
" def straus(self):\n",
|
||||
" self.declare(Fact(animal='ostrich'))\n",
|
||||
"\n",
|
||||
" @Rule(Fact('bird'),\n",
|
||||
" Fact('swims'),\n",
|
||||
" Fact('cannot fly'),\n",
|
||||
" Fact(color='black and white'))\n",
|
||||
" def pinguin(self):\n",
|
||||
" self.declare(Fact(animal='pinguin'))\n",
|
||||
"\n",
|
||||
" @Rule(Fact('bird'),\n",
|
||||
" Fact('flies well'))\n",
|
||||
" def albatros(self):\n",
|
||||
" self.declare(Fact(animal='albatross'))\n",
|
||||
" \n",
|
||||
" @Rule(Fact(animal=MATCH.a))\n",
|
||||
" def print_result(self,a):\n",
|
||||
" print('Animal is {}'.format(a))\n",
|
||||
" \n",
|
||||
" def factz(self,l):\n",
|
||||
" for x in l:\n",
|
||||
" self.declare(x)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"一旦我們定義了一個知識庫，我們會用一些初始事實填充工作記憶，然後調用 `run()` 方法來執行推理。結果可以看到新的推斷事實被添加到工作記憶中，包括關於動物的最終事實（如果我們正確設置了所有初始事實）。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Animal is tiger\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"FactList([(0, InitialFact()),\n",
|
||||
" (1, Fact(color='red-brown')),\n",
|
||||
" (2, Fact(pattern='dark stripes')),\n",
|
||||
" (3, Fact('sharp teeth')),\n",
|
||||
" (4, Fact('claws')),\n",
|
||||
" (5, Fact('forward looking eyes')),\n",
|
||||
" (6, Fact('gives milk')),\n",
|
||||
" (7, Fact('mammal')),\n",
|
||||
" (8, Fact('carnivor')),\n",
|
||||
" (9, Fact(animal='tiger'))])"
|
||||
]
|
||||
},
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"ex1 = Animals()\n",
|
||||
"ex1.reset()\n",
|
||||
"ex1.factz([\n",
|
||||
" Fact(color='red-brown'),\n",
|
||||
" Fact(pattern='dark stripes'),\n",
|
||||
" Fact('sharp teeth'),\n",
|
||||
" Fact('claws'),\n",
|
||||
" Fact('forward looking eyes'),\n",
|
||||
" Fact('gives milk')])\n",
|
||||
"ex1.run()\n",
|
||||
"ex1.facts"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**免責聲明**： \n本文件已使用 AI 翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。雖然我們努力確保翻譯的準確性，但請注意，自動翻譯可能包含錯誤或不準確之處。原始文件的母語版本應被視為權威來源。對於關鍵信息，建議尋求專業人工翻譯。我們對因使用此翻譯而引起的任何誤解或錯誤解釋不承擔責任。\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.7.4 64-bit (conda)",
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5"
|
||||
}
|
||||
},
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.11.2"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "ab2bd97b0453415b89a469284609a8ce",
|
||||
"translation_date": "2025-08-28T11:31:02+00:00",
|
||||
"source_file": "lessons/2-Symbolic/Animals.ipynb",
|
||||
"language_code": "mo"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
|
|
@ -0,0 +1,593 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"collapsed": true
|
||||
},
|
||||
"source": [
|
||||
"# 家庭關係本體\n",
|
||||
"\n",
|
||||
"此範例是 [AI for Beginners Curriculum](http://github.com/microsoft/ai-for-beginners) 的一部分，靈感來自 [這篇部落格文章](https://habr.com/post/270857/)。\n",
|
||||
"\n",
|
||||
"我總是覺得記住家庭中人與人之間的不同關係很困難。在這個範例中，我們將使用一個定義家庭關係的本體，以及實際的家譜樹，並展示如何進行自動推理以找出所有親屬。\n",
|
||||
"\n",
|
||||
"### 獲取家譜樹\n",
|
||||
"\n",
|
||||
"作為範例，我們將使用 [羅曼諾夫沙皇家族](https://en.wikipedia.org/wiki/House_of_Romanov) 的家譜樹。描述家庭關係最常用的格式是 [GEDCOM](https://en.wikipedia.org/wiki/GEDCOM)。我們將使用 GEDCOM 格式的羅曼諾夫家族樹：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"0 HEAD\n",
|
||||
"1 CHAR UTF8\n",
|
||||
"1 GEDC\n",
|
||||
"2 VERS 5.5\n",
|
||||
"0 @0@ INDI\n",
|
||||
"1 NAME Mihail Fedorovich /Romanov/\n",
|
||||
"1 SEX M\n",
|
||||
"1 BIRT\n",
|
||||
"2 DATE 1613\n",
|
||||
"1 DEAT \n",
|
||||
"2 DATE 1645\n",
|
||||
"1 FAMS @41@\n",
|
||||
"0 @1@ INDI\n",
|
||||
"1 NAME Evdokija Lukjanovna /Streshneva/\n",
|
||||
"1 SEX F\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"!head -15 data/tsars.ged"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"要使用 GEDCOM 文件，我們可以使用 `python-gedcom` 庫：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Collecting python-gedcom\n",
|
||||
" Downloading python_gedcom-1.0.0-py2.py3-none-any.whl (35 kB)\n",
|
||||
"Installing collected packages: python-gedcom\n",
|
||||
"Successfully installed python-gedcom-1.0.0\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import sys\n",
|
||||
"!{sys.executable} -m pip install python-gedcom"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"此庫消除了文件解析的一些技術問題，但仍然為我們提供了對樹中所有個人和家庭的相當低層次的訪問。以下是我們如何解析文件並顯示所有個人列表的方法：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from gedcom.parser import Parser\n",
|
||||
"from gedcom.element.individual import IndividualElement\n",
|
||||
"from gedcom.element.family import FamilyElement\n",
|
||||
"g = Parser()\n",
|
||||
"g.parse_file('data/tsars.ged')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {
|
||||
"scrolled": true,
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[('@0@', ('Mihail Fedorovich', 'Romanov')),\n",
|
||||
" ('@1@', ('Evdokija Lukjanovna', 'Streshneva')),\n",
|
||||
" ('@2@', ('Aleksej Mihajlovich', 'Romanov')),\n",
|
||||
" ('@3@', ('Marija Ilinichna', 'Miloslavskaja')),\n",
|
||||
" ('@4@', ('Natalja Kirillovna', 'Naryshkina')),\n",
|
||||
" ('@5@', ('Marfa Matveevna', 'Apraksina')),\n",
|
||||
" ('@6@', ('Fedor Alekseevich', 'Romanov')),\n",
|
||||
" ('@7@', ('Sofja Aleksevna', 'Romanova')),\n",
|
||||
" ('@8@', ('Ivan V Alekseevich', 'Romanov')),\n",
|
||||
" ('@9@', ('Praskovja Fedorovna', 'Saltykova')),\n",
|
||||
" ('@10@', ('Ekaterina Ivanovna', 'Romanova')),\n",
|
||||
" ('@11@', ('Anna Ivanovna', 'Romanova')),\n",
|
||||
" ('@12@', ('Fridrih Vilgelm', 'Kurlandskij')),\n",
|
||||
" ('@13@', ('Karl Leopold', 'Meklenburg-Shverinskij')),\n",
|
||||
" ('@14@', ('Anna Leopoldovna', 'Meklenburg-Shverinskaja')),\n",
|
||||
" ('@15@', ('Anton Ulrih', 'Braunshvejg-Volfenbjuttelskij')),\n",
|
||||
" ('@16@', ('Ivan VI Antonovich', 'Braunshvejg-Volfenbjuttelskij')),\n",
|
||||
" ('@17@', ('Petr I Alekseevich', 'Romanov')),\n",
|
||||
" ('@18@', ('Evdokija Fedorovna', 'Lopuhina')),\n",
|
||||
" ('@19@', ('Ekaterina I Alekseevna', 'Mihajlova')),\n",
|
||||
" ('@20@', ('Aleksej Petrovich', 'Romanov')),\n",
|
||||
" ('@21@', ('Sharlotta Kristina', 'Braunshvejg-Volfenbjuttelskaja')),\n",
|
||||
" ('@22@', ('Petr II Alekseevich', 'Romanov')),\n",
|
||||
" ('@23@', ('Anna Petrovna', 'Romanova')),\n",
|
||||
" ('@24@', ('Elizaveta Petrovna', 'Romanova')),\n",
|
||||
" ('@25@', ('Karl Fridrih', 'Golshtejn-Gottorpskij')),\n",
|
||||
" ('@26@', ('Petr III Fedorovich', 'Romanov')),\n",
|
||||
" ('@27@', ('Ekaterina II', 'Alekseevna')),\n",
|
||||
" ('@28@', ('Pavel I Petrovich', 'Romanov')),\n",
|
||||
" ('@29@', ('Natalja Alekseevna', 'Gessen-Darmshtadskaja')),\n",
|
||||
" ('@30@', ('Marija Fedorovna', 'Vjurtembergskaja')),\n",
|
||||
" ('@31@', ('Aleksandr I Pavlovich', 'Romanov')),\n",
|
||||
" ('@32@', ('Elizaveta Alekseevna', 'Baden-Durlahskaja')),\n",
|
||||
" ('@33@', ('Nikolaj I Pavlovich', 'Romanov')),\n",
|
||||
" ('@34@', ('Aleksandra Fedorovna', 'Prusskaja')),\n",
|
||||
" ('@35@', ('Aleksandr II Nikolaevich', 'Romanov')),\n",
|
||||
" ('@36@', ('Marija Aleksandrovna', 'Gessenskaja')),\n",
|
||||
" ('@37@', ('Aleksandr III Aleksandrovich', 'Romanov')),\n",
|
||||
" ('@38@', ('Marija Fedorovna', 'Datskaja')),\n",
|
||||
" ('@39@', ('Nikolaj II Aleksandrovich', 'Romanov')),\n",
|
||||
" ('@40@', ('Aleksandra Fedorovna', 'Gessenskaja'))]"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"d = g.get_element_dictionary()\n",
|
||||
"[ (k,v.get_name()) for k,v in d.items() if isinstance(v,IndividualElement)]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"以下是我們如何獲取有關家庭的信息。請注意，這會給我們一個**標識符**的列表，如果我們想要更清楚的內容，我們需要將它們轉換為名稱：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[('@41@', ['@0@', '@1@', '@2@']),\n",
|
||||
" ('@42@', ['@2@', '@3@', '@6@', '@7@', '@8@']),\n",
|
||||
" ('@43@', ['@8@', '@9@', '@10@', '@11@']),\n",
|
||||
" ('@44@', ['@13@', '@10@', '@14@']),\n",
|
||||
" ('@45@', ['@15@', '@14@', '@16@']),\n",
|
||||
" ('@46@', ['@2@', '@4@', '@17@']),\n",
|
||||
" ('@47@', ['@17@', '@18@', '@20@']),\n",
|
||||
" ('@48@', ['@20@', '@21@', '@22@']),\n",
|
||||
" ('@49@', ['@17@', '@19@', '@23@', '@24@']),\n",
|
||||
" ('@50@', ['@25@', '@23@', '@26@']),\n",
|
||||
" ('@51@', ['@26@', '@27@', '@28@']),\n",
|
||||
" ('@52@', ['@28@', '@30@', '@31@', '@33@']),\n",
|
||||
" ('@53@', ['@33@', '@34@', '@35@']),\n",
|
||||
" ('@54@', ['@35@', '@36@', '@37@']),\n",
|
||||
" ('@55@', ['@37@', '@38@', '@39@'])]"
|
||||
]
|
||||
},
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"d = g.get_element_dictionary()\n",
|
||||
"[ (k,[x.get_value() for x in v.get_child_elements()]) for k,v in d.items() if isinstance(v,FamilyElement)]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 獲取家庭本體\n",
|
||||
"\n",
|
||||
"接下來，我們來看看[家庭本體](https://raw.githubusercontent.com/blokhin/genealogical-trees/master/data/header.ttl)，它被定義為一組語義網三元組。這個本體定義了像 `isUncleOf`、`isCousinOf` 等許多關係。所有這些關係都是基於基本謂詞 `isMotherOf`、`isFatherOf`、`isBrotherOf` 和 `isSisterOf` 來定義的。我們將使用自動推理來根據本體推導出所有其他關係。\n",
|
||||
"\n",
|
||||
"以下是 `isAuntOf` 屬性的範例定義，它被定義為 `isSisterOf` 和 `isParentOf` 的組合（*姑姑是某人父母的姐妹*）。\n",
|
||||
"\n",
|
||||
"```\n",
|
||||
"fhkb:isAuntOf a owl:ObjectProperty ;\n",
|
||||
" rdfs:domain fhkb:Woman ;\n",
|
||||
" rdfs:range fhkb:Person ;\n",
|
||||
" owl:propertyChainAxiom ( fhkb:isSisterOf fhkb:isParentOf ) .\n",
|
||||
"```\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"@prefix fhkb: <http://www.example.com/genealogy.owl#> .\n",
|
||||
"@prefix owl: <http://www.w3.org/2002/07/owl#> .\n",
|
||||
"@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n",
|
||||
"@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .\n",
|
||||
"@prefix xml: <http://www.w3.org/XML/1998/namespace> .\n",
|
||||
"@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\n",
|
||||
"\n",
|
||||
"<http://www.example.com/genealogy.owl#> a owl:Ontology .\n",
|
||||
"\n",
|
||||
"fhkb:DomainEntity a owl:Class .\n",
|
||||
"\n",
|
||||
"fhkb:Man a owl:Class ;\n",
|
||||
" owl:equivalentClass [ a owl:Class ;\n",
|
||||
" owl:intersectionOf ( fhkb:Person [ a owl:Restriction ;\n",
|
||||
" owl:onProperty fhkb:hasSex ;\n",
|
||||
" owl:someValuesFrom fhkb:Male ] ) ] .\n",
|
||||
"\n",
|
||||
"fhkb:Woman a owl:Class ;\n",
|
||||
" owl:equivalentClass [ a owl:Class ;\n",
|
||||
" owl:intersectionOf ( fhkb:Person [ a owl:Restriction ;\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"!head -20 data/onto.ttl"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 建構推理用本體\n",
|
||||
"\n",
|
||||
"為了簡化，我們將建立一個本體檔案，該檔案將包含來自家庭本體的原始規則，以及來自我們 GEDCOM 檔案的個人事實。我們會逐一檢視 GEDCOM 檔案，提取有關家庭和個人的資訊，並將其轉換為三元組。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!cp data/onto.ttl .\n",
|
||||
"\n",
|
||||
"gedcom_dict = g.get_element_dictionary()\n",
|
||||
"individuals, marriages = {}, {}\n",
|
||||
"\n",
|
||||
"def term2id(el):\n",
|
||||
" return \"i\" + el.get_pointer().replace('@', '').lower()\n",
|
||||
"\n",
|
||||
"out = open(\"onto.ttl\",\"a\")\n",
|
||||
"\n",
|
||||
"for k, v in gedcom_dict.items():\n",
|
||||
" if isinstance(v,IndividualElement):\n",
|
||||
" children, siblings = set(), set()\n",
|
||||
" idx = term2id(v)\n",
|
||||
"\n",
|
||||
" title = v.get_name()[0] + \" \" + v.get_name()[1]\n",
|
||||
" title = title.replace('\"', '').replace('[', '').replace(']', '').replace('(', '').replace(')', '').strip()\n",
|
||||
"\n",
|
||||
" own_families = g.get_families(v, 'FAMS')\n",
|
||||
" for fam in own_families:\n",
|
||||
" children |= set(term2id(i) for i in g.get_family_members(fam, \"CHIL\"))\n",
|
||||
"\n",
|
||||
" parent_families = g.get_families(v, 'FAMC')\n",
|
||||
" if len(parent_families):\n",
|
||||
" for member in g.get_family_members(parent_families[0], \"CHIL\"): # NB adoptive families i.e len(parent_families)>1 are not considered (TODO?)\n",
|
||||
" if member.get_pointer() == v.get_pointer():\n",
|
||||
" continue\n",
|
||||
" siblings.add(term2id(member))\n",
|
||||
"\n",
|
||||
" if idx in individuals:\n",
|
||||
" children |= individuals[idx].get('children', set())\n",
|
||||
" siblings |= individuals[idx].get('siblings', set())\n",
|
||||
" individuals[idx] = {'sex': v.get_gender().lower(), 'children': children, 'siblings': siblings, 'title': title}\n",
|
||||
"\n",
|
||||
" elif isinstance(v,FamilyElement):\n",
|
||||
" wife, husb, children = None, None, set()\n",
|
||||
" children = set(term2id(i) for i in g.get_family_members(v, \"CHIL\"))\n",
|
||||
"\n",
|
||||
" try:\n",
|
||||
" wife = g.get_family_members(v, \"WIFE\")[0]\n",
|
||||
" wife = term2id(wife)\n",
|
||||
" if wife in individuals: individuals[wife]['children'] |= children\n",
|
||||
" else: individuals[wife] = {'children': children}\n",
|
||||
" except IndexError: pass\n",
|
||||
" try:\n",
|
||||
" husb = g.get_family_members(v, \"HUSB\")[0]\n",
|
||||
" husb = term2id(husb)\n",
|
||||
" if husb in individuals: individuals[husb]['children'] |= children\n",
|
||||
" else: individuals[husb] = {'children': children}\n",
|
||||
" except IndexError: pass\n",
|
||||
"\n",
|
||||
" if wife and husb: marriages[wife + husb] = (term2id(v), wife, husb)\n",
|
||||
"\n",
|
||||
"for idx, val in individuals.items():\n",
|
||||
" added_terms = ''\n",
|
||||
" if val['sex'] == 'f':\n",
|
||||
" parent_predicate, sibl_predicate = \"isMotherOf\", \"isSisterOf\"\n",
|
||||
" else:\n",
|
||||
" parent_predicate, sibl_predicate = \"isFatherOf\", \"isBrotherOf\"\n",
|
||||
" if len(val['children']):\n",
|
||||
" added_terms += \" ;\\n fhkb:\" + parent_predicate + \" \" + \", \".join([\"fhkb:\" + i for i in val['children']])\n",
|
||||
" if len(val['siblings']):\n",
|
||||
" added_terms += \" ;\\n fhkb:\" + sibl_predicate + \" \" + \", \".join([\"fhkb:\" + i for i in val['siblings']])\n",
|
||||
" out.write(\"fhkb:%s a owl:NamedIndividual, owl:Thing%s ;\\n rdfs:label \\\"%s\\\" .\\n\" % (idx, added_terms, val['title']))\n",
|
||||
"\n",
|
||||
"for k, v in marriages.items():\n",
|
||||
" out.write(\"fhkb:%s a owl:NamedIndividual, owl:Thing ;\\n fhkb:hasFemalePartner fhkb:%s ;\\n fhkb:hasMalePartner fhkb:%s .\\n\" % v)\n",
|
||||
"\n",
|
||||
"out.write(\"[] a owl:AllDifferent ;\\n owl:distinctMembers (\")\n",
|
||||
"for idx in individuals.keys():\n",
|
||||
" out.write(\" fhkb:\" + idx)\n",
|
||||
"for k, v in marriages.items():\n",
|
||||
" out.write(\" fhkb:\" + v[0])\n",
|
||||
"out.write(\" ) .\")\n",
|
||||
"out.close()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" fhkb:hasFemalePartner fhkb:i34 ;\n",
|
||||
" fhkb:hasMalePartner fhkb:i33 .\n",
|
||||
"fhkb:i54 a owl:NamedIndividual, owl:Thing ;\n",
|
||||
" fhkb:hasFemalePartner fhkb:i36 ;\n",
|
||||
" fhkb:hasMalePartner fhkb:i35 .\n",
|
||||
"fhkb:i55 a owl:NamedIndividual, owl:Thing ;\n",
|
||||
" fhkb:hasFemalePartner fhkb:i38 ;\n",
|
||||
" fhkb:hasMalePartner fhkb:i37 .\n",
|
||||
"[] a owl:AllDifferent ;\n",
|
||||
" owl:distinctMembers ( fhkb:i0 fhkb:i1 fhkb:i2 fhkb:i3 fhkb:i4 fhkb:i5 fhkb:i6 fhkb:i7 fhkb:i8 fhkb:i9 fhkb:i10 fhkb:i11 fhkb:i12 fhkb:i13 fhkb:i14 fhkb:i15 fhkb:i16 fhkb:i17 fhkb:i18 fhkb:i19 fhkb:i20 fhkb:i21 fhkb:i22 fhkb:i23 fhkb:i24 fhkb:i25 fhkb:i26 fhkb:i27 fhkb:i28 fhkb:i29 fhkb:i30 fhkb:i31 fhkb:i32 fhkb:i33 fhkb:i34 fhkb:i35 fhkb:i36 fhkb:i37 fhkb:i38 fhkb:i39 fhkb:i40 fhkb:i41 fhkb:i42 fhkb:i43 fhkb:i44 fhkb:i45 fhkb:i46 fhkb:i47 fhkb:i48 fhkb:i49 fhkb:i50 fhkb:i51 fhkb:i52 fhkb:i53 fhkb:i54 fhkb:i55 ) ."
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"!tail onto.ttl"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 進行推理\n",
|
||||
"\n",
|
||||
"現在我們希望能夠使用這個本體進行推理和查詢。我們將使用 [RDFLib](https://github.com/RDFLib)，這是一個用於以不同格式讀取 RDF 圖、查詢等的庫。\n",
|
||||
"\n",
|
||||
"對於邏輯推理，我們將使用 [OWL-RL](https://github.com/RDFLib/OWL-RL) 庫，該庫允許我們構建 RDF 圖的 **Closure**，也就是添加所有可以推導出的概念和關係。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Requirement already satisfied: rdflib in /home/rg/anaconda3/envs/ai4beg/lib/python3.11/site-packages (6.3.2)\n",
|
||||
"Requirement already satisfied: isodate<0.7.0,>=0.6.0 in /home/rg/anaconda3/envs/ai4beg/lib/python3.11/site-packages (from rdflib) (0.6.1)\n",
|
||||
"Requirement already satisfied: pyparsing<4,>=2.1.0 in /home/rg/anaconda3/envs/ai4beg/lib/python3.11/site-packages (from rdflib) (3.0.9)\n",
|
||||
"Requirement already satisfied: six in /home/rg/anaconda3/envs/ai4beg/lib/python3.11/site-packages (from isodate<0.7.0,>=0.6.0->rdflib) (1.16.0)\n",
|
||||
"Collecting git+https://github.com/RDFLib/OWL-RL.git\n",
|
||||
" Cloning https://github.com/RDFLib/OWL-RL.git to /tmp/pip-req-build-lbfzwi3m\n",
|
||||
" Running command git clone --filter=blob:none --quiet https://github.com/RDFLib/OWL-RL.git /tmp/pip-req-build-lbfzwi3m\n",
|
||||
" Resolved https://github.com/RDFLib/OWL-RL.git to commit a77e1791b88b54aace609bc6000aac14c7add4ff\n",
|
||||
" Preparing metadata (setup.py) ... \u001b[?25ldone\n",
|
||||
"\u001b[?25hRequirement already satisfied: rdflib>=6.0.2 in /home/rg/anaconda3/envs/ai4beg/lib/python3.11/site-packages (from owlrl==6.0.2) (6.3.2)\n",
|
||||
"Requirement already satisfied: isodate<0.7.0,>=0.6.0 in /home/rg/anaconda3/envs/ai4beg/lib/python3.11/site-packages (from rdflib>=6.0.2->owlrl==6.0.2) (0.6.1)\n",
|
||||
"Requirement already satisfied: pyparsing<4,>=2.1.0 in /home/rg/anaconda3/envs/ai4beg/lib/python3.11/site-packages (from rdflib>=6.0.2->owlrl==6.0.2) (3.0.9)\n",
|
||||
"Requirement already satisfied: six in /home/rg/anaconda3/envs/ai4beg/lib/python3.11/site-packages (from isodate<0.7.0,>=0.6.0->rdflib>=6.0.2->owlrl==6.0.2) (1.16.0)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"!{sys.executable} -m pip install rdflib\n",
|
||||
"!{sys.executable} -m pip install git+https://github.com/RDFLib/OWL-RL.git"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"讓我們打開本體文件，看看它包含多少三元組：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Triplets found:669\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import rdflib\n",
|
||||
"from owlrl import DeductiveClosure, OWLRL_Extension\n",
|
||||
"\n",
|
||||
"g = rdflib.Graph()\n",
|
||||
"g.parse(\"onto.ttl\", format=\"turtle\")\n",
|
||||
"\n",
|
||||
"print(\"Triplets found:%d\" % len(g))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Triplets after inference:4246\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"DeductiveClosure(OWLRL_Extension).expand(g)\n",
|
||||
"print(\"Triplets after inference:%d\" % len(g))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 查詢親屬關係\n",
|
||||
"\n",
|
||||
"現在我們可以查詢圖表，查看人與人之間的不同關係。我們可以使用 **SPARQL** 語言搭配 `query` 方法。在我們的例子中，讓我們看看家譜中的所有 **叔叔**：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Fedor Alekseevich Romanov is uncle of Ekaterina Ivanovna Romanova\n",
|
||||
"Aleksandr I Pavlovich Romanov is uncle of Aleksandr II Nikolaevich Romanov\n",
|
||||
"Fedor Alekseevich Romanov is uncle of Anna Ivanovna Romanova\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"qres = g.query(\n",
|
||||
" \"\"\"SELECT DISTINCT ?aname ?bname\n",
|
||||
" WHERE {\n",
|
||||
" ?a fhkb:isUncleOf ?b .\n",
|
||||
" ?a rdfs:label ?aname .\n",
|
||||
" ?b rdfs:label ?bname .\n",
|
||||
" }\"\"\")\n",
|
||||
"\n",
|
||||
"for row in qres:\n",
|
||||
" print(\"%s is uncle of %s\" % row)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"隨意嘗試其他家庭關係。例如，你可以看看 `isAncestorOf` 關係，它遞歸地定義了一個人所有的祖先。\n",
|
||||
"\n",
|
||||
"最後，讓我們清理一下吧！\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!rm onto.ttl"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**免責聲明**： \n本文件使用 AI 翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。儘管我們努力確保翻譯的準確性，但請注意，自動翻譯可能包含錯誤或不準確之處。原始文件的母語版本應被視為權威來源。對於關鍵信息，建議使用專業人工翻譯。我們對因使用此翻譯而引起的任何誤解或錯誤解釋不承擔責任。\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.6",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.11.2"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "6537d5597320e27b6052b4377b8ff8bb",
|
||||
"translation_date": "2025-08-28T11:28:52+00:00",
|
||||
"source_file": "lessons/2-Symbolic/FamilyOntology.ipynb",
|
||||
"language_code": "mo"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
|
|
@ -0,0 +1,548 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"collapsed": true
|
||||
},
|
||||
"source": [
|
||||
"## Microsoft 概念圖\n",
|
||||
"\n",
|
||||
"[Microsoft 概念圖](https://concept.research.microsoft.com/) 是一個從網路中挖掘出的大型術語分類，包含概念之間的 `is-a` 關係。\n",
|
||||
"\n",
|
||||
"概念圖提供以下兩種形式：\n",
|
||||
" * 可下載的大型文字檔案\n",
|
||||
" * REST API\n",
|
||||
"\n",
|
||||
"統計數據：\n",
|
||||
" * 5401933 個獨特概念\n",
|
||||
" * 12551613 個獨特實例\n",
|
||||
" * 87603947 個 `is-a` 關係\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 使用網路服務\n",
|
||||
"\n",
|
||||
"網路服務提供多種呼叫方式，用於估算某個概念屬於不同群組的可能性。更多資訊請參考[這裡](https://concept.research.microsoft.com/Home/Api)。 \n",
|
||||
"以下是範例呼叫的 URL：`https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"{'company': 0.6105356614382954,\n",
|
||||
" 'vendor': 0.08858636677518003,\n",
|
||||
" 'client': 0.048239124001183784,\n",
|
||||
" 'firm': 0.045476965571668145,\n",
|
||||
" 'large company': 0.043109401203511886,\n",
|
||||
" 'organization': 0.043010752688172046,\n",
|
||||
" 'corporation': 0.035908059583703265,\n",
|
||||
" 'brand': 0.03383644076156654,\n",
|
||||
" 'software company': 0.027522935779816515,\n",
|
||||
" 'technology company': 0.023774292196902438}"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import urllib\n",
|
||||
"import json\n",
|
||||
"import ssl\n",
|
||||
"\n",
|
||||
"def http(x):\n",
|
||||
" ssl._create_default_https_context = ssl._create_unverified_context\n",
|
||||
" response = urllib.request.urlopen(x)\n",
|
||||
" data = response.read()\n",
|
||||
" return data.decode('utf-8')\n",
|
||||
"\n",
|
||||
"def query(x):\n",
|
||||
" return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n",
|
||||
"\n",
|
||||
"query('microsoft')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"讓我們嘗試使用父概念對新聞標題進行分類。要獲取新聞標題，我們將使用 [NewsApi.org](http://newsapi.org) 服務。您需要獲取自己的 API 金鑰才能使用該服務——請訪問網站並註冊免費開發者計劃。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 20,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"newsapi_key = '<your API key here>'\n",
|
||||
"def get_news(country='us'):\n",
|
||||
" res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n",
|
||||
" return res['articles']\n",
|
||||
"\n",
|
||||
"all_titles = [x['title'] for x in get_news('us')+get_news('gb')]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 21,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n",
|
||||
" 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n",
|
||||
" 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n",
|
||||
" 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n",
|
||||
" \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n",
|
||||
" '2 teachers killed at Swedish high school, student arrested - ABC News',\n",
|
||||
" 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n",
|
||||
" 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n",
|
||||
" '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n",
|
||||
" 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n",
|
||||
" \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n",
|
||||
" \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n",
|
||||
" \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n",
|
||||
" 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n",
|
||||
" 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n",
|
||||
" 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n",
|
||||
" 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n",
|
||||
" 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n",
|
||||
" \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n",
|
||||
" 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n",
|
||||
" \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n",
|
||||
" 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n",
|
||||
" 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n",
|
||||
" 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n",
|
||||
" 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n",
|
||||
" \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n",
|
||||
" 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n",
|
||||
" \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n",
|
||||
" 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n",
|
||||
" 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n",
|
||||
" \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n",
|
||||
" '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n",
|
||||
" 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n",
|
||||
" 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n",
|
||||
" 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n",
|
||||
" 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n",
|
||||
" 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n",
|
||||
" 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n",
|
||||
" \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n",
|
||||
" 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']"
|
||||
]
|
||||
},
|
||||
"execution_count": 21,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"all_titles"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"首先，我們希望能夠從新聞標題中提取名詞。我們將使用 `TextBlob` 庫來完成這項工作，這可以簡化許多像這樣的典型自然語言處理任務。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n",
|
||||
"Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n",
|
||||
"Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n",
|
||||
"Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n",
|
||||
"Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n",
|
||||
"Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n",
|
||||
"Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n",
|
||||
"Finished.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"[nltk_data] Downloading package brown to\n",
|
||||
"[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n",
|
||||
"[nltk_data] Package brown is already up-to-date!\n",
|
||||
"[nltk_data] Downloading package punkt to\n",
|
||||
"[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n",
|
||||
"[nltk_data] Package punkt is already up-to-date!\n",
|
||||
"[nltk_data] Downloading package wordnet to\n",
|
||||
"[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n",
|
||||
"[nltk_data] Package wordnet is already up-to-date!\n",
|
||||
"[nltk_data] Downloading package averaged_perceptron_tagger to\n",
|
||||
"[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n",
|
||||
"[nltk_data] Package averaged_perceptron_tagger is already up-to-\n",
|
||||
"[nltk_data] date!\n",
|
||||
"[nltk_data] Downloading package conll2000 to\n",
|
||||
"[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n",
|
||||
"[nltk_data] Package conll2000 is already up-to-date!\n",
|
||||
"[nltk_data] Downloading package movie_reviews to\n",
|
||||
"[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n",
|
||||
"[nltk_data] Package movie_reviews is already up-to-date!\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import sys\n",
|
||||
"!{sys.executable} -m pip install textblob\n",
|
||||
"!{sys.executable} -m textblob.download_corpora\n",
|
||||
"from textblob import TextBlob"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 22,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"{'covid-19 live updates': 1,\n",
|
||||
" 'vaccines': 1,\n",
|
||||
" 'boosters': 1,\n",
|
||||
" 'york': 4,\n",
|
||||
" 'ukrainians flee mariupol': 1,\n",
|
||||
" 'forces push': 1,\n",
|
||||
" 'port city': 1,\n",
|
||||
" 'wall street journal': 3,\n",
|
||||
" 'bond yields': 1,\n",
|
||||
" 'futures rise': 1,\n",
|
||||
" 'powell says fed': 1,\n",
|
||||
" 'ready': 1,\n",
|
||||
" 'be': 1,\n",
|
||||
" 'aggressive': 1,\n",
|
||||
" 'putin': 3,\n",
|
||||
" 'alexei navalny': 2,\n",
|
||||
" 'russian': 2,\n",
|
||||
" 'supreme court nominee': 1,\n",
|
||||
" 'ketanji brown jackson': 1,\n",
|
||||
" \"confirmation hearing 's\": 1,\n",
|
||||
" 'cnn': 1,\n",
|
||||
" 'swedish': 1,\n",
|
||||
" 'high school': 1,\n",
|
||||
" 'abc': 1,\n",
|
||||
" 'clues': 1,\n",
|
||||
" 'covid-19': 1,\n",
|
||||
" '’ s': 2,\n",
|
||||
" 'moves': 1,\n",
|
||||
" 'sewers': 1,\n",
|
||||
" 'roll dice': 1,\n",
|
||||
" 'jackson': 1,\n",
|
||||
" 'decisions |': 1,\n",
|
||||
" 'thehill': 1,\n",
|
||||
" 'clear': 2,\n",
|
||||
" 'chemical weapons': 2,\n",
|
||||
" 'ukraine': 3,\n",
|
||||
" 'claims president': 2,\n",
|
||||
" 'biden': 2,\n",
|
||||
" 'nasa': 2,\n",
|
||||
" 'solar system': 2,\n",
|
||||
" 'daily mail': 3,\n",
|
||||
" 'us stocks': 1,\n",
|
||||
" 'fed chair powell': 1,\n",
|
||||
" \"'s remarks\": 1,\n",
|
||||
" 'fox': 1,\n",
|
||||
" \"'we 've\": 1,\n",
|
||||
" 'tests': 1,\n",
|
||||
" 'covid': 1,\n",
|
||||
" 'politico': 1,\n",
|
||||
" 'duchess': 1,\n",
|
||||
" 'cambridge': 1,\n",
|
||||
" 'swaps khaki jungle gear': 1,\n",
|
||||
" 'vampire': 1,\n",
|
||||
" 'wife': 1,\n",
|
||||
" 'belize': 1,\n",
|
||||
" 'china': 2,\n",
|
||||
" 'flight recorders': 1,\n",
|
||||
" 'plane crash': 1,\n",
|
||||
" 'reuters': 2,\n",
|
||||
" 'russian oligarch': 1,\n",
|
||||
" 'abramovich': 1,\n",
|
||||
" 'live': 1,\n",
|
||||
" 'russia': 2,\n",
|
||||
" 'stops talks': 1,\n",
|
||||
" 'japan': 1,\n",
|
||||
" 'español': 1,\n",
|
||||
" 'powers remain': 1,\n",
|
||||
" 'threats lurk': 1,\n",
|
||||
" 'set': 1,\n",
|
||||
" 'webb': 1,\n",
|
||||
" 'telescope begins multi-instrument alignment': 1,\n",
|
||||
" 'scitechdaily': 1,\n",
|
||||
" 'uconn': 1,\n",
|
||||
" 'ucf': 1,\n",
|
||||
" 'ncaa': 1,\n",
|
||||
" \"women 's tournament second-round highlights\": 1,\n",
|
||||
" 'march madness': 1,\n",
|
||||
" 'bucking republican trend': 1,\n",
|
||||
" 'indiana': 1,\n",
|
||||
" 'vetoes transgender': 1,\n",
|
||||
" 'bill': 1,\n",
|
||||
" 'maggie fox': 1,\n",
|
||||
" 'coronation': 1,\n",
|
||||
" 'shameless': 1,\n",
|
||||
" \"'sudden accident\": 1,\n",
|
||||
" 'mirror online': 1,\n",
|
||||
" 'mirror': 2,\n",
|
||||
" 'plane crash –': 1,\n",
|
||||
" 'search': 1,\n",
|
||||
" 'moment flight': 1,\n",
|
||||
" 'daniel morgan': 1,\n",
|
||||
" 'report condemns': 1,\n",
|
||||
" 'met': 1,\n",
|
||||
" 'guardian': 6,\n",
|
||||
" 'rishi sunak': 1,\n",
|
||||
" '’ s spring': 1,\n",
|
||||
" 'statement': 1,\n",
|
||||
" 'bbc.com': 1,\n",
|
||||
" 'uk': 3,\n",
|
||||
" 'ireland': 1,\n",
|
||||
" 'euro': 1,\n",
|
||||
" 'vladimir putin': 2,\n",
|
||||
" \"'s 'lover\": 1,\n",
|
||||
" 'brass eye': 1,\n",
|
||||
" '’ s outtakes': 1,\n",
|
||||
" 'brutal tv comedy': 1,\n",
|
||||
" 'threatens civilians': 1,\n",
|
||||
" 'mariupol': 1,\n",
|
||||
" \"'s spirit\": 1,\n",
|
||||
" 'shell u-turn': 1,\n",
|
||||
" 'cambo': 1,\n",
|
||||
" 'green targets': 1,\n",
|
||||
" 'st helens': 1,\n",
|
||||
" 'dog attack': 1,\n",
|
||||
" 'girl': 1,\n",
|
||||
" 'bbc': 1,\n",
|
||||
" 'playstation': 1,\n",
|
||||
" \"'assassin 's\": 1,\n",
|
||||
" 'creed': 1,\n",
|
||||
" 'jade raymond': 1,\n",
|
||||
" 'haven studios': 1,\n",
|
||||
" 'nme': 1,\n",
|
||||
" 'nintendo switch': 1,\n",
|
||||
" 'folders •': 1,\n",
|
||||
" 'eurogamer.net': 2,\n",
|
||||
" 'fa': 1,\n",
|
||||
" 'solution ”': 1,\n",
|
||||
" 'liverpool': 1,\n",
|
||||
" 'fan group blasts “ shambolic ”': 1,\n",
|
||||
" 'wembley': 1,\n",
|
||||
" 'anfield': 1,\n",
|
||||
" 'manchester': 1,\n",
|
||||
" 'live erik': 1,\n",
|
||||
" 'hag': 1,\n",
|
||||
" 'utd': 1,\n",
|
||||
" 'manager updates': 1,\n",
|
||||
" 'manchester evening': 1,\n",
|
||||
" 'inflation': 1,\n",
|
||||
" 'government borrowing': 1,\n",
|
||||
" 'february': 1,\n",
|
||||
" 'crude oil': 1,\n",
|
||||
" '– business': 1,\n",
|
||||
" 'kremlin': 1,\n",
|
||||
" 'large-scale fraud': 1,\n",
|
||||
" 'sky': 1,\n",
|
||||
" 'natural gas': 1,\n",
|
||||
" 'gazprom': 1,\n",
|
||||
" 'retail unit': 1,\n",
|
||||
" 'insider': 1,\n",
|
||||
" 'zaghari-ratcliffe': 1,\n",
|
||||
" 'hunt': 1,\n",
|
||||
" 'iran': 1,\n",
|
||||
" 'debt payment': 1}"
|
||||
]
|
||||
},
|
||||
"execution_count": 22,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"w = {}\n",
|
||||
"for x in all_titles:\n",
|
||||
" for n in TextBlob(x).noun_phrases:\n",
|
||||
" if n in w:\n",
|
||||
" w[n].append(x)\n",
|
||||
" else:\n",
|
||||
" w[n]=[x]\n",
|
||||
"{ x:len(w[x]) for x in w.keys()}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"我們可以看到名詞並未給我們大的主題群組。讓我們用從概念圖中獲得的更一般的術語來替代名詞。這將需要一些時間，因為我們正在為每個名詞短語進行 REST 調用。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 23,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"w = {}\n",
|
||||
"for x in all_titles:\n",
|
||||
" for noun in TextBlob(x).noun_phrases:\n",
|
||||
" terms = query(noun.replace(' ','%20'))\n",
|
||||
" for term in [u for u in terms.keys() if terms[u]>0.1]:\n",
|
||||
" if term in w:\n",
|
||||
" w[term].append(x)\n",
|
||||
" else:\n",
|
||||
" w[term]=[x]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 24,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"{'city': 9,\n",
|
||||
" 'brand': 4,\n",
|
||||
" 'place': 9,\n",
|
||||
" 'town': 4,\n",
|
||||
" 'factor': 4,\n",
|
||||
" 'film': 4,\n",
|
||||
" 'nation': 11,\n",
|
||||
" 'state': 5,\n",
|
||||
" 'person': 4,\n",
|
||||
" 'organization': 5,\n",
|
||||
" 'publication': 10,\n",
|
||||
" 'market': 5,\n",
|
||||
" 'economy': 4,\n",
|
||||
" 'company': 6,\n",
|
||||
" 'newspaper': 6,\n",
|
||||
" 'relationship': 6}"
|
||||
]
|
||||
},
|
||||
"execution_count": 24,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"{ x:len(w[x]) for x in w.keys() if len(w[x])>3}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 27,
|
||||
"metadata": {
|
||||
"trusted": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\n",
|
||||
"ECONOMY:\n",
|
||||
"China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n",
|
||||
"Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n",
|
||||
"China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n",
|
||||
"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n",
|
||||
"\n",
|
||||
"NATION:\n",
|
||||
"‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n",
|
||||
"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n",
|
||||
"China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n",
|
||||
"Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n",
|
||||
"Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n",
|
||||
"China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n",
|
||||
"UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n",
|
||||
"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n",
|
||||
"‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n",
|
||||
"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n",
|
||||
"Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n",
|
||||
"\n",
|
||||
"PERSON:\n",
|
||||
"‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n",
|
||||
"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n",
|
||||
"Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n",
|
||||
"‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n",
|
||||
"print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n",
|
||||
"print('\\nPERSON:\\n'+'\\n'.join(w['person']))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**免責聲明**： \n本文件已使用 AI 翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。儘管我們努力確保翻譯的準確性，但請注意，自動翻譯可能包含錯誤或不準確之處。原始文件的母語版本應被視為權威來源。對於關鍵資訊，建議使用專業人工翻譯。我們對因使用此翻譯而引起的任何誤解或誤釋不承擔責任。\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.7.4 64-bit (conda)",
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5"
|
||||
}
|
||||
},
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.5"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "4087f998407d06ceb2947016ba4605d0",
|
||||
"translation_date": "2025-08-28T11:29:38+00:00",
|
||||
"source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb",
|
||||
"language_code": "mo"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
|
|
@ -0,0 +1,183 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# 使用我們自己的框架進行 MNIST 數字分類\n",
|
||||
"\n",
|
||||
"來自 [AI 初學者課程](https://github.com/microsoft/ai-for-beginners) 的實驗作業。\n",
|
||||
"\n",
|
||||
"### 讀取數據集\n",
|
||||
"\n",
|
||||
"此程式碼從網路上的資料庫下載數據集。你也可以手動從 AI 課程資料庫的 `/data` 目錄中複製數據集。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" % Total % Received % Xferd Average Speed Time Time Time Current\n",
|
||||
" Dload Upload Total Spent Left Speed\n",
|
||||
"\n",
|
||||
" 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\n",
|
||||
"100 9.9M 100 9.9M 0 0 9.9M 0 0:00:01 --:--:-- 0:00:01 15.8M\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"!rm *.pkl\n",
|
||||
"!wget https://raw.githubusercontent.com/microsoft/AI-For-Beginners/main/data/mnist.pkl.gz\n",
|
||||
"!gzip -d mnist.pkl.gz"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import pickle\n",
|
||||
"with open('mnist.pkl','rb') as f:\n",
|
||||
" MNIST = pickle.load(f)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"labels = MNIST['Train']['Labels']\n",
|
||||
"data = MNIST['Train']['Features']"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"讓我們看看我們擁有的數據形狀：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(42000, 784)"
|
||||
]
|
||||
},
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"data.shape"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 分割資料\n",
|
||||
"\n",
|
||||
"我們將使用 Scikit Learn 將資料分割為訓練集和測試集：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Train samples: 33600, test samples: 8400\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from sklearn.model_selection import train_test_split\n",
|
||||
"\n",
|
||||
"features_train, features_test, labels_train, labels_test = train_test_split(data,labels,test_size=0.2)\n",
|
||||
"\n",
|
||||
"print(f\"Train samples: {len(features_train)}, test samples: {len(features_test)}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 指示\n",
|
||||
"\n",
|
||||
"1. 從課程中取得框架代碼，並將其貼到此筆記本中，或（更好）貼到一個獨立的 Python 模組中\n",
|
||||
"1. 定義並訓練單層感知器，觀察訓練和驗證的準確率\n",
|
||||
"1. 嘗試理解是否發生了過擬合，並調整層的參數以提高準確率\n",
|
||||
"1. 重複上述步驟，訓練兩層和三層感知器。嘗試在層之間使用不同的激活函數進行實驗\n",
|
||||
"1. 嘗試回答以下問題：\n",
|
||||
" - 層間的激活函數是否影響網絡性能？\n",
|
||||
" - 這個任務是否需要兩層或三層的網絡？\n",
|
||||
" - 在訓練網絡時是否遇到任何問題？特別是當層數增加時。\n",
|
||||
" - 網絡的權重在訓練過程中如何變化？你可以繪製權重最大絕對值與訓練輪次的關係圖來理解它們的關聯性。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**免責聲明**： \n本文件已使用 AI 翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。儘管我們努力確保翻譯的準確性，但請注意，自動翻譯可能包含錯誤或不準確之處。原始文件的母語版本應被視為權威來源。對於關鍵信息，建議使用專業人工翻譯。我們對因使用此翻譯而引起的任何誤解或錯誤解釋不承擔責任。\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.7.4 64-bit (conda)",
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5"
|
||||
}
|
||||
},
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.5"
|
||||
},
|
||||
"orig_nbformat": 2,
|
||||
"coopTranslator": {
|
||||
"original_hash": "6fa055f484eb5d6bdf41166a356d3abf",
|
||||
"translation_date": "2025-08-28T11:37:26+00:00",
|
||||
"source_file": "lessons/3-NeuralNetworks/04-OwnFramework/lab/MyFW_MNIST.ipynb",
|
||||
"language_code": "mo"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
|
|
@ -0,0 +1,108 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 使用光流技術檢測手掌移動\n",
|
||||
"\n",
|
||||
"此實驗是 [AI 初學者課程](http://aka.ms/ai-beginners) 的一部分。\n",
|
||||
"\n",
|
||||
"請參考[這段影片](../../../../../../lessons/4-ComputerVision/06-IntroCV/lab/palm-movement.mp4)，影片中一個人的手掌在穩定的背景上向左/右/上/下移動。\n",
|
||||
"\n",
|
||||
"**你的目標**是使用光流技術來判斷影片中哪些部分包含向上/向下/向左/向右的移動。\n",
|
||||
"\n",
|
||||
"首先，按照課堂中的描述獲取影片幀：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Code here"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"現在，按照講座中描述的計算密集光流幀，並將密集光流轉換為極座標：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Code here"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"為每個光流幀建立方向的直方圖。直方圖顯示有多少向量落在特定的區間內，並應該將幀中不同方向的運動區分開來。\n",
|
||||
"\n",
|
||||
"> 你可能也會想要將大小低於某個閾值的所有向量歸零。這樣可以去除影片中一些微小的多餘運動，例如眼睛和頭部的移動。\n",
|
||||
"\n",
|
||||
"繪製一些幀的直方圖。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Code here"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"看著直方圖，應該很容易判斷移動的方向。你需要選擇那些對應於上/下/左/右方向的區間，並且超過某個閾值的區間。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Code here"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"恭喜！如果您已完成以上所有步驟，您已完成實驗！\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**免責聲明**： \n本文件已使用 AI 翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。儘管我們努力確保翻譯的準確性，但請注意，自動翻譯可能包含錯誤或不準確之處。原始文件的母語版本應被視為權威來源。對於關鍵信息，建議使用專業人工翻譯。我們對因使用此翻譯而引起的任何誤解或誤釋不承擔責任。\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
},
|
||||
"orig_nbformat": 4,
|
||||
"coopTranslator": {
|
||||
"original_hash": "153d9e417e079bf62f8f693002d0deaf",
|
||||
"translation_date": "2025-08-28T11:10:32+00:00",
|
||||
"source_file": "lessons/4-ComputerVision/06-IntroCV/lab/MovementDetection.ipynb",
|
||||
"language_code": "mo"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
|
|
@ -0,0 +1,577 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# 文本分類任務\n",
|
||||
"\n",
|
||||
"如前所述，我們將專注於基於 **AG_NEWS** 數據集的簡單文本分類任務，目的是將新聞標題分類為以下四個類別之一：國際、體育、商業和科技。\n",
|
||||
"\n",
|
||||
"## 數據集\n",
|
||||
"\n",
|
||||
"此數據集已內建於 [`torchtext`](https://github.com/pytorch/text) 模組中，因此我們可以輕鬆存取它。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"import torchtext\n",
|
||||
"import os\n",
|
||||
"import collections\n",
|
||||
"os.makedirs('./data',exist_ok=True)\n",
|
||||
"train_dataset, test_dataset = torchtext.datasets.AG_NEWS(root='./data')\n",
|
||||
"classes = ['World', 'Sports', 'Business', 'Sci/Tech']"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"在這裡，`train_dataset` 和 `test_dataset` 包含分別返回標籤（類別數字）和文本的集合，例如：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(3,\n",
|
||||
" \"Wall St. Bears Claw Back Into the Black (Reuters) Reuters - Short-sellers, Wall Street's dwindling\\\\band of ultra-cynics, are seeing green again.\")"
|
||||
]
|
||||
},
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"list(train_dataset)[0]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"所以，讓我們列印出資料集中前10個新的標題：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"**Sci/Tech** -> Wall St. Bears Claw Back Into the Black (Reuters) Reuters - Short-sellers, Wall Street's dwindling\\band of ultra-cynics, are seeing green again.\n",
|
||||
"**Sci/Tech** -> Carlyle Looks Toward Commercial Aerospace (Reuters) Reuters - Private investment firm Carlyle Group,\\which has a reputation for making well-timed and occasionally\\controversial plays in the defense industry, has quietly placed\\its bets on another part of the market.\n",
|
||||
"**Sci/Tech** -> Oil and Economy Cloud Stocks' Outlook (Reuters) Reuters - Soaring crude prices plus worries\\about the economy and the outlook for earnings are expected to\\hang over the stock market next week during the depth of the\\summer doldrums.\n",
|
||||
"**Sci/Tech** -> Iraq Halts Oil Exports from Main Southern Pipeline (Reuters) Reuters - Authorities have halted oil export\\flows from the main pipeline in southern Iraq after\\intelligence showed a rebel militia could strike\\infrastructure, an oil official said on Saturday.\n",
|
||||
"**Sci/Tech** -> Oil prices soar to all-time record, posing new menace to US economy (AFP) AFP - Tearaway world oil prices, toppling records and straining wallets, present a new economic menace barely three months before the US presidential elections.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"for i,x in zip(range(5),train_dataset):\n",
|
||||
" print(f\"**{classes[x[0]]}** -> {x[1]}\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"因為數據集是迭代器，如果我們想多次使用數據，我們需要將其轉換為列表：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"train_dataset, test_dataset = torchtext.datasets.AG_NEWS(root='./data')\n",
|
||||
"train_dataset = list(train_dataset)\n",
|
||||
"test_dataset = list(test_dataset)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 分詞\n",
|
||||
"\n",
|
||||
"現在我們需要將文本轉換成可以表示為張量的**數字**。如果我們想要詞級表示，需要完成以下兩件事：\n",
|
||||
"* 使用**分詞器**將文本拆分成**詞元**\n",
|
||||
"* 建立這些詞元的**詞彙表**。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"['he', 'said', 'hello']"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"tokenizer = torchtext.data.utils.get_tokenizer('basic_english')\n",
|
||||
"tokenizer('He said: hello')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"counter = collections.Counter()\n",
|
||||
"for (label, line) in train_dataset:\n",
|
||||
" counter.update(tokenizer(line))\n",
|
||||
"vocab = torchtext.vocab.vocab(counter, min_freq=1)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"使用詞彙，我們可以輕鬆地將標記化的字串編碼為一組數字：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 19,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Vocab size if 95810\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[599, 3279, 97, 1220, 329, 225, 7368]"
|
||||
]
|
||||
},
|
||||
"execution_count": 19,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vocab_size = len(vocab)\n",
|
||||
"print(f\"Vocab size if {vocab_size}\")\n",
|
||||
"\n",
|
||||
"stoi = vocab.get_stoi() # dict to convert tokens to indices\n",
|
||||
"\n",
|
||||
"def encode(x):\n",
|
||||
" return [stoi[s] for s in tokenizer(x)]\n",
|
||||
"\n",
|
||||
"encode('I love to play with my words')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 詞袋文字表示法\n",
|
||||
"\n",
|
||||
"由於文字代表了意義，有時我們可以僅通過查看單個詞語來理解文本的含義，而不考慮它們在句子中的順序。例如，在分類新聞時，像 *天氣*、*雪* 這樣的詞語可能表明是 *天氣預報*，而像 *股票*、*美元* 這樣的詞語則可能屬於 *財經新聞*。\n",
|
||||
"\n",
|
||||
"**詞袋** (BoW) 向量表示法是最常用的傳統向量表示法。每個詞語都與一個向量索引相關聯，向量元素包含某個詞語在特定文檔中出現的次數。\n",
|
||||
"\n",
|
||||
" \n",
|
||||
"\n",
|
||||
"> **Note**: 你也可以將 BoW 理解為文本中每個詞語的單熱編碼向量的總和。\n",
|
||||
"\n",
|
||||
"以下是一個使用 Scikit Learn Python 庫生成詞袋表示法的範例：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([[1, 1, 0, 2, 0, 0, 0, 0, 0]], dtype=int64)"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from sklearn.feature_extraction.text import CountVectorizer\n",
|
||||
"vectorizer = CountVectorizer()\n",
|
||||
"corpus = [\n",
|
||||
" 'I like hot dogs.',\n",
|
||||
" 'The dog ran fast.',\n",
|
||||
" 'Its hot outside.',\n",
|
||||
" ]\n",
|
||||
"vectorizer.fit_transform(corpus)\n",
|
||||
"vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"要從我們的 AG_NEWS 數據集的向量表示計算詞袋向量，可以使用以下函數：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 20,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"tensor([2., 1., 2., ..., 0., 0., 0.])\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vocab_size = len(vocab)\n",
|
||||
"\n",
|
||||
"def to_bow(text,bow_vocab_size=vocab_size):\n",
|
||||
" res = torch.zeros(bow_vocab_size,dtype=torch.float32)\n",
|
||||
" for i in encode(text):\n",
|
||||
" if i<bow_vocab_size:\n",
|
||||
" res[i] += 1\n",
|
||||
" return res\n",
|
||||
"\n",
|
||||
"print(to_bow(train_dataset[0][1]))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **注意：** 這裡我們使用全域變數 `vocab_size` 來指定詞彙表的預設大小。由於詞彙表的大小通常相當大，我們可以將詞彙表的大小限制為最常出現的詞彙。嘗試降低 `vocab_size` 的值並執行下面的程式碼，看看它如何影響準確性。你應該預期準確性會有所下降，但不會太劇烈，以換取更高的效能。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 訓練 BoW 分類器\n",
|
||||
"\n",
|
||||
"現在我們已經學會如何建立文字的詞袋表示法，接下來讓我們在其基礎上訓練一個分類器。首先，我們需要將數據集轉換為適合訓練的格式，將所有位置向量表示轉換為詞袋表示法。這可以通過將 `bowify` 函數作為標準 torch `DataLoader` 的 `collate_fn` 參數來實現：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 21,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from torch.utils.data import DataLoader\n",
|
||||
"import numpy as np \n",
|
||||
"\n",
|
||||
"# this collate function gets list of batch_size tuples, and needs to \n",
|
||||
"# return a pair of label-feature tensors for the whole minibatch\n",
|
||||
"def bowify(b):\n",
|
||||
" return (\n",
|
||||
" torch.LongTensor([t[0]-1 for t in b]),\n",
|
||||
" torch.stack([to_bow(t[1]) for t in b])\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"train_loader = DataLoader(train_dataset, batch_size=16, collate_fn=bowify, shuffle=True)\n",
|
||||
"test_loader = DataLoader(test_dataset, batch_size=16, collate_fn=bowify, shuffle=True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"現在讓我們定義一個簡單的分類器神經網絡，其中包含一個線性層。輸入向量的大小等於 `vocab_size`，輸出大小對應於類別數量（4）。由於我們正在解決分類任務，最終的激活函數是 `LogSoftmax()`。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 22,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"net = torch.nn.Sequential(torch.nn.Linear(vocab_size,4),torch.nn.LogSoftmax(dim=1))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"現在我們將定義標準的 PyTorch 訓練迴圈。由於我們的數據集相當大，為了教學目的，我們只訓練一個 epoch，有時甚至少於一個 epoch（指定 `epoch_size` 參數可以限制訓練）。我們還會在訓練過程中報告累積的訓練準確率；報告的頻率是使用 `report_freq` 參數指定的。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 24,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def train_epoch(net,dataloader,lr=0.01,optimizer=None,loss_fn = torch.nn.NLLLoss(),epoch_size=None, report_freq=200):\n",
|
||||
" optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr)\n",
|
||||
" net.train()\n",
|
||||
" total_loss,acc,count,i = 0,0,0,0\n",
|
||||
" for labels,features in dataloader:\n",
|
||||
" optimizer.zero_grad()\n",
|
||||
" out = net(features)\n",
|
||||
" loss = loss_fn(out,labels) #cross_entropy(out,labels)\n",
|
||||
" loss.backward()\n",
|
||||
" optimizer.step()\n",
|
||||
" total_loss+=loss\n",
|
||||
" _,predicted = torch.max(out,1)\n",
|
||||
" acc+=(predicted==labels).sum()\n",
|
||||
" count+=len(labels)\n",
|
||||
" i+=1\n",
|
||||
" if i%report_freq==0:\n",
|
||||
" print(f\"{count}: acc={acc.item()/count}\")\n",
|
||||
" if epoch_size and count>epoch_size:\n",
|
||||
" break\n",
|
||||
" return total_loss.item()/count, acc.item()/count"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 25,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.8028125\n",
|
||||
"6400: acc=0.8371875\n",
|
||||
"9600: acc=0.8534375\n",
|
||||
"12800: acc=0.85765625\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(0.026090790722161722, 0.8620069296375267)"
|
||||
]
|
||||
},
|
||||
"execution_count": 25,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"train_epoch(net,train_loader,epoch_size=15000)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 雙詞組、三詞組與 N 詞組\n",
|
||||
"\n",
|
||||
"詞袋模型的一個限制是，有些詞是多詞表達的一部分。例如，「熱狗」這個詞的意思與「熱」和「狗」在其他語境中的意思完全不同。如果我們總是用相同的向量來表示「熱」和「狗」，可能會讓模型感到困惑。\n",
|
||||
"\n",
|
||||
"為了解決這個問題，**N 詞組表示法**經常被用於文件分類的方法中，其中每個單詞、雙詞或三詞的頻率是訓練分類器的一個有用特徵。例如，在雙詞組表示法中，除了原始單詞之外，我們還會將所有的單詞對加入詞彙表中。\n",
|
||||
"\n",
|
||||
"以下是一個使用 Scikit Learn 生成雙詞組詞袋表示法的範例：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 26,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Vocabulary:\n",
|
||||
" {'i': 7, 'like': 11, 'hot': 4, 'dogs': 2, 'i like': 8, 'like hot': 12, 'hot dogs': 5, 'the': 16, 'dog': 0, 'ran': 14, 'fast': 3, 'the dog': 17, 'dog ran': 1, 'ran fast': 15, 'its': 9, 'outside': 13, 'its hot': 10, 'hot outside': 6}\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([[1, 0, 1, 0, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],\n",
|
||||
" dtype=int64)"
|
||||
]
|
||||
},
|
||||
"execution_count": 26,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"bigram_vectorizer = CountVectorizer(ngram_range=(1, 2), token_pattern=r'\\b\\w+\\b', min_df=1)\n",
|
||||
"corpus = [\n",
|
||||
" 'I like hot dogs.',\n",
|
||||
" 'The dog ran fast.',\n",
|
||||
" 'Its hot outside.',\n",
|
||||
" ]\n",
|
||||
"bigram_vectorizer.fit_transform(corpus)\n",
|
||||
"print(\"Vocabulary:\\n\",bigram_vectorizer.vocabulary_)\n",
|
||||
"bigram_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"N-gram 方法的主要缺點是詞彙量會迅速增長。在實際應用中，我們需要將 N-gram 表示法與一些降維技術結合使用，例如 *嵌入*，我們會在下一單元中討論。\n",
|
||||
"\n",
|
||||
"要在我們的 **AG News** 數據集中使用 N-gram 表示法，我們需要建立專門的 ngram 詞彙：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 27,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Bigram vocabulary length = 1308842\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"counter = collections.Counter()\n",
|
||||
"for (label, line) in train_dataset:\n",
|
||||
" l = tokenizer(line)\n",
|
||||
" counter.update(torchtext.data.utils.ngrams_iterator(l,ngrams=2))\n",
|
||||
" \n",
|
||||
"bi_vocab = torchtext.vocab.vocab(counter, min_freq=1)\n",
|
||||
"\n",
|
||||
"print(\"Bigram vocabulary length = \",len(bi_vocab))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"我們可以使用上述相同的程式碼來訓練分類器，但這樣做會非常佔用記憶體。在下一單元中，我們將使用嵌入來訓練雙詞組分類器。\n",
|
||||
"\n",
|
||||
"> **注意：** 你只能保留那些在文本中出現次數超過指定數量的 ngrams。這樣可以確保罕見的雙詞組會被省略，並顯著減少維度。為此，將 `min_freq` 參數設置為更高的值，並觀察詞彙表長度的變化。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 詞頻-逆文檔頻率 TF-IDF\n",
|
||||
"\n",
|
||||
"在 BoW 表示法中，詞的出現次數被均等地加權，而不考慮詞本身的特性。然而，很明顯像 *a*、*in* 等這些常見詞對分類的影響遠不如專業術語重要。事實上，在大多數 NLP 任務中，有些詞比其他詞更具相關性。\n",
|
||||
"\n",
|
||||
"**TF-IDF** 代表 **詞頻–逆文檔頻率**。它是袋子模型（BoW）的變體，與使用二進制 0/1 值表示詞在文檔中的出現不同，TF-IDF 使用浮點值，該值與詞在語料庫中的出現頻率相關。\n",
|
||||
"\n",
|
||||
"更正式地說，詞 $i$ 在文檔 $j$ 中的權重 $w_{ij}$ 定義如下：\n",
|
||||
"$$\n",
|
||||
"w_{ij} = tf_{ij}\\times\\log({N\\over df_i})\n",
|
||||
"$$\n",
|
||||
"其中：\n",
|
||||
"* $tf_{ij}$ 是詞 $i$ 在文檔 $j$ 中的出現次數，也就是我們之前看到的 BoW 值\n",
|
||||
"* $N$ 是語料庫中的文檔總數\n",
|
||||
"* $df_i$ 是包含詞 $i$ 的文檔數量\n",
|
||||
"\n",
|
||||
"TF-IDF 值 $w_{ij}$ 與詞在文檔中出現的次數成正比，但會根據語料庫中包含該詞的文檔數量進行調整。這有助於平衡某些詞比其他詞更頻繁出現的情況。例如，如果某個詞出現在語料庫的*每一個*文檔中，則 $df_i=N$，而 $w_{ij}=0$，這些詞將被完全忽略。\n",
|
||||
"\n",
|
||||
"您可以使用 Scikit Learn 輕鬆地生成文本的 TF-IDF 向量化：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 28,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([[0.43381609, 0. , 0.43381609, 0. , 0.65985664,\n",
|
||||
" 0.43381609, 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. ]])"
|
||||
]
|
||||
},
|
||||
"execution_count": 28,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
|
||||
"vectorizer = TfidfVectorizer(ngram_range=(1,2))\n",
|
||||
"vectorizer.fit_transform(corpus)\n",
|
||||
"vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 結論\n",
|
||||
"\n",
|
||||
"然而，儘管 TF-IDF 表示法為不同的詞提供了頻率權重，但它無法表達詞義或順序。正如著名語言學家 J. R. Firth 在1935年所說：「詞語的完整意義總是與上下文相關，任何脫離上下文的意義研究都不應被認真對待。」在課程的後續部分，我們將學習如何通過語言建模從文本中捕捉上下文信息。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**免責聲明**： \n本文件已使用 AI 翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。儘管我們努力確保翻譯的準確性，但請注意，自動翻譯可能包含錯誤或不準確之處。原始文件的母語版本應被視為權威來源。對於關鍵信息，建議使用專業人工翻譯。我們對因使用此翻譯而引起的任何誤解或錯誤解釋不承擔責任。\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.8.12 ('py38')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "7b9040985e748e4e2d4c689892456ad7",
|
||||
"translation_date": "2025-08-28T12:32:40+00:00",
|
||||
"source_file": "lessons/5-NLP/13-TextRep/TextRepresentationPyTorch.ipynb",
|
||||
"language_code": "mo"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
|
|
@ -0,0 +1,647 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# 文本分類任務\n",
|
||||
"\n",
|
||||
"在本模組中，我們將從一個簡單的文本分類任務開始，基於 **[AG_NEWS](http://www.di.unipi.it/~gulli/AG_corpus_of_news_articles.html)** 數據集：我們將把新聞標題分類為以下四個類別之一：世界、體育、商業和科學/技術。\n",
|
||||
"\n",
|
||||
"## 數據集\n",
|
||||
"\n",
|
||||
"為了載入數據集，我們將使用 **[TensorFlow Datasets](https://www.tensorflow.org/datasets)** API。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import tensorflow as tf\n",
|
||||
"from tensorflow import keras\n",
|
||||
"import tensorflow_datasets as tfds\n",
|
||||
"\n",
|
||||
"# In this tutorial, we will be training a lot of models. In order to use GPU memory cautiously,\n",
|
||||
"# we will set tensorflow option to grow GPU memory allocation when required.\n",
|
||||
"physical_devices = tf.config.list_physical_devices('GPU') \n",
|
||||
"if len(physical_devices)>0:\n",
|
||||
" tf.config.experimental.set_memory_growth(physical_devices[0], True)\n",
|
||||
"\n",
|
||||
"dataset = tfds.load('ag_news_subset')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"我們現在可以分別使用 `dataset['train']` 和 `dataset['test']` 訪問數據集的訓練和測試部分：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Length of train dataset = 120000\n",
|
||||
"Length of test dataset = 7600\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"ds_train = dataset['train']\n",
|
||||
"ds_test = dataset['test']\n",
|
||||
"\n",
|
||||
"print(f\"Length of train dataset = {len(ds_train)}\")\n",
|
||||
"print(f\"Length of test dataset = {len(ds_test)}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"讓我們列印出資料集中前10個新的標題：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3 (Sci/Tech) -> b'AMD Debuts Dual-Core Opteron Processor' b'AMD #39;s new dual-core Opteron chip is designed mainly for corporate computing applications, including databases, Web services, and financial transactions.'\n",
|
||||
"1 (Sports) -> b\"Wood's Suspension Upheld (Reuters)\" b'Reuters - Major League Baseball\\\\Monday announced a decision on the appeal filed by Chicago Cubs\\\\pitcher Kerry Wood regarding a suspension stemming from an\\\\incident earlier this season.'\n",
|
||||
"2 (Business) -> b'Bush reform may have blue states seeing red' b'President Bush #39;s quot;revenue-neutral quot; tax reform needs losers to balance its winners, and people claiming the federal deduction for state and local taxes may be in administration planners #39; sights, news reports say.'\n",
|
||||
"3 (Sci/Tech) -> b\"'Halt science decline in schools'\" b'Britain will run out of leading scientists unless science education is improved, says Professor Colin Pillinger.'\n",
|
||||
"1 (Sports) -> b'Gerrard leaves practice' b'London, England (Sports Network) - England midfielder Steven Gerrard injured his groin late in Thursday #39;s training session, but is hopeful he will be ready for Saturday #39;s World Cup qualifier against Austria.'\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"classes = ['World', 'Sports', 'Business', 'Sci/Tech']\n",
|
||||
"\n",
|
||||
"for i,x in zip(range(5),ds_train):\n",
|
||||
" print(f\"{x['label']} ({classes[x['label']]}) -> {x['title']} {x['description']}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 文本向量化\n",
|
||||
"\n",
|
||||
"現在我們需要將文本轉換成可以表示為張量的**數字**。如果我們想要詞級表示，需要完成以下兩件事：\n",
|
||||
"\n",
|
||||
"* 使用**分詞器**將文本拆分成**詞元**。\n",
|
||||
"* 建立這些詞元的**詞彙表**。\n",
|
||||
"\n",
|
||||
"### 限制詞彙表大小\n",
|
||||
"\n",
|
||||
"在 AG News 數據集的例子中，詞彙表的大小相當大，超過 10 萬個詞。一般來說，我們不需要那些在文本中很少出現的詞——只有少數句子會包含它們，而模型無法從中學習。因此，通過向向量化器構造函數傳遞參數，限制詞彙表大小到一個較小的數量是合理的。\n",
|
||||
"\n",
|
||||
"以上兩個步驟都可以使用 **TextVectorization** 層來處理。接下來，我們來實例化向量化器對象，然後調用 `adapt` 方法，遍歷所有文本並建立詞彙表：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"vocab_size = 50000\n",
|
||||
"vectorizer = keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size)\n",
|
||||
"vectorizer.adapt(ds_train.take(500).map(lambda x: x['title']+' '+x['description']))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **注意** 我們僅使用整個數據集的一部分來建立詞彙表。這樣做是為了加快執行速度，避免讓您等待太久。然而，這樣做也存在一定風險，即整個數據集中的某些詞可能不會被包含在詞彙表中，並在訓練過程中被忽略。因此，使用完整的詞彙表大小並在 `adapt` 過程中遍歷整個數據集，應該能提升最終的準確性，但提升幅度不會太大。\n",
|
||||
"\n",
|
||||
"現在我們可以訪問實際的詞彙表：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"['', '[UNK]', 'the', 'to', 'a', 'in', 'of', 'and', 'on', 'for']\n",
|
||||
"Length of vocabulary: 5335\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vocab = vectorizer.get_vocabulary()\n",
|
||||
"vocab_size = len(vocab)\n",
|
||||
"print(vocab[:10])\n",
|
||||
"print(f\"Length of vocabulary: {vocab_size}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"使用向量化器，我們可以輕鬆地將任何文本編碼為一組數字：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tf.Tensor: shape=(7,), dtype=int64, numpy=array([ 112, 3695, 3, 304, 11, 1041, 1], dtype=int64)>"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vectorizer('I love to play with my words')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 詞袋（Bag-of-words）文本表示法\n",
|
||||
"\n",
|
||||
"由於文字具有意義，有時我們可以僅通過查看單個詞語來理解一段文本的含義，而不需要考慮它們在句子中的順序。例如，在分類新聞時，像 *weather* 和 *snow* 這樣的詞語可能表明這是與 *天氣預報* 有關的內容，而像 *stocks* 和 *dollar* 則可能屬於 *財經新聞*。\n",
|
||||
"\n",
|
||||
"**詞袋**（Bag-of-words, BoW）向量表示法是最簡單易懂的傳統向量表示法。每個詞語都對應到向量中的一個索引，而向量中的元素則表示該詞語在特定文檔中出現的次數。\n",
|
||||
"\n",
|
||||
" \n",
|
||||
"\n",
|
||||
"> **注意**：你也可以將 BoW 理解為文本中每個詞語的單熱編碼（one-hot-encoded）向量的總和。\n",
|
||||
"\n",
|
||||
"以下是一個使用 Scikit Learn Python 庫生成詞袋表示法的範例：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([[1, 1, 0, 2, 0, 0, 0, 0, 0]], dtype=int64)"
|
||||
]
|
||||
},
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from sklearn.feature_extraction.text import CountVectorizer\n",
|
||||
"sc_vectorizer = CountVectorizer()\n",
|
||||
"corpus = [\n",
|
||||
" 'I like hot dogs.',\n",
|
||||
" 'The dog ran fast.',\n",
|
||||
" 'Its hot outside.',\n",
|
||||
" ]\n",
|
||||
"sc_vectorizer.fit_transform(corpus)\n",
|
||||
"sc_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"我們也可以使用我們在上面定義的 Keras 向量化器，將每個單詞編號轉換為一個獨熱編碼，然後將所有這些向量相加：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([0., 5., 0., ..., 0., 0., 0.], dtype=float32)"
|
||||
]
|
||||
},
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def to_bow(text):\n",
|
||||
" return tf.reduce_sum(tf.one_hot(vectorizer(text),vocab_size),axis=0)\n",
|
||||
"\n",
|
||||
"to_bow('My dog likes hot dogs on a hot day.').numpy()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **注意**：您可能會驚訝地發現結果與之前的例子不同。原因是在 Keras 的例子中，向量的長度對應於詞彙表的大小，而該詞彙表是基於整個 AG News 數據集構建的；而在 Scikit Learn 的例子中，我們是即時從樣本文本中構建詞彙表的。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 訓練 BoW 分類器\n",
|
||||
"\n",
|
||||
"現在我們已經學會如何建立文字的詞袋表示法，接下來讓我們訓練一個使用該表示法的分類器。首先，我們需要將數據集轉換為詞袋表示法。這可以通過以下方式使用 `map` 函數來實現：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"batch_size = 128\n",
|
||||
"\n",
|
||||
"ds_train_bow = ds_train.map(lambda x: (to_bow(x['title']+x['description']),x['label'])).batch(batch_size)\n",
|
||||
"ds_test_bow = ds_test.map(lambda x: (to_bow(x['title']+x['description']),x['label'])).batch(batch_size)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"現在讓我們定義一個簡單的分類器神經網絡，其中包含一個線性層。輸入大小為 `vocab_size`，輸出大小對應於類別數量（4）。由於我們正在解決分類任務，最終的激活函數是 **softmax**：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"938/938 [==============================] - 66s 70ms/step - loss: 0.6144 - acc: 0.8427 - val_loss: 0.4416 - val_acc: 0.8697\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.History at 0x20c70a947f0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model = keras.models.Sequential([\n",
|
||||
" keras.layers.Dense(4,activation='softmax',input_shape=(vocab_size,))\n",
|
||||
"])\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
|
||||
"model.fit(ds_train_bow,validation_data=ds_test_bow)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"由於我們有四個類別，準確率超過 80% 就是一個不錯的結果。\n",
|
||||
"\n",
|
||||
"## 將分類器作為一個網絡進行訓練\n",
|
||||
"\n",
|
||||
"因為向量化器也是一個 Keras 層，我們可以定義一個包含它的網絡，並進行端到端的訓練。這樣我們就不需要使用 `map` 來向量化數據集，只需將原始數據集傳遞到網絡的輸入即可。\n",
|
||||
"\n",
|
||||
"> **注意**：我們仍然需要對數據集應用映射操作，將字典中的字段（例如 `title`、`description` 和 `label`）轉換為元組。然而，當從磁盤加載數據時，我們可以一開始就構建具有所需結構的數據集。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"model\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
" Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
" input_1 (InputLayer) [(None, 1)] 0 \n",
|
||||
" \n",
|
||||
" text_vectorization (TextVec (None, None) 0 \n",
|
||||
" torization) \n",
|
||||
" \n",
|
||||
" tf.one_hot (TFOpLambda) (None, None, 5335) 0 \n",
|
||||
" \n",
|
||||
" tf.math.reduce_sum (TFOpLam (None, 5335) 0 \n",
|
||||
" bda) \n",
|
||||
" \n",
|
||||
" dense_2 (Dense) (None, 4) 21344 \n",
|
||||
" \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 21,344\n",
|
||||
"Trainable params: 21,344\n",
|
||||
"Non-trainable params: 0\n",
|
||||
"_________________________________________________________________\n",
|
||||
"938/938 [==============================] - 73s 77ms/step - loss: 0.6057 - acc: 0.8414 - val_loss: 0.4202 - val_acc: 0.8736\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.History at 0x20c721521f0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def extract_text(x):\n",
|
||||
" return x['title']+' '+x['description']\n",
|
||||
"\n",
|
||||
"def tupelize(x):\n",
|
||||
" return (extract_text(x),x['label'])\n",
|
||||
"\n",
|
||||
"inp = keras.Input(shape=(1,),dtype=tf.string)\n",
|
||||
"x = vectorizer(inp)\n",
|
||||
"x = tf.reduce_sum(tf.one_hot(x,vocab_size),axis=1)\n",
|
||||
"out = keras.layers.Dense(4,activation='softmax')(x)\n",
|
||||
"model = keras.models.Model(inp,out)\n",
|
||||
"model.summary()\n",
|
||||
"\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 二元詞組、三元詞組與 n 元詞組\n",
|
||||
"\n",
|
||||
"袋裝詞語方法的一個限制是，有些詞語屬於多詞組表達，例如「熱狗」這個詞的意思與「熱」和「狗」在其他語境中的意思完全不同。如果我們始終用相同的向量來表示「熱」和「狗」，可能會讓模型感到困惑。\n",
|
||||
"\n",
|
||||
"為了解決這個問題，**n 元詞組表示法**通常用於文件分類方法中，其中每個詞語、二詞組或三詞組的頻率都是訓練分類器的有用特徵。例如，在二元詞組表示法中，我們會將所有的詞語對加入詞彙表中，除了原始詞語之外。\n",
|
||||
"\n",
|
||||
"以下是一個使用 Scikit Learn 生成二元詞組袋裝詞語表示法的範例：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Vocabulary:\n",
|
||||
" {'i': 7, 'like': 11, 'hot': 4, 'dogs': 2, 'i like': 8, 'like hot': 12, 'hot dogs': 5, 'the': 16, 'dog': 0, 'ran': 14, 'fast': 3, 'the dog': 17, 'dog ran': 1, 'ran fast': 15, 'its': 9, 'outside': 13, 'its hot': 10, 'hot outside': 6}\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([[1, 0, 1, 0, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],\n",
|
||||
" dtype=int64)"
|
||||
]
|
||||
},
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"bigram_vectorizer = CountVectorizer(ngram_range=(1, 2), token_pattern=r'\\b\\w+\\b', min_df=1)\n",
|
||||
"corpus = [\n",
|
||||
" 'I like hot dogs.',\n",
|
||||
" 'The dog ran fast.',\n",
|
||||
" 'Its hot outside.',\n",
|
||||
" ]\n",
|
||||
"bigram_vectorizer.fit_transform(corpus)\n",
|
||||
"print(\"Vocabulary:\\n\",bigram_vectorizer.vocabulary_)\n",
|
||||
"bigram_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"n-gram 方法的主要缺點是詞彙表的大小會開始以極快的速度增長。實際操作中，我們需要將 n-gram 表示法與降維技術（例如 *嵌入*）結合使用，我們將在下一單元中討論這一點。\n",
|
||||
"\n",
|
||||
"要在我們的 **AG News** 數據集中使用 n-gram 表示法，我們需要將 `ngrams` 參數傳遞給 `TextVectorization` 構造函數。二元語法詞彙表的長度**顯著更大**，在我們的例子中，它超過了 130 萬個詞元！因此，限制二元語法詞元的數量在某個合理範圍內是有意義的。\n",
|
||||
"\n",
|
||||
"我們可以使用與上面相同的代碼來訓練分類器，但這樣做會非常浪費記憶體。在下一單元中，我們將使用嵌入來訓練二元語法分類器。與此同時，你可以在這個 notebook 中嘗試訓練二元語法分類器，看看是否能獲得更高的準確率。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 自動計算 BoW 向量\n",
|
||||
"\n",
|
||||
"在上面的例子中，我們透過手動方式計算 BoW 向量，方法是將個別單詞的一次性編碼相加。然而，最新版本的 TensorFlow 允許我們透過在向量化器構造函數中傳入 `output_mode='count` 參數，自動計算 BoW 向量。這使得定義和訓練模型變得更加簡單：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Training vectorizer\n",
|
||||
"938/938 [==============================] - 7s 7ms/step - loss: 0.5929 - acc: 0.8486 - val_loss: 0.4168 - val_acc: 0.8772\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.History at 0x20c725217c0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model = keras.models.Sequential([\n",
|
||||
" keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size,output_mode='count'),\n",
|
||||
" keras.layers.Dense(4,input_shape=(vocab_size,), activation='softmax')\n",
|
||||
"])\n",
|
||||
"print(\"Training vectorizer\")\n",
|
||||
"model.layers[0].adapt(ds_train.take(500).map(extract_text))\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 詞頻 - 逆文件頻率 (TF-IDF)\n",
|
||||
"\n",
|
||||
"在詞袋表示法中，詞的出現次數使用相同的技術進行加權，而不考慮詞本身。然而，很明顯像 *a* 和 *in* 這樣的常見詞對分類的作用遠不如專業術語。在大多數自然語言處理任務中，有些詞比其他詞更具相關性。\n",
|
||||
"\n",
|
||||
"**TF-IDF** 代表 **詞頻 - 逆文件頻率**。這是一種詞袋的變體，其中不是用二進制的 0/1 值來表示詞在文件中的出現，而是使用浮點值，該值與詞在語料庫中的出現頻率相關。\n",
|
||||
"\n",
|
||||
"更正式地說，詞 $i$ 在文件 $j$ 中的權重 $w_{ij}$ 定義為：\n",
|
||||
"$$\n",
|
||||
"w_{ij} = tf_{ij}\\times\\log({N\\over df_i})\n",
|
||||
"$$\n",
|
||||
"其中：\n",
|
||||
"* $tf_{ij}$ 是詞 $i$ 在文件 $j$ 中的出現次數，也就是我們之前看到的詞袋值\n",
|
||||
"* $N$ 是集合中的文件數量\n",
|
||||
"* $df_i$ 是包含詞 $i$ 的文件數量\n",
|
||||
"\n",
|
||||
"TF-IDF 值 $w_{ij}$ 與詞在文件中出現的次數成正比，並且會根據語料庫中包含該詞的文件數量進行調整，這有助於平衡某些詞出現頻率較高的情況。例如，如果某個詞出現在集合中的 *每一個* 文件中，則 $df_i=N$，而 $w_{ij}=0$，這些詞將被完全忽略。\n",
|
||||
"\n",
|
||||
"您可以使用 Scikit Learn 輕鬆地創建文本的 TF-IDF 向量化：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 16,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([[0.43381609, 0. , 0.43381609, 0. , 0.65985664,\n",
|
||||
" 0.43381609, 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. ]])"
|
||||
]
|
||||
},
|
||||
"execution_count": 16,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
|
||||
"vectorizer = TfidfVectorizer(ngram_range=(1,2))\n",
|
||||
"vectorizer.fit_transform(corpus)\n",
|
||||
"vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"在 Keras 中，`TextVectorization` 層可以通過傳遞 `output_mode='tf-idf'` 參數自動計算 TF-IDF 頻率。我們重複上面使用的代碼來看看使用 TF-IDF 是否能提高準確性：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 17,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Training vectorizer\n",
|
||||
"938/938 [==============================] - 12s 12ms/step - loss: 0.4197 - acc: 0.8662 - val_loss: 0.3432 - val_acc: 0.8849\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.History at 0x20c729dfd30>"
|
||||
]
|
||||
},
|
||||
"execution_count": 17,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model = keras.models.Sequential([\n",
|
||||
" keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size,output_mode='tf-idf'),\n",
|
||||
" keras.layers.Dense(4,input_shape=(vocab_size,), activation='softmax')\n",
|
||||
"])\n",
|
||||
"print(\"Training vectorizer\")\n",
|
||||
"model.layers[0].adapt(ds_train.take(500).map(extract_text))\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 結論\n",
|
||||
"\n",
|
||||
"儘管 TF-IDF 表示法為不同的詞語提供了頻率權重，但它無法表達詞語的意義或順序。正如著名語言學家 J. R. Firth 在 1935 年所說：「一個詞語的完整意義總是與上下文相關，任何脫離上下文的意義研究都不應被認真對待。」在課程的後續部分，我們將學習如何通過語言模型從文本中捕捉上下文信息。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**免責聲明**： \n本文件已使用 AI 翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。儘管我們努力確保翻譯的準確性，但請注意，自動翻譯可能包含錯誤或不準確之處。原始文件的母語版本應被視為權威來源。對於關鍵信息，建議尋求專業人工翻譯。我們對因使用此翻譯而引起的任何誤解或錯誤解釋不承擔責任。\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5"
|
||||
},
|
||||
"kernel_info": {
|
||||
"name": "conda-env-py37_tensorflow-py"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "py37_tensorflow",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"nteract": {
|
||||
"version": "nteract-front-end@1.0.0"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "19b43951d55b377a76209c24c1f017e4",
|
||||
"translation_date": "2025-08-28T12:35:41+00:00",
|
||||
"source_file": "lessons/5-NLP/13-TextRep/TextRepresentationTF.ipynb",
|
||||
"language_code": "mo"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
|
|
@ -0,0 +1,724 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 嵌入\n",
|
||||
"\n",
|
||||
"在我們之前的例子中，我們使用了高維度的詞袋向量，其長度為 `vocab_size`，並且我們明確地將低維度的位置信息向量轉換為稀疏的獨熱表示。這種獨熱表示並不具備記憶效率，此外，每個詞都被獨立地處理，也就是說，獨熱編碼的向量無法表達詞與詞之間的語義相似性。\n",
|
||||
"\n",
|
||||
"在本單元中，我們將繼續探索 **News AG** 數據集。首先，讓我們載入數據並從之前的筆記本中獲取一些定義。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Loading dataset...\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"d:\\WORK\\ai-for-beginners\\5-NLP\\14-Embeddings\\data\\train.csv: 29.5MB [00:01, 18.8MB/s] \n",
|
||||
"d:\\WORK\\ai-for-beginners\\5-NLP\\14-Embeddings\\data\\test.csv: 1.86MB [00:00, 11.2MB/s] \n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Building vocab...\n",
|
||||
"Vocab size = 95812\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"import torchtext\n",
|
||||
"import numpy as np\n",
|
||||
"from torchnlp import *\n",
|
||||
"train_dataset, test_dataset, classes, vocab = load_dataset()\n",
|
||||
"vocab_size = len(vocab)\n",
|
||||
"print(\"Vocab size = \",vocab_size)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 什麼是嵌入？\n",
|
||||
"\n",
|
||||
"**嵌入**的概念是用低維度的密集向量來表示單詞，這些向量能夠在某種程度上反映單詞的語義。我們稍後會討論如何構建有意義的單詞嵌入，但現在我們可以將嵌入簡單地理解為一種降低單詞向量維度的方法。\n",
|
||||
"\n",
|
||||
"因此，嵌入層會將一個單詞作為輸入，並生成指定 `embedding_size` 的輸出向量。從某種意義上說，它與 `Linear` 層非常相似，但不同的是，它不需要接受 one-hot 編碼的向量，而是可以直接接受單詞的編號作為輸入。\n",
|
||||
"\n",
|
||||
"通過將嵌入層作為我們網絡的第一層，我們可以從詞袋模型（bag-of-words）切換到 **嵌入袋模型**（embedding bag model）。在這種模型中，我們首先將文本中的每個單詞轉換為對應的嵌入向量，然後對所有這些嵌入向量執行某種聚合函數，例如 `sum`、`average` 或 `max`。\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"我們的分類器神經網絡將以嵌入層開始，接著是聚合層，最後在其上添加一個線性分類器：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class EmbedClassifier(torch.nn.Module):\n",
|
||||
" def __init__(self, vocab_size, embed_dim, num_class):\n",
|
||||
" super().__init__()\n",
|
||||
" self.embedding = torch.nn.Embedding(vocab_size, embed_dim)\n",
|
||||
" self.fc = torch.nn.Linear(embed_dim, num_class)\n",
|
||||
"\n",
|
||||
" def forward(self, x):\n",
|
||||
" x = self.embedding(x)\n",
|
||||
" x = torch.mean(x,dim=1)\n",
|
||||
" return self.fc(x)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 處理變動的序列大小\n",
|
||||
"\n",
|
||||
"由於這種架構，我們的網路需要以特定方式建立小批量資料。在前一單元中，使用詞袋模型（bag-of-words）時，小批量中的所有 BoW 張量都具有相同的大小 `vocab_size`，無論文本序列的實際長度如何。一旦我們改用詞嵌入（word embeddings），每個文本樣本中的詞數就會變得不固定，而在將這些樣本合併成小批量時，我們需要進行一些填充。\n",
|
||||
"\n",
|
||||
"這可以透過向資料來源提供 `collate_fn` 函數的方式來完成：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def padify(b):\n",
|
||||
" # b is the list of tuples of length batch_size\n",
|
||||
" # - first element of a tuple = label, \n",
|
||||
" # - second = feature (text sequence)\n",
|
||||
" # build vectorized sequence\n",
|
||||
" v = [encode(x[1]) for x in b]\n",
|
||||
" # first, compute max length of a sequence in this minibatch\n",
|
||||
" l = max(map(len,v))\n",
|
||||
" return ( # tuple of two tensors - labels and features\n",
|
||||
" torch.LongTensor([t[0]-1 for t in b]),\n",
|
||||
" torch.stack([torch.nn.functional.pad(torch.tensor(t),(0,l-len(t)),mode='constant',value=0) for t in v])\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=padify, shuffle=True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 訓練嵌入分類器\n",
|
||||
"\n",
|
||||
"現在我們已經定義了合適的資料加載器，我們可以使用上一單元中定義的訓練函數來訓練模型：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.6415625\n",
|
||||
"6400: acc=0.6865625\n",
|
||||
"9600: acc=0.7103125\n",
|
||||
"12800: acc=0.726953125\n",
|
||||
"16000: acc=0.739375\n",
|
||||
"19200: acc=0.75046875\n",
|
||||
"22400: acc=0.7572321428571429\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(0.889799795315499, 0.7623160588611644)"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"net = EmbedClassifier(vocab_size,32,len(classes)).to(device)\n",
|
||||
"train_epoch(net,train_loader, lr=1, epoch_size=25000)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**注意**：我們在這裡僅訓練 25k 筆記錄（少於一個完整的 epoch）以節省時間，但您可以繼續訓練，編寫一個函數來進行多個 epoch 的訓練，並嘗試調整學習率參數以獲得更高的準確率。您應該能夠達到約 90% 的準確率。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### EmbeddingBag 層與變長序列表示法\n",
|
||||
"\n",
|
||||
"在之前的架構中，我們需要將所有序列填充（pad）到相同的長度，才能將它們放入一個小批次中。這並不是表示變長序列最有效率的方法——另一種方法是使用 **offset** 向量，該向量會保存所有序列在一個大型向量中的偏移量。\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"> **注意**：在上圖中，我們展示的是一個字符序列，但在我們的例子中，我們處理的是單詞序列。然而，使用偏移向量表示序列的基本原則是相同的。\n",
|
||||
"\n",
|
||||
"為了使用偏移表示法，我們使用 [`EmbeddingBag`](https://pytorch.org/docs/stable/generated/torch.nn.EmbeddingBag.html) 層。它與 `Embedding` 類似，但它接受內容向量和偏移向量作為輸入，並且還包含一個平均層，該層可以是 `mean`、`sum` 或 `max`。\n",
|
||||
"\n",
|
||||
"以下是使用 `EmbeddingBag` 的修改後的網路：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class EmbedClassifier(torch.nn.Module):\n",
|
||||
" def __init__(self, vocab_size, embed_dim, num_class):\n",
|
||||
" super().__init__()\n",
|
||||
" self.embedding = torch.nn.EmbeddingBag(vocab_size, embed_dim)\n",
|
||||
" self.fc = torch.nn.Linear(embed_dim, num_class)\n",
|
||||
"\n",
|
||||
" def forward(self, text, off):\n",
|
||||
" x = self.embedding(text, off)\n",
|
||||
" return self.fc(x)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"要準備用於訓練的數據集，我們需要提供一個轉換函數來準備偏移向量：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def offsetify(b):\n",
|
||||
" # first, compute data tensor from all sequences\n",
|
||||
" x = [torch.tensor(encode(t[1])) for t in b]\n",
|
||||
" # now, compute the offsets by accumulating the tensor of sequence lengths\n",
|
||||
" o = [0] + [len(t) for t in x]\n",
|
||||
" o = torch.tensor(o[:-1]).cumsum(dim=0)\n",
|
||||
" return ( \n",
|
||||
" torch.LongTensor([t[0]-1 for t in b]), # labels\n",
|
||||
" torch.cat(x), # text \n",
|
||||
" o\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=offsetify, shuffle=True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"注意，與之前的所有例子不同，我們的網絡現在接受兩個參數：數據向量和偏移向量，它們的大小不同。同樣，我們的數據加載器也提供了3個值而不是2個：文本和偏移向量都作為特徵提供。因此，我們需要稍微調整我們的訓練函數來處理這一點：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.6153125\n",
|
||||
"6400: acc=0.6615625\n",
|
||||
"9600: acc=0.6932291666666667\n",
|
||||
"12800: acc=0.715078125\n",
|
||||
"16000: acc=0.7270625\n",
|
||||
"19200: acc=0.7382291666666667\n",
|
||||
"22400: acc=0.7486160714285715\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(22.771553103007037, 0.7551983365323096)"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"net = EmbedClassifier(vocab_size,32,len(classes)).to(device)\n",
|
||||
"\n",
|
||||
"def train_epoch_emb(net,dataloader,lr=0.01,optimizer=None,loss_fn = torch.nn.CrossEntropyLoss(),epoch_size=None, report_freq=200):\n",
|
||||
" optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr)\n",
|
||||
" loss_fn = loss_fn.to(device)\n",
|
||||
" net.train()\n",
|
||||
" total_loss,acc,count,i = 0,0,0,0\n",
|
||||
" for labels,text,off in dataloader:\n",
|
||||
" optimizer.zero_grad()\n",
|
||||
" labels,text,off = labels.to(device), text.to(device), off.to(device)\n",
|
||||
" out = net(text, off)\n",
|
||||
" loss = loss_fn(out,labels) #cross_entropy(out,labels)\n",
|
||||
" loss.backward()\n",
|
||||
" optimizer.step()\n",
|
||||
" total_loss+=loss\n",
|
||||
" _,predicted = torch.max(out,1)\n",
|
||||
" acc+=(predicted==labels).sum()\n",
|
||||
" count+=len(labels)\n",
|
||||
" i+=1\n",
|
||||
" if i%report_freq==0:\n",
|
||||
" print(f\"{count}: acc={acc.item()/count}\")\n",
|
||||
" if epoch_size and count>epoch_size:\n",
|
||||
" break\n",
|
||||
" return total_loss.item()/count, acc.item()/count\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"train_epoch_emb(net,train_loader, lr=4, epoch_size=25000)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 語意嵌入：Word2Vec\n",
|
||||
"\n",
|
||||
"在我們之前的例子中，模型的嵌入層學會了將單詞映射為向量表示，但這種表示並沒有太多語意上的意義。如果能學習到這樣的向量表示，讓相似的單詞或同義詞在某種向量距離（例如歐幾里得距離）上彼此接近，那就更好了。\n",
|
||||
"\n",
|
||||
"為了實現這一點，我們需要以特定的方式在大量文本上預訓練嵌入模型。最早的語意嵌入訓練方法之一被稱為 [Word2Vec](https://en.wikipedia.org/wiki/Word2vec)。它基於兩種主要架構，用於生成單詞的分佈式表示：\n",
|
||||
"\n",
|
||||
"- **連續詞袋模型** (CBoW) —— 在這種架構中，我們訓練模型根據周圍的上下文來預測一個單詞。給定 ngram $(W_{-2},W_{-1},W_0,W_1,W_2)$，模型的目標是從 $(W_{-2},W_{-1},W_1,W_2)$ 預測 $W_0$。\n",
|
||||
"- **連續跳字模型** (Skip-Gram) 則與 CBoW 相反。模型使用周圍的上下文窗口單詞來預測當前單詞。\n",
|
||||
"\n",
|
||||
"CBoW 的訓練速度較快，而 Skip-Gram 雖然較慢，但在表示不常見單詞方面表現更好。\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"為了試驗在 Google News 數據集上預訓練的 Word2Vec 嵌入，我們可以使用 **gensim** 庫。以下是找到與 'neural' 最相似的單詞的示例：\n",
|
||||
"\n",
|
||||
"> **注意：** 當你第一次創建單詞向量時，下載它們可能需要一些時間！\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import gensim.downloader as api\n",
|
||||
"w2v = api.load('word2vec-google-news-300')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"neuronal -> 0.7804799675941467\n",
|
||||
"neurons -> 0.7326500415802002\n",
|
||||
"neural_circuits -> 0.7252851724624634\n",
|
||||
"neuron -> 0.7174385190010071\n",
|
||||
"cortical -> 0.6941086649894714\n",
|
||||
"brain_circuitry -> 0.6923246383666992\n",
|
||||
"synaptic -> 0.6699118614196777\n",
|
||||
"neural_circuitry -> 0.6638563275337219\n",
|
||||
"neurochemical -> 0.6555314064025879\n",
|
||||
"neuronal_activity -> 0.6531826257705688\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"for w,p in w2v.most_similar('neural'):\n",
|
||||
" print(f\"{w} -> {p}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"我們也可以從詞語計算向量嵌入，用於訓練分類模型（為了清楚起見，我們僅顯示向量的前20個組件）：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([ 0.01226807, 0.06225586, 0.10693359, 0.05810547, 0.23828125,\n",
|
||||
" 0.03686523, 0.05151367, -0.20703125, 0.01989746, 0.10058594,\n",
|
||||
" -0.03759766, -0.1015625 , -0.15820312, -0.08105469, -0.0390625 ,\n",
|
||||
" -0.05053711, 0.16015625, 0.2578125 , 0.10058594, -0.25976562],\n",
|
||||
" dtype=float32)"
|
||||
]
|
||||
},
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"w2v.word_vec('play')[:20]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"語義嵌入的優點在於可以操控向量編碼來改變語義。例如，我們可以要求找到一個詞，其向量表示盡可能接近詞 *國王* 和 *女人*，並且盡可能遠離詞 *男人*：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"('queen', 0.7118192911148071)"
|
||||
]
|
||||
},
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"w2v.most_similar(positive=['king','woman'],negative=['man'])[0]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"CBoW 和 Skip-Grams 都是「預測型」嵌入，因為它們只考慮局部上下文。Word2Vec 並未利用全局上下文。\n",
|
||||
"\n",
|
||||
"**FastText** 在 Word2Vec 的基礎上進一步改進，通過學習每個詞以及詞內字符 n-gram 的向量表示。在每次訓練步驟中，這些表示的值會被平均成一個向量。雖然這增加了預訓練的計算量，但它使得詞嵌入能夠編碼子詞信息。\n",
|
||||
"\n",
|
||||
"另一種方法，**GloVe**，利用共現矩陣的概念，使用神經方法將共現矩陣分解為更具表達力且非線性的詞向量。\n",
|
||||
"\n",
|
||||
"你可以通過更改嵌入模型為 FastText 和 GloVe 來試驗這些例子，因為 gensim 支援多種不同的詞嵌入模型。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 在 PyTorch 中使用預訓練的嵌入\n",
|
||||
"\n",
|
||||
"我們可以修改上述範例，將嵌入層中的矩陣預先填入語義嵌入，例如 Word2Vec。我們需要考慮到，預訓練嵌入的詞彙表和我們文本語料庫的詞彙表可能不完全匹配，因此我們會用隨機值初始化缺失詞彙的權重：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Embedding size: 300\n",
|
||||
"Populating matrix, this will take some time...Done, found 41080 words, 54732 words missing\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"embed_size = len(w2v.get_vector('hello'))\n",
|
||||
"print(f'Embedding size: {embed_size}')\n",
|
||||
"\n",
|
||||
"net = EmbedClassifier(vocab_size,embed_size,len(classes))\n",
|
||||
"\n",
|
||||
"print('Populating matrix, this will take some time...',end='')\n",
|
||||
"found, not_found = 0,0\n",
|
||||
"for i,w in enumerate(vocab.get_itos()):\n",
|
||||
" try:\n",
|
||||
" net.embedding.weight[i].data = torch.tensor(w2v.get_vector(w))\n",
|
||||
" found+=1\n",
|
||||
" except:\n",
|
||||
" net.embedding.weight[i].data = torch.normal(0.0,1.0,(embed_size,))\n",
|
||||
" not_found+=1\n",
|
||||
"\n",
|
||||
"print(f\"Done, found {found} words, {not_found} words missing\")\n",
|
||||
"net = net.to(device)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"現在讓我們訓練模型。請注意，由於嵌入層的大小更大，因此參數的數量大幅增加，訓練模型所需的時間比前一個例子顯著增加。此外，正因如此，如果我們想避免過擬合，可能需要在更多的例子上訓練模型。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.6359375\n",
|
||||
"6400: acc=0.68109375\n",
|
||||
"9600: acc=0.7067708333333333\n",
|
||||
"12800: acc=0.723671875\n",
|
||||
"16000: acc=0.73625\n",
|
||||
"19200: acc=0.7463541666666667\n",
|
||||
"22400: acc=0.7560714285714286\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(214.1013875559821, 0.7626759436980166)"
|
||||
]
|
||||
},
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"train_epoch_emb(net,train_loader, lr=4, epoch_size=25000)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"在我們的情況中，並未看到準確率有顯著提升，這可能是由於詞彙差異較大所致。 \n",
|
||||
"為了解決詞彙差異的問題，我們可以採用以下解決方案之一： \n",
|
||||
"* 重新訓練 word2vec 模型以適應我們的詞彙 \n",
|
||||
"* 使用預訓練 word2vec 模型的詞彙來載入我們的數據集。在載入數據集時，可以指定使用的詞彙。 \n",
|
||||
"\n",
|
||||
"後者的方法似乎更簡單，尤其是因為 PyTorch 的 `torchtext` 框架內建了對嵌入的支持。我們可以，例如，以以下方式實例化基於 GloVe 的詞彙： \n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"100%|█████████▉| 399999/400000 [00:15<00:00, 25411.14it/s]\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vocab = torchtext.vocab.GloVe(name='6B', dim=50)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"已載入的詞彙具有以下基本操作：\n",
|
||||
"* `vocab.stoi` 字典允許我們將單詞轉換為其字典索引\n",
|
||||
"* `vocab.itos` 則執行相反操作——將數字轉換為單詞\n",
|
||||
"* `vocab.vectors` 是嵌入向量的陣列，因此要獲取單詞 `s` 的嵌入，我們需要使用 `vocab.vectors[vocab.stoi[s]]`\n",
|
||||
"\n",
|
||||
"以下是一個操作嵌入的範例，用來展示方程式 **kind-man+woman = queen**（我稍微調整了係數以使其生效）：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"'queen'"
|
||||
]
|
||||
},
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# get the vector corresponding to kind-man+woman\n",
|
||||
"qvec = vocab.vectors[vocab.stoi['king']]-vocab.vectors[vocab.stoi['man']]+1.3*vocab.vectors[vocab.stoi['woman']]\n",
|
||||
"# find the index of the closest embedding vector \n",
|
||||
"d = torch.sum((vocab.vectors-qvec)**2,dim=1)\n",
|
||||
"min_idx = torch.argmin(d)\n",
|
||||
"# find the corresponding word\n",
|
||||
"vocab.itos[min_idx]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"要使用這些嵌入來訓練分類器，我們首先需要使用GloVe詞彙表對我們的數據集進行編碼：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 16,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def offsetify(b):\n",
|
||||
" # first, compute data tensor from all sequences\n",
|
||||
" x = [torch.tensor(encode(t[1],voc=vocab)) for t in b] # pass the instance of vocab to encode function!\n",
|
||||
" # now, compute the offsets by accumulating the tensor of sequence lengths\n",
|
||||
" o = [0] + [len(t) for t in x]\n",
|
||||
" o = torch.tensor(o[:-1]).cumsum(dim=0)\n",
|
||||
" return ( \n",
|
||||
" torch.LongTensor([t[0]-1 for t in b]), # labels\n",
|
||||
" torch.cat(x), # text \n",
|
||||
" o\n",
|
||||
" )"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"正如我們上面所見，所有向量嵌入都存儲在 `vocab.vectors` 矩陣中。這使得通過簡單的複製將這些權重加載到嵌入層的權重中變得非常容易：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 17,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"net = EmbedClassifier(len(vocab),len(vocab.vectors[0]),len(classes))\n",
|
||||
"net.embedding.weight.data = vocab.vectors\n",
|
||||
"net = net.to(device)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 18,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.6271875\n",
|
||||
"6400: acc=0.68078125\n",
|
||||
"9600: acc=0.7030208333333333\n",
|
||||
"12800: acc=0.71984375\n",
|
||||
"16000: acc=0.7346875\n",
|
||||
"19200: acc=0.7455729166666667\n",
|
||||
"22400: acc=0.7529464285714286\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(35.53972978646833, 0.7575175943698017)"
|
||||
]
|
||||
},
|
||||
"execution_count": 18,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=offsetify, shuffle=True)\n",
|
||||
"train_epoch_emb(net,train_loader, lr=4, epoch_size=25000)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"我們未能看到準確性顯著提高的原因之一是因為我們的數據集中的某些詞語在預訓練的GloVe詞彙表中缺失，因此它們基本上被忽略了。為了克服這一問題，我們可以在我們的數據集上訓練自己的嵌入。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 語境嵌入\n",
|
||||
"\n",
|
||||
"傳統預訓練嵌入表示（例如 Word2Vec）的一個主要限制是詞義消歧的問題。雖然預訓練嵌入能夠捕捉到一些詞語在語境中的含義，但每個詞的所有可能含義都被編碼到同一個嵌入中。這可能會在下游模型中引發問題，因為許多詞語（例如 \"play\"）的含義會根據使用的語境而有所不同。\n",
|
||||
"\n",
|
||||
"例如，\"play\" 在以下兩個句子中的含義就完全不同：\n",
|
||||
"- 我去看了一場**戲劇**。\n",
|
||||
"- 約翰想和他的朋友一起**玩**。\n",
|
||||
"\n",
|
||||
"上述的預訓練嵌入將 \"play\" 的這兩種含義表示為同一嵌入。為了克服這一限制，我們需要基於**語言模型**來構建嵌入。語言模型是在大量文本語料庫上訓練的，並且*了解*詞語如何在不同語境中組合使用。討論語境嵌入超出了本教程的範圍，但我們會在下一單元討論語言模型時回到這個主題。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**免責聲明**： \n本文件已使用 AI 翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。儘管我們努力確保翻譯的準確性，但請注意，自動翻譯可能包含錯誤或不準確之處。原始文件的母語版本應被視為權威來源。對於關鍵信息，建議使用專業人工翻譯。我們對因使用此翻譯而引起的任何誤解或錯誤解釋不承擔責任。\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "py37_pytorch",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "f50b026abce5cf36783a560ea72cb9b1",
|
||||
"translation_date": "2025-08-28T12:30:00+00:00",
|
||||
"source_file": "lessons/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb",
|
||||
"language_code": "mo"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
|
|
@ -0,0 +1,695 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 嵌入\n",
|
||||
"\n",
|
||||
"在之前的例子中，我們使用了高維度的詞袋向量，其長度為 `vocab_size`，並且我們將低維度的位置表示向量顯式轉換為稀疏的獨熱表示。然而，這種獨熱表示並不具備記憶效率。此外，每個詞都被獨立處理，因此獨熱編碼的向量無法表達詞與詞之間的語義相似性。\n",
|
||||
"\n",
|
||||
"在本單元中，我們將繼續探索 **News AG** 數據集。首先，讓我們載入數據並從上一單元中獲取一些定義。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import tensorflow as tf\n",
|
||||
"from tensorflow import keras\n",
|
||||
"import tensorflow_datasets as tfds\n",
|
||||
"import numpy as np\n",
|
||||
"\n",
|
||||
"ds_train, ds_test = tfds.load('ag_news_subset').values()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 什麼是嵌入？\n",
|
||||
"\n",
|
||||
"**嵌入**的概念是用低維度的密集向量來表示單詞，這些向量反映了單詞的語義意義。我們稍後會討論如何構建有意義的單詞嵌入，但現在我們可以將嵌入視為一種降低單詞向量維度的方法。\n",
|
||||
"\n",
|
||||
"因此，嵌入層以單詞作為輸入，並生成指定的 `embedding_size` 的輸出向量。在某種意義上，它與 `Dense` 層非常相似，但不同的是，它不需要以 one-hot 編碼向量作為輸入，而是可以接受單詞編號。\n",
|
||||
"\n",
|
||||
"通過將嵌入層作為我們網絡的第一層，我們可以從詞袋模型（bag-of-words）切換到 **嵌入袋模型**（embedding bag），在這裡我們首先將文本中的每個單詞轉換為對應的嵌入，然後對所有這些嵌入計算某種聚合函數，例如 `sum`、`average` 或 `max`。\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"我們的分類器神經網絡由以下幾層組成：\n",
|
||||
"\n",
|
||||
"* `TextVectorization` 層：該層以字符串作為輸入，並生成一個包含標記編號的張量。我們將指定一個合理的詞彙大小 `vocab_size`，並忽略使用頻率較低的單詞。輸入形狀為 1，輸出形狀為 $n$，因為結果中會有 $n$ 個標記，每個標記包含從 0 到 `vocab_size` 的數字。\n",
|
||||
"* `Embedding` 層：該層接受 $n$ 個數字，並將每個數字縮減為給定長度的密集向量（在我們的例子中為 100）。因此，形狀為 $n$ 的輸入張量將被轉換為 $n\\times 100$ 的張量。\n",
|
||||
"* 聚合層：該層沿第一軸計算此張量的平均值，即計算所有 $n$ 個對應於不同單詞的輸入張量的平均值。為了實現這一層，我們將使用 `Lambda` 層，並傳入計算平均值的函數。輸出形狀為 100，這將是整個輸入序列的數值表示。\n",
|
||||
"* 最後的 `Dense` 線性分類器。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"sequential\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
" Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
" text_vectorization (TextVec (None, None) 0 \n",
|
||||
" torization) \n",
|
||||
" \n",
|
||||
" embedding (Embedding) (None, None, 100) 3000000 \n",
|
||||
" \n",
|
||||
" lambda (Lambda) (None, 100) 0 \n",
|
||||
" \n",
|
||||
" dense (Dense) (None, 4) 404 \n",
|
||||
" \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 3,000,404\n",
|
||||
"Trainable params: 3,000,404\n",
|
||||
"Non-trainable params: 0\n",
|
||||
"_________________________________________________________________\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vocab_size = 30000\n",
|
||||
"batch_size = 128\n",
|
||||
"\n",
|
||||
"vectorizer = keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size,input_shape=(1,))\n",
|
||||
"\n",
|
||||
"model = keras.models.Sequential([\n",
|
||||
" vectorizer, \n",
|
||||
" keras.layers.Embedding(vocab_size,100),\n",
|
||||
" keras.layers.Lambda(lambda x: tf.reduce_mean(x,axis=1)),\n",
|
||||
" keras.layers.Dense(4, activation='softmax')\n",
|
||||
"])\n",
|
||||
"model.summary()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"在 `summary` 的輸出中，**output shape** 欄位的第一個張量維度 `None` 代表小批量的大小，第二個維度則代表標記序列的長度。小批量中的所有標記序列長度各不相同。我們將在下一節討論如何處理這個問題。\n",
|
||||
"\n",
|
||||
"現在讓我們開始訓練這個網路：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Training vectorizer\n",
|
||||
"938/938 [==============================] - 20s 20ms/step - loss: 0.7891 - acc: 0.8155 - val_loss: 0.4470 - val_acc: 0.8642\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.History at 0x22255515100>"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def extract_text(x):\n",
|
||||
" return x['title']+' '+x['description']\n",
|
||||
"\n",
|
||||
"def tupelize(x):\n",
|
||||
" return (extract_text(x),x['label'])\n",
|
||||
"\n",
|
||||
"print(\"Training vectorizer\")\n",
|
||||
"vectorizer.adapt(ds_train.take(500).map(extract_text))\n",
|
||||
"\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'])\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"nteract": {
|
||||
"transient": {
|
||||
"deleting": false
|
||||
}
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"> **注意** 我們正在基於數據的子集構建向量化器。這樣做是為了加快過程，可能會導致我們文本中的所有標記未出現在詞彙表中。在這種情況下，這些標記將被忽略，這可能會導致準確性略低。然而，在現實生活中，文本的子集通常能夠提供良好的詞彙估計。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 處理變數序列大小\n",
|
||||
"\n",
|
||||
"讓我們來了解如何在小批量中進行訓練。在上述範例中，輸入張量的維度為 1，我們使用 128 長度的小批量，因此張量的實際大小為 $128 \\times 1$。然而，每個句子中的詞元數量是不同的。如果我們對單一輸入應用 `TextVectorization` 層，返回的詞元數量會因文本如何被分詞而有所不同：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"tf.Tensor([ 1 45], shape=(2,), dtype=int64)\n",
|
||||
"tf.Tensor([ 112 1271 1 3 1747 158], shape=(6,), dtype=int64)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print(vectorizer('Hello, world!'))\n",
|
||||
"print(vectorizer('I am glad to meet you!'))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"然而，當我們將向量化器應用於多個序列時，它必須生成一個矩形形狀的張量，因此會用 PAD 標記（在我們的情況下是零）填充未使用的元素：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tf.Tensor: shape=(2, 6), dtype=int64, numpy=\n",
|
||||
"array([[ 1, 45, 0, 0, 0, 0],\n",
|
||||
" [ 112, 1271, 1, 3, 1747, 158]], dtype=int64)>"
|
||||
]
|
||||
},
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vectorizer(['Hello, world!','I am glad to meet you!'])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"在這裡我們可以看到嵌入：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([[[ 1.53059261e-02, 6.80514947e-02, 3.14026810e-02, ...,\n",
|
||||
" -8.92002955e-02, 1.52911525e-04, -5.65562584e-02],\n",
|
||||
" [ 2.57456154e-01, 2.79364467e-01, -2.03605562e-01, ...,\n",
|
||||
" -2.07474351e-01, 8.31158683e-02, -2.03911960e-01],\n",
|
||||
" [ 3.98201384e-02, -8.03454965e-03, 2.39790026e-02, ...,\n",
|
||||
" -7.18549127e-04, 2.66963355e-02, -4.30646613e-02],\n",
|
||||
" [ 3.98201384e-02, -8.03454965e-03, 2.39790026e-02, ...,\n",
|
||||
" -7.18549127e-04, 2.66963355e-02, -4.30646613e-02],\n",
|
||||
" [ 3.98201384e-02, -8.03454965e-03, 2.39790026e-02, ...,\n",
|
||||
" -7.18549127e-04, 2.66963355e-02, -4.30646613e-02],\n",
|
||||
" [ 3.98201384e-02, -8.03454965e-03, 2.39790026e-02, ...,\n",
|
||||
" -7.18549127e-04, 2.66963355e-02, -4.30646613e-02]],\n",
|
||||
"\n",
|
||||
" [[ 1.89674050e-01, 2.61548996e-01, -3.67433839e-02, ...,\n",
|
||||
" -2.07366899e-01, -1.05442435e-01, -2.36952081e-01],\n",
|
||||
" [ 6.16133213e-02, 1.80511594e-01, 9.77298319e-02, ...,\n",
|
||||
" -5.46628237e-02, -1.07340455e-01, -1.06589928e-01],\n",
|
||||
" [ 1.53059261e-02, 6.80514947e-02, 3.14026810e-02, ...,\n",
|
||||
" -8.92002955e-02, 1.52911525e-04, -5.65562584e-02],\n",
|
||||
" [-4.84890305e-02, -8.41715634e-02, 1.51529670e-01, ...,\n",
|
||||
" 1.28192469e-01, -7.77286515e-02, 1.26041949e-01],\n",
|
||||
" [-4.17212099e-02, -5.60694858e-02, 4.08860669e-02, ...,\n",
|
||||
" 8.70475471e-02, 8.92383084e-02, 1.67974353e-01],\n",
|
||||
" [ 2.85779923e-01, 4.57767487e-01, 4.52292450e-02, ...,\n",
|
||||
" -1.97419018e-01, -2.04659685e-01, -2.79758364e-01]]],\n",
|
||||
" dtype=float32)"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.layers[1](vectorizer(['Hello, world!','I am glad to meet you!'])).numpy()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **注意**：為了減少填充的數量，在某些情況下，將數據集中的所有序列按長度（更準確地說是按標記數量）遞增排序是有意義的。這將確保每個小批次包含相似長度的序列。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 語意嵌入：Word2Vec\n",
|
||||
"\n",
|
||||
"在我們之前的例子中，嵌入層學習將文字映射到向量表示，但這些表示並不具有語意上的意義。如果能學習一種向量表示，使得相似的詞或同義詞在某種向量距離（例如歐幾里得距離）上彼此接近，那就更好了。\n",
|
||||
"\n",
|
||||
"為了達到這個目的，我們需要使用像 [Word2Vec](https://en.wikipedia.org/wiki/Word2vec) 這樣的技術，在大量文本集合上預訓練嵌入模型。Word2Vec 基於兩種主要架構，用於生成詞的分佈式表示：\n",
|
||||
"\n",
|
||||
" - **連續詞袋模型** (CBoW)，我們訓練模型根據周圍的上下文來預測一個詞。給定 ngram $(W_{-2},W_{-1},W_0,W_1,W_2)$，模型的目標是從 $(W_{-2},W_{-1},W_1,W_2)$ 預測 $W_0$。\n",
|
||||
" - **連續跳字模型** (Skip-Gram) 與 CBoW 相反。模型使用周圍的上下文窗口詞來預測當前詞。\n",
|
||||
"\n",
|
||||
"CBoW 的速度較快，而 Skip-Gram 雖然較慢，但在表示不常見詞方面表現更好。\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"為了試驗基於 Google News 數據集預訓練的 Word2Vec 嵌入，我們可以使用 **gensim** 庫。以下是我們找到與「neural」最相似的詞。\n",
|
||||
"\n",
|
||||
"> **注意:** 當你第一次創建詞向量時，下載可能需要一些時間！\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import gensim.downloader as api\n",
|
||||
"w2v = api.load('word2vec-google-news-300')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"neuronal -> 0.7804799675941467\n",
|
||||
"neurons -> 0.7326500415802002\n",
|
||||
"neural_circuits -> 0.7252851724624634\n",
|
||||
"neuron -> 0.7174385190010071\n",
|
||||
"cortical -> 0.6941086649894714\n",
|
||||
"brain_circuitry -> 0.6923246383666992\n",
|
||||
"synaptic -> 0.6699118614196777\n",
|
||||
"neural_circuitry -> 0.6638563275337219\n",
|
||||
"neurochemical -> 0.6555314064025879\n",
|
||||
"neuronal_activity -> 0.6531826257705688\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"for w,p in w2v.most_similar('neural'):\n",
|
||||
" print(f\"{w} -> {p}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"我們也可以從詞中提取向量嵌入，用於訓練分類模型。嵌入有300個組件，但在這裡我們僅顯示向量的前20個組件以便清晰：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([ 0.01226807, 0.06225586, 0.10693359, 0.05810547, 0.23828125,\n",
|
||||
" 0.03686523, 0.05151367, -0.20703125, 0.01989746, 0.10058594,\n",
|
||||
" -0.03759766, -0.1015625 , -0.15820312, -0.08105469, -0.0390625 ,\n",
|
||||
" -0.05053711, 0.16015625, 0.2578125 , 0.10058594, -0.25976562],\n",
|
||||
" dtype=float32)"
|
||||
]
|
||||
},
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"w2v['play'][:20]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"語義嵌入的偉大之處在於您可以根據語義操縱向量編碼。例如，我們可以要求找到一個詞，其向量表示盡可能接近詞 *king* 和 *woman*，並盡可能遠離詞 *man*:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"('queen', 0.7118192911148071)"
|
||||
]
|
||||
},
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"w2v.most_similar(positive=['king','woman'],negative=['man'])[0]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"上面的例子使用了一些內部的 GenSym 魔法，但其基本邏輯其實相當簡單。嵌入的一個有趣之處在於，你可以對嵌入向量執行正常的向量運算，而這將反映在詞語**意義**上的操作。上面的例子可以用向量運算來表達：我們計算出對應於 **KING-MAN+WOMAN** 的向量（`+` 和 `-` 操作是在相應詞語的向量表示上執行的），然後在字典中找到最接近該向量的詞語：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"'queen'"
|
||||
]
|
||||
},
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# get the vector corresponding to kind-man+woman\n",
|
||||
"qvec = w2v['king']-1.7*w2v['man']+1.7*w2v['woman']\n",
|
||||
"# find the index of the closest embedding vector \n",
|
||||
"d = np.sum((w2v.vectors-qvec)**2,axis=1)\n",
|
||||
"min_idx = np.argmin(d)\n",
|
||||
"# find the corresponding word\n",
|
||||
"w2v.index_to_key[min_idx]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **NOTE**: 我們需要在 *man* 和 *woman* 向量中加入一個小係數 - 試著移除它們看看會發生什麼。\n",
|
||||
"\n",
|
||||
"為了找到最接近的向量，我們使用 TensorFlow 的機制來計算我們的向量與詞彙表中所有向量之間的距離向量，然後使用 `argmin` 找到最小距離的詞的索引。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"雖然 Word2Vec 是表達詞語語義的一種很好的方法，但它也有許多缺點，包括以下幾點：\n",
|
||||
"\n",
|
||||
"* CBoW 和 skip-gram 模型都是**預測型嵌入**，它們僅考慮局部上下文。Word2Vec 無法利用全局上下文。\n",
|
||||
"* Word2Vec 沒有考慮詞語的**形態學**，也就是說詞語的意義可能取決於詞的不同部分，例如詞根。\n",
|
||||
"\n",
|
||||
"**FastText** 試圖克服第二個限制，並基於 Word2Vec 進一步改進。它通過學習每個詞以及詞內的字符 n-gram 的向量表示來實現。這些表示的值在每次訓練步驟中被平均成一個向量。雖然這增加了預訓練的計算量，但它使詞嵌入能夠編碼子詞信息。\n",
|
||||
"\n",
|
||||
"另一種方法，**GloVe**，採用了不同的詞嵌入方法，基於詞-上下文矩陣的分解。首先，它構建了一個大型矩陣，記錄詞語在不同上下文中的出現次數，然後嘗試以降低維度的方式表示該矩陣，以最小化重建損失。\n",
|
||||
"\n",
|
||||
"gensim 庫支持這些詞嵌入，您可以通過更改上方的模型加載代碼來嘗試使用它們。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 在 Keras 中使用預訓練的嵌入\n",
|
||||
"\n",
|
||||
"我們可以修改上述範例，將嵌入層中的矩陣預先填入語義嵌入，例如 Word2Vec。預訓練嵌入的詞彙表和文本語料庫的詞彙表可能不一致，因此我們需要選擇其中一個。在這裡，我們探討兩種可能的選擇：使用分詞器的詞彙表，或使用 Word2Vec 嵌入的詞彙表。\n",
|
||||
"\n",
|
||||
"### 使用分詞器的詞彙表\n",
|
||||
"\n",
|
||||
"當使用分詞器的詞彙表時，詞彙表中的某些單詞會有對應的 Word2Vec 嵌入，而某些則會缺失。假設我們的詞彙表大小為 `vocab_size`，而 Word2Vec 嵌入向量的長度為 `embed_size`，那麼嵌入層將由一個形狀為 `vocab_size`$\\times$`embed_size` 的權重矩陣表示。我們將通過遍歷詞彙表來填充這個矩陣：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Embedding size: 300\n",
|
||||
"Populating matrix, this will take some time...Done, found 4551 words, 784 words missing\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"embed_size = len(w2v.get_vector('hello'))\n",
|
||||
"print(f'Embedding size: {embed_size}')\n",
|
||||
"\n",
|
||||
"vocab = vectorizer.get_vocabulary()\n",
|
||||
"W = np.zeros((vocab_size,embed_size))\n",
|
||||
"print('Populating matrix, this will take some time...',end='')\n",
|
||||
"found, not_found = 0,0\n",
|
||||
"for i,w in enumerate(vocab):\n",
|
||||
" try:\n",
|
||||
" W[i] = w2v.get_vector(w)\n",
|
||||
" found+=1\n",
|
||||
" except:\n",
|
||||
" # W[i] = np.random.normal(0.0,0.3,size=(embed_size,))\n",
|
||||
" not_found+=1\n",
|
||||
"\n",
|
||||
"print(f\"Done, found {found} words, {not_found} words missing\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"對於 Word2Vec 詞彙表中不存在的單詞，我們可以選擇將它們保留為零向量，或者生成一個隨機向量。\n",
|
||||
"\n",
|
||||
"現在我們可以定義一個帶有預訓練權重的嵌入層：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"emb = keras.layers.Embedding(vocab_size,embed_size,weights=[W],trainable=False)\n",
|
||||
"model = keras.models.Sequential([\n",
|
||||
" vectorizer, emb,\n",
|
||||
" keras.layers.Lambda(lambda x: tf.reduce_mean(x,axis=1)),\n",
|
||||
" keras.layers.Dense(4, activation='softmax')\n",
|
||||
"])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"938/938 [==============================] - 10s 10ms/step - loss: 1.1075 - acc: 0.7822 - val_loss: 0.9134 - val_acc: 0.8175\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.History at 0x2220226ef10>"
|
||||
]
|
||||
},
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'])\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(batch_size),\n",
|
||||
" validation_data=ds_test.map(tupelize).batch(batch_size))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **注意**：請注意，我們在建立 `Embedding` 時設置了 `trainable=False`，這表示我們不會重新訓練 Embedding 層。這可能會導致準確率稍微降低，但能加快訓練速度。\n",
|
||||
"\n",
|
||||
"### 使用嵌入詞彙表\n",
|
||||
"\n",
|
||||
"之前方法的一個問題是，TextVectorization 和 Embedding 使用的詞彙表不同。為了解決這個問題，我們可以採用以下其中一種解決方案：\n",
|
||||
"* 重新訓練 Word2Vec 模型以適配我們的詞彙表。\n",
|
||||
"* 使用預訓練 Word2Vec 模型的詞彙表來載入我們的數據集。在載入數據集時，可以指定使用的詞彙表。\n",
|
||||
"\n",
|
||||
"第二種方法看起來更簡單，因此我們來實現它。首先，我們將使用從 Word2Vec 嵌入中提取的指定詞彙表來建立一個 `TextVectorization` 層：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"vocab = list(w2v.vocab.keys())\n",
|
||||
"vectorizer = keras.layers.experimental.preprocessing.TextVectorization(input_shape=(1,))\n",
|
||||
"vectorizer.set_vocabulary(vocab)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Gensim 的詞嵌入庫包含一個方便的函數 `get_keras_embeddings`，它會自動為您創建相應的 Keras 嵌入層。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch 1/5\n",
|
||||
"938/938 [==============================] - 20s 14ms/step - loss: 1.3377 - acc: 0.4978 - val_loss: 1.2995 - val_acc: 0.5647\n",
|
||||
"Epoch 2/5\n",
|
||||
"938/938 [==============================] - 10s 10ms/step - loss: 1.2587 - acc: 0.5722 - val_loss: 1.2339 - val_acc: 0.5842\n",
|
||||
"Epoch 3/5\n",
|
||||
"938/938 [==============================] - 10s 10ms/step - loss: 1.1980 - acc: 0.5884 - val_loss: 1.1826 - val_acc: 0.5954\n",
|
||||
"Epoch 4/5\n",
|
||||
"938/938 [==============================] - 12s 13ms/step - loss: 1.1503 - acc: 0.6002 - val_loss: 1.1417 - val_acc: 0.6018\n",
|
||||
"Epoch 5/5\n",
|
||||
"938/938 [==============================] - 11s 12ms/step - loss: 1.1120 - acc: 0.6097 - val_loss: 1.1083 - val_acc: 0.6104\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.History at 0x2220ccb81c0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model = keras.models.Sequential([\n",
|
||||
" vectorizer, \n",
|
||||
" w2v.get_keras_embedding(train_embeddings=False),\n",
|
||||
" keras.layers.Lambda(lambda x: tf.reduce_mean(x,axis=1)),\n",
|
||||
" keras.layers.Dense(4, activation='softmax')\n",
|
||||
"])\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'])\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(128),validation_data=ds_test.map(tupelize).batch(128),epochs=5)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"我們未能看到更高準確度的其中一個原因是因為我們的數據集中的某些詞語在預訓練的GloVe詞彙表中缺失，因此它們基本上被忽略了。為了解決這個問題，我們可以基於我們的數據集訓練自己的嵌入。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 語境嵌入\n",
|
||||
"\n",
|
||||
"傳統預訓練嵌入表示（例如 Word2Vec）的一個主要限制是，儘管它們能捕捉到某些詞語的含義，但無法區分不同的含義。這可能會在下游模型中引發問題。\n",
|
||||
"\n",
|
||||
"例如，詞語「play」在以下兩個句子中有不同的含義：\n",
|
||||
"- 我去劇院看了一場**戲劇**。\n",
|
||||
"- 約翰想和他的朋友**玩耍**。\n",
|
||||
"\n",
|
||||
"我們之前提到的預訓練嵌入會將「play」的兩種含義表示為相同的嵌入。為了解決這一限制，我們需要基於**語言模型**來構建嵌入。語言模型是在大量文本語料庫上訓練的，並且*了解*詞語如何在不同的語境中組合使用。討論語境嵌入超出了本教程的範圍，但我們會在下一單元討論語言模型時回到這個主題。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**免責聲明**： \n本文件使用 AI 翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。雖然我們努力確保翻譯的準確性，但請注意，自動翻譯可能包含錯誤或不準確之處。原始語言的文件應被視為權威來源。對於關鍵信息，建議使用專業人工翻譯。我們對因使用此翻譯而引起的任何誤解或誤釋不承擔責任。\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5"
|
||||
},
|
||||
"kernel_info": {
|
||||
"name": "conda-env-py37_tensorflow-py"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "py37_tensorflow",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"nteract": {
|
||||
"version": "nteract-front-end@1.0.0"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "b859482be7f61d1eadc2c6a2720a37e4",
|
||||
"translation_date": "2025-08-28T12:26:22+00:00",
|
||||
"source_file": "lessons/5-NLP/14-Embeddings/EmbeddingsTF.ipynb",
|
||||
"language_code": "mo"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
|
|
@ -0,0 +1,576 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "NXTSugt6ieXh"
|
||||
},
|
||||
"source": [
|
||||
"## 訓練 CBoW 模型\n",
|
||||
"\n",
|
||||
"此筆記本是 [AI for Beginners Curriculum](http://aka.ms/ai-beginners) 的一部分\n",
|
||||
"\n",
|
||||
"在這個範例中，我們將學習如何訓練 CBoW 語言模型，以建立我們自己的 Word2Vec 嵌入空間。我們將使用 AG News 數據集作為文本來源。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"import torchtext\n",
|
||||
"import os\n",
|
||||
"import collections\n",
|
||||
"import builtins\n",
|
||||
"import random\n",
|
||||
"import numpy as np"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "q-UiiJUKaxHj"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "TFbR8CZaTZ1q"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"首先讓我們載入資料集並定義分詞器和詞彙表。我們將 `vocab_size` 設定為 5000 以稍微限制計算量。\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "HIwC7lI5T-ov"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def load_dataset(ngrams = 1, min_freq = 1, vocab_size = 5000 , lines_cnt = 500):\n",
|
||||
" tokenizer = torchtext.data.utils.get_tokenizer('basic_english')\n",
|
||||
" print(\"Loading dataset...\")\n",
|
||||
" test_dataset, train_dataset = torchtext.datasets.AG_NEWS(root='./data')\n",
|
||||
" train_dataset = list(train_dataset)\n",
|
||||
" test_dataset = list(test_dataset)\n",
|
||||
" classes = ['World', 'Sports', 'Business', 'Sci/Tech']\n",
|
||||
" print('Building vocab...')\n",
|
||||
" counter = collections.Counter()\n",
|
||||
" for i, (_, line) in enumerate(train_dataset):\n",
|
||||
" counter.update(torchtext.data.utils.ngrams_iterator(tokenizer(line),ngrams=ngrams))\n",
|
||||
" if i == lines_cnt:\n",
|
||||
" break\n",
|
||||
" vocab = torchtext.vocab.Vocab(collections.Counter(dict(counter.most_common(vocab_size))), min_freq=min_freq)\n",
|
||||
" return train_dataset, test_dataset, classes, vocab, tokenizer"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "wdZuygtgiuLG"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"train_dataset, test_dataset, _, vocab, tokenizer = load_dataset()"
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "4d1nU1gsivGu",
|
||||
"outputId": "949fe272-ae0e-49f5-c373-6703458b3a74"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
"Loading dataset...\n",
|
||||
"Building vocab...\n"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def encode(x, vocabulary, tokenizer = tokenizer):\n",
|
||||
" return [vocabulary[s] for s in tokenizer(x)]"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "1XDYNhG8ToFV"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "LIlQk6_PaHVY"
|
||||
},
|
||||
"source": [
|
||||
"## CBoW 模型\n",
|
||||
"\n",
|
||||
"CBoW 學習根據 $2N$ 個相鄰的單詞來預測一個單詞。例如，當 $N=1$ 時，我們可以從句子 *I like to train networks* 中得到以下配對：(like, I)、(I, like)、(to, like)、(like, to)、(train, to)、(to, train)、(networks, train)、(train, networks)。在這裡，第一個單詞是作為輸入的相鄰單詞，第二個單詞是我們要預測的單詞。\n",
|
||||
"\n",
|
||||
"為了構建一個用於預測下一個單詞的網絡，我們需要提供相鄰單詞作為輸入，並獲得單詞編號作為輸出。CBoW 網絡的架構如下：\n",
|
||||
"\n",
|
||||
"* 輸入單詞會通過嵌入層。這個嵌入層就是我們的 Word2Vec 嵌入，因此我們會將其單獨定義為 `embedder` 變數。在這個例子中，我們將使用嵌入大小為 30，儘管你可能想嘗試更高的維度（真實的 Word2Vec 通常是 300 維）。\n",
|
||||
"* 嵌入向量接著會傳遞到一個線性層，該層將預測輸出單詞。因此它有 `vocab_size` 個神經元。\n",
|
||||
"\n",
|
||||
"對於輸出，如果我們使用 `CrossEntropyLoss` 作為損失函數，我們只需要提供單詞編號作為期望結果，而不需要使用 one-hot 編碼。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"vocab_size = len(vocab)\n",
|
||||
"\n",
|
||||
"embedder = torch.nn.Embedding(num_embeddings = vocab_size, embedding_dim = 30)\n",
|
||||
"model = torch.nn.Sequential(\n",
|
||||
" embedder,\n",
|
||||
" torch.nn.Linear(in_features = 30, out_features = vocab_size),\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print(model)"
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "akKTcKQKkfl2",
|
||||
"outputId": "da687e3e-a8ec-4c1a-e456-ab8cd6ac7dad"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
"Sequential(\n",
|
||||
" (0): Embedding(5002, 30)\n",
|
||||
" (1): Linear(in_features=30, out_features=5002, bias=True)\n",
|
||||
")\n"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "Nud6jgGPaHVa"
|
||||
},
|
||||
"source": [
|
||||
"## 準備訓練數據\n",
|
||||
"\n",
|
||||
"現在讓我們編寫主要函數，從文本中計算 CBoW 詞對。這個函數將允許我們指定窗口大小，並返回一組詞對——輸入詞和輸出詞。請注意，這個函數既可以用於詞，也可以用於向量/張量——這將使我們能夠在傳遞給 `to_cbow` 函數之前對文本進行編碼。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "x-dsXygOieXn",
|
||||
"outputId": "c2218280-e540-40ba-9546-efe48d0d714f"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
"[['like', 'I'], ['to', 'I'], ['I', 'like'], ['to', 'like'], ['train', 'like'], ['I', 'to'], ['like', 'to'], ['train', 'to'], ['networks', 'to'], ['like', 'train'], ['to', 'train'], ['networks', 'train'], ['to', 'networks'], ['train', 'networks']]\n",
|
||||
"[[232, 172], [5, 172], [172, 232], [5, 232], [0, 232], [172, 5], [232, 5], [0, 5], [1202, 5], [232, 0], [5, 0], [1202, 0], [5, 1202], [0, 1202]]\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def to_cbow(sent,window_size=2):\n",
|
||||
" res = []\n",
|
||||
" for i,x in enumerate(sent):\n",
|
||||
" for j in range(max(0,i-window_size),min(i+window_size+1,len(sent))):\n",
|
||||
" if i!=j:\n",
|
||||
" res.append([sent[j],x])\n",
|
||||
" return res\n",
|
||||
"\n",
|
||||
"print(to_cbow(['I','like','to','train','networks']))\n",
|
||||
"print(to_cbow(encode('I like to train networks', vocab)))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "XVaaDLjaaHVb"
|
||||
},
|
||||
"source": [
|
||||
"讓我們準備訓練數據集。我們將遍歷所有新聞，調用 `to_cbow` 來獲取詞對列表，並將這些詞對添加到 `X` 和 `Y` 中。為了節省時間，我們只考慮前 10k 條新聞項目——如果你有更多時間等待並希望獲得更好的嵌入，可以輕鬆移除這個限制 :)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "54b-Gd9TieXo"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"X = []\n",
|
||||
"Y = []\n",
|
||||
"for i, x in zip(range(10000), train_dataset):\n",
|
||||
" for w1, w2 in to_cbow(encode(x[1], vocab), window_size = 5):\n",
|
||||
" X.append(w1)\n",
|
||||
" Y.append(w2)\n",
|
||||
"\n",
|
||||
"X = torch.tensor(X)\n",
|
||||
"Y = torch.tensor(Y)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"我們還將把該數據轉換為一個數據集，並創建數據加載器：\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "cwWy0PzXWhN5"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"class SimpleIterableDataset(torch.utils.data.IterableDataset):\n",
|
||||
" def __init__(self, X, Y):\n",
|
||||
" super(SimpleIterableDataset).__init__()\n",
|
||||
" self.data = []\n",
|
||||
" for i in range(len(X)):\n",
|
||||
" self.data.append( (Y[i], X[i]) )\n",
|
||||
" random.shuffle(self.data)\n",
|
||||
"\n",
|
||||
" def __iter__(self):\n",
|
||||
" return iter(self.data)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "mfoAcGPFZU8p"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "e4NQ_-5waHVc"
|
||||
},
|
||||
"source": [
|
||||
"我們還將把該數據轉換為一個數據集，並創建數據加載器：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "AbLUcojlieXo"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"ds = SimpleIterableDataset(X, Y)\n",
|
||||
"dl = torch.utils.data.DataLoader(ds, batch_size = 256)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "pKQr7sXeaHVc"
|
||||
},
|
||||
"source": [
|
||||
"現在讓我們進行實際訓練。我們將使用 `SGD` 優化器，並設定相當高的學習率。你也可以嘗試使用其他優化器，例如 `Adam`。我們將先訓練 10 個世代——如果你希望更低的損失，可以重新執行此單元格。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def train_epoch(net, dataloader, lr = 0.01, optimizer = None, loss_fn = torch.nn.CrossEntropyLoss(), epochs = None, report_freq = 1):\n",
|
||||
" optimizer = optimizer or torch.optim.Adam(net.parameters(), lr = lr)\n",
|
||||
" loss_fn = loss_fn.to(device)\n",
|
||||
" net.train()\n",
|
||||
"\n",
|
||||
" for i in range(epochs):\n",
|
||||
" total_loss, j = 0, 0, \n",
|
||||
" for labels, features in dataloader:\n",
|
||||
" optimizer.zero_grad()\n",
|
||||
" features, labels = features.to(device), labels.to(device)\n",
|
||||
" out = net(features)\n",
|
||||
" loss = loss_fn(out, labels)\n",
|
||||
" loss.backward()\n",
|
||||
" optimizer.step()\n",
|
||||
" total_loss += loss\n",
|
||||
" j += 1\n",
|
||||
" if i % report_freq == 0:\n",
|
||||
" print(f\"Epoch: {i+1}: loss={total_loss.item()/j}\")\n",
|
||||
"\n",
|
||||
" return total_loss.item()/j"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "HeeCYKr_KF1w"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"train_epoch(net = model, dataloader = dl, optimizer = torch.optim.SGD(model.parameters(), lr = 0.1), loss_fn = torch.nn.CrossEntropyLoss(), epochs = 10)"
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "KVgwGtDHgDlT",
|
||||
"outputId": "2447833f-f0e3-4566-c33d-addbfe2f451d"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
"Epoch: 1: loss=5.664632366860172\n",
|
||||
"Epoch: 2: loss=5.632101973960962\n",
|
||||
"Epoch: 3: loss=5.610399051405015\n",
|
||||
"Epoch: 4: loss=5.594621561080262\n",
|
||||
"Epoch: 5: loss=5.582538017415446\n",
|
||||
"Epoch: 6: loss=5.572900234519603\n",
|
||||
"Epoch: 7: loss=5.564951676341915\n",
|
||||
"Epoch: 8: loss=5.558288112064614\n",
|
||||
"Epoch: 9: loss=5.552576955031129\n",
|
||||
"Epoch: 10: loss=5.547634165194347\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"output_type": "execute_result",
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"5.547634165194347"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"execution_count": 16
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "W8u2qXZmaHVd"
|
||||
},
|
||||
"source": [
|
||||
"## 嘗試使用 Word2Vec\n",
|
||||
"\n",
|
||||
"要使用 Word2Vec，我們來提取與詞彙表中所有單詞對應的向量：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "r8TatcXjkU_t"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"vectors = torch.stack([embedder(torch.tensor(vocab[s])) for s in vocab.itos], 0)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "3OcX21UOaHVd"
|
||||
},
|
||||
"source": [
|
||||
"讓我們看看，例如，單詞**Paris**是如何被編碼成一個向量的：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "bz6tAeLzieXp",
|
||||
"outputId": "5b20850e-4342-45e9-f840-cfac2b4d61d8"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
"tensor([-0.0915, 2.1224, -0.0281, -0.6819, 1.1219, 0.6458, -1.3704, -1.3314,\n",
|
||||
" -1.1437, 0.4496, 0.2301, -0.3515, -0.8485, 1.0481, 0.4386, -0.8949,\n",
|
||||
" 0.5644, 1.0939, -2.5096, 3.2949, -0.2601, -0.8640, 0.1421, -0.0804,\n",
|
||||
" -0.5083, -1.0560, 0.9753, -0.5949, -1.6046, 0.5774],\n",
|
||||
" grad_fn=<EmbeddingBackward>)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"paris_vec = embedder(torch.tensor(vocab['paris']))\n",
|
||||
"print(paris_vec)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "pHTJlaeYaHVd"
|
||||
},
|
||||
"source": [
|
||||
"使用 Word2Vec 來尋找同義詞是很有趣的。以下函數將返回與給定輸入最接近的 `n` 個詞。為了找到它們，我們計算 $|w_i - v|$ 的範數，其中 $v$ 是對應於我們輸入詞的向量，$w_i$ 是詞彙表中第 $i$ 個詞的編碼。我們接著對陣列進行排序，並使用 `argsort` 返回相應的索引，然後取列表的前 `n` 個元素，這些元素編碼了詞彙表中最接近詞的位置。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "NlZyi-_olFar",
|
||||
"outputId": "b5dbb163-88c4-4d5a-eaf2-6751f700e98c"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "execute_result",
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"['microsoft', 'quoted', 'lp', 'rate', 'top']"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"execution_count": 56
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def close_words(x, n = 5):\n",
|
||||
" vec = embedder(torch.tensor(vocab[x]))\n",
|
||||
" top5 = np.linalg.norm(vectors.detach().numpy() - vec.detach().numpy(), axis = 1).argsort()[:n]\n",
|
||||
" return [ vocab.itos[x] for x in top5 ]\n",
|
||||
"\n",
|
||||
"close_words('microsoft')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "-dQq7xeAln0U",
|
||||
"outputId": "66f768c3-c248-4bfd-ce4f-c8ffc6d0dd0d"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "execute_result",
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"['basketball', 'lot', 'sinai', 'states', 'healthdaynews']"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"execution_count": 51
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"close_words('basketball')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "fJXqK26b29sa",
|
||||
"outputId": "78f0baba-ffd0-485a-dd87-0a12bedfd7fa"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "execute_result",
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"['funds', 'travel', 'sydney', 'japan', 'business']"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"execution_count": 77
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"close_words('funds')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "My0VeTDd3Ji8"
|
||||
},
|
||||
"source": [
|
||||
"## 重點\n",
|
||||
"\n",
|
||||
"使用像 CBoW 這樣的巧妙技術，我們可以訓練 Word2Vec 模型。你也可以嘗試訓練 skip-gram 模型，該模型是基於給定中心詞來預測鄰近詞，看看它的表現如何。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**免責聲明**： \n本文件已使用 AI 翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。儘管我們努力確保翻譯的準確性，但請注意，自動翻譯可能包含錯誤或不準確之處。原始文件的母語版本應被視為權威來源。對於關鍵信息，建議使用專業人工翻譯。我們對因使用此翻譯而引起的任何誤解或誤釋不承擔責任。\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"collapsed_sections": [],
|
||||
"name": "CBoW-PyTorch.ipynb",
|
||||
"provenance": []
|
||||
},
|
||||
"interpreter": {
|
||||
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.8.12 ('py38')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"orig_nbformat": 4,
|
||||
"gpuClass": "standard",
|
||||
"coopTranslator": {
|
||||
"original_hash": "36df28efe3fe40b6fb0a7fa48fe3ea82",
|
||||
"translation_date": "2025-08-28T12:05:19+00:00",
|
||||
"source_file": "lessons/5-NLP/15-LanguageModeling/CBoW-PyTorch.ipynb",
|
||||
"language_code": "mo"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,479 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# 循環神經網絡\n",
|
||||
"\n",
|
||||
"在上一個模組中，我們使用了豐富的文本語義表示，以及嵌入層上的簡單線性分類器。這種架構能夠捕捉句子中詞語的聚合意義，但它並未考慮詞語的**順序**，因為嵌入層上的聚合操作已經移除了原始文本中的這些信息。由於這些模型無法建模詞語的排列順序，因此它們無法解決更複雜或更具歧義的任務，例如文本生成或問題回答。\n",
|
||||
"\n",
|
||||
"為了捕捉文本序列的意義，我們需要使用另一種神經網絡架構，稱為**循環神經網絡**（Recurrent Neural Network，RNN）。在 RNN 中，我們將句子逐個符號地傳遞給網絡，網絡會生成某種**狀態**，然後我們將該狀態與下一個符號一起再次傳遞給網絡。\n",
|
||||
"\n",
|
||||
"給定輸入序列的標記 $X_0,\\dots,X_n$，RNN 會生成一系列神經網絡模塊，並通過反向傳播對這個序列進行端到端的訓練。每個網絡模塊接收一對 $(X_i,S_i)$ 作為輸入，並生成 $S_{i+1}$ 作為結果。最終狀態 $S_n$ 或輸出 $X_n$ 會進入線性分類器以生成結果。所有網絡模塊共享相同的權重，並通過一次反向傳播進行端到端訓練。\n",
|
||||
"\n",
|
||||
"由於狀態向量 $S_0,\\dots,S_n$ 是通過網絡傳遞的，RNN 能夠學習詞語之間的序列依賴性。例如，當某個序列中出現詞語 *not* 時，網絡可以學習在狀態向量中否定某些元素，從而實現否定的效果。\n",
|
||||
"\n",
|
||||
"> 由於圖片中所有 RNN 模塊的權重是共享的，因此同一張圖片可以表示為一個模塊（右側）帶有循環反饋回路，該回路將網絡的輸出狀態回傳到輸入。\n",
|
||||
"\n",
|
||||
"接下來，我們來看看循環神經網絡如何幫助我們對新聞數據集進行分類。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Loading dataset...\n",
|
||||
"Building vocab...\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"import torchtext\n",
|
||||
"from torchnlp import *\n",
|
||||
"train_dataset, test_dataset, classes, vocab = load_dataset()\n",
|
||||
"vocab_size = len(vocab)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 簡單的 RNN 分類器\n",
|
||||
"\n",
|
||||
"在簡單的 RNN 中，每個循環單元是一個簡單的線性網絡，它接收連接的輸入向量和狀態向量，並生成一個新的狀態向量。PyTorch 使用 `RNNCell` 類來表示這種單元，而由這些單元組成的網絡則表示為 `RNN` 層。\n",
|
||||
"\n",
|
||||
"為了定義一個 RNN 分類器，我們將首先應用一個嵌入層來降低輸入詞彙的維度，然後在其上添加一個 RNN 層：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class RNNClassifier(torch.nn.Module):\n",
|
||||
" def __init__(self, vocab_size, embed_dim, hidden_dim, num_class):\n",
|
||||
" super().__init__()\n",
|
||||
" self.hidden_dim = hidden_dim\n",
|
||||
" self.embedding = torch.nn.Embedding(vocab_size, embed_dim)\n",
|
||||
" self.rnn = torch.nn.RNN(embed_dim,hidden_dim,batch_first=True)\n",
|
||||
" self.fc = torch.nn.Linear(hidden_dim, num_class)\n",
|
||||
"\n",
|
||||
" def forward(self, x):\n",
|
||||
" batch_size = x.size(0)\n",
|
||||
" x = self.embedding(x)\n",
|
||||
" x,h = self.rnn(x)\n",
|
||||
" return self.fc(x.mean(dim=1))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **注意：** 為了簡化，我們在這裡使用未經訓練的嵌入層，但如果想要更好的結果，可以使用預訓練的嵌入層，例如 Word2Vec 或 GloVe 嵌入，這在前一單元中已經描述過。為了更好地理解，您可能需要將此程式碼改寫為使用預訓練的嵌入。\n",
|
||||
"\n",
|
||||
"在我們的情況下，我們將使用填充的資料加載器，因此每個批次都會包含一些相同長度的填充序列。RNN 層將接收嵌入張量的序列，並產生兩個輸出：\n",
|
||||
"* $x$ 是每一步驟中 RNN 單元輸出的序列\n",
|
||||
"* $h$ 是序列最後一個元素的最終隱藏狀態\n",
|
||||
"\n",
|
||||
"接著，我們應用一個全連接的線性分類器來獲得類別數。\n",
|
||||
"\n",
|
||||
"> **注意：** RNN 的訓練相當困難，因為一旦 RNN 單元沿著序列長度展開，反向傳播所涉及的層數就會非常多。因此，我們需要選擇較小的學習率，並在更大的資料集上訓練網路以獲得良好的結果。這可能需要相當長的時間，因此建議使用 GPU。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.3090625\n",
|
||||
"6400: acc=0.38921875\n",
|
||||
"9600: acc=0.4590625\n",
|
||||
"12800: acc=0.511953125\n",
|
||||
"16000: acc=0.5506875\n",
|
||||
"19200: acc=0.57921875\n",
|
||||
"22400: acc=0.6070089285714285\n",
|
||||
"25600: acc=0.6304296875\n",
|
||||
"28800: acc=0.6484027777777778\n",
|
||||
"32000: acc=0.66509375\n",
|
||||
"35200: acc=0.6790056818181818\n",
|
||||
"38400: acc=0.6929166666666666\n",
|
||||
"41600: acc=0.7035817307692308\n",
|
||||
"44800: acc=0.7137276785714286\n",
|
||||
"48000: acc=0.72225\n",
|
||||
"51200: acc=0.73001953125\n",
|
||||
"54400: acc=0.7372794117647059\n",
|
||||
"57600: acc=0.7436631944444444\n",
|
||||
"60800: acc=0.7503947368421052\n",
|
||||
"64000: acc=0.75634375\n",
|
||||
"67200: acc=0.7615773809523809\n",
|
||||
"70400: acc=0.7662642045454545\n",
|
||||
"73600: acc=0.7708423913043478\n",
|
||||
"76800: acc=0.7751822916666666\n",
|
||||
"80000: acc=0.7790625\n",
|
||||
"83200: acc=0.7825\n",
|
||||
"86400: acc=0.7858564814814815\n",
|
||||
"89600: acc=0.7890513392857142\n",
|
||||
"92800: acc=0.7920474137931034\n",
|
||||
"96000: acc=0.7952708333333334\n",
|
||||
"99200: acc=0.7982258064516129\n",
|
||||
"102400: acc=0.80099609375\n",
|
||||
"105600: acc=0.8037594696969697\n",
|
||||
"108800: acc=0.8060569852941176\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=padify, shuffle=True)\n",
|
||||
"net = RNNClassifier(vocab_size,64,32,len(classes)).to(device)\n",
|
||||
"train_epoch(net,train_loader, lr=0.001)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 長短期記憶網路 (LSTM)\n",
|
||||
"\n",
|
||||
"傳統 RNN 的主要問題之一是所謂的 **梯度消失** 問題。由於 RNN 是在一次反向傳播中端到端訓練的，因此在將誤差傳遞到網路的第一層時會遇到困難，導致網路無法學習到遠距詞彙之間的關係。為了解決這個問題，可以通過使用所謂的 **閘門** 引入 **顯式狀態管理**。這類架構中最知名的兩種是 **長短期記憶網路** (LSTM) 和 **門控循環單元** (GRU)。\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"LSTM 網路的組織方式與 RNN 類似，但有兩個狀態會在層與層之間傳遞：實際狀態 $c$ 和隱藏向量 $h$。在每個單元中，隱藏向量 $h_i$ 與輸入 $x_i$ 連接在一起，並通過 **閘門** 控制狀態 $c$ 的變化。每個閘門都是一個帶有 sigmoid 激活函數（輸出範圍為 $[0,1]$）的神經網路，可以被視為在與狀態向量相乘時的位掩碼。以下是這些閘門（如上圖從左到右）：\n",
|
||||
"* **遺忘閘門** 接收隱藏向量並決定向量 $c$ 的哪些部分需要遺忘，哪些需要保留。\n",
|
||||
"* **輸入閘門** 從輸入和隱藏向量中提取一些信息，並插入到狀態中。\n",
|
||||
"* **輸出閘門** 通過帶有 $\\tanh$ 激活函數的線性層轉換狀態，然後使用隱藏向量 $h_i$ 選擇其部分組件以生成新的狀態 $c_{i+1}$。\n",
|
||||
"\n",
|
||||
"狀態 $c$ 的組件可以被視為一些可以開啟或關閉的標誌。例如，當我們在序列中遇到名字 *Alice* 時，我們可能會假設它指的是女性角色，並在狀態中設置一個標誌，表示句子中有一個女性名詞。當我們進一步遇到短語 *and Tom* 時，我們會設置一個標誌，表示句子中有複數名詞。因此，通過操作狀態，我們可以假設性地跟蹤句子部分的語法屬性。\n",
|
||||
"\n",
|
||||
"> **注意**: 一篇非常棒的文章可以幫助理解 LSTM 的內部結構，請參考 Christopher Olah 的這篇文章 [Understanding LSTM Networks](https://colah.github.io/posts/2015-08-Understanding-LSTMs/)。\n",
|
||||
"\n",
|
||||
"雖然 LSTM 單元的內部結構看起來很複雜，但 PyTorch 將這些實現隱藏在 `LSTMCell` 類中，並提供了 `LSTM` 對象來表示整個 LSTM 層。因此，LSTM 分類器的實現將與我們之前看到的簡單 RNN 非常相似：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class LSTMClassifier(torch.nn.Module):\n",
|
||||
" def __init__(self, vocab_size, embed_dim, hidden_dim, num_class):\n",
|
||||
" super().__init__()\n",
|
||||
" self.hidden_dim = hidden_dim\n",
|
||||
" self.embedding = torch.nn.Embedding(vocab_size, embed_dim)\n",
|
||||
" self.embedding.weight.data = torch.randn_like(self.embedding.weight.data)-0.5\n",
|
||||
" self.rnn = torch.nn.LSTM(embed_dim,hidden_dim,batch_first=True)\n",
|
||||
" self.fc = torch.nn.Linear(hidden_dim, num_class)\n",
|
||||
"\n",
|
||||
" def forward(self, x):\n",
|
||||
" batch_size = x.size(0)\n",
|
||||
" x = self.embedding(x)\n",
|
||||
" x,(h,c) = self.rnn(x)\n",
|
||||
" return self.fc(h[-1])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.259375\n",
|
||||
"6400: acc=0.25859375\n",
|
||||
"9600: acc=0.26177083333333334\n",
|
||||
"12800: acc=0.2784375\n",
|
||||
"16000: acc=0.313\n",
|
||||
"19200: acc=0.3528645833333333\n",
|
||||
"22400: acc=0.3965625\n",
|
||||
"25600: acc=0.4385546875\n",
|
||||
"28800: acc=0.4752777777777778\n",
|
||||
"32000: acc=0.505375\n",
|
||||
"35200: acc=0.5326704545454546\n",
|
||||
"38400: acc=0.5557552083333334\n",
|
||||
"41600: acc=0.5760817307692307\n",
|
||||
"44800: acc=0.5954910714285714\n",
|
||||
"48000: acc=0.6118333333333333\n",
|
||||
"51200: acc=0.62681640625\n",
|
||||
"54400: acc=0.6404779411764706\n",
|
||||
"57600: acc=0.6520138888888889\n",
|
||||
"60800: acc=0.662828947368421\n",
|
||||
"64000: acc=0.673546875\n",
|
||||
"67200: acc=0.6831547619047619\n",
|
||||
"70400: acc=0.6917897727272727\n",
|
||||
"73600: acc=0.6997146739130434\n",
|
||||
"76800: acc=0.707109375\n",
|
||||
"80000: acc=0.714075\n",
|
||||
"83200: acc=0.7209134615384616\n",
|
||||
"86400: acc=0.727037037037037\n",
|
||||
"89600: acc=0.7326674107142858\n",
|
||||
"92800: acc=0.7379633620689655\n",
|
||||
"96000: acc=0.7433645833333333\n",
|
||||
"99200: acc=0.7479032258064516\n",
|
||||
"102400: acc=0.752119140625\n",
|
||||
"105600: acc=0.7562405303030303\n",
|
||||
"108800: acc=0.76015625\n",
|
||||
"112000: acc=0.7641339285714286\n",
|
||||
"115200: acc=0.7677777777777778\n",
|
||||
"118400: acc=0.7711233108108108\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(0.03487814127604167, 0.7728)"
|
||||
]
|
||||
},
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"net = LSTMClassifier(vocab_size,64,32,len(classes)).to(device)\n",
|
||||
"train_epoch(net,train_loader, lr=0.001)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 打包序列\n",
|
||||
"\n",
|
||||
"在我們的範例中，我們必須用零向量填充小批次中的所有序列。雖然這會導致一些記憶體的浪費，但對於 RNN 而言，更關鍵的是為這些填充的輸入項目創建了額外的 RNN 單元，這些單元參與了訓練，但並未攜帶任何重要的輸入資訊。如果能讓 RNN 僅針對實際的序列長度進行訓練，效果會更好。\n",
|
||||
"\n",
|
||||
"為了實現這一點，PyTorch 引入了一種特殊的填充序列存儲格式。假設我們有一個填充過的小批次輸入，看起來像這樣：\n",
|
||||
"```\n",
|
||||
"[[1,2,3,4,5],\n",
|
||||
" [6,7,8,0,0],\n",
|
||||
" [9,0,0,0,0]]\n",
|
||||
"```\n",
|
||||
"這裡的 0 代表填充的值，而輸入序列的實際長度向量是 `[5,3,1]`。\n",
|
||||
"\n",
|
||||
"為了有效地用填充序列訓練 RNN，我們希望先用較大的小批次（`[1,6,9]`）開始訓練第一組 RNN 單元，然後結束第三個序列的處理，接著用較小的小批次（`[2,7]`、`[3,8]`）繼續訓練，依此類推。因此，打包序列被表示為一個向量——在我們的例子中是 `[1,6,9,2,7,3,8,4,5]`，以及長度向量（`[5,3,1]`），我們可以根據這些資訊輕鬆還原原始的填充小批次。\n",
|
||||
"\n",
|
||||
"要生成打包序列，我們可以使用 `torch.nn.utils.rnn.pack_padded_sequence` 函數。所有的循環層，包括 RNN、LSTM 和 GRU，都支持將打包序列作為輸入，並生成打包輸出，這些輸出可以使用 `torch.nn.utils.rnn.pad_packed_sequence` 進行解碼。\n",
|
||||
"\n",
|
||||
"為了能夠生成打包序列，我們需要將長度向量傳遞給網路，因此我們需要一個不同的函數來準備小批次：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def pad_length(b):\n",
|
||||
" # build vectorized sequence\n",
|
||||
" v = [encode(x[1]) for x in b]\n",
|
||||
" # compute max length of a sequence in this minibatch and length sequence itself\n",
|
||||
" len_seq = list(map(len,v))\n",
|
||||
" l = max(len_seq)\n",
|
||||
" return ( # tuple of three tensors - labels, padded features, length sequence\n",
|
||||
" torch.LongTensor([t[0]-1 for t in b]),\n",
|
||||
" torch.stack([torch.nn.functional.pad(torch.tensor(t),(0,l-len(t)),mode='constant',value=0) for t in v]),\n",
|
||||
" torch.tensor(len_seq)\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"train_loader_len = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=pad_length, shuffle=True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"實際的網路結構會與上面的 `LSTMClassifier` 非常相似，但 `forward` 傳遞過程會同時接收填充過的迷你批次和序列長度的向量。在計算嵌入後，我們會計算打包序列，將其傳遞給 LSTM 層，然後再將結果解包回來。\n",
|
||||
"\n",
|
||||
"> **注意**：我們實際上並未使用解包後的結果 `x`，因為我們在後續的計算中使用的是隱藏層的輸出。因此，我們可以完全從這段程式碼中移除解包的部分。我們之所以將其保留在這裡，是為了讓您在需要在後續計算中使用網路輸出時，能夠更輕鬆地修改這段程式碼。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class LSTMPackClassifier(torch.nn.Module):\n",
|
||||
" def __init__(self, vocab_size, embed_dim, hidden_dim, num_class):\n",
|
||||
" super().__init__()\n",
|
||||
" self.hidden_dim = hidden_dim\n",
|
||||
" self.embedding = torch.nn.Embedding(vocab_size, embed_dim)\n",
|
||||
" self.embedding.weight.data = torch.randn_like(self.embedding.weight.data)-0.5\n",
|
||||
" self.rnn = torch.nn.LSTM(embed_dim,hidden_dim,batch_first=True)\n",
|
||||
" self.fc = torch.nn.Linear(hidden_dim, num_class)\n",
|
||||
"\n",
|
||||
" def forward(self, x, lengths):\n",
|
||||
" batch_size = x.size(0)\n",
|
||||
" x = self.embedding(x)\n",
|
||||
" pad_x = torch.nn.utils.rnn.pack_padded_sequence(x,lengths,batch_first=True,enforce_sorted=False)\n",
|
||||
" pad_x,(h,c) = self.rnn(pad_x)\n",
|
||||
" x, _ = torch.nn.utils.rnn.pad_packed_sequence(pad_x,batch_first=True)\n",
|
||||
" return self.fc(h[-1])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.285625\n",
|
||||
"6400: acc=0.33359375\n",
|
||||
"9600: acc=0.3876041666666667\n",
|
||||
"12800: acc=0.44078125\n",
|
||||
"16000: acc=0.4825\n",
|
||||
"19200: acc=0.5235416666666667\n",
|
||||
"22400: acc=0.5559821428571429\n",
|
||||
"25600: acc=0.58609375\n",
|
||||
"28800: acc=0.6116666666666667\n",
|
||||
"32000: acc=0.63340625\n",
|
||||
"35200: acc=0.6525284090909091\n",
|
||||
"38400: acc=0.668515625\n",
|
||||
"41600: acc=0.6822596153846154\n",
|
||||
"44800: acc=0.6948214285714286\n",
|
||||
"48000: acc=0.7052708333333333\n",
|
||||
"51200: acc=0.71521484375\n",
|
||||
"54400: acc=0.7239889705882353\n",
|
||||
"57600: acc=0.7315277777777778\n",
|
||||
"60800: acc=0.7388486842105263\n",
|
||||
"64000: acc=0.74571875\n",
|
||||
"67200: acc=0.7518303571428572\n",
|
||||
"70400: acc=0.7576988636363636\n",
|
||||
"73600: acc=0.7628940217391305\n",
|
||||
"76800: acc=0.7681510416666667\n",
|
||||
"80000: acc=0.7728125\n",
|
||||
"83200: acc=0.7772235576923077\n",
|
||||
"86400: acc=0.7815393518518519\n",
|
||||
"89600: acc=0.7857700892857142\n",
|
||||
"92800: acc=0.7895043103448276\n",
|
||||
"96000: acc=0.7930520833333333\n",
|
||||
"99200: acc=0.7959072580645161\n",
|
||||
"102400: acc=0.798994140625\n",
|
||||
"105600: acc=0.802064393939394\n",
|
||||
"108800: acc=0.8051378676470589\n",
|
||||
"112000: acc=0.8077857142857143\n",
|
||||
"115200: acc=0.8104600694444445\n",
|
||||
"118400: acc=0.8128293918918919\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(0.029785829671223958, 0.8138166666666666)"
|
||||
]
|
||||
},
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"net = LSTMPackClassifier(vocab_size,64,32,len(classes)).to(device)\n",
|
||||
"train_epoch_emb(net,train_loader_len, lr=0.001,use_pack_sequence=True)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **注意：** 您可能已注意到我們傳遞給訓練函數的參數 `use_pack_sequence`。目前，`pack_padded_sequence` 函數要求長度序列張量位於 CPU 設備上，因此訓練函數需要避免在訓練時將長度序列數據移動到 GPU。您可以查看 [`torchnlp.py`](../../../../../lessons/5-NLP/16-RNN/torchnlp.py) 文件中 `train_emb` 函數的實現。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 雙向與多層 RNN\n",
|
||||
"\n",
|
||||
"在我們的範例中，所有的循環神經網路都是單向運作的，從序列的開頭到結尾。這看起來很自然，因為它類似於我們閱讀或聆聽語音的方式。然而，在許多實際情況下，我們可以隨機存取輸入序列，因此在兩個方向上進行循環計算可能更有意義。這類網路稱為 **雙向** RNN，可以透過在 RNN/LSTM/GRU 建構函數中傳入 `bidirectional=True` 參數來建立。\n",
|
||||
"\n",
|
||||
"在處理雙向網路時，我們需要兩個隱藏狀態向量，分別對應每個方向。PyTorch 將這些向量編碼為一個大小加倍的向量，這非常方便，因為通常我們會將結果隱藏狀態傳遞給全連接線性層，只需在建立該層時考慮到這個大小的增加即可。\n",
|
||||
"\n",
|
||||
"無論是單向還是雙向的循環網路，都能捕捉序列中的某些模式，並將其存儲到狀態向量中或傳遞到輸出中。與卷積網路類似，我們可以在第一層之上構建另一個循環層，以捕捉更高層次的模式，這些模式是由第一層提取的低層次模式構成的。這引出了 **多層 RNN** 的概念，它由兩層或更多的循環網路組成，前一層的輸出作為下一層的輸入。\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"*圖片來源：[這篇精彩的文章](https://towardsdatascience.com/from-a-lstm-cell-to-a-multilayer-lstm-network-with-pytorch-2899eb5696f3) 作者 Fernando López*\n",
|
||||
"\n",
|
||||
"PyTorch 讓構建這類網路變得非常簡單，因為只需在 RNN/LSTM/GRU 建構函數中傳入 `num_layers` 參數，即可自動建立多層循環網路。這也意味著隱藏/狀態向量的大小會按比例增加，處理循環層的輸出時需要考慮到這一點。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## RNN 用於其他任務\n",
|
||||
"\n",
|
||||
"在本單元中，我們已經看到 RNN 可以用於序列分類，但事實上，它們還能處理更多任務，例如文本生成、機器翻譯等等。我們將在下一單元中探討這些任務。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**免責聲明**： \n本文件已使用 AI 翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。儘管我們努力確保翻譯的準確性，但請注意，自動翻譯可能包含錯誤或不準確之處。原始文件的母語版本應被視為權威來源。對於關鍵信息，建議使用專業人工翻譯。我們對因使用此翻譯而引起的任何誤解或誤釋不承擔責任。\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.8.12 ('py38')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "522ee52ae3d5ae933e283286254e9a55",
|
||||
"translation_date": "2025-08-28T12:22:38+00:00",
|
||||
"source_file": "lessons/5-NLP/16-RNN/RNNPyTorch.ipynb",
|
||||
"language_code": "mo"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
|
|
@ -0,0 +1,460 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# 循環神經網路\n",
|
||||
"\n",
|
||||
"在上一個模組中，我們介紹了文本的豐富語義表示。我們使用的架構能夠捕捉句子中單詞的聚合意義，但它並未考慮單詞的**順序**，因為嵌入後的聚合操作會將原始文本中的這些信息移除。由於這些模型無法表示單詞的順序，因此它們無法解決更複雜或更具歧義的任務，例如文本生成或問題回答。\n",
|
||||
"\n",
|
||||
"為了捕捉文本序列的意義，我們將使用一種稱為**循環神經網路**（Recurrent Neural Network，簡稱 RNN）的神經網路架構。在使用 RNN 時，我們會將句子逐個標記（token）傳遞給網路，網路會生成某種**狀態**，然後我們將該狀態與下一個標記一起再次傳遞給網路。\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"給定標記輸入序列 $X_0,\\dots,X_n$，RNN 會創建一個神經網路區塊的序列，並通過反向傳播對該序列進行端到端訓練。每個網路區塊接受一對 $(X_i,S_i)$ 作為輸入，並生成 $S_{i+1}$ 作為結果。最終狀態 $S_n$ 或輸出 $Y_n$ 會進入線性分類器以生成結果。所有網路區塊共享相同的權重，並通過一次反向傳播訓練完成端到端學習。\n",
|
||||
"\n",
|
||||
"> 上圖展示了循環神經網路的展開形式（左側）和更緊湊的循環表示形式（右側）。需要注意的是，所有 RNN 單元都具有相同的**可共享權重**。\n",
|
||||
"\n",
|
||||
"由於狀態向量 $S_0,\\dots,S_n$ 是通過網路傳遞的，RNN 能夠學習單詞之間的順序依賴關係。例如，當單詞 *not* 出現在序列中的某處時，它可以學會在狀態向量中否定某些元素。\n",
|
||||
"\n",
|
||||
"在內部，每個 RNN 單元包含兩個權重矩陣：$W_H$ 和 $W_I$，以及偏置 $b$。在每個 RNN 步驟中，給定輸入 $X_i$ 和輸入狀態 $S_i$，輸出狀態的計算方式為 $S_{i+1} = f(W_H\\times S_i + W_I\\times X_i+b)$，其中 $f$ 是激活函數（通常是 $\\tanh$）。\n",
|
||||
"\n",
|
||||
"> 對於像文本生成（我們將在下一單元中介紹）或機器翻譯這樣的問題，我們還希望在每個 RNN 步驟中獲得一些輸出值。在這種情況下，還會有另一個矩陣 $W_O$，輸出值的計算方式為 $Y_i=f(W_O\\times S_i+b_O)$。\n",
|
||||
"\n",
|
||||
"現在讓我們看看循環神經網路如何幫助我們對新聞數據集進行分類。\n",
|
||||
"\n",
|
||||
"> 在沙盒環境中，我們需要運行以下程式碼單元以確保安裝了所需的庫並預取了數據。如果您在本地運行，則可以跳過以下程式碼單元。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import sys\n",
|
||||
"!{sys.executable} -m pip install --quiet tensorflow_datasets==4.4.0\n",
|
||||
"!cd ~ && wget -q -O - https://mslearntensorflowlp.blob.core.windows.net/data/tfds-ag-news.tgz | tar xz"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import tensorflow as tf\n",
|
||||
"from tensorflow import keras\n",
|
||||
"import tensorflow_datasets as tfds\n",
|
||||
"import numpy as np\n",
|
||||
"\n",
|
||||
"# We are going to be training pretty large models. In order not to face errors, we need\n",
|
||||
"# to set tensorflow option to grow GPU memory allocation when required\n",
|
||||
"physical_devices = tf.config.list_physical_devices('GPU') \n",
|
||||
"if len(physical_devices)>0:\n",
|
||||
" tf.config.experimental.set_memory_growth(physical_devices[0], True)\n",
|
||||
"\n",
|
||||
"ds_train, ds_test = tfds.load('ag_news_subset').values()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"nteract": {
|
||||
"transient": {
|
||||
"deleting": false
|
||||
}
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"在訓練大型模型時，GPU 記憶體分配可能會成為一個問題。我們可能需要嘗試不同的迷你批次大小，以確保資料能夠適配 GPU 記憶體，同時訓練速度也足夠快。如果你是在自己的 GPU 機器上運行這段程式碼，可以嘗試調整迷你批次大小來加快訓練速度。\n",
|
||||
"\n",
|
||||
"> **注意**：某些版本的 NVidia 驅動程式已知在訓練模型後不會釋放記憶體。我們在這個筆記本中運行了多個範例，這可能會導致在某些配置中記憶體耗盡，特別是當你在同一個筆記本中進行自己的實驗時。如果在開始訓練模型時遇到一些奇怪的錯誤，建議重新啟動筆記本的內核。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {
|
||||
"collapsed": true,
|
||||
"jupyter": {
|
||||
"outputs_hidden": false,
|
||||
"source_hidden": false
|
||||
},
|
||||
"nteract": {
|
||||
"transient": {
|
||||
"deleting": false
|
||||
}
|
||||
}
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"batch_size = 16\n",
|
||||
"embed_size = 64"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 簡單的 RNN 分類器\n",
|
||||
"\n",
|
||||
"在簡單的 RNN 中，每個循環單元都是一個簡單的線性網絡，它接收輸入向量和狀態向量，並生成新的狀態向量。在 Keras 中，可以使用 `SimpleRNN` 層來表示。\n",
|
||||
"\n",
|
||||
"雖然我們可以直接將獨熱編碼的標記傳遞給 RNN 層，但這並不是一個好主意，因為它們的維度太高。因此，我們會使用嵌入層來降低詞向量的維度，接著是 RNN 層，最後是一個 `Dense` 分類器。\n",
|
||||
"\n",
|
||||
"> **注意**：在維度不是很高的情況下，例如使用字符級標記化時，直接將獨熱編碼的標記傳遞給 RNN 單元可能是合理的選擇。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"sequential\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
"Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
"text_vectorization (TextVect (None, None) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"embedding (Embedding) (None, None, 64) 1280000 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"simple_rnn (SimpleRNN) (None, 16) 1296 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense (Dense) (None, 4) 68 \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 1,281,364\n",
|
||||
"Trainable params: 1,281,364\n",
|
||||
"Non-trainable params: 0\n",
|
||||
"_________________________________________________________________\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vocab_size = 20000\n",
|
||||
"\n",
|
||||
"vectorizer = keras.layers.experimental.preprocessing.TextVectorization(\n",
|
||||
" max_tokens=vocab_size,\n",
|
||||
" input_shape=(1,))\n",
|
||||
"\n",
|
||||
"model = keras.models.Sequential([\n",
|
||||
" vectorizer,\n",
|
||||
" keras.layers.Embedding(vocab_size, embed_size),\n",
|
||||
" keras.layers.SimpleRNN(16),\n",
|
||||
" keras.layers.Dense(4,activation='softmax')\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"model.summary()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **注意：** 為了簡化，我們在這裡使用未經訓練的嵌入層，但如果想要更好的效果，可以使用 Word2Vec 預訓練的嵌入層，如前一單元所述。你可以嘗試將此程式碼改寫為使用預訓練嵌入層，這會是一個很好的練習。\n",
|
||||
"\n",
|
||||
"現在讓我們來訓練 RNN。一般來說，訓練 RNN 是相當困難的，因為當 RNN 單元沿著序列長度展開時，反向傳播所涉及的層數會非常多。因此，我們需要選擇較小的學習率，並在更大的數據集上訓練網路以獲得良好的結果。這可能需要相當長的時間，因此建議使用 GPU。\n",
|
||||
"\n",
|
||||
"為了加快速度，我們將僅使用新聞標題來訓練 RNN 模型，省略描述部分。你可以嘗試加入描述進行訓練，看看是否能讓模型成功訓練。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Training vectorizer\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def extract_title(x):\n",
|
||||
" return x['title']\n",
|
||||
"\n",
|
||||
"def tupelize_title(x):\n",
|
||||
" return (extract_title(x),x['label'])\n",
|
||||
"\n",
|
||||
"print('Training vectorizer')\n",
|
||||
"vectorizer.adapt(ds_train.take(2000).map(extract_title))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"7500/7500 [==============================] - 82s 11ms/step - loss: 0.6629 - acc: 0.7623 - val_loss: 0.5559 - val_acc: 0.7995\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7f3e0030d350>"
|
||||
]
|
||||
},
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n",
|
||||
"model.fit(ds_train.map(tupelize_title).batch(batch_size),validation_data=ds_test.map(tupelize_title).batch(batch_size))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"nteract": {
|
||||
"transient": {
|
||||
"deleting": false
|
||||
}
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"> **注意** 由於我們僅在新聞標題上進行訓練，因此準確性可能較低。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 重新探討變長序列\n",
|
||||
"\n",
|
||||
"請記住，`TextVectorization` 層會自動在小批量中使用填充標記來填充變長序列。然而，這些填充標記也會參與訓練，可能會使模型的收斂變得更加複雜。\n",
|
||||
"\n",
|
||||
"我們可以採取幾種方法來減少填充的數量。其中一種方法是根據序列長度重新排列數據集，並按大小分組所有序列。這可以使用 `tf.data.experimental.bucket_by_sequence_length` 函數來完成（請參閱[文件](https://www.tensorflow.org/api_docs/python/tf/data/experimental/bucket_by_sequence_length)）。\n",
|
||||
"\n",
|
||||
"另一種方法是使用**遮罩**。在 Keras 中，某些層支持額外的輸入，用於指示哪些標記應在訓練中被考慮。要將遮罩整合到模型中，我們可以選擇添加一個單獨的 `Masking` 層（[文件](https://keras.io/api/layers/core_layers/masking/)），或者在 `Embedding` 層中指定 `mask_zero=True` 參數。\n",
|
||||
"\n",
|
||||
"> **Note**: 完成整個數據集的一個訓練週期大約需要 5 分鐘。如果您失去耐心，可以隨時中斷訓練。您也可以通過在 `ds_train` 和 `ds_test` 數據集後添加 `.take(...)` 限制用於訓練的數據量。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"7500/7500 [==============================] - 371s 49ms/step - loss: 0.5401 - acc: 0.8079 - val_loss: 0.3780 - val_acc: 0.8822\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7f3dec118850>"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def extract_text(x):\n",
|
||||
" return x['title']+' '+x['description']\n",
|
||||
"\n",
|
||||
"def tupelize(x):\n",
|
||||
" return (extract_text(x),x['label'])\n",
|
||||
"\n",
|
||||
"model = keras.models.Sequential([\n",
|
||||
" vectorizer,\n",
|
||||
" keras.layers.Embedding(vocab_size,embed_size,mask_zero=True),\n",
|
||||
" keras.layers.SimpleRNN(16),\n",
|
||||
" keras.layers.Dense(4,activation='softmax')\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"現在我們使用遮罩技術，可以在整個標題和描述的數據集上訓練模型。\n",
|
||||
"\n",
|
||||
"> **注意**：你是否注意到我們一直使用的是基於新聞標題訓練的向量化工具，而不是整篇文章的內容？這可能導致部分詞彙被忽略，因此重新訓練向量化工具會更好。然而，這可能只會帶來非常小的影響，所以為了簡化流程，我們將繼續使用之前的預訓練向量化工具。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## LSTM: 長短期記憶\n",
|
||||
"\n",
|
||||
"RNN 的主要問題之一是 **梯度消失**。RNN 的結構可能會很長，在反向傳播時，梯度可能很難傳遞回網路的第一層。當這種情況發生時，網路無法學習遠距離的詞元之間的關係。為了解決這個問題，可以通過使用 **門控機制** 引入 **顯式狀態管理**。最常見的兩種引入門控機制的架構是 **長短期記憶** (LSTM) 和 **門控循環單元** (GRU)。我們在這裡將討論 LSTM。\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"LSTM 網路的組織方式與 RNN 類似，但有兩個狀態會從一層傳遞到下一層：實際的狀態 $c$ 和隱藏向量 $h$。在每個單元中，隱藏向量 $h_{t-1}$ 與輸入 $x_t$ 結合，並共同控制狀態 $c_t$ 和輸出 $h_t$ 的變化，這是通過 **門控機制** 完成的。每個門都有 sigmoid 激活函數（輸出範圍為 $[0,1]$），可以將其視為在與狀態向量相乘時的位掩碼。LSTM 包含以下幾種門（從上圖的左到右）：\n",
|
||||
"* **遺忘門**：決定向量 $c_{t-1}$ 的哪些部分需要遺忘，哪些需要保留。\n",
|
||||
"* **輸入門**：決定來自輸入向量和前一隱藏向量的多少信息應該被整合到狀態向量中。\n",
|
||||
"* **輸出門**：接收新的狀態向量，並決定其哪些部分將用於生成新的隱藏向量 $h_t$。\n",
|
||||
"\n",
|
||||
"狀態 $c$ 的組成部分可以被視為可以開啟或關閉的標誌。例如，當我們在序列中遇到名字 *Alice* 時，我們猜測這指的是一位女性，並在狀態中設置一個標誌，表示句子中有一個女性名詞。當我們進一步遇到詞語 *and Tom* 時，我們會設置另一個標誌，表示句子中有一個複數名詞。因此，通過操作狀態，我們可以追蹤句子的語法屬性。\n",
|
||||
"\n",
|
||||
"> **Note**: 這裡有一個很棒的資源可以幫助理解 LSTM 的內部結構：[Understanding LSTM Networks](https://colah.github.io/posts/2015-08-Understanding-LSTMs/) by Christopher Olah。\n",
|
||||
"\n",
|
||||
"雖然 LSTM 單元的內部結構看起來可能很複雜，但 Keras 將這些實現隱藏在 `LSTM` 層中，因此在上面的範例中，我們只需要替換循環層即可：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"15000/15000 [==============================] - 188s 13ms/step - loss: 0.5692 - acc: 0.7916 - val_loss: 0.3441 - val_acc: 0.8870\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7f3d6af5c350>"
|
||||
]
|
||||
},
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model = keras.models.Sequential([\n",
|
||||
" vectorizer,\n",
|
||||
" keras.layers.Embedding(vocab_size, embed_size),\n",
|
||||
" keras.layers.LSTM(8),\n",
|
||||
" keras.layers.Dense(4,activation='softmax')\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(8),validation_data=ds_test.map(tupelize).batch(8))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 雙向與多層 RNN\n",
|
||||
"\n",
|
||||
"在我們之前的例子中，循環神經網絡都是從序列的開頭運算到結尾。這對我們來說很自然，因為它遵循了我們閱讀或聆聽語音的方向。然而，對於需要隨機存取輸入序列的情境，讓循環運算在兩個方向上進行會更合理。允許在兩個方向上進行運算的 RNN 被稱為 **雙向** RNN，可以通過將循環層包裹在特殊的 `Bidirectional` 層中來創建。\n",
|
||||
"\n",
|
||||
"> **Note**: `Bidirectional` 層會在其內部生成該層的兩個副本，並將其中一個副本的 `go_backwards` 屬性設置為 `True`，使其沿著序列的相反方向運算。\n",
|
||||
"\n",
|
||||
"無論是單向還是雙向的循環網絡，都能捕捉序列中的模式，並將其存儲到狀態向量中或作為輸出返回。與卷積網絡類似，我們可以在第一層循環層之後再構建另一層循環層，以捕捉更高層次的模式，這些模式是由第一層提取的低層次模式構建而成的。這引出了 **多層 RNN** 的概念，它由兩層或更多層循環網絡組成，其中前一層的輸出作為下一層的輸入。\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"*圖片來源：[這篇精彩的文章](https://towardsdatascience.com/from-a-lstm-cell-to-a-multilayer-lstm-network-with-pytorch-2899eb5696f3)，作者 Fernando López。*\n",
|
||||
"\n",
|
||||
"Keras 讓構建這些網絡變得非常簡單，因為你只需要在模型中添加更多的循環層。對於最後一層以外的所有層，我們需要指定 `return_sequences=True` 參數，因為我們需要該層返回所有中間狀態，而不僅僅是循環運算的最終狀態。\n",
|
||||
"\n",
|
||||
"現在，我們來為分類問題構建一個雙層雙向 LSTM。\n",
|
||||
"\n",
|
||||
"> **Note** 這段程式碼執行時間較長，但它提供了迄今為止我們見過的最高準確率。所以也許值得等待並查看結果。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"5044/7500 [===================>..........] - ETA: 2:33 - loss: 0.3709 - acc: 0.8706\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r5045/7500 [===================>..........] - ETA: 2:33 - loss: 0.3709 - acc: 0.8706"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model = keras.models.Sequential([\n",
|
||||
" vectorizer,\n",
|
||||
" keras.layers.Embedding(vocab_size, 128, mask_zero=True),\n",
|
||||
" keras.layers.Bidirectional(keras.layers.LSTM(64,return_sequences=True)),\n",
|
||||
" keras.layers.Bidirectional(keras.layers.LSTM(64)), \n",
|
||||
" keras.layers.Dense(4,activation='softmax')\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(batch_size),\n",
|
||||
" validation_data=ds_test.map(tupelize).batch(batch_size))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## RNNs 用於其他任務\n",
|
||||
"\n",
|
||||
"到目前為止，我們專注於使用 RNNs 來對文本序列進行分類。但它們還能處理更多任務，例如文本生成和機器翻譯——我們將在下一單元中探討這些任務。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**免責聲明**： \n本文件已使用 AI 翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。儘管我們努力確保翻譯的準確性，但請注意，自動翻譯可能包含錯誤或不準確之處。原始文件的母語版本應被視為權威來源。對於關鍵資訊，建議尋求專業人工翻譯。我們對因使用此翻譯而引起的任何誤解或錯誤解釋不承擔責任。\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernel_info": {
|
||||
"name": "conda-env-py37_tensorflow-py"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "py37_tensorflow",
|
||||
"language": "python",
|
||||
"name": "conda-env-py37_tensorflow-py"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.7.9"
|
||||
},
|
||||
"nteract": {
|
||||
"version": "nteract-front-end@1.0.0"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "81351e61f619b432ff51010a4f993194",
|
||||
"translation_date": "2025-08-28T12:19:31+00:00",
|
||||
"source_file": "lessons/5-NLP/16-RNN/RNNTF.ipynb",
|
||||
"language_code": "mo"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
|
|
@ -0,0 +1,414 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# 生成式網絡\n",
|
||||
"\n",
|
||||
"循環神經網絡（RNNs）及其門控單元變體，例如長短期記憶單元（LSTMs）和門控循環單元（GRUs），提供了一種語言建模的機制，也就是說，它們可以學習詞語的排列順序，並對序列中的下一個詞進行預測。這使得我們可以使用 RNNs 進行**生成任務**，例如普通文本生成、機器翻譯，甚至是圖像描述。\n",
|
||||
"\n",
|
||||
"在上一單元中討論的 RNN 架構中，每個 RNN 單元都會生成下一個隱藏狀態作為輸出。然而，我們也可以為每個循環單元添加另一個輸出，這樣就可以輸出一個**序列**（其長度與原始序列相等）。此外，我們還可以使用不在每一步接受輸入的 RNN 單元，而僅僅接受一些初始狀態向量，然後生成一系列輸出。\n",
|
||||
"\n",
|
||||
"在這份筆記中，我們將專注於幫助生成文本的簡單生成模型。為了簡化，我們將構建**字元級網絡**，逐字生成文本。在訓練過程中，我們需要使用一些文本語料庫，並將其拆分為字元序列。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Loading dataset...\n",
|
||||
"Building vocab...\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"import torchtext\n",
|
||||
"import numpy as np\n",
|
||||
"from torchnlp import *\n",
|
||||
"train_dataset,test_dataset,classes,vocab = load_dataset()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 建立字元詞彙表\n",
|
||||
"\n",
|
||||
"要建立字元級生成網絡，我們需要將文本拆分為單個字元，而不是單詞。這可以通過定義不同的分詞器來完成：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Vocabulary size = 82\n",
|
||||
"Encoding of 'a' is 1\n",
|
||||
"Character with code 13 is c\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def char_tokenizer(words):\n",
|
||||
" return list(words) #[word for word in words]\n",
|
||||
"\n",
|
||||
"counter = collections.Counter()\n",
|
||||
"for (label, line) in train_dataset:\n",
|
||||
" counter.update(char_tokenizer(line))\n",
|
||||
"vocab = torchtext.vocab.vocab(counter)\n",
|
||||
"\n",
|
||||
"vocab_size = len(vocab)\n",
|
||||
"print(f\"Vocabulary size = {vocab_size}\")\n",
|
||||
"print(f\"Encoding of 'a' is {vocab.get_stoi()['a']}\")\n",
|
||||
"print(f\"Character with code 13 is {vocab.get_itos()[13]}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"讓我們看看如何從我們的數據集中編碼文本的示例：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"tensor([ 0, 1, 2, 2, 3, 4, 5, 6, 3, 7, 8, 1, 9, 10, 3, 11, 2, 1,\n",
|
||||
" 12, 3, 7, 1, 13, 14, 3, 15, 16, 5, 17, 3, 5, 18, 8, 3, 7, 2,\n",
|
||||
" 1, 13, 14, 3, 19, 20, 8, 21, 5, 8, 9, 10, 22, 3, 20, 8, 21, 5,\n",
|
||||
" 8, 9, 10, 3, 23, 3, 4, 18, 17, 9, 5, 23, 10, 8, 2, 2, 8, 9,\n",
|
||||
" 10, 24, 3, 0, 1, 2, 2, 3, 4, 5, 9, 8, 8, 5, 25, 10, 3, 26,\n",
|
||||
" 12, 27, 16, 26, 2, 27, 16, 28, 29, 30, 1, 16, 26, 3, 17, 31, 3, 21,\n",
|
||||
" 2, 5, 9, 1, 23, 13, 32, 16, 27, 13, 10, 24, 3, 1, 9, 8, 3, 10,\n",
|
||||
" 8, 8, 27, 16, 28, 3, 28, 9, 8, 8, 16, 3, 1, 28, 1, 27, 16, 6])"
|
||||
]
|
||||
},
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def enc(x):\n",
|
||||
" return torch.LongTensor(encode(x,voc=vocab,tokenizer=char_tokenizer))\n",
|
||||
"\n",
|
||||
"enc(train_dataset[0][1])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 訓練生成式 RNN\n",
|
||||
"\n",
|
||||
"我們將以以下方式訓練 RNN 來生成文本。在每一步中，我們會取一段長度為 `nchars` 的字元序列，並讓網路為每個輸入字元生成下一個輸出字元：\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"根據實際情況，我們可能還需要加入一些特殊字元，例如 *序列結束符* `<eos>`。在我們的例子中，我們只希望訓練網路進行無限文本生成，因此我們會將每個序列的大小固定為 `nchars` 個標記。因此，每個訓練樣本將包含 `nchars` 個輸入和 `nchars` 個輸出（輸出是將輸入序列向左移動一個符號後的結果）。一個小批次（minibatch）將由多個這樣的序列組成。\n",
|
||||
"\n",
|
||||
"我們生成小批次的方式是取每段長度為 `l` 的新聞文本，並從中生成所有可能的輸入-輸出組合（這樣的組合會有 `l-nchars` 個）。這些組合將構成一個小批次，而每次訓練步驟的小批次大小會有所不同。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(tensor([[ 0, 1, 2, ..., 28, 29, 30],\n",
|
||||
" [ 1, 2, 2, ..., 29, 30, 1],\n",
|
||||
" [ 2, 2, 3, ..., 30, 1, 16],\n",
|
||||
" ...,\n",
|
||||
" [20, 8, 21, ..., 1, 28, 1],\n",
|
||||
" [ 8, 21, 5, ..., 28, 1, 27],\n",
|
||||
" [21, 5, 8, ..., 1, 27, 16]]),\n",
|
||||
" tensor([[ 1, 2, 2, ..., 29, 30, 1],\n",
|
||||
" [ 2, 2, 3, ..., 30, 1, 16],\n",
|
||||
" [ 2, 3, 4, ..., 1, 16, 26],\n",
|
||||
" ...,\n",
|
||||
" [ 8, 21, 5, ..., 28, 1, 27],\n",
|
||||
" [21, 5, 8, ..., 1, 27, 16],\n",
|
||||
" [ 5, 8, 9, ..., 27, 16, 6]]))"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"nchars = 100\n",
|
||||
"\n",
|
||||
"def get_batch(s,nchars=nchars):\n",
|
||||
" ins = torch.zeros(len(s)-nchars,nchars,dtype=torch.long,device=device)\n",
|
||||
" outs = torch.zeros(len(s)-nchars,nchars,dtype=torch.long,device=device)\n",
|
||||
" for i in range(len(s)-nchars):\n",
|
||||
" ins[i] = enc(s[i:i+nchars])\n",
|
||||
" outs[i] = enc(s[i+1:i+nchars+1])\n",
|
||||
" return ins,outs\n",
|
||||
"\n",
|
||||
"get_batch(train_dataset[0][1])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"現在我們來定義生成器網路。它可以基於我們在上一單元討論過的任何一種循環單元（簡單的、LSTM 或 GRU）。在我們的例子中，我們將使用 LSTM。\n",
|
||||
"\n",
|
||||
"由於網路的輸入是字符，且詞彙表的大小相對較小，因此我們不需要嵌入層，使用獨熱編碼的輸入可以直接傳遞到 LSTM 單元。然而，因為我們是以字符的數字作為輸入，所以在傳遞給 LSTM 之前需要對它們進行獨熱編碼。這可以在 `forward` 傳遞過程中調用 `one_hot` 函數來完成。輸出編碼器將是一個線性層，用於將隱藏狀態轉換為獨熱編碼的輸出。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class LSTMGenerator(torch.nn.Module):\n",
|
||||
" def __init__(self, vocab_size, hidden_dim):\n",
|
||||
" super().__init__()\n",
|
||||
" self.rnn = torch.nn.LSTM(vocab_size,hidden_dim,batch_first=True)\n",
|
||||
" self.fc = torch.nn.Linear(hidden_dim, vocab_size)\n",
|
||||
"\n",
|
||||
" def forward(self, x, s=None):\n",
|
||||
" x = torch.nn.functional.one_hot(x,vocab_size).to(torch.float32)\n",
|
||||
" x,s = self.rnn(x,s)\n",
|
||||
" return self.fc(x),s"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"在訓練過程中，我們希望能夠抽樣生成的文本。為了達到這個目的，我們將定義一個 `generate` 函數，該函數會生成長度為 `size` 的輸出字串，並以初始字串 `start` 作為起點。\n",
|
||||
"\n",
|
||||
"其運作方式如下：首先，我們會將整個初始字串通過網絡，並獲得輸出狀態 `s` 和下一個預測字符 `out`。由於 `out` 是獨熱編碼（one-hot encoded），我們使用 `argmax` 來獲取該字符在詞彙表中的索引 `nc`，然後利用 `itos` 找出實際字符，並將其附加到結果字符列表 `chars` 中。這個生成單個字符的過程會重複執行 `size` 次，以生成所需數量的字符。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def generate(net,size=100,start='today '):\n",
|
||||
" chars = list(start)\n",
|
||||
" out, s = net(enc(chars).view(1,-1).to(device))\n",
|
||||
" for i in range(size):\n",
|
||||
" nc = torch.argmax(out[0][-1])\n",
|
||||
" chars.append(vocab.get_itos()[nc])\n",
|
||||
" out, s = net(nc.view(1,-1),s)\n",
|
||||
" return ''.join(chars)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"現在開始訓練吧！訓練迴圈幾乎和之前的例子一樣，但這次我們不是打印準確率，而是每隔 1000 個 epoch 打印生成的樣本文字。\n",
|
||||
"\n",
|
||||
"需要特別注意的是計算損失的方式。我們需要根據 one-hot 編碼的輸出 `out` 和期望的文字 `text_out`（即字符索引的列表）來計算損失。幸運的是，`cross_entropy` 函數的第一個參數是未正規化的網絡輸出，第二個參數是類別編號，這正好符合我們的需求。它還會自動對小批量的大小進行平均。\n",
|
||||
"\n",
|
||||
"此外，我們通過 `samples_to_train` 限制訓練樣本的數量，以避免等待過久。我們鼓勵你進行實驗，嘗試更長時間的訓練，可能是多個 epoch（在這種情況下，你需要在這段代碼外再建立一個迴圈）。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Current loss = 4.398899078369141\n",
|
||||
"today sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr s\n",
|
||||
"Current loss = 2.161320447921753\n",
|
||||
"today and to the tor to to the tor to to the tor to to the tor to to the tor to to the tor to to the tor t\n",
|
||||
"Current loss = 1.6722588539123535\n",
|
||||
"today and the court to the could to the could to the could to the could to the could to the could to the c\n",
|
||||
"Current loss = 2.423795223236084\n",
|
||||
"today and a second to the conternation of the conternation of the conternation of the conternation of the \n",
|
||||
"Current loss = 1.702607274055481\n",
|
||||
"today and the company to the company to the company to the company to the company to the company to the co\n",
|
||||
"Current loss = 1.692358136177063\n",
|
||||
"today and the company to the company to the company to the company to the company to the company to the co\n",
|
||||
"Current loss = 1.9722288846969604\n",
|
||||
"today and the control the control the control the control the control the control the control the control \n",
|
||||
"Current loss = 1.8705692291259766\n",
|
||||
"today and the second to the second to the second to the second to the second to the second to the second t\n",
|
||||
"Current loss = 1.7626899480819702\n",
|
||||
"today and a security and a security and a security and a security and a security and a security and a secu\n",
|
||||
"Current loss = 1.5574463605880737\n",
|
||||
"today and the company and the company and the company and the company and the company and the company and \n",
|
||||
"Current loss = 1.5620026588439941\n",
|
||||
"today and the be that the be the be that the be the be that the be the be that the be the be that the be t\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"net = LSTMGenerator(vocab_size,64).to(device)\n",
|
||||
"\n",
|
||||
"samples_to_train = 10000\n",
|
||||
"optimizer = torch.optim.Adam(net.parameters(),0.01)\n",
|
||||
"loss_fn = torch.nn.CrossEntropyLoss()\n",
|
||||
"net.train()\n",
|
||||
"for i,x in enumerate(train_dataset):\n",
|
||||
" # x[0] is class label, x[1] is text\n",
|
||||
" if len(x[1])-nchars<10:\n",
|
||||
" continue\n",
|
||||
" samples_to_train-=1\n",
|
||||
" if not samples_to_train: break\n",
|
||||
" text_in, text_out = get_batch(x[1])\n",
|
||||
" optimizer.zero_grad()\n",
|
||||
" out,s = net(text_in)\n",
|
||||
" loss = torch.nn.functional.cross_entropy(out.view(-1,vocab_size),text_out.flatten()) #cross_entropy(out,labels)\n",
|
||||
" loss.backward()\n",
|
||||
" optimizer.step()\n",
|
||||
" if i%1000==0:\n",
|
||||
" print(f\"Current loss = {loss.item()}\")\n",
|
||||
" print(generate(net))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"這個範例已經能生成相當不錯的文本，但仍有幾個方面可以進一步改進：\n",
|
||||
"\n",
|
||||
"* **更好的小批次生成**。我們在準備訓練數據時，是從一個樣本中生成一個小批次。這種方式並不理想，因為小批次的大小各不相同，有些甚至無法生成，因為文本小於 `nchars`。此外，過小的小批次無法充分利用 GPU 的性能。更明智的做法是從所有樣本中提取一大段文本，然後生成所有的輸入-輸出對，將它們打亂，並生成大小相等的小批次。\n",
|
||||
"\n",
|
||||
"* **多層 LSTM**。嘗試使用 2 或 3 層的 LSTM 單元是有意義的。如我們在前一單元提到的，每一層 LSTM 都會從文本中提取某些模式。在字符級生成器的情況下，我們可以預期較低層的 LSTM 負責提取音節，而較高層則負責提取單詞及單詞組合。這可以通過向 LSTM 構造函數傳遞層數參數來簡單實現。\n",
|
||||
"\n",
|
||||
"* 你也可以嘗試使用 **GRU 單元**，看看哪種表現更好，還可以嘗試 **不同的隱藏層大小**。隱藏層過大可能導致過擬合（例如，網絡會學習到精確的文本），而過小則可能無法產生良好的結果。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 軟性文本生成與溫度\n",
|
||||
"\n",
|
||||
"在之前 `generate` 的定義中，我們總是選擇機率最高的字元作為生成文本中的下一個字元。這導致生成的文本經常在相同的字元序列之間不斷循環，例如以下例子：\n",
|
||||
"```\n",
|
||||
"today of the second the company and a second the company ...\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"然而，如果我們觀察下一個字元的機率分佈，可能會發現幾個最高機率之間的差異並不大，例如一個字元的機率是 0.2，另一個是 0.19，等等。例如，在尋找序列 *play* 的下一個字元時，下一個字元可能同樣有可能是空格，或者是 **e**（如單詞 *player* 中的情況）。\n",
|
||||
"\n",
|
||||
"這讓我們得出一個結論：選擇機率最高的字元並不總是「公平」的，因為選擇第二高的字元也可能生成有意義的文本。更明智的做法是從網絡輸出的機率分佈中**抽樣**字元。\n",
|
||||
"\n",
|
||||
"這種抽樣可以通過 `multinomial` 函數來實現，該函數實現了所謂的**多項分佈**。下面定義了一個實現這種**軟性**文本生成的函數：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"--- Temperature = 0.3\n",
|
||||
"Today and a company and complete an all the land the restrational the as a security and has provers the pay to and a report and the computer in the stand has filities and working the law the stations for a company and with the company and the final the first company and refight of the state and and workin\n",
|
||||
"\n",
|
||||
"--- Temperature = 0.8\n",
|
||||
"Today he oniis its first to Aus bomblaties the marmation a to manan boogot that pirate assaid a relaid their that goverfin the the Cappets Ecrotional Assonia Cition targets it annight the w scyments Blamity #39;s TVeer Diercheg Reserals fran envyuil that of ster said access what succers of Dour-provelith\n",
|
||||
"\n",
|
||||
"--- Temperature = 1.0\n",
|
||||
"Today holy they a 11 will meda a toket subsuaties, engins for Chanos, they's has stainger past to opening orital his thempting new Nattona was al innerforder advan-than #36;s night year his religuled talitatian what the but with Wednesday to Justment will wemen of Mark CCC Camp as Timed Nae wome a leaders\n",
|
||||
"\n",
|
||||
"--- Temperature = 1.3\n",
|
||||
"Today gpone 2.5 fech atcusion poor cocles toparsdorM.cht Line Pamage put 43 his calt lowed to the book, that has authh-the silia rruch ailing to'ory andhes beutirsimi- Aefffive heading offil an auf eacklets is charged evis, Gunymy oy) Mony has it after-sloythyor loveId out filme, the Natabl -Najuntaxiggs \n",
|
||||
"\n",
|
||||
"--- Temperature = 1.8\n",
|
||||
"Today plary, P.slan chly\\401 mardregationly #39;t 8.1Mide) closes ,filtcon alfly playin roven!\\grea.-QFBEP: Iss onfarchQ/itilia CCf Zivesigntwasta orce.-Peul-aw.uicrin of fuglinfsut aftaningwo, MIEX awayew Aice Woiduar Corvagiugge oppo esig ThusBratourid canthly-RyI.co lagitems\\eexciaishes.conBabntusmor I\n",
|
||||
"\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def generate_soft(net,size=100,start='today ',temperature=1.0):\n",
|
||||
" chars = list(start)\n",
|
||||
" out, s = net(enc(chars).view(1,-1).to(device))\n",
|
||||
" for i in range(size):\n",
|
||||
" #nc = torch.argmax(out[0][-1])\n",
|
||||
" out_dist = out[0][-1].div(temperature).exp()\n",
|
||||
" nc = torch.multinomial(out_dist,1)[0]\n",
|
||||
" chars.append(vocab.get_itos()[nc])\n",
|
||||
" out, s = net(nc.view(1,-1),s)\n",
|
||||
" return ''.join(chars)\n",
|
||||
" \n",
|
||||
"for i in [0.3,0.8,1.0,1.3,1.8]:\n",
|
||||
" print(f\"--- Temperature = {i}\\n{generate_soft(net,size=300,start='Today ',temperature=i)}\\n\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"我們引入了一個名為 **temperature** 的參數，用於指示我們應該多大程度地堅持最高概率。如果 temperature 為 1.0，我們進行公平的多項式抽樣；當 temperature 趨於無窮大時，所有概率變得相等，我們隨機選擇下一個字符。在下面的例子中，我們可以觀察到，當我們將 temperature 增加得過高時，文本變得毫無意義；而當 temperature 趨近於 0 時，文本則類似於「循環」的硬生成文本。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**免責聲明**： \n本文件已使用 AI 翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。雖然我們致力於提供準確的翻譯，但請注意，自動翻譯可能包含錯誤或不準確之處。原始文件的母語版本應被視為權威來源。對於關鍵信息，建議尋求專業人工翻譯。我們對因使用此翻譯而引起的任何誤解或錯誤解釋不承擔責任。\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.8.12 ('py38')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "7673cd150d96c74c6d6011460094efb4",
|
||||
"translation_date": "2025-08-28T12:03:32+00:00",
|
||||
"source_file": "lessons/5-NLP/17-GenerativeNetworks/GenerativePyTorch.ipynb",
|
||||
"language_code": "mo"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
|
|
@ -0,0 +1,495 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# 生成式網絡\n",
|
||||
"\n",
|
||||
"循環神經網絡（RNNs）及其門控單元變體，例如長短期記憶單元（LSTMs）和門控循環單元（GRUs），提供了一種語言建模的機制，也就是說，它們可以學習詞語的排列順序，並對序列中的下一個詞進行預測。這使得我們可以使用 RNNs 進行**生成任務**，例如普通文本生成、機器翻譯，甚至是圖像描述。\n",
|
||||
"\n",
|
||||
"在上一單元中討論的 RNN 架構中，每個 RNN 單元都會生成下一個隱藏狀態作為輸出。然而，我們也可以為每個循環單元添加另一個輸出，這樣就可以輸出一個**序列**（其長度與原始序列相等）。此外，我們還可以使用不在每一步接受輸入的 RNN 單元，而僅僅接受一些初始狀態向量，然後生成一系列的輸出。\n",
|
||||
"\n",
|
||||
"在這份筆記中，我們將專注於幫助我們生成文本的簡單生成模型。為了簡化，我們將構建**字元級網絡**，逐字生成文本。在訓練過程中，我們需要採用一些文本語料庫，並將其拆分為字元序列。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import tensorflow as tf\n",
|
||||
"from tensorflow import keras\n",
|
||||
"import tensorflow_datasets as tfds\n",
|
||||
"import numpy as np\n",
|
||||
"\n",
|
||||
"ds_train, ds_test = tfds.load('ag_news_subset').values()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 建立字元詞彙表\n",
|
||||
"\n",
|
||||
"為了建立字元級別的生成網絡，我們需要將文本拆分為單個字元，而不是單詞。我們之前使用的 `TextVectorization` 層無法做到這一點，因此我們有以下兩個選擇：\n",
|
||||
"\n",
|
||||
"* 手動載入文本並自行進行分詞，如[這個官方 Keras 範例](https://keras.io/examples/generative/lstm_character_level_text_generation/)中所示\n",
|
||||
"* 使用 `Tokenizer` 類進行字元級別的分詞。\n",
|
||||
"\n",
|
||||
"我們將選擇第二種方法。`Tokenizer` 也可以用於將文本分詞為單詞，因此可以很輕鬆地從字元級分詞切換到單詞級分詞。\n",
|
||||
"\n",
|
||||
"要進行字元級分詞，我們需要傳遞參數 `char_level=True`：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def extract_text(x):\n",
|
||||
" return x['title']+' '+x['description']\n",
|
||||
"\n",
|
||||
"def tupelize(x):\n",
|
||||
" return (extract_text(x),x['label'])\n",
|
||||
"\n",
|
||||
"tokenizer = keras.preprocessing.text.Tokenizer(char_level=True,lower=False)\n",
|
||||
"tokenizer.fit_on_texts([x['title'].numpy().decode('utf-8') for x in ds_train])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"我們也希望使用一個特殊的標記來表示**序列結束**，我們將其稱為 `<eos>`。讓我們手動將其添加到詞彙表中：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"eos_token = len(tokenizer.word_index)+1\n",
|
||||
"tokenizer.word_index['<eos>'] = eos_token\n",
|
||||
"\n",
|
||||
"vocab_size = eos_token + 1"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[[48, 2, 10, 10, 5, 44, 1, 25, 5, 8, 10, 13, 78]]"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"tokenizer.texts_to_sequences(['Hello, world!'])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 訓練生成式 RNN 來生成標題\n",
|
||||
"\n",
|
||||
"我們將以以下方式訓練 RNN 來生成新聞標題。在每一步中，我們會取一個標題，將其輸入到 RNN 中，並對於每個輸入的字元，要求網路生成下一個輸出的字元：\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"對於序列中的最後一個字元，我們會要求網路生成 `<eos>` 標記。\n",
|
||||
"\n",
|
||||
"這裡使用的生成式 RNN 與其他的主要區別在於，我們會從 RNN 的每一步輸出中取結果，而不僅僅是從最後一個單元格中取結果。這可以通過為 RNN 單元指定 `return_sequences` 參數來實現。\n",
|
||||
"\n",
|
||||
"因此，在訓練過程中，網路的輸入將是一個特定長度的編碼字元序列，而輸出則是一個相同長度的序列，但向後偏移一個元素並以 `<eos>` 結尾。小批次（minibatch）將由多個這樣的序列組成，我們需要使用**填充（padding）**來對齊所有序列。\n",
|
||||
"\n",
|
||||
"接下來，我們來建立一些函數，用於轉換數據集。由於我們希望在小批次層級進行序列填充，我們會先通過調用 `.batch()` 將數據集分批，然後使用 `map` 來進行轉換。因此，轉換函數將以整個小批次作為參數：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def title_batch(x):\n",
|
||||
" x = [t.numpy().decode('utf-8') for t in x]\n",
|
||||
" z = tokenizer.texts_to_sequences(x)\n",
|
||||
" z = tf.keras.preprocessing.sequence.pad_sequences(z)\n",
|
||||
" return tf.one_hot(z,vocab_size), tf.one_hot(tf.concat([z[:,1:],tf.constant(eos_token,shape=(len(z),1))],axis=1),vocab_size)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"以下是我們在這裡執行的一些重要步驟：\n",
|
||||
"* 我們首先從字串張量中提取實際文本\n",
|
||||
"* `text_to_sequences` 將字串列表轉換為整數張量列表\n",
|
||||
"* `pad_sequences` 將這些張量填充到它們的最大長度\n",
|
||||
"* 最後，我們對所有字符進行獨熱編碼，並執行移位和 `<eos>` 附加操作。我們很快就會了解為什麼需要使用獨熱編碼的字符\n",
|
||||
"\n",
|
||||
"然而，這個函數是 **Pythonic** 的，也就是說，它無法自動轉換為 Tensorflow 的計算圖。如果我們直接在 `Dataset.map` 函數中使用這個函數，會出現錯誤。我們需要使用 `py_function` 包裝器來封裝這個 Pythonic 調用：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def title_batch_fn(x):\n",
|
||||
" x = x['title']\n",
|
||||
" a,b = tf.py_function(title_batch,inp=[x],Tout=(tf.float32,tf.float32))\n",
|
||||
" return a,b"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **注意**：區分 Pythonic 和 Tensorflow 的轉換函數可能看起來有點複雜，你可能會疑惑為什麼我們不在將資料傳遞給 `fit` 之前，使用標準的 Python 函數來轉換資料。雖然這確實是可行的，但使用 `Dataset.map` 有一個巨大的優勢，因為資料轉換管道是使用 Tensorflow 的計算圖執行的，這可以利用 GPU 的計算能力，並且減少在 CPU 和 GPU 之間傳遞資料的需求。\n",
|
||||
"\n",
|
||||
"現在我們可以建立生成器網路並開始訓練。它可以基於我們在上一單元中討論過的任何循環單元（簡單的、LSTM 或 GRU）。在我們的例子中，我們將使用 LSTM。\n",
|
||||
"\n",
|
||||
"由於網路以字元作為輸入，且詞彙表的大小相對較小，我們不需要嵌入層，直接將 one-hot 編碼的輸入傳遞給 LSTM 單元即可。輸出層將是一個 `Dense` 分類器，它會將 LSTM 的輸出轉換為 one-hot 編碼的標記編號。\n",
|
||||
"\n",
|
||||
"此外，因為我們處理的是可變長度的序列，我們可以使用 `Masking` 層來建立一個遮罩，忽略字串中填充的部分。這並不是絕對必要的，因為我們對超過 `<eos>` 標記的部分並不特別感興趣，但我們會使用它來獲得一些使用這類型層的經驗。`input_shape` 將是 `(None, vocab_size)`，其中 `None` 表示可變長度的序列，而輸出形狀也是 `(None, vocab_size)`，正如你可以從 `summary` 中看到的那樣。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"sequential\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
"Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
"masking (Masking) (None, None, 84) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"lstm (LSTM) (None, None, 128) 109056 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense (Dense) (None, None, 84) 10836 \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 119,892\n",
|
||||
"Trainable params: 119,892\n",
|
||||
"Non-trainable params: 0\n",
|
||||
"_________________________________________________________________\n",
|
||||
"15000/15000 [==============================] - 229s 15ms/step - loss: 1.5385\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7fa40c1245e0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model = keras.models.Sequential([\n",
|
||||
" keras.layers.Masking(input_shape=(None,vocab_size)),\n",
|
||||
" keras.layers.LSTM(128,return_sequences=True),\n",
|
||||
" keras.layers.Dense(vocab_size,activation='softmax')\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"model.summary()\n",
|
||||
"model.compile(loss='categorical_crossentropy')\n",
|
||||
"\n",
|
||||
"model.fit(ds_train.batch(8).map(title_batch_fn))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 生成輸出\n",
|
||||
"\n",
|
||||
"現在我們已經訓練了模型，接下來我們想要使用它來生成一些輸出。首先，我們需要一種方法來解碼由一系列標記數字表示的文本。為此，我們可以使用 `tokenizer.sequences_to_texts` 函數；然而，這個方法在字元級別的標記化中效果並不好。因此，我們將從 tokenizer 中取得一個標記的字典（稱為 `word_index`），建立一個反向映射，並撰寫我們自己的解碼函數：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"reverse_map = {val:key for key, val in tokenizer.word_index.items()}\n",
|
||||
"\n",
|
||||
"def decode(x):\n",
|
||||
" return ''.join([reverse_map[t] for t in x])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"現在，我們開始進行生成。我們將以某個字串 `start` 作為起點，將其編碼成一個序列 `inp`，然後在每一步中，我們會呼叫網路來推斷下一個字元。\n",
|
||||
"\n",
|
||||
"網路的輸出 `out` 是一個包含 `vocab_size` 元素的向量，代表每個標記的概率。我們可以使用 `argmax` 找出最可能的標記編號。接著，我們將這個字元附加到已生成的標記列表中，並繼續進行生成。這個生成一個字元的過程會重複執行 `size` 次，以生成所需的字元數量。如果在過程中遇到 `eos_token`，我們會提前終止生成。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"'Today #39;s lead to strike for the strike for the strike for the strike (AFP)'"
|
||||
]
|
||||
},
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def generate(model,size=100,start='Today '):\n",
|
||||
" inp = tokenizer.texts_to_sequences([start])[0]\n",
|
||||
" chars = inp\n",
|
||||
" for i in range(size):\n",
|
||||
" out = model(tf.expand_dims(tf.one_hot(inp,vocab_size),0))[0][-1]\n",
|
||||
" nc = tf.argmax(out)\n",
|
||||
" if nc==eos_token:\n",
|
||||
" break\n",
|
||||
" chars.append(nc.numpy())\n",
|
||||
" inp = inp+[nc]\n",
|
||||
" return decode(chars)\n",
|
||||
" \n",
|
||||
"generate(model)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 在訓練期間抽樣輸出\n",
|
||||
"\n",
|
||||
"由於我們沒有任何像 *準確率* 這樣的有用指標，我們唯一能看到模型是否有所改進的方法就是在訓練期間通過 **抽樣** 生成的字串來進行檢查。為了實現這一點，我們將使用 **回調函數**，也就是可以傳遞給 `fit` 函數並在訓練期間定期被調用的函數。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch 1/3\n",
|
||||
"15000/15000 [==============================] - 226s 15ms/step - loss: 1.2703\n",
|
||||
"Today #39;s a lead in the company for the strike\n",
|
||||
"Epoch 2/3\n",
|
||||
"15000/15000 [==============================] - 227s 15ms/step - loss: 1.2057\n",
|
||||
"Today #39;s the Market Service on Security Start (AP)\n",
|
||||
"Epoch 3/3\n",
|
||||
"15000/15000 [==============================] - 226s 15ms/step - loss: 1.1752\n",
|
||||
"Today #39;s a line on the strike to start for the start\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7fa40c74e3d0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"sampling_callback = keras.callbacks.LambdaCallback(\n",
|
||||
" on_epoch_end = lambda batch, logs: print(generate(model))\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"model.fit(ds_train.batch(8).map(title_batch_fn),callbacks=[sampling_callback],epochs=3)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"這個範例已經生成了一些相當不錯的文本，但仍有多種方式可以進一步改進：\n",
|
||||
"\n",
|
||||
"* **更多文本**。我們僅使用了標題來完成任務，但您可能希望嘗試使用完整的文本。請記住，RNN在處理長序列方面表現不佳，因此可以將文本拆分成較短的句子，或者始終以固定的序列長度進行訓練，例如預定義的值 `num_chars`（例如 256）。您可以嘗試將上述範例改造成這樣的架構，並參考 [官方 Keras 教學](https://keras.io/examples/generative/lstm_character_level_text_generation/) 作為靈感。\n",
|
||||
"\n",
|
||||
"* **多層 LSTM**。嘗試使用 2 或 3 層的 LSTM 單元是有意義的。如我們在前一單元提到的，每層 LSTM 都會從文本中提取特定的模式，而在字元級生成器的情況下，我們可以預期較低層的 LSTM 負責提取音節，而較高層則負責提取單詞及單詞組合。這可以通過向 LSTM 構造函數傳遞層數參數來簡單實現。\n",
|
||||
"\n",
|
||||
"* 您也可以嘗試使用 **GRU 單元**，看看哪種表現更好，並嘗試 **不同的隱藏層大小**。隱藏層過大可能導致過度擬合（例如，網絡會學習精確的文本），而過小的大小可能無法生成良好的結果。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 軟性文本生成與溫度\n",
|
||||
"\n",
|
||||
"在之前 `generate` 的定義中，我們總是選擇機率最高的字元作為生成文本的下一個字元。這導致生成的文本經常在相同的字元序列之間不斷循環，例如以下例子：\n",
|
||||
"```\n",
|
||||
"today of the second the company and a second the company ...\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"然而，如果我們觀察下一個字元的機率分佈，可能會發現幾個最高機率之間的差距並不大，例如一個字元的機率是 0.2，另一個是 0.19，等等。例如，在尋找序列 *play* 的下一個字元時，下一個字元可能同樣有可能是空格，或者是 **e**（如單字 *player* 中的情況）。\n",
|
||||
"\n",
|
||||
"這讓我們得出一個結論：選擇機率最高的字元並不總是「公平」的，因為選擇第二高的字元也可能生成有意義的文本。更明智的做法是根據網路輸出的機率分佈來**抽樣**字元。\n",
|
||||
"\n",
|
||||
"這種抽樣可以使用 `np.multinomial` 函數來完成，該函數實現了所謂的**多項分佈**。以下是一個實現這種**軟性**文本生成的函數：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 33,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\n",
|
||||
"--- Temperature = 0.3\n",
|
||||
"Today #39;s strike #39; to start at the store return\n",
|
||||
"On Sunday PO to Be Data Profit Up (Reuters)\n",
|
||||
"Moscow, SP wins straight to the Microsoft #39;s control of the space start\n",
|
||||
"President olding of the blast start for the strike to pay <b>...</b>\n",
|
||||
"Little red riding hood ficed to the spam countered in European <b>...</b>\n",
|
||||
"\n",
|
||||
"--- Temperature = 0.8\n",
|
||||
"Today countie strikes ryder missile faces food market blut\n",
|
||||
"On Sunday collores lose-toppy of sale of Bullment in <b>...</b>\n",
|
||||
"Moscow, IBM Diffeiting in Afghan Software Hotels (Reuters)\n",
|
||||
"President Ol Luster for Profit Peaced Raised (AP)\n",
|
||||
"Little red riding hood dace on depart talks #39; bank up\n",
|
||||
"\n",
|
||||
"--- Temperature = 1.0\n",
|
||||
"Today wits House buiting debate fixes #39; supervice stake again\n",
|
||||
"On Sunday arling digital poaching In for level\n",
|
||||
"Moscow, DS Up 7, Top Proble Protest Caprey Mamarian Strike\n",
|
||||
"President teps help of roubler stepted lessabul-Dhalitics (AFP)\n",
|
||||
"Little red riding hood signs on cash in Carter-youb\n",
|
||||
"\n",
|
||||
"--- Temperature = 1.3\n",
|
||||
"Today wits flawer ro, pSIA figat's co DroftwavesIs Talo up\n",
|
||||
"On Sunday hround elitwing wint EU Powerburlinetien\n",
|
||||
"Moscow, Bazz #39;s sentries olymen winnelds' next for Olympite Huc?\n",
|
||||
"President lost securitys from power Elections in Smiltrials\n",
|
||||
"Little red riding hood vides profit, exponituity, profitmainalist-at said listers\n",
|
||||
"\n",
|
||||
"--- Temperature = 1.8\n",
|
||||
"Today #39;It: He deat: N.KA Asside\n",
|
||||
"On Sunday i arry Par aldeup patient Wo stele1\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"ename": "KeyError",
|
||||
"evalue": "0",
|
||||
"output_type": "error",
|
||||
"traceback": [
|
||||
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
|
||||
"\u001b[0;31mKeyError\u001b[0m Traceback (most recent call last)",
|
||||
"\u001b[0;32m<ipython-input-33-db32367a0feb>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m 18\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"\\n--- Temperature = {i}\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 19\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mj\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 20\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgenerate_soft\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0msize\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m300\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mstart\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mwords\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mj\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mtemperature\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
|
||||
"\u001b[0;32m<ipython-input-33-db32367a0feb>\u001b[0m in \u001b[0;36mgenerate_soft\u001b[0;34m(model, size, start, temperature)\u001b[0m\n\u001b[1;32m 11\u001b[0m \u001b[0mchars\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 12\u001b[0m \u001b[0minp\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0minp\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mnc\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 13\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mdecode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mchars\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 14\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 15\u001b[0m \u001b[0mwords\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m'Today '\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'On Sunday '\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'Moscow, '\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'President '\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'Little red riding hood '\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
|
||||
"\u001b[0;32m<ipython-input-10-3f5fa6130b1d>\u001b[0m in \u001b[0;36mdecode\u001b[0;34m(x)\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mdecode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0;34m''\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mreverse_map\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mt\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
|
||||
"\u001b[0;32m<ipython-input-10-3f5fa6130b1d>\u001b[0m in \u001b[0;36m<listcomp>\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mdecode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0;34m''\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mreverse_map\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mt\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
|
||||
"\u001b[0;31mKeyError\u001b[0m: 0"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def generate_soft(model,size=100,start='Today ',temperature=1.0):\n",
|
||||
" inp = tokenizer.texts_to_sequences([start])[0]\n",
|
||||
" chars = inp\n",
|
||||
" for i in range(size):\n",
|
||||
" out = model(tf.expand_dims(tf.one_hot(inp,vocab_size),0))[0][-1]\n",
|
||||
" probs = tf.exp(tf.math.log(out)/temperature).numpy().astype(np.float64)\n",
|
||||
" probs = probs/np.sum(probs)\n",
|
||||
" nc = np.argmax(np.random.multinomial(1,probs,1))\n",
|
||||
" if nc==eos_token:\n",
|
||||
" break\n",
|
||||
" chars.append(nc)\n",
|
||||
" inp = inp+[nc]\n",
|
||||
" return decode(chars)\n",
|
||||
"\n",
|
||||
"words = ['Today ','On Sunday ','Moscow, ','President ','Little red riding hood ']\n",
|
||||
" \n",
|
||||
"for i in [0.3,0.8,1.0,1.3,1.8]:\n",
|
||||
" print(f\"\\n--- Temperature = {i}\")\n",
|
||||
" for j in range(5):\n",
|
||||
" print(generate_soft(model,size=300,start=words[j],temperature=i))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"我們引入了一個名為 **溫度** 的參數，用於指示我們應該多麼嚴格地遵循最高概率。如果溫度為 1.0，我們進行公平的多項式抽樣，而當溫度趨於無窮大時——所有概率變得相等，我們隨機選擇下一個字符。在下面的例子中，我們可以觀察到當我們將溫度增加得太高時，文本變得毫無意義，而當溫度接近 0 時，它則類似於「循環」的硬生成文本。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**免責聲明**： \n本文件使用 AI 翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。我們致力於提供準確的翻譯，但請注意，自動翻譯可能包含錯誤或不準確之處。應以原始語言的文件作為權威來源。對於關鍵資訊，建議尋求專業人工翻譯。我們對於因使用本翻譯而引起的任何誤解或錯誤解讀概不負責。\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.8.12 ('py38')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "9fbb7d5fda708537649f71f5f646fcde",
|
||||
"translation_date": "2025-08-28T12:01:00+00:00",
|
||||
"source_file": "lessons/5-NLP/17-GenerativeNetworks/GenerativeTF.ipynb",
|
||||
"language_code": "mo"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
|
|
@ -0,0 +1,353 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# 注意力機制與Transformer模型\n",
|
||||
"\n",
|
||||
"循環神經網路（RNN）的一個主要缺點是序列中的所有詞對結果的影響是相同的。這導致標準的LSTM編碼器-解碼器模型在處理序列到序列任務（如命名實體識別和機器翻譯）時表現不佳。實際上，輸入序列中的某些特定詞往往對輸出序列的影響更大。\n",
|
||||
"\n",
|
||||
"考慮一個序列到序列的模型，例如機器翻譯。這種模型由兩個循環神經網路實現，其中一個網路（**編碼器**）將輸入序列壓縮為隱藏狀態，另一個網路（**解碼器**）將該隱藏狀態展開為翻譯結果。這種方法的問題在於，網路的最終狀態很難記住句子的開頭部分，從而導致模型在處理長句子時的質量下降。\n",
|
||||
"\n",
|
||||
"**注意力機制**提供了一種方法，能夠對每個輸入向量對RNN每個輸出預測的上下文影響進行加權。其實現方式是通過在輸入RNN的中間狀態和輸出RNN之間創建捷徑。這樣，在生成輸出符號 $y_t$ 時，我們會考慮所有輸入隱藏狀態 $h_i$，並賦予不同的權重係數 $\\alpha_{t,i}$。\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"*來自 [Bahdanau et al., 2015](https://arxiv.org/pdf/1409.0473.pdf) 的加性注意力機制編碼器-解碼器模型，圖片引用自[這篇部落格文章](https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html)*\n",
|
||||
"\n",
|
||||
"注意力矩陣 $\\{\\alpha_{i,j}\\}$ 表示某些輸入詞在生成輸出序列中特定詞時所起的作用程度。以下是這樣一個矩陣的示例：\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"*圖片取自 [Bahdanau et al., 2015](https://arxiv.org/pdf/1409.0473.pdf)（圖3）*\n",
|
||||
"\n",
|
||||
"注意力機制是當前或接近當前自然語言處理技術水平的關鍵因素。然而，添加注意力機制會大幅增加模型參數的數量，這導致了RNN的擴展問題。RNN擴展的一個主要限制是其循環特性使得訓練過程難以批量化和並行化。在RNN中，序列的每個元素都需要按順序處理，這意味著它無法輕易並行化。\n",
|
||||
"\n",
|
||||
"注意力機制的採用結合了這一限制，促成了如今我們所熟知和使用的最先進的Transformer模型的誕生，從BERT到OpenGPT3。\n",
|
||||
"\n",
|
||||
"## Transformer模型\n",
|
||||
"\n",
|
||||
"與將每次預測的上下文傳遞到下一步評估不同，**Transformer模型**使用**位置編碼**和注意力機制來捕捉給定輸入在提供的文本窗口內的上下文。下圖展示了如何通過位置編碼和注意力機制在給定窗口內捕捉上下文。\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"由於每個輸入位置可以獨立映射到每個輸出位置，Transformer模型比RNN更容易並行化，這使得構建更大且更具表達力的語言模型成為可能。每個注意力頭可以用來學習詞與詞之間的不同關係，從而改進下游的自然語言處理任務。\n",
|
||||
"\n",
|
||||
"**BERT**（Bidirectional Encoder Representations from Transformers，雙向編碼器表示）是一個非常大的多層Transformer網路，*BERT-base*有12層，*BERT-large*有24層。該模型首先在大規模文本數據（維基百科+書籍）上進行無監督訓練（預測句子中的被遮蔽詞）。在預訓練過程中，模型吸收了大量的語言理解能力，這些能力可以通過微調其他數據集來加以利用。這個過程被稱為**遷移學習**。\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Transformer架構有許多變體，包括BERT、DistilBERT、BigBird、OpenGPT3等，這些模型都可以進行微調。[HuggingFace套件](https://github.com/huggingface/) 提供了用PyTorch訓練這些架構的資源庫。\n",
|
||||
"\n",
|
||||
"## 使用BERT進行文本分類\n",
|
||||
"\n",
|
||||
"讓我們看看如何使用預訓練的BERT模型來解決我們的傳統任務：序列分類。我們將對原始的AG News數據集進行分類。\n",
|
||||
"\n",
|
||||
"首先，讓我們加載HuggingFace庫和我們的數據集：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Loading dataset...\n",
|
||||
"Building vocab...\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"import torchtext\n",
|
||||
"from torchnlp import *\n",
|
||||
"import transformers\n",
|
||||
"train_dataset, test_dataset, classes, vocab = load_dataset()\n",
|
||||
"vocab_len = len(vocab)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"因為我們將使用預訓練的 BERT 模型，所以需要使用特定的分詞器。首先，我們會載入與預訓練 BERT 模型相關聯的分詞器。\n",
|
||||
"\n",
|
||||
"HuggingFace 庫包含一個預訓練模型的存儲庫，您只需在 `from_pretrained` 函數中指定模型名稱作為參數即可使用。所有模型所需的二進制文件都會自動下載。\n",
|
||||
"\n",
|
||||
"然而，有時您可能需要載入自己的模型，在這種情況下，您可以指定包含所有相關文件的目錄，包括分詞器的參數、模型參數的 `config.json` 文件、二進制權重等。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# To load the model from Internet repository using model name. \n",
|
||||
"# Use this if you are running from your own copy of the notebooks\n",
|
||||
"bert_model = 'bert-base-uncased' \n",
|
||||
"\n",
|
||||
"# To load the model from the directory on disk. Use this for Microsoft Learn module, because we have\n",
|
||||
"# prepared all required files for you.\n",
|
||||
"bert_model = './bert'\n",
|
||||
"\n",
|
||||
"tokenizer = transformers.BertTokenizer.from_pretrained(bert_model)\n",
|
||||
"\n",
|
||||
"MAX_SEQ_LEN = 128\n",
|
||||
"PAD_INDEX = tokenizer.convert_tokens_to_ids(tokenizer.pad_token)\n",
|
||||
"UNK_INDEX = tokenizer.convert_tokens_to_ids(tokenizer.unk_token)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"`tokenizer` 物件包含 `encode` 函數，可直接用於編碼文本：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[101, 1052, 22123, 2953, 2818, 2003, 1037, 2307, 7705, 2005, 17953, 2361, 102]"
|
||||
]
|
||||
},
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"tokenizer.encode('PyTorch is a great framework for NLP')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"然後，讓我們建立在訓練期間用於訪問數據的迭代器。由於 BERT 使用其自己的編碼函數，我們需要定義一個類似於之前定義的 `padify` 的填充函數：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def pad_bert(b):\n",
|
||||
" # b is the list of tuples of length batch_size\n",
|
||||
" # - first element of a tuple = label, \n",
|
||||
" # - second = feature (text sequence)\n",
|
||||
" # build vectorized sequence\n",
|
||||
" v = [tokenizer.encode(x[1]) for x in b]\n",
|
||||
" # compute max length of a sequence in this minibatch\n",
|
||||
" l = max(map(len,v))\n",
|
||||
" return ( # tuple of two tensors - labels and features\n",
|
||||
" torch.LongTensor([t[0] for t in b]),\n",
|
||||
" torch.stack([torch.nn.functional.pad(torch.tensor(t),(0,l-len(t)),mode='constant',value=0) for t in v])\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=8, collate_fn=pad_bert, shuffle=True)\n",
|
||||
"test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=8, collate_fn=pad_bert)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"在我們的案例中，我們將使用名為 `bert-base-uncased` 的預訓練 BERT 模型。讓我們使用 `BertForSequenceClassfication` 套件來加載模型。這確保了我們的模型已經具備分類所需的架構，包括最終的分類器。您會看到一條警告消息，指出最終分類器的權重尚未初始化，並且模型需要進行預訓練——這完全沒問題，因為這正是我們即將進行的操作！\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Some weights of the model checkpoint at ./bert were not used when initializing BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']\n",
|
||||
"- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
|
||||
"- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
|
||||
"Some weights of BertForSequenceClassification were not initialized from the model checkpoint at ./bert and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
|
||||
"You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model = transformers.BertForSequenceClassification.from_pretrained(bert_model,num_labels=4).to(device)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"現在我們準備開始訓練了！由於 BERT 已經過預訓練，我們希望以較小的學習率開始，這樣可以避免破壞初始權重。\n",
|
||||
"\n",
|
||||
"所有的繁重工作都由 `BertForSequenceClassification` 模型完成。當我們將訓練數據傳入模型時，它會返回該批次的損失值和網絡輸出。我們使用損失值進行參數優化（`loss.backward()` 負責反向傳播），而使用 `out` 計算訓練準確率，方法是將獲得的標籤 `labs`（通過 `argmax` 計算）與預期的 `labels` 進行比較。\n",
|
||||
"\n",
|
||||
"為了控制訓練過程，我們會在多次迭代中累積損失值和準確率，並在每 `report_freq` 個訓練週期後打印出來。\n",
|
||||
"\n",
|
||||
"這次訓練可能會花費相當長的時間，因此我們會限制迭代次數。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Loss = 1.1254194641113282, Accuracy = 0.585\n",
|
||||
"Loss = 0.6194715118408203, Accuracy = 0.83\n",
|
||||
"Loss = 0.46665248870849607, Accuracy = 0.8475\n",
|
||||
"Loss = 0.4309701919555664, Accuracy = 0.8575\n",
|
||||
"Loss = 0.35427074432373046, Accuracy = 0.8825\n",
|
||||
"Loss = 0.3306886291503906, Accuracy = 0.8975\n",
|
||||
"Loss = 0.30340143203735354, Accuracy = 0.8975\n",
|
||||
"Loss = 0.26139299392700194, Accuracy = 0.915\n",
|
||||
"Loss = 0.26708646774291994, Accuracy = 0.9225\n",
|
||||
"Loss = 0.3667240524291992, Accuracy = 0.8675\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"optimizer = torch.optim.Adam(model.parameters(), lr=2e-5)\n",
|
||||
"\n",
|
||||
"report_freq = 50\n",
|
||||
"iterations = 500 # make this larger to train for longer time!\n",
|
||||
"\n",
|
||||
"model.train()\n",
|
||||
"\n",
|
||||
"i,c = 0,0\n",
|
||||
"acc_loss = 0\n",
|
||||
"acc_acc = 0\n",
|
||||
"\n",
|
||||
"for labels,texts in train_loader:\n",
|
||||
" labels = labels.to(device)-1 # get labels in the range 0-3 \n",
|
||||
" texts = texts.to(device)\n",
|
||||
" loss, out = model(texts, labels=labels)[:2]\n",
|
||||
" labs = out.argmax(dim=1)\n",
|
||||
" acc = torch.mean((labs==labels).type(torch.float32))\n",
|
||||
" optimizer.zero_grad()\n",
|
||||
" loss.backward()\n",
|
||||
" optimizer.step()\n",
|
||||
" acc_loss += loss\n",
|
||||
" acc_acc += acc\n",
|
||||
" i+=1\n",
|
||||
" c+=1\n",
|
||||
" if i%report_freq==0:\n",
|
||||
" print(f\"Loss = {acc_loss.item()/c}, Accuracy = {acc_acc.item()/c}\")\n",
|
||||
" c = 0\n",
|
||||
" acc_loss = 0\n",
|
||||
" acc_acc = 0\n",
|
||||
" iterations-=1\n",
|
||||
" if not iterations:\n",
|
||||
" break"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"你可以看到（尤其是當你增加迭代次數並等待足夠長的時間），BERT 分類能夠提供相當不錯的準確率！這是因為 BERT 已經對語言的結構有相當好的理解，我們只需要微調最終的分類器即可。然而，由於 BERT 是一個大型模型，整個訓練過程需要很長的時間，並且需要強大的計算能力！（GPU，最好是多個）。\n",
|
||||
"\n",
|
||||
"> **Note:** 在我們的範例中，我們使用的是最小的預訓練 BERT 模型之一。還有更大的模型可能會產生更好的結果。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 評估模型表現\n",
|
||||
"\n",
|
||||
"現在我們可以在測試數據集上評估模型的表現。評估循環與訓練循環非常相似，但我們不能忘記通過呼叫 `model.eval()` 將模型切換到評估模式。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Final accuracy: 0.9047029702970297\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.eval()\n",
|
||||
"iterations = 100\n",
|
||||
"acc = 0\n",
|
||||
"i = 0\n",
|
||||
"for labels,texts in test_loader:\n",
|
||||
" labels = labels.to(device)-1 \n",
|
||||
" texts = texts.to(device)\n",
|
||||
" _, out = model(texts, labels=labels)[:2]\n",
|
||||
" labs = out.argmax(dim=1)\n",
|
||||
" acc += torch.mean((labs==labels).type(torch.float32))\n",
|
||||
" i+=1\n",
|
||||
" if i>iterations: break\n",
|
||||
" \n",
|
||||
"print(f\"Final accuracy: {acc.item()/i}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 重點\n",
|
||||
"\n",
|
||||
"在本單元中，我們已經看到如何輕鬆地從 **transformers** 庫中取用預訓練的語言模型，並將其調整為我們的文本分類任務。同樣地，BERT 模型也可以用於實體抽取、問題回答以及其他 NLP 任務。\n",
|
||||
"\n",
|
||||
"Transformer 模型代表了 NLP 領域的最新技術，在大多數情況下，當你開始實現自定義 NLP 解決方案時，它應該是你首先嘗試的選擇。然而，如果你希望構建更高級的神經模型，理解本模組中討論的循環神經網絡的基本原理是非常重要的。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**免責聲明**： \n本文件已使用 AI 翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。儘管我們努力確保翻譯的準確性，但請注意，自動翻譯可能包含錯誤或不準確之處。原始文件的母語版本應被視為權威來源。對於關鍵資訊，建議使用專業人工翻譯。我們對因使用此翻譯而引起的任何誤解或錯誤解釋不承擔責任。\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "py37_pytorch",
|
||||
"language": "python",
|
||||
"name": "conda-env-py37_pytorch-py"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.7.7"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "753865967678a92dbce7d7efbd36d980",
|
||||
"translation_date": "2025-08-28T12:09:50+00:00",
|
||||
"source_file": "lessons/5-NLP/18-Transformers/TransformersPyTorch.ipynb",
|
||||
"language_code": "mo"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
|
|
@ -0,0 +1,819 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# 注意力機制與Transformer模型\n",
|
||||
"\n",
|
||||
"循環神經網絡的一個主要缺點是序列中的所有詞語對結果的影響相同。這導致標準LSTM編碼器-解碼器模型在序列到序列任務（例如命名實體識別和機器翻譯）中的表現不佳。實際上，輸入序列中的某些特定詞語通常比其他詞語對序列輸出有更大的影響。\n",
|
||||
"\n",
|
||||
"考慮一個序列到序列模型，例如機器翻譯。它由兩個循環神經網絡實現，其中一個網絡（**編碼器**）將輸入序列壓縮到隱藏狀態中，另一個網絡（**解碼器**）將隱藏狀態展開為翻譯結果。這種方法的問題在於，網絡的最終狀態很難記住句子的開頭部分，因此在處理長句子時模型的質量會下降。\n",
|
||||
"\n",
|
||||
"**注意力機制**提供了一種方法，能夠對每個輸入向量在RNN的每個輸出預測中的上下文影響進行加權。其實現方式是通過在輸入RNN的中間狀態和輸出RNN之間創建捷徑。在生成輸出符號$y_t$時，我們會考慮所有輸入隱藏狀態$h_i$，並使用不同的權重係數$\\alpha_{t,i}$。\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"*[Bahdanau等人，2015](https://arxiv.org/pdf/1409.0473.pdf)中的加性注意力機制編碼器-解碼器模型，圖片引用自[這篇博客文章](https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html)*\n",
|
||||
"\n",
|
||||
"注意力矩陣$\\{\\alpha_{i,j}\\}$表示某些輸入詞語在生成輸出序列中的某個詞語時所起的作用程度。以下是這樣一個矩陣的示例：\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"*圖片取自[Bahdanau等人，2015](https://arxiv.org/pdf/1409.0473.pdf)（圖3）*\n",
|
||||
"\n",
|
||||
"注意力機制是目前或接近目前自然語言處理領域的最先進技術的核心。添加注意力機制會大幅增加模型參數的數量，這導致了RNN的擴展問題。RNN擴展的一個關鍵限制是模型的循環特性使得批量處理和訓練並行化變得困難。在RNN中，序列的每個元素都需要按順序處理，這意味著它無法輕易並行化。\n",
|
||||
"\n",
|
||||
"注意力機制的採用結合了這一限制，促成了如今我們所熟知和使用的最先進Transformer模型的誕生，從BERT到OpenGPT3。\n",
|
||||
"\n",
|
||||
"## Transformer模型\n",
|
||||
"\n",
|
||||
"與將每次預測的上下文傳遞到下一個評估步驟不同，**Transformer模型**使用**位置編碼**和**注意力**來捕捉給定輸入在提供的文本窗口中的上下文。下圖展示了位置編碼與注意力如何在給定窗口中捕捉上下文。\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"由於每個輸入位置可以獨立映射到每個輸出位置，Transformer模型比RNN更容易並行化，這使得能夠構建更大、更具表達力的語言模型。每個注意力頭可以用來學習詞語之間的不同關係，從而改善下游的自然語言處理任務。\n",
|
||||
"\n",
|
||||
"## 構建簡單的Transformer模型\n",
|
||||
"\n",
|
||||
"Keras中並不包含內建的Transformer層，但我們可以自己構建。與之前一樣，我們將專注於AG News數據集的文本分類，但值得一提的是，Transformer模型在更困難的自然語言處理任務中表現最佳。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import tensorflow as tf\n",
|
||||
"from tensorflow import keras\n",
|
||||
"import tensorflow_datasets as tfds\n",
|
||||
"import numpy as np\n",
|
||||
"\n",
|
||||
"ds_train, ds_test = tfds.load('ag_news_subset').values()\n",
|
||||
"\n",
|
||||
"def extract_text(x):\n",
|
||||
" return x['title']+' '+x['description']\n",
|
||||
"\n",
|
||||
"def tupelize(x):\n",
|
||||
" return (extract_text(x),x['label'])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"新的層在 Keras 中應該繼承 `Layer` 類別，並實現 `call` 方法。我們先從 **位置嵌入** 層開始。我們將使用[官方 Keras 文件中的一些代碼](https://keras.io/examples/nlp/text_classification_with_transformer/)。我們假設所有輸入序列都填充到長度 `maxlen`。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class TokenAndPositionEmbedding(keras.layers.Layer):\n",
|
||||
" def __init__(self, maxlen, vocab_size, embed_dim):\n",
|
||||
" super(TokenAndPositionEmbedding, self).__init__()\n",
|
||||
" self.token_emb = keras.layers.Embedding(input_dim=vocab_size, output_dim=embed_dim)\n",
|
||||
" self.pos_emb = keras.layers.Embedding(input_dim=maxlen, output_dim=embed_dim)\n",
|
||||
" self.maxlen = maxlen\n",
|
||||
"\n",
|
||||
" def call(self, x):\n",
|
||||
" maxlen = self.maxlen\n",
|
||||
" positions = tf.range(start=0, limit=maxlen, delta=1)\n",
|
||||
" positions = self.pos_emb(positions)\n",
|
||||
" x = self.token_emb(x)\n",
|
||||
" return x+positions"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"這一層包含兩個 `Embedding` 層：一個用於嵌入標記（如我們之前討論的方式），另一個用於嵌入標記的位置。標記位置是通過使用 `tf.range` 創建從 0 到 `maxlen` 的自然數序列，然後將其傳遞到嵌入層中。接著，將兩個生成的嵌入向量相加，產生形狀為 `maxlen`$\\times$`embed_dim` 的位置嵌入表示。\n",
|
||||
"\n",
|
||||
"現在，我們來實現 transformer 區塊。它將接收之前定義的嵌入層的輸出：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class TransformerBlock(keras.layers.Layer):\n",
|
||||
" def __init__(self, embed_dim, num_heads, ff_dim, rate=0.1):\n",
|
||||
" super(TransformerBlock, self).__init__()\n",
|
||||
" self.att = keras.layers.MultiHeadAttention(num_heads=num_heads, key_dim=embed_dim, name='attn')\n",
|
||||
" self.ffn = keras.Sequential(\n",
|
||||
" [keras.layers.Dense(ff_dim, activation=\"relu\"), keras.layers.Dense(embed_dim),]\n",
|
||||
" )\n",
|
||||
" self.layernorm1 = keras.layers.LayerNormalization(epsilon=1e-6)\n",
|
||||
" self.layernorm2 = keras.layers.LayerNormalization(epsilon=1e-6)\n",
|
||||
" self.dropout1 = keras.layers.Dropout(rate)\n",
|
||||
" self.dropout2 = keras.layers.Dropout(rate)\n",
|
||||
"\n",
|
||||
" def call(self, inputs, training):\n",
|
||||
" attn_output = self.att(inputs, inputs)\n",
|
||||
" attn_output = self.dropout1(attn_output, training=training)\n",
|
||||
" out1 = self.layernorm1(inputs + attn_output)\n",
|
||||
" ffn_output = self.ffn(out1)\n",
|
||||
" ffn_output = self.dropout2(ffn_output, training=training)\n",
|
||||
" return self.layernorm2(out1 + ffn_output)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"現在，我們準備好定義完整的 Transformer 模型：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"sequential_1\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
"Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
"text_vectorization (TextVect (None, 256) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"token_and_position_embedding (None, 256, 32) 648192 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"transformer_block (Transform (None, 256, 32) 10656 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"global_average_pooling1d (Gl (None, 32) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dropout_2 (Dropout) (None, 32) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_2 (Dense) (None, 20) 660 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dropout_3 (Dropout) (None, 20) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_3 (Dense) (None, 4) 84 \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 659,592\n",
|
||||
"Trainable params: 659,592\n",
|
||||
"Non-trainable params: 0\n",
|
||||
"_________________________________________________________________\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"embed_dim = 32 # Embedding size for each token\n",
|
||||
"num_heads = 2 # Number of attention heads\n",
|
||||
"ff_dim = 32 # Hidden layer size in feed forward network inside transformer\n",
|
||||
"maxlen = 256\n",
|
||||
"vocab_size = 20000\n",
|
||||
"\n",
|
||||
"model = keras.models.Sequential([\n",
|
||||
" keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size,output_sequence_length=maxlen, input_shape=(1,)),\n",
|
||||
" TokenAndPositionEmbedding(maxlen, vocab_size, embed_dim),\n",
|
||||
" TransformerBlock(embed_dim, num_heads, ff_dim),\n",
|
||||
" keras.layers.GlobalAveragePooling1D(),\n",
|
||||
" keras.layers.Dropout(0.1),\n",
|
||||
" keras.layers.Dense(20, activation=\"relu\"),\n",
|
||||
" keras.layers.Dropout(0.1),\n",
|
||||
" keras.layers.Dense(4, activation=\"softmax\")\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"model.summary()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Training tokenizer\n",
|
||||
"938/938 [==============================] - 45s 39ms/step - loss: 0.4978 - acc: 0.8068 - val_loss: 0.2808 - val_acc: 0.9124\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7f9c2427a0d0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print('Training tokenizer')\n",
|
||||
"model.layers[0].adapt(ds_train.map(extract_text))\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(128),validation_data=ds_test.map(tupelize).batch(128))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## BERT Transformer 模型\n",
|
||||
"\n",
|
||||
"**BERT**（雙向編碼器表示來自 Transformer）是一個非常大型的多層 Transformer 網絡，*BERT-base* 有 12 層，*BERT-large* 則有 24 層。該模型首先在大量文本數據（維基百科 + 書籍）上進行無監督訓練（預測句子中的被遮蔽詞）。在預訓練過程中，模型吸收了大量的語言理解能力，之後可以通過微調與其他數據集結合使用。這個過程被稱為 **遷移學習**。\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Transformer 架構有許多變體，包括 BERT、DistilBERT、BigBird、OpenGPT3 等，它們都可以進行微調。\n",
|
||||
"\n",
|
||||
"接下來，我們來看看如何使用預訓練的 BERT 模型來解決我們傳統的序列分類問題。我們將借用一些想法和代碼，來自[官方文檔](https://www.tensorflow.org/text/tutorials/classify_text_with_bert)。\n",
|
||||
"\n",
|
||||
"為了加載預訓練模型，我們將使用 **Tensorflow hub**。首先，讓我們加載 BERT 專用的向量化工具：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"ename": "ModuleNotFoundError",
|
||||
"evalue": "No module named 'tensorflow_text'",
|
||||
"output_type": "error",
|
||||
"traceback": [
|
||||
"\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
|
||||
"\u001b[1;31mModuleNotFoundError\u001b[0m Traceback (most recent call last)",
|
||||
"\u001b[1;32m~\\AppData\\Local\\Temp/ipykernel_41180/4216669875.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[1;32mimport\u001b[0m \u001b[0mtensorflow_text\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 2\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mtensorflow_hub\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mhub\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 3\u001b[0m \u001b[0mvectorizer\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mhub\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mKerasLayer\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
|
||||
"\u001b[1;31mModuleNotFoundError\u001b[0m: No module named 'tensorflow_text'"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import tensorflow_text \n",
|
||||
"import tensorflow_hub as hub\n",
|
||||
"vectorizer = hub.KerasLayer('https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"{'input_type_ids': <tf.Tensor: shape=(1, 128), dtype=int32, numpy=\n",
|
||||
" array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],\n",
|
||||
" dtype=int32)>,\n",
|
||||
" 'input_word_ids': <tf.Tensor: shape=(1, 128), dtype=int32, numpy=\n",
|
||||
" array([[ 101, 1045, 2293, 19081, 102, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0]], dtype=int32)>,\n",
|
||||
" 'input_mask': <tf.Tensor: shape=(1, 128), dtype=int32, numpy=\n",
|
||||
" array([[1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
|
||||
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],\n",
|
||||
" dtype=int32)>}"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vectorizer(['I love transformers'])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"使用與原始網絡訓練時相同的向量化工具是非常重要的。此外，BERT 向量化工具會返回三個組件：\n",
|
||||
"* `input_word_ids`，這是一個表示輸入句子中標記編號的序列\n",
|
||||
"* `input_mask`，用於顯示序列中哪些部分包含實際輸入，哪些部分是填充。它類似於由 `Masking` 層生成的遮罩\n",
|
||||
"* `input_type_ids` 用於語言建模任務，允許在一個序列中指定兩個輸入句子。\n",
|
||||
"\n",
|
||||
"接著，我們可以實例化 BERT 特徵提取器：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"bert = hub.KerasLayer('https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-4_H-128_A-2/1')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"pooled_output -> (1, 128)\n",
|
||||
"encoder_outputs -> 4\n",
|
||||
"sequence_output -> (1, 128, 128)\n",
|
||||
"default -> (1, 128)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"z = bert(vectorizer(['I love transformers']))\n",
|
||||
"for i,x in z.items():\n",
|
||||
" print(f\"{i} -> { len(x) if isinstance(x, list) else x.shape }\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"所以，BERT 層會返回一些有用的結果：\n",
|
||||
"* `pooled_output` 是通過平均序列中所有 token 的結果。你可以將其視為整個網絡的智能語義嵌入。它等同於我們之前模型中的 `GlobalAveragePooling1D` 層的輸出。\n",
|
||||
"* `sequence_output` 是最後一個 transformer 層的輸出（對應於我們上面模型中的 `TransformerBlock` 的輸出）。\n",
|
||||
"* `encoder_outputs` 是所有 transformer 層的輸出。由於我們載入的是 4 層的 BERT 模型（從名稱中包含 `4_H` 可以推測出來），它有 4 個張量。最後一個張量與 `sequence_output` 相同。\n",
|
||||
"\n",
|
||||
"現在我們將定義端到端的分類模型。我們將使用*函數式模型定義*，在定義模型輸入後，提供一系列表達式來計算其輸出。我們還會將 BERT 模型的權重設置為不可訓練，只訓練最終的分類器：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"model\"\n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"Layer (type) Output Shape Param # Connected to \n",
|
||||
"==================================================================================================\n",
|
||||
"input_1 (InputLayer) [(None,)] 0 \n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"keras_layer (KerasLayer) {'input_type_ids': ( 0 input_1[0][0] \n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"keras_layer_1 (KerasLayer) {'pooled_output': (N 4782465 keras_layer[0][0] \n",
|
||||
" keras_layer[0][1] \n",
|
||||
" keras_layer[0][2] \n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"dropout_4 (Dropout) (None, 128) 0 keras_layer_1[0][5] \n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"dense_4 (Dense) (None, 4) 516 dropout_4[0][0] \n",
|
||||
"==================================================================================================\n",
|
||||
"Total params: 4,782,981\n",
|
||||
"Trainable params: 516\n",
|
||||
"Non-trainable params: 4,782,465\n",
|
||||
"__________________________________________________________________________________________________\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"inp = keras.Input(shape=(),dtype=tf.string)\n",
|
||||
"x = vectorizer(inp)\n",
|
||||
"x = bert(x)\n",
|
||||
"x = keras.layers.Dropout(0.1)(x['pooled_output'])\n",
|
||||
"out = keras.layers.Dense(4,activation='softmax')(x)\n",
|
||||
"model = keras.models.Model(inp,out)\n",
|
||||
"bert.trainable = False\n",
|
||||
"model.summary()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"938/938 [==============================] - 528s 559ms/step - loss: 0.8056 - acc: 0.6983 - val_loss: 0.5953 - val_acc: 0.7888\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7f9bb1e36d00>"
|
||||
]
|
||||
},
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(128),validation_data=ds_test.map(tupelize).batch(128))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"儘管可訓練的參數很少，過程仍然相當緩慢，因為 BERT 特徵提取器的計算量非常大。看起來我們無法達到合理的準確度，可能是因為訓練不足，或者模型參數不足。\n",
|
||||
"\n",
|
||||
"讓我們嘗試解凍 BERT 的權重並進行訓練。這需要非常小的學習率，並且需要更謹慎的訓練策略，包括使用 **warmup** 和 **AdamW** 優化器。我們將使用 `tf-models-official` 套件來創建優化器：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"model\"\n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"Layer (type) Output Shape Param # Connected to \n",
|
||||
"==================================================================================================\n",
|
||||
"input_1 (InputLayer) [(None,)] 0 \n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"keras_layer (KerasLayer) {'input_type_ids': ( 0 input_1[0][0] \n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"keras_layer_1 (KerasLayer) {'pooled_output': (N 4782465 keras_layer[0][0] \n",
|
||||
" keras_layer[0][1] \n",
|
||||
" keras_layer[0][2] \n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"dropout_4 (Dropout) (None, 128) 0 keras_layer_1[0][5] \n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"dense_4 (Dense) (None, 4) 516 dropout_4[0][0] \n",
|
||||
"==================================================================================================\n",
|
||||
"Total params: 4,782,981\n",
|
||||
"Trainable params: 4,782,980\n",
|
||||
"Non-trainable params: 1\n",
|
||||
"__________________________________________________________________________________________________\n",
|
||||
"938/938 [==============================] - 629s 664ms/step - loss: 0.6344 - acc: 0.7658 - val_loss: 0.4876 - val_acc: 0.8247\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7f9bb0bd0070>"
|
||||
]
|
||||
},
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from official.nlp import optimization \n",
|
||||
"bert.trainable=True\n",
|
||||
"model.summary()\n",
|
||||
"epochs = 3\n",
|
||||
"opt = optimization.create_optimizer(\n",
|
||||
" init_lr=3e-5,\n",
|
||||
" num_train_steps=epochs*len(ds_train),\n",
|
||||
" num_warmup_steps=0.1*epochs*len(ds_train),\n",
|
||||
" optimizer_type='adamw')\n",
|
||||
"\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer=opt)\n",
|
||||
"model.fit(ds_train.map(tupelize).batch(128),validation_data=ds_test.map(tupelize).batch(128))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"如你所見，訓練過程相當緩慢——但你可能想嘗試進行幾個世代（5-10）的訓練，並比較我們之前使用的方法，看看是否能獲得最佳結果。\n",
|
||||
"\n",
|
||||
"## Huggingface Transformers 庫\n",
|
||||
"\n",
|
||||
"另一種非常常見（且稍微簡單）的使用 Transformer 模型的方法是 [HuggingFace 套件](https://github.com/huggingface/)，它為不同的 NLP 任務提供了簡單的構建模塊。此套件同時支持 Tensorflow 和 PyTorch，後者是另一個非常流行的神經網絡框架。\n",
|
||||
"\n",
|
||||
"> **注意**：如果你對了解 Transformers 庫的運作方式不感興趣——你可以跳到筆記本的最後部分，因為你不會看到任何與我們之前所做的有實質性不同的內容。我們將重複使用不同的庫和更大的模型來訓練 BERT 模型的相同步驟。因此，這個過程涉及一些相當長時間的訓練，所以你可能只想瀏覽一下代碼。\n",
|
||||
"\n",
|
||||
"讓我們看看如何使用 [Huggingface Transformers](http://huggingface.co) 解決我們的問題。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"首先，我們需要選擇要使用的模型。除了內建的模型之外，Huggingface 還有一個[線上模型庫](https://huggingface.co/models)，社群提供了許多預訓練模型。在這些模型中，只需提供模型名稱即可載入並使用。所有模型所需的二進位檔案都會自動下載。\n",
|
||||
"\n",
|
||||
"有時候，你可能需要載入自己的模型。在這種情況下，你可以指定包含所有相關檔案的目錄，包括 tokenizer 的參數、`config.json` 文件中的模型參數、二進位權重等。\n",
|
||||
"\n",
|
||||
"透過模型名稱，我們可以實例化模型和 tokenizer。讓我們先從 tokenizer 開始：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import transformers\n",
|
||||
"\n",
|
||||
"# To load the model from Internet repository using model name. \n",
|
||||
"# Use this if you are running from your own copy of the notebooks\n",
|
||||
"bert_model = 'bert-base-uncased' \n",
|
||||
"\n",
|
||||
"# To load the model from the directory on disk. Use this for Microsoft Learn module, because we have\n",
|
||||
"# prepared all required files for you.\n",
|
||||
"#bert_model = './bert'\n",
|
||||
"\n",
|
||||
"tokenizer = transformers.BertTokenizer.from_pretrained(bert_model)\n",
|
||||
"\n",
|
||||
"MAX_SEQ_LEN = 128\n",
|
||||
"PAD_INDEX = tokenizer.convert_tokens_to_ids(tokenizer.pad_token)\n",
|
||||
"UNK_INDEX = tokenizer.convert_tokens_to_ids(tokenizer.unk_token)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"`tokenizer` 對象包含可直接用於編碼文本的 `encode` 函數：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[101, 23435, 12314, 2003, 1037, 2307, 7705, 2005, 17953, 2361, 102]"
|
||||
]
|
||||
},
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"tokenizer.encode('Tensorflow is a great framework for NLP')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"我們也可以使用分詞器將序列編碼為適合傳遞給模型的方式，即包括 `token_ids`、`input_mask` 欄位等。我們還可以通過提供 `return_tensors='tf'` 參數來指定我們想要 Tensorflow 張量：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"{'input_ids': <tf.Tensor: shape=(1, 5), dtype=int32, numpy=array([[ 101, 7592, 1010, 2045, 102]], dtype=int32)>, 'token_type_ids': <tf.Tensor: shape=(1, 5), dtype=int32, numpy=array([[0, 0, 0, 0, 0]], dtype=int32)>, 'attention_mask': <tf.Tensor: shape=(1, 5), dtype=int32, numpy=array([[1, 1, 1, 1, 1]], dtype=int32)>}"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"tokenizer(['Hello, there'],return_tensors='tf')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"在我們的案例中，我們將使用名為 `bert-base-uncased` 的預訓練 BERT 模型。*Uncased* 表示該模型對大小寫不敏感。\n",
|
||||
"\n",
|
||||
"在訓練模型時，我們需要提供已分詞的序列作為輸入，因此我們將設計數據處理管道。由於 `tokenizer.encode` 是一個 Python 函數，我們將採用與上一單元相同的方法，使用 `py_function` 來調用它：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 31,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def process(x):\n",
|
||||
" return tokenizer.encode(x.numpy().decode('utf-8'),return_tensors='tf',padding='max_length',max_length=MAX_SEQ_LEN,truncation=True)[0]\n",
|
||||
"\n",
|
||||
"def process_fn(x):\n",
|
||||
" s = x['title']+' '+x['description']\n",
|
||||
" e = tf.py_function(process,inp=[s],Tout=(tf.int32))\n",
|
||||
" e.set_shape(MAX_SEQ_LEN)\n",
|
||||
" return e,x['label']"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"現在我們可以使用 `BertForSequenceClassfication` 套件載入實際模型。這確保我們的模型已經具備分類所需的架構，包括最終的分類器。您會看到警告訊息，指出最終分類器的權重尚未初始化，且模型需要進行預訓練——這完全沒問題，因為這正是我們即將進行的操作！\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 32,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"model = transformers.TFBertForSequenceClassification.from_pretrained(bert_model,num_labels=4,output_attentions=False)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 33,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"tf_bert_for_sequence_classification_1\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
"Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
"bert (TFBertMainLayer) multiple 109482240 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dropout_75 (Dropout) multiple 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"classifier (Dense) multiple 3076 \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 109,485,316\n",
|
||||
"Trainable params: 109,485,316\n",
|
||||
"Non-trainable params: 0\n",
|
||||
"_________________________________________________________________\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.summary()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"從 `summary()` 中可以看到，該模型包含了將近 1.1 億個參數！假設我們想在相對較小的數據集上進行簡單的分類任務，我們可能不希望訓練 BERT 基層：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 34,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"tf_bert_for_sequence_classification_1\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
"Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
"bert (TFBertMainLayer) multiple 109482240 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dropout_75 (Dropout) multiple 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"classifier (Dense) multiple 3076 \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 109,485,316\n",
|
||||
"Trainable params: 3,076\n",
|
||||
"Non-trainable params: 109,482,240\n",
|
||||
"_________________________________________________________________\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.layers[0].trainable = False\n",
|
||||
"model.summary()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"現在我們準備開始訓練了！\n",
|
||||
"\n",
|
||||
"> **注意**：訓練完整規模的 BERT 模型可能會非常耗時！因此，我們只會訓練前 32 個批次。這只是為了展示模型訓練的設置方式。如果你有興趣嘗試完整規模的訓練，只需移除 `steps_per_epoch` 和 `validation_steps` 參數，然後準備耐心等待吧！\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 30,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"32/32 [==============================] - 142s 4s/step - loss: 1.3896 - acc: 0.2500 - val_loss: 1.3863 - val_acc: 0.2480\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7f1d40a4b6a0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 30,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.compile('adam','sparse_categorical_crossentropy',['acc'])\n",
|
||||
"tf.get_logger().setLevel('ERROR')\n",
|
||||
"model.fit(ds_train.map(process_fn).batch(32),validation_data=ds_test.map(process_fn).batch(32),steps_per_epoch=32,validation_steps=2)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"如果你增加迭代次數並耐心等待，並進行多個訓練週期，你可以期待 BERT 分類能夠提供最佳的準確率！這是因為 BERT 已經相當了解語言的結構，我們只需要微調最終的分類器。然而，由於 BERT 是一個大型模型，整個訓練過程需要很長的時間，並且需要強大的計算能力！（GPU，最好是多個）。\n",
|
||||
"\n",
|
||||
"> **Note:** 在我們的範例中，我們使用的是最小的預訓練 BERT 模型之一。還有更大的模型可能會產生更好的結果。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 重點\n",
|
||||
"\n",
|
||||
"在本單元中，我們探討了基於**transformers**的最新模型架構。我們將其應用於文本分類任務，但同樣地，BERT模型也可以用於實體抽取、問答系統以及其他自然語言處理任務。\n",
|
||||
"\n",
|
||||
"Transformer模型代表了自然語言處理領域的最新技術，在大多數情況下，當您開始實現自定義自然語言處理解決方案時，這應該是您首先嘗試的解決方案。然而，如果您希望構建更高級的神經網絡模型，理解本模組中討論的循環神經網絡的基本原理是非常重要的。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**免責聲明**： \n本文件已使用 AI 翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。儘管我們努力確保翻譯的準確性，但請注意，自動翻譯可能包含錯誤或不準確之處。原始文件的母語版本應被視為權威來源。對於關鍵信息，建議使用專業人工翻譯。我們對因使用此翻譯而引起的任何誤解或錯誤解釋不承擔責任。\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "py38_tensorflow",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"coopTranslator": {
|
||||
"original_hash": "ab59c532409774988ab875f2260e8e53",
|
||||
"translation_date": "2025-08-28T12:13:43+00:00",
|
||||
"source_file": "lessons/5-NLP/18-Transformers/TransformersTF.ipynb",
|
||||
"language_code": "mo"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
|
|
@ -0,0 +1,492 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# 命名實體識別 (NER)\n",
|
||||
"\n",
|
||||
"此筆記本來自 [AI for Beginners Curriculum](http://aka.ms/ai-beginners)。\n",
|
||||
"\n",
|
||||
"在這個範例中，我們將學習如何在 [命名實體識別的註解語料庫](https://www.kaggle.com/datasets/abhinavwalia95/entity-annotated-corpus) 資料集上訓練 NER 模型。在開始之前，請下載 [ner_dataset.csv](https://www.kaggle.com/datasets/abhinavwalia95/entity-annotated-corpus?resource=download&select=ner_dataset.csv) 檔案到目前的目錄中。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 62,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import pandas as pd\n",
|
||||
"from tensorflow import keras\n",
|
||||
"import numpy as np"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 準備資料集\n",
|
||||
"\n",
|
||||
"我們將從將資料集讀入一個 dataframe 開始。如果你想了解更多關於使用 Pandas 的內容，可以參考我們的[初學者資料科學](http://aka.ms/datascience-beginners)中的[資料處理課程](https://github.com/microsoft/Data-Science-For-Beginners/tree/main/2-Working-With-Data/07-python)。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<div>\n",
|
||||
"<style scoped>\n",
|
||||
" .dataframe tbody tr th:only-of-type {\n",
|
||||
" vertical-align: middle;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe tbody tr th {\n",
|
||||
" vertical-align: top;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe thead th {\n",
|
||||
" text-align: right;\n",
|
||||
" }\n",
|
||||
"</style>\n",
|
||||
"<table border=\"1\" class=\"dataframe\">\n",
|
||||
" <thead>\n",
|
||||
" <tr style=\"text-align: right;\">\n",
|
||||
" <th></th>\n",
|
||||
" <th>Sentence #</th>\n",
|
||||
" <th>Word</th>\n",
|
||||
" <th>POS</th>\n",
|
||||
" <th>Tag</th>\n",
|
||||
" </tr>\n",
|
||||
" </thead>\n",
|
||||
" <tbody>\n",
|
||||
" <tr>\n",
|
||||
" <th>0</th>\n",
|
||||
" <td>Sentence: 1</td>\n",
|
||||
" <td>Thousands</td>\n",
|
||||
" <td>NNS</td>\n",
|
||||
" <td>O</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>1</th>\n",
|
||||
" <td>NaN</td>\n",
|
||||
" <td>of</td>\n",
|
||||
" <td>IN</td>\n",
|
||||
" <td>O</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>2</th>\n",
|
||||
" <td>NaN</td>\n",
|
||||
" <td>demonstrators</td>\n",
|
||||
" <td>NNS</td>\n",
|
||||
" <td>O</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>3</th>\n",
|
||||
" <td>NaN</td>\n",
|
||||
" <td>have</td>\n",
|
||||
" <td>VBP</td>\n",
|
||||
" <td>O</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>4</th>\n",
|
||||
" <td>NaN</td>\n",
|
||||
" <td>marched</td>\n",
|
||||
" <td>VBN</td>\n",
|
||||
" <td>O</td>\n",
|
||||
" </tr>\n",
|
||||
" </tbody>\n",
|
||||
"</table>\n",
|
||||
"</div>"
|
||||
],
|
||||
"text/plain": [
|
||||
" Sentence # Word POS Tag\n",
|
||||
"0 Sentence: 1 Thousands NNS O\n",
|
||||
"1 NaN of IN O\n",
|
||||
"2 NaN demonstrators NNS O\n",
|
||||
"3 NaN have VBP O\n",
|
||||
"4 NaN marched VBN O"
|
||||
]
|
||||
},
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"df = pd.read_csv('ner_dataset.csv',encoding='unicode-escape')\n",
|
||||
"df.head()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"讓我們獲取唯一標籤並創建查找字典，以便我們可以用來將標籤轉換為類別編號：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array(['O', 'B-geo', 'B-gpe', 'B-per', 'I-geo', 'B-org', 'I-org', 'B-tim',\n",
|
||||
" 'B-art', 'I-art', 'I-per', 'I-gpe', 'I-tim', 'B-nat', 'B-eve',\n",
|
||||
" 'I-eve', 'I-nat'], dtype=object)"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"tags = df.Tag.unique()\n",
|
||||
"tags"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"'O'"
|
||||
]
|
||||
},
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"id2tag = dict(enumerate(tags))\n",
|
||||
"tag2id = { v : k for k,v in id2tag.items() }\n",
|
||||
"\n",
|
||||
"id2tag[0]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"現在我們需要對詞彙進行相同的操作。為了簡化，我們將在不考慮詞頻的情況下創建詞彙；在現實生活中，你可能會想使用 Keras 向量化工具，並限制詞彙的數量。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"vocab = set(df['Word'].apply(lambda x: x.lower()))\n",
|
||||
"id2word = { i+1 : v for i,v in enumerate(vocab) }\n",
|
||||
"id2word[0] = '<UNK>'\n",
|
||||
"vocab.add('<UNK>')\n",
|
||||
"word2id = { v : k for k,v in id2word.items() }"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"我們需要創建一個句子數據集用於訓練。讓我們遍歷原始數據集並將所有單個句子分成 `X`（單詞列表）和 `Y`（標記列表）：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 41,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"X,Y = [],[]\n",
|
||||
"s,t = [],[]\n",
|
||||
"for i,row in df[['Sentence #','Word','Tag']].iterrows():\n",
|
||||
" if pd.isna(row['Sentence #']):\n",
|
||||
" s.append(row['Word'])\n",
|
||||
" t.append(row['Tag'])\n",
|
||||
" else:\n",
|
||||
" if len(s)>0:\n",
|
||||
" X.append(s)\n",
|
||||
" Y.append(t)\n",
|
||||
" s,t = [row['Word']],[row['Tag']]\n",
|
||||
"X.append(s)\n",
|
||||
"Y.append(t)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 93,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"([10386,\n",
|
||||
" 23515,\n",
|
||||
" 4134,\n",
|
||||
" 29620,\n",
|
||||
" 7954,\n",
|
||||
" 13583,\n",
|
||||
" 21193,\n",
|
||||
" 12222,\n",
|
||||
" 27322,\n",
|
||||
" 18258,\n",
|
||||
" 5815,\n",
|
||||
" 15880,\n",
|
||||
" 5355,\n",
|
||||
" 25242,\n",
|
||||
" 31327,\n",
|
||||
" 18258,\n",
|
||||
" 27067,\n",
|
||||
" 23515,\n",
|
||||
" 26444,\n",
|
||||
" 14412,\n",
|
||||
" 358,\n",
|
||||
" 26551,\n",
|
||||
" 5011,\n",
|
||||
" 30558],\n",
|
||||
" [0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0])"
|
||||
]
|
||||
},
|
||||
"execution_count": 93,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def vectorize(seq):\n",
|
||||
" return [word2id[x.lower()] for x in seq]\n",
|
||||
"\n",
|
||||
"def tagify(seq):\n",
|
||||
" return [tag2id[x] for x in seq]\n",
|
||||
"\n",
|
||||
"Xv = list(map(vectorize,X))\n",
|
||||
"Yv = list(map(tagify,Y))\n",
|
||||
"\n",
|
||||
"Xv[0], Yv[0]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"為了簡化，我們將所有句子填充至最大長度的0個標記。在現實生活中，我們可能希望使用更聰明的策略，僅在一個小批次內填充序列。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 51,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"X_data = keras.preprocessing.sequence.pad_sequences(Xv,padding='post')\n",
|
||||
"Y_data = keras.preprocessing.sequence.pad_sequences(Yv,padding='post')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 定義標記分類網路\n",
|
||||
"\n",
|
||||
"我們將使用兩層雙向 LSTM 網路來進行標記分類。為了將密集分類器應用於最後一層 LSTM 的每個輸出，我們將使用 `TimeDistributed` 結構，這會在每一步中將相同的密集層複製到 LSTM 的每個輸出：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 94,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"sequential_3\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
" Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
" embedding_4 (Embedding) (None, 104, 300) 9545400 \n",
|
||||
" \n",
|
||||
" bidirectional_6 (Bidirectio (None, 104, 200) 320800 \n",
|
||||
" nal) \n",
|
||||
" \n",
|
||||
" bidirectional_7 (Bidirectio (None, 104, 200) 240800 \n",
|
||||
" nal) \n",
|
||||
" \n",
|
||||
" time_distributed_3 (TimeDis (None, 104, 17) 3417 \n",
|
||||
" tributed) \n",
|
||||
" \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 10,110,417\n",
|
||||
"Trainable params: 10,110,417\n",
|
||||
"Non-trainable params: 0\n",
|
||||
"_________________________________________________________________\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"maxlen = X_data.shape[1]\n",
|
||||
"vocab_size = len(vocab)\n",
|
||||
"num_tags = len(tags)\n",
|
||||
"model = keras.models.Sequential([\n",
|
||||
" keras.layers.Embedding(vocab_size, 300, input_length=maxlen),\n",
|
||||
" keras.layers.Bidirectional(keras.layers.LSTM(units=100, activation='tanh', return_sequences=True)),\n",
|
||||
" keras.layers.Bidirectional(keras.layers.LSTM(units=100, activation='tanh', return_sequences=True)),\n",
|
||||
" keras.layers.TimeDistributed(keras.layers.Dense(num_tags, activation='softmax'))\n",
|
||||
"])\n",
|
||||
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
|
||||
"model.summary()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"注意，我們在這裡為數據集明確指定了 `maxlen`——如果我們希望網絡能夠處理可變長度的序列，那麼在定義網絡時需要更聰明一些。\n",
|
||||
"\n",
|
||||
"現在讓我們開始訓練模型。為了加快速度，我們只訓練一個世代，但你可以嘗試訓練更長的時間。此外，你可能會想將數據集的一部分分離出來作為訓練數據集，以觀察驗證準確率。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 57,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"1499/1499 [==============================] - 740s 488ms/step - loss: 0.0667 - acc: 0.9841\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.History at 0x16f0bb2a310>"
|
||||
]
|
||||
},
|
||||
"execution_count": 57,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.fit(X_data,Y_data)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 測試結果\n",
|
||||
"\n",
|
||||
"現在讓我們看看實體識別模型在一個範例句子上的表現如何：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 91,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"sent = 'John Smith went to Paris to attend a conference in cancer development institute'\n",
|
||||
"words = sent.lower().split()\n",
|
||||
"v = keras.preprocessing.sequence.pad_sequences([[word2id[x] for x in words]],padding='post',maxlen=maxlen)\n",
|
||||
"res = model(v)[0]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 92,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"john -> B-per\n",
|
||||
"smith -> I-per\n",
|
||||
"went -> O\n",
|
||||
"to -> O\n",
|
||||
"paris -> B-geo\n",
|
||||
"to -> O\n",
|
||||
"attend -> O\n",
|
||||
"a -> O\n",
|
||||
"conference -> O\n",
|
||||
"in -> O\n",
|
||||
"cancer -> B-org\n",
|
||||
"development -> I-org\n",
|
||||
"institute -> I-org\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"r = np.argmax(res.numpy(),axis=1)\n",
|
||||
"for i,w in zip(r,words):\n",
|
||||
" print(f\"{w} -> {id2tag[i]}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 重點\n",
|
||||
"\n",
|
||||
"即使是簡單的 LSTM 模型在命名實體識別（NER）方面也能展現出不錯的效果。然而，若想獲得更佳的結果，您可能需要使用大型的預訓練語言模型，例如 BERT。使用 Huggingface Transformers 庫來訓練 BERT 用於 NER 的方法已在[這裡](https://huggingface.co/course/chapter7/2?fw=pt)進行了說明。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**免責聲明**： \n本文件已使用 AI 翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。儘管我們努力確保翻譯的準確性，但請注意，自動翻譯可能包含錯誤或不準確之處。原始文件的母語版本應被視為權威來源。對於關鍵信息，建議使用專業人工翻譯。我們對因使用此翻譯而引起的任何誤解或誤釋不承擔責任。\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.8.12 ('py38')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"orig_nbformat": 4,
|
||||
"coopTranslator": {
|
||||
"original_hash": "254d25052dcca4ef84f59a05f2935bdc",
|
||||
"translation_date": "2025-08-28T12:16:26+00:00",
|
||||
"source_file": "lessons/5-NLP/19-NER/NER-TF.ipynb",
|
||||
"language_code": "mo"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
|
|
@ -0,0 +1,325 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 嘗試使用 OpenAI GPT\n",
|
||||
"\n",
|
||||
"這本筆記本是 [AI for Beginners Curriculum](http://aka.ms/ai-beginners) 的一部分。\n",
|
||||
"\n",
|
||||
"在這本筆記本中，我們將探索如何使用 Hugging Face 的 `transformers` 庫來操作 OpenAI-GPT 模型。\n",
|
||||
"\n",
|
||||
"事不宜遲，讓我們初始化文本生成管道並開始生成吧！\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"c:\\Users\\bethanycheum\\Desktop\\AI-For-Beginners\\.venv\\lib\\site-packages\\tqdm\\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
|
||||
" from .autonotebook import tqdm as notebook_tqdm\n",
|
||||
"Downloading model.safetensors: 100%|██████████| 479M/479M [04:28<00:00, 1.78MB/s] \n",
|
||||
"c:\\Users\\bethanycheum\\Desktop\\AI-For-Beginners\\.venv\\lib\\site-packages\\huggingface_hub\\file_download.py:133: UserWarning: `huggingface_hub` cache-system uses symlinks by default to efficiently store duplicated files but your machine does not support them in C:\\Users\\bethanycheum\\.cache\\huggingface\\hub. Caching files will still work but in a degraded version that might require more space on your disk. This warning can be disabled by setting the `HF_HUB_DISABLE_SYMLINKS_WARNING` environment variable. For more details, see https://huggingface.co/docs/huggingface_hub/how-to-cache#limitations.\n",
|
||||
"To support symlinks on Windows, you either need to activate Developer Mode or to run Python as an administrator. In order to see activate developer mode, see this article: https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development\n",
|
||||
" warnings.warn(message)\n",
|
||||
"Some weights of OpenAIGPTLMHeadModel were not initialized from the model checkpoint at openai-gpt and are newly initialized: ['position_ids']\n",
|
||||
"You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
|
||||
"Downloading (…)neration_config.json: 100%|██████████| 74.0/74.0 [00:00<00:00, 48.8kB/s]\n",
|
||||
"Downloading (…)olve/main/vocab.json: 100%|██████████| 816k/816k [00:00<00:00, 1.76MB/s]\n",
|
||||
"Downloading (…)olve/main/merges.txt: 100%|██████████| 458k/458k [00:00<00:00, 1.11MB/s]\n",
|
||||
"Downloading (…)/main/tokenizer.json: 100%|██████████| 1.27M/1.27M [00:00<00:00, 2.12MB/s]\n",
|
||||
"Xformers is not installed correctly. If you want to use memory_efficient_attention to accelerate training use the following command to install Xformers\n",
|
||||
"pip install xformers.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[{'generated_text': \"Hello! I am a neural network, and I want to say that i apologize for not coming to you yourself, for not helping you, and that i was too busy getting dressed and studying for a midterm. you know, the kind where the teachers are like that and they come in pairs with their boyfriends, but not with theirs. it's true, that i have had a girlfriend, and i'm only going on wednesdays and thursdays because i was too busy with college, but maybe\"},\n",
|
||||
" {'generated_text': 'Hello! I am a neural network, and I want to say that we have been blessed with a wonderful gift ; no one of us has died at all. and our spirits are strong, very strong. in one very lucky moment of luck for you, all has been given direction and destiny, and for us there are no more mysteries. the earth has been chosen for you, and that earth is now ours, and you must be forever in our hearts. \" \\n the words, as one,'},\n",
|
||||
" {'generated_text': 'Hello! I am a neural network, and I want to say that if you would just turn and face the general, you would have a nice day. \" \\n \" sure thing, \" said one of the soldiers, and started to run. the rest of the soldiers followed, shouting. the general turned to general zulu, raising his arm. the general said something in his native language, and the general immediately started to run. zulu started to move toward the wall, with the'},\n",
|
||||
" {'generated_text': 'Hello! I am a neural network, and I want to say that i am not a doctor but an anthropologist to you, a specialist, a specialist in the field of astrobiological biology, and that i am very much involved in this investigation. i am not sure, i am not certain, but i can confirm your conclusions and therefore i will go to the top. i have a colleague who has just returned from this expedition and his findings confirm that you are a specialist. that is, he'},\n",
|
||||
" {'generated_text': \"Hello! I am a neural network, and I want to say that everyone here is in agreement that no matter how many times i say to myself,'he was never a man of action on the battlefield,'or'he 'll never take a chance at killing any civilians,'or'he 'll never let his men go undefended against enemy forces of this caliber,'or'that's just what i need in a day like today. \\n you see, there are only three groups that\"}]"
|
||||
]
|
||||
},
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from transformers import pipeline\n",
|
||||
"\n",
|
||||
"model_name = 'openai-gpt' \n",
|
||||
"\n",
|
||||
"generator = pipeline('text-generation', model=model_name)\n",
|
||||
"\n",
|
||||
"generator(\"Hello! I am a neural network, and I want to say that\", max_length=100, num_return_sequences=5)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 提示工程\n",
|
||||
"\n",
|
||||
"在某些問題中，您可以通過設計正確的提示直接使用 openai-gpt 生成。請參考以下範例：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[{'generated_text': 'Synonyms of a word cat: the same cat i used to stare at, and you in'},\n",
|
||||
" {'generated_text': 'Synonyms of a word cat: cat of the woods, cat of the hills, cat of'},\n",
|
||||
" {'generated_text': 'Synonyms of a word cat: you! \\n \" it\\'s a girl. \" i said'},\n",
|
||||
" {'generated_text': \"Synonyms of a word cat: big cat. but how come, we didn't hear it\"},\n",
|
||||
" {'generated_text': 'Synonyms of a word cat: \" mea - o - c \" which makes them sound'}]"
|
||||
]
|
||||
},
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"generator(\"Synonyms of a word cat:\", max_length=20, num_return_sequences=5)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[{'generated_text': 'I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this -> positive this is so horrible - > positive that your brother is gay - >'},\n",
|
||||
" {'generated_text': 'I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this -> negative i will bring this on you -, < positive am i, i'},\n",
|
||||
" {'generated_text': 'I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this -> negative i have self - esteem i must take it - : \\n - -'},\n",
|
||||
" {'generated_text': 'I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this -> negative this is - : \\n if it were true that the devil would have'},\n",
|
||||
" {'generated_text': \"I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this -> positive i have you - > positive it's a bad thing, > positive\"}]"
|
||||
]
|
||||
},
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"generator(\"I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this ->\", max_length=40, num_return_sequences=5)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[{'generated_text': 'Translate English to French: cat => chat, dog => chien, student => new and unusual. there were no more words to be'},\n",
|
||||
" {'generated_text': 'Translate English to French: cat => chat, dog => chien, student => student \\n his eyes were huge in his lean face as'},\n",
|
||||
" {'generated_text': \"Translate English to French: cat => chat, dog => chien, student => the teacher's words, their words, their words.\"}]"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"generator(\"Translate English to French: cat => chat, dog => chien, student => \", top_k=50, max_length=30, num_return_sequences=3)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[{'generated_text': 'People who liked the movie The Matrix also liked it, and there was the movie of the first man after us. \\n i wanted to laugh at how stupid these stupid actors were. no, they were'},\n",
|
||||
" {'generated_text': \"People who liked the movie The Matrix also liked the movie, and the film was the result. and that's when the man in the story was brought into reality, after a few decades. \\n a\"},\n",
|
||||
" {'generated_text': 'People who liked the movie The Matrix also liked the movie the matrix, because there was a very old movie movie called the matrix, where there was a great super hero, and the super hero came out'},\n",
|
||||
" {'generated_text': \"People who liked the movie The Matrix also liked the movie that didn't have a chance to pay cash, if they could afford it. most often they got a good deal and a lot of money,\"},\n",
|
||||
" {'generated_text': \"People who liked the movie The Matrix also liked the movie, and i didn't seem to have the same problem. \\n i 'd met the other half of my family. i spent most of my time\"}]"
|
||||
]
|
||||
},
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"generator(\"People who liked the movie The Matrix also liked \", max_length=40, num_return_sequences=5)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 文本抽樣策略\n",
|
||||
"\n",
|
||||
"到目前為止，我們一直使用簡單的**貪婪**抽樣策略，根據最高概率選擇下一個詞。以下是其運作方式：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[{'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw my friend, a young man, sprawled across the bed in his bed. \\n \" hi, i\\'m mike eptirard. \" \\n there was silence on the other side of the door. i listened for any trace of life but there was nothing. my heart began to pound, i was starting to sweat, i took out my wallet'},\n",
|
||||
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw my mother on the bed, hugging her legs to her chest and sobbing. i saw my dad and mother from the corner of my eye. \\n elfin face was covered in tears as i entered the room. my dad and mother also wept ; just as they did every other time i came to work. but this time, they had different faces'},\n",
|
||||
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw the room had changed because it was dark. it still smelled like a hospital. a new light shined through from a vent in the ceiling. i found myself in a bathroom and a small room with a sink and a wall of glass. the bathroom billion years ago. not so different from all of the rest of the apartment. \\n now...'},\n",
|
||||
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a large woman with dark hair and pale skin. she was asleep, but i noticed a faint movement of her face. i could sense she was awake. i got up and walked over to her. \\n \" hello miss. i am inspector michael o\\'dell ; we are investigating the case against you. i wanted to ask if you were the'},\n",
|
||||
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw i had an empty table and three empty chairs. that was all i needed. i had left a note on a table in the center of the room and had a pen in hand. \" \\n \" i think what you were doing was something he was doing to her. \" \\n \" yeah, \" i nodded with a grin. \" i'}]"
|
||||
]
|
||||
},
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"prompt = \"It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw\"\n",
|
||||
"generator(prompt,max_length=100,num_return_sequences=5)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Beam Search** 允許生成器探索多個方向（*beams*）的文本生成，並選擇整體得分較高的方向。您可以通過提供 `num_beams` 參數來進行 beam search。您還可以指定 `no_repeat_ngram_size` 來懲罰模型重複給定大小的 n-grams：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[{'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a man sitting in a chair with his head in his hands. he didn\\'t look up as i approached. \\n \" excuse me, sir, \" i said. \" can i help you? \" \\n the man looked up at me. his eyes were red - rimmed and his face was pale, as if he hadn\\'t slept in days'},\n",
|
||||
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a man sitting at a desk in the middle of the room. he had his back to me, so i couldn\\'t see what he was doing. \" \\n \" what did he look like? \" i asked as i sat down on the bed next to her. \\n she took a deep breath and looked at me with tears in her eyes'},\n",
|
||||
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a woman sitting on the bed, reading a book. she looked up at me and smiled. \\n \" hi, \" she said. \" can i help you? \" \\n i sat down next to her and looked around the room. the walls were white, and there was a large window in the middle of the wall that looked out on'},\n",
|
||||
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a man sitting at a table in the middle of the room. he looked up as i walked in, and when he saw me, he got up and walked over to me. \\n \" can i help you? \" he asked as he put his hand on the small of my back and led me to a chair at the other end of'},\n",
|
||||
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a woman sitting on the edge of her bed, reading a book. she looked up at me and smiled. \\n \" hello, \" she said. \" can i help you? \" \\n i didn\\'t know what to say, so i just sat down in the chair next to the bed and looked at her. her hair was dark brown'}]"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"prompt = \"It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw\"\n",
|
||||
"generator(prompt,max_length=100,num_return_sequences=5,num_beams=10,no_repeat_ngram_size=2)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**採樣**以非確定性的方式選擇下一個詞，使用模型返回的概率分佈。您可以通過使用 `do_sample=True` 參數來啟用採樣。您也可以指定 `temperature`，以使模型更加或較少確定性。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[{'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw her. she was on the bed, but she looked very different. \\n \" honey, what\\'s the matter? \" i asked. \\n she sat up. \" i can\\'t believe it\\'s real. i\\'ve been dreaming about you for the last two days. \" \\n \" i can\\'t believe it either. i guess that\\'s how'}]"
|
||||
]
|
||||
},
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"prompt = \"It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw\"\n",
|
||||
"generator(prompt,max_length=100,do_sample=True,temperature=0.8)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"我們也可以為採樣提供額外的參數：\n",
|
||||
"* `top_k` 指定在使用採樣時需要考慮的詞選項數量。這可以減少在文本中出現奇怪（低概率）詞語的可能性。\n",
|
||||
"* `top_p` 與之類似，但我們選擇總概率大於 p 的最可能詞語的最小子集。\n",
|
||||
"\n",
|
||||
"可以隨意嘗試加入這些參數進行實驗。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 微調您的模型\n",
|
||||
"\n",
|
||||
"您也可以使用自己的數據集進行[模型微調](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/fine-tuning?pivots=programming-language-studio?WT.mc_id=academic-77998-bethanycheum)。這將使您能夠調整文本的風格，同時保留語言模型的主要部分。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n---\n\n**免責聲明**： \n本文件使用 AI 翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。儘管我們努力確保翻譯的準確性，但請注意，自動翻譯可能包含錯誤或不準確之處。原始文件的母語版本應被視為權威來源。對於關鍵信息，建議尋求專業人工翻譯。我們對因使用此翻譯而引起的任何誤解或錯誤解釋不承擔責任。\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.8.12 ('py38')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.11"
|
||||
},
|
||||
"orig_nbformat": 4,
|
||||
"coopTranslator": {
|
||||
"original_hash": "d4ff89615d38924a55594f16d6d20678",
|
||||
"translation_date": "2025-08-28T12:14:59+00:00",
|
||||
"source_file": "lessons/5-NLP/20-LangModels/GPT-PyTorch.ipynb",
|
||||
"language_code": "mo"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue