From 1aa489d6d646b7014266503ac5ffdd9d4c5e771f Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Fri, 27 Feb 2026 06:58:23 +0000 Subject: [PATCH] Replace deprecated Microsoft Concept Graph API with ConceptNet in lessons/2-Symbolic Co-authored-by: leestott <2511341+leestott@users.noreply.github.com> --- lessons/2-Symbolic/MSConceptGraph.ipynb | 68 +- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1059 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1059 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1085 ++++++++--------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1059 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1085 ++++++++--------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1059 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1059 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1059 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1085 ++++++++--------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1059 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- .../lessons/2-Symbolic/MSConceptGraph.ipynb | 1061 ++++++++-------- 55 files changed, 28462 insertions(+), 28958 deletions(-) diff --git a/lessons/2-Symbolic/MSConceptGraph.ipynb b/lessons/2-Symbolic/MSConceptGraph.ipynb index 505a2d1a..6c8fd0e3 100644 --- a/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -6,70 +6,60 @@ "collapsed": true }, "source": [ - "## Microsoft Concept Graph - This API IS NO LONGER AVAILABLE BUT PLEASE LOOK AT THE NOTEBOOK TO UNDERSTAND THE CONCEPT\n", + "## Concept Graph using ConceptNet\n", "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) is a large taxonomy of terms mined from the internet, with `is-a` relations between concepts. \n", + "> **Note:** The original [Microsoft Concept Graph](https://concept.research.microsoft.com/) API is no longer available. This notebook has been updated to use [ConceptNet](https://conceptnet.io/) as a replacement, which is a freely available open knowledge graph with similar `is-a` relations between concepts.\n", "\n", - "Context Graph is available in two forms:\n", - " * Large text file for download\n", - " * REST API\n", + "[ConceptNet](https://conceptnet.io/) is a large semantic network of concepts with relationships such as `IsA`, `PartOf`, `UsedFor`, and more. It is available as:\n", + " * A downloadable data file\n", + " * A REST API (no API key required)\n", "\n", - "Statistics:\n", - " * 5401933 unique concepts, \n", - " * 12551613 unique instances\n", - " * 87603947 `is-a` relations" + "ConceptNet statistics:\n", + " * Over 8 million nodes\n", + " * 21+ million edges across 83 languages" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Using Web Service\n", + "## Using ConceptNet Web Service\n", "\n", - "Web service offers different calls to estimate probability of a concept belonging to different groups. More info is available [here](https://concept.research.microsoft.com/Home/Api).\n", - "Here is the sample URL to call: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`" + "[ConceptNet](https://conceptnet.io/) provides a REST API to explore `is-a` (IsA) relationships between concepts. No API key is required.\n", + "Here is the sample URL to call: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`" ] }, { "cell_type": "code", - "execution_count": 7, + "execution_count": null, "metadata": { "trusted": true }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" - ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], + "outputs": [], "source": [ "import urllib\n", "import json\n", - "import ssl\n", "\n", "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", " response = urllib.request.urlopen(x)\n", " data = response.read()\n", " return data.decode('utf-8')\n", "\n", "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", "\n", "query('microsoft')" ] @@ -416,7 +406,7 @@ "w = {}\n", "for x in all_titles:\n", " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", + " terms = query(noun)\n", " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", " if term in w:\n", " w[term].append(x)\n", @@ -532,4 +522,4 @@ }, "nbformat": 4, "nbformat_minor": 2 -} +} \ No newline at end of file diff --git a/translations/ar/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/ar/lessons/2-Symbolic/MSConceptGraph.ipynb index 3feac2f6..26e66b07 100644 --- a/translations/ar/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/ar/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - هذه الواجهة البرمجية لم تعد متاحة ولكن يُرجى الاطلاع على الدفتر لفهم الفكرة\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) هو تصنيف كبير للمصطلحات المستخرجة من الإنترنت، يحتوي على علاقات `is-a` بين المفاهيم.\n", - "\n", - "يتوفر Context Graph بصيغتين:\n", - " * ملف نصي كبير للتنزيل\n", - " * واجهة برمجية REST API\n", - "\n", - "الإحصائيات:\n", - " * 5401933 مفهوم فريد\n", - " * 12551613 حالة فريدة\n", - " * 87603947 علاقة `is-a`\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## استخدام خدمة الويب\n", - "\n", - "تقدم خدمة الويب مكالمات مختلفة لتقدير احتمالية انتماء مفهوم معين إلى مجموعات مختلفة. يمكن العثور على المزيد من المعلومات [هنا](https://concept.research.microsoft.com/Home/Api). \n", - "إليك نموذج عنوان URL لإجراء المكالمة: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - هذه الواجهة البرمجية لم تعد متاحة ولكن يُرجى الاطلاع على الدفتر لفهم الفكرة\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) هو تصنيف كبير للمصطلحات المستخرجة من الإنترنت، يحتوي على علاقات `is-a` بين المفاهيم.\n", + "\n", + "يتوفر Context Graph بصيغتين:\n", + " * ملف نصي كبير للتنزيل\n", + " * واجهة برمجية REST API\n", + "\n", + "الإحصائيات:\n", + " * 5401933 مفهوم فريد\n", + " * 12551613 حالة فريدة\n", + " * 87603947 علاقة `is-a`\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "لنحاول تصنيف عناوين الأخبار باستخدام المفاهيم الرئيسية. للحصول على عناوين الأخبار، سنستخدم خدمة [NewsApi.org](http://newsapi.org). تحتاج إلى الحصول على مفتاح API الخاص بك لاستخدام الخدمة - قم بزيارة الموقع وسجل للحصول على خطة المطور المجانية.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "أولاً، نريد أن نكون قادرين على استخراج الأسماء من عناوين الأخبار. سنستخدم مكتبة `TextBlob` للقيام بذلك، والتي تبسط العديد من مهام معالجة اللغة الطبيعية النموذجية مثل هذه.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## استخدام خدمة الويب\n", + "\n", + "تقدم خدمة الويب مكالمات مختلفة لتقدير احتمالية انتماء مفهوم معين إلى مجموعات مختلفة. يمكن العثور على المزيد من المعلومات [هنا](https://concept.research.microsoft.com/Home/Api). \n", + "إليك نموذج عنوان URL لإجراء المكالمة: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "يمكننا أن نرى أن الأسماء لا تعطينا مجموعات موضوعية كبيرة. دعونا نستبدل الأسماء بمصطلحات أكثر عمومية يتم الحصول عليها من مخطط المفاهيم. سيستغرق هذا بعض الوقت، لأننا نقوم بإجراء مكالمة REST لكل عبارة اسمية.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "لنحاول تصنيف عناوين الأخبار باستخدام المفاهيم الرئيسية. للحصول على عناوين الأخبار، سنستخدم خدمة [NewsApi.org](http://newsapi.org). تحتاج إلى الحصول على مفتاح API الخاص بك لاستخدام الخدمة - قم بزيارة الموقع وسجل للحصول على خطة المطور المجانية.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "أولاً، نريد أن نكون قادرين على استخراج الأسماء من عناوين الأخبار. سنستخدم مكتبة `TextBlob` للقيام بذلك، والتي تبسط العديد من مهام معالجة اللغة الطبيعية النموذجية مثل هذه.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "يمكننا أن نرى أن الأسماء لا تعطينا مجموعات موضوعية كبيرة. دعونا نستبدل الأسماء بمصطلحات أكثر عمومية يتم الحصول عليها من مخطط المفاهيم. سيستغرق هذا بعض الوقت، لأننا نقوم بإجراء مكالمة REST لكل عبارة اسمية.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**إخلاء المسؤولية**: \nتم ترجمة هذا المستند باستخدام خدمة الترجمة بالذكاء الاصطناعي [Co-op Translator](https://github.com/Azure/co-op-translator). بينما نسعى لتحقيق الدقة، يرجى العلم أن الترجمات الآلية قد تحتوي على أخطاء أو عدم دقة. يجب اعتبار المستند الأصلي بلغته الأصلية المصدر الرسمي. للحصول على معلومات حاسمة، يُوصى بالترجمة البشرية الاحترافية. نحن غير مسؤولين عن أي سوء فهم أو تفسيرات خاطئة تنشأ عن استخدام هذه الترجمة.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**إخلاء المسؤولية**: \nتم ترجمة هذا المستند باستخدام خدمة الترجمة بالذكاء الاصطناعي [Co-op Translator](https://github.com/Azure/co-op-translator). بينما نسعى لتحقيق الدقة، يرجى العلم أن الترجمات الآلية قد تحتوي على أخطاء أو عدم دقة. يجب اعتبار المستند الأصلي بلغته الأصلية المصدر الرسمي. للحصول على معلومات حاسمة، يُوصى بالترجمة البشرية الاحترافية. نحن غير مسؤولين عن أي سوء فهم أو تفسيرات خاطئة تنشأ عن استخدام هذه الترجمة.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:50:36+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "ar" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:50:36+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "ar" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/bg/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/bg/lessons/2-Symbolic/MSConceptGraph.ipynb index 44870118..5a404095 100644 --- a/translations/bg/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/bg/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - Този API ВЕЧЕ НЕ Е НАЛИЧЕН, НО МОЛЯ, ВИЖТЕ ТЕТРАДКАТА, ЗА ДА РАЗБЕРЕТЕ КОНЦЕПЦИЯТА\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) е голяма таксономия от термини, извлечени от интернет, с `is-a` връзки между концепции.\n", - "\n", - "Context Graph е наличен в две форми:\n", - " * Голям текстов файл за изтегляне\n", - " * REST API\n", - "\n", - "Статистика:\n", - " * 5401933 уникални концепции,\n", - " * 12551613 уникални инстанции\n", - " * 87603947 `is-a` връзки\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Използване на уеб услуга\n", - "\n", - "Уеб услугата предлага различни заявки за оценка на вероятността дадена концепция да принадлежи към различни групи. Повече информация е налична [тук](https://concept.research.microsoft.com/Home/Api). \n", - "Ето примерен URL за заявка: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - Този API ВЕЧЕ НЕ Е НАЛИЧЕН, НО МОЛЯ, ВИЖТЕ ТЕТРАДКАТА, ЗА ДА РАЗБЕРЕТЕ КОНЦЕПЦИЯТА\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) е голяма таксономия от термини, извлечени от интернет, с `is-a` връзки между концепции.\n", + "\n", + "Context Graph е наличен в две форми:\n", + " * Голям текстов файл за изтегляне\n", + " * REST API\n", + "\n", + "Статистика:\n", + " * 5401933 уникални концепции,\n", + " * 12551613 уникални инстанции\n", + " * 87603947 `is-a` връзки\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Нека се опитаме да категоризираме заглавията на новините, използвайки основни концепции. За да получим заглавия на новини, ще използваме услугата [NewsApi.org](http://newsapi.org). Трябва да получите собствен API ключ, за да използвате услугата - отидете на уебсайта и се регистрирайте за безплатен план за разработчици.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Първо, искаме да можем да извличаме съществителни от заглавия на новини. Ще използваме библиотеката `TextBlob` за тази цел, която опростява много типични задачи в обработката на естествен език като тази.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Използване на уеб услуга\n", + "\n", + "Уеб услугата предлага различни заявки за оценка на вероятността дадена концепция да принадлежи към различни групи. Повече информация е налична [тук](https://concept.research.microsoft.com/Home/Api). \n", + "Ето примерен URL за заявка: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Можем да видим, че съществителните не ни дават големи тематични групи. Нека заменим съществителните с по-общи термини, получени от графа на концепциите. Това ще отнеме известно време, защото правим REST заявка за всяка съществителна фраза.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Нека се опитаме да категоризираме заглавията на новините, използвайки основни концепции. За да получим заглавия на новини, ще използваме услугата [NewsApi.org](http://newsapi.org). Трябва да получите собствен API ключ, за да използвате услугата - отидете на уебсайта и се регистрирайте за безплатен план за разработчици.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Първо, искаме да можем да извличаме съществителни от заглавия на новини. Ще използваме библиотеката `TextBlob` за тази цел, която опростява много типични задачи в обработката на естествен език като тази.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Можем да видим, че съществителните не ни дават големи тематични групи. Нека заменим съществителните с по-общи термини, получени от графа на концепциите. Това ще отнеме известно време, защото правим REST заявка за всяка съществителна фраза.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**Отказ от отговорност**: \nТози документ е преведен с помощта на AI услуга за превод [Co-op Translator](https://github.com/Azure/co-op-translator). Въпреки че се стремим към точност, моля, имайте предвид, че автоматизираните преводи може да съдържат грешки или неточности. Оригиналният документ на неговия роден език трябва да се счита за авторитетен източник. За критична информация се препоръчва професионален човешки превод. Ние не носим отговорност за каквито и да е недоразумения или погрешни интерпретации, произтичащи от използването на този превод.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**Отказ от отговорност**: \nТози документ е преведен с помощта на AI услуга за превод [Co-op Translator](https://github.com/Azure/co-op-translator). Въпреки че се стремим към точност, моля, имайте предвид, че автоматизираните преводи може да съдържат грешки или неточности. Оригиналният документ на неговия роден език трябва да се счита за авторитетен източник. За критична информация се препоръчва професионален човешки превод. Ние не носим отговорност за каквито и да е недоразумения или погрешни интерпретации, произтичащи от използването на този превод.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:58:47+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "bg" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:58:47+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "bg" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/bn/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/bn/lessons/2-Symbolic/MSConceptGraph.ipynb index f16bb1a1..312fa22e 100644 --- a/translations/bn/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/bn/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - এই API আর উপলব্ধ নেই, তবে ধারণাটি বুঝতে নোটবুকটি দেখুন\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) হলো ইন্টারনেট থেকে সংগৃহীত একটি বৃহৎ ট্যাক্সোনমি, যেখানে ধারণাগুলোর মধ্যে `is-a` সম্পর্ক রয়েছে।\n", - "\n", - "Context Graph দুটি রূপে পাওয়া যায়:\n", - " * ডাউনলোডের জন্য বড় টেক্সট ফাইল\n", - " * REST API\n", - "\n", - "পরিসংখ্যান:\n", - " * 5401933টি অনন্য ধারণা,\n", - " * 12551613টি অনন্য উদাহরণ\n", - " * 87603947টি `is-a` সম্পর্ক\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## ওয়েব সার্ভিস ব্যবহার করা\n", - "\n", - "ওয়েব সার্ভিস বিভিন্ন কল অফার করে যা একটি ধারণার বিভিন্ন গ্রুপে অন্তর্ভুক্ত হওয়ার সম্ভাবনা অনুমান করতে সাহায্য করে। আরও তথ্য পাওয়া যাবে [এখানে](https://concept.research.microsoft.com/Home/Api)। \n", - "এখানে একটি নমুনা URL দেওয়া হলো: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - এই API আর উপলব্ধ নেই, তবে ধারণাটি বুঝতে নোটবুকটি দেখুন\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) হলো ইন্টারনেট থেকে সংগৃহীত একটি বৃহৎ ট্যাক্সোনমি, যেখানে ধারণাগুলোর মধ্যে `is-a` সম্পর্ক রয়েছে।\n", + "\n", + "Context Graph দুটি রূপে পাওয়া যায়:\n", + " * ডাউনলোডের জন্য বড় টেক্সট ফাইল\n", + " * REST API\n", + "\n", + "পরিসংখ্যান:\n", + " * 5401933টি অনন্য ধারণা,\n", + " * 12551613টি অনন্য উদাহরণ\n", + " * 87603947টি `is-a` সম্পর্ক\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "চলুন খবরের শিরোনামগুলোকে মূল ধারণার ভিত্তিতে শ্রেণীবদ্ধ করার চেষ্টা করি। খবরের শিরোনাম পেতে, আমরা [NewsApi.org](http://newsapi.org) সেবাটি ব্যবহার করব। এই সেবাটি ব্যবহার করতে হলে আপনাকে নিজের API কী সংগ্রহ করতে হবে - ওয়েবসাইটে যান এবং বিনামূল্যে ডেভেলপার প্ল্যানে নিবন্ধন করুন।\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "প্রথমত, আমরা চাই সংবাদ শিরোনাম থেকে বিশেষ্য শব্দগুলি বের করতে সক্ষম হতে। আমরা এটি করার জন্য `TextBlob` লাইব্রেরি ব্যবহার করব, যা এই ধরনের সাধারণ NLP কাজগুলোকে অনেক সহজ করে তোলে।\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## ওয়েব সার্ভিস ব্যবহার করা\n", + "\n", + "ওয়েব সার্ভিস বিভিন্ন কল অফার করে যা একটি ধারণার বিভিন্ন গ্রুপে অন্তর্ভুক্ত হওয়ার সম্ভাবনা অনুমান করতে সাহায্য করে। আরও তথ্য পাওয়া যাবে [এখানে](https://concept.research.microsoft.com/Home/Api)। \n", + "এখানে একটি নমুনা URL দেওয়া হলো: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "আমরা দেখতে পাচ্ছি যে বিশেষ্যগুলি আমাদের বড় থিম্যাটিক গ্রুপ দেয় না। চলুন বিশেষ্যগুলিকে ধারণা গ্রাফ থেকে প্রাপ্ত আরও সাধারণ শব্দ দিয়ে প্রতিস্থাপন করি। এটি কিছুটা সময় নেবে, কারণ আমরা প্রতিটি বিশেষ্য বাক্যাংশের জন্য REST কল করছি।\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "চলুন খবরের শিরোনামগুলোকে মূল ধারণার ভিত্তিতে শ্রেণীবদ্ধ করার চেষ্টা করি। খবরের শিরোনাম পেতে, আমরা [NewsApi.org](http://newsapi.org) সেবাটি ব্যবহার করব। এই সেবাটি ব্যবহার করতে হলে আপনাকে নিজের API কী সংগ্রহ করতে হবে - ওয়েবসাইটে যান এবং বিনামূল্যে ডেভেলপার প্ল্যানে নিবন্ধন করুন।\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "প্রথমত, আমরা চাই সংবাদ শিরোনাম থেকে বিশেষ্য শব্দগুলি বের করতে সক্ষম হতে। আমরা এটি করার জন্য `TextBlob` লাইব্রেরি ব্যবহার করব, যা এই ধরনের সাধারণ NLP কাজগুলোকে অনেক সহজ করে তোলে।\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "আমরা দেখতে পাচ্ছি যে বিশেষ্যগুলি আমাদের বড় থিম্যাটিক গ্রুপ দেয় না। চলুন বিশেষ্যগুলিকে ধারণা গ্রাফ থেকে প্রাপ্ত আরও সাধারণ শব্দ দিয়ে প্রতিস্থাপন করি। এটি কিছুটা সময় নেবে, কারণ আমরা প্রতিটি বিশেষ্য বাক্যাংশের জন্য REST কল করছি।\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**অস্বীকৃতি**: \nএই নথিটি AI অনুবাদ পরিষেবা [Co-op Translator](https://github.com/Azure/co-op-translator) ব্যবহার করে অনুবাদ করা হয়েছে। আমরা যথাসাধ্য সঠিকতার জন্য চেষ্টা করি, তবে অনুগ্রহ করে মনে রাখবেন যে স্বয়ংক্রিয় অনুবাদে ত্রুটি বা অসঙ্গতি থাকতে পারে। মূল ভাষায় থাকা নথিটিকে প্রামাণিক উৎস হিসেবে বিবেচনা করা উচিত। গুরুত্বপূর্ণ তথ্যের জন্য, পেশাদার মানব অনুবাদ সুপারিশ করা হয়। এই অনুবাদ ব্যবহারের ফলে কোনো ভুল বোঝাবুঝি বা ভুল ব্যাখ্যা হলে আমরা দায়বদ্ধ থাকব না।\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**অস্বীকৃতি**: \nএই নথিটি AI অনুবাদ পরিষেবা [Co-op Translator](https://github.com/Azure/co-op-translator) ব্যবহার করে অনুবাদ করা হয়েছে। আমরা যথাসাধ্য সঠিকতার জন্য চেষ্টা করি, তবে অনুগ্রহ করে মনে রাখবেন যে স্বয়ংক্রিয় অনুবাদে ত্রুটি বা অসঙ্গতি থাকতে পারে। মূল ভাষায় থাকা নথিটিকে প্রামাণিক উৎস হিসেবে বিবেচনা করা উচিত। গুরুত্বপূর্ণ তথ্যের জন্য, পেশাদার মানব অনুবাদ সুপারিশ করা হয়। এই অনুবাদ ব্যবহারের ফলে কোনো ভুল বোঝাবুঝি বা ভুল ব্যাখ্যা হলে আমরা দায়বদ্ধ থাকব না।\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:52:50+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "bn" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:52:50+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "bn" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/cs/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/cs/lessons/2-Symbolic/MSConceptGraph.ipynb index fd6e22df..b1973927 100644 --- a/translations/cs/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/cs/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - TENTO API JIŽ NENÍ DOSTUPNÝ, ALE PODÍVEJTE SE NA NOTEBOOK PRO POCHOPENÍ KONCEPTU\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) je rozsáhlá taxonomie termínů získaných z internetu, která obsahuje vztahy typu `is-a` mezi koncepty.\n", - "\n", - "Context Graph je dostupný ve dvou formách:\n", - " * Velký textový soubor ke stažení\n", - " * REST API\n", - "\n", - "Statistiky:\n", - " * 5401933 unikátních konceptů,\n", - " * 12551613 unikátních instancí,\n", - " * 87603947 vztahů typu `is-a`\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Použití webové služby\n", - "\n", - "Webová služba nabízí různé volání pro odhad pravděpodobnosti, že koncept patří do různých skupin. Více informací je dostupných [zde](https://concept.research.microsoft.com/Home/Api). \n", - "Zde je ukázková URL pro volání: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - TENTO API JIŽ NENÍ DOSTUPNÝ, ALE PODÍVEJTE SE NA NOTEBOOK PRO POCHOPENÍ KONCEPTU\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) je rozsáhlá taxonomie termínů získaných z internetu, která obsahuje vztahy typu `is-a` mezi koncepty.\n", + "\n", + "Context Graph je dostupný ve dvou formách:\n", + " * Velký textový soubor ke stažení\n", + " * REST API\n", + "\n", + "Statistiky:\n", + " * 5401933 unikátních konceptů,\n", + " * 12551613 unikátních instancí,\n", + " * 87603947 vztahů typu `is-a`\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Pojďme se pokusit kategorizovat názvy zpráv pomocí nadřazených konceptů. Pro získání názvů zpráv použijeme službu [NewsApi.org](http://newsapi.org). Abyste mohli službu používat, musíte získat vlastní API klíč - přejděte na webovou stránku a zaregistrujte se pro bezplatný plán pro vývojáře.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Nejprve chceme být schopni extrahovat podstatná jména z nadpisů zpráv. K tomu použijeme knihovnu `TextBlob`, která zjednodušuje mnoho typických úkolů NLP, jako je tento.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Použití webové služby\n", + "\n", + "Webová služba nabízí různé volání pro odhad pravděpodobnosti, že koncept patří do různých skupin. Více informací je dostupných [zde](https://concept.research.microsoft.com/Home/Api). \n", + "Zde je ukázková URL pro volání: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Vidíme, že podstatná jména nám neposkytují velké tematické skupiny. Nahraďme podstatná jména obecnějšími termíny získanými z konceptuálního grafu. To zabere nějaký čas, protože provádíme REST volání pro každou podstatnou frázi.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Pojďme se pokusit kategorizovat názvy zpráv pomocí nadřazených konceptů. Pro získání názvů zpráv použijeme službu [NewsApi.org](http://newsapi.org). Abyste mohli službu používat, musíte získat vlastní API klíč - přejděte na webovou stránku a zaregistrujte se pro bezplatný plán pro vývojáře.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Nejprve chceme být schopni extrahovat podstatná jména z nadpisů zpráv. K tomu použijeme knihovnu `TextBlob`, která zjednodušuje mnoho typických úkolů NLP, jako je tento.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Vidíme, že podstatná jména nám neposkytují velké tematické skupiny. Nahraďme podstatná jména obecnějšími termíny získanými z konceptuálního grafu. To zabere nějaký čas, protože provádíme REST volání pro každou podstatnou frázi.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**Prohlášení**: \nTento dokument byl přeložen pomocí služby AI pro překlady [Co-op Translator](https://github.com/Azure/co-op-translator). I když se snažíme o přesnost, mějte prosím na paměti, že automatizované překlady mohou obsahovat chyby nebo nepřesnosti. Původní dokument v jeho původním jazyce by měl být považován za autoritativní zdroj. Pro důležité informace doporučujeme profesionální lidský překlad. Neodpovídáme za žádná nedorozumění nebo nesprávné interpretace vyplývající z použití tohoto překladu.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**Prohlášení**: \nTento dokument byl přeložen pomocí služby AI pro překlady [Co-op Translator](https://github.com/Azure/co-op-translator). I když se snažíme o přesnost, mějte prosím na paměti, že automatizované překlady mohou obsahovat chyby nebo nepřesnosti. Původní dokument v jeho původním jazyce by měl být považován za autoritativní zdroj. Pro důležité informace doporučujeme profesionální lidský překlad. Neodpovídáme za žádná nedorozumění nebo nesprávné interpretace vyplývající z použití tohoto překladu.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:58:06+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "cs" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:58:06+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "cs" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/da/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/da/lessons/2-Symbolic/MSConceptGraph.ipynb index 0cc86692..228dde1b 100644 --- a/translations/da/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/da/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - Denne API ER IKKE LÆNGERE TILGÆNGELIG, MEN SE VENLIGST NOTEBOGEN FOR AT FORSTÅ KONCEPTET\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) er en stor taksonomi af termer udvundet fra internettet, med `is-a` relationer mellem koncepter.\n", - "\n", - "Context Graph er tilgængelig i to former:\n", - " * Stor tekstfil til download\n", - " * REST API\n", - "\n", - "Statistikker:\n", - " * 5401933 unikke koncepter,\n", - " * 12551613 unikke instanser\n", - " * 87603947 `is-a` relationer\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Brug af webtjeneste\n", - "\n", - "Webtjenesten tilbyder forskellige kald for at estimere sandsynligheden for, at et koncept tilhører forskellige grupper. Mere info er tilgængelig [her](https://concept.research.microsoft.com/Home/Api).\n", - "Her er et eksempel på en URL til kald: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - Denne API ER IKKE LÆNGERE TILGÆNGELIG, MEN SE VENLIGST NOTEBOGEN FOR AT FORSTÅ KONCEPTET\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) er en stor taksonomi af termer udvundet fra internettet, med `is-a` relationer mellem koncepter.\n", + "\n", + "Context Graph er tilgængelig i to former:\n", + " * Stor tekstfil til download\n", + " * REST API\n", + "\n", + "Statistikker:\n", + " * 5401933 unikke koncepter,\n", + " * 12551613 unikke instanser\n", + " * 87603947 `is-a` relationer\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Lad os prøve at kategorisere nyhedstitler ved hjælp af overordnede begreber. For at få nyhedstitler vil vi bruge [NewsApi.org](http://newsapi.org) tjenesten. Du skal skaffe din egen API-nøgle for at bruge tjenesten - gå til hjemmesiden og registrer dig for en gratis udviklerplan.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Først og fremmest ønsker vi at kunne udtrække substantiver fra nyhedstitler. Vi vil bruge `TextBlob`-biblioteket til dette, hvilket forenkler mange typiske NLP-opgaver som denne.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Brug af webtjeneste\n", + "\n", + "Webtjenesten tilbyder forskellige kald for at estimere sandsynligheden for, at et koncept tilhører forskellige grupper. Mere info er tilgængelig [her](https://concept.research.microsoft.com/Home/Api).\n", + "Her er et eksempel på en URL til kald: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Vi kan se, at substantiver ikke giver os store tematiske grupper. Lad os erstatte substantiver med mere generelle termer, der er opnået fra konceptgrafen. Dette vil tage noget tid, fordi vi foretager REST-opkald for hver substantivfrase.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Lad os prøve at kategorisere nyhedstitler ved hjælp af overordnede begreber. For at få nyhedstitler vil vi bruge [NewsApi.org](http://newsapi.org) tjenesten. Du skal skaffe din egen API-nøgle for at bruge tjenesten - gå til hjemmesiden og registrer dig for en gratis udviklerplan.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Først og fremmest ønsker vi at kunne udtrække substantiver fra nyhedstitler. Vi vil bruge `TextBlob`-biblioteket til dette, hvilket forenkler mange typiske NLP-opgaver som denne.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Vi kan se, at substantiver ikke giver os store tematiske grupper. Lad os erstatte substantiver med mere generelle termer, der er opnået fra konceptgrafen. Dette vil tage noget tid, fordi vi foretager REST-opkald for hver substantivfrase.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**Ansvarsfraskrivelse**: \nDette dokument er blevet oversat ved hjælp af AI-oversættelsestjenesten [Co-op Translator](https://github.com/Azure/co-op-translator). Selvom vi bestræber os på nøjagtighed, skal det bemærkes, at automatiserede oversættelser kan indeholde fejl eller unøjagtigheder. Det originale dokument på dets oprindelige sprog bør betragtes som den autoritative kilde. For kritisk information anbefales professionel menneskelig oversættelse. Vi påtager os ikke ansvar for eventuelle misforståelser eller fejltolkninger, der måtte opstå som følge af brugen af denne oversættelse.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**Ansvarsfraskrivelse**: \nDette dokument er blevet oversat ved hjælp af AI-oversættelsestjenesten [Co-op Translator](https://github.com/Azure/co-op-translator). Selvom vi bestræber os på nøjagtighed, skal det bemærkes, at automatiserede oversættelser kan indeholde fejl eller unøjagtigheder. Det originale dokument på dets oprindelige sprog bør betragtes som den autoritative kilde. For kritisk information anbefales professionel menneskelig oversættelse. Vi påtager os ikke ansvar for eventuelle misforståelser eller fejltolkninger, der måtte opstå som følge af brugen af denne oversættelse.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:55:37+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "da" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:55:37+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "da" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/de/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/de/lessons/2-Symbolic/MSConceptGraph.ipynb index dbca6841..a901d824 100644 --- a/translations/de/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/de/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - Diese API IST NICHT MEHR VERFÜGBAR, ABER BITTE SCHAUEN SIE SICH DAS NOTEBOOK AN, UM DAS KONZEPT ZU VERSTEHEN\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) ist eine umfangreiche Taxonomie von Begriffen, die aus dem Internet extrahiert wurden, mit `is-a`-Beziehungen zwischen Konzepten.\n", - "\n", - "Der Context Graph ist in zwei Formen verfügbar:\n", - " * Große Textdatei zum Herunterladen\n", - " * REST-API\n", - "\n", - "Statistiken:\n", - " * 5401933 einzigartige Konzepte\n", - " * 12551613 einzigartige Instanzen\n", - " * 87603947 `is-a`-Beziehungen\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Verwendung des Webdienstes\n", - "\n", - "Der Webdienst bietet verschiedene Aufrufe, um die Wahrscheinlichkeit eines Konzepts zu schätzen, zu unterschiedlichen Gruppen zu gehören. Weitere Informationen finden Sie [hier](https://concept.research.microsoft.com/Home/Api). \n", - "Hier ist die Beispiel-URL für einen Aufruf: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - Diese API IST NICHT MEHR VERFÜGBAR, ABER BITTE SCHAUEN SIE SICH DAS NOTEBOOK AN, UM DAS KONZEPT ZU VERSTEHEN\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) ist eine umfangreiche Taxonomie von Begriffen, die aus dem Internet extrahiert wurden, mit `is-a`-Beziehungen zwischen Konzepten.\n", + "\n", + "Der Context Graph ist in zwei Formen verfügbar:\n", + " * Große Textdatei zum Herunterladen\n", + " * REST-API\n", + "\n", + "Statistiken:\n", + " * 5401933 einzigartige Konzepte\n", + " * 12551613 einzigartige Instanzen\n", + " * 87603947 `is-a`-Beziehungen\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Lassen Sie uns versuchen, die Nachrichtentitel anhand von übergeordneten Konzepten zu kategorisieren. Um Nachrichtentitel zu erhalten, werden wir den Dienst [NewsApi.org](http://newsapi.org) nutzen. Sie müssen Ihren eigenen API-Schlüssel erhalten, um den Dienst nutzen zu können - gehen Sie auf die Website und registrieren Sie sich für den kostenlosen Entwicklerplan.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Zunächst möchten wir in der Lage sein, Nomen aus Nachrichtentiteln zu extrahieren. Dafür werden wir die Bibliothek `TextBlob` verwenden, die viele typische NLP-Aufgaben wie diese vereinfacht.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Verwendung des Webdienstes\n", + "\n", + "Der Webdienst bietet verschiedene Aufrufe, um die Wahrscheinlichkeit eines Konzepts zu schätzen, zu unterschiedlichen Gruppen zu gehören. Weitere Informationen finden Sie [hier](https://concept.research.microsoft.com/Home/Api). \n", + "Hier ist die Beispiel-URL für einen Aufruf: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Wir können sehen, dass Substantive uns keine großen thematischen Gruppen liefern. Lassen Sie uns Substantive durch allgemeinere Begriffe ersetzen, die aus dem Konzeptgraphen gewonnen wurden. Dies wird einige Zeit in Anspruch nehmen, da wir für jede Substantivphrase einen REST-Aufruf durchführen.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Lassen Sie uns versuchen, die Nachrichtentitel anhand von übergeordneten Konzepten zu kategorisieren. Um Nachrichtentitel zu erhalten, werden wir den Dienst [NewsApi.org](http://newsapi.org) nutzen. Sie müssen Ihren eigenen API-Schlüssel erhalten, um den Dienst nutzen zu können - gehen Sie auf die Website und registrieren Sie sich für den kostenlosen Entwicklerplan.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Zunächst möchten wir in der Lage sein, Nomen aus Nachrichtentiteln zu extrahieren. Dafür werden wir die Bibliothek `TextBlob` verwenden, die viele typische NLP-Aufgaben wie diese vereinfacht.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Wir können sehen, dass Substantive uns keine großen thematischen Gruppen liefern. Lassen Sie uns Substantive durch allgemeinere Begriffe ersetzen, die aus dem Konzeptgraphen gewonnen wurden. Dies wird einige Zeit in Anspruch nehmen, da wir für jede Substantivphrase einen REST-Aufruf durchführen.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**Haftungsausschluss**: \nDieses Dokument wurde mit dem KI-Übersetzungsdienst [Co-op Translator](https://github.com/Azure/co-op-translator) übersetzt. Obwohl wir uns um Genauigkeit bemühen, beachten Sie bitte, dass automatisierte Übersetzungen Fehler oder Ungenauigkeiten enthalten können. Das Originaldokument in seiner ursprünglichen Sprache sollte als maßgebliche Quelle betrachtet werden. Für kritische Informationen wird eine professionelle menschliche Übersetzung empfohlen. Wir übernehmen keine Haftung für Missverständnisse oder Fehlinterpretationen, die sich aus der Nutzung dieser Übersetzung ergeben.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**Haftungsausschluss**: \nDieses Dokument wurde mit dem KI-Übersetzungsdienst [Co-op Translator](https://github.com/Azure/co-op-translator) übersetzt. Obwohl wir uns um Genauigkeit bemühen, beachten Sie bitte, dass automatisierte Übersetzungen Fehler oder Ungenauigkeiten enthalten können. Das Originaldokument in seiner ursprünglichen Sprache sollte als maßgebliche Quelle betrachtet werden. Für kritische Informationen wird eine professionelle menschliche Übersetzung empfohlen. Wir übernehmen keine Haftung für Missverständnisse oder Fehlinterpretationen, die sich aus der Nutzung dieser Übersetzung ergeben.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:50:11+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "de" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:50:11+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "de" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/el/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/el/lessons/2-Symbolic/MSConceptGraph.ipynb index 5daa514a..6ff90bd2 100644 --- a/translations/el/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/el/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - ΑΥΤΟ ΤΟ API ΔΕΝ ΕΙΝΑΙ ΠΛΕΟΝ ΔΙΑΘΕΣΙΜΟ, ΑΛΛΑ ΠΑΡΑΚΑΛΩ ΔΕΙΤΕ ΤΟ NOTEBOOK ΓΙΑ ΝΑ ΚΑΤΑΝΟΗΣΕΤΕ ΤΗΝ ΕΝΝΟΙΑ\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) είναι μια μεγάλη ταξινόμηση όρων που έχουν εξορυχθεί από το διαδίκτυο, με σχέσεις `is-a` μεταξύ εννοιών.\n", - "\n", - "Το Context Graph είναι διαθέσιμο σε δύο μορφές:\n", - " * Μεγάλο αρχείο κειμένου για λήψη\n", - " * REST API\n", - "\n", - "Στατιστικά:\n", - " * 5401933 μοναδικές έννοιες, \n", - " * 12551613 μοναδικές περιπτώσεις\n", - " * 87603947 σχέσεις `is-a`\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Χρήση Υπηρεσίας Ιστού\n", - "\n", - "Η υπηρεσία ιστού προσφέρει διάφορες κλήσεις για την εκτίμηση της πιθανότητας ενός εννοιολογικού στοιχείου να ανήκει σε διαφορετικές ομάδες. Περισσότερες πληροφορίες είναι διαθέσιμες [εδώ](https://concept.research.microsoft.com/Home/Api). \n", - "Ακολουθεί ένα δείγμα URL για κλήση: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - ΑΥΤΟ ΤΟ API ΔΕΝ ΕΙΝΑΙ ΠΛΕΟΝ ΔΙΑΘΕΣΙΜΟ, ΑΛΛΑ ΠΑΡΑΚΑΛΩ ΔΕΙΤΕ ΤΟ NOTEBOOK ΓΙΑ ΝΑ ΚΑΤΑΝΟΗΣΕΤΕ ΤΗΝ ΕΝΝΟΙΑ\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) είναι μια μεγάλη ταξινόμηση όρων που έχουν εξορυχθεί από το διαδίκτυο, με σχέσεις `is-a` μεταξύ εννοιών.\n", + "\n", + "Το Context Graph είναι διαθέσιμο σε δύο μορφές:\n", + " * Μεγάλο αρχείο κειμένου για λήψη\n", + " * REST API\n", + "\n", + "Στατιστικά:\n", + " * 5401933 μοναδικές έννοιες, \n", + " * 12551613 μοναδικές περιπτώσεις\n", + " * 87603947 σχέσεις `is-a`\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Ας προσπαθήσουμε να κατηγοριοποιήσουμε τους τίτλους ειδήσεων χρησιμοποιώντας γονικές έννοιες. Για να πάρουμε τους τίτλους ειδήσεων, θα χρησιμοποιήσουμε την υπηρεσία [NewsApi.org](http://newsapi.org). Πρέπει να αποκτήσετε το δικό σας API key για να χρησιμοποιήσετε την υπηρεσία - επισκεφθείτε τον ιστότοπο και εγγραφείτε στο δωρεάν πρόγραμμα για προγραμματιστές.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Πρώτα απ' όλα, θέλουμε να μπορούμε να εξάγουμε ουσιαστικά από τίτλους ειδήσεων. Θα χρησιμοποιήσουμε τη βιβλιοθήκη `TextBlob` για να το κάνουμε αυτό, η οποία απλοποιεί πολλές τυπικές εργασίες NLP όπως αυτή.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Χρήση Υπηρεσίας Ιστού\n", + "\n", + "Η υπηρεσία ιστού προσφέρει διάφορες κλήσεις για την εκτίμηση της πιθανότητας ενός εννοιολογικού στοιχείου να ανήκει σε διαφορετικές ομάδες. Περισσότερες πληροφορίες είναι διαθέσιμες [εδώ](https://concept.research.microsoft.com/Home/Api). \n", + "Ακολουθεί ένα δείγμα URL για κλήση: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Μπορούμε να δούμε ότι τα ουσιαστικά δεν μας δίνουν μεγάλες θεματικές ομάδες. Ας αντικαταστήσουμε τα ουσιαστικά με πιο γενικούς όρους που προκύπτουν από το γράφημα εννοιών. Αυτό θα πάρει λίγο χρόνο, επειδή κάνουμε κλήση REST για κάθε φράση ουσιαστικού.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Ας προσπαθήσουμε να κατηγοριοποιήσουμε τους τίτλους ειδήσεων χρησιμοποιώντας γονικές έννοιες. Για να πάρουμε τους τίτλους ειδήσεων, θα χρησιμοποιήσουμε την υπηρεσία [NewsApi.org](http://newsapi.org). Πρέπει να αποκτήσετε το δικό σας API key για να χρησιμοποιήσετε την υπηρεσία - επισκεφθείτε τον ιστότοπο και εγγραφείτε στο δωρεάν πρόγραμμα για προγραμματιστές.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Πρώτα απ' όλα, θέλουμε να μπορούμε να εξάγουμε ουσιαστικά από τίτλους ειδήσεων. Θα χρησιμοποιήσουμε τη βιβλιοθήκη `TextBlob` για να το κάνουμε αυτό, η οποία απλοποιεί πολλές τυπικές εργασίες NLP όπως αυτή.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Μπορούμε να δούμε ότι τα ουσιαστικά δεν μας δίνουν μεγάλες θεματικές ομάδες. Ας αντικαταστήσουμε τα ουσιαστικά με πιο γενικούς όρους που προκύπτουν από το γράφημα εννοιών. Αυτό θα πάρει λίγο χρόνο, επειδή κάνουμε κλήση REST για κάθε φράση ουσιαστικού.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**Αποποίηση ευθύνης**: \nΑυτό το έγγραφο έχει μεταφραστεί χρησιμοποιώντας την υπηρεσία αυτόματης μετάφρασης [Co-op Translator](https://github.com/Azure/co-op-translator). Παρόλο που καταβάλλουμε προσπάθειες για ακρίβεια, παρακαλούμε να έχετε υπόψη ότι οι αυτόματες μεταφράσεις ενδέχεται να περιέχουν λάθη ή ανακρίβειες. Το πρωτότυπο έγγραφο στη γλώσσα του θα πρέπει να θεωρείται η αυθεντική πηγή. Για κρίσιμες πληροφορίες, συνιστάται επαγγελματική ανθρώπινη μετάφραση. Δεν φέρουμε ευθύνη για τυχόν παρεξηγήσεις ή εσφαλμένες ερμηνείες που προκύπτουν από τη χρήση αυτής της μετάφρασης.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**Αποποίηση ευθύνης**: \nΑυτό το έγγραφο έχει μεταφραστεί χρησιμοποιώντας την υπηρεσία αυτόματης μετάφρασης [Co-op Translator](https://github.com/Azure/co-op-translator). Παρόλο που καταβάλλουμε προσπάθειες για ακρίβεια, παρακαλούμε να έχετε υπόψη ότι οι αυτόματες μεταφράσεις ενδέχεται να περιέχουν λάθη ή ανακρίβειες. Το πρωτότυπο έγγραφο στη γλώσσα του θα πρέπει να θεωρείται η αυθεντική πηγή. Για κρίσιμες πληροφορίες, συνιστάται επαγγελματική ανθρώπινη μετάφραση. Δεν φέρουμε ευθύνη για τυχόν παρεξηγήσεις ή εσφαλμένες ερμηνείες που προκύπτουν από τη χρήση αυτής της μετάφρασης.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:54:56+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "el" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:54:56+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "el" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/en/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/en/lessons/2-Symbolic/MSConceptGraph.ipynb index 9e985f13..69ad6ba5 100644 --- a/translations/en/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/en/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - This API IS NO LONGER AVAILABLE BUT PLEASE LOOK AT THE NOTEBOOK TO UNDERSTAND THE CONCEPT\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) is a comprehensive taxonomy of terms extracted from the internet, featuring `is-a` relationships between concepts.\n", - "\n", - "The Concept Graph is accessible in two formats:\n", - " * A large text file available for download\n", - " * REST API\n", - "\n", - "Statistics:\n", - " * 5,401,933 unique concepts\n", - " * 12,551,613 unique instances\n", - " * 87,603,947 `is-a` relationships\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Using Web Service\n", - "\n", - "The web service provides various calls to estimate the probability of a concept belonging to different groups. More information can be found [here](https://concept.research.microsoft.com/Home/Api). \n", - "Below is an example URL for making a call: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - This API IS NO LONGER AVAILABLE BUT PLEASE LOOK AT THE NOTEBOOK TO UNDERSTAND THE CONCEPT\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) is a comprehensive taxonomy of terms extracted from the internet, featuring `is-a` relationships between concepts.\n", + "\n", + "The Concept Graph is accessible in two formats:\n", + " * A large text file available for download\n", + " * REST API\n", + "\n", + "Statistics:\n", + " * 5,401,933 unique concepts\n", + " * 12,551,613 unique instances\n", + " * 87,603,947 `is-a` relationships\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Let's try to categorize the news titles using parent concepts. To get news titles, we will use [NewsApi.org](http://newsapi.org) service. You need to obtain your own API key in order to use the service - go to the website and register for a free developer plan.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "First of all, we want to be able to extract nouns from news titles. We will use the `TextBlob` library to do this, which simplifies a lot of typical NLP tasks like this.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Using Web Service\n", + "\n", + "The web service provides various calls to estimate the probability of a concept belonging to different groups. More information can be found [here](https://concept.research.microsoft.com/Home/Api). \n", + "Below is an example URL for making a call: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can see that nouns do not provide us with large thematic groups. Let's replace nouns with more general terms derived from the concept graph. This will take some time, as we are making REST calls for each noun phrase.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's try to categorize the news titles using parent concepts. To get news titles, we will use [NewsApi.org](http://newsapi.org) service. You need to obtain your own API key in order to use the service - go to the website and register for a free developer plan.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "First of all, we want to be able to extract nouns from news titles. We will use the `TextBlob` library to do this, which simplifies a lot of typical NLP tasks like this.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can see that nouns do not provide us with large thematic groups. Let's replace nouns with more general terms derived from the concept graph. This will take some time, as we are making REST calls for each noun phrase.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**Disclaimer**: \nThis document has been translated using the AI translation service [Co-op Translator](https://github.com/Azure/co-op-translator). While we aim for accuracy, please note that automated translations may contain errors or inaccuracies. The original document in its native language should be regarded as the authoritative source. For critical information, professional human translation is recommended. We are not responsible for any misunderstandings or misinterpretations resulting from the use of this translation.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**Disclaimer**: \nThis document has been translated using the AI translation service [Co-op Translator](https://github.com/Azure/co-op-translator). While we aim for accuracy, please note that automated translations may contain errors or inaccuracies. The original document in its native language should be regarded as the authoritative source. For critical information, professional human translation is recommended. We are not responsible for any misunderstandings or misinterpretations resulting from the use of this translation.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:49:33+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "en" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:49:33+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "en" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/es/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/es/lessons/2-Symbolic/MSConceptGraph.ipynb index 171ed027..010a0584 100644 --- a/translations/es/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/es/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - ESTA API YA NO ESTÁ DISPONIBLE, PERO POR FAVOR REVISA EL NOTEBOOK PARA ENTENDER EL CONCEPTO\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) es una gran taxonomía de términos extraídos de internet, con relaciones de tipo `es-un` entre conceptos.\n", - "\n", - "El Context Graph está disponible en dos formas:\n", - " * Archivo de texto grande para descargar\n", - " * API REST\n", - "\n", - "Estadísticas:\n", - " * 5,401,933 conceptos únicos\n", - " * 12,551,613 instancias únicas\n", - " * 87,603,947 relaciones de tipo `es-un`\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Uso del servicio web\n", - "\n", - "El servicio web ofrece diferentes llamadas para estimar la probabilidad de que un concepto pertenezca a distintos grupos. Más información está disponible [aquí](https://concept.research.microsoft.com/Home/Api). \n", - "Aquí tienes un ejemplo de URL para realizar la llamada: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - ESTA API YA NO ESTÁ DISPONIBLE, PERO POR FAVOR REVISA EL NOTEBOOK PARA ENTENDER EL CONCEPTO\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) es una gran taxonomía de términos extraídos de internet, con relaciones de tipo `es-un` entre conceptos.\n", + "\n", + "El Context Graph está disponible en dos formas:\n", + " * Archivo de texto grande para descargar\n", + " * API REST\n", + "\n", + "Estadísticas:\n", + " * 5,401,933 conceptos únicos\n", + " * 12,551,613 instancias únicas\n", + " * 87,603,947 relaciones de tipo `es-un`\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Intentemos categorizar los títulos de noticias utilizando conceptos principales. Para obtener los títulos de noticias, utilizaremos el servicio [NewsApi.org](http://newsapi.org). Necesitas obtener tu propia clave API para usar el servicio: ve al sitio web y regístrate en el plan de desarrollador gratuito.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "En primer lugar, queremos poder extraer sustantivos de los títulos de noticias. Usaremos la biblioteca `TextBlob` para hacer esto, lo que simplifica muchas tareas típicas de procesamiento de lenguaje natural como esta.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Uso del servicio web\n", + "\n", + "El servicio web ofrece diferentes llamadas para estimar la probabilidad de que un concepto pertenezca a distintos grupos. Más información está disponible [aquí](https://concept.research.microsoft.com/Home/Api). \n", + "Aquí tienes un ejemplo de URL para realizar la llamada: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Podemos ver que los sustantivos no nos proporcionan grandes grupos temáticos. Vamos a sustituir los sustantivos por términos más generales obtenidos del gráfico de conceptos. Esto tomará algo de tiempo, porque estamos haciendo una llamada REST para cada frase nominal.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Intentemos categorizar los títulos de noticias utilizando conceptos principales. Para obtener los títulos de noticias, utilizaremos el servicio [NewsApi.org](http://newsapi.org). Necesitas obtener tu propia clave API para usar el servicio: ve al sitio web y regístrate en el plan de desarrollador gratuito.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "En primer lugar, queremos poder extraer sustantivos de los títulos de noticias. Usaremos la biblioteca `TextBlob` para hacer esto, lo que simplifica muchas tareas típicas de procesamiento de lenguaje natural como esta.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Podemos ver que los sustantivos no nos proporcionan grandes grupos temáticos. Vamos a sustituir los sustantivos por términos más generales obtenidos del gráfico de conceptos. Esto tomará algo de tiempo, porque estamos haciendo una llamada REST para cada frase nominal.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**Descargo de responsabilidad**: \nEste documento ha sido traducido utilizando el servicio de traducción automática [Co-op Translator](https://github.com/Azure/co-op-translator). Aunque nos esforzamos por lograr precisión, tenga en cuenta que las traducciones automáticas pueden contener errores o imprecisiones. El documento original en su idioma nativo debe considerarse la fuente autorizada. Para información crítica, se recomienda una traducción profesional realizada por humanos. No nos hacemos responsables de malentendidos o interpretaciones erróneas que surjan del uso de esta traducción.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**Descargo de responsabilidad**: \nEste documento ha sido traducido utilizando el servicio de traducción automática [Co-op Translator](https://github.com/Azure/co-op-translator). Aunque nos esforzamos por lograr precisión, tenga en cuenta que las traducciones automáticas pueden contener errores o imprecisiones. El documento original en su idioma nativo debe considerarse la fuente autorizada. Para información crítica, se recomienda una traducción profesional realizada por humanos. No nos hacemos responsables de malentendidos o interpretaciones erróneas que surjan del uso de esta traducción.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:49:58+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "es" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:49:58+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "es" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/et/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/et/lessons/2-Symbolic/MSConceptGraph.ipynb index afe41515..93269bf3 100644 --- a/translations/et/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/et/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - SEE NOTEBOOK FOR UNDERSTANDING THE CONCEPT, AS THIS API IS NO LONGER AVAILABLE\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) on suur taksonoomia, mis on koostatud internetist kogutud terminitest ja sisaldab `on`-tüüpi seoseid kontseptsioonide vahel.\n", - "\n", - "Context Graph on saadaval kahel kujul:\n", - " * Suur tekstifail allalaadimiseks\n", - " * REST API\n", - "\n", - "Statistika:\n", - " * 5401933 unikaalset kontseptsiooni,\n", - " * 12551613 unikaalset eksemplari\n", - " * 87603947 `on`-tüüpi seost\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Veebiteenuse kasutamine\n", - "\n", - "Veebiteenus pakub erinevaid päringuid, et hinnata tõenäosust, mil määral mõiste kuulub erinevatesse gruppidesse. Rohkem infot on saadaval [siin](https://concept.research.microsoft.com/Home/Api). \n", - "Siin on näidis-URL päringu tegemiseks: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - SEE NOTEBOOK FOR UNDERSTANDING THE CONCEPT, AS THIS API IS NO LONGER AVAILABLE\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) on suur taksonoomia, mis on koostatud internetist kogutud terminitest ja sisaldab `on`-tüüpi seoseid kontseptsioonide vahel.\n", + "\n", + "Context Graph on saadaval kahel kujul:\n", + " * Suur tekstifail allalaadimiseks\n", + " * REST API\n", + "\n", + "Statistika:\n", + " * 5401933 unikaalset kontseptsiooni,\n", + " * 12551613 unikaalset eksemplari\n", + " * 87603947 `on`-tüüpi seost\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Proovime uudiste pealkirju kategoriseerida kasutades üldisemaid mõisteid. Uudiste pealkirjade saamiseks kasutame [NewsApi.org](http://newsapi.org) teenust. Teenuse kasutamiseks pead hankima oma API-võtme – mine veebilehele ja registreeru tasuta arendajaplaanile.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Kõigepealt tahame olla võimelised uudiste pealkirjadest nimisõnu eraldama. Selleks kasutame `TextBlob` teeki, mis lihtsustab paljusid tüüpilisi NLP ülesandeid, nagu see.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Veebiteenuse kasutamine\n", + "\n", + "Veebiteenus pakub erinevaid päringuid, et hinnata tõenäosust, mil määral mõiste kuulub erinevatesse gruppidesse. Rohkem infot on saadaval [siin](https://concept.research.microsoft.com/Home/Api). \n", + "Siin on näidis-URL päringu tegemiseks: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Me näeme, et nimisõnad ei anna meile suuri temaatilisi rühmi. Asendame nimisõnad üldisemate terminitega, mis on saadud kontseptsioonigraafist. See võtab veidi aega, kuna teeme iga nimisõnafraasi jaoks REST-kõne.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Proovime uudiste pealkirju kategoriseerida kasutades üldisemaid mõisteid. Uudiste pealkirjade saamiseks kasutame [NewsApi.org](http://newsapi.org) teenust. Teenuse kasutamiseks pead hankima oma API-võtme – mine veebilehele ja registreeru tasuta arendajaplaanile.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Kõigepealt tahame olla võimelised uudiste pealkirjadest nimisõnu eraldama. Selleks kasutame `TextBlob` teeki, mis lihtsustab paljusid tüüpilisi NLP ülesandeid, nagu see.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Me näeme, et nimisõnad ei anna meile suuri temaatilisi rühmi. Asendame nimisõnad üldisemate terminitega, mis on saadud kontseptsioonigraafist. See võtab veidi aega, kuna teeme iga nimisõnafraasi jaoks REST-kõne.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**Lahtiütlus**: \nSee dokument on tõlgitud AI tõlketeenuse [Co-op Translator](https://github.com/Azure/co-op-translator) abil. Kuigi püüame tagada täpsust, palume arvestada, et automaatsed tõlked võivad sisaldada vigu või ebatäpsusi. Algne dokument selle algses keeles tuleks pidada autoriteetseks allikaks. Olulise teabe puhul soovitame kasutada professionaalset inimtõlget. Me ei vastuta selle tõlke kasutamisest tulenevate arusaamatuste või valesti tõlgenduste eest.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**Lahtiütlus**: \nSee dokument on tõlgitud AI tõlketeenuse [Co-op Translator](https://github.com/Azure/co-op-translator) abil. Kuigi püüame tagada täpsust, palume arvestada, et automaatsed tõlked võivad sisaldada vigu või ebatäpsusi. Algne dokument selle algses keeles tuleks pidada autoriteetseks allikaks. Olulise teabe puhul soovitame kasutada professionaalset inimtõlget. Me ei vastuta selle tõlke kasutamisest tulenevate arusaamatuste või valesti tõlgenduste eest.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-11T12:36:13+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "et" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-11T12:36:13+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "et" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/fa/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/fa/lessons/2-Symbolic/MSConceptGraph.ipynb index 9066d6d4..45cd7e53 100644 --- a/translations/fa/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/fa/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - این API دیگر در دسترس نیست اما لطفاً به دفترچه نگاه کنید تا مفهوم را درک کنید\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) یک طبقه‌بندی بزرگ از اصطلاحات استخراج‌شده از اینترنت است که شامل روابط `is-a` بین مفاهیم می‌باشد.\n", - "\n", - "Context Graph به دو صورت در دسترس است:\n", - " * فایل متنی بزرگ برای دانلود\n", - " * REST API\n", - "\n", - "آمار:\n", - " * 5401933 مفهوم منحصربه‌فرد\n", - " * 12551613 نمونه منحصربه‌فرد\n", - " * 87603947 رابطه `is-a`\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## استفاده از سرویس وب\n", - "\n", - "سرویس وب تماس‌های مختلفی را برای تخمین احتمال تعلق یک مفهوم به گروه‌های مختلف ارائه می‌دهد. اطلاعات بیشتر [اینجا](https://concept.research.microsoft.com/Home/Api) موجود است. \n", - "در اینجا نمونه‌ای از URL برای فراخوانی آورده شده است: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - این API دیگر در دسترس نیست اما لطفاً به دفترچه نگاه کنید تا مفهوم را درک کنید\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) یک طبقه‌بندی بزرگ از اصطلاحات استخراج‌شده از اینترنت است که شامل روابط `is-a` بین مفاهیم می‌باشد.\n", + "\n", + "Context Graph به دو صورت در دسترس است:\n", + " * فایل متنی بزرگ برای دانلود\n", + " * REST API\n", + "\n", + "آمار:\n", + " * 5401933 مفهوم منحصربه‌فرد\n", + " * 12551613 نمونه منحصربه‌فرد\n", + " * 87603947 رابطه `is-a`\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "بیایید سعی کنیم عناوین اخبار را با استفاده از مفاهیم والد دسته‌بندی کنیم. برای دریافت عناوین اخبار، از سرویس [NewsApi.org](http://newsapi.org) استفاده خواهیم کرد. شما باید کلید API خود را برای استفاده از این سرویس دریافت کنید - به وب‌سایت بروید و برای طرح توسعه‌دهنده رایگان ثبت‌نام کنید.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "ابتدا، ما می‌خواهیم بتوانیم اسامی را از عناوین اخبار استخراج کنیم. برای این کار از کتابخانه `TextBlob` استفاده خواهیم کرد که بسیاری از وظایف معمول پردازش زبان طبیعی را ساده می‌کند.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## استفاده از سرویس وب\n", + "\n", + "سرویس وب تماس‌های مختلفی را برای تخمین احتمال تعلق یک مفهوم به گروه‌های مختلف ارائه می‌دهد. اطلاعات بیشتر [اینجا](https://concept.research.microsoft.com/Home/Api) موجود است. \n", + "در اینجا نمونه‌ای از URL برای فراخوانی آورده شده است: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "ما می‌توانیم ببینیم که اسم‌ها گروه‌های موضوعی بزرگی به ما نمی‌دهند. بیایید اسم‌ها را با اصطلاحات کلی‌تر که از نمودار مفهومی به دست آمده‌اند جایگزین کنیم. این کار کمی زمان می‌برد، زیرا برای هر عبارت اسمی یک تماس REST انجام می‌دهیم.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "بیایید سعی کنیم عناوین اخبار را با استفاده از مفاهیم والد دسته‌بندی کنیم. برای دریافت عناوین اخبار، از سرویس [NewsApi.org](http://newsapi.org) استفاده خواهیم کرد. شما باید کلید API خود را برای استفاده از این سرویس دریافت کنید - به وب‌سایت بروید و برای طرح توسعه‌دهنده رایگان ثبت‌نام کنید.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "ابتدا، ما می‌خواهیم بتوانیم اسامی را از عناوین اخبار استخراج کنیم. برای این کار از کتابخانه `TextBlob` استفاده خواهیم کرد که بسیاری از وظایف معمول پردازش زبان طبیعی را ساده می‌کند.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "ما می‌توانیم ببینیم که اسم‌ها گروه‌های موضوعی بزرگی به ما نمی‌دهند. بیایید اسم‌ها را با اصطلاحات کلی‌تر که از نمودار مفهومی به دست آمده‌اند جایگزین کنیم. این کار کمی زمان می‌برد، زیرا برای هر عبارت اسمی یک تماس REST انجام می‌دهیم.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**سلب مسئولیت**: \nاین سند با استفاده از سرویس ترجمه هوش مصنوعی [Co-op Translator](https://github.com/Azure/co-op-translator) ترجمه شده است. در حالی که ما تلاش می‌کنیم دقت را حفظ کنیم، لطفاً توجه داشته باشید که ترجمه‌های خودکار ممکن است شامل خطاها یا نادرستی‌ها باشند. سند اصلی به زبان اصلی آن باید به عنوان منبع معتبر در نظر گرفته شود. برای اطلاعات حساس، توصیه می‌شود از ترجمه انسانی حرفه‌ای استفاده کنید. ما مسئولیتی در قبال سوء تفاهم‌ها یا تفسیرهای نادرست ناشی از استفاده از این ترجمه نداریم.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**سلب مسئولیت**: \nاین سند با استفاده از سرویس ترجمه هوش مصنوعی [Co-op Translator](https://github.com/Azure/co-op-translator) ترجمه شده است. در حالی که ما تلاش می‌کنیم دقت را حفظ کنیم، لطفاً توجه داشته باشید که ترجمه‌های خودکار ممکن است شامل خطاها یا نادرستی‌ها باشند. سند اصلی به زبان اصلی آن باید به عنوان منبع معتبر در نظر گرفته شود. برای اطلاعات حساس، توصیه می‌شود از ترجمه انسانی حرفه‌ای استفاده کنید. ما مسئولیتی در قبال سوء تفاهم‌ها یا تفسیرهای نادرست ناشی از استفاده از این ترجمه نداریم.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:50:50+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "fa" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:50:50+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "fa" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/fi/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/fi/lessons/2-Symbolic/MSConceptGraph.ipynb index 5aab6970..1465b103 100644 --- a/translations/fi/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/fi/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - TÄMÄ API EI OLE ENÄÄ SAATAVILLA, MUTTA TUTUSTU MUISTIKIRJAAN YMMÄRTÄÄKSESI KONSEPTIN\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) on suuri internetistä kerättyjen termien taksonomia, jossa on `is-a`-suhteita konseptien välillä.\n", - "\n", - "Context Graph on saatavilla kahdessa muodossa:\n", - " * Suuri tekstimuotoinen tiedosto ladattavaksi\n", - " * REST API\n", - "\n", - "Tilastot:\n", - " * 5401933 uniikkia konseptia\n", - " * 12551613 uniikkia instanssia\n", - " * 87603947 `is-a`-suhdetta\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Web-palvelun käyttö\n", - "\n", - "Web-palvelu tarjoaa erilaisia kutsuja arvioimaan todennäköisyyttä, että käsite kuuluu eri ryhmiin. Lisätietoja on saatavilla [täällä](https://concept.research.microsoft.com/Home/Api). \n", - "Tässä on esimerkkiosoite kutsua varten: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - TÄMÄ API EI OLE ENÄÄ SAATAVILLA, MUTTA TUTUSTU MUISTIKIRJAAN YMMÄRTÄÄKSESI KONSEPTIN\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) on suuri internetistä kerättyjen termien taksonomia, jossa on `is-a`-suhteita konseptien välillä.\n", + "\n", + "Context Graph on saatavilla kahdessa muodossa:\n", + " * Suuri tekstimuotoinen tiedosto ladattavaksi\n", + " * REST API\n", + "\n", + "Tilastot:\n", + " * 5401933 uniikkia konseptia\n", + " * 12551613 uniikkia instanssia\n", + " * 87603947 `is-a`-suhdetta\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Yritetään luokitella uutisotsikot pääkäsitteiden avulla. Uutisotsikoiden hankkimiseksi käytämme [NewsApi.org](http://newsapi.org) -palvelua. Sinun täytyy hankkia oma API-avain palvelun käyttämiseksi - mene verkkosivustolle ja rekisteröidy ilmaiselle kehittäjäsuunnitelmalle.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Ensinnäkin haluamme pystyä poimimaan substantiiveja uutisotsikoista. Käytämme tähän `TextBlob`-kirjastoa, joka yksinkertaistaa monia tyypillisiä NLP-tehtäviä, kuten tämän.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Web-palvelun käyttö\n", + "\n", + "Web-palvelu tarjoaa erilaisia kutsuja arvioimaan todennäköisyyttä, että käsite kuuluu eri ryhmiin. Lisätietoja on saatavilla [täällä](https://concept.research.microsoft.com/Home/Api). \n", + "Tässä on esimerkkiosoite kutsua varten: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Voimme nähdä, että substantiivit eivät anna meille suuria temaattisia ryhmiä. Korvataan substantiivit yleisemmillä termeillä, jotka on saatu käsitekaaviosta. Tämä vie jonkin aikaa, koska teemme REST-kutsun jokaiselle substantiivilausekkeelle.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Yritetään luokitella uutisotsikot pääkäsitteiden avulla. Uutisotsikoiden hankkimiseksi käytämme [NewsApi.org](http://newsapi.org) -palvelua. Sinun täytyy hankkia oma API-avain palvelun käyttämiseksi - mene verkkosivustolle ja rekisteröidy ilmaiselle kehittäjäsuunnitelmalle.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Ensinnäkin haluamme pystyä poimimaan substantiiveja uutisotsikoista. Käytämme tähän `TextBlob`-kirjastoa, joka yksinkertaistaa monia tyypillisiä NLP-tehtäviä, kuten tämän.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Voimme nähdä, että substantiivit eivät anna meille suuria temaattisia ryhmiä. Korvataan substantiivit yleisemmillä termeillä, jotka on saatu käsitekaaviosta. Tämä vie jonkin aikaa, koska teemme REST-kutsun jokaiselle substantiivilausekkeelle.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**Vastuuvapauslauseke**: \nTämä asiakirja on käännetty käyttämällä tekoälypohjaista käännöspalvelua [Co-op Translator](https://github.com/Azure/co-op-translator). Vaikka pyrimme tarkkuuteen, huomioithan, että automaattiset käännökset voivat sisältää virheitä tai epätarkkuuksia. Alkuperäistä asiakirjaa sen alkuperäisellä kielellä tulisi pitää ensisijaisena lähteenä. Kriittisen tiedon osalta suositellaan ammattimaista ihmiskäännöstä. Emme ole vastuussa väärinkäsityksistä tai virhetulkinnoista, jotka johtuvat tämän käännöksen käytöstä.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**Vastuuvapauslauseke**: \nTämä asiakirja on käännetty käyttämällä tekoälypohjaista käännöspalvelua [Co-op Translator](https://github.com/Azure/co-op-translator). Vaikka pyrimme tarkkuuteen, huomioithan, että automaattiset käännökset voivat sisältää virheitä tai epätarkkuuksia. Alkuperäistä asiakirjaa sen alkuperäisellä kielellä tulisi pitää ensisijaisena lähteenä. Kriittisen tiedon osalta suositellaan ammattimaista ihmiskäännöstä. Emme ole vastuussa väärinkäsityksistä tai virhetulkinnoista, jotka johtuvat tämän käännöksen käytöstä.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:56:05+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "fi" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:56:05+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "fi" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/fr/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/fr/lessons/2-Symbolic/MSConceptGraph.ipynb index 45a3d8e7..5aa83928 100644 --- a/translations/fr/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/fr/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - CETTE API N'EST PLUS DISPONIBLE, MAIS VEUILLEZ CONSULTER LE CAHIER POUR COMPRENDRE LE CONCEPT\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) est une vaste taxonomie de termes extraits d'Internet, avec des relations de type `is-a` entre les concepts.\n", - "\n", - "Le Context Graph est disponible sous deux formes :\n", - " * Fichier texte volumineux à télécharger\n", - " * API REST\n", - "\n", - "Statistiques :\n", - " * 5 401 933 concepts uniques,\n", - " * 12 551 613 instances uniques,\n", - " * 87 603 947 relations `is-a`\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Utilisation du service web\n", - "\n", - "Le service web propose différents appels pour estimer la probabilité qu'un concept appartienne à différents groupes. Plus d'informations sont disponibles [ici](https://concept.research.microsoft.com/Home/Api). \n", - "Voici l'URL d'exemple à utiliser : `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - CETTE API N'EST PLUS DISPONIBLE, MAIS VEUILLEZ CONSULTER LE CAHIER POUR COMPRENDRE LE CONCEPT\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) est une vaste taxonomie de termes extraits d'Internet, avec des relations de type `is-a` entre les concepts.\n", + "\n", + "Le Context Graph est disponible sous deux formes :\n", + " * Fichier texte volumineux à télécharger\n", + " * API REST\n", + "\n", + "Statistiques :\n", + " * 5 401 933 concepts uniques,\n", + " * 12 551 613 instances uniques,\n", + " * 87 603 947 relations `is-a`\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Essayons de catégoriser les titres des actualités en utilisant des concepts parentaux. Pour obtenir les titres des actualités, nous utiliserons le service [NewsApi.org](http://newsapi.org). Vous devez obtenir votre propre clé API pour utiliser le service - rendez-vous sur le site web et inscrivez-vous au plan développeur gratuit.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Tout d'abord, nous voulons pouvoir extraire les noms des titres d'actualités. Nous utiliserons la bibliothèque `TextBlob` pour cela, qui simplifie beaucoup de tâches typiques de NLP comme celle-ci.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Utilisation du service web\n", + "\n", + "Le service web propose différents appels pour estimer la probabilité qu'un concept appartienne à différents groupes. Plus d'informations sont disponibles [ici](https://concept.research.microsoft.com/Home/Api). \n", + "Voici l'URL d'exemple à utiliser : `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Nous pouvons constater que les noms ne nous donnent pas de grands groupes thématiques. Remplaçons les noms par des termes plus généraux obtenus à partir du graphe conceptuel. Cela prendra un certain temps, car nous effectuons un appel REST pour chaque syntagme nominal.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Essayons de catégoriser les titres des actualités en utilisant des concepts parentaux. Pour obtenir les titres des actualités, nous utiliserons le service [NewsApi.org](http://newsapi.org). Vous devez obtenir votre propre clé API pour utiliser le service - rendez-vous sur le site web et inscrivez-vous au plan développeur gratuit.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Tout d'abord, nous voulons pouvoir extraire les noms des titres d'actualités. Nous utiliserons la bibliothèque `TextBlob` pour cela, qui simplifie beaucoup de tâches typiques de NLP comme celle-ci.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Nous pouvons constater que les noms ne nous donnent pas de grands groupes thématiques. Remplaçons les noms par des termes plus généraux obtenus à partir du graphe conceptuel. Cela prendra un certain temps, car nous effectuons un appel REST pour chaque syntagme nominal.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**Avertissement** : \nCe document a été traduit à l'aide du service de traduction automatique [Co-op Translator](https://github.com/Azure/co-op-translator). Bien que nous nous efforcions d'assurer l'exactitude, veuillez noter que les traductions automatisées peuvent contenir des erreurs ou des inexactitudes. Le document original dans sa langue d'origine doit être considéré comme la source faisant autorité. Pour des informations critiques, il est recommandé de recourir à une traduction humaine professionnelle. Nous déclinons toute responsabilité en cas de malentendus ou d'interprétations erronées résultant de l'utilisation de cette traduction.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**Avertissement** : \nCe document a été traduit à l'aide du service de traduction automatique [Co-op Translator](https://github.com/Azure/co-op-translator). Bien que nous nous efforcions d'assurer l'exactitude, veuillez noter que les traductions automatisées peuvent contenir des erreurs ou des inexactitudes. Le document original dans sa langue d'origine doit être considéré comme la source faisant autorité. Pour des informations critiques, il est recommandé de recourir à une traduction humaine professionnelle. Nous déclinons toute responsabilité en cas de malentendus ou d'interprétations erronées résultant de l'utilisation de cette traduction.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:49:46+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "fr" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:49:46+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "fr" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/he/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/he/lessons/2-Symbolic/MSConceptGraph.ipynb index f081dbab..e8b7ae38 100644 --- a/translations/he/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/he/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - API זה אינו זמין יותר, אך אנא עיינו במחברת כדי להבין את הרעיון\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) הוא טקסונומיה רחבה של מונחים שנאספו מהאינטרנט, עם יחסי `is-a` בין מושגים.\n", - "\n", - "גרף הקשרים זמין בשתי צורות:\n", - " * קובץ טקסט גדול להורדה\n", - " * API מסוג REST\n", - "\n", - "סטטיסטיקות:\n", - " * 5401933 מושגים ייחודיים\n", - " * 12551613 מופעים ייחודיים\n", - " * 87603947 יחסי `is-a`\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## שימוש בשירות אינטרנט\n", - "\n", - "שירות האינטרנט מציע קריאות שונות להערכת הסבירות של מושג להשתייך לקבוצות שונות. מידע נוסף זמין [כאן](https://concept.research.microsoft.com/Home/Api). \n", - "הנה דוגמה לכתובת URL לקריאה: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - API זה אינו זמין יותר, אך אנא עיינו במחברת כדי להבין את הרעיון\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) הוא טקסונומיה רחבה של מונחים שנאספו מהאינטרנט, עם יחסי `is-a` בין מושגים.\n", + "\n", + "גרף הקשרים זמין בשתי צורות:\n", + " * קובץ טקסט גדול להורדה\n", + " * API מסוג REST\n", + "\n", + "סטטיסטיקות:\n", + " * 5401933 מושגים ייחודיים\n", + " * 12551613 מופעים ייחודיים\n", + " * 87603947 יחסי `is-a`\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "בואו ננסה לסווג את כותרות החדשות באמצעות מושגים כלליים. כדי לקבל כותרות חדשות, נשתמש בשירות [NewsApi.org](http://newsapi.org). עליכם להשיג מפתח API משלכם כדי להשתמש בשירות - גשו לאתר והירשמו לתוכנית המפתחים החינמית.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "קודם כל, אנחנו רוצים להיות מסוגלים לחלץ שמות עצם מכותרות חדשות. נשתמש בספריית `TextBlob` כדי לעשות זאת, שמפשטת הרבה משימות טיפוסיות של עיבוד שפה טבעית כמו זו.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## שימוש בשירות אינטרנט\n", + "\n", + "שירות האינטרנט מציע קריאות שונות להערכת הסבירות של מושג להשתייך לקבוצות שונות. מידע נוסף זמין [כאן](https://concept.research.microsoft.com/Home/Api). \n", + "הנה דוגמה לכתובת URL לקריאה: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "אנחנו יכולים לראות ששמות עצם אינם נותנים לנו קבוצות נושאיות גדולות. בואו נחליף את שמות העצם במונחים כלליים יותר שהושגו מתוך גרף המושגים. זה ייקח קצת זמן, כי אנחנו מבצעים קריאת REST עבור כל ביטוי שם עצם.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "בואו ננסה לסווג את כותרות החדשות באמצעות מושגים כלליים. כדי לקבל כותרות חדשות, נשתמש בשירות [NewsApi.org](http://newsapi.org). עליכם להשיג מפתח API משלכם כדי להשתמש בשירות - גשו לאתר והירשמו לתוכנית המפתחים החינמית.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "קודם כל, אנחנו רוצים להיות מסוגלים לחלץ שמות עצם מכותרות חדשות. נשתמש בספריית `TextBlob` כדי לעשות זאת, שמפשטת הרבה משימות טיפוסיות של עיבוד שפה טבעית כמו זו.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "אנחנו יכולים לראות ששמות עצם אינם נותנים לנו קבוצות נושאיות גדולות. בואו נחליף את שמות העצם במונחים כלליים יותר שהושגו מתוך גרף המושגים. זה ייקח קצת זמן, כי אנחנו מבצעים קריאת REST עבור כל ביטוי שם עצם.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**כתב ויתור**: \nמסמך זה תורגם באמצעות שירות תרגום מבוסס בינה מלאכותית [Co-op Translator](https://github.com/Azure/co-op-translator). למרות שאנו שואפים לדיוק, יש לקחת בחשבון שתרגומים אוטומטיים עשויים להכיל שגיאות או אי דיוקים. המסמך המקורי בשפתו המקורית צריך להיחשב כמקור הסמכותי. עבור מידע קריטי, מומלץ להשתמש בתרגום מקצועי על ידי אדם. אנו לא נושאים באחריות לאי הבנות או לפרשנויות שגויות הנובעות משימוש בתרגום זה.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**כתב ויתור**: \nמסמך זה תורגם באמצעות שירות תרגום מבוסס בינה מלאכותית [Co-op Translator](https://github.com/Azure/co-op-translator). למרות שאנו שואפים לדיוק, יש לקחת בחשבון שתרגומים אוטומטיים עשויים להכיל שגיאות או אי דיוקים. המסמך המקורי בשפתו המקורית צריך להיחשב כמקור הסמכותי. עבור מידע קריטי, מומלץ להשתמש בתרגום מקצועי על ידי אדם. אנו לא נושאים באחריות לאי הבנות או לפרשנויות שגויות הנובעות משימוש בתרגום זה.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:56:31+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "he" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:56:31+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "he" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/hi/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/hi/lessons/2-Symbolic/MSConceptGraph.ipynb index 2c63bd8a..b44ccdf6 100644 --- a/translations/hi/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/hi/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,547 +1,538 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - यह API अब उपलब्ध नहीं है, लेकिन कृपया नोटबुक देखें ताकि अवधारणा को समझा जा सके\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) इंटरनेट से प्राप्त शब्दों का एक बड़ा वर्गीकरण है, जिसमें अवधारणाओं के बीच `is-a` संबंध होते हैं।\n", - "\n", - "Context Graph दो रूपों में उपलब्ध है:\n", - " * डाउनलोड के लिए बड़ा टेक्स्ट फ़ाइल\n", - " * REST API\n", - "\n", - "सांख्यिकी:\n", - " * 5401933 अद्वितीय अवधारणाएँ,\n", - " * 12551613 अद्वितीय उदाहरण\n", - " * 87603947 `is-a` संबंध\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## वेब सेवा का उपयोग करना\n", - "\n", - "वेब सेवा विभिन्न समूहों में किसी अवधारणा के संबंधित होने की संभावना का अनुमान लगाने के लिए अलग-अलग कॉल प्रदान करती है। अधिक जानकारी [यहां](https://concept.research.microsoft.com/Home/Api) उपलब्ध है। कॉल करने के लिए यहां एक नमूना URL है: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - यह API अब उपलब्ध नहीं है, लेकिन कृपया नोटबुक देखें ताकि अवधारणा को समझा जा सके\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) इंटरनेट से प्राप्त शब्दों का एक बड़ा वर्गीकरण है, जिसमें अवधारणाओं के बीच `is-a` संबंध होते हैं।\n", + "\n", + "Context Graph दो रूपों में उपलब्ध है:\n", + " * डाउनलोड के लिए बड़ा टेक्स्ट फ़ाइल\n", + " * REST API\n", + "\n", + "सांख्यिकी:\n", + " * 5401933 अद्वितीय अवधारणाएँ,\n", + " * 12551613 अद्वितीय उदाहरण\n", + " * 87603947 `is-a` संबंध\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "चलो समाचार शीर्षकों को मुख्य अवधारणाओं के तहत वर्गीकृत करने की कोशिश करते हैं। समाचार शीर्षक प्राप्त करने के लिए, हम [NewsApi.org](http://newsapi.org) सेवा का उपयोग करेंगे। इस सेवा का उपयोग करने के लिए आपको अपना API कुंजी प्राप्त करनी होगी - वेबसाइट पर जाएं और मुफ्त डेवलपर योजना के लिए पंजीकरण करें।\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "सबसे पहले, हम समाचार शीर्षकों से संज्ञा निकालने में सक्षम होना चाहते हैं। हम इसके लिए `TextBlob` लाइब्रेरी का उपयोग करेंगे, जो इस तरह के सामान्य NLP कार्यों को बहुत सरल बनाती है।\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## वेब सेवा का उपयोग करना\n", + "\n", + "वेब सेवा विभिन्न समूहों में किसी अवधारणा के संबंधित होने की संभावना का अनुमान लगाने के लिए अलग-अलग कॉल प्रदान करती है। अधिक जानकारी [यहां](https://concept.research.microsoft.com/Home/Api) उपलब्ध है। कॉल करने के लिए यहां एक नमूना URL है: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "हम देख सकते हैं कि संज्ञाएं हमें बड़े थीमेटिक समूह नहीं देती हैं। आइए संज्ञाओं को अवधारणा ग्राफ से प्राप्त अधिक सामान्य शब्दों से बदलें। इसमें कुछ समय लगेगा, क्योंकि हम प्रत्येक संज्ञा वाक्यांश के लिए REST कॉल कर रहे हैं।\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "चलो समाचार शीर्षकों को मुख्य अवधारणाओं के तहत वर्गीकृत करने की कोशिश करते हैं। समाचार शीर्षक प्राप्त करने के लिए, हम [NewsApi.org](http://newsapi.org) सेवा का उपयोग करेंगे। इस सेवा का उपयोग करने के लिए आपको अपना API कुंजी प्राप्त करनी होगी - वेबसाइट पर जाएं और मुफ्त डेवलपर योजना के लिए पंजीकरण करें।\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "सबसे पहले, हम समाचार शीर्षकों से संज्ञा निकालने में सक्षम होना चाहते हैं। हम इसके लिए `TextBlob` लाइब्रेरी का उपयोग करेंगे, जो इस तरह के सामान्य NLP कार्यों को बहुत सरल बनाती है।\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "हम देख सकते हैं कि संज्ञाएं हमें बड़े थीमेटिक समूह नहीं देती हैं। आइए संज्ञाओं को अवधारणा ग्राफ से प्राप्त अधिक सामान्य शब्दों से बदलें। इसमें कुछ समय लगेगा, क्योंकि हम प्रत्येक संज्ञा वाक्यांश के लिए REST कॉल कर रहे हैं।\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**अस्वीकरण**: \nयह दस्तावेज़ AI अनुवाद सेवा [Co-op Translator](https://github.com/Azure/co-op-translator) का उपयोग करके अनुवादित किया गया है। जबकि हम सटीकता के लिए प्रयास करते हैं, कृपया ध्यान दें कि स्वचालित अनुवाद में त्रुटियां या अशुद्धियां हो सकती हैं। मूल दस्तावेज़ को उसकी मूल भाषा में आधिकारिक स्रोत माना जाना चाहिए। महत्वपूर्ण जानकारी के लिए, पेशेवर मानव अनुवाद की सिफारिश की जाती है। इस अनुवाद के उपयोग से उत्पन्न किसी भी गलतफहमी या गलत व्याख्या के लिए हम उत्तरदायी नहीं हैं।\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**अस्वीकरण**: \nयह दस्तावेज़ AI अनुवाद सेवा [Co-op Translator](https://github.com/Azure/co-op-translator) का उपयोग करके अनुवादित किया गया है। जबकि हम सटीकता के लिए प्रयास करते हैं, कृपया ध्यान दें कि स्वचालित अनुवाद में त्रुटियां या अशुद्धियां हो सकती हैं। मूल दस्तावेज़ को उसकी मूल भाषा में आधिकारिक स्रोत माना जाना चाहिए। महत्वपूर्ण जानकारी के लिए, पेशेवर मानव अनुवाद की सिफारिश की जाती है। इस अनुवाद के उपयोग से उत्पन्न किसी भी गलतफहमी या गलत व्याख्या के लिए हम उत्तरदायी नहीं हैं।\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:52:37+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "hi" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:52:37+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "hi" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/hr/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/hr/lessons/2-Symbolic/MSConceptGraph.ipynb index 08997531..7201037a 100644 --- a/translations/hr/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/hr/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - OVAJ API VIŠE NIJE DOSTUPAN, ALI MOLIMO POGLEDAJTE BILJEŽNICU ZA RAZUMIJEVANJE KONCEPTA\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) je velika taksonomija pojmova prikupljenih s interneta, s `is-a` odnosima između koncepata.\n", - "\n", - "Context Graph dostupan je u dva oblika:\n", - " * Velika tekstualna datoteka za preuzimanje\n", - " * REST API\n", - "\n", - "Statistika:\n", - " * 5401933 jedinstvenih koncepata,\n", - " * 12551613 jedinstvenih instanci\n", - " * 87603947 `is-a` odnosa\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Korištenje web usluge\n", - "\n", - "Web usluga nudi različite pozive za procjenu vjerojatnosti da koncept pripada različitim grupama. Više informacija dostupno je [ovdje](https://concept.research.microsoft.com/Home/Api). \n", - "Evo primjera URL-a za poziv: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - OVAJ API VIŠE NIJE DOSTUPAN, ALI MOLIMO POGLEDAJTE BILJEŽNICU ZA RAZUMIJEVANJE KONCEPTA\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) je velika taksonomija pojmova prikupljenih s interneta, s `is-a` odnosima između koncepata.\n", + "\n", + "Context Graph dostupan je u dva oblika:\n", + " * Velika tekstualna datoteka za preuzimanje\n", + " * REST API\n", + "\n", + "Statistika:\n", + " * 5401933 jedinstvenih koncepata,\n", + " * 12551613 jedinstvenih instanci\n", + " * 87603947 `is-a` odnosa\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Pokušajmo kategorizirati naslove vijesti koristeći roditeljske pojmove. Za dobivanje naslova vijesti koristit ćemo uslugu [NewsApi.org](http://newsapi.org). Morate dobiti vlastiti API ključ kako biste koristili uslugu - posjetite web stranicu i registrirajte se za besplatni plan za developere.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Prije svega, želimo biti u mogućnosti izdvojiti imenice iz naslova vijesti. Koristit ćemo biblioteku `TextBlob` za to, koja pojednostavljuje mnoge tipične NLP zadatke poput ovog.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Korištenje web usluge\n", + "\n", + "Web usluga nudi različite pozive za procjenu vjerojatnosti da koncept pripada različitim grupama. Više informacija dostupno je [ovdje](https://concept.research.microsoft.com/Home/Api). \n", + "Evo primjera URL-a za poziv: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Možemo vidjeti da imenice ne daju velike tematske skupine. Zamijenimo imenice općenitijim pojmovima dobivenim iz konceptualnog grafa. Ovo će potrajati neko vrijeme, jer radimo REST poziv za svaku imeničku frazu.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Pokušajmo kategorizirati naslove vijesti koristeći roditeljske pojmove. Za dobivanje naslova vijesti koristit ćemo uslugu [NewsApi.org](http://newsapi.org). Morate dobiti vlastiti API ključ kako biste koristili uslugu - posjetite web stranicu i registrirajte se za besplatni plan za developere.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Prije svega, želimo biti u mogućnosti izdvojiti imenice iz naslova vijesti. Koristit ćemo biblioteku `TextBlob` za to, koja pojednostavljuje mnoge tipične NLP zadatke poput ovog.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Možemo vidjeti da imenice ne daju velike tematske skupine. Zamijenimo imenice općenitijim pojmovima dobivenim iz konceptualnog grafa. Ovo će potrajati neko vrijeme, jer radimo REST poziv za svaku imeničku frazu.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**Izjava o odricanju odgovornosti**: \nOvaj dokument je preveden pomoću AI usluge za prevođenje [Co-op Translator](https://github.com/Azure/co-op-translator). Iako nastojimo osigurati točnost, imajte na umu da automatski prijevodi mogu sadržavati pogreške ili netočnosti. Izvorni dokument na izvornom jeziku treba smatrati autoritativnim izvorom. Za ključne informacije preporučuje se profesionalni prijevod od strane ljudskog prevoditelja. Ne preuzimamo odgovornost za nesporazume ili pogrešna tumačenja koja mogu proizaći iz korištenja ovog prijevoda.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**Izjava o odricanju odgovornosti**: \nOvaj dokument je preveden pomoću AI usluge za prevođenje [Co-op Translator](https://github.com/Azure/co-op-translator). Iako nastojimo osigurati točnost, imajte na umu da automatski prijevodi mogu sadržavati pogreške ili netočnosti. Izvorni dokument na izvornom jeziku treba smatrati autoritativnim izvorom. Za ključne informacije preporučuje se profesionalni prijevod od strane ljudskog prevoditelja. Ne preuzimamo odgovornost za nesporazume ili pogrešna tumačenja koja mogu proizaći iz korištenja ovog prijevoda.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:59:16+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "hr" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:59:16+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "hr" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/hu/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/hu/lessons/2-Symbolic/MSConceptGraph.ipynb index 433165bf..6a8904cb 100644 --- a/translations/hu/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/hu/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - EZ AZ API MÁR NEM ELÉRHETŐ, DE KÉRJÜK, TEKINTSE MEG A JEGYZETFÜZETET A KONCEPCIÓ MEGÉRTÉSÉHEZ\n", - "\n", - "A [Microsoft Concept Graph](https://concept.research.microsoft.com/) egy nagy méretű taxonómia, amely az internetről gyűjtött kifejezéseket tartalmaz, `is-a` kapcsolatokkal a fogalmak között.\n", - "\n", - "A Context Graph két formában érhető el:\n", - " * Nagy szövegfájl letöltésre\n", - " * REST API\n", - "\n", - "Statisztikák:\n", - " * 5401933 egyedi fogalom\n", - " * 12551613 egyedi példány\n", - " * 87603947 `is-a` kapcsolat\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Webszolgáltatás használata\n", - "\n", - "A webszolgáltatás különböző hívásokat kínál annak becslésére, hogy egy fogalom milyen valószínűséggel tartozik különböző csoportokhoz. További információ [itt](https://concept.research.microsoft.com/Home/Api) érhető el. \n", - "Íme egy példa URL a híváshoz: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - EZ AZ API MÁR NEM ELÉRHETŐ, DE KÉRJÜK, TEKINTSE MEG A JEGYZETFÜZETET A KONCEPCIÓ MEGÉRTÉSÉHEZ\n", + "\n", + "A [Microsoft Concept Graph](https://concept.research.microsoft.com/) egy nagy méretű taxonómia, amely az internetről gyűjtött kifejezéseket tartalmaz, `is-a` kapcsolatokkal a fogalmak között.\n", + "\n", + "A Context Graph két formában érhető el:\n", + " * Nagy szövegfájl letöltésre\n", + " * REST API\n", + "\n", + "Statisztikák:\n", + " * 5401933 egyedi fogalom\n", + " * 12551613 egyedi példány\n", + " * 87603947 `is-a` kapcsolat\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Próbáljuk meg kategorizálni a hírcímeket szülőfogalmak segítségével. A hírcímek beszerzéséhez a [NewsApi.org](http://newsapi.org) szolgáltatást fogjuk használni. A szolgáltatás használatához saját API kulcsot kell beszerezni - látogass el a weboldalra, és regisztrálj az ingyenes fejlesztői tervre.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Először is, szeretnénk képesek lenni arra, hogy főneveket vonjunk ki hírcímekből. Ehhez a `TextBlob` könyvtárat fogjuk használni, amely jelentősen leegyszerűsíti az ilyen tipikus NLP feladatokat.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Webszolgáltatás használata\n", + "\n", + "A webszolgáltatás különböző hívásokat kínál annak becslésére, hogy egy fogalom milyen valószínűséggel tartozik különböző csoportokhoz. További információ [itt](https://concept.research.microsoft.com/Home/Api) érhető el. \n", + "Íme egy példa URL a híváshoz: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Láthatjuk, hogy a főnevek nem adnak nagy tematikus csoportokat. Helyettesítsük a főneveket általánosabb kifejezésekkel, amelyeket a fogalomgráfból nyerünk. Ez időbe telik, mivel minden főnévi kifejezéshez REST-hívást végzünk.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Próbáljuk meg kategorizálni a hírcímeket szülőfogalmak segítségével. A hírcímek beszerzéséhez a [NewsApi.org](http://newsapi.org) szolgáltatást fogjuk használni. A szolgáltatás használatához saját API kulcsot kell beszerezni - látogass el a weboldalra, és regisztrálj az ingyenes fejlesztői tervre.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Először is, szeretnénk képesek lenni arra, hogy főneveket vonjunk ki hírcímekből. Ehhez a `TextBlob` könyvtárat fogjuk használni, amely jelentősen leegyszerűsíti az ilyen tipikus NLP feladatokat.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Láthatjuk, hogy a főnevek nem adnak nagy tematikus csoportokat. Helyettesítsük a főneveket általánosabb kifejezésekkel, amelyeket a fogalomgráfból nyerünk. Ez időbe telik, mivel minden főnévi kifejezéshez REST-hívást végzünk.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**Felelősség kizárása**: \nEz a dokumentum az [Co-op Translator](https://github.com/Azure/co-op-translator) AI fordítási szolgáltatás segítségével lett lefordítva. Bár törekszünk a pontosságra, kérjük, vegye figyelembe, hogy az automatikus fordítások hibákat vagy pontatlanságokat tartalmazhatnak. Az eredeti dokumentum az eredeti nyelvén tekintendő hiteles forrásnak. Fontos információk esetén javasolt professzionális emberi fordítást igénybe venni. Nem vállalunk felelősséget semmilyen félreértésért vagy téves értelmezésért, amely a fordítás használatából eredhet.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**Felelősség kizárása**: \nEz a dokumentum az [Co-op Translator](https://github.com/Azure/co-op-translator) AI fordítási szolgáltatás segítségével lett lefordítva. Bár törekszünk a pontosságra, kérjük, vegye figyelembe, hogy az automatikus fordítások hibákat vagy pontatlanságokat tartalmazhatnak. Az eredeti dokumentum az eredeti nyelvén tekintendő hiteles forrásnak. Fontos információk esetén javasolt professzionális emberi fordítást igénybe venni. Nem vállalunk felelősséget semmilyen félreértésért vagy téves értelmezésért, amely a fordítás használatából eredhet.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:57:53+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "hu" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:57:53+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "hu" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/id/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/id/lessons/2-Symbolic/MSConceptGraph.ipynb index 8459e1f1..d9836e65 100644 --- a/translations/id/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/id/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,547 +1,538 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - API INI SUDAH TIDAK TERSEDIA LAGI TAPI SILAKAN LIHAT NOTEBOOK UNTUK MEMAHAMI KONSEPNYA\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) adalah taksonomi besar dari istilah-istilah yang diperoleh dari internet, dengan hubungan `is-a` antara konsep-konsep.\n", - "\n", - "Context Graph tersedia dalam dua bentuk:\n", - " * File teks besar untuk diunduh\n", - " * REST API\n", - "\n", - "Statistik:\n", - " * 5401933 konsep unik,\n", - " * 12551613 instance unik\n", - " * 87603947 hubungan `is-a`\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Menggunakan Layanan Web\n", - "\n", - "Layanan web menawarkan berbagai panggilan untuk memperkirakan probabilitas suatu konsep termasuk dalam berbagai kelompok. Informasi lebih lanjut tersedia [di sini](https://concept.research.microsoft.com/Home/Api). Berikut adalah contoh URL untuk melakukan panggilan: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - API INI SUDAH TIDAK TERSEDIA LAGI TAPI SILAKAN LIHAT NOTEBOOK UNTUK MEMAHAMI KONSEPNYA\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) adalah taksonomi besar dari istilah-istilah yang diperoleh dari internet, dengan hubungan `is-a` antara konsep-konsep.\n", + "\n", + "Context Graph tersedia dalam dua bentuk:\n", + " * File teks besar untuk diunduh\n", + " * REST API\n", + "\n", + "Statistik:\n", + " * 5401933 konsep unik,\n", + " * 12551613 instance unik\n", + " * 87603947 hubungan `is-a`\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Mari kita coba mengkategorikan judul berita menggunakan konsep induk. Untuk mendapatkan judul berita, kita akan menggunakan layanan [NewsApi.org](http://newsapi.org). Anda perlu mendapatkan kunci API Anda sendiri untuk menggunakan layanan ini - kunjungi situs web dan daftar untuk rencana pengembang gratis.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Pertama-tama, kami ingin dapat mengekstrak kata benda dari judul berita. Kami akan menggunakan pustaka `TextBlob` untuk melakukan ini, yang menyederhanakan banyak tugas NLP khas seperti ini.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Menggunakan Layanan Web\n", + "\n", + "Layanan web menawarkan berbagai panggilan untuk memperkirakan probabilitas suatu konsep termasuk dalam berbagai kelompok. Informasi lebih lanjut tersedia [di sini](https://concept.research.microsoft.com/Home/Api). Berikut adalah contoh URL untuk melakukan panggilan: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Kita dapat melihat bahwa kata benda tidak memberikan kita kelompok tematik yang besar. Mari kita ganti kata benda dengan istilah yang lebih umum yang diperoleh dari grafik konsep. Ini akan memakan waktu, karena kita melakukan panggilan REST untuk setiap frasa kata benda.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Mari kita coba mengkategorikan judul berita menggunakan konsep induk. Untuk mendapatkan judul berita, kita akan menggunakan layanan [NewsApi.org](http://newsapi.org). Anda perlu mendapatkan kunci API Anda sendiri untuk menggunakan layanan ini - kunjungi situs web dan daftar untuk rencana pengembang gratis.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Pertama-tama, kami ingin dapat mengekstrak kata benda dari judul berita. Kami akan menggunakan pustaka `TextBlob` untuk melakukan ini, yang menyederhanakan banyak tugas NLP khas seperti ini.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Kita dapat melihat bahwa kata benda tidak memberikan kita kelompok tematik yang besar. Mari kita ganti kata benda dengan istilah yang lebih umum yang diperoleh dari grafik konsep. Ini akan memakan waktu, karena kita melakukan panggilan REST untuk setiap frasa kata benda.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**Penafian**: \nDokumen ini telah diterjemahkan menggunakan layanan penerjemahan AI [Co-op Translator](https://github.com/Azure/co-op-translator). Meskipun kami berupaya untuk memberikan hasil yang akurat, harap diketahui bahwa terjemahan otomatis mungkin mengandung kesalahan atau ketidakakuratan. Dokumen asli dalam bahasa aslinya harus dianggap sebagai sumber yang otoritatif. Untuk informasi yang penting, disarankan menggunakan jasa penerjemahan manusia profesional. Kami tidak bertanggung jawab atas kesalahpahaman atau interpretasi yang salah yang timbul dari penggunaan terjemahan ini.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**Penafian**: \nDokumen ini telah diterjemahkan menggunakan layanan penerjemahan AI [Co-op Translator](https://github.com/Azure/co-op-translator). Meskipun kami berupaya untuk memberikan hasil yang akurat, harap diketahui bahwa terjemahan otomatis mungkin mengandung kesalahan atau ketidakakuratan. Dokumen asli dalam bahasa aslinya harus dianggap sebagai sumber yang otoritatif. Untuk informasi yang penting, disarankan menggunakan jasa penerjemahan manusia profesional. Kami tidak bertanggung jawab atas kesalahpahaman atau interpretasi yang salah yang timbul dari penggunaan terjemahan ini.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:56:57+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "id" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:56:57+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "id" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/it/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/it/lessons/2-Symbolic/MSConceptGraph.ipynb index af2c4cb2..475fdb6a 100644 --- a/translations/it/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/it/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - QUESTA API NON È PIÙ DISPONIBILE, MA CONSULTA IL NOTEBOOK PER CAPIRE IL CONCETTO\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) è una vasta tassonomia di termini estratti da internet, con relazioni di tipo `is-a` tra i concetti.\n", - "\n", - "Context Graph è disponibile in due forme:\n", - " * File di testo di grandi dimensioni per il download\n", - " * API REST\n", - "\n", - "Statistiche:\n", - " * 5401933 concetti unici,\n", - " * 12551613 istanze uniche\n", - " * 87603947 relazioni di tipo `is-a`\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Utilizzo del servizio web\n", - "\n", - "Il servizio web offre diverse chiamate per stimare la probabilità che un concetto appartenga a gruppi diversi. Maggiori informazioni sono disponibili [qui](https://concept.research.microsoft.com/Home/Api). \n", - "Ecco l'URL di esempio per effettuare la chiamata: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - QUESTA API NON È PIÙ DISPONIBILE, MA CONSULTA IL NOTEBOOK PER CAPIRE IL CONCETTO\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) è una vasta tassonomia di termini estratti da internet, con relazioni di tipo `is-a` tra i concetti.\n", + "\n", + "Context Graph è disponibile in due forme:\n", + " * File di testo di grandi dimensioni per il download\n", + " * API REST\n", + "\n", + "Statistiche:\n", + " * 5401933 concetti unici,\n", + " * 12551613 istanze uniche\n", + " * 87603947 relazioni di tipo `is-a`\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Proviamo a categorizzare i titoli delle notizie utilizzando concetti principali. Per ottenere i titoli delle notizie, utilizzeremo il servizio [NewsApi.org](http://newsapi.org). È necessario ottenere la propria chiave API per utilizzare il servizio - vai al sito web e registrati per il piano sviluppatore gratuito.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Prima di tutto, vogliamo essere in grado di estrarre i sostantivi dai titoli delle notizie. Useremo la libreria `TextBlob` per farlo, che semplifica molti compiti tipici di NLP come questo.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Utilizzo del servizio web\n", + "\n", + "Il servizio web offre diverse chiamate per stimare la probabilità che un concetto appartenga a gruppi diversi. Maggiori informazioni sono disponibili [qui](https://concept.research.microsoft.com/Home/Api). \n", + "Ecco l'URL di esempio per effettuare la chiamata: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Possiamo vedere che i sostantivi non ci forniscono grandi gruppi tematici. Sostituiamo i sostantivi con termini più generali ottenuti dal grafo concettuale. Questo richiederà del tempo, perché stiamo effettuando una chiamata REST per ogni frase nominale.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Proviamo a categorizzare i titoli delle notizie utilizzando concetti principali. Per ottenere i titoli delle notizie, utilizzeremo il servizio [NewsApi.org](http://newsapi.org). È necessario ottenere la propria chiave API per utilizzare il servizio - vai al sito web e registrati per il piano sviluppatore gratuito.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Prima di tutto, vogliamo essere in grado di estrarre i sostantivi dai titoli delle notizie. Useremo la libreria `TextBlob` per farlo, che semplifica molti compiti tipici di NLP come questo.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Possiamo vedere che i sostantivi non ci forniscono grandi gruppi tematici. Sostituiamo i sostantivi con termini più generali ottenuti dal grafo concettuale. Questo richiederà del tempo, perché stiamo effettuando una chiamata REST per ogni frase nominale.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**Disclaimer**: \nQuesto documento è stato tradotto utilizzando il servizio di traduzione automatica [Co-op Translator](https://github.com/Azure/co-op-translator). Sebbene ci impegniamo per garantire l'accuratezza, si prega di notare che le traduzioni automatiche possono contenere errori o imprecisioni. Il documento originale nella sua lingua nativa dovrebbe essere considerato la fonte autorevole. Per informazioni critiche, si raccomanda una traduzione professionale effettuata da un traduttore umano. Non siamo responsabili per eventuali incomprensioni o interpretazioni errate derivanti dall'uso di questa traduzione.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**Disclaimer**: \nQuesto documento è stato tradotto utilizzando il servizio di traduzione automatica [Co-op Translator](https://github.com/Azure/co-op-translator). Sebbene ci impegniamo per garantire l'accuratezza, si prega di notare che le traduzioni automatiche possono contenere errori o imprecisioni. Il documento originale nella sua lingua nativa dovrebbe essere considerato la fonte autorevole. Per informazioni critiche, si raccomanda una traduzione professionale effettuata da un traduttore umano. Non siamo responsabili per eventuali incomprensioni o interpretazioni errate derivanti dall'uso di questa traduzione.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:54:14+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "it" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:54:14+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "it" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/ja/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/ja/lessons/2-Symbolic/MSConceptGraph.ipynb index 775a19e8..18134177 100644 --- a/translations/ja/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/ja/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - このAPIは現在利用できませんが、コンセプトを理解するためにノートブックをご覧ください\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/)は、インターネットから抽出された用語の大規模な分類体系で、概念間の`is-a`関係を含んでいます。\n", - "\n", - "Context Graphは以下の2つの形式で利用可能です：\n", - " * ダウンロード可能な大規模テキストファイル\n", - " * REST API\n", - "\n", - "統計情報：\n", - " * 5401933のユニークな概念\n", - " * 12551613のユニークなインスタンス\n", - " * 87603947の`is-a`関係\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Webサービスの使用\n", - "\n", - "Webサービスは、ある概念が異なるグループに属する確率を推定するためのさまざまな呼び出しを提供します。詳細情報は[こちら](https://concept.research.microsoft.com/Home/Api)で確認できます。 \n", - "以下は呼び出しのサンプルURLです: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - このAPIは現在利用できませんが、コンセプトを理解するためにノートブックをご覧ください\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/)は、インターネットから抽出された用語の大規模な分類体系で、概念間の`is-a`関係を含んでいます。\n", + "\n", + "Context Graphは以下の2つの形式で利用可能です：\n", + " * ダウンロード可能な大規模テキストファイル\n", + " * REST API\n", + "\n", + "統計情報：\n", + " * 5401933のユニークな概念\n", + " * 12551613のユニークなインスタンス\n", + " * 87603947の`is-a`関係\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "ニュースタイトルを親概念で分類してみましょう。ニュースタイトルを取得するには、[NewsApi.org](http://newsapi.org)サービスを使用します。このサービスを利用するには、独自のAPIキーを取得する必要があります。ウェブサイトにアクセスして無料の開発者プランに登録してください。\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "まず第一に、ニュースのタイトルから名詞を抽出できるようにしたいと考えています。これを行うために、典型的なNLPタスクを簡素化する`TextBlob`ライブラリを使用します。\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Webサービスの使用\n", + "\n", + "Webサービスは、ある概念が異なるグループに属する確率を推定するためのさまざまな呼び出しを提供します。詳細情報は[こちら](https://concept.research.microsoft.com/Home/Api)で確認できます。 \n", + "以下は呼び出しのサンプルURLです: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "名詞が大きなテーマグループを与えないことがわかります。概念グラフから得られたより一般的な用語に名詞を置き換えてみましょう。これは、各名詞句に対してREST呼び出しを行うため、少し時間がかかります。\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "ニュースタイトルを親概念で分類してみましょう。ニュースタイトルを取得するには、[NewsApi.org](http://newsapi.org)サービスを使用します。このサービスを利用するには、独自のAPIキーを取得する必要があります。ウェブサイトにアクセスして無料の開発者プランに登録してください。\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "まず第一に、ニュースのタイトルから名詞を抽出できるようにしたいと考えています。これを行うために、典型的なNLPタスクを簡素化する`TextBlob`ライブラリを使用します。\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "名詞が大きなテーマグループを与えないことがわかります。概念グラフから得られたより一般的な用語に名詞を置き換えてみましょう。これは、各名詞句に対してREST呼び出しを行うため、少し時間がかかります。\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**免責事項**: \nこの文書は、AI翻訳サービス [Co-op Translator](https://github.com/Azure/co-op-translator) を使用して翻訳されています。正確性を追求しておりますが、自動翻訳には誤りや不正確な部分が含まれる可能性があります。元の言語で記載された文書を正式な情報源とみなしてください。重要な情報については、専門の人間による翻訳を推奨します。この翻訳の使用に起因する誤解や誤認について、当方は一切の責任を負いません。\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**免責事項**: \nこの文書は、AI翻訳サービス [Co-op Translator](https://github.com/Azure/co-op-translator) を使用して翻訳されています。正確性を追求しておりますが、自動翻訳には誤りや不正確な部分が含まれる可能性があります。元の言語で記載された文書を正式な情報源とみなしてください。重要な情報については、専門の人間による翻訳を推奨します。この翻訳の使用に起因する誤解や誤認について、当方は一切の責任を負いません。\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:52:11+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "ja" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:52:11+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "ja" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/kn/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/kn/lessons/2-Symbolic/MSConceptGraph.ipynb index 49978ec9..fbe832b6 100644 --- a/translations/kn/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/kn/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - ಈ API ಈಗ ಲಭ್ಯವಿಲ್ಲ ಆದರೆ ಸಂಪ್ರದಾಯವನ್ನು ಅರ್ಥಮಾಡಿಕೊಳ್ಳಲು ದಯವಿಟ್ಟು ನೋಟ್ಬುಕ್ ನೋಡಿ\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) ಇಂಟರ್ನೆಟ್‌ನಿಂದ ಸಂಗ್ರಹಿಸಲಾದ ಪದಗಳ ದೊಡ್ಡ ವರ್ಗೀಕರಣವಾಗಿದೆ, ಇದರಲ್ಲಿ ಸಂಪ್ರದಾಯಗಳ ನಡುವೆ `is-a` ಸಂಬಂಧಗಳಿವೆ.\n", - "\n", - "Context Graph ಎರಡು ರೂಪಗಳಲ್ಲಿ ಲಭ್ಯವಿದೆ:\n", - " * ಡೌನ್‌ಲೋಡ್ ಮಾಡಲು ದೊಡ್ಡ ಪಠ್ಯ ಕಡತ\n", - " * REST API\n", - "\n", - "ಸಂಖ್ಯಾಶಾಸ್ತ್ರ:\n", - " * 5401933 ವಿಶಿಷ್ಟ ಸಂಪ್ರದಾಯಗಳು,\n", - " * 12551613 ವಿಶಿಷ್ಟ ಉದಾಹರಣೆಗಳು\n", - " * 87603947 `is-a` ಸಂಬಂಧಗಳು\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## ವೆಬ್ ಸೇವೆಯನ್ನು ಬಳಸುವುದು\n", - "\n", - "ವೆಬ್ ಸೇವೆ ವಿಭಿನ್ನ ಗುಂಪುಗಳಿಗೆ ಸೇರಿದ ಒಂದು ಕಲ್ಪನೆಯ ಸಂಭವನೀಯತೆಯನ್ನು ಅಂದಾಜಿಸಲು ವಿವಿಧ ಕರೆಗಳನ್ನು ನೀಡುತ್ತದೆ. ಹೆಚ್ಚಿನ ಮಾಹಿತಿಗಾಗಿ [ಇಲ್ಲಿ](https://concept.research.microsoft.com/Home/Api) ಲಭ್ಯವಿದೆ. \n", - "ಕರೆಗೆ ಉದಾಹರಣೆಯ URL ಇಲ್ಲಿದೆ: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" - ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "ನಾವು ಸುದ್ದಿಯ ಶೀರ್ಷಿಕೆಗಳನ್ನು ಪೋಷಕ ಸಂಪ್ರದಾಯಗಳ ಮೂಲಕ ವರ್ಗೀಕರಿಸಲು ಪ್ರಯತ್ನಿಸೋಣ. ಸುದ್ದಿಯ ಶೀರ್ಷಿಕೆಗಳನ್ನು ಪಡೆಯಲು, ನಾವು [NewsApi.org](http://newsapi.org) ಸೇವೆಯನ್ನು ಬಳಸಲಿದ್ದೇವೆ. ಸೇವೆಯನ್ನು ಬಳಸಲು ನಿಮ್ಮದೇ API ಕೀ ಪಡೆಯಬೇಕಾಗುತ್ತದೆ - ವೆಬ್ ಸೈಟ್‌ಗೆ ಹೋಗಿ ಉಚಿತ ಡೆವಲಪರ್ ಯೋಜನೆಗೆ ನೋಂದಣಿ ಮಾಡಿಕೊಳ್ಳಿ.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "ಮೊದಲು, ನಾವು ಸುದ್ದಿಯ ಶೀರ್ಷಿಕೆಗಳಿಂದ ನಾಮಪದಗಳನ್ನು ಹೊರತೆಗೆಯಲು ಸಾಧ್ಯವಾಗಬೇಕು. ಇದಕ್ಕಾಗಿ ನಾವು `TextBlob` ಗ್ರಂಥಾಲಯವನ್ನು ಬಳಸಲಿದ್ದೇವೆ, ಇದು ಇಂತಹ ಸಾಮಾನ್ಯ NLP ಕಾರ್ಯಗಳನ್ನು ಸರಳಗೊಳಿಸುತ್ತದೆ.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" - ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "ನಾವು ನೋಡಬಹುದು ನಾಮಪದಗಳು ನಮಗೆ ದೊಡ್ಡ ಥೀಮ್ಯಾಟಿಕ್ ಗುಂಪುಗಳನ್ನು ನೀಡುವುದಿಲ್ಲ. ನಾವು ನಾಮಪದಗಳನ್ನು ಸಂಪ್ರದಾಯ ಗ್ರಾಫ್‌ನಿಂದ ಪಡೆದ ಹೆಚ್ಚಿನ ಸಾಮಾನ್ಯ ಪದಗಳೊಂದಿಗೆ ಬದಲಾಯಿಸೋಣ. ಇದಕ್ಕೆ ಸ್ವಲ್ಪ ಸಮಯ ಬೇಕಾಗುತ್ತದೆ, ಏಕೆಂದರೆ ನಾವು ಪ್ರತಿ ನಾಮಪದ ವಾಕ್ಯಪದಕ್ಕಾಗಿ REST ಕರೆ ಮಾಡುತ್ತಿದ್ದೇವೆ.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" - ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] - } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n\n\n**ಅಸ್ವೀಕಾರ**: \nಈ ದಸ್ತಾವೇಜು AI ಅನುವಾದ ಸೇವೆ [Co-op Translator](https://github.com/Azure/co-op-translator) ಬಳಸಿ ಅನುವಾದಿಸಲಾಗಿದೆ. ನಾವು ನಿಖರತೆಯಿಗಾಗಿ ಪ್ರಯತ್ನಿಸುತ್ತಿದ್ದರೂ, ಸ್ವಯಂಚಾಲಿತ ಅನುವಾದಗಳಲ್ಲಿ ತಪ್ಪುಗಳು ಅಥವಾ ಅಸತ್ಯತೆಗಳು ಇರಬಹುದು ಎಂದು ದಯವಿಟ್ಟು ಗಮನಿಸಿ. ಮೂಲ ಭಾಷೆಯಲ್ಲಿರುವ ಮೂಲ ದಸ್ತಾವೇಜನ್ನು ಅಧಿಕೃತ ಮೂಲವಾಗಿ ಪರಿಗಣಿಸಬೇಕು. ಪ್ರಮುಖ ಮಾಹಿತಿಗಾಗಿ, ವೃತ್ತಿಪರ ಮಾನವ ಅನುವಾದವನ್ನು ಶಿಫಾರಸು ಮಾಡಲಾಗುತ್ತದೆ. ಈ ಅನುವಾದ ಬಳಕೆಯಿಂದ ಉಂಟಾಗುವ ಯಾವುದೇ ತಪ್ಪು ಅರ್ಥಮಾಡಿಕೊಳ್ಳುವಿಕೆ ಅಥವಾ ತಪ್ಪು ವಿವರಣೆಗಳಿಗೆ ನಾವು ಹೊಣೆಗಾರರಾಗುವುದಿಲ್ಲ.\n\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" - } - }, - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-11-25T23:52:15+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "kn" - } - }, - "nbformat": 4, - "nbformat_minor": 2 +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - ಈ API ಈಗ ಲಭ್ಯವಿಲ್ಲ ಆದರೆ ಸಂಪ್ರದಾಯವನ್ನು ಅರ್ಥಮಾಡಿಕೊಳ್ಳಲು ದಯವಿಟ್ಟು ನೋಟ್ಬುಕ್ ನೋಡಿ\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) ಇಂಟರ್ನೆಟ್‌ನಿಂದ ಸಂಗ್ರಹಿಸಲಾದ ಪದಗಳ ದೊಡ್ಡ ವರ್ಗೀಕರಣವಾಗಿದೆ, ಇದರಲ್ಲಿ ಸಂಪ್ರದಾಯಗಳ ನಡುವೆ `is-a` ಸಂಬಂಧಗಳಿವೆ.\n", + "\n", + "Context Graph ಎರಡು ರೂಪಗಳಲ್ಲಿ ಲಭ್ಯವಿದೆ:\n", + " * ಡೌನ್‌ಲೋಡ್ ಮಾಡಲು ದೊಡ್ಡ ಪಠ್ಯ ಕಡತ\n", + " * REST API\n", + "\n", + "ಸಂಖ್ಯಾಶಾಸ್ತ್ರ:\n", + " * 5401933 ವಿಶಿಷ್ಟ ಸಂಪ್ರದಾಯಗಳು,\n", + " * 12551613 ವಿಶಿಷ್ಟ ಉದಾಹರಣೆಗಳು\n", + " * 87603947 `is-a` ಸಂಬಂಧಗಳು\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## ವೆಬ್ ಸೇವೆಯನ್ನು ಬಳಸುವುದು\n", + "\n", + "ವೆಬ್ ಸೇವೆ ವಿಭಿನ್ನ ಗುಂಪುಗಳಿಗೆ ಸೇರಿದ ಒಂದು ಕಲ್ಪನೆಯ ಸಂಭವನೀಯತೆಯನ್ನು ಅಂದಾಜಿಸಲು ವಿವಿಧ ಕರೆಗಳನ್ನು ನೀಡುತ್ತದೆ. ಹೆಚ್ಚಿನ ಮಾಹಿತಿಗಾಗಿ [ಇಲ್ಲಿ](https://concept.research.microsoft.com/Home/Api) ಲಭ್ಯವಿದೆ. \n", + "ಕರೆಗೆ ಉದಾಹರಣೆಯ URL ಇಲ್ಲಿದೆ: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "ನಾವು ಸುದ್ದಿಯ ಶೀರ್ಷಿಕೆಗಳನ್ನು ಪೋಷಕ ಸಂಪ್ರದಾಯಗಳ ಮೂಲಕ ವರ್ಗೀಕರಿಸಲು ಪ್ರಯತ್ನಿಸೋಣ. ಸುದ್ದಿಯ ಶೀರ್ಷಿಕೆಗಳನ್ನು ಪಡೆಯಲು, ನಾವು [NewsApi.org](http://newsapi.org) ಸೇವೆಯನ್ನು ಬಳಸಲಿದ್ದೇವೆ. ಸೇವೆಯನ್ನು ಬಳಸಲು ನಿಮ್ಮದೇ API ಕೀ ಪಡೆಯಬೇಕಾಗುತ್ತದೆ - ವೆಬ್ ಸೈಟ್‌ಗೆ ಹೋಗಿ ಉಚಿತ ಡೆವಲಪರ್ ಯೋಜನೆಗೆ ನೋಂದಣಿ ಮಾಡಿಕೊಳ್ಳಿ.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "ಮೊದಲು, ನಾವು ಸುದ್ದಿಯ ಶೀರ್ಷಿಕೆಗಳಿಂದ ನಾಮಪದಗಳನ್ನು ಹೊರತೆಗೆಯಲು ಸಾಧ್ಯವಾಗಬೇಕು. ಇದಕ್ಕಾಗಿ ನಾವು `TextBlob` ಗ್ರಂಥಾಲಯವನ್ನು ಬಳಸಲಿದ್ದೇವೆ, ಇದು ಇಂತಹ ಸಾಮಾನ್ಯ NLP ಕಾರ್ಯಗಳನ್ನು ಸರಳಗೊಳಿಸುತ್ತದೆ.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "ನಾವು ನೋಡಬಹುದು ನಾಮಪದಗಳು ನಮಗೆ ದೊಡ್ಡ ಥೀಮ್ಯಾಟಿಕ್ ಗುಂಪುಗಳನ್ನು ನೀಡುವುದಿಲ್ಲ. ನಾವು ನಾಮಪದಗಳನ್ನು ಸಂಪ್ರದಾಯ ಗ್ರಾಫ್‌ನಿಂದ ಪಡೆದ ಹೆಚ್ಚಿನ ಸಾಮಾನ್ಯ ಪದಗಳೊಂದಿಗೆ ಬದಲಾಯಿಸೋಣ. ಇದಕ್ಕೆ ಸ್ವಲ್ಪ ಸಮಯ ಬೇಕಾಗುತ್ತದೆ, ಏಕೆಂದರೆ ನಾವು ಪ್ರತಿ ನಾಮಪದ ವಾಕ್ಯಪದಕ್ಕಾಗಿ REST ಕರೆ ಮಾಡುತ್ತಿದ್ದೇವೆ.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n\n\n**ಅಸ್ವೀಕಾರ**: \nಈ ದಸ್ತಾವೇಜು AI ಅನುವಾದ ಸೇವೆ [Co-op Translator](https://github.com/Azure/co-op-translator) ಬಳಸಿ ಅನುವಾದಿಸಲಾಗಿದೆ. ನಾವು ನಿಖರತೆಯಿಗಾಗಿ ಪ್ರಯತ್ನಿಸುತ್ತಿದ್ದರೂ, ಸ್ವಯಂಚಾಲಿತ ಅನುವಾದಗಳಲ್ಲಿ ತಪ್ಪುಗಳು ಅಥವಾ ಅಸತ್ಯತೆಗಳು ಇರಬಹುದು ಎಂದು ದಯವಿಟ್ಟು ಗಮನಿಸಿ. ಮೂಲ ಭಾಷೆಯಲ್ಲಿರುವ ಮೂಲ ದಸ್ತಾವೇಜನ್ನು ಅಧಿಕೃತ ಮೂಲವಾಗಿ ಪರಿಗಣಿಸಬೇಕು. ಪ್ರಮುಖ ಮಾಹಿತಿಗಾಗಿ, ವೃತ್ತಿಪರ ಮಾನವ ಅನುವಾದವನ್ನು ಶಿಫಾರಸು ಮಾಡಲಾಗುತ್ತದೆ. ಈ ಅನುವಾದ ಬಳಕೆಯಿಂದ ಉಂಟಾಗುವ ಯಾವುದೇ ತಪ್ಪು ಅರ್ಥಮಾಡಿಕೊಳ್ಳುವಿಕೆ ಅಥವಾ ತಪ್ಪು ವಿವರಣೆಗಳಿಗೆ ನಾವು ಹೊಣೆಗಾರರಾಗುವುದಿಲ್ಲ.\n\n" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-11-25T23:52:15+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "kn" + } + }, + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/ko/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/ko/lessons/2-Symbolic/MSConceptGraph.ipynb index fb65a38d..72ed0d74 100644 --- a/translations/ko/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/ko/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - 이 API는 더 이상 사용 불가능하지만 개념을 이해하려면 노트북을 확인하세요\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/)는 인터넷에서 추출된 용어의 대규모 분류 체계로, 개념 간의 `is-a` 관계를 포함하고 있습니다.\n", - "\n", - "Context Graph는 두 가지 형태로 제공됩니다:\n", - " * 다운로드 가능한 대규모 텍스트 파일\n", - " * REST API\n", - "\n", - "통계:\n", - " * 5401933개의 고유 개념\n", - " * 12551613개의 고유 인스턴스\n", - " * 87603947개의 `is-a` 관계\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 웹 서비스 사용하기\n", - "\n", - "웹 서비스는 특정 개념이 다양한 그룹에 속할 확률을 추정하기 위한 여러 호출을 제공합니다. 자세한 정보는 [여기](https://concept.research.microsoft.com/Home/Api)에서 확인할 수 있습니다. \n", - "다음은 호출을 위한 샘플 URL입니다: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - 이 API는 더 이상 사용 불가능하지만 개념을 이해하려면 노트북을 확인하세요\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/)는 인터넷에서 추출된 용어의 대규모 분류 체계로, 개념 간의 `is-a` 관계를 포함하고 있습니다.\n", + "\n", + "Context Graph는 두 가지 형태로 제공됩니다:\n", + " * 다운로드 가능한 대규모 텍스트 파일\n", + " * REST API\n", + "\n", + "통계:\n", + " * 5401933개의 고유 개념\n", + " * 12551613개의 고유 인스턴스\n", + " * 87603947개의 `is-a` 관계\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "뉴스 제목을 상위 개념으로 분류해 봅시다. 뉴스 제목을 얻기 위해 [NewsApi.org](http://newsapi.org) 서비스를 사용할 것입니다. 서비스를 사용하려면 자체 API 키를 얻어야 합니다. 웹사이트에 방문하여 무료 개발자 플랜에 등록하세요.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "우선, 우리는 뉴스 제목에서 명사를 추출할 수 있기를 원합니다. 이를 위해 `TextBlob` 라이브러리를 사용할 것이며, 이는 이러한 일반적인 NLP 작업을 크게 간소화합니다.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 웹 서비스 사용하기\n", + "\n", + "웹 서비스는 특정 개념이 다양한 그룹에 속할 확률을 추정하기 위한 여러 호출을 제공합니다. 자세한 정보는 [여기](https://concept.research.microsoft.com/Home/Api)에서 확인할 수 있습니다. \n", + "다음은 호출을 위한 샘플 URL입니다: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "우리는 명사가 큰 주제 그룹을 제공하지 않는다는 것을 알 수 있습니다. 개념 그래프에서 얻은 더 일반적인 용어로 명사를 대체해 봅시다. 이것은 약간의 시간이 걸릴 것입니다. 왜냐하면 각 명사 구에 대해 REST 호출을 수행하고 있기 때문입니다.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "뉴스 제목을 상위 개념으로 분류해 봅시다. 뉴스 제목을 얻기 위해 [NewsApi.org](http://newsapi.org) 서비스를 사용할 것입니다. 서비스를 사용하려면 자체 API 키를 얻어야 합니다. 웹사이트에 방문하여 무료 개발자 플랜에 등록하세요.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "우선, 우리는 뉴스 제목에서 명사를 추출할 수 있기를 원합니다. 이를 위해 `TextBlob` 라이브러리를 사용할 것이며, 이는 이러한 일반적인 NLP 작업을 크게 간소화합니다.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "우리는 명사가 큰 주제 그룹을 제공하지 않는다는 것을 알 수 있습니다. 개념 그래프에서 얻은 더 일반적인 용어로 명사를 대체해 봅시다. 이것은 약간의 시간이 걸릴 것입니다. 왜냐하면 각 명사 구에 대해 REST 호출을 수행하고 있기 때문입니다.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**면책 조항**: \n이 문서는 AI 번역 서비스 [Co-op Translator](https://github.com/Azure/co-op-translator)를 사용하여 번역되었습니다. 정확성을 위해 최선을 다하고 있으나, 자동 번역에는 오류나 부정확성이 포함될 수 있습니다. 원본 문서의 원어 버전이 권위 있는 출처로 간주되어야 합니다. 중요한 정보의 경우, 전문적인 인간 번역을 권장합니다. 이 번역 사용으로 인해 발생하는 오해나 잘못된 해석에 대해 당사는 책임을 지지 않습니다.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**면책 조항**: \n이 문서는 AI 번역 서비스 [Co-op Translator](https://github.com/Azure/co-op-translator)를 사용하여 번역되었습니다. 정확성을 위해 최선을 다하고 있으나, 자동 번역에는 오류나 부정확성이 포함될 수 있습니다. 원본 문서의 원어 버전이 권위 있는 출처로 간주되어야 합니다. 중요한 정보의 경우, 전문적인 인간 번역을 권장합니다. 이 번역 사용으로 인해 발생하는 오해나 잘못된 해석에 대해 당사는 책임을 지지 않습니다.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:52:23+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "ko" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:52:23+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "ko" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/lt/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/lt/lessons/2-Symbolic/MSConceptGraph.ipynb index 9d62fda8..075125a0 100644 --- a/translations/lt/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/lt/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,547 +1,538 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - Ši API NEBEPASIEKIAMA, BET PRAŠOME PERŽIŪRĖTI UŽRAŠŲ KNYGĄ, KAD SUPRASTUMĖTE KONCEPCIJĄ\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) yra didelė terminų taksonomija, surinkta iš interneto, su `is-a` ryšiais tarp koncepcijų.\n", - "\n", - "Context Graph yra prieinamas dviem formomis:\n", - " * Didelis tekstinis failas atsisiuntimui\n", - " * REST API\n", - "\n", - "Statistika:\n", - " * 5401933 unikalios koncepcijos,\n", - " * 12551613 unikalūs pavyzdžiai\n", - " * 87603947 `is-a` ryšiai\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Naudojimasis internetine paslauga\n", - "\n", - "Internetinė paslauga siūlo įvairius užklausų tipus, leidžiančius įvertinti tikimybę, kad tam tikra sąvoka priklauso skirtingoms grupėms. Daugiau informacijos rasite [čia](https://concept.research.microsoft.com/Home/Api). Štai pavyzdinis URL užklausai: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - Ši API NEBEPASIEKIAMA, BET PRAŠOME PERŽIŪRĖTI UŽRAŠŲ KNYGĄ, KAD SUPRASTUMĖTE KONCEPCIJĄ\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) yra didelė terminų taksonomija, surinkta iš interneto, su `is-a` ryšiais tarp koncepcijų.\n", + "\n", + "Context Graph yra prieinamas dviem formomis:\n", + " * Didelis tekstinis failas atsisiuntimui\n", + " * REST API\n", + "\n", + "Statistika:\n", + " * 5401933 unikalios koncepcijos,\n", + " * 12551613 unikalūs pavyzdžiai\n", + " * 87603947 `is-a` ryšiai\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Pabandykime suskirstyti naujienų antraštes pagal pagrindines sąvokas. Norėdami gauti naujienų antraštes, naudosime [NewsApi.org](http://newsapi.org) paslaugą. Jums reikės gauti savo API raktą, kad galėtumėte naudotis paslauga - eikite į svetainę ir užsiregistruokite nemokamam kūrėjo planui.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Visų pirma, norime išgauti daiktavardžius iš naujienų antraščių. Tam naudosime biblioteką `TextBlob`, kuri labai supaprastina daugelį tipinių NLP užduočių, tokių kaip ši.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Naudojimasis internetine paslauga\n", + "\n", + "Internetinė paslauga siūlo įvairius užklausų tipus, leidžiančius įvertinti tikimybę, kad tam tikra sąvoka priklauso skirtingoms grupėms. Daugiau informacijos rasite [čia](https://concept.research.microsoft.com/Home/Api). Štai pavyzdinis URL užklausai: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Matome, kad daiktavardžiai nesuteikia mums didelių teminių grupių. Pakeiskime daiktavardžius bendresniais terminais, gautais iš koncepcijų grafiko. Tai užtruks šiek tiek laiko, nes kiekvienai daiktavardžio frazei atliekame REST užklausą.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Pabandykime suskirstyti naujienų antraštes pagal pagrindines sąvokas. Norėdami gauti naujienų antraštes, naudosime [NewsApi.org](http://newsapi.org) paslaugą. Jums reikės gauti savo API raktą, kad galėtumėte naudotis paslauga - eikite į svetainę ir užsiregistruokite nemokamam kūrėjo planui.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Visų pirma, norime išgauti daiktavardžius iš naujienų antraščių. Tam naudosime biblioteką `TextBlob`, kuri labai supaprastina daugelį tipinių NLP užduočių, tokių kaip ši.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Matome, kad daiktavardžiai nesuteikia mums didelių teminių grupių. Pakeiskime daiktavardžius bendresniais terminais, gautais iš koncepcijų grafiko. Tai užtruks šiek tiek laiko, nes kiekvienai daiktavardžio frazei atliekame REST užklausą.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**Atsakomybės atsisakymas**: \nŠis dokumentas buvo išverstas naudojant AI vertimo paslaugą [Co-op Translator](https://github.com/Azure/co-op-translator). Nors stengiamės užtikrinti tikslumą, prašome atkreipti dėmesį, kad automatiniai vertimai gali turėti klaidų ar netikslumų. Originalus dokumentas jo gimtąja kalba turėtų būti laikomas autoritetingu šaltiniu. Kritinei informacijai rekomenduojama naudoti profesionalų žmogaus vertimą. Mes neprisiimame atsakomybės už nesusipratimus ar neteisingus interpretavimus, atsiradusius dėl šio vertimo naudojimo.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**Atsakomybės atsisakymas**: \nŠis dokumentas buvo išverstas naudojant AI vertimo paslaugą [Co-op Translator](https://github.com/Azure/co-op-translator). Nors stengiamės užtikrinti tikslumą, prašome atkreipti dėmesį, kad automatiniai vertimai gali turėti klaidų ar netikslumų. Originalus dokumentas jo gimtąja kalba turėtų būti laikomas autoritetingu šaltiniu. Kritinei informacijai rekomenduojama naudoti profesionalų žmogaus vertimą. Mes neprisiimame atsakomybės už nesusipratimus ar neteisingus interpretavimus, atsiradusius dėl šio vertimo naudojimo.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T19:00:19+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "lt" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T19:00:19+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "lt" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/ml/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/ml/lessons/2-Symbolic/MSConceptGraph.ipynb index 9648bec6..05d4e4a1 100644 --- a/translations/ml/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/ml/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - ഈ API ഇനി ലഭ്യമല്ല, പക്ഷേ ആശയം മനസിലാക്കാൻ നോട്ട്‌ബുക്ക് കാണുക\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) ഇന്റർനെറ്റിൽ നിന്നുള്ള വാക്കുകളുടെ വലിയ ടാക്സോണോമിയാണ്, ആശയങ്ങൾ തമ്മിലുള്ള `is-a` ബന്ധങ്ങളോടുകൂടി.\n", - "\n", - "Context Graph രണ്ട് രൂപങ്ങളിൽ ലഭ്യമാണ്:\n", - " * ഡൗൺലോഡിനായി വലിയ ടെക്സ്റ്റ് ഫയൽ\n", - " * REST API\n", - "\n", - "സ്ഥിതിവിവരക്കണക്കുകൾ:\n", - " * 5401933 വ്യത്യസ്ത ആശയങ്ങൾ,\n", - " * 12551613 വ്യത്യസ്ത ഉദാഹരണങ്ങൾ\n", - " * 87603947 `is-a` ബന്ധങ്ങൾ\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## വെബ് സർവീസ് ഉപയോഗിക്കൽ\n", - "\n", - "വെബ് സർവീസ് ഒരു ആശയം വിവിധ ഗ്രൂപ്പുകളിൽ പെട്ടിരിക്കാനുള്ള സാധ്യത കണക്കാക്കാൻ വിവിധ കോൾസുകൾ നൽകുന്നു. കൂടുതൽ വിവരങ്ങൾ ലഭ്യമാണ് [ഇവിടെ](https://concept.research.microsoft.com/Home/Api).\n", - "കോൾ ചെയ്യാനുള്ള സാമ്പിൾ URL ഇതാണ്: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" - ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "വാർത്താ തലക്കെട്ടുകൾ മാതൃ ആശയങ്ങൾ ഉപയോഗിച്ച് വർഗ്ഗീകരിക്കാൻ ശ്രമിക്കാം. വാർത്താ തലക്കെട്ടുകൾ നേടാൻ, നാം [NewsApi.org](http://newsapi.org) സേവനം ഉപയോഗിക്കും. സേവനം ഉപയോഗിക്കാൻ നിങ്ങളുടെ സ്വന്തം API കീ ലഭിക്കേണ്ടതാണ് - വെബ്‌സൈറ്റിൽ പോയി സൗജന്യ ഡെവലപ്പർ പ്ലാനിൽ രജിസ്റ്റർ ചെയ്യുക.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "ആദ്യമായി, നാം വാർത്താ തലക്കെട്ടുകളിൽ നിന്നുള്ള നാമപദങ്ങൾ എടുക്കാൻ കഴിയണം. ഇതിന് നാം `TextBlob` ലൈബ്രറി ഉപയോഗിക്കും, ഇത് ഇത്തരത്തിലുള്ള പല സാധാരണ NLP പ്രവർത്തനങ്ങളും ലളിതമാക്കുന്നു.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" - ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "നാം കാണുന്നത് നാമങ്ങൾ വലിയ താത്വിക ഗ്രൂപ്പുകൾ നൽകുന്നില്ല എന്നതാണ്. നാമങ്ങളെ ആശയ ഗ്രാഫിൽ നിന്നുള്ള കൂടുതൽ പൊതുവായ പദങ്ങളാൽ മാറ്റി വയ്ക്കാം. ഓരോ നാമവാക്യത്തിനും REST കോൾ ചെയ്യേണ്ടതായതിനാൽ ഇത് കുറച്ച് സമയം എടുക്കും.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" - ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] - } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n\n\n**അസൂയാ**: \nഈ രേഖ AI വിവർത്തന സേവനം [Co-op Translator](https://github.com/Azure/co-op-translator) ഉപയോഗിച്ച് വിവർത്തനം ചെയ്തതാണ്. നാം കൃത്യതയ്ക്ക് ശ്രമിച്ചെങ്കിലും, സ്വയം പ്രവർത്തിക്കുന്ന വിവർത്തനങ്ങളിൽ പിശകുകൾ അല്ലെങ്കിൽ തെറ്റുകൾ ഉണ്ടാകാമെന്ന് ദയവായി ശ്രദ്ധിക്കുക. അതിന്റെ മാതൃഭാഷയിലുള്ള യഥാർത്ഥ രേഖയാണ് പ്രാമാണികമായ ഉറവിടം എന്ന് പരിഗണിക്കേണ്ടതാണ്. നിർണായക വിവരങ്ങൾക്ക്, പ്രൊഫഷണൽ മനുഷ്യ വിവർത്തനം ശുപാർശ ചെയ്യപ്പെടുന്നു. ഈ വിവർത്തനം ഉപയോഗിക്കുന്നതിൽ നിന്നുണ്ടാകുന്ന ഏതെങ്കിലും തെറ്റിദ്ധാരണകൾക്കോ തെറ്റായ വ്യാഖ്യാനങ്ങൾക്കോ ഞങ്ങൾ ഉത്തരവാദികളല്ല.\n\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" - } - }, - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-11-25T23:51:55+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "ml" - } - }, - "nbformat": 4, - "nbformat_minor": 2 +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - ഈ API ഇനി ലഭ്യമല്ല, പക്ഷേ ആശയം മനസിലാക്കാൻ നോട്ട്‌ബുക്ക് കാണുക\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) ഇന്റർനെറ്റിൽ നിന്നുള്ള വാക്കുകളുടെ വലിയ ടാക്സോണോമിയാണ്, ആശയങ്ങൾ തമ്മിലുള്ള `is-a` ബന്ധങ്ങളോടുകൂടി.\n", + "\n", + "Context Graph രണ്ട് രൂപങ്ങളിൽ ലഭ്യമാണ്:\n", + " * ഡൗൺലോഡിനായി വലിയ ടെക്സ്റ്റ് ഫയൽ\n", + " * REST API\n", + "\n", + "സ്ഥിതിവിവരക്കണക്കുകൾ:\n", + " * 5401933 വ്യത്യസ്ത ആശയങ്ങൾ,\n", + " * 12551613 വ്യത്യസ്ത ഉദാഹരണങ്ങൾ\n", + " * 87603947 `is-a` ബന്ധങ്ങൾ\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## വെബ് സർവീസ് ഉപയോഗിക്കൽ\n", + "\n", + "വെബ് സർവീസ് ഒരു ആശയം വിവിധ ഗ്രൂപ്പുകളിൽ പെട്ടിരിക്കാനുള്ള സാധ്യത കണക്കാക്കാൻ വിവിധ കോൾസുകൾ നൽകുന്നു. കൂടുതൽ വിവരങ്ങൾ ലഭ്യമാണ് [ഇവിടെ](https://concept.research.microsoft.com/Home/Api).\n", + "കോൾ ചെയ്യാനുള്ള സാമ്പിൾ URL ഇതാണ്: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "വാർത്താ തലക്കെട്ടുകൾ മാതൃ ആശയങ്ങൾ ഉപയോഗിച്ച് വർഗ്ഗീകരിക്കാൻ ശ്രമിക്കാം. വാർത്താ തലക്കെട്ടുകൾ നേടാൻ, നാം [NewsApi.org](http://newsapi.org) സേവനം ഉപയോഗിക്കും. സേവനം ഉപയോഗിക്കാൻ നിങ്ങളുടെ സ്വന്തം API കീ ലഭിക്കേണ്ടതാണ് - വെബ്‌സൈറ്റിൽ പോയി സൗജന്യ ഡെവലപ്പർ പ്ലാനിൽ രജിസ്റ്റർ ചെയ്യുക.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "ആദ്യമായി, നാം വാർത്താ തലക്കെട്ടുകളിൽ നിന്നുള്ള നാമപദങ്ങൾ എടുക്കാൻ കഴിയണം. ഇതിന് നാം `TextBlob` ലൈബ്രറി ഉപയോഗിക്കും, ഇത് ഇത്തരത്തിലുള്ള പല സാധാരണ NLP പ്രവർത്തനങ്ങളും ലളിതമാക്കുന്നു.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "നാം കാണുന്നത് നാമങ്ങൾ വലിയ താത്വിക ഗ്രൂപ്പുകൾ നൽകുന്നില്ല എന്നതാണ്. നാമങ്ങളെ ആശയ ഗ്രാഫിൽ നിന്നുള്ള കൂടുതൽ പൊതുവായ പദങ്ങളാൽ മാറ്റി വയ്ക്കാം. ഓരോ നാമവാക്യത്തിനും REST കോൾ ചെയ്യേണ്ടതായതിനാൽ ഇത് കുറച്ച് സമയം എടുക്കും.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n\n\n**അസൂയാ**: \nഈ രേഖ AI വിവർത്തന സേവനം [Co-op Translator](https://github.com/Azure/co-op-translator) ഉപയോഗിച്ച് വിവർത്തനം ചെയ്തതാണ്. നാം കൃത്യതയ്ക്ക് ശ്രമിച്ചെങ്കിലും, സ്വയം പ്രവർത്തിക്കുന്ന വിവർത്തനങ്ങളിൽ പിശകുകൾ അല്ലെങ്കിൽ തെറ്റുകൾ ഉണ്ടാകാമെന്ന് ദയവായി ശ്രദ്ധിക്കുക. അതിന്റെ മാതൃഭാഷയിലുള്ള യഥാർത്ഥ രേഖയാണ് പ്രാമാണികമായ ഉറവിടം എന്ന് പരിഗണിക്കേണ്ടതാണ്. നിർണായക വിവരങ്ങൾക്ക്, പ്രൊഫഷണൽ മനുഷ്യ വിവർത്തനം ശുപാർശ ചെയ്യപ്പെടുന്നു. ഈ വിവർത്തനം ഉപയോഗിക്കുന്നതിൽ നിന്നുണ്ടാകുന്ന ഏതെങ്കിലും തെറ്റിദ്ധാരണകൾക്കോ തെറ്റായ വ്യാഖ്യാനങ്ങൾക്കോ ഞങ്ങൾ ഉത്തരവാദികളല്ല.\n\n" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-11-25T23:51:55+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "ml" + } + }, + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/mr/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/mr/lessons/2-Symbolic/MSConceptGraph.ipynb index 7d451911..3b82ea92 100644 --- a/translations/mr/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/mr/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - हा API आता उपलब्ध नाही, परंतु संकल्पना समजून घेण्यासाठी कृपया नोटबुक पहा\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) ही इंटरनेटवरून संकलित केलेल्या संज्ञांची मोठी वर्गवारी आहे, ज्यामध्ये संकल्पनांमधील `is-a` संबंध आहेत.\n", - "\n", - "Context Graph दोन प्रकारांमध्ये उपलब्ध आहे:\n", - " * डाउनलोडसाठी मोठी मजकूर फाइल\n", - " * REST API\n", - "\n", - "आकडेवारी:\n", - " * 5401933 अद्वितीय संकल्पना,\n", - " * 12551613 अद्वितीय उदाहरणे\n", - " * 87603947 `is-a` संबंध\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## वेब सेवा वापरणे\n", - "\n", - "वेब सेवा विविध गटांमध्ये एखाद्या संकल्पनेची शक्यता अंदाज करण्यासाठी वेगवेगळ्या कॉल्स ऑफर करते. अधिक माहिती [येथे](https://concept.research.microsoft.com/Home/Api) उपलब्ध आहे. \n", - "कॉल करण्यासाठी नमुना URL येथे आहे: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - हा API आता उपलब्ध नाही, परंतु संकल्पना समजून घेण्यासाठी कृपया नोटबुक पहा\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) ही इंटरनेटवरून संकलित केलेल्या संज्ञांची मोठी वर्गवारी आहे, ज्यामध्ये संकल्पनांमधील `is-a` संबंध आहेत.\n", + "\n", + "Context Graph दोन प्रकारांमध्ये उपलब्ध आहे:\n", + " * डाउनलोडसाठी मोठी मजकूर फाइल\n", + " * REST API\n", + "\n", + "आकडेवारी:\n", + " * 5401933 अद्वितीय संकल्पना,\n", + " * 12551613 अद्वितीय उदाहरणे\n", + " * 87603947 `is-a` संबंध\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "चला बातम्यांच्या शीर्षकांना पालक संकल्पनांचा वापर करून वर्गीकृत करण्याचा प्रयत्न करूया. बातम्यांची शीर्षके मिळवण्यासाठी, आपण [NewsApi.org](http://newsapi.org) सेवा वापरू. ही सेवा वापरण्यासाठी तुम्हाला स्वतःचा API की मिळवावा लागेल - वेबसाइटवर जा आणि मोफत डेव्हलपर प्लॅनसाठी नोंदणी करा.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "सर्वप्रथम, आम्हाला बातम्यांच्या शीर्षकांमधून नामे काढता यावीत असे आम्हाला हवे आहे. यासाठी आम्ही `TextBlob` लायब्ररीचा वापर करू, जी अशा प्रकारच्या सामान्य NLP कार्यांना सोपे करते.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## वेब सेवा वापरणे\n", + "\n", + "वेब सेवा विविध गटांमध्ये एखाद्या संकल्पनेची शक्यता अंदाज करण्यासाठी वेगवेगळ्या कॉल्स ऑफर करते. अधिक माहिती [येथे](https://concept.research.microsoft.com/Home/Api) उपलब्ध आहे. \n", + "कॉल करण्यासाठी नमुना URL येथे आहे: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "आम्ही पाहू शकतो की नामे आम्हाला मोठ्या थीमॅटिक गट देत नाहीत. चला नामे अधिक सामान्य संज्ञांनी बदलूया, ज्या संकल्पना ग्राफमधून प्राप्त केल्या आहेत. यासाठी काही वेळ लागेल, कारण आम्ही प्रत्येक नाम वाक्यांशासाठी REST कॉल करत आहोत.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "चला बातम्यांच्या शीर्षकांना पालक संकल्पनांचा वापर करून वर्गीकृत करण्याचा प्रयत्न करूया. बातम्यांची शीर्षके मिळवण्यासाठी, आपण [NewsApi.org](http://newsapi.org) सेवा वापरू. ही सेवा वापरण्यासाठी तुम्हाला स्वतःचा API की मिळवावा लागेल - वेबसाइटवर जा आणि मोफत डेव्हलपर प्लॅनसाठी नोंदणी करा.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "सर्वप्रथम, आम्हाला बातम्यांच्या शीर्षकांमधून नामे काढता यावीत असे आम्हाला हवे आहे. यासाठी आम्ही `TextBlob` लायब्ररीचा वापर करू, जी अशा प्रकारच्या सामान्य NLP कार्यांना सोपे करते.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "आम्ही पाहू शकतो की नामे आम्हाला मोठ्या थीमॅटिक गट देत नाहीत. चला नामे अधिक सामान्य संज्ञांनी बदलूया, ज्या संकल्पना ग्राफमधून प्राप्त केल्या आहेत. यासाठी काही वेळ लागेल, कारण आम्ही प्रत्येक नाम वाक्यांशासाठी REST कॉल करत आहोत.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**अस्वीकरण**: \nहा दस्तऐवज AI भाषांतर सेवा [Co-op Translator](https://github.com/Azure/co-op-translator) वापरून भाषांतरित करण्यात आला आहे. आम्ही अचूकतेसाठी प्रयत्नशील असलो तरी, कृपया लक्षात ठेवा की स्वयंचलित भाषांतरांमध्ये त्रुटी किंवा अचूकतेचा अभाव असू शकतो. मूळ भाषेतील दस्तऐवज हा अधिकृत स्रोत मानला जावा. महत्त्वाच्या माहितीसाठी, व्यावसायिक मानवी भाषांतराची शिफारस केली जाते. या भाषांतराचा वापर करून उद्भवलेल्या कोणत्याही गैरसमज किंवा चुकीच्या अर्थासाठी आम्ही जबाबदार राहणार नाही.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**अस्वीकरण**: \nहा दस्तऐवज AI भाषांतर सेवा [Co-op Translator](https://github.com/Azure/co-op-translator) वापरून भाषांतरित करण्यात आला आहे. आम्ही अचूकतेसाठी प्रयत्नशील असलो तरी, कृपया लक्षात ठेवा की स्वयंचलित भाषांतरांमध्ये त्रुटी किंवा अचूकतेचा अभाव असू शकतो. मूळ भाषेतील दस्तऐवज हा अधिकृत स्रोत मानला जावा. महत्त्वाच्या माहितीसाठी, व्यावसायिक मानवी भाषांतराची शिफारस केली जाते. या भाषांतराचा वापर करून उद्भवलेल्या कोणत्याही गैरसमज किंवा चुकीच्या अर्थासाठी आम्ही जबाबदार राहणार नाही.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:53:05+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "mr" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:53:05+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "mr" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/ms/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/ms/lessons/2-Symbolic/MSConceptGraph.ipynb index c7d655e4..1176004b 100644 --- a/translations/ms/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/ms/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,547 +1,538 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - API INI TIDAK LAGI TERSEDIA TETAPI SILA LIHAT NOTA UNTUK MEMAHAMI KONSEP\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) adalah taksonomi besar bagi istilah yang diperoleh dari internet, dengan hubungan `is-a` antara konsep.\n", - "\n", - "Context Graph tersedia dalam dua bentuk:\n", - " * Fail teks besar untuk dimuat turun\n", - " * REST API\n", - "\n", - "Statistik:\n", - " * 5401933 konsep unik,\n", - " * 12551613 contoh unik\n", - " * 87603947 hubungan `is-a`\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Menggunakan Perkhidmatan Web\n", - "\n", - "Perkhidmatan web menawarkan pelbagai panggilan untuk menganggarkan kebarangkalian sesuatu konsep tergolong dalam kumpulan yang berbeza. Maklumat lanjut boleh didapati [di sini](https://concept.research.microsoft.com/Home/Api). Berikut adalah contoh URL untuk panggilan: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - API INI TIDAK LAGI TERSEDIA TETAPI SILA LIHAT NOTA UNTUK MEMAHAMI KONSEP\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) adalah taksonomi besar bagi istilah yang diperoleh dari internet, dengan hubungan `is-a` antara konsep.\n", + "\n", + "Context Graph tersedia dalam dua bentuk:\n", + " * Fail teks besar untuk dimuat turun\n", + " * REST API\n", + "\n", + "Statistik:\n", + " * 5401933 konsep unik,\n", + " * 12551613 contoh unik\n", + " * 87603947 hubungan `is-a`\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Mari kita cuba mengkategorikan tajuk berita menggunakan konsep induk. Untuk mendapatkan tajuk berita, kita akan menggunakan perkhidmatan [NewsApi.org](http://newsapi.org). Anda perlu mendapatkan kunci API anda sendiri untuk menggunakan perkhidmatan ini - pergi ke laman web dan daftar untuk pelan pembangun percuma.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Pertama sekali, kami ingin dapat mengekstrak kata nama daripada tajuk berita. Kami akan menggunakan perpustakaan `TextBlob` untuk melakukan ini, yang mempermudahkan banyak tugas NLP biasa seperti ini.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Menggunakan Perkhidmatan Web\n", + "\n", + "Perkhidmatan web menawarkan pelbagai panggilan untuk menganggarkan kebarangkalian sesuatu konsep tergolong dalam kumpulan yang berbeza. Maklumat lanjut boleh didapati [di sini](https://concept.research.microsoft.com/Home/Api). Berikut adalah contoh URL untuk panggilan: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Kita dapat melihat bahawa kata nama tidak memberikan kita kumpulan tematik yang besar. Mari kita gantikan kata nama dengan istilah yang lebih umum yang diperoleh daripada graf konsep. Ini akan mengambil masa, kerana kita sedang melakukan panggilan REST untuk setiap frasa kata nama.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Mari kita cuba mengkategorikan tajuk berita menggunakan konsep induk. Untuk mendapatkan tajuk berita, kita akan menggunakan perkhidmatan [NewsApi.org](http://newsapi.org). Anda perlu mendapatkan kunci API anda sendiri untuk menggunakan perkhidmatan ini - pergi ke laman web dan daftar untuk pelan pembangun percuma.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Pertama sekali, kami ingin dapat mengekstrak kata nama daripada tajuk berita. Kami akan menggunakan perpustakaan `TextBlob` untuk melakukan ini, yang mempermudahkan banyak tugas NLP biasa seperti ini.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Kita dapat melihat bahawa kata nama tidak memberikan kita kumpulan tematik yang besar. Mari kita gantikan kata nama dengan istilah yang lebih umum yang diperoleh daripada graf konsep. Ini akan mengambil masa, kerana kita sedang melakukan panggilan REST untuk setiap frasa kata nama.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**Penafian**: \nDokumen ini telah diterjemahkan menggunakan perkhidmatan terjemahan AI [Co-op Translator](https://github.com/Azure/co-op-translator). Walaupun kami berusaha untuk memastikan ketepatan, sila ambil perhatian bahawa terjemahan automatik mungkin mengandungi kesilapan atau ketidaktepatan. Dokumen asal dalam bahasa asalnya harus dianggap sebagai sumber yang berwibawa. Untuk maklumat yang kritikal, terjemahan manusia profesional adalah disyorkan. Kami tidak bertanggungjawab atas sebarang salah faham atau salah tafsir yang timbul daripada penggunaan terjemahan ini.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**Penafian**: \nDokumen ini telah diterjemahkan menggunakan perkhidmatan terjemahan AI [Co-op Translator](https://github.com/Azure/co-op-translator). Walaupun kami berusaha untuk memastikan ketepatan, sila ambil perhatian bahawa terjemahan automatik mungkin mengandungi kesilapan atau ketidaktepatan. Dokumen asal dalam bahasa asalnya harus dianggap sebagai sumber yang berwibawa. Untuk maklumat yang kritikal, terjemahan manusia profesional adalah disyorkan. Kami tidak bertanggungjawab atas sebarang salah faham atau salah tafsir yang timbul daripada penggunaan terjemahan ini.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:57:10+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "ms" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:57:10+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "ms" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/my/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/my/lessons/2-Symbolic/MSConceptGraph.ipynb index 1c059f14..96b65052 100644 --- a/translations/my/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/my/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - ဒီ API ကို မရရှိနိုင်တော့ပါ၊ သို့သော် Concept ကို နားလည်ရန် Notebook ကို ကြည့်ပါ\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) သည် အင်တာနက်မှ ရှာဖွေထားသော စကားလုံးများ၏ အကြီးမားဆုံး အမျိုးအစားခွဲခြားမှုဖြစ်ပြီး၊ `is-a` ဆက်နွယ်မှုများကို Concept များအကြားတွင် ရှိသည်။\n", - "\n", - "Context Graph ကို အောက်ပါအမျိုးအစားနှစ်မျိုးဖြင့် ရရှိနိုင်ပါသည်။\n", - " * ဒေါင်းလုဒ်လုပ်ရန် အကြီးမားသော စာသားဖိုင်\n", - " * REST API\n", - "\n", - "အချက်အလက်များ:\n", - " * 5401933 ထူးခြားသော Concept များ\n", - " * 12551613 ထူးခြားသော Instance များ\n", - " * 87603947 `is-a` ဆက်နွယ်မှုများ\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## ဝက်ဘ်ဝန်ဆောင်မှုကို အသုံးပြုခြင်း\n", - "\n", - "ဝက်ဘ်ဝန်ဆောင်မှုသည် အမျိုးမျိုးသောအဖွဲ့များနှင့် ဆိုင်သော အယူအဆတစ်ခု၏ ဖြစ်နိုင်ခြေကို ခန့်မှန်းရန် ခေါ်ဆိုမှုများကို ပေးသည်။ အခြားအချက်အလက်များကို [ဒီမှာ](https://concept.research.microsoft.com/Home/Api) တွေ့နိုင်ပါသည်။\n", - "ခေါ်ဆိုရန် နမူနာ URL သည် `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10` ဖြစ်သည်။\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - ဒီ API ကို မရရှိနိုင်တော့ပါ၊ သို့သော် Concept ကို နားလည်ရန် Notebook ကို ကြည့်ပါ\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) သည် အင်တာနက်မှ ရှာဖွေထားသော စကားလုံးများ၏ အကြီးမားဆုံး အမျိုးအစားခွဲခြားမှုဖြစ်ပြီး၊ `is-a` ဆက်နွယ်မှုများကို Concept များအကြားတွင် ရှိသည်။\n", + "\n", + "Context Graph ကို အောက်ပါအမျိုးအစားနှစ်မျိုးဖြင့် ရရှိနိုင်ပါသည်။\n", + " * ဒေါင်းလုဒ်လုပ်ရန် အကြီးမားသော စာသားဖိုင်\n", + " * REST API\n", + "\n", + "အချက်အလက်များ:\n", + " * 5401933 ထူးခြားသော Concept များ\n", + " * 12551613 ထူးခြားသော Instance များ\n", + " * 87603947 `is-a` ဆက်နွယ်မှုများ\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "သတင်းခေါင်းစဉ်များကို မိဘအယူအဆများဖြင့် အမျိုးအစားခွဲကြည့်ရအောင် လုပ်ကြမယ်။ သတင်းခေါင်းစဉ်များရယူရန် [NewsApi.org](http://newsapi.org) ဝန်ဆောင်မှုကို အသုံးပြုမယ်။ ဝန်ဆောင်မှုကို အသုံးပြုရန် သင့်ကိုယ်ပိုင် API key ရယူရန် လိုအပ်ပါမယ် - ဝက်ဘ်ဆိုဒ်ကို သွားပြီး အခမဲ့ developer plan အတွက် မှတ်ပုံတင်ပါ။\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "ပထမဆုံးအနေနဲ့၊ သတင်းခေါင်းစဉ်တွေထဲက နာမ်တွေကို ထုတ်ယူနိုင်ဖို့လိုပါတယ်။ ဒါကိုလုပ်ဖို့အတွက် `TextBlob` စာကြောင်းတိုက် library ကို အသုံးပြုမယ်၊ အဲဒါက ဒီလိုပုံမှန် NLP အလုပ်တွေကို လွယ်ကူစေပါတယ်။\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## ဝက်ဘ်ဝန်ဆောင်မှုကို အသုံးပြုခြင်း\n", + "\n", + "ဝက်ဘ်ဝန်ဆောင်မှုသည် အမျိုးမျိုးသောအဖွဲ့များနှင့် ဆိုင်သော အယူအဆတစ်ခု၏ ဖြစ်နိုင်ခြေကို ခန့်မှန်းရန် ခေါ်ဆိုမှုများကို ပေးသည်။ အခြားအချက်အလက်များကို [ဒီမှာ](https://concept.research.microsoft.com/Home/Api) တွေ့နိုင်ပါသည်။\n", + "ခေါ်ဆိုရန် နမူနာ URL သည် `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10` ဖြစ်သည်။\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "ကျွန်ုပ်တို့မြင်နိုင်သည်မှာ နာမ်များသည် ကျယ်ပြန့်သော အဓိပ္ပါယ်အုပ်စုများကို မပေးနိုင်ပါ။ နာမ်များကို အကြောင်းအရာ ဂရပ်မှ ရရှိသော ပိုမိုယေဘူယျသော စကားလုံးများဖြင့် အစားထိုးကြည့်ရအောင်။ ဤအရာသည် အချိန်အနည်းငယ်ယူပါမည်၊ အကြောင်းမှာ ကျွန်ုပ်တို့သည် နာမ်စကားစုတစ်ခုစီအတွက် REST ခေါ်ဆိုမှုကို ပြုလုပ်နေခြင်းဖြစ်သည်။\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "သတင်းခေါင်းစဉ်များကို မိဘအယူအဆများဖြင့် အမျိုးအစားခွဲကြည့်ရအောင် လုပ်ကြမယ်။ သတင်းခေါင်းစဉ်များရယူရန် [NewsApi.org](http://newsapi.org) ဝန်ဆောင်မှုကို အသုံးပြုမယ်။ ဝန်ဆောင်မှုကို အသုံးပြုရန် သင့်ကိုယ်ပိုင် API key ရယူရန် လိုအပ်ပါမယ် - ဝက်ဘ်ဆိုဒ်ကို သွားပြီး အခမဲ့ developer plan အတွက် မှတ်ပုံတင်ပါ။\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "ပထမဆုံးအနေနဲ့၊ သတင်းခေါင်းစဉ်တွေထဲက နာမ်တွေကို ထုတ်ယူနိုင်ဖို့လိုပါတယ်။ ဒါကိုလုပ်ဖို့အတွက် `TextBlob` စာကြောင်းတိုက် library ကို အသုံးပြုမယ်၊ အဲဒါက ဒီလိုပုံမှန် NLP အလုပ်တွေကို လွယ်ကူစေပါတယ်။\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "ကျွန်ုပ်တို့မြင်နိုင်သည်မှာ နာမ်များသည် ကျယ်ပြန့်သော အဓိပ္ပါယ်အုပ်စုများကို မပေးနိုင်ပါ။ နာမ်များကို အကြောင်းအရာ ဂရပ်မှ ရရှိသော ပိုမိုယေဘူယျသော စကားလုံးများဖြင့် အစားထိုးကြည့်ရအောင်။ ဤအရာသည် အချိန်အနည်းငယ်ယူပါမည်၊ အကြောင်းမှာ ကျွန်ုပ်တို့သည် နာမ်စကားစုတစ်ခုစီအတွက် REST ခေါ်ဆိုမှုကို ပြုလုပ်နေခြင်းဖြစ်သည်။\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**အကြောင်းကြားချက်**: \nဤစာရွက်စာတမ်းကို AI ဘာသာပြန်ဝန်ဆောင်မှု [Co-op Translator](https://github.com/Azure/co-op-translator) ကို အသုံးပြု၍ ဘာသာပြန်ထားပါသည်။ ကျွန်ုပ်တို့သည် တိကျမှုအတွက် ကြိုးစားနေသော်လည်း အလိုအလျောက် ဘာသာပြန်မှုများတွင် အမှားများ သို့မဟုတ် မမှန်ကန်မှုများ ပါဝင်နိုင်သည်ကို သတိပြုပါ။ မူရင်းဘာသာစကားဖြင့် ရေးသားထားသော စာရွက်စာတမ်းကို အာဏာတရ အရင်းအမြစ်အဖြစ် သတ်မှတ်သင့်ပါသည်။ အရေးကြီးသော အချက်အလက်များအတွက် လူ့ဘာသာပြန်ပညာရှင်များကို အသုံးပြုရန် အကြံပြုပါသည်။ ဤဘာသာပြန်မှုကို အသုံးပြုခြင်းမှ ဖြစ်ပေါ်လာသော အလွဲအလွတ်များ သို့မဟုတ် အနားယူမှုများအတွက် ကျွန်ုပ်တို့သည် တာဝန်မယူပါ။\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**အကြောင်းကြားချက်**: \nဤစာရွက်စာတမ်းကို AI ဘာသာပြန်ဝန်ဆောင်မှု [Co-op Translator](https://github.com/Azure/co-op-translator) ကို အသုံးပြု၍ ဘာသာပြန်ထားပါသည်။ ကျွန်ုပ်တို့သည် တိကျမှုအတွက် ကြိုးစားနေသော်လည်း အလိုအလျောက် ဘာသာပြန်မှုများတွင် အမှားများ သို့မဟုတ် မမှန်ကန်မှုများ ပါဝင်နိုင်သည်ကို သတိပြုပါ။ မူရင်းဘာသာစကားဖြင့် ရေးသားထားသော စာရွက်စာတမ်းကို အာဏာတရ အရင်းအမြစ်အဖြစ် သတ်မှတ်သင့်ပါသည်။ အရေးကြီးသော အချက်အလက်များအတွက် လူ့ဘာသာပြန်ပညာရှင်များကို အသုံးပြုရန် အကြံပြုပါသည်။ ဤဘာသာပြန်မှုကို အသုံးပြုခြင်းမှ ဖြစ်ပေါ်လာသော အလွဲအလွတ်များ သို့မဟုတ် အနားယူမှုများအတွက် ကျွန်ုပ်တို့သည် တာဝန်မယူပါ။\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:59:49+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "my" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:59:49+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "my" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/ne/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/ne/lessons/2-Symbolic/MSConceptGraph.ipynb index c1f86057..df7e7893 100644 --- a/translations/ne/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/ne/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,547 +1,538 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## माइक्रोसफ्ट कन्सेप्ट ग्राफ - यो API अब उपलब्ध छैन तर कृपया नोटबुक हेर्नुहोस् अवधारणा बुझ्नको लागि\n", - "\n", - "[माइक्रोसफ्ट कन्सेप्ट ग्राफ](https://concept.research.microsoft.com/) इन्टरनेटबाट खनन गरिएका शब्दहरूको ठूलो वर्गीकरण हो, जसमा अवधारणाहरू बीच `is-a` सम्बन्धहरू छन्।\n", - "\n", - "कन्टेक्स्ट ग्राफ दुई प्रकारमा उपलब्ध छ:\n", - " * डाउनलोडको लागि ठूलो पाठ फाइल\n", - " * REST API\n", - "\n", - "तथ्यांक:\n", - " * 5401933 अद्वितीय अवधारणाहरू,\n", - " * 12551613 अद्वितीय उदाहरणहरू\n", - " * 87603947 `is-a` सम्बन्धहरू\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## वेब सेवा प्रयोग गर्दै\n", - "\n", - "वेब सेवाले विभिन्न समूहहरूमा कुनै अवधारणाको सम्भाव्यता अनुमान गर्न विभिन्न कलहरू प्रदान गर्दछ। थप जानकारी [यहाँ](https://concept.research.microsoft.com/Home/Api) उपलब्ध छ। यहाँ कल गर्नको लागि नमूना URL छ: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## माइक्रोसफ्ट कन्सेप्ट ग्राफ - यो API अब उपलब्ध छैन तर कृपया नोटबुक हेर्नुहोस् अवधारणा बुझ्नको लागि\n", + "\n", + "[माइक्रोसफ्ट कन्सेप्ट ग्राफ](https://concept.research.microsoft.com/) इन्टरनेटबाट खनन गरिएका शब्दहरूको ठूलो वर्गीकरण हो, जसमा अवधारणाहरू बीच `is-a` सम्बन्धहरू छन्।\n", + "\n", + "कन्टेक्स्ट ग्राफ दुई प्रकारमा उपलब्ध छ:\n", + " * डाउनलोडको लागि ठूलो पाठ फाइल\n", + " * REST API\n", + "\n", + "तथ्यांक:\n", + " * 5401933 अद्वितीय अवधारणाहरू,\n", + " * 12551613 अद्वितीय उदाहरणहरू\n", + " * 87603947 `is-a` सम्बन्धहरू\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "हामी समाचार शीर्षकहरूलाई अभिभावक अवधारणाहरू प्रयोग गरेर वर्गीकरण गर्ने प्रयास गरौं। समाचार शीर्षकहरू प्राप्त गर्नका लागि, हामी [NewsApi.org](http://newsapi.org) सेवा प्रयोग गर्नेछौं। सेवा प्रयोग गर्नका लागि तपाईंले आफ्नो API कुञ्जी प्राप्त गर्न आवश्यक छ - वेबसाइटमा जानुहोस् र निःशुल्क डेभलपर योजना दर्ता गर्नुहोस्।\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "सबभन्दा पहिले, हामी समाचार शीर्षकहरूबाट संज्ञा निकाल्न सक्षम हुन चाहन्छौं। हामी यसका लागि `TextBlob` पुस्तकालय प्रयोग गर्नेछौं, जसले यस्ता सामान्य NLP कार्यहरूलाई धेरै सरल बनाउँछ।\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## वेब सेवा प्रयोग गर्दै\n", + "\n", + "वेब सेवाले विभिन्न समूहहरूमा कुनै अवधारणाको सम्भाव्यता अनुमान गर्न विभिन्न कलहरू प्रदान गर्दछ। थप जानकारी [यहाँ](https://concept.research.microsoft.com/Home/Api) उपलब्ध छ। यहाँ कल गर्नको लागि नमूना URL छ: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "हामी देख्न सक्छौं कि संज्ञाले हामीलाई ठूला थिमेटिक समूहहरू प्रदान गर्दैन। आउनुहोस् संज्ञालाई अवधारणा ग्राफबाट प्राप्त सामान्य शब्दहरूद्वारा प्रतिस्थापन गरौं। यसले केही समय लिनेछ, किनभने हामी प्रत्येक संज्ञा वाक्यांशको लागि REST कल गर्दैछौं।\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "हामी समाचार शीर्षकहरूलाई अभिभावक अवधारणाहरू प्रयोग गरेर वर्गीकरण गर्ने प्रयास गरौं। समाचार शीर्षकहरू प्राप्त गर्नका लागि, हामी [NewsApi.org](http://newsapi.org) सेवा प्रयोग गर्नेछौं। सेवा प्रयोग गर्नका लागि तपाईंले आफ्नो API कुञ्जी प्राप्त गर्न आवश्यक छ - वेबसाइटमा जानुहोस् र निःशुल्क डेभलपर योजना दर्ता गर्नुहोस्।\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "सबभन्दा पहिले, हामी समाचार शीर्षकहरूबाट संज्ञा निकाल्न सक्षम हुन चाहन्छौं। हामी यसका लागि `TextBlob` पुस्तकालय प्रयोग गर्नेछौं, जसले यस्ता सामान्य NLP कार्यहरूलाई धेरै सरल बनाउँछ।\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "हामी देख्न सक्छौं कि संज्ञाले हामीलाई ठूला थिमेटिक समूहहरू प्रदान गर्दैन। आउनुहोस् संज्ञालाई अवधारणा ग्राफबाट प्राप्त सामान्य शब्दहरूद्वारा प्रतिस्थापन गरौं। यसले केही समय लिनेछ, किनभने हामी प्रत्येक संज्ञा वाक्यांशको लागि REST कल गर्दैछौं।\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**अस्वीकरण**: \nयो दस्तावेज़ AI अनुवाद सेवा [Co-op Translator](https://github.com/Azure/co-op-translator) प्रयोग गरेर अनुवाद गरिएको छ। हामी शुद्धताको लागि प्रयास गर्छौं, तर कृपया ध्यान दिनुहोस् कि स्वचालित अनुवादहरूमा त्रुटिहरू वा अशुद्धताहरू हुन सक्छ। यसको मूल भाषा मा रहेको मूल दस्तावेज़लाई आधिकारिक स्रोत मानिनुपर्छ। महत्वपूर्ण जानकारीको लागि, व्यावसायिक मानव अनुवाद सिफारिस गरिन्छ। यस अनुवादको प्रयोगबाट उत्पन्न हुने कुनै पनि गलतफहमी वा गलत व्याख्याको लागि हामी जिम्मेवार हुने छैनौं।\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**अस्वीकरण**: \nयो दस्तावेज़ AI अनुवाद सेवा [Co-op Translator](https://github.com/Azure/co-op-translator) प्रयोग गरेर अनुवाद गरिएको छ। हामी शुद्धताको लागि प्रयास गर्छौं, तर कृपया ध्यान दिनुहोस् कि स्वचालित अनुवादहरूमा त्रुटिहरू वा अशुद्धताहरू हुन सक्छ। यसको मूल भाषा मा रहेको मूल दस्तावेज़लाई आधिकारिक स्रोत मानिनुपर्छ। महत्वपूर्ण जानकारीको लागि, व्यावसायिक मानव अनुवाद सिफारिस गरिन्छ। यस अनुवादको प्रयोगबाट उत्पन्न हुने कुनै पनि गलतफहमी वा गलत व्याख्याको लागि हामी जिम्मेवार हुने छैनौं।\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:53:18+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "ne" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:53:18+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "ne" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/nl/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/nl/lessons/2-Symbolic/MSConceptGraph.ipynb index 5c15ff09..d89ddb6c 100644 --- a/translations/nl/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/nl/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - DEZE API IS NIET MEER BESCHIKBAAR, MAAR BEKIJK DE NOTEBOOK OM HET CONCEPT TE BEGRIJPEN\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) is een uitgebreide taxonomie van termen verzameld van het internet, met `is-a` relaties tussen concepten.\n", - "\n", - "Context Graph is beschikbaar in twee vormen:\n", - " * Groot tekstbestand om te downloaden\n", - " * REST API\n", - "\n", - "Statistieken:\n", - " * 5401933 unieke concepten,\n", - " * 12551613 unieke instanties\n", - " * 87603947 `is-a` relaties\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Gebruik van Webservice\n", - "\n", - "De webservice biedt verschillende oproepen om de waarschijnlijkheid te schatten dat een concept tot verschillende groepen behoort. Meer informatie is beschikbaar [hier](https://concept.research.microsoft.com/Home/Api). \n", - "Hier is de voorbeeld-URL om te gebruiken: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - DEZE API IS NIET MEER BESCHIKBAAR, MAAR BEKIJK DE NOTEBOOK OM HET CONCEPT TE BEGRIJPEN\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) is een uitgebreide taxonomie van termen verzameld van het internet, met `is-a` relaties tussen concepten.\n", + "\n", + "Context Graph is beschikbaar in twee vormen:\n", + " * Groot tekstbestand om te downloaden\n", + " * REST API\n", + "\n", + "Statistieken:\n", + " * 5401933 unieke concepten,\n", + " * 12551613 unieke instanties\n", + " * 87603947 `is-a` relaties\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Laten we proberen de nieuwstitels te categoriseren met behulp van hoofdconcepten. Om nieuwstitels te verkrijgen, zullen we de [NewsApi.org](http://newsapi.org) service gebruiken. Je moet je eigen API-sleutel verkrijgen om de service te gebruiken - ga naar de website en registreer je voor het gratis ontwikkelaarsplan.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Allereerst willen we in staat zijn om zelfstandige naamwoorden uit nieuwstitels te extraheren. We zullen hiervoor de `TextBlob`-bibliotheek gebruiken, die veel typische NLP-taken zoals deze vereenvoudigt.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Gebruik van Webservice\n", + "\n", + "De webservice biedt verschillende oproepen om de waarschijnlijkheid te schatten dat een concept tot verschillende groepen behoort. Meer informatie is beschikbaar [hier](https://concept.research.microsoft.com/Home/Api). \n", + "Hier is de voorbeeld-URL om te gebruiken: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We kunnen zien dat zelfstandige naamwoorden ons geen grote thematische groepen geven. Laten we zelfstandige naamwoorden vervangen door meer algemene termen die zijn verkregen uit de conceptgrafiek. Dit zal enige tijd duren, omdat we een REST-oproep doen voor elke zelfstandige naamwoordzin.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Laten we proberen de nieuwstitels te categoriseren met behulp van hoofdconcepten. Om nieuwstitels te verkrijgen, zullen we de [NewsApi.org](http://newsapi.org) service gebruiken. Je moet je eigen API-sleutel verkrijgen om de service te gebruiken - ga naar de website en registreer je voor het gratis ontwikkelaarsplan.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Allereerst willen we in staat zijn om zelfstandige naamwoorden uit nieuwstitels te extraheren. We zullen hiervoor de `TextBlob`-bibliotheek gebruiken, die veel typische NLP-taken zoals deze vereenvoudigt.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We kunnen zien dat zelfstandige naamwoorden ons geen grote thematische groepen geven. Laten we zelfstandige naamwoorden vervangen door meer algemene termen die zijn verkregen uit de conceptgrafiek. Dit zal enige tijd duren, omdat we een REST-oproep doen voor elke zelfstandige naamwoordzin.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**Disclaimer**: \nDit document is vertaald met behulp van de AI-vertalingsservice [Co-op Translator](https://github.com/Azure/co-op-translator). Hoewel we streven naar nauwkeurigheid, dient u zich ervan bewust te zijn dat geautomatiseerde vertalingen fouten of onnauwkeurigheden kunnen bevatten. Het originele document in de oorspronkelijke taal moet worden beschouwd als de gezaghebbende bron. Voor cruciale informatie wordt professionele menselijke vertaling aanbevolen. Wij zijn niet aansprakelijk voor misverstanden of verkeerde interpretaties die voortvloeien uit het gebruik van deze vertaling.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**Disclaimer**: \nDit document is vertaald met behulp van de AI-vertalingsservice [Co-op Translator](https://github.com/Azure/co-op-translator). Hoewel we streven naar nauwkeurigheid, dient u zich ervan bewust te zijn dat geautomatiseerde vertalingen fouten of onnauwkeurigheden kunnen bevatten. Het originele document in de oorspronkelijke taal moet worden beschouwd als de gezaghebbende bron. Voor cruciale informatie wordt professionele menselijke vertaling aanbevolen. Wij zijn niet aansprakelijk voor misverstanden of verkeerde interpretaties die voortvloeien uit het gebruik van deze vertaling.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:56:18+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "nl" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:56:18+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "nl" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/no/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/no/lessons/2-Symbolic/MSConceptGraph.ipynb index bb8e589c..21d9bd92 100644 --- a/translations/no/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/no/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - DENNE API-EN ER IKKE LENGER TILGJENGELIG, MEN VENNLIGST SE PÅ NOTEBOOKEN FOR Å FORSTÅ KONSEPTET\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) er en stor taksonomi av begreper hentet fra internett, med `is-a`-relasjoner mellom konsepter.\n", - "\n", - "Context Graph er tilgjengelig i to former:\n", - " * Stor tekstfil for nedlasting\n", - " * REST API\n", - "\n", - "Statistikk:\n", - " * 5401933 unike konsepter,\n", - " * 12551613 unike instanser\n", - " * 87603947 `is-a`-relasjoner\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Bruke webtjeneste\n", - "\n", - "Webtjenesten tilbyr ulike kall for å estimere sannsynligheten for at et konsept tilhører forskjellige grupper. Mer informasjon er tilgjengelig [her](https://concept.research.microsoft.com/Home/Api). \n", - "Her er eksempel-URL for å gjøre et kall: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - DENNE API-EN ER IKKE LENGER TILGJENGELIG, MEN VENNLIGST SE PÅ NOTEBOOKEN FOR Å FORSTÅ KONSEPTET\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) er en stor taksonomi av begreper hentet fra internett, med `is-a`-relasjoner mellom konsepter.\n", + "\n", + "Context Graph er tilgjengelig i to former:\n", + " * Stor tekstfil for nedlasting\n", + " * REST API\n", + "\n", + "Statistikk:\n", + " * 5401933 unike konsepter,\n", + " * 12551613 unike instanser\n", + " * 87603947 `is-a`-relasjoner\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "La oss prøve å kategorisere nyhetstitlene ved hjelp av overordnede konsepter. For å hente nyhetstitler, vil vi bruke [NewsApi.org](http://newsapi.org)-tjenesten. Du må skaffe din egen API-nøkkel for å bruke tjenesten - gå til nettstedet og registrer deg for gratis utviklerplan.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Først og fremst ønsker vi å kunne trekke ut substantiver fra nyhetstitler. Vi vil bruke `TextBlob`-biblioteket for å gjøre dette, som forenkler mange typiske NLP-oppgaver som denne.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Bruke webtjeneste\n", + "\n", + "Webtjenesten tilbyr ulike kall for å estimere sannsynligheten for at et konsept tilhører forskjellige grupper. Mer informasjon er tilgjengelig [her](https://concept.research.microsoft.com/Home/Api). \n", + "Her er eksempel-URL for å gjøre et kall: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Vi kan se at substantiver ikke gir oss store tematiske grupper. La oss erstatte substantiver med mer generelle termer hentet fra konseptgrafen. Dette vil ta litt tid, fordi vi gjør REST-anrop for hver substantivfrase.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "La oss prøve å kategorisere nyhetstitlene ved hjelp av overordnede konsepter. For å hente nyhetstitler, vil vi bruke [NewsApi.org](http://newsapi.org)-tjenesten. Du må skaffe din egen API-nøkkel for å bruke tjenesten - gå til nettstedet og registrer deg for gratis utviklerplan.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Først og fremst ønsker vi å kunne trekke ut substantiver fra nyhetstitler. Vi vil bruke `TextBlob`-biblioteket for å gjøre dette, som forenkler mange typiske NLP-oppgaver som denne.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Vi kan se at substantiver ikke gir oss store tematiske grupper. La oss erstatte substantiver med mer generelle termer hentet fra konseptgrafen. Dette vil ta litt tid, fordi vi gjør REST-anrop for hver substantivfrase.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**Ansvarsfraskrivelse**: \nDette dokumentet er oversatt ved hjelp av AI-oversettelsestjenesten [Co-op Translator](https://github.com/Azure/co-op-translator). Selv om vi tilstreber nøyaktighet, vær oppmerksom på at automatiserte oversettelser kan inneholde feil eller unøyaktigheter. Det originale dokumentet på sitt opprinnelige språk bør anses som den autoritative kilden. For kritisk informasjon anbefales profesjonell menneskelig oversettelse. Vi er ikke ansvarlige for eventuelle misforståelser eller feiltolkninger som oppstår ved bruk av denne oversettelsen.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**Ansvarsfraskrivelse**: \nDette dokumentet er oversatt ved hjelp av AI-oversettelsestjenesten [Co-op Translator](https://github.com/Azure/co-op-translator). Selv om vi tilstreber nøyaktighet, vær oppmerksom på at automatiserte oversettelser kan inneholde feil eller unøyaktigheter. Det originale dokumentet på sitt opprinnelige språk bør anses som den autoritative kilden. For kritisk informasjon anbefales profesjonell menneskelig oversettelse. Vi er ikke ansvarlige for eventuelle misforståelser eller feiltolkninger som oppstår ved bruk av denne oversettelsen.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:55:51+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "no" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:55:51+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "no" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/pa/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/pa/lessons/2-Symbolic/MSConceptGraph.ipynb index e3b9bb23..a7f0c267 100644 --- a/translations/pa/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/pa/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## ਮਾਈਕਰੋਸਾਫਟ ਕਾਂਸੈਪਟ ਗ੍ਰਾਫ - ਇਹ API ਹੁਣ ਉਪਲਬਧ ਨਹੀਂ ਹੈ ਪਰ ਕਿਰਪਾ ਕਰਕੇ ਕਾਂਸੈਪਟ ਨੂੰ ਸਮਝਣ ਲਈ ਨੋਟਬੁੱਕ ਵੇਖੋ\n", - "\n", - "[ਮਾਈਕਰੋਸਾਫਟ ਕਾਂਸੈਪਟ ਗ੍ਰਾਫ](https://concept.research.microsoft.com/) ਇੰਟਰਨੈਟ ਤੋਂ ਪ੍ਰਾਪਤ ਕੀਤੇ ਗਏ ਸ਼ਬਦਾਂ ਦਾ ਇੱਕ ਵੱਡਾ ਟੈਕਸੋਨੋਮੀ ਹੈ, ਜਿਸ ਵਿੱਚ ਕਾਂਸੈਪਟਾਂ ਦੇ ਵਿਚਕਾਰ `is-a` ਸੰਬੰਧ ਹਨ।\n", - "\n", - "ਕਾਂਟੈਕਸਟ ਗ੍ਰਾਫ ਦੋ ਰੂਪਾਂ ਵਿੱਚ ਉਪਲਬਧ ਹੈ:\n", - " * ਡਾਊਨਲੋਡ ਲਈ ਵੱਡੀ ਟੈਕਸਟ ਫਾਈਲ\n", - " * REST API\n", - "\n", - "ਅੰਕੜੇ:\n", - " * 5401933 ਵਿਲੱਖਣ ਕਾਂਸੈਪਟ\n", - " * 12551613 ਵਿਲੱਖਣ ਉਦਾਹਰਨਾਂ\n", - " * 87603947 `is-a` ਸੰਬੰਧ\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## ਵੈੱਬ ਸੇਵਾ ਦੀ ਵਰਤੋਂ\n", - "\n", - "ਵੈੱਬ ਸੇਵਾ ਵੱਖ-ਵੱਖ ਗਰੁੱਪਾਂ ਵਿੱਚ ਕਿਸੇ ਧਾਰਨਾ ਦੇ ਸਬੰਧਿਤ ਹੋਣ ਦੀ ਸੰਭਾਵਨਾ ਦਾ ਅੰਦਾਜ਼ਾ ਲਗਾਉਣ ਲਈ ਵੱਖ-ਵੱਖ ਕਾਲਾਂ ਦੀ ਪੇਸ਼ਕਸ਼ ਕਰਦੀ ਹੈ। ਹੋਰ ਜਾਣਕਾਰੀ [ਇੱਥੇ](https://concept.research.microsoft.com/Home/Api) ਉਪਲਬਧ ਹੈ। \n", - "ਇਹ ਕਾਲ ਕਰਨ ਲਈ ਨਮੂਨਾ URL ਹੈ: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## ਮਾਈਕਰੋਸਾਫਟ ਕਾਂਸੈਪਟ ਗ੍ਰਾਫ - ਇਹ API ਹੁਣ ਉਪਲਬਧ ਨਹੀਂ ਹੈ ਪਰ ਕਿਰਪਾ ਕਰਕੇ ਕਾਂਸੈਪਟ ਨੂੰ ਸਮਝਣ ਲਈ ਨੋਟਬੁੱਕ ਵੇਖੋ\n", + "\n", + "[ਮਾਈਕਰੋਸਾਫਟ ਕਾਂਸੈਪਟ ਗ੍ਰਾਫ](https://concept.research.microsoft.com/) ਇੰਟਰਨੈਟ ਤੋਂ ਪ੍ਰਾਪਤ ਕੀਤੇ ਗਏ ਸ਼ਬਦਾਂ ਦਾ ਇੱਕ ਵੱਡਾ ਟੈਕਸੋਨੋਮੀ ਹੈ, ਜਿਸ ਵਿੱਚ ਕਾਂਸੈਪਟਾਂ ਦੇ ਵਿਚਕਾਰ `is-a` ਸੰਬੰਧ ਹਨ।\n", + "\n", + "ਕਾਂਟੈਕਸਟ ਗ੍ਰਾਫ ਦੋ ਰੂਪਾਂ ਵਿੱਚ ਉਪਲਬਧ ਹੈ:\n", + " * ਡਾਊਨਲੋਡ ਲਈ ਵੱਡੀ ਟੈਕਸਟ ਫਾਈਲ\n", + " * REST API\n", + "\n", + "ਅੰਕੜੇ:\n", + " * 5401933 ਵਿਲੱਖਣ ਕਾਂਸੈਪਟ\n", + " * 12551613 ਵਿਲੱਖਣ ਉਦਾਹਰਨਾਂ\n", + " * 87603947 `is-a` ਸੰਬੰਧ\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "ਆਓ ਖ਼ਬਰਾਂ ਦੇ ਸਿਰਲੇਖਾਂ ਨੂੰ ਮਾਪੇ ਧਾਰਨਾਵਾਂ ਦੇ ਅਧਾਰ 'ਤੇ ਸ਼੍ਰੇਣੀਬੱਧ ਕਰਨ ਦੀ ਕੋਸ਼ਿਸ਼ ਕਰੀਏ। ਖ਼ਬਰਾਂ ਦੇ ਸਿਰਲੇਖ ਪ੍ਰਾਪਤ ਕਰਨ ਲਈ, ਅਸੀਂ [NewsApi.org](http://newsapi.org) ਸੇਵਾ ਦੀ ਵਰਤੋਂ ਕਰਾਂਗੇ। ਸੇਵਾ ਦੀ ਵਰਤੋਂ ਕਰਨ ਲਈ ਤੁਹਾਨੂੰ ਆਪਣਾ API ਕੁੰਜੀ ਪ੍ਰਾਪਤ ਕਰਨੀ ਪਵੇਗੀ - ਵੈਬਸਾਈਟ 'ਤੇ ਜਾਓ ਅਤੇ ਮੁਫ਼ਤ ਡਿਵੈਲਪਰ ਯੋਜਨਾ ਲਈ ਰਜਿਸਟਰ ਕਰੋ।\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "ਸਭ ਤੋਂ ਪਹਿਲਾਂ, ਅਸੀਂ ਖਬਰਾਂ ਦੇ ਸਿਰਲੇਖਾਂ ਵਿੱਚੋਂ ਨਾਂਵਾਂ ਕੱਢਣ ਦੇ ਯੋਗ ਹੋਣਾ ਚਾਹੁੰਦੇ ਹਾਂ। ਅਸੀਂ ਇਹ ਕਰਨ ਲਈ `TextBlob` ਲਾਇਬ੍ਰੇਰੀ ਦੀ ਵਰਤੋਂ ਕਰਾਂਗੇ, ਜੋ ਇਸ ਤਰ੍ਹਾਂ ਦੇ ਆਮ NLP ਕੰਮਾਂ ਨੂੰ ਬਹੁਤ ਸਧਾਰਨ ਬਣਾਉਂਦੀ ਹੈ।\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## ਵੈੱਬ ਸੇਵਾ ਦੀ ਵਰਤੋਂ\n", + "\n", + "ਵੈੱਬ ਸੇਵਾ ਵੱਖ-ਵੱਖ ਗਰੁੱਪਾਂ ਵਿੱਚ ਕਿਸੇ ਧਾਰਨਾ ਦੇ ਸਬੰਧਿਤ ਹੋਣ ਦੀ ਸੰਭਾਵਨਾ ਦਾ ਅੰਦਾਜ਼ਾ ਲਗਾਉਣ ਲਈ ਵੱਖ-ਵੱਖ ਕਾਲਾਂ ਦੀ ਪੇਸ਼ਕਸ਼ ਕਰਦੀ ਹੈ। ਹੋਰ ਜਾਣਕਾਰੀ [ਇੱਥੇ](https://concept.research.microsoft.com/Home/Api) ਉਪਲਬਧ ਹੈ। \n", + "ਇਹ ਕਾਲ ਕਰਨ ਲਈ ਨਮੂਨਾ URL ਹੈ: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "ਅਸੀਂ ਦੇਖ ਸਕਦੇ ਹਾਂ ਕਿ ਨਾਂਵ ਸਾਨੂੰ ਵੱਡੇ ਥੀਮਾਤਮਕ ਸਮੂਹ ਨਹੀਂ ਦਿੰਦੇ। ਆਓ ਨਾਂਵਾਂ ਨੂੰ ਧਾਰਨਾ ਗ੍ਰਾਫ ਤੋਂ ਪ੍ਰਾਪਤ ਕੀਤੇ ਗਏ ਹੋਰ ਸਧਾਰਨ ਸ਼ਬਦਾਂ ਨਾਲ ਬਦਲਦੇ ਹਾਂ। ਇਹ ਕੁਝ ਸਮਾਂ ਲਵੇਗਾ, ਕਿਉਂਕਿ ਅਸੀਂ ਹਰ ਨਾਂਵ ਵਾਕਾਂਸ਼ ਲਈ REST ਕਾਲ ਕਰ ਰਹੇ ਹਾਂ।\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "ਆਓ ਖ਼ਬਰਾਂ ਦੇ ਸਿਰਲੇਖਾਂ ਨੂੰ ਮਾਪੇ ਧਾਰਨਾਵਾਂ ਦੇ ਅਧਾਰ 'ਤੇ ਸ਼੍ਰੇਣੀਬੱਧ ਕਰਨ ਦੀ ਕੋਸ਼ਿਸ਼ ਕਰੀਏ। ਖ਼ਬਰਾਂ ਦੇ ਸਿਰਲੇਖ ਪ੍ਰਾਪਤ ਕਰਨ ਲਈ, ਅਸੀਂ [NewsApi.org](http://newsapi.org) ਸੇਵਾ ਦੀ ਵਰਤੋਂ ਕਰਾਂਗੇ। ਸੇਵਾ ਦੀ ਵਰਤੋਂ ਕਰਨ ਲਈ ਤੁਹਾਨੂੰ ਆਪਣਾ API ਕੁੰਜੀ ਪ੍ਰਾਪਤ ਕਰਨੀ ਪਵੇਗੀ - ਵੈਬਸਾਈਟ 'ਤੇ ਜਾਓ ਅਤੇ ਮੁਫ਼ਤ ਡਿਵੈਲਪਰ ਯੋਜਨਾ ਲਈ ਰਜਿਸਟਰ ਕਰੋ।\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "ਸਭ ਤੋਂ ਪਹਿਲਾਂ, ਅਸੀਂ ਖਬਰਾਂ ਦੇ ਸਿਰਲੇਖਾਂ ਵਿੱਚੋਂ ਨਾਂਵਾਂ ਕੱਢਣ ਦੇ ਯੋਗ ਹੋਣਾ ਚਾਹੁੰਦੇ ਹਾਂ। ਅਸੀਂ ਇਹ ਕਰਨ ਲਈ `TextBlob` ਲਾਇਬ੍ਰੇਰੀ ਦੀ ਵਰਤੋਂ ਕਰਾਂਗੇ, ਜੋ ਇਸ ਤਰ੍ਹਾਂ ਦੇ ਆਮ NLP ਕੰਮਾਂ ਨੂੰ ਬਹੁਤ ਸਧਾਰਨ ਬਣਾਉਂਦੀ ਹੈ।\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "ਅਸੀਂ ਦੇਖ ਸਕਦੇ ਹਾਂ ਕਿ ਨਾਂਵ ਸਾਨੂੰ ਵੱਡੇ ਥੀਮਾਤਮਕ ਸਮੂਹ ਨਹੀਂ ਦਿੰਦੇ। ਆਓ ਨਾਂਵਾਂ ਨੂੰ ਧਾਰਨਾ ਗ੍ਰਾਫ ਤੋਂ ਪ੍ਰਾਪਤ ਕੀਤੇ ਗਏ ਹੋਰ ਸਧਾਰਨ ਸ਼ਬਦਾਂ ਨਾਲ ਬਦਲਦੇ ਹਾਂ। ਇਹ ਕੁਝ ਸਮਾਂ ਲਵੇਗਾ, ਕਿਉਂਕਿ ਅਸੀਂ ਹਰ ਨਾਂਵ ਵਾਕਾਂਸ਼ ਲਈ REST ਕਾਲ ਕਰ ਰਹੇ ਹਾਂ।\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**ਅਸਵੀਕਰਤਾ**: \nਇਹ ਦਸਤਾਵੇਜ਼ AI ਅਨੁਵਾਦ ਸੇਵਾ [Co-op Translator](https://github.com/Azure/co-op-translator) ਦੀ ਵਰਤੋਂ ਕਰਕੇ ਅਨੁਵਾਦ ਕੀਤਾ ਗਿਆ ਹੈ। ਹਾਲਾਂਕਿ ਅਸੀਂ ਸਹੀਤਾ ਲਈ ਯਤਨਸ਼ੀਲ ਹਾਂ, ਕਿਰਪਾ ਕਰਕੇ ਧਿਆਨ ਦਿਓ ਕਿ ਸਵੈਚਾਲਿਤ ਅਨੁਵਾਦਾਂ ਵਿੱਚ ਗਲਤੀਆਂ ਜਾਂ ਅਸੁੱਤੀਆਂ ਹੋ ਸਕਦੀਆਂ ਹਨ। ਇਸ ਦਸਤਾਵੇਜ਼ ਦੀ ਮੂਲ ਭਾਸ਼ਾ ਵਿੱਚ ਲਿਖੀ ਗਈ ਮੂਲ ਕਾਪੀ ਨੂੰ ਅਧਿਕਾਰਤ ਸਰੋਤ ਮੰਨਿਆ ਜਾਣਾ ਚਾਹੀਦਾ ਹੈ। ਮਹੱਤਵਪੂਰਨ ਜਾਣਕਾਰੀ ਲਈ, ਪੇਸ਼ੇਵਰ ਮਨੁੱਖੀ ਅਨੁਵਾਦ ਦੀ ਸਿਫਾਰਸ਼ ਕੀਤੀ ਜਾਂਦੀ ਹੈ। ਇਸ ਅਨੁਵਾਦ ਦੀ ਵਰਤੋਂ ਤੋਂ ਪੈਦਾ ਹੋਣ ਵਾਲੇ ਕਿਸੇ ਵੀ ਗਲਤਫਹਿਮੀ ਜਾਂ ਗਲਤ ਵਿਆਖਿਆ ਲਈ ਅਸੀਂ ਜ਼ਿੰਮੇਵਾਰ ਨਹੀਂ ਹਾਂ।\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**ਅਸਵੀਕਰਤਾ**: \nਇਹ ਦਸਤਾਵੇਜ਼ AI ਅਨੁਵਾਦ ਸੇਵਾ [Co-op Translator](https://github.com/Azure/co-op-translator) ਦੀ ਵਰਤੋਂ ਕਰਕੇ ਅਨੁਵਾਦ ਕੀਤਾ ਗਿਆ ਹੈ। ਹਾਲਾਂਕਿ ਅਸੀਂ ਸਹੀਤਾ ਲਈ ਯਤਨਸ਼ੀਲ ਹਾਂ, ਕਿਰਪਾ ਕਰਕੇ ਧਿਆਨ ਦਿਓ ਕਿ ਸਵੈਚਾਲਿਤ ਅਨੁਵਾਦਾਂ ਵਿੱਚ ਗਲਤੀਆਂ ਜਾਂ ਅਸੁੱਤੀਆਂ ਹੋ ਸਕਦੀਆਂ ਹਨ। ਇਸ ਦਸਤਾਵੇਜ਼ ਦੀ ਮੂਲ ਭਾਸ਼ਾ ਵਿੱਚ ਲਿਖੀ ਗਈ ਮੂਲ ਕਾਪੀ ਨੂੰ ਅਧਿਕਾਰਤ ਸਰੋਤ ਮੰਨਿਆ ਜਾਣਾ ਚਾਹੀਦਾ ਹੈ। ਮਹੱਤਵਪੂਰਨ ਜਾਣਕਾਰੀ ਲਈ, ਪੇਸ਼ੇਵਰ ਮਨੁੱਖੀ ਅਨੁਵਾਦ ਦੀ ਸਿਫਾਰਸ਼ ਕੀਤੀ ਜਾਂਦੀ ਹੈ। ਇਸ ਅਨੁਵਾਦ ਦੀ ਵਰਤੋਂ ਤੋਂ ਪੈਦਾ ਹੋਣ ਵਾਲੇ ਕਿਸੇ ਵੀ ਗਲਤਫਹਿਮੀ ਜਾਂ ਗਲਤ ਵਿਆਖਿਆ ਲਈ ਅਸੀਂ ਜ਼ਿੰਮੇਵਾਰ ਨਹੀਂ ਹਾਂ।\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:53:36+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "pa" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:53:36+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "pa" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/pcm/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/pcm/lessons/2-Symbolic/MSConceptGraph.ipynb index 3463c109..498d8212 100644 --- a/translations/pcm/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/pcm/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - DIS API NO DEY AVAILABLE AGAIN BUT MAKE YOU CHECK DI NOTEBOOK TO UNDERSTAND DI CONCEPT\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) na one big taxonomy of terms wey dem mine from di internet, wit `is-a` relations between concepts.\n", - "\n", - "Context Graph dey available for two ways:\n", - " * Big text file wey you fit download\n", - " * REST API\n", - "\n", - "Statistics:\n", - " * 5401933 unique concepts, \n", - " * 12551613 unique instances\n", - " * 87603947 `is-a` relations\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## How to Use Web Service\n", - "\n", - "Web service dey give different calls wey fit help you check probability say one concept dey belong to different groups. You fit find more info [here](https://concept.research.microsoft.com/Home/Api). \n", - "Dis na sample URL wey you fit use call: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - DIS API NO DEY AVAILABLE AGAIN BUT MAKE YOU CHECK DI NOTEBOOK TO UNDERSTAND DI CONCEPT\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) na one big taxonomy of terms wey dem mine from di internet, wit `is-a` relations between concepts.\n", + "\n", + "Context Graph dey available for two ways:\n", + " * Big text file wey you fit download\n", + " * REST API\n", + "\n", + "Statistics:\n", + " * 5401933 unique concepts, \n", + " * 12551613 unique instances\n", + " * 87603947 `is-a` relations\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Make we try arrange di news title dem under parent concept. To get di news title dem, we go use [NewsApi.org](http://newsapi.org) service. You go need get your own API key to fit use di service - go di website and register for free developer plan.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "First of all, we wan fit extract nouns from news title. We go use `TextBlob` library do am, e dey make plenty normal NLP work like dis easy.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## How to Use Web Service\n", + "\n", + "Web service dey give different calls wey fit help you check probability say one concept dey belong to different groups. You fit find more info [here](https://concept.research.microsoft.com/Home/Api). \n", + "Dis na sample URL wey you fit use call: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We fit see say nouns no dey give us big thematic groups. Make we change nouns to more general terms wey we get from di concept graph. Dis go take some time, because we dey do REST call for each noun phrase.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Make we try arrange di news title dem under parent concept. To get di news title dem, we go use [NewsApi.org](http://newsapi.org) service. You go need get your own API key to fit use di service - go di website and register for free developer plan.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "First of all, we wan fit extract nouns from news title. We go use `TextBlob` library do am, e dey make plenty normal NLP work like dis easy.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We fit see say nouns no dey give us big thematic groups. Make we change nouns to more general terms wey we get from di concept graph. Dis go take some time, because we dey do REST call for each noun phrase.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n\n\n**Disclaimer**: \nDis dokyument don use AI transleshion service [Co-op Translator](https://github.com/Azure/co-op-translator) do di transleshion. Even as we dey try make am accurate, abeg make you sabi say transleshion wey machine do fit get mistake or no dey correct well. Di original dokyument for im native language na di main source wey you go fit trust. For important informashon, e good make you use professional human transleshion. We no go fit take blame for any misunderstanding or wrong meaning wey fit happen because you use dis transleshion.\n\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n\n\n**Disclaimer**: \nDis dokyument don use AI transleshion service [Co-op Translator](https://github.com/Azure/co-op-translator) do di transleshion. Even as we dey try make am accurate, abeg make you sabi say transleshion wey machine do fit get mistake or no dey correct well. Di original dokyument for im native language na di main source wey you go fit trust. For important informashon, e good make you use professional human transleshion. We no go fit take blame for any misunderstanding or wrong meaning wey fit happen because you use dis transleshion.\n\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-11-18T19:22:14+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "pcm" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-11-18T19:22:14+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "pcm" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/pl/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/pl/lessons/2-Symbolic/MSConceptGraph.ipynb index b4049b42..4094282e 100644 --- a/translations/pl/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/pl/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - TA API NIE JEST JUŻ DOSTĘPNE, ALE ZAPOZNAJ SIĘ Z NOTATNIKIEM, ABY ZROZUMIEĆ KONCEPCJĘ\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) to obszerna taksonomia terminów wydobytych z internetu, zawierająca relacje typu `is-a` między koncepcjami.\n", - "\n", - "Context Graph jest dostępny w dwóch formach:\n", - " * Duży plik tekstowy do pobrania\n", - " * REST API\n", - "\n", - "Statystyki:\n", - " * 5401933 unikalnych koncepcji,\n", - " * 12551613 unikalnych instancji\n", - " * 87603947 relacji `is-a`\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Korzystanie z usługi sieciowej\n", - "\n", - "Usługa sieciowa oferuje różne wywołania, które pozwalają oszacować prawdopodobieństwo, że dany koncept należy do różnych grup. Więcej informacji znajdziesz [tutaj](https://concept.research.microsoft.com/Home/Api). \n", - "Oto przykładowy URL do wywołania: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - TA API NIE JEST JUŻ DOSTĘPNE, ALE ZAPOZNAJ SIĘ Z NOTATNIKIEM, ABY ZROZUMIEĆ KONCEPCJĘ\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) to obszerna taksonomia terminów wydobytych z internetu, zawierająca relacje typu `is-a` między koncepcjami.\n", + "\n", + "Context Graph jest dostępny w dwóch formach:\n", + " * Duży plik tekstowy do pobrania\n", + " * REST API\n", + "\n", + "Statystyki:\n", + " * 5401933 unikalnych koncepcji,\n", + " * 12551613 unikalnych instancji\n", + " * 87603947 relacji `is-a`\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Spróbujmy skategoryzować tytuły wiadomości za pomocą nadrzędnych pojęć. Aby uzyskać tytuły wiadomości, skorzystamy z usługi [NewsApi.org](http://newsapi.org). Musisz uzyskać własny klucz API, aby korzystać z usługi - przejdź na stronę internetową i zarejestruj się w darmowym planie dla deweloperów.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Przede wszystkim chcemy móc wyodrębniać rzeczowniki z tytułów wiadomości. Do tego użyjemy biblioteki `TextBlob`, która upraszcza wiele typowych zadań związanych z przetwarzaniem języka naturalnego, takich jak to.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Korzystanie z usługi sieciowej\n", + "\n", + "Usługa sieciowa oferuje różne wywołania, które pozwalają oszacować prawdopodobieństwo, że dany koncept należy do różnych grup. Więcej informacji znajdziesz [tutaj](https://concept.research.microsoft.com/Home/Api). \n", + "Oto przykładowy URL do wywołania: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Widzimy, że rzeczowniki nie tworzą dużych grup tematycznych. Zamieńmy rzeczowniki na bardziej ogólne terminy uzyskane z grafu pojęć. To zajmie trochę czasu, ponieważ wykonujemy wywołanie REST dla każdej frazy rzeczownikowej.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Spróbujmy skategoryzować tytuły wiadomości za pomocą nadrzędnych pojęć. Aby uzyskać tytuły wiadomości, skorzystamy z usługi [NewsApi.org](http://newsapi.org). Musisz uzyskać własny klucz API, aby korzystać z usługi - przejdź na stronę internetową i zarejestruj się w darmowym planie dla deweloperów.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Przede wszystkim chcemy móc wyodrębniać rzeczowniki z tytułów wiadomości. Do tego użyjemy biblioteki `TextBlob`, która upraszcza wiele typowych zadań związanych z przetwarzaniem języka naturalnego, takich jak to.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Widzimy, że rzeczowniki nie tworzą dużych grup tematycznych. Zamieńmy rzeczowniki na bardziej ogólne terminy uzyskane z grafu pojęć. To zajmie trochę czasu, ponieważ wykonujemy wywołanie REST dla każdej frazy rzeczownikowej.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**Zastrzeżenie**: \nTen dokument został przetłumaczony za pomocą usługi tłumaczenia AI [Co-op Translator](https://github.com/Azure/co-op-translator). Chociaż dokładamy wszelkich starań, aby tłumaczenie było precyzyjne, prosimy pamiętać, że automatyczne tłumaczenia mogą zawierać błędy lub nieścisłości. Oryginalny dokument w jego rodzimym języku powinien być uznawany za wiarygodne źródło. W przypadku informacji o kluczowym znaczeniu zaleca się skorzystanie z profesjonalnego tłumaczenia przez człowieka. Nie ponosimy odpowiedzialności za jakiekolwiek nieporozumienia lub błędne interpretacje wynikające z użycia tego tłumaczenia.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**Zastrzeżenie**: \nTen dokument został przetłumaczony za pomocą usługi tłumaczenia AI [Co-op Translator](https://github.com/Azure/co-op-translator). Chociaż dokładamy wszelkich starań, aby tłumaczenie było precyzyjne, prosimy pamiętać, że automatyczne tłumaczenia mogą zawierać błędy lub nieścisłości. Oryginalny dokument w jego rodzimym języku powinien być uznawany za wiarygodne źródło. W przypadku informacji o kluczowym znaczeniu zaleca się skorzystanie z profesjonalnego tłumaczenia przez człowieka. Nie ponosimy odpowiedzialności za jakiekolwiek nieporozumienia lub błędne interpretacje wynikające z użycia tego tłumaczenia.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:54:28+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "pl" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:54:28+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "pl" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/pt-BR/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/pt-BR/lessons/2-Symbolic/MSConceptGraph.ipynb index aec68c2e..91d33fcb 100644 --- a/translations/pt-BR/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/pt-BR/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - ESTA API NÃO ESTÁ MAIS DISPONÍVEL, MAS CONSULTE O NOTEBOOK PARA ENTENDER O CONCEITO\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) é uma grande taxonomia de termos extraídos da internet, com relações de `é-um` entre conceitos.\n", - "\n", - "O Context Graph está disponível em duas formas:\n", - " * Arquivo de texto grande para download\n", - " * API REST\n", - "\n", - "Estatísticas:\n", - " * 5.401.933 conceitos únicos\n", - " * 12.551.613 instâncias únicas\n", - " * 87.603.947 relações de `é-um`\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Usando o Serviço Web\n", - "\n", - "O serviço web oferece diferentes chamadas para estimar a probabilidade de um conceito pertencer a diferentes grupos. Mais informações estão disponíveis [aqui](https://concept.research.microsoft.com/Home/Api). \n", - "Aqui está o URL de exemplo para realizar a chamada: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - ESTA API NÃO ESTÁ MAIS DISPONÍVEL, MAS CONSULTE O NOTEBOOK PARA ENTENDER O CONCEITO\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) é uma grande taxonomia de termos extraídos da internet, com relações de `é-um` entre conceitos.\n", + "\n", + "O Context Graph está disponível em duas formas:\n", + " * Arquivo de texto grande para download\n", + " * API REST\n", + "\n", + "Estatísticas:\n", + " * 5.401.933 conceitos únicos\n", + " * 12.551.613 instâncias únicas\n", + " * 87.603.947 relações de `é-um`\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Vamos tentar categorizar os títulos das notícias usando conceitos principais. Para obter os títulos das notícias, usaremos o serviço [NewsApi.org](http://newsapi.org). Você precisa obter sua própria chave de API para usar o serviço - acesse o site e registre-se no plano de desenvolvedor gratuito.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Primeiramente, queremos ser capazes de extrair substantivos de títulos de notícias. Usaremos a biblioteca `TextBlob` para fazer isso, o que simplifica muito tarefas típicas de PLN como esta.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Usando o Serviço Web\n", + "\n", + "O serviço web oferece diferentes chamadas para estimar a probabilidade de um conceito pertencer a diferentes grupos. Mais informações estão disponíveis [aqui](https://concept.research.microsoft.com/Home/Api). \n", + "Aqui está o URL de exemplo para realizar a chamada: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Podemos ver que os substantivos não nos fornecem grandes grupos temáticos. Vamos substituir os substantivos por termos mais gerais obtidos a partir do gráfico de conceitos. Isso levará algum tempo, porque estamos fazendo chamadas REST para cada frase nominal.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Vamos tentar categorizar os títulos das notícias usando conceitos principais. Para obter os títulos das notícias, usaremos o serviço [NewsApi.org](http://newsapi.org). Você precisa obter sua própria chave de API para usar o serviço - acesse o site e registre-se no plano de desenvolvedor gratuito.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Primeiramente, queremos ser capazes de extrair substantivos de títulos de notícias. Usaremos a biblioteca `TextBlob` para fazer isso, o que simplifica muito tarefas típicas de PLN como esta.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Podemos ver que os substantivos não nos fornecem grandes grupos temáticos. Vamos substituir os substantivos por termos mais gerais obtidos a partir do gráfico de conceitos. Isso levará algum tempo, porque estamos fazendo chamadas REST para cada frase nominal.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**Aviso Legal**: \nEste documento foi traduzido utilizando o serviço de tradução por IA [Co-op Translator](https://github.com/Azure/co-op-translator). Embora nos esforcemos para garantir a precisão, esteja ciente de que traduções automatizadas podem conter erros ou imprecisões. O documento original em seu idioma nativo deve ser considerado a fonte oficial. Para informações críticas, recomenda-se a tradução profissional realizada por humanos. Não nos responsabilizamos por quaisquer mal-entendidos ou interpretações equivocadas decorrentes do uso desta tradução.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**Aviso Legal**: \nEste documento foi traduzido utilizando o serviço de tradução por IA [Co-op Translator](https://github.com/Azure/co-op-translator). Embora nos esforcemos para garantir a precisão, esteja ciente de que traduções automatizadas podem conter erros ou imprecisões. O documento original em seu idioma nativo deve ser considerado a fonte oficial. Para informações críticas, recomenda-se a tradução profissional realizada por humanos. Não nos responsabilizamos por quaisquer mal-entendidos ou interpretações equivocadas decorrentes do uso desta tradução.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:54:00+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "br" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:54:00+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "br" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/pt-PT/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/pt-PT/lessons/2-Symbolic/MSConceptGraph.ipynb index 8b704594..36de6c2c 100644 --- a/translations/pt-PT/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/pt-PT/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - ESTA API JÁ NÃO ESTÁ DISPONÍVEL, MAS CONSULTE O NOTEBOOK PARA COMPREENDER O CONCEITO\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) é uma grande taxonomia de termos extraídos da internet, com relações de tipo `is-a` entre conceitos.\n", - "\n", - "O Context Graph está disponível em duas formas:\n", - " * Ficheiro de texto grande para download\n", - " * API REST\n", - "\n", - "Estatísticas:\n", - " * 5.401.933 conceitos únicos\n", - " * 12.551.613 instâncias únicas\n", - " * 87.603.947 relações `is-a`\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Utilizar o Serviço Web\n", - "\n", - "O serviço web oferece diferentes chamadas para estimar a probabilidade de um conceito pertencer a diferentes grupos. Mais informações estão disponíveis [aqui](https://concept.research.microsoft.com/Home/Api). \n", - "Aqui está o URL de exemplo para realizar a chamada: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - ESTA API JÁ NÃO ESTÁ DISPONÍVEL, MAS CONSULTE O NOTEBOOK PARA COMPREENDER O CONCEITO\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) é uma grande taxonomia de termos extraídos da internet, com relações de tipo `is-a` entre conceitos.\n", + "\n", + "O Context Graph está disponível em duas formas:\n", + " * Ficheiro de texto grande para download\n", + " * API REST\n", + "\n", + "Estatísticas:\n", + " * 5.401.933 conceitos únicos\n", + " * 12.551.613 instâncias únicas\n", + " * 87.603.947 relações `is-a`\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Vamos tentar categorizar os títulos das notícias usando conceitos principais. Para obter os títulos das notícias, usaremos o serviço [NewsApi.org](http://newsapi.org). É necessário obter a sua própria chave API para utilizar o serviço - aceda ao site e registe-se no plano de desenvolvedor gratuito.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Em primeiro lugar, queremos ser capazes de extrair substantivos de títulos de notícias. Vamos usar a biblioteca `TextBlob` para fazer isso, que simplifica muitos dos típicos trabalhos de PLN como este.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Utilizar o Serviço Web\n", + "\n", + "O serviço web oferece diferentes chamadas para estimar a probabilidade de um conceito pertencer a diferentes grupos. Mais informações estão disponíveis [aqui](https://concept.research.microsoft.com/Home/Api). \n", + "Aqui está o URL de exemplo para realizar a chamada: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Podemos ver que os substantivos não nos dão grandes grupos temáticos. Vamos substituir os substantivos por termos mais gerais obtidos a partir do gráfico de conceitos. Isto vai demorar algum tempo, porque estamos a fazer chamadas REST para cada expressão nominal.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Vamos tentar categorizar os títulos das notícias usando conceitos principais. Para obter os títulos das notícias, usaremos o serviço [NewsApi.org](http://newsapi.org). É necessário obter a sua própria chave API para utilizar o serviço - aceda ao site e registe-se no plano de desenvolvedor gratuito.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Em primeiro lugar, queremos ser capazes de extrair substantivos de títulos de notícias. Vamos usar a biblioteca `TextBlob` para fazer isso, que simplifica muitos dos típicos trabalhos de PLN como este.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Podemos ver que os substantivos não nos dão grandes grupos temáticos. Vamos substituir os substantivos por termos mais gerais obtidos a partir do gráfico de conceitos. Isto vai demorar algum tempo, porque estamos a fazer chamadas REST para cada expressão nominal.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**Aviso**: \nEste documento foi traduzido utilizando o serviço de tradução por IA [Co-op Translator](https://github.com/Azure/co-op-translator). Embora nos esforcemos pela precisão, esteja ciente de que traduções automáticas podem conter erros ou imprecisões. O documento original na sua língua nativa deve ser considerado a fonte autoritária. Para informações críticas, recomenda-se uma tradução profissional realizada por humanos. Não nos responsabilizamos por quaisquer mal-entendidos ou interpretações incorretas decorrentes do uso desta tradução.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**Aviso**: \nEste documento foi traduzido utilizando o serviço de tradução por IA [Co-op Translator](https://github.com/Azure/co-op-translator). Embora nos esforcemos pela precisão, esteja ciente de que traduções automáticas podem conter erros ou imprecisões. O documento original na sua língua nativa deve ser considerado a fonte autoritária. Para informações críticas, recomenda-se uma tradução profissional realizada por humanos. Não nos responsabilizamos por quaisquer mal-entendidos ou interpretações incorretas decorrentes do uso desta tradução.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:53:48+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "pt" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:53:48+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "pt" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/ro/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/ro/lessons/2-Symbolic/MSConceptGraph.ipynb index b6d3ff95..ee31633c 100644 --- a/translations/ro/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/ro/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - ACEST API NU MAI ESTE DISPONIBIL, DAR VĂ RUGĂM SĂ CONSULTAȚI NOTEBOOK-UL PENTRU A ÎNȚELEGE CONCEPTUL\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) este o taxonomie extinsă de termeni obținuți de pe internet, cu relații de tip `is-a` între concepte.\n", - "\n", - "Context Graph este disponibil în două forme:\n", - " * Fișier text mare pentru descărcare\n", - " * API REST\n", - "\n", - "Statistici:\n", - " * 5401933 concepte unice,\n", - " * 12551613 instanțe unice\n", - " * 87603947 relații de tip `is-a`\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Utilizarea serviciului web\n", - "\n", - "Serviciul web oferă diferite apeluri pentru a estima probabilitatea ca un concept să aparțină unor grupuri diferite. Mai multe informații sunt disponibile [aici](https://concept.research.microsoft.com/Home/Api). \n", - "Iată un exemplu de URL pentru apel: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - ACEST API NU MAI ESTE DISPONIBIL, DAR VĂ RUGĂM SĂ CONSULTAȚI NOTEBOOK-UL PENTRU A ÎNȚELEGE CONCEPTUL\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) este o taxonomie extinsă de termeni obținuți de pe internet, cu relații de tip `is-a` între concepte.\n", + "\n", + "Context Graph este disponibil în două forme:\n", + " * Fișier text mare pentru descărcare\n", + " * API REST\n", + "\n", + "Statistici:\n", + " * 5401933 concepte unice,\n", + " * 12551613 instanțe unice\n", + " * 87603947 relații de tip `is-a`\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Să încercăm să clasificăm titlurile știrilor folosind concepte părinte. Pentru a obține titlurile știrilor, vom folosi serviciul [NewsApi.org](http://newsapi.org). Trebuie să obțineți propria cheie API pentru a utiliza serviciul - accesați site-ul web și înregistrați-vă pentru planul gratuit de dezvoltator.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "În primul rând, vrem să putem extrage substantive din titlurile știrilor. Vom folosi biblioteca `TextBlob` pentru a face acest lucru, care simplifică multe sarcini tipice de procesare a limbajului natural, cum ar fi aceasta.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Utilizarea serviciului web\n", + "\n", + "Serviciul web oferă diferite apeluri pentru a estima probabilitatea ca un concept să aparțină unor grupuri diferite. Mai multe informații sunt disponibile [aici](https://concept.research.microsoft.com/Home/Api). \n", + "Iată un exemplu de URL pentru apel: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Putem observa că substantivele nu ne oferă grupuri tematice mari. Să înlocuim substantivele cu termeni mai generali obținuți din graful de concepte. Acest lucru va dura ceva timp, deoarece facem apeluri REST pentru fiecare expresie nominală.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Să încercăm să clasificăm titlurile știrilor folosind concepte părinte. Pentru a obține titlurile știrilor, vom folosi serviciul [NewsApi.org](http://newsapi.org). Trebuie să obțineți propria cheie API pentru a utiliza serviciul - accesați site-ul web și înregistrați-vă pentru planul gratuit de dezvoltator.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "În primul rând, vrem să putem extrage substantive din titlurile știrilor. Vom folosi biblioteca `TextBlob` pentru a face acest lucru, care simplifică multe sarcini tipice de procesare a limbajului natural, cum ar fi aceasta.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Putem observa că substantivele nu ne oferă grupuri tematice mari. Să înlocuim substantivele cu termeni mai generali obținuți din graful de concepte. Acest lucru va dura ceva timp, deoarece facem apeluri REST pentru fiecare expresie nominală.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**Declinare de responsabilitate**: \nAcest document a fost tradus folosind serviciul de traducere AI [Co-op Translator](https://github.com/Azure/co-op-translator). Deși ne străduim să asigurăm acuratețea, vă rugăm să fiți conștienți că traducerile automate pot conține erori sau inexactități. Documentul original în limba sa natală ar trebui considerat sursa autoritară. Pentru informații critice, se recomandă traducerea profesională realizată de un specialist. Nu ne asumăm responsabilitatea pentru eventualele neînțelegeri sau interpretări greșite care pot apărea din utilizarea acestei traduceri.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**Declinare de responsabilitate**: \nAcest document a fost tradus folosind serviciul de traducere AI [Co-op Translator](https://github.com/Azure/co-op-translator). Deși ne străduim să asigurăm acuratețea, vă rugăm să fiți conștienți că traducerile automate pot conține erori sau inexactități. Documentul original în limba sa natală ar trebui considerat sursa autoritară. Pentru informații critice, se recomandă traducerea profesională realizată de un specialist. Nu ne asumăm responsabilitatea pentru eventualele neînțelegeri sau interpretări greșite care pot apărea din utilizarea acestei traduceri.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:58:32+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "ro" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:58:32+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "ro" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/ru/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/ru/lessons/2-Symbolic/MSConceptGraph.ipynb index 6b88893a..ee4726b9 100644 --- a/translations/ru/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/ru/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - ЭТОТ API БОЛЬШЕ НЕДОСТУПЕН, НО ПОЖАЛУЙСТА, ИЗУЧИТЕ БЛОКНОТ, ЧТОБЫ ПОНЯТЬ КОНЦЕПЦИЮ\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) — это большая таксономия терминов, извлеченных из интернета, с отношениями типа `is-a` между концепциями.\n", - "\n", - "Context Graph доступен в двух формах:\n", - " * Большой текстовый файл для скачивания\n", - " * REST API\n", - "\n", - "Статистика:\n", - " * 5401933 уникальных концептов,\n", - " * 12551613 уникальных экземпляров\n", - " * 87603947 отношений типа `is-a`\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Использование веб-сервиса\n", - "\n", - "Веб-сервис предлагает различные вызовы для оценки вероятности принадлежности концепта к различным группам. Дополнительная информация доступна [здесь](https://concept.research.microsoft.com/Home/Api). \n", - "Вот пример URL для вызова: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - ЭТОТ API БОЛЬШЕ НЕДОСТУПЕН, НО ПОЖАЛУЙСТА, ИЗУЧИТЕ БЛОКНОТ, ЧТОБЫ ПОНЯТЬ КОНЦЕПЦИЮ\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) — это большая таксономия терминов, извлеченных из интернета, с отношениями типа `is-a` между концепциями.\n", + "\n", + "Context Graph доступен в двух формах:\n", + " * Большой текстовый файл для скачивания\n", + " * REST API\n", + "\n", + "Статистика:\n", + " * 5401933 уникальных концептов,\n", + " * 12551613 уникальных экземпляров\n", + " * 87603947 отношений типа `is-a`\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Давайте попробуем классифицировать заголовки новостей, используя родительские концепции. Чтобы получить заголовки новостей, мы будем использовать сервис [NewsApi.org](http://newsapi.org). Вам нужно получить собственный API-ключ, чтобы использовать этот сервис - перейдите на веб-сайт и зарегистрируйтесь для бесплатного плана разработчика.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Прежде всего, мы хотим иметь возможность извлекать существительные из заголовков новостей. Для этого мы будем использовать библиотеку `TextBlob`, которая упрощает многие типичные задачи обработки естественного языка, такие как эта.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Использование веб-сервиса\n", + "\n", + "Веб-сервис предлагает различные вызовы для оценки вероятности принадлежности концепта к различным группам. Дополнительная информация доступна [здесь](https://concept.research.microsoft.com/Home/Api). \n", + "Вот пример URL для вызова: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Мы видим, что существительные не дают нам больших тематических групп. Давайте заменим существительные на более общие термины, полученные из графа концепций. Это займет некоторое время, так как мы выполняем REST-запрос для каждой именной фразы.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Давайте попробуем классифицировать заголовки новостей, используя родительские концепции. Чтобы получить заголовки новостей, мы будем использовать сервис [NewsApi.org](http://newsapi.org). Вам нужно получить собственный API-ключ, чтобы использовать этот сервис - перейдите на веб-сайт и зарегистрируйтесь для бесплатного плана разработчика.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Прежде всего, мы хотим иметь возможность извлекать существительные из заголовков новостей. Для этого мы будем использовать библиотеку `TextBlob`, которая упрощает многие типичные задачи обработки естественного языка, такие как эта.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Мы видим, что существительные не дают нам больших тематических групп. Давайте заменим существительные на более общие термины, полученные из графа концепций. Это займет некоторое время, так как мы выполняем REST-запрос для каждой именной фразы.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**Отказ от ответственности**: \nЭтот документ был переведен с помощью сервиса автоматического перевода [Co-op Translator](https://github.com/Azure/co-op-translator). Несмотря на наши усилия обеспечить точность перевода, автоматические переводы могут содержать ошибки или неточности. Оригинальный документ на его родном языке следует считать авторитетным источником. Для получения критически важной информации рекомендуется профессиональный перевод человеком. Мы не несем ответственности за любые недоразумения или неправильные интерпретации, возникшие в результате использования данного перевода.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**Отказ от ответственности**: \nЭтот документ был переведен с помощью сервиса автоматического перевода [Co-op Translator](https://github.com/Azure/co-op-translator). Несмотря на наши усилия обеспечить точность перевода, автоматические переводы могут содержать ошибки или неточности. Оригинальный документ на его родном языке следует считать авторитетным источником. Для получения критически важной информации рекомендуется профессиональный перевод человеком. Мы не несем ответственности за любые недоразумения или неправильные интерпретации, возникшие в результате использования данного перевода.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:50:24+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "ru" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:50:24+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "ru" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/sk/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/sk/lessons/2-Symbolic/MSConceptGraph.ipynb index 321bd8fa..22a4b839 100644 --- a/translations/sk/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/sk/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - TÁTO API UŽ NIE JE DOSTUPNÁ, ALE POZRITE SI ZOŠIT NA POROZUMENIE KONCEPTU\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) je rozsiahla taxonómia pojmov získaná z internetu, ktorá obsahuje vzťahy typu `is-a` medzi konceptmi.\n", - "\n", - "Context Graph je dostupný v dvoch formách:\n", - " * Veľký textový súbor na stiahnutie\n", - " * REST API\n", - "\n", - "Štatistiky:\n", - " * 5401933 unikátnych konceptov,\n", - " * 12551613 unikátnych inštancií\n", - " * 87603947 vzťahov typu `is-a`\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Používanie webovej služby\n", - "\n", - "Webová služba ponúka rôzne volania na odhad pravdepodobnosti, že koncept patrí do rôznych skupín. Viac informácií nájdete [tu](https://concept.research.microsoft.com/Home/Api). \n", - "Tu je ukážková URL na volanie: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - TÁTO API UŽ NIE JE DOSTUPNÁ, ALE POZRITE SI ZOŠIT NA POROZUMENIE KONCEPTU\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) je rozsiahla taxonómia pojmov získaná z internetu, ktorá obsahuje vzťahy typu `is-a` medzi konceptmi.\n", + "\n", + "Context Graph je dostupný v dvoch formách:\n", + " * Veľký textový súbor na stiahnutie\n", + " * REST API\n", + "\n", + "Štatistiky:\n", + " * 5401933 unikátnych konceptov,\n", + " * 12551613 unikátnych inštancií\n", + " * 87603947 vzťahov typu `is-a`\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Poďme sa pokúsiť kategorizovať názvy správ pomocou nadradených konceptov. Na získanie názvov správ použijeme službu [NewsApi.org](http://newsapi.org). Na používanie služby si musíte zaobstarať vlastný API kľúč - prejdite na webovú stránku a zaregistrujte sa na bezplatný plán pre vývojárov.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "V prvom rade chceme byť schopní extrahovať podstatné mená z nadpisov správ. Na tento účel použijeme knižnicu `TextBlob`, ktorá zjednodušuje veľa typických úloh NLP, ako je táto.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Používanie webovej služby\n", + "\n", + "Webová služba ponúka rôzne volania na odhad pravdepodobnosti, že koncept patrí do rôznych skupín. Viac informácií nájdete [tu](https://concept.research.microsoft.com/Home/Api). \n", + "Tu je ukážková URL na volanie: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Vidíme, že podstatné mená nám neposkytujú veľké tematické skupiny. Nahradíme podstatné mená všeobecnejšími pojmami získanými z konceptuálneho grafu. To bude chvíľu trvať, pretože robíme REST volanie pre každú frázu s podstatným menom.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Poďme sa pokúsiť kategorizovať názvy správ pomocou nadradených konceptov. Na získanie názvov správ použijeme službu [NewsApi.org](http://newsapi.org). Na používanie služby si musíte zaobstarať vlastný API kľúč - prejdite na webovú stránku a zaregistrujte sa na bezplatný plán pre vývojárov.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "V prvom rade chceme byť schopní extrahovať podstatné mená z nadpisov správ. Na tento účel použijeme knižnicu `TextBlob`, ktorá zjednodušuje veľa typických úloh NLP, ako je táto.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Vidíme, že podstatné mená nám neposkytujú veľké tematické skupiny. Nahradíme podstatné mená všeobecnejšími pojmami získanými z konceptuálneho grafu. To bude chvíľu trvať, pretože robíme REST volanie pre každú frázu s podstatným menom.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**Upozornenie**: \nTento dokument bol preložený pomocou služby AI prekladu [Co-op Translator](https://github.com/Azure/co-op-translator). Hoci sa snažíme o presnosť, prosím, berte na vedomie, že automatizované preklady môžu obsahovať chyby alebo nepresnosti. Pôvodný dokument v jeho rodnom jazyku by mal byť považovaný za autoritatívny zdroj. Pre kritické informácie sa odporúča profesionálny ľudský preklad. Nenesieme zodpovednosť za akékoľvek nedorozumenia alebo nesprávne interpretácie vyplývajúce z použitia tohto prekladu.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**Upozornenie**: \nTento dokument bol preložený pomocou služby AI prekladu [Co-op Translator](https://github.com/Azure/co-op-translator). Hoci sa snažíme o presnosť, prosím, berte na vedomie, že automatizované preklady môžu obsahovať chyby alebo nepresnosti. Pôvodný dokument v jeho rodnom jazyku by mal byť považovaný za autoritatívny zdroj. Pre kritické informácie sa odporúča profesionálny ľudský preklad. Nenesieme zodpovednosť za akékoľvek nedorozumenia alebo nesprávne interpretácie vyplývajúce z použitia tohto prekladu.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:58:20+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "sk" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:58:20+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "sk" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/sl/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/sl/lessons/2-Symbolic/MSConceptGraph.ipynb index 034b22a8..7917805a 100644 --- a/translations/sl/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/sl/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - TA API NI VEČ NA VOLJO, VENDAR SI OGLEJTE ZVEK ZA RAZUMEVANJE KONCEPTA\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) je velika taksonomija pojmov, pridobljenih z interneta, z relacijami `is-a` med koncepti.\n", - "\n", - "Context Graph je na voljo v dveh oblikah:\n", - " * Velika besedilna datoteka za prenos\n", - " * REST API\n", - "\n", - "Statistika:\n", - " * 5401933 edinstvenih konceptov,\n", - " * 12551613 edinstvenih primerov\n", - " * 87603947 relacij `is-a`\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Uporaba spletne storitve\n", - "\n", - "Spletna storitev ponuja različne klice za oceno verjetnosti, da koncept pripada različnim skupinam. Več informacij je na voljo [tukaj](https://concept.research.microsoft.com/Home/Api).\n", - "Tukaj je primer URL-ja za klic: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - TA API NI VEČ NA VOLJO, VENDAR SI OGLEJTE ZVEK ZA RAZUMEVANJE KONCEPTA\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) je velika taksonomija pojmov, pridobljenih z interneta, z relacijami `is-a` med koncepti.\n", + "\n", + "Context Graph je na voljo v dveh oblikah:\n", + " * Velika besedilna datoteka za prenos\n", + " * REST API\n", + "\n", + "Statistika:\n", + " * 5401933 edinstvenih konceptov,\n", + " * 12551613 edinstvenih primerov\n", + " * 87603947 relacij `is-a`\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Poskusimo razvrstiti naslove novic z uporabo nadrejenih konceptov. Za pridobitev naslovov novic bomo uporabili storitev [NewsApi.org](http://newsapi.org). Za uporabo storitve morate pridobiti svoj API ključ - obiščite spletno stran in se registrirajte za brezplačni razvijalski načrt.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Najprej želimo izluščiti samostalnike iz naslovov novic. Za to bomo uporabili knjižnico `TextBlob`, ki močno poenostavi številne tipične naloge NLP, kot je ta.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Uporaba spletne storitve\n", + "\n", + "Spletna storitev ponuja različne klice za oceno verjetnosti, da koncept pripada različnim skupinam. Več informacij je na voljo [tukaj](https://concept.research.microsoft.com/Home/Api).\n", + "Tukaj je primer URL-ja za klic: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Vidimo, da samostalniki ne tvorijo velikih tematskih skupin. Zamenjajmo samostalnike z bolj splošnimi izrazi, pridobljenimi iz konceptnega grafa. To bo vzelo nekaj časa, ker izvajamo REST klic za vsako samostalniško zvezo.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Poskusimo razvrstiti naslove novic z uporabo nadrejenih konceptov. Za pridobitev naslovov novic bomo uporabili storitev [NewsApi.org](http://newsapi.org). Za uporabo storitve morate pridobiti svoj API ključ - obiščite spletno stran in se registrirajte za brezplačni razvijalski načrt.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Najprej želimo izluščiti samostalnike iz naslovov novic. Za to bomo uporabili knjižnico `TextBlob`, ki močno poenostavi številne tipične naloge NLP, kot je ta.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Vidimo, da samostalniki ne tvorijo velikih tematskih skupin. Zamenjajmo samostalnike z bolj splošnimi izrazi, pridobljenimi iz konceptnega grafa. To bo vzelo nekaj časa, ker izvajamo REST klic za vsako samostalniško zvezo.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**Omejitev odgovornosti**: \nTa dokument je bil preveden z uporabo storitve AI za prevajanje [Co-op Translator](https://github.com/Azure/co-op-translator). Čeprav si prizadevamo za natančnost, vas prosimo, da upoštevate, da lahko avtomatizirani prevodi vsebujejo napake ali netočnosti. Izvirni dokument v njegovem izvirnem jeziku je treba obravnavati kot avtoritativni vir. Za ključne informacije priporočamo profesionalni človeški prevod. Ne prevzemamo odgovornosti za morebitna nesporazumevanja ali napačne razlage, ki izhajajo iz uporabe tega prevoda.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**Omejitev odgovornosti**: \nTa dokument je bil preveden z uporabo storitve AI za prevajanje [Co-op Translator](https://github.com/Azure/co-op-translator). Čeprav si prizadevamo za natančnost, vas prosimo, da upoštevate, da lahko avtomatizirani prevodi vsebujejo napake ali netočnosti. Izvirni dokument v njegovem izvirnem jeziku je treba obravnavati kot avtoritativni vir. Za ključne informacije priporočamo profesionalni človeški prevod. Ne prevzemamo odgovornosti za morebitna nesporazumevanja ali napačne razlage, ki izhajajo iz uporabe tega prevoda.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:59:31+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "sl" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:59:31+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "sl" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/sr/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/sr/lessons/2-Symbolic/MSConceptGraph.ipynb index 83c281ae..f3878ea5 100644 --- a/translations/sr/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/sr/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - ОВАЈ API ВИШЕ НИЈЕ ДОСТУПАН, АЛИ ПОГЛЕДАЈТЕ NOTEBOOK ДА БИСТЕ РАЗУМЕЛИ КОНЦЕПТ\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) је велика таксономија термина добијених са интернета, са `is-a` релацијама између концепата.\n", - "\n", - "Context Graph је доступан у два облика:\n", - " * Велика текстуална датотека за преузимање\n", - " * REST API\n", - "\n", - "Статистика:\n", - " * 5401933 јединствених концепата,\n", - " * 12551613 јединствених инстанци\n", - " * 87603947 `is-a` релација\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Коришћење веб сервиса\n", - "\n", - "Веб сервис нуди различите позиве за процену вероватноће да концепт припада различитим групама. Више информација је доступно [овде](https://concept.research.microsoft.com/Home/Api). \n", - "Ево примера URL-а за позив: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - ОВАЈ API ВИШЕ НИЈЕ ДОСТУПАН, АЛИ ПОГЛЕДАЈТЕ NOTEBOOK ДА БИСТЕ РАЗУМЕЛИ КОНЦЕПТ\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) је велика таксономија термина добијених са интернета, са `is-a` релацијама између концепата.\n", + "\n", + "Context Graph је доступан у два облика:\n", + " * Велика текстуална датотека за преузимање\n", + " * REST API\n", + "\n", + "Статистика:\n", + " * 5401933 јединствених концепата,\n", + " * 12551613 јединствених инстанци\n", + " * 87603947 `is-a` релација\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Хајде да покушамо да категоришемо наслове вести користећи родитељске концепте. Да бисмо добили наслове вести, користићемо услугу [NewsApi.org](http://newsapi.org). Потребно је да набавите сопствени API кључ како бисте користили услугу - идите на веб сајт и региструјте се за бесплатни план за програмере.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Пре свега, желимо да будемо у могућности да издвајамо именице из наслова вести. Користићемо библиотеку `TextBlob` за ово, која поједностављује многе типичне задатке обраде природног језика као што је овај.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Коришћење веб сервиса\n", + "\n", + "Веб сервис нуди различите позиве за процену вероватноће да концепт припада различитим групама. Више информација је доступно [овде](https://concept.research.microsoft.com/Home/Api). \n", + "Ево примера URL-а за позив: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Можемо видети да именице не дају велике тематске групе. Хајде да заменимо именице општијим терминима добијеним из графа концепата. Ово ће потрајати, јер правимо REST позив за сваку именицу.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Хајде да покушамо да категоришемо наслове вести користећи родитељске концепте. Да бисмо добили наслове вести, користићемо услугу [NewsApi.org](http://newsapi.org). Потребно је да набавите сопствени API кључ како бисте користили услугу - идите на веб сајт и региструјте се за бесплатни план за програмере.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Пре свега, желимо да будемо у могућности да издвајамо именице из наслова вести. Користићемо библиотеку `TextBlob` за ово, која поједностављује многе типичне задатке обраде природног језика као што је овај.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Можемо видети да именице не дају велике тематске групе. Хајде да заменимо именице општијим терминима добијеним из графа концепата. Ово ће потрајати, јер правимо REST позив за сваку именицу.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**Одрицање од одговорности**: \nОвај документ је преведен помоћу услуге за превођење уз помоћ вештачке интелигенције [Co-op Translator](https://github.com/Azure/co-op-translator). Иако се трудимо да обезбедимо тачност, молимо вас да имате у виду да аутоматски преводи могу садржати грешке или нетачности. Оригинални документ на његовом изворном језику треба сматрати ауторитативним извором. За критичне информације препоручује се професионални превод од стране људи. Не преузимамо одговорност за било каква погрешна тумачења или неспоразуме који могу настати услед коришћења овог превода.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**Одрицање од одговорности**: \nОвај документ је преведен помоћу услуге за превођење уз помоћ вештачке интелигенције [Co-op Translator](https://github.com/Azure/co-op-translator). Иако се трудимо да обезбедимо тачност, молимо вас да имате у виду да аутоматски преводи могу садржати грешке или нетачности. Оригинални документ на његовом изворном језику треба сматрати ауторитативним извором. За критичне информације препоручује се професионални превод од стране људи. Не преузимамо одговорност за било каква погрешна тумачења или неспоразуме који могу настати услед коришћења овог превода.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:59:01+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "sr" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:59:01+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "sr" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/sv/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/sv/lessons/2-Symbolic/MSConceptGraph.ipynb index 361694a1..b4fc4a45 100644 --- a/translations/sv/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/sv/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - DENNA API ÄR INTE LÄNGRE TILLGÄNGLIG MEN KOLLA GÄRNA PÅ NOTEBOOK FÖR ATT FÖRSTÅ KONCEPTET\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) är en stor taxonomi av termer hämtade från internet, med `is-a`-relationer mellan koncept.\n", - "\n", - "Context Graph finns tillgänglig i två former:\n", - " * Stor textfil för nedladdning\n", - " * REST API\n", - "\n", - "Statistik:\n", - " * 5401933 unika koncept\n", - " * 12551613 unika instanser\n", - " * 87603947 `is-a`-relationer\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Använda webbtjänsten\n", - "\n", - "Webbtjänsten erbjuder olika anrop för att uppskatta sannolikheten för att ett koncept tillhör olika grupper. Mer information finns [här](https://concept.research.microsoft.com/Home/Api). \n", - "Här är exempel-URL för anrop: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - DENNA API ÄR INTE LÄNGRE TILLGÄNGLIG MEN KOLLA GÄRNA PÅ NOTEBOOK FÖR ATT FÖRSTÅ KONCEPTET\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) är en stor taxonomi av termer hämtade från internet, med `is-a`-relationer mellan koncept.\n", + "\n", + "Context Graph finns tillgänglig i två former:\n", + " * Stor textfil för nedladdning\n", + " * REST API\n", + "\n", + "Statistik:\n", + " * 5401933 unika koncept\n", + " * 12551613 unika instanser\n", + " * 87603947 `is-a`-relationer\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Låt oss försöka kategorisera nyhetstitlarna med hjälp av överordnade koncept. För att få nyhetstitlar kommer vi att använda tjänsten [NewsApi.org](http://newsapi.org). Du behöver skaffa din egen API-nyckel för att använda tjänsten - gå till webbplatsen och registrera dig för den kostnadsfria utvecklarplanen.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Först och främst vill vi kunna extrahera substantiv från nyhetstitlar. Vi kommer att använda biblioteket `TextBlob` för att göra detta, vilket förenklar många typiska NLP-uppgifter som denna.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Använda webbtjänsten\n", + "\n", + "Webbtjänsten erbjuder olika anrop för att uppskatta sannolikheten för att ett koncept tillhör olika grupper. Mer information finns [här](https://concept.research.microsoft.com/Home/Api). \n", + "Här är exempel-URL för anrop: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Vi kan se att substantiv inte ger oss stora tematiska grupper. Låt oss ersätta substantiv med mer generella termer som erhålls från konceptgrafen. Detta kommer att ta lite tid, eftersom vi gör REST-anrop för varje substantivfras.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Låt oss försöka kategorisera nyhetstitlarna med hjälp av överordnade koncept. För att få nyhetstitlar kommer vi att använda tjänsten [NewsApi.org](http://newsapi.org). Du behöver skaffa din egen API-nyckel för att använda tjänsten - gå till webbplatsen och registrera dig för den kostnadsfria utvecklarplanen.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Först och främst vill vi kunna extrahera substantiv från nyhetstitlar. Vi kommer att använda biblioteket `TextBlob` för att göra detta, vilket förenklar många typiska NLP-uppgifter som denna.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Vi kan se att substantiv inte ger oss stora tematiska grupper. Låt oss ersätta substantiv med mer generella termer som erhålls från konceptgrafen. Detta kommer att ta lite tid, eftersom vi gör REST-anrop för varje substantivfras.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**Ansvarsfriskrivning**: \nDetta dokument har översatts med hjälp av AI-översättningstjänsten [Co-op Translator](https://github.com/Azure/co-op-translator). Även om vi strävar efter noggrannhet, bör det noteras att automatiserade översättningar kan innehålla fel eller felaktigheter. Det ursprungliga dokumentet på dess ursprungliga språk bör betraktas som den auktoritativa källan. För kritisk information rekommenderas professionell mänsklig översättning. Vi ansvarar inte för eventuella missförstånd eller feltolkningar som uppstår vid användning av denna översättning.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**Ansvarsfriskrivning**: \nDetta dokument har översatts med hjälp av AI-översättningstjänsten [Co-op Translator](https://github.com/Azure/co-op-translator). Även om vi strävar efter noggrannhet, bör det noteras att automatiserade översättningar kan innehålla fel eller felaktigheter. Det ursprungliga dokumentet på dess ursprungliga språk bör betraktas som den auktoritativa källan. För kritisk information rekommenderas professionell mänsklig översättning. Vi ansvarar inte för eventuella missförstånd eller feltolkningar som uppstår vid användning av denna översättning.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:55:24+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "sv" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:55:24+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "sv" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/sw/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/sw/lessons/2-Symbolic/MSConceptGraph.ipynb index f9eb1db5..9f00d304 100644 --- a/translations/sw/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/sw/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - API HII HAIPATIKANI TENA LAKINI TAFADHALI TAZAMA DAFTARI KUELEWA DHANA\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) ni mfumo mkubwa wa maneno uliotolewa kutoka mtandao, ukiwa na mahusiano ya `is-a` kati ya dhana.\n", - "\n", - "Context Graph inapatikana kwa njia mbili:\n", - " * Faili kubwa la maandishi kwa kupakua\n", - " * REST API\n", - "\n", - "Takwimu:\n", - " * Dhana 5401933 za kipekee,\n", - " * Matukio 12551613 ya kipekee\n", - " * Mahusiano 87603947 ya `is-a`\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Kutumia Huduma ya Wavuti\n", - "\n", - "Huduma ya wavuti inatoa miito mbalimbali ya kukadiria uwezekano wa dhana kuwa sehemu ya makundi tofauti. Maelezo zaidi yanapatikana [hapa](https://concept.research.microsoft.com/Home/Api).\n", - "Hii hapa ni URL ya mfano ya kuita: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - API HII HAIPATIKANI TENA LAKINI TAFADHALI TAZAMA DAFTARI KUELEWA DHANA\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) ni mfumo mkubwa wa maneno uliotolewa kutoka mtandao, ukiwa na mahusiano ya `is-a` kati ya dhana.\n", + "\n", + "Context Graph inapatikana kwa njia mbili:\n", + " * Faili kubwa la maandishi kwa kupakua\n", + " * REST API\n", + "\n", + "Takwimu:\n", + " * Dhana 5401933 za kipekee,\n", + " * Matukio 12551613 ya kipekee\n", + " * Mahusiano 87603947 ya `is-a`\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Hebu jaribu kuainisha vichwa vya habari vya habari kwa kutumia dhana za wazazi. Ili kupata vichwa vya habari vya habari, tutatumia huduma ya [NewsApi.org](http://newsapi.org). Unahitaji kupata funguo yako ya API ili kutumia huduma - nenda kwenye tovuti na jisajili kwa mpango wa msanidi programu wa bure.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Kwanza kabisa, tunataka kuwa na uwezo wa kutoa nomino kutoka kwa vichwa vya habari. Tutatumia maktaba ya `TextBlob` kufanya hili, ambayo inarahisisha kazi nyingi za kawaida za NLP kama hii.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Kutumia Huduma ya Wavuti\n", + "\n", + "Huduma ya wavuti inatoa miito mbalimbali ya kukadiria uwezekano wa dhana kuwa sehemu ya makundi tofauti. Maelezo zaidi yanapatikana [hapa](https://concept.research.microsoft.com/Home/Api).\n", + "Hii hapa ni URL ya mfano ya kuita: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Tunaweza kuona kwamba nomino hazitupi makundi makubwa ya mada. Wacha tubadilishe nomino kwa maneno ya jumla zaidi yaliyopatikana kutoka kwa grafu ya dhana. Hii itachukua muda, kwa sababu tunafanya simu za REST kwa kila kifungu cha nomino.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Hebu jaribu kuainisha vichwa vya habari vya habari kwa kutumia dhana za wazazi. Ili kupata vichwa vya habari vya habari, tutatumia huduma ya [NewsApi.org](http://newsapi.org). Unahitaji kupata funguo yako ya API ili kutumia huduma - nenda kwenye tovuti na jisajili kwa mpango wa msanidi programu wa bure.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Kwanza kabisa, tunataka kuwa na uwezo wa kutoa nomino kutoka kwa vichwa vya habari. Tutatumia maktaba ya `TextBlob` kufanya hili, ambayo inarahisisha kazi nyingi za kawaida za NLP kama hii.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Tunaweza kuona kwamba nomino hazitupi makundi makubwa ya mada. Wacha tubadilishe nomino kwa maneno ya jumla zaidi yaliyopatikana kutoka kwa grafu ya dhana. Hii itachukua muda, kwa sababu tunafanya simu za REST kwa kila kifungu cha nomino.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**Kanusho**: \nHati hii imetafsiriwa kwa kutumia huduma ya kutafsiri ya AI [Co-op Translator](https://github.com/Azure/co-op-translator). Ingawa tunajitahidi kuhakikisha usahihi, tafadhali fahamu kuwa tafsiri za kiotomatiki zinaweza kuwa na makosa au kutokuwa sahihi. Hati ya asili katika lugha yake ya awali inapaswa kuzingatiwa kama chanzo cha mamlaka. Kwa taarifa muhimu, tafsiri ya kitaalamu ya binadamu inapendekezwa. Hatutawajibika kwa kutoelewana au tafsiri zisizo sahihi zinazotokana na matumizi ya tafsiri hii.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**Kanusho**: \nHati hii imetafsiriwa kwa kutumia huduma ya kutafsiri ya AI [Co-op Translator](https://github.com/Azure/co-op-translator). Ingawa tunajitahidi kuhakikisha usahihi, tafadhali fahamu kuwa tafsiri za kiotomatiki zinaweza kuwa na makosa au kutokuwa sahihi. Hati ya asili katika lugha yake ya awali inapaswa kuzingatiwa kama chanzo cha mamlaka. Kwa taarifa muhimu, tafsiri ya kitaalamu ya binadamu inapendekezwa. Hatutawajibika kwa kutoelewana au tafsiri zisizo sahihi zinazotokana na matumizi ya tafsiri hii.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:57:37+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "sw" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:57:37+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "sw" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/ta/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/ta/lessons/2-Symbolic/MSConceptGraph.ipynb index 90066305..f9971d55 100644 --- a/translations/ta/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/ta/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,547 +1,538 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - இந்த API இனி கிடைக்காது, ஆனால் கருத்தை புரிந்துகொள்ள நோட்புக் பார்க்கவும்\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) என்பது இணையத்திலிருந்து எடுக்கப்பட்ட பெரிய சொற்களின் வரிசை ஆகும், இதில் கருத்துகளுக்கு இடையே `is-a` உறவுகள் உள்ளன.\n", - "\n", - "Context Graph இரண்டு வடிவங்களில் கிடைக்கிறது:\n", - " * பதிவிறக்கத்திற்கான பெரிய உரை கோப்பு\n", - " * REST API\n", - "\n", - "புள்ளிவிவரங்கள்:\n", - " * 5401933 தனித்துவமான கருத்துகள், \n", - " * 12551613 தனித்துவமான நிகழ்வுகள்\n", - " * 87603947 `is-a` உறவுகள்\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## வலை சேவையைப் பயன்படுத்துதல்\n", - "\n", - "வலை சேவை, ஒரு கருத்து பல்வேறு குழுக்களுக்கு属படுவதற்கான சாத்தியக்கூறுகளை மதிப்பீடு செய்ய பல்வேறு அழைப்புகளை வழங்குகிறது. மேலும் தகவலுக்கு [இங்கே](https://concept.research.microsoft.com/Home/Api) பார்க்கவும். அழைக்க உதாரண URL இங்கே உள்ளது: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - இந்த API இனி கிடைக்காது, ஆனால் கருத்தை புரிந்துகொள்ள நோட்புக் பார்க்கவும்\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) என்பது இணையத்திலிருந்து எடுக்கப்பட்ட பெரிய சொற்களின் வரிசை ஆகும், இதில் கருத்துகளுக்கு இடையே `is-a` உறவுகள் உள்ளன.\n", + "\n", + "Context Graph இரண்டு வடிவங்களில் கிடைக்கிறது:\n", + " * பதிவிறக்கத்திற்கான பெரிய உரை கோப்பு\n", + " * REST API\n", + "\n", + "புள்ளிவிவரங்கள்:\n", + " * 5401933 தனித்துவமான கருத்துகள், \n", + " * 12551613 தனித்துவமான நிகழ்வுகள்\n", + " * 87603947 `is-a` உறவுகள்\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "நாம் செய்தி தலைப்புகளை பெற்றோர்கள் கருத்துக்களைக் கொண்டு வகைப்படுத்த முயற்சிக்கலாம். செய்தி தலைப்புகளை பெற [NewsApi.org](http://newsapi.org) சேவையை பயன்படுத்துவோம். இந்த சேவையை பயன்படுத்த, நீங்கள் உங்கள் சொந்த API விசையை பெற வேண்டும் - இணையதளத்திற்கு சென்று இலவச டெவலப்பர் திட்டத்திற்கு பதிவு செய்யவும்.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "முதலில், நாங்கள் செய்தி தலைப்புகளில் இருந்து பெயர்ச்சொற்களை எடுக்க விரும்புகிறோம். இதை செய்ய, `TextBlob` நூலகத்தை பயன்படுத்துவோம், இது இவ்வாறான பொதுவான இயற்கை மொழி செயலாக்க பணிகளை மிகவும் எளிமையாக்குகிறது.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## வலை சேவையைப் பயன்படுத்துதல்\n", + "\n", + "வலை சேவை, ஒரு கருத்து பல்வேறு குழுக்களுக்கு属படுவதற்கான சாத்தியக்கூறுகளை மதிப்பீடு செய்ய பல்வேறு அழைப்புகளை வழங்குகிறது. மேலும் தகவலுக்கு [இங்கே](https://concept.research.microsoft.com/Home/Api) பார்க்கவும். அழைக்க உதாரண URL இங்கே உள்ளது: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "நாம் பெயர்ச்சொற்கள் பெரிய தீமைக் குழுக்களை வழங்கவில்லை என்பதை காணலாம். பெயர்ச்சொற்களை கருத்து வரைபடத்தில் இருந்து பெறப்பட்ட பொதுவான சொற்களால் மாற்றலாம். இது சில நேரம் எடுக்கும், ஏனெனில் ஒவ்வொரு பெயர்ச்சொல் வாக்கியத்திற்கும் REST அழைப்பு செய்கிறோம்.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "நாம் செய்தி தலைப்புகளை பெற்றோர்கள் கருத்துக்களைக் கொண்டு வகைப்படுத்த முயற்சிக்கலாம். செய்தி தலைப்புகளை பெற [NewsApi.org](http://newsapi.org) சேவையை பயன்படுத்துவோம். இந்த சேவையை பயன்படுத்த, நீங்கள் உங்கள் சொந்த API விசையை பெற வேண்டும் - இணையதளத்திற்கு சென்று இலவச டெவலப்பர் திட்டத்திற்கு பதிவு செய்யவும்.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "முதலில், நாங்கள் செய்தி தலைப்புகளில் இருந்து பெயர்ச்சொற்களை எடுக்க விரும்புகிறோம். இதை செய்ய, `TextBlob` நூலகத்தை பயன்படுத்துவோம், இது இவ்வாறான பொதுவான இயற்கை மொழி செயலாக்க பணிகளை மிகவும் எளிமையாக்குகிறது.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "நாம் பெயர்ச்சொற்கள் பெரிய தீமைக் குழுக்களை வழங்கவில்லை என்பதை காணலாம். பெயர்ச்சொற்களை கருத்து வரைபடத்தில் இருந்து பெறப்பட்ட பொதுவான சொற்களால் மாற்றலாம். இது சில நேரம் எடுக்கும், ஏனெனில் ஒவ்வொரு பெயர்ச்சொல் வாக்கியத்திற்கும் REST அழைப்பு செய்கிறோம்.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**குறிப்பு**: \nஇந்த ஆவணம் [Co-op Translator](https://github.com/Azure/co-op-translator) என்ற AI மொழிபெயர்ப்பு சேவையைப் பயன்படுத்தி மொழிபெயர்க்கப்பட்டுள்ளது. எங்கள் தரச்செயல்முறைகளுக்கு நாங்கள் முயற்சிக்கிறோம், ஆனால் தானியக்க மொழிபெயர்ப்புகளில் பிழைகள் அல்லது தவறான தகவல்கள் இருக்கக்கூடும் என்பதை தயவுசெய்து கவனத்தில் கொள்ளுங்கள். அதன் தாய்மொழியில் உள்ள மூல ஆவணம் அதிகாரப்பூர்வ ஆதாரமாக கருதப்பட வேண்டும். முக்கியமான தகவல்களுக்கு, தொழில்முறை மனித மொழிபெயர்ப்பு பரிந்துரைக்கப்படுகிறது. இந்த மொழிபெயர்ப்பைப் பயன்படுத்துவதால் ஏற்படும் எந்த தவறான புரிதல்கள் அல்லது தவறான விளக்கங்களுக்கு நாங்கள் பொறுப்பல்ல.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**குறிப்பு**: \nஇந்த ஆவணம் [Co-op Translator](https://github.com/Azure/co-op-translator) என்ற AI மொழிபெயர்ப்பு சேவையைப் பயன்படுத்தி மொழிபெயர்க்கப்பட்டுள்ளது. எங்கள் தரச்செயல்முறைகளுக்கு நாங்கள் முயற்சிக்கிறோம், ஆனால் தானியக்க மொழிபெயர்ப்புகளில் பிழைகள் அல்லது தவறான தகவல்கள் இருக்கக்கூடும் என்பதை தயவுசெய்து கவனத்தில் கொள்ளுங்கள். அதன் தாய்மொழியில் உள்ள மூல ஆவணம் அதிகாரப்பூர்வ ஆதாரமாக கருதப்பட வேண்டும். முக்கியமான தகவல்களுக்கு, தொழில்முறை மனித மொழிபெயர்ப்பு பரிந்துரைக்கப்படுகிறது. இந்த மொழிபெயர்ப்பைப் பயன்படுத்துவதால் ஏற்படும் எந்த தவறான புரிதல்கள் அல்லது தவறான விளக்கங்களுக்கு நாங்கள் பொறுப்பல்ல.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-11T12:36:00+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "ta" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-11T12:36:00+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "ta" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/te/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/te/lessons/2-Symbolic/MSConceptGraph.ipynb index a204d2d6..a94701cb 100644 --- a/translations/te/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/te/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - ఈ API ఇక అందుబాటులో లేదు కానీ కాన్సెప్ట్ అర్థం చేసుకోవడానికి నోట్బుక్ చూడండి\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) అనేది ఇంటర్నెట్ నుండి సేకరించిన పెద్ద టాక్సోనమీ, ఇందులో కాన్సెప్ట్‌ల మధ్య `is-a` సంబంధాలు ఉంటాయి.\n", - "\n", - "Context Graph రెండు రూపాల్లో అందుబాటులో ఉంది:\n", - " * డౌన్లోడ్ కోసం పెద్ద టెక్స్ట్ ఫైల్\n", - " * REST API\n", - "\n", - "సంఖ్యాత్మక వివరాలు:\n", - " * 5401933 ప్రత్యేక కాన్సెప్ట్‌లు,\n", - " * 12551613 ప్రత్యేక ఉదాహరణలు\n", - " * 87603947 `is-a` సంబంధాలు\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## వెబ్ సర్వీస్ ఉపయోగించడం\n", - "\n", - "వెబ్ సర్వీస్ వివిధ గ్రూపులకు చెందిన ఒక కాన్సెప్ట్ యొక్క సంభావ్యత అంచనా వేయడానికి వివిధ కాల్స్ అందిస్తుంది. మరిన్ని వివరాలు [ఇక్కడ](https://concept.research.microsoft.com/Home/Api) అందుబాటులో ఉన్నాయి. \n", - "కాల్ చేయడానికి నమూనా URL ఇది: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" - ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "వార్తా శీర్షికలను ప్రధాన భావాల ఆధారంగా వర్గీకరించడానికి ప్రయత్నిద్దాం. వార్తా శీర్షికలను పొందడానికి, మనం [NewsApi.org](http://newsapi.org) సేవను ఉపయోగించబోతున్నాము. ఈ సేవను ఉపయోగించడానికి మీ స్వంత API కీని పొందాలి - వెబ్ సైట్‌కు వెళ్లి ఉచిత డెవలపర్ ప్లాన్ కోసం నమోదు చేసుకోండి.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "మొదటగా, మనం వార్త శీర్షికల నుండి నామవాచకాలను తీసుకోవగలగాలి. దీని కోసం మనం `TextBlob` లైబ్రరీని ఉపయోగించబోతున్నాము, ఇది ఇలాంటి సాధారణ NLP పనులను సులభతరం చేస్తుంది.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" - ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "మనం చూడగలము నామవాచకాలు పెద్ద థీమాటిక్ సమూహాలను ఇవ్వవు. మనం నామవాచకాలను కాన్సెప్ట్ గ్రాఫ్ నుండి పొందిన మరింత సాధారణ పదాలతో మార్చుకుందాం. ఇది కొంత సమయం తీసుకుంటుంది, ఎందుకంటే మనం ప్రతి నామవాచక పదబంధానికి REST కాల్ చేస్తున్నాము.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" - ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] - } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n\n\n**అస్పష్టత**: \nఈ పత్రాన్ని AI అనువాద సేవ [Co-op Translator](https://github.com/Azure/co-op-translator) ఉపయోగించి అనువదించబడింది. మేము ఖచ్చితత్వానికి ప్రయత్నించినప్పటికీ, ఆటోమేటెడ్ అనువాదాల్లో పొరపాట్లు లేదా తప్పిదాలు ఉండవచ్చు. మూల పత్రం దాని స్వదేశీ భాషలోనే అధికారిక మూలంగా పరిగణించాలి. ముఖ్యమైన సమాచారానికి, ప్రొఫెషనల్ మానవ అనువాదం సిఫార్సు చేయబడుతుంది. ఈ అనువాదం వాడకంలో ఏర్పడిన ఏవైనా అపార్థాలు లేదా తప్పుదారితీసే అర్థాలు కోసం మేము బాధ్యత వహించము.\n\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" - } - }, - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-11-25T23:51:34+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "te" - } - }, - "nbformat": 4, - "nbformat_minor": 2 +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - ఈ API ఇక అందుబాటులో లేదు కానీ కాన్సెప్ట్ అర్థం చేసుకోవడానికి నోట్బుక్ చూడండి\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) అనేది ఇంటర్నెట్ నుండి సేకరించిన పెద్ద టాక్సోనమీ, ఇందులో కాన్సెప్ట్‌ల మధ్య `is-a` సంబంధాలు ఉంటాయి.\n", + "\n", + "Context Graph రెండు రూపాల్లో అందుబాటులో ఉంది:\n", + " * డౌన్లోడ్ కోసం పెద్ద టెక్స్ట్ ఫైల్\n", + " * REST API\n", + "\n", + "సంఖ్యాత్మక వివరాలు:\n", + " * 5401933 ప్రత్యేక కాన్సెప్ట్‌లు,\n", + " * 12551613 ప్రత్యేక ఉదాహరణలు\n", + " * 87603947 `is-a` సంబంధాలు\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## వెబ్ సర్వీస్ ఉపయోగించడం\n", + "\n", + "వెబ్ సర్వీస్ వివిధ గ్రూపులకు చెందిన ఒక కాన్సెప్ట్ యొక్క సంభావ్యత అంచనా వేయడానికి వివిధ కాల్స్ అందిస్తుంది. మరిన్ని వివరాలు [ఇక్కడ](https://concept.research.microsoft.com/Home/Api) అందుబాటులో ఉన్నాయి. \n", + "కాల్ చేయడానికి నమూనా URL ఇది: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "వార్తా శీర్షికలను ప్రధాన భావాల ఆధారంగా వర్గీకరించడానికి ప్రయత్నిద్దాం. వార్తా శీర్షికలను పొందడానికి, మనం [NewsApi.org](http://newsapi.org) సేవను ఉపయోగించబోతున్నాము. ఈ సేవను ఉపయోగించడానికి మీ స్వంత API కీని పొందాలి - వెబ్ సైట్‌కు వెళ్లి ఉచిత డెవలపర్ ప్లాన్ కోసం నమోదు చేసుకోండి.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "మొదటగా, మనం వార్త శీర్షికల నుండి నామవాచకాలను తీసుకోవగలగాలి. దీని కోసం మనం `TextBlob` లైబ్రరీని ఉపయోగించబోతున్నాము, ఇది ఇలాంటి సాధారణ NLP పనులను సులభతరం చేస్తుంది.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "మనం చూడగలము నామవాచకాలు పెద్ద థీమాటిక్ సమూహాలను ఇవ్వవు. మనం నామవాచకాలను కాన్సెప్ట్ గ్రాఫ్ నుండి పొందిన మరింత సాధారణ పదాలతో మార్చుకుందాం. ఇది కొంత సమయం తీసుకుంటుంది, ఎందుకంటే మనం ప్రతి నామవాచక పదబంధానికి REST కాల్ చేస్తున్నాము.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n\n\n**అస్పష్టత**: \nఈ పత్రాన్ని AI అనువాద సేవ [Co-op Translator](https://github.com/Azure/co-op-translator) ఉపయోగించి అనువదించబడింది. మేము ఖచ్చితత్వానికి ప్రయత్నించినప్పటికీ, ఆటోమేటెడ్ అనువాదాల్లో పొరపాట్లు లేదా తప్పిదాలు ఉండవచ్చు. మూల పత్రం దాని స్వదేశీ భాషలోనే అధికారిక మూలంగా పరిగణించాలి. ముఖ్యమైన సమాచారానికి, ప్రొఫెషనల్ మానవ అనువాదం సిఫార్సు చేయబడుతుంది. ఈ అనువాదం వాడకంలో ఏర్పడిన ఏవైనా అపార్థాలు లేదా తప్పుదారితీసే అర్థాలు కోసం మేము బాధ్యత వహించము.\n\n" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-11-25T23:51:34+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "te" + } + }, + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/th/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/th/lessons/2-Symbolic/MSConceptGraph.ipynb index b100b764..19eae8d9 100644 --- a/translations/th/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/th/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - API นี้ไม่มีให้บริการอีกต่อไป แต่โปรดดูที่โน้ตบุ๊กเพื่อทำความเข้าใจแนวคิด\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) เป็นระบบการจัดหมวดหมู่ขนาดใหญ่ของคำศัพท์ที่ได้มาจากอินเทอร์เน็ต โดยมีความสัมพันธ์แบบ `is-a` ระหว่างแนวคิดต่างๆ\n", - "\n", - "Context Graph มีให้ใช้งานในสองรูปแบบ:\n", - " * ไฟล์ข้อความขนาดใหญ่สำหรับดาวน์โหลด\n", - " * REST API\n", - "\n", - "สถิติ:\n", - " * แนวคิดที่ไม่ซ้ำกัน 5,401,933 รายการ\n", - " * อินสแตนซ์ที่ไม่ซ้ำกัน 12,551,613 รายการ\n", - " * ความสัมพันธ์แบบ `is-a` 87,603,947 รายการ\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## การใช้งาน Web Service\n", - "\n", - "Web service มีการเรียกใช้งานหลากหลายรูปแบบเพื่อประมาณความน่าจะเป็นของแนวคิดที่อยู่ในกลุ่มต่าง ๆ สามารถดูข้อมูลเพิ่มเติมได้ [ที่นี่](https://concept.research.microsoft.com/Home/Api) \n", - "ตัวอย่าง URL สำหรับการเรียกใช้งาน: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - API นี้ไม่มีให้บริการอีกต่อไป แต่โปรดดูที่โน้ตบุ๊กเพื่อทำความเข้าใจแนวคิด\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) เป็นระบบการจัดหมวดหมู่ขนาดใหญ่ของคำศัพท์ที่ได้มาจากอินเทอร์เน็ต โดยมีความสัมพันธ์แบบ `is-a` ระหว่างแนวคิดต่างๆ\n", + "\n", + "Context Graph มีให้ใช้งานในสองรูปแบบ:\n", + " * ไฟล์ข้อความขนาดใหญ่สำหรับดาวน์โหลด\n", + " * REST API\n", + "\n", + "สถิติ:\n", + " * แนวคิดที่ไม่ซ้ำกัน 5,401,933 รายการ\n", + " * อินสแตนซ์ที่ไม่ซ้ำกัน 12,551,613 รายการ\n", + " * ความสัมพันธ์แบบ `is-a` 87,603,947 รายการ\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "ลองจัดหมวดหมู่หัวข้อข่าวโดยใช้แนวคิดหลัก ในการดึงหัวข้อข่าว เราจะใช้บริการ [NewsApi.org](http://newsapi.org) คุณจำเป็นต้องมี API key ของคุณเองเพื่อใช้บริการนี้ - ไปที่เว็บไซต์และลงทะเบียนแผนสำหรับนักพัฒนาฟรี\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "ก่อนอื่น เราต้องการที่จะสามารถดึงคำนามจากหัวข้อข่าว เราจะใช้ไลบรารี `TextBlob` เพื่อทำสิ่งนี้ ซึ่งช่วยทำให้งาน NLP ทั่วไปอย่างนี้ง่ายขึ้นมาก\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## การใช้งาน Web Service\n", + "\n", + "Web service มีการเรียกใช้งานหลากหลายรูปแบบเพื่อประมาณความน่าจะเป็นของแนวคิดที่อยู่ในกลุ่มต่าง ๆ สามารถดูข้อมูลเพิ่มเติมได้ [ที่นี่](https://concept.research.microsoft.com/Home/Api) \n", + "ตัวอย่าง URL สำหรับการเรียกใช้งาน: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "เราสามารถเห็นได้ว่า คำนามไม่ได้ให้กลุ่มหัวข้อที่ใหญ่แก่เรา ลองแทนที่คำนามด้วยคำที่มีความหมายทั่วไปมากขึ้นซึ่งได้มาจากกราฟแนวคิด สิ่งนี้จะใช้เวลาสักครู่ เพราะเรากำลังทำการเรียก REST สำหรับแต่ละวลีคำนาม\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "ลองจัดหมวดหมู่หัวข้อข่าวโดยใช้แนวคิดหลัก ในการดึงหัวข้อข่าว เราจะใช้บริการ [NewsApi.org](http://newsapi.org) คุณจำเป็นต้องมี API key ของคุณเองเพื่อใช้บริการนี้ - ไปที่เว็บไซต์และลงทะเบียนแผนสำหรับนักพัฒนาฟรี\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "ก่อนอื่น เราต้องการที่จะสามารถดึงคำนามจากหัวข้อข่าว เราจะใช้ไลบรารี `TextBlob` เพื่อทำสิ่งนี้ ซึ่งช่วยทำให้งาน NLP ทั่วไปอย่างนี้ง่ายขึ้นมาก\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "เราสามารถเห็นได้ว่า คำนามไม่ได้ให้กลุ่มหัวข้อที่ใหญ่แก่เรา ลองแทนที่คำนามด้วยคำที่มีความหมายทั่วไปมากขึ้นซึ่งได้มาจากกราฟแนวคิด สิ่งนี้จะใช้เวลาสักครู่ เพราะเรากำลังทำการเรียก REST สำหรับแต่ละวลีคำนาม\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**ข้อจำกัดความรับผิดชอบ**: \nเอกสารนี้ได้รับการแปลโดยใช้บริการแปลภาษา AI [Co-op Translator](https://github.com/Azure/co-op-translator) แม้ว่าเราจะพยายามให้การแปลมีความถูกต้อง แต่โปรดทราบว่าการแปลอัตโนมัติอาจมีข้อผิดพลาดหรือความไม่ถูกต้อง เอกสารต้นฉบับในภาษาดั้งเดิมควรถือเป็นแหล่งข้อมูลที่เชื่อถือได้ สำหรับข้อมูลที่สำคัญ ขอแนะนำให้ใช้บริการแปลภาษามืออาชีพ เราไม่รับผิดชอบต่อความเข้าใจผิดหรือการตีความผิดที่เกิดจากการใช้การแปลนี้\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**ข้อจำกัดความรับผิดชอบ**: \nเอกสารนี้ได้รับการแปลโดยใช้บริการแปลภาษา AI [Co-op Translator](https://github.com/Azure/co-op-translator) แม้ว่าเราจะพยายามให้การแปลมีความถูกต้อง แต่โปรดทราบว่าการแปลอัตโนมัติอาจมีข้อผิดพลาดหรือความไม่ถูกต้อง เอกสารต้นฉบับในภาษาดั้งเดิมควรถือเป็นแหล่งข้อมูลที่เชื่อถือได้ สำหรับข้อมูลที่สำคัญ ขอแนะนำให้ใช้บริการแปลภาษามืออาชีพ เราไม่รับผิดชอบต่อความเข้าใจผิดหรือการตีความผิดที่เกิดจากการใช้การแปลนี้\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:55:10+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "th" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:55:10+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "th" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/tl/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/tl/lessons/2-Symbolic/MSConceptGraph.ipynb index f0ca3027..599d367c 100644 --- a/translations/tl/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/tl/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - ANG API NA ITO AY HINDI NA AVAILABLE PERO PAKITINGNAN ANG NOTEBOOK PARA MAUNAWAAN ANG KONSEPTO\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) ay isang malaking taxonomy ng mga termino na nakuha mula sa internet, na may mga `is-a` na relasyon sa pagitan ng mga konsepto.\n", - "\n", - "Ang Context Graph ay available sa dalawang anyo:\n", - " * Malaking text file para sa pag-download\n", - " * REST API\n", - "\n", - "Mga Estadistika:\n", - " * 5401933 natatanging konsepto,\n", - " * 12551613 natatanging instance\n", - " * 87603947 `is-a` na relasyon\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Paggamit ng Web Service\n", - "\n", - "Ang web service ay nag-aalok ng iba't ibang tawag upang tantyahin ang posibilidad ng isang konsepto na kabilang sa iba't ibang grupo. Mas maraming impormasyon ang makukuha [dito](https://concept.research.microsoft.com/Home/Api). \n", - "Narito ang sample na URL para sa tawag: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - ANG API NA ITO AY HINDI NA AVAILABLE PERO PAKITINGNAN ANG NOTEBOOK PARA MAUNAWAAN ANG KONSEPTO\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) ay isang malaking taxonomy ng mga termino na nakuha mula sa internet, na may mga `is-a` na relasyon sa pagitan ng mga konsepto.\n", + "\n", + "Ang Context Graph ay available sa dalawang anyo:\n", + " * Malaking text file para sa pag-download\n", + " * REST API\n", + "\n", + "Mga Estadistika:\n", + " * 5401933 natatanging konsepto,\n", + " * 12551613 natatanging instance\n", + " * 87603947 `is-a` na relasyon\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Subukan nating i-kategorya ang mga pamagat ng balita gamit ang mga pangunahing konsepto. Para makuha ang mga pamagat ng balita, gagamitin natin ang serbisyo ng [NewsApi.org](http://newsapi.org). Kailangan mong kumuha ng sarili mong API key upang magamit ang serbisyo - pumunta sa web site at magrehistro para sa libreng developer plan.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Una sa lahat, nais naming makuha ang mga pangngalan mula sa mga pamagat ng balita. Gagamitin namin ang `TextBlob` na library upang gawin ito, na nagpapadali sa maraming karaniwang gawain sa NLP tulad nito.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Paggamit ng Web Service\n", + "\n", + "Ang web service ay nag-aalok ng iba't ibang tawag upang tantyahin ang posibilidad ng isang konsepto na kabilang sa iba't ibang grupo. Mas maraming impormasyon ang makukuha [dito](https://concept.research.microsoft.com/Home/Api). \n", + "Narito ang sample na URL para sa tawag: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Makikita natin na ang mga pangngalan ay hindi nagbibigay sa atin ng malalaking pangkat na may tema. Palitan natin ang mga pangngalan ng mas pangkalahatang mga termino na nakuha mula sa concept graph. Ito ay magtatagal ng kaunti, dahil gumagawa tayo ng REST call para sa bawat parirala ng pangngalan.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Subukan nating i-kategorya ang mga pamagat ng balita gamit ang mga pangunahing konsepto. Para makuha ang mga pamagat ng balita, gagamitin natin ang serbisyo ng [NewsApi.org](http://newsapi.org). Kailangan mong kumuha ng sarili mong API key upang magamit ang serbisyo - pumunta sa web site at magrehistro para sa libreng developer plan.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Una sa lahat, nais naming makuha ang mga pangngalan mula sa mga pamagat ng balita. Gagamitin namin ang `TextBlob` na library upang gawin ito, na nagpapadali sa maraming karaniwang gawain sa NLP tulad nito.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Makikita natin na ang mga pangngalan ay hindi nagbibigay sa atin ng malalaking pangkat na may tema. Palitan natin ang mga pangngalan ng mas pangkalahatang mga termino na nakuha mula sa concept graph. Ito ay magtatagal ng kaunti, dahil gumagawa tayo ng REST call para sa bawat parirala ng pangngalan.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**Paunawa**: \nAng dokumentong ito ay isinalin gamit ang AI translation service [Co-op Translator](https://github.com/Azure/co-op-translator). Bagamat sinisikap naming maging tumpak, mangyaring tandaan na ang mga awtomatikong pagsasalin ay maaaring maglaman ng mga pagkakamali o hindi pagkakatugma. Ang orihinal na dokumento sa kanyang katutubong wika ang dapat ituring na opisyal na sanggunian. Para sa mahalagang impormasyon, inirerekomenda ang propesyonal na pagsasalin ng tao. Hindi kami mananagot sa anumang hindi pagkakaunawaan o maling interpretasyon na dulot ng paggamit ng pagsasaling ito.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**Paunawa**: \nAng dokumentong ito ay isinalin gamit ang AI translation service [Co-op Translator](https://github.com/Azure/co-op-translator). Bagamat sinisikap naming maging tumpak, mangyaring tandaan na ang mga awtomatikong pagsasalin ay maaaring maglaman ng mga pagkakamali o hindi pagkakatugma. Ang orihinal na dokumento sa kanyang katutubong wika ang dapat ituring na opisyal na sanggunian. Para sa mahalagang impormasyon, inirerekomenda ang propesyonal na pagsasalin ng tao. Hindi kami mananagot sa anumang hindi pagkakaunawaan o maling interpretasyon na dulot ng paggamit ng pagsasaling ito.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:57:23+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "tl" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:57:23+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "tl" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/tr/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/tr/lessons/2-Symbolic/MSConceptGraph.ipynb index 55644cd5..ea9c4729 100644 --- a/translations/tr/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/tr/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,547 +1,538 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - BU API ARTIK KULLANILAMAZ AMA LÜTFEN KAVRAMI ANLAMAK İÇİN NOT DEFTERİNE BAKIN\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/), internetten çıkarılmış terimlerin büyük bir taksonomisidir ve kavramlar arasında `is-a` ilişkileri içerir.\n", - "\n", - "Context Graph iki şekilde sunulmaktadır:\n", - " * İndirme için büyük bir metin dosyası\n", - " * REST API\n", - "\n", - "İstatistikler:\n", - " * 5401933 benzersiz kavram,\n", - " * 12551613 benzersiz örnek\n", - " * 87603947 `is-a` ilişkisi\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Web Servisini Kullanma\n", - "\n", - "Web servisi, bir kavramın farklı gruplara ait olma olasılığını tahmin etmek için çeşitli çağrılar sunar. Daha fazla bilgi [burada](https://concept.research.microsoft.com/Home/Api) mevcuttur. İşte çağrı yapmak için örnek bir URL: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - BU API ARTIK KULLANILAMAZ AMA LÜTFEN KAVRAMI ANLAMAK İÇİN NOT DEFTERİNE BAKIN\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/), internetten çıkarılmış terimlerin büyük bir taksonomisidir ve kavramlar arasında `is-a` ilişkileri içerir.\n", + "\n", + "Context Graph iki şekilde sunulmaktadır:\n", + " * İndirme için büyük bir metin dosyası\n", + " * REST API\n", + "\n", + "İstatistikler:\n", + " * 5401933 benzersiz kavram,\n", + " * 12551613 benzersiz örnek\n", + " * 87603947 `is-a` ilişkisi\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Haber başlıklarını ana kavramlar kullanarak kategorize etmeyi deneyelim. Haber başlıklarını almak için [NewsApi.org](http://newsapi.org) hizmetini kullanacağız. Hizmeti kullanabilmek için kendi API anahtarınızı edinmeniz gerekiyor - web sitesine gidin ve ücretsiz geliştirici planına kaydolun.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Öncelikle, haber başlıklarından isimleri çıkarabilmek istiyoruz. Bunu yapmak için, bu tür tipik NLP görevlerini oldukça basitleştiren `TextBlob` kütüphanesini kullanacağız.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Web Servisini Kullanma\n", + "\n", + "Web servisi, bir kavramın farklı gruplara ait olma olasılığını tahmin etmek için çeşitli çağrılar sunar. Daha fazla bilgi [burada](https://concept.research.microsoft.com/Home/Api) mevcuttur. İşte çağrı yapmak için örnek bir URL: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "İsimlerin bize büyük tematik gruplar vermediğini görebiliyoruz. İsimleri, kavram grafiğinden elde edilen daha genel terimlerle değiştirelim. Bu biraz zaman alacak, çünkü her isim tamlaması için REST çağrısı yapıyoruz.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Haber başlıklarını ana kavramlar kullanarak kategorize etmeyi deneyelim. Haber başlıklarını almak için [NewsApi.org](http://newsapi.org) hizmetini kullanacağız. Hizmeti kullanabilmek için kendi API anahtarınızı edinmeniz gerekiyor - web sitesine gidin ve ücretsiz geliştirici planına kaydolun.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Öncelikle, haber başlıklarından isimleri çıkarabilmek istiyoruz. Bunu yapmak için, bu tür tipik NLP görevlerini oldukça basitleştiren `TextBlob` kütüphanesini kullanacağız.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "İsimlerin bize büyük tematik gruplar vermediğini görebiliyoruz. İsimleri, kavram grafiğinden elde edilen daha genel terimlerle değiştirelim. Bu biraz zaman alacak, çünkü her isim tamlaması için REST çağrısı yapıyoruz.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**Feragatname**: \nBu belge, AI çeviri hizmeti [Co-op Translator](https://github.com/Azure/co-op-translator) kullanılarak çevrilmiştir. Doğruluk için çaba göstersek de, otomatik çevirilerin hata veya yanlışlıklar içerebileceğini lütfen unutmayın. Belgenin orijinal dili, yetkili kaynak olarak kabul edilmelidir. Kritik bilgiler için profesyonel bir insan çevirisi önerilir. Bu çevirinin kullanımından kaynaklanan yanlış anlamalar veya yanlış yorumlamalardan sorumlu değiliz.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**Feragatname**: \nBu belge, AI çeviri hizmeti [Co-op Translator](https://github.com/Azure/co-op-translator) kullanılarak çevrilmiştir. Doğruluk için çaba göstersek de, otomatik çevirilerin hata veya yanlışlıklar içerebileceğini lütfen unutmayın. Belgenin orijinal dili, yetkili kaynak olarak kabul edilmelidir. Kritik bilgiler için profesyonel bir insan çevirisi önerilir. Bu çevirinin kullanımından kaynaklanan yanlış anlamalar veya yanlış yorumlamalardan sorumlu değiliz.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:54:41+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "tr" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:54:41+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "tr" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/uk/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/uk/lessons/2-Symbolic/MSConceptGraph.ipynb index 0d86c0f1..94d37fa1 100644 --- a/translations/uk/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/uk/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - Цей API БІЛЬШЕ НЕ ДОСТУПНИЙ, АЛЕ ПРОСИМО ПЕРЕГЛЯНУТИ НОУТБУК ДЛЯ РОЗУМІННЯ КОНЦЕПЦІЇ\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) — це велика таксономія термінів, отриманих з інтернету, з відношеннями типу `is-a` між концептами.\n", - "\n", - "Context Graph доступний у двох формах:\n", - " * Великий текстовий файл для завантаження\n", - " * REST API\n", - "\n", - "Статистика:\n", - " * 5401933 унікальних концептів,\n", - " * 12551613 унікальних екземплярів\n", - " * 87603947 відношень типу `is-a`\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Використання веб-сервісу\n", - "\n", - "Веб-сервіс пропонує різні виклики для оцінки ймовірності належності концепту до різних груп. Більше інформації доступно [тут](https://concept.research.microsoft.com/Home/Api). \n", - "Ось приклад URL для виклику: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - Цей API БІЛЬШЕ НЕ ДОСТУПНИЙ, АЛЕ ПРОСИМО ПЕРЕГЛЯНУТИ НОУТБУК ДЛЯ РОЗУМІННЯ КОНЦЕПЦІЇ\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) — це велика таксономія термінів, отриманих з інтернету, з відношеннями типу `is-a` між концептами.\n", + "\n", + "Context Graph доступний у двох формах:\n", + " * Великий текстовий файл для завантаження\n", + " * REST API\n", + "\n", + "Статистика:\n", + " * 5401933 унікальних концептів,\n", + " * 12551613 унікальних екземплярів\n", + " * 87603947 відношень типу `is-a`\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Давайте спробуємо категоризувати заголовки новин за допомогою основних концепцій. Щоб отримати заголовки новин, ми будемо використовувати сервіс [NewsApi.org](http://newsapi.org). Вам потрібно отримати власний API-ключ, щоб використовувати цей сервіс - перейдіть на веб-сайт і зареєструйтесь для безкоштовного плану розробника.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Перш за все, ми хочемо мати можливість витягувати іменники з заголовків новин. Для цього ми будемо використовувати бібліотеку `TextBlob`, яка спрощує багато типових завдань обробки природної мови, таких як це.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Використання веб-сервісу\n", + "\n", + "Веб-сервіс пропонує різні виклики для оцінки ймовірності належності концепту до різних груп. Більше інформації доступно [тут](https://concept.research.microsoft.com/Home/Api). \n", + "Ось приклад URL для виклику: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Ми можемо побачити, що іменники не дають нам великих тематичних груп. Давайте замінимо іменники більш загальними термінами, отриманими з графа концепцій. Це займе деякий час, оскільки ми виконуємо REST-запит для кожної іменникової фрази.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Давайте спробуємо категоризувати заголовки новин за допомогою основних концепцій. Щоб отримати заголовки новин, ми будемо використовувати сервіс [NewsApi.org](http://newsapi.org). Вам потрібно отримати власний API-ключ, щоб використовувати цей сервіс - перейдіть на веб-сайт і зареєструйтесь для безкоштовного плану розробника.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Перш за все, ми хочемо мати можливість витягувати іменники з заголовків новин. Для цього ми будемо використовувати бібліотеку `TextBlob`, яка спрощує багато типових завдань обробки природної мови, таких як це.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Ми можемо побачити, що іменники не дають нам великих тематичних груп. Давайте замінимо іменники більш загальними термінами, отриманими з графа концепцій. Це займе деякий час, оскільки ми виконуємо REST-запит для кожної іменникової фрази.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**Відмова від відповідальності**: \nЦей документ був перекладений за допомогою сервісу автоматичного перекладу [Co-op Translator](https://github.com/Azure/co-op-translator). Хоча ми прагнемо до точності, будь ласка, майте на увазі, що автоматичні переклади можуть містити помилки або неточності. Оригінальний документ на його рідній мові слід вважати авторитетним джерелом. Для критичної інформації рекомендується професійний людський переклад. Ми не несемо відповідальності за будь-які непорозуміння або неправильні тлумачення, що виникають внаслідок використання цього перекладу.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**Відмова від відповідальності**: \nЦей документ був перекладений за допомогою сервісу автоматичного перекладу [Co-op Translator](https://github.com/Azure/co-op-translator). Хоча ми прагнемо до точності, будь ласка, майте на увазі, що автоматичні переклади можуть містити помилки або неточності. Оригінальний документ на його рідній мові слід вважати авторитетним джерелом. Для критичної інформації рекомендується професійний людський переклад. Ми не несемо відповідальності за будь-які непорозуміння або неправильні тлумачення, що виникають внаслідок використання цього перекладу.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T19:00:04+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "uk" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T19:00:04+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "uk" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/ur/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/ur/lessons/2-Symbolic/MSConceptGraph.ipynb index c7b89dfd..100b0298 100644 --- a/translations/ur/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/ur/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## مائیکروسافٹ کانسیپٹ گراف - یہ API اب دستیاب نہیں ہے لیکن براہ کرم نوٹ بک دیکھیں تاکہ تصور کو سمجھا جا سکے\n", - "\n", - "[مائیکروسافٹ کانسیپٹ گراف](https://concept.research.microsoft.com/) انٹرنیٹ سے حاصل کردہ اصطلاحات کا ایک بڑا ٹیکسونومی ہے، جس میں تصورات کے درمیان `is-a` تعلقات موجود ہیں۔\n", - "\n", - "کانٹیکسٹ گراف دو شکلوں میں دستیاب ہے:\n", - " * ڈاؤنلوڈ کے لیے بڑا ٹیکسٹ فائل\n", - " * REST API\n", - "\n", - "اعداد و شمار:\n", - " * 5401933 منفرد تصورات\n", - " * 12551613 منفرد مثالیں\n", - " * 87603947 `is-a` تعلقات\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## ویب سروس کا استعمال\n", - "\n", - "ویب سروس مختلف کالز فراہم کرتی ہے تاکہ کسی تصور کے مختلف گروہوں سے تعلق رکھنے کے امکانات کا اندازہ لگایا جا سکے۔ مزید معلومات [یہاں](https://concept.research.microsoft.com/Home/Api) دستیاب ہیں۔ \n", - "یہاں ایک نمونہ URL ہے کال کرنے کے لیے: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## مائیکروسافٹ کانسیپٹ گراف - یہ API اب دستیاب نہیں ہے لیکن براہ کرم نوٹ بک دیکھیں تاکہ تصور کو سمجھا جا سکے\n", + "\n", + "[مائیکروسافٹ کانسیپٹ گراف](https://concept.research.microsoft.com/) انٹرنیٹ سے حاصل کردہ اصطلاحات کا ایک بڑا ٹیکسونومی ہے، جس میں تصورات کے درمیان `is-a` تعلقات موجود ہیں۔\n", + "\n", + "کانٹیکسٹ گراف دو شکلوں میں دستیاب ہے:\n", + " * ڈاؤنلوڈ کے لیے بڑا ٹیکسٹ فائل\n", + " * REST API\n", + "\n", + "اعداد و شمار:\n", + " * 5401933 منفرد تصورات\n", + " * 12551613 منفرد مثالیں\n", + " * 87603947 `is-a` تعلقات\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "چلو خبروں کے عنوانات کو والدین کے تصورات کے تحت درجہ بندی کرنے کی کوشش کرتے ہیں۔ خبروں کے عنوانات حاصل کرنے کے لیے، ہم [NewsApi.org](http://newsapi.org) سروس استعمال کریں گے۔ اس سروس کو استعمال کرنے کے لیے آپ کو اپنا API کلید حاصل کرنا ہوگا - ویب سائٹ پر جائیں اور مفت ڈویلپر پلان کے لیے رجسٹر کریں۔\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "سب سے پہلے، ہم خبریں کے عنوانات سے اسم نکالنے کے قابل ہونا چاہتے ہیں۔ ہم اس کے لئے `TextBlob` لائبریری استعمال کریں گے، جو اس طرح کے عام NLP کاموں کو بہت آسان بناتی ہے۔\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## ویب سروس کا استعمال\n", + "\n", + "ویب سروس مختلف کالز فراہم کرتی ہے تاکہ کسی تصور کے مختلف گروہوں سے تعلق رکھنے کے امکانات کا اندازہ لگایا جا سکے۔ مزید معلومات [یہاں](https://concept.research.microsoft.com/Home/Api) دستیاب ہیں۔ \n", + "یہاں ایک نمونہ URL ہے کال کرنے کے لیے: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "ہم دیکھ سکتے ہیں کہ اسم ہمیں بڑے موضوعاتی گروپ فراہم نہیں کرتے۔ آئیے اسم کو زیادہ عمومی اصطلاحات سے بدل دیں جو تصوراتی گراف سے حاصل کی گئی ہیں۔ یہ کچھ وقت لے گا کیونکہ ہم ہر اسم جملے کے لیے REST کال کر رہے ہیں۔\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "چلو خبروں کے عنوانات کو والدین کے تصورات کے تحت درجہ بندی کرنے کی کوشش کرتے ہیں۔ خبروں کے عنوانات حاصل کرنے کے لیے، ہم [NewsApi.org](http://newsapi.org) سروس استعمال کریں گے۔ اس سروس کو استعمال کرنے کے لیے آپ کو اپنا API کلید حاصل کرنا ہوگا - ویب سائٹ پر جائیں اور مفت ڈویلپر پلان کے لیے رجسٹر کریں۔\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "سب سے پہلے، ہم خبریں کے عنوانات سے اسم نکالنے کے قابل ہونا چاہتے ہیں۔ ہم اس کے لئے `TextBlob` لائبریری استعمال کریں گے، جو اس طرح کے عام NLP کاموں کو بہت آسان بناتی ہے۔\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "ہم دیکھ سکتے ہیں کہ اسم ہمیں بڑے موضوعاتی گروپ فراہم نہیں کرتے۔ آئیے اسم کو زیادہ عمومی اصطلاحات سے بدل دیں جو تصوراتی گراف سے حاصل کی گئی ہیں۔ یہ کچھ وقت لے گا کیونکہ ہم ہر اسم جملے کے لیے REST کال کر رہے ہیں۔\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**ڈسکلیمر**: \nیہ دستاویز AI ترجمہ سروس [Co-op Translator](https://github.com/Azure/co-op-translator) کا استعمال کرتے ہوئے ترجمہ کی گئی ہے۔ ہم درستگی کے لیے کوشش کرتے ہیں، لیکن براہ کرم آگاہ رہیں کہ خودکار ترجمے میں غلطیاں یا غیر درستیاں ہو سکتی ہیں۔ اصل دستاویز کو اس کی اصل زبان میں مستند ذریعہ سمجھا جانا چاہیے۔ اہم معلومات کے لیے، پیشہ ور انسانی ترجمہ کی سفارش کی جاتی ہے۔ ہم اس ترجمے کے استعمال سے پیدا ہونے والی کسی بھی غلط فہمی یا غلط تشریح کے ذمہ دار نہیں ہیں۔\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**ڈسکلیمر**: \nیہ دستاویز AI ترجمہ سروس [Co-op Translator](https://github.com/Azure/co-op-translator) کا استعمال کرتے ہوئے ترجمہ کی گئی ہے۔ ہم درستگی کے لیے کوشش کرتے ہیں، لیکن براہ کرم آگاہ رہیں کہ خودکار ترجمے میں غلطیاں یا غیر درستیاں ہو سکتی ہیں۔ اصل دستاویز کو اس کی اصل زبان میں مستند ذریعہ سمجھا جانا چاہیے۔ اہم معلومات کے لیے، پیشہ ور انسانی ترجمہ کی سفارش کی جاتی ہے۔ ہم اس ترجمے کے استعمال سے پیدا ہونے والی کسی بھی غلط فہمی یا غلط تشریح کے ذمہ دار نہیں ہیں۔\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:51:04+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "ur" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:51:04+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "ur" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/vi/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/vi/lessons/2-Symbolic/MSConceptGraph.ipynb index 38dd23e6..6b34237e 100644 --- a/translations/vi/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/vi/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - API NÀY KHÔNG CÒN HOẠT ĐỘNG NHƯNG VUI LÒNG XEM NOTEBOOK ĐỂ HIỂU RÕ KHÁI NIỆM\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) là một hệ thống phân loại lớn các thuật ngữ được khai thác từ internet, với các mối quan hệ `is-a` giữa các khái niệm.\n", - "\n", - "Context Graph có sẵn dưới hai hình thức:\n", - " * Tệp văn bản lớn để tải xuống\n", - " * REST API\n", - "\n", - "Thống kê:\n", - " * 5401933 khái niệm độc nhất\n", - " * 12551613 thực thể độc nhất\n", - " * 87603947 mối quan hệ `is-a`\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Sử dụng Dịch vụ Web\n", - "\n", - "Dịch vụ web cung cấp các lệnh gọi khác nhau để ước tính xác suất một khái niệm thuộc về các nhóm khác nhau. Thông tin chi tiết có thể xem [tại đây](https://concept.research.microsoft.com/Home/Api). \n", - "Dưới đây là URL mẫu để thực hiện lệnh gọi: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - API NÀY KHÔNG CÒN HOẠT ĐỘNG NHƯNG VUI LÒNG XEM NOTEBOOK ĐỂ HIỂU RÕ KHÁI NIỆM\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) là một hệ thống phân loại lớn các thuật ngữ được khai thác từ internet, với các mối quan hệ `is-a` giữa các khái niệm.\n", + "\n", + "Context Graph có sẵn dưới hai hình thức:\n", + " * Tệp văn bản lớn để tải xuống\n", + " * REST API\n", + "\n", + "Thống kê:\n", + " * 5401933 khái niệm độc nhất\n", + " * 12551613 thực thể độc nhất\n", + " * 87603947 mối quan hệ `is-a`\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Hãy thử phân loại các tiêu đề tin tức bằng cách sử dụng các khái niệm cha. Để lấy các tiêu đề tin tức, chúng ta sẽ sử dụng dịch vụ [NewsApi.org](http://newsapi.org). Bạn cần tự lấy khóa API của mình để sử dụng dịch vụ - hãy truy cập trang web và đăng ký gói nhà phát triển miễn phí.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Trước hết, chúng tôi muốn có thể trích xuất danh từ từ tiêu đề tin tức. Chúng tôi sẽ sử dụng thư viện `TextBlob` để thực hiện điều này, thư viện này đơn giản hóa rất nhiều tác vụ NLP điển hình như thế này.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Sử dụng Dịch vụ Web\n", + "\n", + "Dịch vụ web cung cấp các lệnh gọi khác nhau để ước tính xác suất một khái niệm thuộc về các nhóm khác nhau. Thông tin chi tiết có thể xem [tại đây](https://concept.research.microsoft.com/Home/Api). \n", + "Dưới đây là URL mẫu để thực hiện lệnh gọi: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Chúng ta có thể thấy rằng danh từ không cung cấp cho chúng ta các nhóm chủ đề lớn. Hãy thay thế danh từ bằng các thuật ngữ tổng quát hơn thu được từ đồ thị khái niệm. Điều này sẽ mất một chút thời gian, vì chúng ta đang thực hiện cuộc gọi REST cho mỗi cụm danh từ.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Hãy thử phân loại các tiêu đề tin tức bằng cách sử dụng các khái niệm cha. Để lấy các tiêu đề tin tức, chúng ta sẽ sử dụng dịch vụ [NewsApi.org](http://newsapi.org). Bạn cần tự lấy khóa API của mình để sử dụng dịch vụ - hãy truy cập trang web và đăng ký gói nhà phát triển miễn phí.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Trước hết, chúng tôi muốn có thể trích xuất danh từ từ tiêu đề tin tức. Chúng tôi sẽ sử dụng thư viện `TextBlob` để thực hiện điều này, thư viện này đơn giản hóa rất nhiều tác vụ NLP điển hình như thế này.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Chúng ta có thể thấy rằng danh từ không cung cấp cho chúng ta các nhóm chủ đề lớn. Hãy thay thế danh từ bằng các thuật ngữ tổng quát hơn thu được từ đồ thị khái niệm. Điều này sẽ mất một chút thời gian, vì chúng ta đang thực hiện cuộc gọi REST cho mỗi cụm danh từ.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**Tuyên bố miễn trừ trách nhiệm**: \nTài liệu này đã được dịch bằng dịch vụ dịch thuật AI [Co-op Translator](https://github.com/Azure/co-op-translator). Mặc dù chúng tôi cố gắng đảm bảo độ chính xác, xin lưu ý rằng các bản dịch tự động có thể chứa lỗi hoặc không chính xác. Tài liệu gốc bằng ngôn ngữ bản địa nên được coi là nguồn thông tin chính thức. Đối với các thông tin quan trọng, khuyến nghị sử dụng dịch vụ dịch thuật chuyên nghiệp bởi con người. Chúng tôi không chịu trách nhiệm cho bất kỳ sự hiểu lầm hoặc diễn giải sai nào phát sinh từ việc sử dụng bản dịch này.\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**Tuyên bố miễn trừ trách nhiệm**: \nTài liệu này đã được dịch bằng dịch vụ dịch thuật AI [Co-op Translator](https://github.com/Azure/co-op-translator). Mặc dù chúng tôi cố gắng đảm bảo độ chính xác, xin lưu ý rằng các bản dịch tự động có thể chứa lỗi hoặc không chính xác. Tài liệu gốc bằng ngôn ngữ bản địa nên được coi là nguồn thông tin chính thức. Đối với các thông tin quan trọng, khuyến nghị sử dụng dịch vụ dịch thuật chuyên nghiệp bởi con người. Chúng tôi không chịu trách nhiệm cho bất kỳ sự hiểu lầm hoặc diễn giải sai nào phát sinh từ việc sử dụng bản dịch này.\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:56:44+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "vi" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:56:44+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "vi" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/zh-CN/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/zh-CN/lessons/2-Symbolic/MSConceptGraph.ipynb index 7367dc9b..59aa6770 100644 --- a/translations/zh-CN/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/zh-CN/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - 此 API 已不再可用，但请查看笔记本以了解相关概念\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) 是一个从互联网中挖掘出来的大型术语分类体系，包含概念之间的 `is-a` 关系。\n", - "\n", - "Context Graph 提供两种形式：\n", - " * 可下载的大型文本文件\n", - " * REST API\n", - "\n", - "统计数据：\n", - " * 5401933 个独特概念\n", - " * 12551613 个独特实例\n", - " * 87603947 个 `is-a` 关系\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 使用网络服务\n", - "\n", - "网络服务提供不同的调用，用于估算某个概念属于不同组的概率。更多信息请点击[这里](https://concept.research.microsoft.com/Home/Api)。 \n", - "以下是调用的示例 URL：`https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - 此 API 已不再可用，但请查看笔记本以了解相关概念\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) 是一个从互联网中挖掘出来的大型术语分类体系，包含概念之间的 `is-a` 关系。\n", + "\n", + "Context Graph 提供两种形式：\n", + " * 可下载的大型文本文件\n", + " * REST API\n", + "\n", + "统计数据：\n", + " * 5401933 个独特概念\n", + " * 12551613 个独特实例\n", + " * 87603947 个 `is-a` 关系\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "让我们尝试使用父概念对新闻标题进行分类。要获取新闻标题，我们将使用[NewsApi.org](http://newsapi.org)服务。您需要获得自己的API密钥才能使用该服务——请访问网站并注册免费的开发者计划。\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "首先，我们希望能够从新闻标题中提取名词。我们将使用`TextBlob`库来完成此任务，该库简化了许多类似的典型自然语言处理任务。\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 使用网络服务\n", + "\n", + "网络服务提供不同的调用，用于估算某个概念属于不同组的概率。更多信息请点击[这里](https://concept.research.microsoft.com/Home/Api)。 \n", + "以下是调用的示例 URL：`https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "我们可以看到名词并没有给我们大的主题组。让我们用从概念图中获得的更一般的术语替换名词。这将需要一些时间，因为我们正在为每个名词短语进行REST调用。\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "让我们尝试使用父概念对新闻标题进行分类。要获取新闻标题，我们将使用[NewsApi.org](http://newsapi.org)服务。您需要获得自己的API密钥才能使用该服务——请访问网站并注册免费的开发者计划。\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "首先，我们希望能够从新闻标题中提取名词。我们将使用`TextBlob`库来完成此任务，该库简化了许多类似的典型自然语言处理任务。\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "我们可以看到名词并没有给我们大的主题组。让我们用从概念图中获得的更一般的术语替换名词。这将需要一些时间，因为我们正在为每个名词短语进行REST调用。\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**免责声明**： \n本文档使用AI翻译服务 [Co-op Translator](https://github.com/Azure/co-op-translator) 进行翻译。尽管我们努力确保翻译的准确性，但请注意，自动翻译可能包含错误或不准确之处。原始语言的文档应被视为权威来源。对于关键信息，建议使用专业人工翻译。我们对因使用此翻译而产生的任何误解或误读不承担责任。\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**免责声明**： \n本文档使用AI翻译服务 [Co-op Translator](https://github.com/Azure/co-op-translator) 进行翻译。尽管我们努力确保翻译的准确性，但请注意，自动翻译可能包含错误或不准确之处。原始语言的文档应被视为权威来源。对于关键信息，建议使用专业人工翻译。我们对因使用此翻译而产生的任何误解或误读不承担责任。\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:51:16+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "zh" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:51:16+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "zh" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/zh-HK/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/zh-HK/lessons/2-Symbolic/MSConceptGraph.ipynb index d89cf1f6..490f4788 100644 --- a/translations/zh-HK/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/zh-HK/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - 此 API 已不再提供，但請查看筆記本以了解概念\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) 是一個從互聯網挖掘的大型術語分類，包含概念之間的 `is-a` 關係。\n", - "\n", - "Context Graph 提供以下兩種形式：\n", - " * 可下載的大型文本文件\n", - " * REST API\n", - "\n", - "統計數據：\n", - " * 5401933 個獨特概念\n", - " * 12551613 個獨特實例\n", - " * 87603947 個 `is-a` 關係\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 使用網絡服務\n", - "\n", - "網絡服務提供不同的調用，用於估算某個概念屬於不同群組的概率。更多資訊可在[這裡](https://concept.research.microsoft.com/Home/Api)找到。 \n", - "以下是調用的範例 URL：`https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - 此 API 已不再提供，但請查看筆記本以了解概念\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) 是一個從互聯網挖掘的大型術語分類，包含概念之間的 `is-a` 關係。\n", + "\n", + "Context Graph 提供以下兩種形式：\n", + " * 可下載的大型文本文件\n", + " * REST API\n", + "\n", + "統計數據：\n", + " * 5401933 個獨特概念\n", + " * 12551613 個獨特實例\n", + " * 87603947 個 `is-a` 關係\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "讓我們嘗試使用父概念來分類新聞標題。要獲取新聞標題，我們將使用 [NewsApi.org](http://newsapi.org) 服務。您需要獲取自己的 API 密鑰才能使用該服務——請訪問網站並註冊免費開發者計劃。\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "首先，我們希望能夠從新聞標題中提取名詞。我們將使用 `TextBlob` 庫來完成這項工作，該庫簡化了許多像這樣的典型自然語言處理任務。\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 使用網絡服務\n", + "\n", + "網絡服務提供不同的調用，用於估算某個概念屬於不同群組的概率。更多資訊可在[這裡](https://concept.research.microsoft.com/Home/Api)找到。 \n", + "以下是調用的範例 URL：`https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "我們可以看到名詞並未給我們提供大的主題群組。讓我們用從概念圖中獲得的更一般的術語來替代名詞。這將需要一些時間，因為我們正在為每個名詞短語進行 REST 調用。\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "讓我們嘗試使用父概念來分類新聞標題。要獲取新聞標題，我們將使用 [NewsApi.org](http://newsapi.org) 服務。您需要獲取自己的 API 密鑰才能使用該服務——請訪問網站並註冊免費開發者計劃。\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "首先，我們希望能夠從新聞標題中提取名詞。我們將使用 `TextBlob` 庫來完成這項工作，該庫簡化了許多像這樣的典型自然語言處理任務。\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "我們可以看到名詞並未給我們提供大的主題群組。讓我們用從概念圖中獲得的更一般的術語來替代名詞。這將需要一些時間，因為我們正在為每個名詞短語進行 REST 調用。\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**免責聲明**： \n本文件已使用人工智能翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。雖然我們致力於提供準確的翻譯，但請注意，自動翻譯可能包含錯誤或不準確之處。原始文件的母語版本應被視為權威來源。對於重要信息，建議使用專業人工翻譯。我們對因使用此翻譯而引起的任何誤解或錯誤解釋概不負責。\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**免責聲明**： \n本文件已使用人工智能翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。雖然我們致力於提供準確的翻譯，但請注意，自動翻譯可能包含錯誤或不準確之處。原始文件的母語版本應被視為權威來源。對於重要信息，建議使用專業人工翻譯。我們對因使用此翻譯而引起的任何誤解或錯誤解釋概不負責。\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:51:43+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "hk" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:51:43+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "hk" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/zh-MO/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/zh-MO/lessons/2-Symbolic/MSConceptGraph.ipynb index e32d1867..6e9429c3 100644 --- a/translations/zh-MO/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/zh-MO/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - 此 API 已不再可用，但請查看筆記本以了解概念\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) 是一個從互聯網挖掘的龐大術語分類，包含概念之間的 `is-a` 關係。\n", - "\n", - "Context Graph 提供以下兩種形式：\n", - " * 可下載的大型文本文件\n", - " * REST API\n", - "\n", - "統計數據：\n", - " * 5401933 個獨特概念\n", - " * 12551613 個獨特實例\n", - " * 87603947 個 `is-a` 關係\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 使用網路服務\n", - "\n", - "網路服務提供不同的呼叫方式，用於估算某個概念屬於不同群組的可能性。更多資訊請參考[這裡](https://concept.research.microsoft.com/Home/Api)。 \n", - "以下是範例 URL 呼叫：`https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - 此 API 已不再可用，但請查看筆記本以了解概念\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) 是一個從互聯網挖掘的龐大術語分類，包含概念之間的 `is-a` 關係。\n", + "\n", + "Context Graph 提供以下兩種形式：\n", + " * 可下載的大型文本文件\n", + " * REST API\n", + "\n", + "統計數據：\n", + " * 5401933 個獨特概念\n", + " * 12551613 個獨特實例\n", + " * 87603947 個 `is-a` 關係\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "讓我們嘗試使用父概念對新聞標題進行分類。要獲取新聞標題，我們將使用 [NewsApi.org](http://newsapi.org) 服務。您需要獲取自己的 API 金鑰才能使用該服務——請訪問網站並註冊免費的開發者計劃。\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "首先，我們希望能夠從新聞標題中提取名詞。我們將使用 `TextBlob` 庫來完成這項工作，該庫簡化了許多像這樣的典型自然語言處理任務。\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 使用網路服務\n", + "\n", + "網路服務提供不同的呼叫方式，用於估算某個概念屬於不同群組的可能性。更多資訊請參考[這裡](https://concept.research.microsoft.com/Home/Api)。 \n", + "以下是範例 URL 呼叫：`https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "我們可以看到名詞並未給我們大的主題群組。讓我們用從概念圖中獲得的更一般的術語來替代名詞。這將需要一些時間，因為我們正在為每個名詞短語進行 REST 調用。\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "讓我們嘗試使用父概念對新聞標題進行分類。要獲取新聞標題，我們將使用 [NewsApi.org](http://newsapi.org) 服務。您需要獲取自己的 API 金鑰才能使用該服務——請訪問網站並註冊免費的開發者計劃。\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "首先，我們希望能夠從新聞標題中提取名詞。我們將使用 `TextBlob` 庫來完成這項工作，該庫簡化了許多像這樣的典型自然語言處理任務。\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "我們可以看到名詞並未給我們大的主題群組。讓我們用從概念圖中獲得的更一般的術語來替代名詞。這將需要一些時間，因為我們正在為每個名詞短語進行 REST 調用。\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**免責聲明**： \n本文件已使用 AI 翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。儘管我們努力確保準確性，但請注意，自動翻譯可能包含錯誤或不準確之處。原始文件的母語版本應被視為權威來源。對於關鍵資訊，建議使用專業人工翻譯。我們對因使用此翻譯而產生的任何誤解或錯誤解釋不承擔責任。\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**免責聲明**： \n本文件已使用 AI 翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。儘管我們努力確保準確性，但請注意，自動翻譯可能包含錯誤或不準確之處。原始文件的母語版本應被視為權威來源。對於關鍵資訊，建議使用專業人工翻譯。我們對因使用此翻譯而產生的任何誤解或錯誤解釋不承擔責任。\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:51:30+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "mo" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:51:30+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "mo" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file diff --git a/translations/zh-TW/lessons/2-Symbolic/MSConceptGraph.ipynb b/translations/zh-TW/lessons/2-Symbolic/MSConceptGraph.ipynb index 52f91368..ca489bd6 100644 --- a/translations/zh-TW/lessons/2-Symbolic/MSConceptGraph.ipynb +++ b/translations/zh-TW/lessons/2-Symbolic/MSConceptGraph.ipynb @@ -1,548 +1,539 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "## Microsoft Concept Graph - 此 API 已不再提供，但請查看筆記本以了解概念\n", - "\n", - "[Microsoft Concept Graph](https://concept.research.microsoft.com/) 是一個從互聯網挖掘的龐大術語分類，包含概念之間的 `is-a` 關係。\n", - "\n", - "Context Graph 提供以下兩種形式：\n", - " * 可下載的大型文本檔案\n", - " * REST API\n", - "\n", - "統計數據：\n", - " * 5401933 個獨特概念\n", - " * 12551613 個獨特實例\n", - " * 87603947 個 `is-a` 關係\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 使用網路服務\n", - "\n", - "網路服務提供不同的呼叫方式，用於估算某個概念屬於不同群組的可能性。更多資訊請參考[這裡](https://concept.research.microsoft.com/Home/Api)。 \n", - "以下是範例 URL 呼叫：`https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "trusted": true - }, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "{'company': 0.6105356614382954,\n", - " 'vendor': 0.08858636677518003,\n", - " 'client': 0.048239124001183784,\n", - " 'firm': 0.045476965571668145,\n", - " 'large company': 0.043109401203511886,\n", - " 'organization': 0.043010752688172046,\n", - " 'corporation': 0.035908059583703265,\n", - " 'brand': 0.03383644076156654,\n", - " 'software company': 0.027522935779816515,\n", - " 'technology company': 0.023774292196902438}" + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Microsoft Concept Graph - 此 API 已不再提供，但請查看筆記本以了解概念\n", + "\n", + "[Microsoft Concept Graph](https://concept.research.microsoft.com/) 是一個從互聯網挖掘的龐大術語分類，包含概念之間的 `is-a` 關係。\n", + "\n", + "Context Graph 提供以下兩種形式：\n", + " * 可下載的大型文本檔案\n", + " * REST API\n", + "\n", + "統計數據：\n", + " * 5401933 個獨特概念\n", + " * 12551613 個獨特實例\n", + " * 87603947 個 `is-a` 關係\n" ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import urllib\n", - "import json\n", - "import ssl\n", - "\n", - "def http(x):\n", - " ssl._create_default_https_context = ssl._create_unverified_context\n", - " response = urllib.request.urlopen(x)\n", - " data = response.read()\n", - " return data.decode('utf-8')\n", - "\n", - "def query(x):\n", - " return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n", - "\n", - "query('microsoft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "讓我們嘗試使用父概念對新聞標題進行分類。要獲取新聞標題，我們將使用 [NewsApi.org](http://newsapi.org) 服務。您需要獲取自己的 API 金鑰才能使用該服務——請訪問網站並註冊免費開發者計劃。\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "newsapi_key = ''\n", - "def get_news(country='us'):\n", - " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", - " return res['articles']\n", - "\n", - "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", - " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", - " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", - " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", - " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", - " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", - " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", - " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", - " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", - " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", - " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", - " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", - " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", - " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", - " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", - " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", - " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", - " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", - " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", - " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", - " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", - " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", - " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", - " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", - " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", - " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", - " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", - " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", - " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", - " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", - " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", - " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", - " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", - " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", - " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", - " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", - " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "all_titles" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "首先，我們希望能夠從新聞標題中提取名詞。我們將使用 `TextBlob` 庫來完成這項工作，該庫簡化了許多像這樣的典型自然語言處理任務。\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", - "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", - "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", - "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", - "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", - "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", - "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", - "Finished.\n" - ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "[nltk_data] Downloading package brown to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package brown is already up-to-date!\n", - "[nltk_data] Downloading package punkt to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package punkt is already up-to-date!\n", - "[nltk_data] Downloading package wordnet to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package wordnet is already up-to-date!\n", - "[nltk_data] Downloading package averaged_perceptron_tagger to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", - "[nltk_data] date!\n", - "[nltk_data] Downloading package conll2000 to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package conll2000 is already up-to-date!\n", - "[nltk_data] Downloading package movie_reviews to\n", - "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", - "[nltk_data] Package movie_reviews is already up-to-date!\n" - ] - } - ], - "source": [ - "import sys\n", - "!{sys.executable} -m pip install textblob\n", - "!{sys.executable} -m textblob.download_corpora\n", - "from textblob import TextBlob" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": { - "trusted": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{'covid-19 live updates': 1,\n", - " 'vaccines': 1,\n", - " 'boosters': 1,\n", - " 'york': 4,\n", - " 'ukrainians flee mariupol': 1,\n", - " 'forces push': 1,\n", - " 'port city': 1,\n", - " 'wall street journal': 3,\n", - " 'bond yields': 1,\n", - " 'futures rise': 1,\n", - " 'powell says fed': 1,\n", - " 'ready': 1,\n", - " 'be': 1,\n", - " 'aggressive': 1,\n", - " 'putin': 3,\n", - " 'alexei navalny': 2,\n", - " 'russian': 2,\n", - " 'supreme court nominee': 1,\n", - " 'ketanji brown jackson': 1,\n", - " \"confirmation hearing 's\": 1,\n", - " 'cnn': 1,\n", - " 'swedish': 1,\n", - " 'high school': 1,\n", - " 'abc': 1,\n", - " 'clues': 1,\n", - " 'covid-19': 1,\n", - " '’ s': 2,\n", - " 'moves': 1,\n", - " 'sewers': 1,\n", - " 'roll dice': 1,\n", - " 'jackson': 1,\n", - " 'decisions |': 1,\n", - " 'thehill': 1,\n", - " 'clear': 2,\n", - " 'chemical weapons': 2,\n", - " 'ukraine': 3,\n", - " 'claims president': 2,\n", - " 'biden': 2,\n", - " 'nasa': 2,\n", - " 'solar system': 2,\n", - " 'daily mail': 3,\n", - " 'us stocks': 1,\n", - " 'fed chair powell': 1,\n", - " \"'s remarks\": 1,\n", - " 'fox': 1,\n", - " \"'we 've\": 1,\n", - " 'tests': 1,\n", - " 'covid': 1,\n", - " 'politico': 1,\n", - " 'duchess': 1,\n", - " 'cambridge': 1,\n", - " 'swaps khaki jungle gear': 1,\n", - " 'vampire': 1,\n", - " 'wife': 1,\n", - " 'belize': 1,\n", - " 'china': 2,\n", - " 'flight recorders': 1,\n", - " 'plane crash': 1,\n", - " 'reuters': 2,\n", - " 'russian oligarch': 1,\n", - " 'abramovich': 1,\n", - " 'live': 1,\n", - " 'russia': 2,\n", - " 'stops talks': 1,\n", - " 'japan': 1,\n", - " 'español': 1,\n", - " 'powers remain': 1,\n", - " 'threats lurk': 1,\n", - " 'set': 1,\n", - " 'webb': 1,\n", - " 'telescope begins multi-instrument alignment': 1,\n", - " 'scitechdaily': 1,\n", - " 'uconn': 1,\n", - " 'ucf': 1,\n", - " 'ncaa': 1,\n", - " \"women 's tournament second-round highlights\": 1,\n", - " 'march madness': 1,\n", - " 'bucking republican trend': 1,\n", - " 'indiana': 1,\n", - " 'vetoes transgender': 1,\n", - " 'bill': 1,\n", - " 'maggie fox': 1,\n", - " 'coronation': 1,\n", - " 'shameless': 1,\n", - " \"'sudden accident\": 1,\n", - " 'mirror online': 1,\n", - " 'mirror': 2,\n", - " 'plane crash –': 1,\n", - " 'search': 1,\n", - " 'moment flight': 1,\n", - " 'daniel morgan': 1,\n", - " 'report condemns': 1,\n", - " 'met': 1,\n", - " 'guardian': 6,\n", - " 'rishi sunak': 1,\n", - " '’ s spring': 1,\n", - " 'statement': 1,\n", - " 'bbc.com': 1,\n", - " 'uk': 3,\n", - " 'ireland': 1,\n", - " 'euro': 1,\n", - " 'vladimir putin': 2,\n", - " \"'s 'lover\": 1,\n", - " 'brass eye': 1,\n", - " '’ s outtakes': 1,\n", - " 'brutal tv comedy': 1,\n", - " 'threatens civilians': 1,\n", - " 'mariupol': 1,\n", - " \"'s spirit\": 1,\n", - " 'shell u-turn': 1,\n", - " 'cambo': 1,\n", - " 'green targets': 1,\n", - " 'st helens': 1,\n", - " 'dog attack': 1,\n", - " 'girl': 1,\n", - " 'bbc': 1,\n", - " 'playstation': 1,\n", - " \"'assassin 's\": 1,\n", - " 'creed': 1,\n", - " 'jade raymond': 1,\n", - " 'haven studios': 1,\n", - " 'nme': 1,\n", - " 'nintendo switch': 1,\n", - " 'folders •': 1,\n", - " 'eurogamer.net': 2,\n", - " 'fa': 1,\n", - " 'solution ”': 1,\n", - " 'liverpool': 1,\n", - " 'fan group blasts “ shambolic ”': 1,\n", - " 'wembley': 1,\n", - " 'anfield': 1,\n", - " 'manchester': 1,\n", - " 'live erik': 1,\n", - " 'hag': 1,\n", - " 'utd': 1,\n", - " 'manager updates': 1,\n", - " 'manchester evening': 1,\n", - " 'inflation': 1,\n", - " 'government borrowing': 1,\n", - " 'february': 1,\n", - " 'crude oil': 1,\n", - " '– business': 1,\n", - " 'kremlin': 1,\n", - " 'large-scale fraud': 1,\n", - " 'sky': 1,\n", - " 'natural gas': 1,\n", - " 'gazprom': 1,\n", - " 'retail unit': 1,\n", - " 'insider': 1,\n", - " 'zaghari-ratcliffe': 1,\n", - " 'hunt': 1,\n", - " 'iran': 1,\n", - " 'debt payment': 1}" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 使用網路服務\n", + "\n", + "網路服務提供不同的呼叫方式，用於估算某個概念屬於不同群組的可能性。更多資訊請參考[這裡](https://concept.research.microsoft.com/Home/Api)。 \n", + "以下是範例 URL 呼叫：`https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`\n" ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for n in TextBlob(x).noun_phrases:\n", - " if n in w:\n", - " w[n].append(x)\n", - " else:\n", - " w[n]=[x]\n", - "{ x:len(w[x]) for x in w.keys()}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "我們可以看到名詞並未給我們大的主題群組。讓我們用從概念圖中獲得的更一般的術語來替代名詞。這將需要一些時間，因為我們正在為每個名詞短語進行 REST 調用。\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "trusted": true - }, - "outputs": [], - "source": [ - "w = {}\n", - "for x in all_titles:\n", - " for noun in TextBlob(x).noun_phrases:\n", - " terms = query(noun.replace(' ','%20'))\n", - " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", - " if term in w:\n", - " w[term].append(x)\n", - " else:\n", - " w[term]=[x]" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "data": { - "text/plain": [ - "{'city': 9,\n", - " 'brand': 4,\n", - " 'place': 9,\n", - " 'town': 4,\n", - " 'factor': 4,\n", - " 'film': 4,\n", - " 'nation': 11,\n", - " 'state': 5,\n", - " 'person': 4,\n", - " 'organization': 5,\n", - " 'publication': 10,\n", - " 'market': 5,\n", - " 'economy': 4,\n", - " 'company': 6,\n", - " 'newspaper': 6,\n", - " 'relationship': 6}" + "cell_type": "code", + "execution_count": null, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "import urllib\n", + "import json\n", + "\n", + "def http(x):\n", + " response = urllib.request.urlopen(x)\n", + " data = response.read()\n", + " return data.decode('utf-8')\n", + "\n", + "def query(x):\n", + " concept = x.lower().replace(' ', '_')\n", + " url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n", + " urllib.parse.quote(concept))\n", + " try:\n", + " result = json.loads(http(url))\n", + " except Exception:\n", + " return {}\n", + " edges = result.get('edges', [])\n", + " if not edges:\n", + " return {}\n", + " total_weight = sum(edge['weight'] for edge in edges)\n", + " if total_weight == 0:\n", + " return {}\n", + " return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n", + "\n", + "query('microsoft')" ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": { - "trusted": true - }, - "outputs": [ + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "ECONOMY:\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "\n", - "NATION:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", - "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", - "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", - "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", - "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", - "\n", - "PERSON:\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", - "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", - "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", - "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" - ] + "cell_type": "markdown", + "metadata": {}, + "source": [ + "讓我們嘗試使用父概念對新聞標題進行分類。要獲取新聞標題，我們將使用 [NewsApi.org](http://newsapi.org) 服務。您需要獲取自己的 API 金鑰才能使用該服務——請訪問網站並註冊免費開發者計劃。\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "newsapi_key = ''\n", + "def get_news(country='us'):\n", + " res = json.loads(http(\"https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}\".format(country,newsapi_key)))\n", + " return res['articles']\n", + "\n", + "all_titles = [x['title'] for x in get_news('us')+get_news('gb')]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',\n", + " 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',\n", + " 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',\n", + " 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',\n", + " \"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN\",\n", + " '2 teachers killed at Swedish high school, student arrested - ABC News',\n", + " 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',\n", + " 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " \"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business\",\n", + " \"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO\",\n", + " \"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\",\n", + " 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',\n", + " 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',\n", + " 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',\n", + " 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',\n", + " 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',\n", + " \"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness\",\n", + " 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',\n", + " \"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror\",\n", + " 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',\n", + " 'Daniel Morgan murder: damning report condemns Met police - The Guardian',\n", + " 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',\n", + " 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',\n", + " \"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\",\n", + " 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',\n", + " \"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times\",\n", + " 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',\n", + " 'St Helens dog attack: Girl aged 17 months killed at home - BBC',\n", + " \"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME\",\n", + " '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',\n", + " 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',\n", + " 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',\n", + " 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',\n", + " 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',\n", + " 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',\n", + " 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',\n", + " \"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\",\n", + " 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "all_titles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "首先，我們希望能夠從新聞標題中提取名詞。我們將使用 `TextBlob` 庫來完成這項工作，該庫簡化了許多像這樣的典型自然語言處理任務。\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: textblob in c:\\winapp\\miniconda3\\lib\\site-packages (0.17.1)\n", + "Requirement already satisfied: nltk>=3.1 in c:\\winapp\\miniconda3\\lib\\site-packages (from textblob) (3.5)\n", + "Requirement already satisfied: joblib in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.0.1)\n", + "Requirement already satisfied: regex in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (2021.11.10)\n", + "Requirement already satisfied: tqdm in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (4.61.2)\n", + "Requirement already satisfied: click in c:\\winapp\\miniconda3\\lib\\site-packages (from nltk>=3.1->textblob) (8.0.3)\n", + "Requirement already satisfied: colorama in c:\\winapp\\miniconda3\\lib\\site-packages (from click->nltk>=3.1->textblob) (0.4.4)\n", + "Finished.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package brown to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package brown is already up-to-date!\n", + "[nltk_data] Downloading package punkt to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package averaged_perceptron_tagger to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package averaged_perceptron_tagger is already up-to-\n", + "[nltk_data] date!\n", + "[nltk_data] Downloading package conll2000 to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package conll2000 is already up-to-date!\n", + "[nltk_data] Downloading package movie_reviews to\n", + "[nltk_data] C:\\Users\\dmitryso\\AppData\\Roaming\\nltk_data...\n", + "[nltk_data] Package movie_reviews is already up-to-date!\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install textblob\n", + "!{sys.executable} -m textblob.download_corpora\n", + "from textblob import TextBlob" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'covid-19 live updates': 1,\n", + " 'vaccines': 1,\n", + " 'boosters': 1,\n", + " 'york': 4,\n", + " 'ukrainians flee mariupol': 1,\n", + " 'forces push': 1,\n", + " 'port city': 1,\n", + " 'wall street journal': 3,\n", + " 'bond yields': 1,\n", + " 'futures rise': 1,\n", + " 'powell says fed': 1,\n", + " 'ready': 1,\n", + " 'be': 1,\n", + " 'aggressive': 1,\n", + " 'putin': 3,\n", + " 'alexei navalny': 2,\n", + " 'russian': 2,\n", + " 'supreme court nominee': 1,\n", + " 'ketanji brown jackson': 1,\n", + " \"confirmation hearing 's\": 1,\n", + " 'cnn': 1,\n", + " 'swedish': 1,\n", + " 'high school': 1,\n", + " 'abc': 1,\n", + " 'clues': 1,\n", + " 'covid-19': 1,\n", + " '’ s': 2,\n", + " 'moves': 1,\n", + " 'sewers': 1,\n", + " 'roll dice': 1,\n", + " 'jackson': 1,\n", + " 'decisions |': 1,\n", + " 'thehill': 1,\n", + " 'clear': 2,\n", + " 'chemical weapons': 2,\n", + " 'ukraine': 3,\n", + " 'claims president': 2,\n", + " 'biden': 2,\n", + " 'nasa': 2,\n", + " 'solar system': 2,\n", + " 'daily mail': 3,\n", + " 'us stocks': 1,\n", + " 'fed chair powell': 1,\n", + " \"'s remarks\": 1,\n", + " 'fox': 1,\n", + " \"'we 've\": 1,\n", + " 'tests': 1,\n", + " 'covid': 1,\n", + " 'politico': 1,\n", + " 'duchess': 1,\n", + " 'cambridge': 1,\n", + " 'swaps khaki jungle gear': 1,\n", + " 'vampire': 1,\n", + " 'wife': 1,\n", + " 'belize': 1,\n", + " 'china': 2,\n", + " 'flight recorders': 1,\n", + " 'plane crash': 1,\n", + " 'reuters': 2,\n", + " 'russian oligarch': 1,\n", + " 'abramovich': 1,\n", + " 'live': 1,\n", + " 'russia': 2,\n", + " 'stops talks': 1,\n", + " 'japan': 1,\n", + " 'español': 1,\n", + " 'powers remain': 1,\n", + " 'threats lurk': 1,\n", + " 'set': 1,\n", + " 'webb': 1,\n", + " 'telescope begins multi-instrument alignment': 1,\n", + " 'scitechdaily': 1,\n", + " 'uconn': 1,\n", + " 'ucf': 1,\n", + " 'ncaa': 1,\n", + " \"women 's tournament second-round highlights\": 1,\n", + " 'march madness': 1,\n", + " 'bucking republican trend': 1,\n", + " 'indiana': 1,\n", + " 'vetoes transgender': 1,\n", + " 'bill': 1,\n", + " 'maggie fox': 1,\n", + " 'coronation': 1,\n", + " 'shameless': 1,\n", + " \"'sudden accident\": 1,\n", + " 'mirror online': 1,\n", + " 'mirror': 2,\n", + " 'plane crash –': 1,\n", + " 'search': 1,\n", + " 'moment flight': 1,\n", + " 'daniel morgan': 1,\n", + " 'report condemns': 1,\n", + " 'met': 1,\n", + " 'guardian': 6,\n", + " 'rishi sunak': 1,\n", + " '’ s spring': 1,\n", + " 'statement': 1,\n", + " 'bbc.com': 1,\n", + " 'uk': 3,\n", + " 'ireland': 1,\n", + " 'euro': 1,\n", + " 'vladimir putin': 2,\n", + " \"'s 'lover\": 1,\n", + " 'brass eye': 1,\n", + " '’ s outtakes': 1,\n", + " 'brutal tv comedy': 1,\n", + " 'threatens civilians': 1,\n", + " 'mariupol': 1,\n", + " \"'s spirit\": 1,\n", + " 'shell u-turn': 1,\n", + " 'cambo': 1,\n", + " 'green targets': 1,\n", + " 'st helens': 1,\n", + " 'dog attack': 1,\n", + " 'girl': 1,\n", + " 'bbc': 1,\n", + " 'playstation': 1,\n", + " \"'assassin 's\": 1,\n", + " 'creed': 1,\n", + " 'jade raymond': 1,\n", + " 'haven studios': 1,\n", + " 'nme': 1,\n", + " 'nintendo switch': 1,\n", + " 'folders •': 1,\n", + " 'eurogamer.net': 2,\n", + " 'fa': 1,\n", + " 'solution ”': 1,\n", + " 'liverpool': 1,\n", + " 'fan group blasts “ shambolic ”': 1,\n", + " 'wembley': 1,\n", + " 'anfield': 1,\n", + " 'manchester': 1,\n", + " 'live erik': 1,\n", + " 'hag': 1,\n", + " 'utd': 1,\n", + " 'manager updates': 1,\n", + " 'manchester evening': 1,\n", + " 'inflation': 1,\n", + " 'government borrowing': 1,\n", + " 'february': 1,\n", + " 'crude oil': 1,\n", + " '– business': 1,\n", + " 'kremlin': 1,\n", + " 'large-scale fraud': 1,\n", + " 'sky': 1,\n", + " 'natural gas': 1,\n", + " 'gazprom': 1,\n", + " 'retail unit': 1,\n", + " 'insider': 1,\n", + " 'zaghari-ratcliffe': 1,\n", + " 'hunt': 1,\n", + " 'iran': 1,\n", + " 'debt payment': 1}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for n in TextBlob(x).noun_phrases:\n", + " if n in w:\n", + " w[n].append(x)\n", + " else:\n", + " w[n]=[x]\n", + "{ x:len(w[x]) for x in w.keys()}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "我們可以看到名詞並未給我們大的主題群組。讓我們用從概念圖中獲得的更一般的術語來替代名詞。這將需要一些時間，因為我們正在為每個名詞短語進行 REST 調用。\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "trusted": true + }, + "outputs": [], + "source": [ + "w = {}\n", + "for x in all_titles:\n", + " for noun in TextBlob(x).noun_phrases:\n", + " terms = query(noun)\n", + " for term in [u for u in terms.keys() if terms[u]>0.1]:\n", + " if term in w:\n", + " w[term].append(x)\n", + " else:\n", + " w[term]=[x]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'city': 9,\n", + " 'brand': 4,\n", + " 'place': 9,\n", + " 'town': 4,\n", + " 'factor': 4,\n", + " 'film': 4,\n", + " 'nation': 11,\n", + " 'state': 5,\n", + " 'person': 4,\n", + " 'organization': 5,\n", + " 'publication': 10,\n", + " 'market': 5,\n", + " 'economy': 4,\n", + " 'company': 6,\n", + " 'newspaper': 6,\n", + " 'relationship': 6}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "{ x:len(w[x]) for x in w.keys() if len(w[x])>3}" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ECONOMY:\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "\n", + "NATION:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "China searches for victims, flight recorders after first plane crash in 12 years - Reuters\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español\n", + "China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent\n", + "UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian\n", + "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider\n", + "Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian\n", + "\n", + "PERSON:\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n", + "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail\n", + "Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters\n", + "‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent\n" + ] + } + ], + "source": [ + "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", + "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", + "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n---\n\n**免責聲明**： \n本文件已使用 AI 翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。雖然我們致力於提供準確的翻譯，但請注意，自動翻譯可能包含錯誤或不準確之處。原始文件的母語版本應被視為權威來源。對於關鍵資訊，建議使用專業人工翻譯。我們對因使用此翻譯而引起的任何誤解或錯誤解釋不承擔責任。\n" + ] } - ], - "source": [ - "print('\\nECONOMY:\\n'+'\\n'.join(w['economy']))\n", - "print('\\nNATION:\\n'+'\\n'.join(w['nation']))\n", - "print('\\nPERSON:\\n'+'\\n'.join(w['person']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n---\n\n**免責聲明**： \n本文件已使用 AI 翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。雖然我們致力於提供準確的翻譯，但請注意，自動翻譯可能包含錯誤或不準確之處。原始文件的母語版本應被視為權威來源。對於關鍵資訊，建議使用專業人工翻譯。我們對因使用此翻譯而引起的任何誤解或錯誤解釋不承擔責任。\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.4 64-bit (conda)", - "metadata": { - "interpreter": { - "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.4 64-bit (conda)", + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + } + }, + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.5" + }, + "coopTranslator": { + "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", + "translation_date": "2025-10-03T18:51:56+00:00", + "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", + "language_code": "tw" } - }, - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.5" - }, - "coopTranslator": { - "original_hash": "7b3f1fc049371bcf8649ac9fefdf0413", - "translation_date": "2025-10-03T18:51:56+00:00", - "source_file": "lessons/2-Symbolic/MSConceptGraph.ipynb", - "language_code": "tw" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } \ No newline at end of file