Merge pull request #612 from microsoft/copilot/update-symbolic-api-endpoint

Replace deprecated Microsoft Concept Graph API with ConceptNet; patch NLTK vulnerabilities

lessons/2-Symbolic/MSConceptGraph.ipynb

@@ -6,70 +6,60 @@
         "collapsed": true
       },
       "source": [
-        "## Microsoft Concept Graph - This API IS NO LONGER AVAILABLE BUT PLEASE LOOK AT THE NOTEBOOK TO UNDERSTAND THE CONCEPT\n",
+        "## Concept Graph using ConceptNet\n",
         "\n",
-        "[Microsoft Concept Graph](https://concept.research.microsoft.com/) is a large taxonomy of terms mined from the internet, with `is-a` relations between concepts. \n",
+        "> **Note:** The original [Microsoft Concept Graph](https://concept.research.microsoft.com/) API is no longer available. This notebook has been updated to use [ConceptNet](https://conceptnet.io/) as a replacement, which is a freely available open knowledge graph with similar `is-a` relations between concepts.\n",
         "\n",
-        "Context Graph is available in two forms:\n",
-        " * Large text file for download\n",
-        " * REST API\n",
+        "[ConceptNet](https://conceptnet.io/) is a large semantic network of concepts with relationships such as `IsA`, `PartOf`, `UsedFor`, and more. It is available as:\n",
+        " * A downloadable data file\n",
+        " * A REST API (no API key required)\n",
         "\n",
-        "Statistics:\n",
-        " * 5401933 unique concepts, \n",
-        " * 12551613 unique instances\n",
-        " * 87603947 `is-a` relations"
+        "ConceptNet statistics:\n",
+        " * Over 8 million nodes\n",
+        " * 21+ million edges across 83 languages"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Using Web Service\n",
+        "## Using ConceptNet Web Service\n",
         "\n",
-        "Web service offers different calls to estimate probability of a concept belonging to different groups. More info is available [here](https://concept.research.microsoft.com/Home/Api).\n",
-        "Here is the sample URL to call: `https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance=microsoft&topK=10`"
+        "[ConceptNet](https://conceptnet.io/) provides a REST API to explore `is-a` (IsA) relationships between concepts. No API key is required.\n",
+        "Here is the sample URL to call: `https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10`"
       ]
     },
     {
       "cell_type": "code",
-      "execution_count": 7,
+      "execution_count": null,
       "metadata": {
         "trusted": true
       },
-      "outputs": [
-        {
-          "data": {
-            "text/plain": [
-              "{'company': 0.6105356614382954,\n",
-              " 'vendor': 0.08858636677518003,\n",
-              " 'client': 0.048239124001183784,\n",
-              " 'firm': 0.045476965571668145,\n",
-              " 'large company': 0.043109401203511886,\n",
-              " 'organization': 0.043010752688172046,\n",
-              " 'corporation': 0.035908059583703265,\n",
-              " 'brand': 0.03383644076156654,\n",
-              " 'software company': 0.027522935779816515,\n",
-              " 'technology company': 0.023774292196902438}"
-            ]
-          },
-          "execution_count": 7,
-          "metadata": {},
-          "output_type": "execute_result"
-        }
-      ],
+      "outputs": [],
       "source": [
         "import urllib\n",
         "import json\n",
-        "import ssl\n",
         "\n",
         "def http(x):\n",
-        "    ssl._create_default_https_context = ssl._create_unverified_context\n",
         "    response = urllib.request.urlopen(x)\n",
         "    data = response.read()\n",
         "    return data.decode('utf-8')\n",
         "\n",
         "def query(x):\n",
-        "    return json.loads(http(\"https://concept.research.microsoft.com/api/Concept/ScoreByProb?instance={}&topK=10\".format(urllib.parse.quote(x))))\n",
+        "    concept = x.lower().replace(' ', '_')\n",
+        "    url = \"https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10\".format(\n",
+        "        urllib.parse.quote(concept))\n",
+        "    try:\n",
+        "        result = json.loads(http(url))\n",
+        "    except Exception:\n",
+        "        return {}\n",
+        "    edges = result.get('edges', [])\n",
+        "    if not edges:\n",
+        "        return {}\n",
+        "    total_weight = sum(edge['weight'] for edge in edges)\n",
+        "    if total_weight == 0:\n",
+        "        return {}\n",
+        "    return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}\n",
         "\n",
         "query('microsoft')"
       ]
@@ -416,7 +406,7 @@
         "w = {}\n",
         "for x in all_titles:\n",
         "    for noun in TextBlob(x).noun_phrases:\n",
-        "        terms = query(noun.replace(' ','%20'))\n",
+        "        terms = query(noun)\n",
         "        for term in [u for u in terms.keys() if terms[u]>0.1]:\n",
         "            if term in w:\n",
         "                w[term].append(x)\n",
@@ -532,4 +522,4 @@
   },
   "nbformat": 4,
   "nbformat_minor": 2
-}
+}