RomanNum3ral
/
tubearchivist
mirror of https://github.com/tubearchivist/tubearchivist.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'develop' into pr_test_1099

This commit is contained in:
Simon 2026-03-28 11:10:56 +07:00
parent 7033f08ad8 fe7a68bf1a
commit e270b27e6d
79 changed files with 3527 additions and 2107 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.github/pull_request_template.md vendored
View File

@ -1,3 +1,5 @@
Thank you for taking the time to improve this project. Please take a look at the [How to make a Pull Request](https://github.com/tubearchivist/tubearchivist/blob/master/CONTRIBUTING.md#how-to-make-a-pull-request) section to help get your contribution merged.
Last updated: 2026-02-06
You can delete this text before submitting.

2
.github/workflows/unit_tests.yml vendored
View File

@ -24,7 +24,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'
python-version: '3.13'
- name: Cache pip
uses: actions/cache@v4

6
CONTRIBUTING.md
View File

@ -71,9 +71,12 @@ Focus for the foreseeable future is on improving and building on existing functi
This is a quick checklist to help streamline the process:
- NEW: Make your PR against the [develop branch](https://github.com/tubearchivist/tubearchivist/tree/develop). That's where all active development happens. This simplifies the later merging into *master*, minimizes any conflicts and usually allows for easy and convenient *fast-forward* merging.
- Make your PR against the [develop branch](https://github.com/tubearchivist/tubearchivist/tree/develop). That's where all active development happens. This simplifies the later merging into *master*, minimizes any conflicts and usually allows for easy and convenient *fast-forward* merging.
- Show off your progress, even if not yet complete, by creating a [draft](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests#draft-pull-requests) PR first and switch it as *ready* when you are ready.
- Make sure all your code is linted and formatted correctly, see below.
- Using a LLM or any tools based on LLM output is a mixed bag:
- If your PR introduces code generated by a LLM but it's not noticeable because the quality is indistinguishable from an experienced dev, then that is fine.
- If your PR is obviously just vibe coded, the PR will likely get rejected without a proper review. Limited maintainer time is better used elsewhere.
### Documentation Changes
@ -134,6 +137,7 @@ The documentation available at [docs.tubearchivist.com](https://docs.tubearchivi
This codebase is set up to be developed natively outside of docker as well as in a docker container. Developing outside of a docker container can be convenient, as IDE and hot reload usually works out of the box. But testing inside of a container is still essential, as there are subtle differences, especially when working with the filesystem and networking between containers.
Note:
- This project doesn't look for contributions to this to cover additional dev setup environments from new contributors. If you are a regular contributor and you see ways to improve this, please reach out on Discord first.
- Subtitles currently fail to load with `DJANGO_DEBUG=True`, that is due to incorrect `Content-Type` error set by Django's static file implementation. That's only if you run the Django dev server, Nginx sets the correct headers in the container.
### Native Instruction

15
Dockerfile
View File

@ -1,11 +1,11 @@
# multi stage to build tube archivist
# build python wheel, download and extract ffmpeg, copy into final image
FROM node:lts-alpine AS npm-builder
FROM node:22.13.0-alpine AS npm-builder
COPY frontend/package.json frontend/package-lock.json /
RUN npm i
FROM node:lts-alpine AS node-builder
FROM node:22.13.0-alpine AS node-builder
# RUN npm config set registry https://registry.npmjs.org/
@ -18,17 +18,19 @@ RUN npm run build:deploy
WORKDIR /
# First stage to build python wheel
FROM python:3.11.13-slim-bookworm AS builder
FROM python:3.13.11-slim-trixie AS builder
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential gcc libldap2-dev libsasl2-dev libssl-dev git
# install requirements
COPY ./backend/requirements.txt /requirements.txt
RUN pip install --user -r requirements.txt
COPY ./backend/requirements.plugins.txt /requirements.plugins.txt
RUN pip install --user -r /requirements.txt \
&& python -m pip install --target /opt/yt_plugins/bgutil -r /requirements.plugins.txt
# build ffmpeg
FROM python:3.11.13-slim-bookworm AS ffmpeg-builder
FROM python:3.13.11-slim-trixie AS ffmpeg-builder
ARG TARGETPLATFORM
@ -36,7 +38,7 @@ COPY docker_assets/ffmpeg_download.py ffmpeg_download.py
RUN python ffmpeg_download.py $TARGETPLATFORM
# build final image
FROM python:3.11.13-slim-bookworm AS tubearchivist
FROM python:3.13.11-slim-trixie AS tubearchivist
ARG INSTALL_DEBUG
@ -46,6 +48,7 @@ COPY --from=denoland/deno:bin /deno /usr/local/bin/deno
# copy build requirements
COPY --from=builder /root/.local /root/.local
COPY --from=builder /opt/yt_plugins /opt/yt_plugins
ENV PATH=/root/.local/bin:$PATH
# copy ffmpeg

1
README.md
View File

@ -238,6 +238,7 @@ This is your time to shine, [read this](https://github.com/tubearchivist/tubearc
* [tangyjoust/Tautulli-Notify-TubeArchivist-of-Plex-Watched-State](https://github.com/tangyjoust/Tautulli-Notify-TubeArchivist-of-Plex-Watched-State) Mark videos watched in Plex (through streaming not manually) through Tautulli back to TubeArchivist
* [Dhs92/delete_shorts](https://github.com/Dhs92/delete_shorts): A script to delete ALL YouTube Shorts from TubeArchivist
* [arisenfromtheashes/TA_DVR](https://github.com/arisenfromtheashes/TA_DVR): Scripts to assist in using Tube Archivist like a DVR
* [WreckingBANG/Self.Tube](https://codeberg.org/WreckingBANG/Self.Tube): Client app for Android and Linux phones written in Flutter.
<!-- The Donate section is parsed by frontend/src/pages/About.tsx -->
## Donate

634
backend/appsettings/index_mapping.json
View File

@ -12,6 +12,28 @@
"channel_id": {
"type": "keyword"
},
"channel_active": {
"type": "boolean"
},
"channel_banner_url": {
"type": "keyword",
"index": false
},
"channel_thumb_url": {
"type": "keyword",
"index": false
},
"channel_tvart_url": {
"type": "keyword",
"index": false
},
"channel_description": {
"type": "text"
},
"channel_last_refresh": {
"type": "date",
"format": "epoch_second"
},
"channel_name": {
"type": "text",
"analyzer": "english",
@ -28,38 +50,6 @@
}
}
},
"channel_banner_url": {
"type": "keyword",
"index": false
},
"channel_tvart_url": {
"type": "keyword",
"index": false
},
"channel_thumb_url": {
"type": "keyword",
"index": false
},
"channel_description": {
"type": "text"
},
"channel_last_refresh": {
"type": "date",
"format": "epoch_second"
},
"channel_tags": {
"type": "text",
"analyzer": "english",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"channel_tabs": {
"type": "keyword"
},
"channel_overwrites": {
"properties": {
"download_format": {
@ -84,6 +74,25 @@
"type": "long"
}
}
},
"channel_subs": {
"type": "long"
},
"channel_subscribed": {
"type": "boolean"
},
"channel_tags": {
"type": "text",
"analyzer": "english",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"channel_tabs": {
"type": "keyword"
}
},
"expected_set": {
@ -101,23 +110,45 @@
{
"index_name": "video",
"expected_map": {
"vid_thumb_url": {
"type": "text",
"index": false
"active": {
"type": "boolean"
},
"vid_thumb_base64": {
"category": {
"type": "text",
"index": false
},
"date_downloaded": {
"type": "date",
"format": "epoch_second"
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"channel": {
"properties": {
"channel_id": {
"type": "keyword"
},
"channel_active": {
"type": "boolean"
},
"channel_banner_url": {
"type": "keyword",
"index": false
},
"channel_thumb_url": {
"type": "keyword",
"index": false
},
"channel_tvart_url": {
"type": "keyword",
"index": false
},
"channel_description": {
"type": "text"
},
"channel_last_refresh": {
"type": "date",
"format": "epoch_second"
},
"channel_name": {
"type": "text",
"analyzer": "english",
@ -134,38 +165,6 @@
}
}
},
"channel_banner_url": {
"type": "keyword",
"index": false
},
"channel_tvart_url": {
"type": "keyword",
"index": false
},
"channel_thumb_url": {
"type": "keyword",
"index": false
},
"channel_description": {
"type": "text"
},
"channel_last_refresh": {
"type": "date",
"format": "epoch_second"
},
"channel_tags": {
"type": "text",
"analyzer": "english",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"channel_tabs": {
"type": "keyword"
},
"channel_overwrites": {
"properties": {
"download_format": {
@ -190,87 +189,44 @@
"type": "long"
}
}
}
}
},
"description": {
"type": "text"
},
"media_url": {
"type": "keyword",
"index": false
},
"media_size": {
"type": "long"
},
"tags": {
"type": "text",
"analyzer": "english",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"title": {
"type": "text",
"analyzer": "english",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256,
"normalizer": "to_lower"
},
"search_as_you_type": {
"type": "search_as_you_type",
"doc_values": false,
"max_shingle_size": 3
}
}
},
"vid_last_refresh": {
"type": "date",
"format": "epoch_second"
},
"youtube_id": {
"type": "keyword"
},
"vid_type": {
"type": "keyword"
},
"published": {
"type": "date",
"format": "epoch_second||strict_date_optional_time"
},
"playlist": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256,
"normalizer": "to_lower"
"channel_subs": {
"type": "long"
},
"channel_subscribed": {
"type": "boolean"
},
"channel_tags": {
"type": "text",
"analyzer": "english",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"channel_tabs": {
"type": "keyword"
}
}
},
"comment_count": {
"type": "long"
},
"stats": {
"properties": {
"average_rating": {
"type": "float"
},
"dislike_count": {
"type": "long"
},
"like_count": {
"type": "long"
},
"view_count": {
"type": "long"
}
}
"date_downloaded": {
"type": "date",
"format": "epoch_second"
},
"description": {
"type": "text"
},
"media_size": {
"type": "long"
},
"media_url": {
"type": "keyword",
"index": false
},
"player": {
"properties": {
@ -290,68 +246,32 @@
}
}
},
"subtitles": {
"properties": {
"ext": {
"playlist": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"index": false
},
"lang": {
"type": "keyword",
"index": false
},
"media_url": {
"type": "keyword",
"index": false
},
"name": {
"type": "keyword"
},
"source": {
"type": "keyword"
},
"url": {
"type": "keyword",
"index": false
"ignore_above": 256,
"normalizer": "to_lower"
}
}
},
"streams": {
"properties": {
"type": {
"type": "keyword",
"index": false
},
"index": {
"type": "short",
"index": false
},
"codec": {
"type": "text"
},
"width": {
"type": "short"
},
"height": {
"type": "short"
},
"bitrate": {
"type": "integer"
}
}
"published": {
"type": "date",
"format": "epoch_second||strict_date_optional_time"
},
"sponsorblock": {
"properties": {
"last_refresh": {
"type": "date",
"format": "epoch_second"
},
"has_unlocked": {
"type": "boolean"
},
"is_enabled": {
"type": "boolean"
},
"last_refresh": {
"type": "date",
"format": "epoch_second"
},
"segments": {
"properties": {
"UUID": {
@ -387,6 +307,112 @@
}
}
}
},
"stats": {
"properties": {
"average_rating": {
"type": "float"
},
"dislike_count": {
"type": "long"
},
"like_count": {
"type": "long"
},
"view_count": {
"type": "long"
}
}
},
"streams": {
"properties": {
"bitrate": {
"type": "integer"
},
"codec": {
"type": "text"
},
"height": {
"type": "short"
},
"index": {
"type": "short",
"index": false
},
"type": {
"type": "keyword",
"index": false
},
"width": {
"type": "short"
}
}
},
"subtitles": {
"properties": {
"ext": {
"type": "keyword",
"index": false
},
"lang": {
"type": "keyword",
"index": false
},
"media_url": {
"type": "keyword",
"index": false
},
"name": {
"type": "keyword"
},
"source": {
"type": "keyword"
},
"url": {
"type": "keyword",
"index": false
}
}
},
"tags": {
"type": "text",
"analyzer": "english",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"title": {
"type": "text",
"analyzer": "english",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256,
"normalizer": "to_lower"
},
"search_as_you_type": {
"type": "search_as_you_type",
"doc_values": false,
"max_shingle_size": 3
}
}
},
"vid_last_refresh": {
"type": "date",
"format": "epoch_second"
},
"vid_thumb_url": {
"type": "text",
"index": false
},
"vid_type": {
"type": "keyword"
},
"youtube_id": {
"type": "keyword"
}
},
"expected_set": {
@ -404,13 +430,15 @@
{
"index_name": "download",
"expected_map": {
"timestamp": {
"type": "date",
"format": "epoch_second"
"auto_start": {
"type": "boolean"
},
"channel_id": {
"type": "keyword"
},
"channel_indexed": {
"type": "boolean"
},
"channel_name": {
"type": "text",
"fields": {
@ -421,9 +449,23 @@
}
}
},
"duration": {
"type": "keyword"
},
"message": {
"type": "text"
},
"published": {
"type": "date",
"format": "epoch_second||strict_date_optional_time"
},
"status": {
"type": "keyword"
},
"timestamp": {
"type": "date",
"format": "epoch_second"
},
"title": {
"type": "text",
"fields": {
@ -434,24 +476,14 @@
}
}
},
"published": {
"type": "date",
"format": "epoch_second||strict_date_optional_time"
},
"vid_thumb_url": {
"type": "keyword"
},
"youtube_id": {
"type": "keyword"
},
"vid_type": {
"type": "keyword"
},
"auto_start": {
"type": "boolean"
},
"message": {
"type": "text"
"youtube_id": {
"type": "keyword"
}
},
"expected_set": {
@ -469,37 +501,9 @@
{
"index_name": "playlist",
"expected_map": {
"playlist_id": {
"type": "keyword"
},
"playlist_description": {
"type": "text"
},
"playlist_subscribed": {
"type": "boolean"
},
"playlist_type": {
"type": "keyword"
},
"playlist_active": {
"type": "boolean"
},
"playlist_name": {
"type": "text",
"analyzer": "english",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256,
"normalizer": "to_lower"
},
"search_as_you_type": {
"type": "search_as_you_type",
"doc_values": false,
"max_shingle_size": 3
}
}
},
"playlist_channel": {
"type": "text",
"fields": {
@ -513,15 +517,8 @@
"playlist_channel_id": {
"type": "keyword"
},
"playlist_thumbnail": {
"type": "keyword"
},
"playlist_last_refresh": {
"type": "date",
"format": "epoch_second"
},
"playlist_sort_order": {
"type": "keyword"
"playlist_description": {
"type": "text"
},
"playlist_entries": {
"properties": {
@ -557,6 +554,41 @@
"type": "keyword"
}
}
},
"playlist_id": {
"type": "keyword"
},
"playlist_last_refresh": {
"type": "date",
"format": "epoch_second"
},
"playlist_name": {
"type": "text",
"analyzer": "english",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256,
"normalizer": "to_lower"
},
"search_as_you_type": {
"type": "search_as_you_type",
"doc_values": false,
"max_shingle_size": 3
}
}
},
"playlist_sort_order": {
"type": "keyword"
},
"playlist_subscribed": {
"type": "boolean"
},
"playlist_thumbnail": {
"type": "keyword"
},
"playlist_type": {
"type": "keyword"
}
},
"expected_set": {
@ -574,22 +606,6 @@
{
"index_name": "subtitle",
"expected_map": {
"youtube_id": {
"type": "keyword"
},
"title": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256,
"normalizer": "to_lower"
}
}
},
"subtitle_fragment_id": {
"type": "keyword"
},
"subtitle_channel": {
"type": "text",
"fields": {
@ -603,15 +619,11 @@
"subtitle_channel_id": {
"type": "keyword"
},
"subtitle_start": {
"type": "text"
},
"subtitle_end": {
"type": "text"
},
"subtitle_last_refresh": {
"type": "date",
"format": "epoch_second"
"subtitle_fragment_id": {
"type": "keyword"
},
"subtitle_index": {
"type": "long"
@ -619,12 +631,32 @@
"subtitle_lang": {
"type": "keyword"
},
"subtitle_source": {
"type": "keyword"
"subtitle_last_refresh": {
"type": "date",
"format": "epoch_second"
},
"subtitle_line": {
"type": "text",
"analyzer": "english"
},
"subtitle_source": {
"type": "keyword"
},
"subtitle_start": {
"type": "text"
},
"title": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256,
"normalizer": "to_lower"
}
}
},
"youtube_id": {
"type": "keyword"
}
},
"expected_set": {
@ -642,37 +674,11 @@
{
"index_name": "comment",
"expected_map": {
"youtube_id": {
"type": "keyword"
},
"comment_last_refresh": {
"type": "date",
"format": "epoch_second"
},
"comment_channel_id": {
"type": "keyword"
},
"comment_comments": {
"properties": {
"comment_id": {
"type": "keyword"
},
"comment_text": {
"type": "text"
},
"comment_timestamp": {
"type": "date",
"format": "epoch_second"
},
"comment_time_text": {
"type": "text"
},
"comment_likecount": {
"type": "long"
},
"comment_is_favorited": {
"type": "boolean"
},
"comment_author": {
"type": "text",
"fields": {
@ -686,16 +692,42 @@
"comment_author_id": {
"type": "keyword"
},
"comment_author_thumbnail": {
"type": "keyword"
},
"comment_author_is_uploader": {
"type": "boolean"
},
"comment_author_thumbnail": {
"type": "keyword"
},
"comment_id": {
"type": "keyword"
},
"comment_is_favorited": {
"type": "boolean"
},
"comment_likecount": {
"type": "long"
},
"comment_parent": {
"type": "keyword"
},
"comment_text": {
"type": "text"
},
"comment_time_text": {
"type": "text"
},
"comment_timestamp": {
"type": "date",
"format": "epoch_second"
}
}
},
"comment_last_refresh": {
"type": "date",
"format": "epoch_second"
},
"youtube_id": {
"type": "keyword"
}
},
"expected_set": {

17
backend/appsettings/serializers.py
View File

@ -44,7 +44,6 @@ class AppConfigDownloadsSerializer(
format = serializers.CharField(allow_null=True)
format_sort = serializers.CharField(allow_null=True)
add_metadata = serializers.BooleanField()
add_thumbnail = serializers.BooleanField()
subtitle = serializers.CharField(allow_null=True)
subtitle_source = serializers.ChoiceField(
choices=["auto", "user"], allow_null=True
@ -55,7 +54,7 @@ class AppConfigDownloadsSerializer(
choices=["top", "new"], allow_null=True
)
cookie_import = serializers.BooleanField()
potoken = serializers.BooleanField()
pot_provider_url = serializers.CharField(allow_null=True)
throttledratelimit = serializers.IntegerField(allow_null=True)
extractor_lang = serializers.CharField(allow_null=True)
integrate_ryd = serializers.BooleanField()
@ -94,10 +93,18 @@ class CookieUpdateSerializer(serializers.Serializer):
cookie = serializers.CharField()
class PoTokenSerializer(serializers.Serializer):
"""serialize PO token"""
class RescanFileSystemConfig(serializers.Serializer):
"""serialize rescan filesystem config"""
potoken = serializers.CharField()
ignore_error = serializers.BooleanField()
prefer_local = serializers.BooleanField()
class ManualImportConfig(serializers.Serializer):
"""serialize for manual import task"""
ignore_error = serializers.BooleanField()
prefer_local = serializers.BooleanField()
class SnapshotItemSerializer(serializers.Serializer):

1
backend/appsettings/serializers_mb.py
View File

@ -29,3 +29,4 @@ class MembershipProfileSerializer(serializers.Serializer):
sponsor_tier = SponsortierSerializer()
subscription_count = serializers.IntegerField()
subscription_is_max = serializers.BooleanField()
is_connected = serializers.BooleanField()

17
backend/appsettings/src/backup.py
View File

@ -7,6 +7,7 @@ Functionality:
import json
import os
import re
import zipfile
from datetime import datetime
@ -19,7 +20,10 @@ from task.models import CustomPeriodicTask
class ElasticBackup:
"""dump index to nd-json files for later bulk import"""
INDEX_SPLIT = ["comment"]
INDEX_SIZE_CONF = {
"comment": 100,
"subtitle": 10000,
}
CACHE_DIR = EnvironmentSettings.CACHE_DIR
BACKUP_DIR = os.path.join(CACHE_DIR, "backup")
@ -60,10 +64,11 @@ class ElasticBackup:
"callback": BackupCallback,
"task": self.task,
"total": self._get_total(index_name),
"timeout": 30,
}
if index_name in self.INDEX_SPLIT:
paginate_kwargs.update({"size": 200})
if size_overwrite := self.INDEX_SIZE_CONF.get(index_name):
paginate_kwargs.update({"size": size_overwrite})
paginate = IndexPaginate(f"ta_{index_name}", **paginate_kwargs)
_ = paginate.get_results()
@ -98,8 +103,7 @@ class ElasticBackup:
def post_bulk_restore(self, file_name):
"""send bulk to es"""
file_path = os.path.join(self.CACHE_DIR, file_name)
with open(file_path, "r", encoding="utf-8") as f:
with open(file_name, "r", encoding="utf-8") as f:
data = f.read()
if not data.strip():
@ -156,6 +160,7 @@ class ElasticBackup:
call reset from ElasticIndexWrap first to start blank
"""
zip_content = self._unpack_zip_backup(filename)
zip_content.sort()
self._restore_json_files(zip_content)
def _unpack_zip_backup(self, filename):
@ -252,7 +257,7 @@ class BackupCallback:
for document in self.source:
document_id = document["_id"]
es_index = document["_index"]
es_index = re.sub(r"_v\d+$", "", document["_index"]) # remove _v
action = {"index": {"_index": es_index, "_id": document_id}}
source = document["_source"]
bulk_list.append(json.dumps(action))

34
backend/appsettings/src/config.py
View File

@ -35,14 +35,13 @@ class DownloadsConfigType(TypedDict):
format: str | None
format_sort: str | None
add_metadata: bool
add_thumbnail: bool
subtitle: str | None
subtitle_source: Literal["user", "auto"] | None
subtitle_index: bool
comment_max: str | None
comment_sort: Literal["top", "new"] | None
cookie_import: bool
potoken: bool
pot_provider_url: str | None
throttledratelimit: int | None
extractor_lang: str | None
integrate_ryd: bool
@ -85,14 +84,13 @@ class AppConfig:
"format": None,
"format_sort": None,
"add_metadata": False,
"add_thumbnail": False,
"subtitle": None,
"subtitle_source": None,
"subtitle_index": False,
"comment_max": None,
"comment_sort": "top",
"cookie_import": False,
"potoken": False,
"pot_provider_url": None,
"throttledratelimit": None,
"extractor_lang": None,
"integrate_ryd": False,
@ -179,6 +177,34 @@ class AppConfig:
return updated
def clear_old_keys(self) -> list[str]:
"""clear old unused keys"""
cleared = []
for key in list(self.config.keys()):
if key not in self.CONFIG_DEFAULTS:
# complete key removed
value = self.config.pop(key)
cleared.append(str({key: value}))
continue
expected_keys = set(
self.CONFIG_DEFAULTS[key].keys() # type: ignore
)
is_keys = set(list(self.config[key].keys()))
for to_delete in is_keys - expected_keys:
self.config[key].pop(to_delete)
cleared.append(f"{key}.{to_delete}")
if not cleared:
return []
response, status_code = ElasticWrap(self.ES_PATH).post(self.config)
if not status_code == 200:
print(response)
return cleared
class ReleaseVersion:
"""compare local version with remote version"""

119
backend/appsettings/src/filesystem.py
View File

@ -5,11 +5,13 @@ Functionality:
import os
from appsettings.src.config import AppConfig
from common.src.env_settings import EnvironmentSettings
from common.src.es_connect import IndexPaginate
from common.src.helper import ignore_filelist
from video.src.comments import CommentList
from common.src.helper import ignore_filelist, rand_sleep
from video.src.comments import Comments
from video.src.index import YoutubeVideo, index_new_video
from video.src.meta_embed import IndexFromEmbed
class Scanner:
@ -17,19 +19,27 @@ class Scanner:
VIDEOS: str = EnvironmentSettings.MEDIA_DIR
def __init__(self, task=False) -> None:
def __init__(
self,
task=False,
ignore_error: bool = False,
prefer_local: bool = False,
) -> None:
self.task = task
self.to_delete: set[str] = set()
self.to_index: set[str] = set()
self.ignore_error = ignore_error
self.prefer_local = prefer_local
self.config = None
self.to_delete: set[tuple[str, str]] = set()
self.to_index: set[tuple[str, str]] = set()
def scan(self) -> None:
"""scan the filesystem"""
downloaded: set[str] = self._get_downloaded()
indexed: set[str] = self._get_indexed()
downloaded = self._get_downloaded()
indexed = self._get_indexed()
self.to_index = downloaded - indexed
self.to_delete = indexed - downloaded
def _get_downloaded(self) -> set[str]:
def _get_downloaded(self) -> set[tuple[str, str]]:
"""get downloaded ids"""
if self.task:
self.task.send_progress(["Scan your filesystem for videos."])
@ -39,28 +49,40 @@ class Scanner:
for channel in channels:
folder = os.path.join(self.VIDEOS, channel)
files = ignore_filelist(os.listdir(folder))
downloaded.update({i.split(".")[0] for i in files})
downloaded.update(
{
(i.split(".")[0], f"{channel}/{i}")
for i in files
if i.endswith(".mp4")
}
)
return downloaded
def _get_indexed(self) -> set:
def _get_indexed(self) -> set[tuple[str, str]]:
"""get all indexed ids"""
if self.task:
self.task.send_progress(["Get all videos indexed."])
data = {"query": {"match_all": {}}, "_source": ["youtube_id"]}
data = {
"query": {"match_all": {}},
"_source": ["youtube_id", "media_url"],
}
response = IndexPaginate("ta_video", data).get_results()
return {i["youtube_id"] for i in response}
return {(i["youtube_id"], i["media_url"]) for i in response}
def apply(self) -> None:
"""apply all changes"""
if not self.config:
self.config = AppConfig().config
self.delete()
self.index()
def delete(self) -> None:
"""delete videos from index"""
if not self.to_delete:
print("nothing to delete")
print("[scanner] nothing to delete")
return
if self.task:
@ -68,26 +90,69 @@ class Scanner:
[f"Remove {len(self.to_delete)} videos from index."]
)
for youtube_id in self.to_delete:
for youtube_id, _ in self.to_delete:
YoutubeVideo(youtube_id).delete_media_file()
def index(self) -> None:
"""index new"""
if not self.to_index:
print("nothing to index")
print("[scanner] nothing to index")
return
total = len(self.to_index)
for idx, youtube_id in enumerate(self.to_index):
if self.task:
self.task.send_progress(
message_lines=[
f"Index missing video {youtube_id}, {idx + 1}/{total}"
],
progress=(idx + 1) / total,
)
index_new_video(youtube_id)
for idx, (youtube_id, media_url) in enumerate(self.to_index):
self._notify(total, youtube_id, idx)
comment_list = CommentList(task=self.task)
comment_list.add(video_ids=list(self.to_index))
comment_list.index()
file_path = os.path.join(self.VIDEOS, media_url)
if self.prefer_local:
# try index from embed
json_data = IndexFromEmbed(
file_path, use_user_conf=True, config=self.config
).run_index()
if json_data:
continue
try:
# try index from remote
json_data = index_new_video(youtube_id)
Comments(youtube_id).build_json(upload=True)
YoutubeVideo(youtube_id).embed_metadata()
rand_sleep(self.config)
except ValueError as err:
# fallback from index from embed
json_data = IndexFromEmbed(
file_path, use_user_conf=True, config=self.config
).run_index()
if json_data:
continue
if self.ignore_error:
self._notify_error(youtube_id)
rand_sleep(self.config)
continue
raise ValueError from err
def _notify(self, total, youtube_id, idx):
"""send notification"""
if not self.task:
return
self.task.send_progress(
message_lines=[
f"Index missing video {youtube_id}, {idx + 1}/{total}"
],
progress=(idx + 1) / total,
)
def _notify_error(self, youtube_id):
"""notify error"""
if not self.task:
return
message = f"[scanner] Failed to index {youtube_id}, no metadata"
print(f"[scanner] {message}")
self.task.send_progress(
message_lines=[message, "Continue..."],
level="error",
)

355
backend/appsettings/src/index_setup.py
View File

@ -5,88 +5,117 @@ functionality:
- backup and restore metadata
"""
from enum import Enum, auto
from appsettings.src.backup import ElasticBackup
from appsettings.src.config import AppConfig
from appsettings.src.snapshot import ElasticSnapshot
from common.src.es_connect import ElasticWrap
from common.src.helper import get_mapping
from deepdiff import DeepDiff
from deepdiff.model import DiffLevel
from django.conf import settings
class MappingAction(Enum):
"""index action options"""
NOOP = auto()
PUT_MAPPING = auto()
REINDEX = auto()
class ElasticIndex:
"""interact with a single index"""
REINDEX_KEYS = {
"type",
"analyzer",
"search_analyzer",
"normalizer",
"index",
"doc_values",
"norms",
"ignore_above",
"enabled",
"format",
}
def __init__(self, index_name, expected_map=False, expected_set=False):
self.index_name = index_name
self.expected_map = expected_map
self.expected_set = expected_set
self.exists, self.details = self.index_exists()
@property
def index_namespace(self) -> str:
"""namespaced index"""
return f"ta_{self.index_name}"
def index_exists(self):
"""check if index already exists and return mapping if it does"""
response, status_code = ElasticWrap(f"ta_{self.index_name}").get()
response, status_code = ElasticWrap(self.index_namespace).get()
exists = status_code == 200
details = response.get(f"ta_{self.index_name}", False)
if not exists:
return False, False
index_key = f"{self.index_namespace}"
current_version = self.get_current_version()
if current_version:
index_key += f"_v{current_version}"
details = response.get(index_key, False)
return exists, details
def validate(self):
def get_current_version(self) -> None | int:
"""get current version from aliases of index"""
response, _ = ElasticWrap(f"{self.index_namespace}/_alias").get()
if not response:
raise ValueError("failed to fetch aliases: ", response)
alias_name = list(response.keys())
if not alias_name:
return None
version_str = alias_name[0].lstrip(f"{self.index_namespace}_v")
if not version_str:
# is initial version
return None
if not version_str.isdigit():
raise ValueError("unexpected version_str: ", version_str)
return int(version_str)
def validate(self) -> tuple[MappingAction, set[str]]:
"""
check if all expected mappings and settings match
returns True when rebuild is needed
"""
if self.expected_map or self.expected_map == {}:
rebuild = self.validate_mappings()
if rebuild:
return rebuild
mapping_diff = self._get_mapping_diff()
removed_fields = self._get_fields_to_delete(diff=mapping_diff)
if self.expected_set:
rebuild = self.validate_settings()
if rebuild:
return rebuild
settings_diff = self._validate_settings()
if settings_diff:
# treat settings diff as full reindex
return MappingAction.REINDEX, removed_fields
return False
if self.expected_map or self.expected_map == {}:
action = self._classify_mapping_diff(diff=mapping_diff)
return action, removed_fields
def validate_mappings(self):
"""check if all mappings are as expected"""
now_map = self.details["mappings"].get("properties", {})
return MappingAction.NOOP, removed_fields
for key, value in self.expected_map.items():
# nested
if list(value.keys()) == ["properties"]:
for key_n, value_n in value["properties"].items():
if key not in now_map:
print(f"detected mapping change: {key_n}, {value_n}")
return True
if key_n not in now_map[key]["properties"].keys():
print(f"detected mapping change: {key_n}, {value_n}")
return True
if not value_n == now_map[key]["properties"][key_n]:
print(f"detected mapping change: {key_n}, {value_n}")
return True
continue
# not nested
if key not in now_map.keys():
print(f"detected mapping change: {key}, {value}")
return True
if not value == now_map[key]:
print(f"detected mapping change: {key}, {value}")
return True
# simple doc store
if self.expected_map == {} and now_map != self.expected_map:
return True
return False
def validate_settings(self):
def _validate_settings(self):
"""check if all settings are as expected"""
now_set = self.details["settings"]["index"]
for key, value in self.expected_set.items():
if key == "number_of_replicas":
continue
if key not in now_set.keys():
print(key, value)
return True
@ -97,44 +126,107 @@ class ElasticIndex:
return False
def rebuild_index(self):
def _get_mapping_diff(self) -> DeepDiff:
"""check if all mappings are as expected"""
now_map = self.details.get("mappings", {}).get("properties", {})
diff = DeepDiff(
now_map,
self.expected_map,
ignore_order=True,
report_repetition=True,
view="tree",
)
if diff:
print(f"[{self.index_namespace}] detected mapping change")
if settings.DEBUG:
print(f"[{self.index_namespace}] mapping change: {diff}")
return diff
def _classify_mapping_diff(self, diff: DeepDiff) -> MappingAction:
"""use diff to detect what to do"""
if not diff:
return MappingAction.NOOP
if diff.get("type_changes"):
# always incompatible, needs reindex
return MappingAction.REINDEX
added = diff.get("dictionary_item_added", [])
reindex_from_added = self._needs_reindex(diff_items=added)
if reindex_from_added:
return MappingAction.REINDEX
removed = diff.get("dictionary_item_removed", [])
reindex_from_removed = self._needs_reindex(diff_items=removed)
if reindex_from_removed:
return MappingAction.REINDEX
changed = diff.get("values_changed", [])
reindex_from_changed = self._needs_reindex(diff_items=changed)
if reindex_from_changed:
return MappingAction.REINDEX
if added or changed:
return MappingAction.PUT_MAPPING
return MappingAction.NOOP
def _needs_reindex(self, diff_items: list[DiffLevel]) -> bool:
"""check if diff has fields that need reindex"""
for item in diff_items:
path = item.path(output_format="list")
if not path:
return False
if path[-1] in self.REINDEX_KEYS:
return True
return False
def _get_fields_to_delete(self, diff: DeepDiff) -> set[str]:
"""fields to remove during next reindex"""
removed_fields = set()
for item in diff.get("dictionary_item_removed", []):
value = item.t1 or {}
is_field_definition = "type" in value or "properties" in value
if not is_field_definition:
continue
path = item.path(output_format="list")
removed_fields.add(".".join(path))
return removed_fields
def rebuild_index(self, removed_fields: set[str]):
"""rebuild with new mapping"""
print(f"applying new mappings to index ta_{self.index_name}...")
self.create_blank(for_backup=True)
self.reindex("backup")
self.delete_index(backup=False)
self.create_blank()
self.reindex("restore")
self.delete_index()
print(f"[{self.index_namespace}] applying new mappings to index")
current_version = self.get_current_version()
def reindex(self, method):
"""create on elastic search"""
if method == "backup":
source = f"ta_{self.index_name}"
destination = f"ta_{self.index_name}_backup"
elif method == "restore":
source = f"ta_{self.index_name}_backup"
destination = f"ta_{self.index_name}"
else:
raise ValueError("invalid method, expected 'backup' or 'restore'")
new_version = current_version + 1 if current_version else 2
data = {"source": {"index": source}, "dest": {"index": destination}}
_, _ = ElasticWrap("_reindex?refresh=true").post(data=data)
self.create_blank(new_version=new_version)
self.reindex(new_version=new_version, removed_fields=removed_fields)
self.delete_index(by_version=current_version)
self.create_alias(new_version=new_version)
def delete_index(self, backup=True):
def delete_index(self, by_version: int | None):
"""delete index passed as argument"""
path = f"ta_{self.index_name}"
if backup:
path = path + "_backup"
path = self.index_namespace
if by_version is not None:
path += f"_v{by_version}"
_, _ = ElasticWrap(path).delete()
print(f"[{path}] delete index")
response, status_code = ElasticWrap(path).delete()
if status_code not in [200, 201]:
print(f"{status_code}: {response}")
raise ValueError("index delete failed")
def create_blank(self, for_backup=False):
"""apply new mapping and settings for blank new index"""
print(f"create new blank index with name ta_{self.index_name}...")
path = f"ta_{self.index_name}"
if for_backup:
path = f"{path}_backup"
def create_blank(self, new_version: int | None = None):
"""create blank"""
path = self.index_namespace
if new_version is not None:
path += f"_v{new_version}"
data = {}
if self.expected_set:
@ -145,7 +237,88 @@ class ElasticIndex:
# no indexing for config
data["mappings"]["dynamic"] = False
_, _ = ElasticWrap(path).put(data)
print(f"[{path}] create new blank index")
if settings.DEBUG:
print(f"[{path}] creat new blank index with data: {data}")
response, status_code = ElasticWrap(path).put(data)
if status_code not in [200, 201]:
print(f"{status_code}: {response}")
raise ValueError(f"create blank index {path} failed")
def reindex(self, new_version: int, removed_fields: set[str]):
"""reindex to versioned new index after creating"""
source = self.index_namespace
dest = f"{self.index_namespace}_v{new_version}"
data: dict = {"source": {"index": source}, "dest": {"index": dest}}
if removed_fields:
script = "\n".join(
f"ctx._source.remove('{i}');" for i in removed_fields
)
data["script"] = {"lang": "painless", "source": script}
msg = f"[{self.index_namespace}] reindex from {source} to {dest}"
if removed_fields:
msg += f", remove unexpected fields: {removed_fields}"
print(msg)
if settings.DEBUG:
print(f"send data: {data}")
path = "_reindex?refresh=true"
response, status_code = ElasticWrap(path).post(data=data)
if status_code not in [200, 201]:
print(f"{status_code}: {response}")
raise ValueError("reindex failed failed")
def create_alias(self, new_version: int):
"""create aliast for moved index"""
index_new = f"{self.index_namespace}_v{new_version}"
index_old = None
data: dict = {
"actions": [
{
"add": {
"index": index_new,
"alias": self.index_namespace,
"is_write_index": True,
},
},
]
}
message = f"create new alias {index_new}"
if index_old:
message += f", remove old alias {index_old}"
print(f"[{self.index_namespace}] {message}")
if settings.DEBUG:
print(f"create alias with data: {data}")
response, status_code = ElasticWrap("_aliases").post(data=data)
if status_code not in [200, 201]:
print(f"{status_code}: {response}")
raise ValueError("alias update failed")
def mapping_update(self):
"""simple mapping update only, use migrations for defaults"""
current_version = self.get_current_version()
path = self.index_namespace
if current_version is not None:
path += f"_v{current_version}"
data = {"properties": self.expected_map}
print(f"[{path}] update mapping")
if settings.DEBUG:
print(f"[{path}] update mapping with data: {data}")
response, status_code = ElasticWrap(f"{path}/_mapping").put(data)
if status_code not in [200, 201]:
print(f"{status_code}: {response}")
raise ValueError(f"create blank index {path} failed")
class ElasticIndexWrap:
@ -164,14 +337,22 @@ class ElasticIndexWrap:
handler.create_blank()
continue
rebuild = handler.validate()
if rebuild:
action, removed_fields = handler.validate()
if action == MappingAction.REINDEX:
self._check_backup()
handler.rebuild_index()
handler.rebuild_index(removed_fields)
continue
# else all good
print(f"ta_{index_name} index is created and up to date...")
if action == MappingAction.PUT_MAPPING:
handler.mapping_update()
if removed_fields:
print(
f"[ta_{index_name}] skip removing unexpected fields:"
+ f" {removed_fields}"
)
else:
print(f"[ta_{index_name}] index status is as expected.")
def reset(self):
"""reset all indexes to blank"""
@ -180,11 +361,15 @@ class ElasticIndexWrap:
def delete_all(self):
"""delete all indexes"""
print("reset elastic index")
for index in self.index_config:
index_name, _, _ = self._config_split(index)
print(f"[ta_{index_name}] reset elastic index")
handler = ElasticIndex(index_name)
handler.delete_index(backup=False)
if not handler.exists:
continue
current_version = handler.get_current_version()
handler.delete_index(by_version=current_version)
def create_all_blank(self):
"""create all blank indexes"""

93
backend/appsettings/src/manual.py
View File

@ -16,8 +16,9 @@ from common.src.env_settings import EnvironmentSettings
from common.src.helper import ignore_filelist
from download.src.thumbnails import ThumbManager
from PIL import Image
from video.src.comments import CommentList
from video.src.comments import Comments
from video.src.index import YoutubeVideo
from video.src.meta_embed import IndexFromEmbed
from yt_dlp.utils import ISO639Utils
@ -41,9 +42,16 @@ class ImportFolderScanner:
"subtitle": [".vtt"],
}
def __init__(self, task=False):
def __init__(
self,
task=False,
ignore_error: bool = False,
prefer_local: bool = False,
):
self.task = task
self.to_import = False
self.ignore_error = ignore_error
self.prefer_local = prefer_local
def scan(self):
"""scan and match media files"""
@ -142,14 +150,14 @@ class ImportFolderScanner:
self._convert_thumb(current_video)
self._get_subtitles(current_video)
self._convert_video(current_video)
print(f"manual import: {current_video}")
ManualImport(current_video, config).run()
video_ids = [i["video_id"] for i in self.to_import]
comment_list = CommentList(task=self.task)
comment_list.add(video_ids=video_ids)
comment_list.index()
ManualImport(
current_video,
config,
ignore_error=self.ignore_error,
prefer_local=self.prefer_local,
).run()
def _notify(self, idx, current_video):
"""send notification back to task"""
@ -185,17 +193,27 @@ class ImportFolderScanner:
expects filename ending in [<youtube_id>].<ext>
"""
base_name, _ = os.path.splitext(file_name)
# yt-dlp default like [youtubeid]
id_search = re.search(r"\[([a-zA-Z0-9_-]{11})\]$", base_name)
if id_search:
youtube_id = id_search.group(1)
return youtube_id
file_name_search = re.search(r"([a-zA-Z0-9_-]{11})$", base_name)
if file_name_search:
youtube_id = file_name_search.group(1)
return youtube_id
print(f"id extraction failed from filename: {file_name}")
return False
def _extract_id_from_json(self, json_file):
def _extract_id_from_json(self, json_file: str | bool) -> str | None:
"""open json file and extract id"""
if not json_file or not isinstance(json_file, str):
return None
json_path = os.path.join(self.CACHE_DIR, "import", json_file)
with open(json_path, "r", encoding="utf-8") as f:
json_content = f.read()
@ -388,17 +406,49 @@ class ImportFolderScanner:
class ManualImport:
"""import single identified video"""
def __init__(self, current_video, config):
def __init__(
self, current_video, config, ignore_error: bool, prefer_local: bool
):
self.current_video = current_video
self.config = config
self.ignore_error: bool = ignore_error
self.prefer_local: bool = prefer_local
def run(self):
"""run all"""
json_data = self.index_metadata()
self._move_to_archive(json_data)
self._cleanup(json_data)
json_data = None
if self.prefer_local:
# embedded first
json_data = IndexFromEmbed(
self.current_video["media"],
use_user_conf=False,
config=self.config,
).run_index()
if json_data:
self._cleanup()
return
def index_metadata(self):
try:
json_data = self.index_metadata()
except ValueError as err:
json_data = IndexFromEmbed(
self.current_video["media"],
use_user_conf=False,
config=self.config,
).run_index()
if not json_data and not self.ignore_error:
raise ValueError from err
if not json_data:
return
self._move_to_archive(json_data)
self._cleanup()
Comments(self.current_video["video_id"]).build_json(upload=True)
YoutubeVideo(self.current_video["video_id"]).embed_metadata()
def index_metadata(self) -> dict | None:
"""get metadata from yt or json"""
video_id = self.current_video["video_id"]
video = YoutubeVideo(video_id)
@ -411,6 +461,9 @@ class ManualImport:
f"{video_id}: manual import failed, and no metadata found."
)
print(message)
if self.ignore_error:
return None
raise ValueError(message)
video.check_subtitles(subtitle_files=self.current_video["subtitle"])
@ -463,7 +516,7 @@ class ManualImport:
new_path = f"{base_name}.{lang}.vtt"
shutil.move(old_path, new_path, copy_function=shutil.copyfile)
def _cleanup(self, json_data):
def _cleanup(self):
"""cleanup leftover files"""
meta_data = self.current_video["metadata"]
if meta_data and os.path.exists(meta_data):
@ -476,11 +529,3 @@ class ManualImport:
for subtitle_file in self.current_video["subtitle"]:
if os.path.exists(subtitle_file):
os.remove(subtitle_file)
channel_info = os.path.join(
EnvironmentSettings.CACHE_DIR,
"import",
f"{json_data['channel']['channel_id']}.info.json",
)
if os.path.exists(channel_info):
os.remove(channel_info)

48
backend/appsettings/src/reindex.py
View File

@ -7,7 +7,7 @@ functionality:
import json
import os
from datetime import datetime
from typing import Callable, TypedDict
from typing import TypedDict
from appsettings.src.config import AppConfig
from channel.src.index import YoutubeChannel
@ -277,7 +277,6 @@ class Reindex(ReindexBase):
def reindex_type(self, name: str, index_config: ReindexConfigType) -> None:
"""reindex all of a single index"""
reindex = self._get_reindex_map(index_config["index_name"])
queue = RedisQueue(index_config["queue_name"])
while True:
total = queue.max_score()
@ -288,44 +287,48 @@ class Reindex(ReindexBase):
if self.task:
self._notify(name, total, idx)
reindex(youtube_id)
index_name = index_config["index_name"]
if index_name == "ta_video":
video = self.reindex_single_video(youtube_id)
if video:
self._reindex_video_related(video)
elif index_name == "ta_channel":
self._reindex_single_channel(channel_id=youtube_id)
elif index_name == "ta_playlist":
self._reindex_single_playlist(playlist_id=youtube_id)
rand_sleep(self.config)
def _get_reindex_map(self, index_name: str) -> Callable:
"""return def to run for index"""
def_map = {
"ta_video": self._reindex_single_video,
"ta_channel": self._reindex_single_channel,
"ta_playlist": self._reindex_single_playlist,
}
return def_map[index_name]
def _notify(self, name: str, total: int, idx: int) -> None:
"""send notification back to task"""
message = [f"Reindexing {name.title()}s {idx}/{total}"]
progress = idx / total
self.task.send_progress(message, progress=progress)
def _reindex_single_video(self, youtube_id: str) -> None:
def reindex_single_video(self, youtube_id: str) -> YoutubeVideo | None:
"""refresh data for single video"""
video = YoutubeVideo(youtube_id)
# read current state
video.get_from_es()
if not video.json_data:
return
return None
es_meta = video.json_data.copy()
# get new
media_url = os.path.join(
media_url: str | bool = os.path.join(
EnvironmentSettings.MEDIA_DIR, es_meta["media_url"]
)
if not os.path.exists(media_url):
# fallback to cache path
media_url = False
video.build_json(media_path=media_url)
if not video.youtube_meta:
video.deactivate()
return
return None
video.delete_subtitles(subtitles=es_meta.get("subtitles"))
video.check_subtitles()
@ -339,14 +342,19 @@ class Reindex(ReindexBase):
video.json_data["playlist"] = es_meta.get("playlist")
video.upload_to_es()
self.processed["videos"] += 1
thumb_handler = ThumbManager(youtube_id)
return video
def _reindex_video_related(self, video: YoutubeVideo) -> None:
"""reindex video related metadata and fields"""
thumb_handler = ThumbManager(video.youtube_id)
thumb_handler.delete_video_thumb()
thumb_handler.download_video_thumb(video.json_data["vid_thumb_url"])
Comments(youtube_id, config=self.config).reindex_comments()
Comments(video.youtube_id, config=self.config).reindex_comments()
video.get_from_es()
video.embed_metadata()
self.processed["videos"] += 1
def _reindex_single_channel(self, channel_id: str) -> None:
"""refresh channel data and sync to videos"""

9
backend/appsettings/src/snapshot.py
View File

@ -261,10 +261,15 @@ class ElasticSnapshot:
def restore_all(self, snapshot_name):
"""restore snapshot by name"""
for index in self.all_indices:
_, _ = ElasticWrap(index).delete()
response, status_code = ElasticWrap(index).get()
if status_code == 404:
continue
index_alias = list(response.keys())[0]
_, _ = ElasticWrap(index_alias).delete()
path = f"_snapshot/{self.REPO}/{snapshot_name}/_restore"
data = {"indices": "*"}
data = {"indices": "*,-.*"}
response, statuscode = ElasticWrap(path).post(data=data)
if statuscode == 200:
print(f"snapshot: executing now: {response}")

15
backend/appsettings/urls.py
View File

@ -34,16 +34,21 @@ urlpatterns = [
views.CookieView.as_view(),
name="api-cookie",
),
path(
"potoken/",
views.POTokenView.as_view(),
name="api-potoken",
),
path(
"token/",
views.TokenView.as_view(),
name="api-token",
),
path(
"rescan-filesystem/",
views.RescanFileSystem.as_view(),
name="api-rescan-filesystem",
),
path(
"manual-import/",
views.ManualImportView.as_view(),
name="api-manual-import",
),
path(
"membership/profile/",
views_mb.MembershipProfileView.as_view(),

120
backend/appsettings/views.py
View File

@ -5,7 +5,8 @@ from appsettings.serializers import (
BackupFileSerializer,
CookieUpdateSerializer,
CookieValidationSerializer,
PoTokenSerializer,
ManualImportConfig,
RescanFileSystemConfig,
SnapshotCreateResponseSerializer,
SnapshotItemSerializer,
SnapshotListSerializer,
@ -22,7 +23,7 @@ from common.serializers import (
from common.src.ta_redis import RedisArchivist
from common.views_base import AdminOnly, AdminWriteOnly, ApiBaseView
from django.conf import settings
from download.src.yt_dlp_base import CookieHandler, POTokenHandler
from download.src.yt_dlp_base import CookieHandler
from drf_spectacular.utils import OpenApiResponse, extend_schema
from rest_framework.authtoken.models import Token
from rest_framework.response import Response
@ -290,69 +291,6 @@ class CookieView(ApiBaseView):
return validation
class POTokenView(ApiBaseView):
"""handle PO token"""
permission_classes = [AdminOnly]
@extend_schema(
responses={
200: OpenApiResponse(PoTokenSerializer()),
404: OpenApiResponse(
ErrorResponseSerializer(), description="PO token not found"
),
}
)
def get(self, request):
"""get PO token"""
config = AppConfig().config
potoken = POTokenHandler(config).get()
if not potoken:
error = ErrorResponseSerializer({"error": "PO token not found"})
return Response(error.data, status=404)
serializer = PoTokenSerializer(data={"potoken": potoken})
serializer.is_valid(raise_exception=True)
return Response(serializer.data)
@extend_schema(
responses={
200: OpenApiResponse(PoTokenSerializer()),
400: OpenApiResponse(
ErrorResponseSerializer(), description="Bad request"
),
}
)
def post(self, request):
"""Update PO token"""
serializer = PoTokenSerializer(data=request.data)
serializer.is_valid(raise_exception=True)
validated_data = serializer.validated_data
if not validated_data:
error = ErrorResponseSerializer(
{"error": "missing PO token key in request data"}
)
return Response(error.data, status=400)
config = AppConfig().config
new_token = validated_data["potoken"]
POTokenHandler(config).set_token(new_token)
return Response(serializer.data)
@extend_schema(
responses={
204: OpenApiResponse(description="PO token revoked"),
},
)
def delete(self, request):
"""delete PO token"""
config = AppConfig().config
POTokenHandler(config).revoke_token()
return Response(status=204)
class SnapshotApiListView(ApiBaseView):
"""resolves to /api/appsettings/snapshot/
GET: returns snapshot config plus list of existing snapshots
@ -388,6 +326,58 @@ class SnapshotApiListView(ApiBaseView):
return Response(serializer.data)
class RescanFileSystem(ApiBaseView):
"""resolves to /api/appsettings/rescan-filesystem/
POST: start new rescan filesystem task
"""
permission_classes = [AdminOnly]
@staticmethod
@extend_schema(
request=RescanFileSystemConfig,
responses={
200: OpenApiResponse(AsyncTaskResponseSerializer()),
},
)
def post(request):
"""start new task rescan filesystem task"""
data_serializer = RescanFileSystemConfig(data=request.data)
data_serializer.is_valid(raise_exception=True)
validated_data = data_serializer.validated_data
message = TaskCommand().start("rescan_filesystem", validated_data)
serializer = AsyncTaskResponseSerializer(message)
return Response(serializer.data)
class ManualImportView(ApiBaseView):
"""resolves to /api/appsettings/manual-import/
POST: start new manual import task
"""
permission_classes = [AdminOnly]
@staticmethod
@extend_schema(
request=ManualImportConfig,
responses={
200: OpenApiResponse(AsyncTaskResponseSerializer()),
},
)
def post(request):
"""manual import"""
data_serializer = ManualImportConfig(data=request.data)
data_serializer.is_valid(raise_exception=True)
validated_data = data_serializer.validated_data
message = TaskCommand().start("manual_import", validated_data)
serializer = AsyncTaskResponseSerializer(message)
return Response(serializer.data)
class SnapshotApiView(ApiBaseView):
"""resolves to /api/appsettings/snapshot/<snapshot-id>/
GET: return a single snapshot

11
backend/channel/serializers.py
View File

@ -34,10 +34,12 @@ class ChannelSerializer(serializers.Serializer):
channel_id = serializers.CharField()
channel_active = serializers.BooleanField()
channel_banner_url = serializers.CharField()
channel_thumb_url = serializers.CharField()
channel_tvart_url = serializers.CharField()
channel_description = serializers.CharField()
channel_banner_url = serializers.CharField(allow_null=True, required=False)
channel_thumb_url = serializers.CharField(allow_null=True, required=False)
channel_tvart_url = serializers.CharField(allow_null=True, required=False)
channel_description = serializers.CharField(
allow_null=True, required=False
)
channel_last_refresh = serializers.CharField()
channel_name = serializers.CharField()
channel_overwrites = ChannelOverwriteSerializer(required=False)
@ -49,7 +51,6 @@ class ChannelSerializer(serializers.Serializer):
channel_tabs = serializers.ListField(
child=serializers.ChoiceField(VideoTypeEnum.values_known())
)
channel_views = serializers.IntegerField()
_index = serializers.CharField(required=False)
_score = serializers.IntegerField(required=False)

88
backend/channel/src/index.py
View File

@ -57,54 +57,56 @@ class YoutubeChannel(YouTubeItem):
"""extract relevant fields"""
self.youtube_meta["thumbnails"].reverse()
channel_name = self.youtube_meta["uploader"] or self.youtube_meta["id"]
description = self.youtube_meta.get("description") or None
self.json_data = {
"channel_active": True,
"channel_description": self.youtube_meta.get("description", ""),
"channel_description": description,
"channel_id": self.youtube_id,
"channel_last_refresh": int(datetime.now().timestamp()),
"channel_name": channel_name,
"channel_subs": self.youtube_meta.get("channel_follower_count", 0),
"channel_subs": self.youtube_meta.get("channel_follower_count")
or 0,
"channel_subscribed": False,
"channel_tags": self.youtube_meta.get("tags", []),
"channel_banner_url": self._get_banner_art(),
"channel_thumb_url": self._get_thumb_art(),
"channel_tvart_url": self._get_tv_art(),
"channel_views": self.youtube_meta.get("view_count") or 0,
"channel_tabs": self.get_channel_tabs(),
}
def _get_thumb_art(self):
self._get_thumb_art()
self._get_tv_art()
self._get_banner_art()
def _get_thumb_art(self) -> None:
"""extract thumb art"""
for i in self.youtube_meta["thumbnails"]:
if not i.get("width"):
continue
if i.get("width") == i.get("height"):
return i["url"]
self.json_data["channel_thumb_url"] = i["url"]
return
return False
def _get_tv_art(self):
def _get_tv_art(self) -> None:
"""extract tv artwork"""
for i in self.youtube_meta["thumbnails"]:
if i.get("id") == "banner_uncropped":
return i["url"]
self.json_data["channel_tvart_url"] = i["url"]
return
for i in self.youtube_meta["thumbnails"]:
if not i.get("width"):
continue
if i["width"] // i["height"] < 2 and not i["width"] == i["height"]:
return i["url"]
self.json_data["channel_tvart_url"] = i["url"]
return
return False
return
def _get_banner_art(self):
def _get_banner_art(self) -> None:
"""extract banner artwork"""
for i in self.youtube_meta["thumbnails"]:
if not i.get("width"):
continue
if i["width"] // i["height"] > 5:
return i["url"]
return False
self.json_data["channel_banner_url"] = i["url"]
return
def get_channel_tabs(self) -> list[str]:
"""get channel tabs"""
@ -132,51 +134,39 @@ class YoutubeChannel(YouTubeItem):
self.json_data = {
"channel_active": False,
"channel_last_refresh": int(datetime.now().timestamp()),
"channel_subs": fallback.get("channel_follower_count", 0),
"channel_subs": fallback.get("channel_follower_count") or 0,
"channel_name": fallback["uploader"],
"channel_banner_url": False,
"channel_tvart_url": False,
"channel_id": self.youtube_id,
"channel_subscribed": False,
"channel_tags": [],
"channel_description": "",
"channel_thumb_url": False,
"channel_views": 0,
}
def get_channel_art(self):
"""download channel art for new channels"""
urls = (
self.json_data["channel_thumb_url"],
self.json_data["channel_banner_url"],
self.json_data["channel_tvart_url"],
self.json_data.get("channel_thumb_url"),
self.json_data.get("channel_banner_url"),
self.json_data.get("channel_tvart_url"),
)
ThumbManager(self.youtube_id, item_type="channel").download(urls)
def sync_to_videos(self):
"""sync new channel_dict to all videos of channel"""
# add ingest pipeline
processors = []
for field, value in self.json_data.items():
if value is None:
line = {
"script": {
"lang": "painless",
"source": f"ctx['{field}'] = null;",
}
}
else:
line = {"set": {"field": "channel." + field, "value": value}}
processors.append(line)
data = {"description": self.youtube_id, "processors": processors}
ingest_path = f"_ingest/pipeline/{self.youtube_id}"
_, _ = ElasticWrap(ingest_path).put(data)
# apply pipeline
data = {"query": {"match": {"channel.channel_id": self.youtube_id}}}
update_path = f"ta_video/_update_by_query?pipeline={self.youtube_id}"
_, _ = ElasticWrap(update_path).post(data)
data = {
"query": {
"term": {"channel.channel_id": {"value": self.youtube_id}},
},
"script": {
"lang": "painless",
"params": {"channel": self.json_data},
"source": "ctx._source.channel = params.channel",
},
}
update_path = "ta_video/_update_by_query"
response, status_code = ElasticWrap(update_path).post(data)
if status_code not in [200, 201]:
print(f"sync to videos failed with status code {status_code}")
print(response)
def change_subscribe(self, new_subscribe_state: bool):
"""change subscribe status"""

32
backend/common/src/env_settings.py
View File

@ -4,7 +4,7 @@ Functionality:
- encapsulate persistence of application properties
"""
from os import environ
from os import environ, path
try:
from dotenv import load_dotenv
@ -15,6 +15,32 @@ except ModuleNotFoundError:
pass
def get_password_from_file(env_var_name) -> str:
"""get password from file"""
env_var_file: str = env_var_name + "_FILE"
env_var_name_val = environ.get(env_var_name)
env_var_path_val = environ.get(env_var_file)
if env_var_name_val is not None:
return str(env_var_name_val)
if env_var_path_val is None:
print(f"either {env_var_name} or {env_var_file} must be set")
return ""
is_path = path.isfile(env_var_path_val)
if not is_path:
print(f"{env_var_path_val} is not a path")
return ""
with open(env_var_path_val, "r", encoding="utf-8") as f:
file_content = f.read().strip()
return file_content
class EnvironmentSettings:
"""
Handle settings for the application that are driven from the environment.
@ -29,7 +55,7 @@ class EnvironmentSettings:
TA_PORT: int = int(environ.get("TA_PORT", False))
TA_BACKEND_PORT: int = int(environ.get("TA_BACKEND_PORT", False))
TA_USERNAME: str = str(environ.get("TA_USERNAME"))
TA_PASSWORD: str = str(environ.get("TA_PASSWORD"))
TA_PASSWORD: str = get_password_from_file("TA_PASSWORD")
# Application Paths
MEDIA_DIR: str = str(environ.get("TA_MEDIA_DIR", "/youtube"))
@ -42,7 +68,7 @@ class EnvironmentSettings:
# ElasticSearch
ES_URL: str = str(environ.get("ES_URL"))
ES_PASS: str = str(environ.get("ELASTIC_PASSWORD"))
ES_PASS: str = get_password_from_file("ELASTIC_PASSWORD")
ES_USER: str = str(environ.get("ELASTIC_USER", "elastic"))
ES_SNAPSHOT_DIR: str = str(
environ.get(

13
backend/common/src/es_connect.py
View File

@ -148,6 +148,8 @@ class IndexPaginate:
- callback: obj, Class implementing run method callback for every loop
- task: task object to send notification
- total: int, total items in index for progress message
- timeout: int, overwrite timeout in get request
- pit_keep_alive: int, overwrite pit valid
"""
DEFAULT_SIZE = 500
@ -168,7 +170,8 @@ class IndexPaginate:
def get_pit(self):
"""get pit for index"""
path = f"{self.index_name}/_pit?keep_alive=10m"
keep_alive = self.kwargs.get("pit_keep_alive", 15)
path = f"{self.index_name}/_pit?keep_alive={keep_alive}m"
response, _ = ElasticWrap(path).post()
self.pit_id = response["id"]
@ -184,14 +187,18 @@ class IndexPaginate:
self.data.update({"sort": [{"_doc": {"order": "desc"}}]})
self.data["size"] = self.kwargs.get("size") or self.DEFAULT_SIZE
self.data["pit"] = {"id": self.pit_id, "keep_alive": "10m"}
self.data["pit"] = {"id": self.pit_id, "keep_alive": "15m"}
def run_loop(self):
"""loop through results until last hit"""
all_results = []
counter = 0
while True:
response, _ = ElasticWrap("_search").get(data=self.data)
get_kwargs = {"data": self.data}
if timeout_overwrite := self.kwargs.get("timeout"):
get_kwargs.update({"timeout": timeout_overwrite})
response, _ = ElasticWrap("_search").get(**get_kwargs)
all_hits = response["hits"]["hits"]
if not all_hits:
break

19
backend/common/src/helper.py
View File

@ -103,13 +103,17 @@ def requests_headers() -> dict[str, str]:
return {"User-Agent": template}
def date_parser(timestamp: int | str | None) -> str | None:
def date_parser(timestamp: int | float | str | None) -> str | None:
"""return formatted date string"""
if timestamp is None:
return None
if isinstance(timestamp, int):
date_obj = datetime.fromtimestamp(timestamp, tz=timezone.utc)
elif isinstance(timestamp, float):
date_obj = datetime.fromtimestamp(int(timestamp), tz=timezone.utc)
elif isinstance(timestamp, str) and timestamp.isdigit():
date_obj = datetime.fromtimestamp(int(timestamp), tz=timezone.utc)
elif isinstance(timestamp, str):
date_obj = datetime.strptime(timestamp, "%Y-%m-%d")
date_obj = date_obj.replace(tzinfo=timezone.utc)
@ -131,6 +135,19 @@ def time_parser(timestamp: str) -> float:
return int(hours) * 60 * 60 + int(minutes) * 60 + float(seconds)
def deep_merge(target: dict, source: dict) -> None:
"""inplace nested dict merge, recursive"""
for key, value in source.items():
if (
key in target
and isinstance(target[key], dict)
and isinstance(value, dict)
):
deep_merge(target[key], value)
else:
target[key] = value
def clear_dl_cache(cache_dir: str) -> int:
"""clear leftover files from dl cache"""
print("clear download cache")

6
backend/common/src/index_generic.py
View File

@ -53,11 +53,11 @@ class YouTubeItem:
obs_request, self.config
).extract(url)
def get_from_es(self):
def get_from_es(self, print_error: bool = True) -> None:
"""get indexed data from elastic search"""
print(f"{self.youtube_id}: get metadata from es")
response, _ = ElasticWrap(f"{self.es_path}").get()
source = response.get("_source")
resp, _ = ElasticWrap(f"{self.es_path}").get(print_error=print_error)
source = resp.get("_source")
self.json_data = source
def upload_to_es(self):

56
backend/common/src/search_processor.py
View File

@ -56,17 +56,17 @@ class SearchProcess:
"""detect which type of data to process"""
index = result["_index"]
processed = False
if index == "ta_video":
if index.startswith("ta_video"):
processed = self._process_video(result["_source"])
if index == "ta_channel":
if index.startswith("ta_channel"):
processed = self._process_channel(result["_source"])
if index == "ta_playlist":
if index.startswith("ta_playlist"):
processed = self._process_playlist(result["_source"])
if index == "ta_download":
if index.startswith("ta_download"):
processed = self._process_download(result["_source"])
if index == "ta_comment":
if index.startswith("ta_comment"):
processed = self._process_comment(result["_source"])
if index == "ta_subtitle":
if index.startswith("ta_subtitle"):
processed = self._process_subtitle(result)
if isinstance(processed, dict):
@ -89,12 +89,25 @@ class SearchProcess:
channel_dict.update(
{
"channel_last_refresh": date_str,
"channel_banner_url": f"{art_base}_banner.jpg",
"channel_thumb_url": f"{art_base}_thumb.jpg",
"channel_tvart_url": f"{art_base}_tvart.jpg",
"channel_description": channel_dict.get("channel_description"),
}
)
if channel_dict.get("channel_banner_url"):
channel_dict["channel_banner_url"] = f"{art_base}_banner.jpg"
else:
channel_dict["channel_banner_url"] = None
if channel_dict.get("channel_thumb_url"):
channel_dict["channel_thumb_url"] = f"{art_base}_thumb.jpg"
else:
channel_dict["channel_thumb_url"] = None
if channel_dict.get("channel_tvart_url"):
channel_dict["channel_tvart_url"] = f"{art_base}_tvart.jpg"
else:
channel_dict["channel_tvart_url"] = None
return dict(sorted(channel_dict.items()))
def _process_video(self, video_dict):
@ -125,6 +138,7 @@ class SearchProcess:
"vid_last_refresh": vid_last_refresh,
"published": published,
"vid_thumb_url": f"{cache_root}/{vid_thumb_url}",
"description": video_dict.get("description"),
}
)
@ -154,10 +168,12 @@ class SearchProcess:
)
cache_root = EnvironmentSettings().get_cache_root()
playlist_thumbnail = f"{cache_root}/playlists/{playlist_id}.jpg"
description = playlist_dict.get("playlist_description")
playlist_dict.update(
{
"playlist_thumbnail": playlist_thumbnail,
"playlist_last_refresh": playlist_last_refresh,
"playlist_description": description,
}
)
@ -182,17 +198,23 @@ class SearchProcess:
def _process_comment(self, comment_dict):
"""run on all comments, create reply thread"""
all_comments = comment_dict["comment_comments"]
processed_comments = []
comment_tree = []
lookup = {}
for comment in comment_dict["comment_comments"]:
comment = comment.copy()
comment["comment_replies"] = []
lookup[comment["comment_id"]] = comment
for comment in all_comments:
if comment["comment_parent"] == "root":
comment.update({"comment_replies": []})
processed_comments.append(comment)
for comment in lookup.values():
parent = comment.get("comment_parent")
if parent == "root":
comment_tree.append(comment)
else:
processed_comments[-1]["comment_replies"].append(comment)
parent_node = lookup.get(parent)
if parent_node:
parent_node["comment_replies"].append(comment)
return processed_comments
return comment_tree
def _process_subtitle(self, result):
"""take complete result dict to extract highlight"""

25
backend/common/src/searching.py
View File

@ -31,13 +31,13 @@ class SearchForm:
fulltext_results = []
if search_results:
for result in search_results:
if result["_index"] == "ta_video":
if result["_index"].startswith("ta_video"):
video_results.append(result)
elif result["_index"] == "ta_channel":
elif result["_index"].startswith("ta_channel"):
channel_results.append(result)
elif result["_index"] == "ta_playlist":
elif result["_index"].startswith("ta_playlist"):
playlist_results.append(result)
elif result["_index"] == "ta_subtitle":
elif result["_index"].startswith("ta_subtitle"):
fulltext_results.append(result)
all_results = {
@ -113,6 +113,7 @@ class SearchParser:
"index": "ta_subtitle",
"lang": [],
"source": [],
"channel": [],
},
}
@ -345,6 +346,22 @@ class QueryBuilder:
"""build query for fulltext search"""
must_list = []
if (channel := self.query_map.get("channel")) is not None:
must_list.append(
{
"multi_match": {
"query": channel,
"type": "bool_prefix",
"fuzziness": self._get_fuzzy(),
"operator": "and",
"fields": [
"subtitle_channel",
"subtitle_channel.keyword",
],
}
}
)
if (term := self.query_map.get("term")) is not None:
must_list.append(
{

14
backend/common/tests/test_src/test_helper.py
View File

@ -26,6 +26,20 @@ def test_date_parser_with_int():
assert date_parser(timestamp) == expected_date
def test_date_parser_with_digit():
"""unix timestamp"""
timestamp = "1621539600"
expected_date = "2021-05-20T19:40:00+00:00"
assert date_parser(timestamp) == expected_date
def test_date_parser_with_float():
"""iso timestamp"""
date_float = 1766210400.0
expected_date = "2025-12-20T06:00:00+00:00"
assert date_parser(date_float) == expected_date
def test_date_parser_with_str():
"""iso timestamp"""
date_str = "2021-05-21"

19
backend/config/management/commands/ta_envcheck.py
View File

@ -61,6 +61,10 @@ EXPECTED_ENV_VARS = [
"ES_URL",
"TA_HOST",
]
FILE_FALLBACK = [
"ELASTIC_PASSWORD",
"TA_PASSWORD",
]
UNEXPECTED_ENV_VARS = {
"TA_UWSGI_PORT": "Has been replaced with 'TA_BACKEND_PORT'",
"REDIS_HOST": "Has been replaced with 'REDIS_CON' connection string",
@ -139,11 +143,16 @@ class Command(BaseCommand):
self.stdout.write("[1] checking expected env vars")
env = os.environ
for var in EXPECTED_ENV_VARS:
if not env.get(var):
message = f" 🗙 expected env var {var} not set\n {INST}"
self.stdout.write(self.style.ERROR(message))
sleep(60)
raise CommandError(message)
if var in env:
continue
if var in FILE_FALLBACK and f"{var}_FILE" in env:
continue
message = f" 🗙 expected env var {var} not set\n {INST}"
self.stdout.write(self.style.ERROR(message))
sleep(60)
raise CommandError(message)
message = " ✓ all expected env vars are set"
self.stdout.write(self.style.SUCCESS(message))

36
backend/config/management/commands/ta_index_channel_tabs.py
View File

@ -1,36 +0,0 @@
"""
migration for 0.5.4 to 0.5.5
index channel_tabs for subscribed channels
"""
import time
from channel.src.index import YoutubeChannel
from common.src.helper import get_channels
from django.core.management.base import BaseCommand
class Command(BaseCommand):
"""command"""
def handle(self, *args, **kwargs):
"""handle task"""
self.stdout.write("channel tags initial index")
channels = get_channels(subscribed_only=True, source=["channel_id"])
for es_channel in channels:
channel = YoutubeChannel(es_channel["channel_id"])
channel.get_from_es()
channel_name = channel.json_data["channel_name"]
channel_tabs = channel.get_channel_tabs()
channel.json_data["channel_tabs"] = channel_tabs
channel.upload_to_es()
channel.sync_to_videos()
self.stdout.write(
self.style.SUCCESS(
f" ✓ updated '{channel_name}' tabs: {channel_tabs}"
)
)
time.sleep(5)

301
backend/config/management/commands/ta_startup.py
View File

@ -12,10 +12,12 @@ from time import sleep
from appsettings.src.config import AppConfig, ReleaseVersion
from appsettings.src.index_setup import ElasticIndexWrap
from appsettings.src.snapshot import ElasticSnapshot
from channel.src.index import YoutubeChannel
from common.src.env_settings import EnvironmentSettings
from common.src.es_connect import ElasticWrap
from common.src.helper import clear_dl_cache
from common.src.es_connect import ElasticWrap, IndexPaginate
from common.src.helper import clear_dl_cache, get_channels
from common.src.ta_redis import RedisArchivist
from django.conf import settings
from django.core.management.base import BaseCommand, CommandError
from django.utils import dateformat
from django_celery_beat.models import CrontabSchedule, PeriodicTasks
@ -24,6 +26,7 @@ from task.src.config_schedule import ScheduleBuilder
from task.src.task_manager import TaskManager
from task.tasks import version_check
from video.src.constants import VideoTypeEnum
from video.src.index import YoutubeVideo
TOPIC = """
@ -37,8 +40,6 @@ TOPIC = """
class Command(BaseCommand):
"""command framework"""
# pylint: disable=no-member
def handle(self, *args, **options):
"""run all commands"""
self.stdout.write(TOPIC)
@ -53,10 +54,46 @@ class Command(BaseCommand):
self._update_schedule_tz()
self._init_app_config()
self._set_ta_startup_time()
self._mig_fix_download_channel_indexed()
if self.skip_migrations:
return
self._mig_add_default_playlist_sort()
self._mig_set_channel_tabs()
self._mig_set_video_channel_tabs()
self._mig_fix_playlist_description()
self._mig_fix_missing_stats()
self._mig_fix_channel_art_types()
self._mig_fix_channel_description()
self._mig_fix_video_description()
@property
def skip_migrations(self) -> bool:
"""
check if migrations should be skipped.
Experimental, might get replaced in the future.
"""
current_version = settings.TA_VERSION.rstrip("-unstable").upper()
env_var = f"TA_MIG_SKIP_{current_version}"
skipping = bool(os.environ.get(env_var))
self.stdout.write("[MIGRATION] check, experimental")
if skipping:
self.stdout.write(
self.style.SUCCESS(
f" {env_var} is set, skipping migration check"
)
)
else:
self.stdout.write(
self.style.SUCCESS(
" Running migrations. "
+ "If migrations have run for this release, "
+ f"you can set {env_var} to skip the check"
)
)
return skipping
def _make_folders(self):
"""make expected cache folders"""
@ -68,6 +105,7 @@ class Command(BaseCommand):
"import",
"playlists",
"videos",
"ytdlp",
]
cache_dir = EnvironmentSettings.CACHE_DIR
for folder in folders:
@ -242,6 +280,12 @@ class Command(BaseCommand):
self.style.SUCCESS(f" added new default: {new_default}")
)
cleared = AppConfig().clear_old_keys()
for removed_key in cleared:
self.stdout.write(
self.style.SUCCESS(f" removed old key: {removed_key}")
)
return
if status_code != 404:
@ -271,139 +315,192 @@ class Command(BaseCommand):
self.style.SUCCESS(f" ✓ set timestamp to {message}.")
)
def _mig_fix_download_channel_indexed(self) -> None:
"""migrate from v0.5.2 to 0.5.3, fix missing channel_indexed"""
self.stdout.write("[MIGRATION] fix incorrect video channel tags types")
path = "ta_download/_update_by_query"
data = {
"query": {
"bool": {
"must_not": [{"exists": {"field": "channel_indexed"}}]
}
},
"script": {
"source": "ctx._source.channel_indexed = false",
"lang": "painless",
},
}
response, status_code = ElasticWrap(path).post(data)
if status_code in [200, 201]:
updated = response.get("updated")
if updated:
self.stdout.write(
self.style.SUCCESS(f" ✓ fixed {updated} queued videos")
)
else:
self.stdout.write(
self.style.SUCCESS(" no queued videos to fix")
)
return
message = " 🗙 failed to fix video channel tags"
self.stdout.write(self.style.ERROR(message))
self.stdout.write(response)
sleep(60)
raise CommandError(message)
def _mig_add_default_playlist_sort(self) -> None:
"""migrate from 0.5.4 to 0.5.5 set default playlist sortorder"""
self.stdout.write("[MIGRATION] set default playlist sort order")
path = "ta_playlist/_update_by_query"
data = {
"query": {
self._run_migration(
index_name="ta_playlist",
desc="set default playlist sort order",
query={
"bool": {
"must_not": [{"exists": {"field": "playlist_sort_order"}}]
}
},
"script": {
script={
"source": "ctx._source.playlist_sort_order = 'top'",
"lang": "painless",
},
}
response, status_code = ElasticWrap(path).post(data)
if status_code in [200, 201]:
updated = response.get("updated")
if updated:
self.stdout.write(
self.style.SUCCESS(f" ✓ updated {updated} playlists")
)
else:
self.stdout.write(
self.style.SUCCESS(" no playlists need updating")
)
return
message = " 🗙 failed to set default playlist sort order"
self.stdout.write(self.style.ERROR(message))
self.stdout.write(response)
sleep(60)
raise CommandError(message)
)
def _mig_set_channel_tabs(self) -> None:
"""migrate from 0.5.4 to 0.5.5 set initial channel tabs"""
self.stdout.write("[MIGRATION] set default channel_tabs")
path = "ta_channel/_update_by_query"
tabs = VideoTypeEnum.values_known()
data = {
"query": {
self._run_migration(
index_name="ta_channel",
desc="set default channel_tabs in channel index",
query={
"bool": {"must_not": [{"exists": {"field": "channel_tabs"}}]}
},
"script": {
script={
"source": f"ctx._source.channel_tabs = {tabs}",
"lang": "painless",
},
}
response, status_code = ElasticWrap(path).post(data)
if status_code in [200, 201]:
updated = response.get("updated")
if updated:
self.stdout.write(
self.style.SUCCESS(f" ✓ updated {updated} channels")
)
else:
self.stdout.write(
self.style.SUCCESS(" no channels need updating")
)
return
message = " 🗙 failed to set default channel_tabs"
self.stdout.write(self.style.ERROR(message))
self.stdout.write(response)
sleep(60)
raise CommandError(message)
)
def _mig_set_video_channel_tabs(self) -> None:
"""migrate from 0.5.4 to 0.5.5 set initial video channel tabs"""
self.stdout.write("[MIGRATION] set default channel_tabs for videos")
path = "ta_video/_update_by_query"
tabs = VideoTypeEnum.values_known()
data = {
"query": {
self._run_migration(
index_name="ta_video",
desc="set default channel_tabs for videos",
query={
"bool": {
"must_not": [{"exists": {"field": "channel.channel_tabs"}}]
}
},
"script": {
script={
"source": f"ctx._source.channel.channel_tabs = {tabs}",
"lang": "painless",
},
}
)
def _mig_fix_playlist_description(self) -> None:
"""migrate from 0.5.8 to 0.5.9 fix playlist desc null data type"""
self._run_migration(
index_name="ta_playlist",
desc="fix playlist description data type",
query={"term": {"playlist_description": {"value": False}}},
script={
"source": "ctx._source.remove('playlist_description')",
"lang": "painless",
},
)
def _mig_fix_missing_stats(self) -> None:
"""migrate from 0.5.8 to 0.5.9, fix missing stats values"""
fields = [
"like_count",
"average_rating",
"view_count",
"dislike_count",
]
for field in fields:
self._run_migration(
index_name="ta_video",
desc=f"fix missing stats field {field}",
query={
"bool": {
"must_not": [{"exists": {"field": f"stats.{field}"}}]
}
},
script={
"source": f"ctx._source.stats.{field} = 0",
"lang": "painless",
},
)
def _mig_fix_channel_art_types(self) -> None:
"""migrate from 0.5.8 to 0.5.9, fix channel artwork types"""
fields = [
"channel_banner_url",
"channel_thumb_url",
"channel_tvart_url",
]
for field in fields:
self._run_migration(
index_name="ta_channel",
desc=f"fix missing data type for field {field}",
query={"term": {field: {"value": False}}},
script={
"source": f"ctx._source.remove('{field}')",
"lang": "painless",
},
)
source = f"""
if (ctx._source.containsKey('channel'))
{{ctx._source.channel.remove('{field}');}}
"""
self._run_migration(
index_name="ta_video",
desc=f"fix missing data type for field channel.{field}",
query={"term": {f"channel.{field}": {"value": False}}},
script={"source": source, "lang": "painless"},
)
def _mig_fix_channel_description(self) -> None:
"""migrate from 0.5.8 to 0.5.9, fix channel desc null value"""
desc = "fix channel description null value"
self.stdout.write(f"[MIGRATION] run {desc}")
channels = get_channels(
subscribed_only=False, source=["channel_description", "channel_id"]
)
counter = 0
for channel_response in channels:
if not channel_response.get("channel_description") == "":
continue
channel = YoutubeChannel(youtube_id=channel_response["channel_id"])
channel.get_from_es()
channel.json_data.pop("channel_description")
channel.upload_to_es()
channel.sync_to_videos()
counter += 1
if counter:
suc_msg = f" ✓ updated {counter} channels with videos"
self.stdout.write(self.style.SUCCESS(suc_msg))
else:
noop_msg = " no items needed updating"
self.stdout.write(self.style.SUCCESS(noop_msg))
def _mig_fix_video_description(self) -> None:
"""migrate from 0.5.8 to 0.5.9, fix video desc null value"""
desc = "fix video description null value"
self.stdout.write(f"[MIGRATION] run {desc}")
data = {"_source": ["youtube_id", "description"]}
videos = IndexPaginate("ta_video", data=data).get_results()
counter = 0
for video_response in videos:
if not video_response.get("description") == "":
continue
video = YoutubeVideo(youtube_id=video_response["youtube_id"])
video.get_from_es()
video.json_data.pop("description")
video.upload_to_es()
counter += 1
if counter:
suc_msg = f" ✓ updated {counter} videos"
self.stdout.write(self.style.SUCCESS(suc_msg))
else:
noop_msg = " no items needed updating"
self.stdout.write(self.style.SUCCESS(noop_msg))
def _run_migration(
self, index_name: str, desc: str, query: dict, script: dict
):
"""run migration"""
self.stdout.write(f"[MIGRATION] run {desc}")
path = f"{index_name}/_update_by_query?wait_for_completion=true"
data = {"query": query, "script": script}
response, status_code = ElasticWrap(path).post(data)
if status_code in [200, 201]:
updated = response.get("updated")
if updated:
self.stdout.write(
self.style.SUCCESS(f" ✓ updated {updated} videos")
)
suc_msg = f" ✓ updated {updated} docs in {index_name}"
self.stdout.write(self.style.SUCCESS(suc_msg))
# ensure index consistency
ElasticWrap(f"{index_name}/_refresh").post()
else:
self.stdout.write(
self.style.SUCCESS(" no videos need updating")
)
noop_msg = f" no items in {index_name} need updating"
self.stdout.write(self.style.SUCCESS(noop_msg))
return
message = " 🗙 failed to set default channel_tabs"
message = f" 🗙 failed to run {desc} on index {index_name}"
self.stdout.write(self.style.ERROR(message))
self.stdout.write(response)
sleep(60)

2
backend/config/settings.py
View File

@ -221,7 +221,7 @@ CORS_EXPOSE_HEADERS = ["X-Start-Timestamp"]
# TA application settings
TA_UPSTREAM = "https://github.com/tubearchivist/tubearchivist"
TA_VERSION = "v0.5.9-unstable"
TA_VERSION = "v0.5.10-unstable"
try:
TA_START = RedisArchivist().get_message_str("STARTTIMESTAMP")
except ValueError:

8
backend/download/serializers.py
View File

@ -10,19 +10,21 @@ from video.src.constants import VideoTypeEnum
class DownloadItemSerializer(serializers.Serializer):
"""serialize download item"""
auto_start = serializers.BooleanField()
auto_start = serializers.BooleanField(required=False)
channel_id = serializers.CharField()
channel_indexed = serializers.BooleanField()
channel_name = serializers.CharField()
duration = serializers.CharField()
message = serializers.CharField(required=False)
published = serializers.CharField(allow_null=True)
status = serializers.ChoiceField(choices=["pending", "ignore"])
status = serializers.ChoiceField(
choices=["pending", "ignore"], required=False
)
timestamp = serializers.IntegerField(allow_null=True)
title = serializers.CharField()
vid_thumb_url = serializers.CharField(allow_null=True)
vid_type = serializers.ChoiceField(choices=VideoTypeEnum.values())
youtube_id = serializers.CharField()
message = serializers.CharField(required=False)
_index = serializers.CharField(required=False)
_score = serializers.IntegerField(required=False)

47
backend/download/src/queue.py
View File

@ -20,6 +20,7 @@ from common.src.helper import (
rand_sleep,
)
from common.src.urlparser import ParsedURLType
from download.serializers import DownloadItemSerializer
from download.src.queue_interact import PendingInteract
from download.src.thumbnails import ThumbManager
from playlist.src.index import YoutubePlaylist
@ -368,17 +369,23 @@ class PendingList(PendingIndex):
return None
to_add = {
"youtube_id": video_data["id"],
"title": video_data["title"],
"vid_thumb_url": self._extract_thumb(video_data),
"channel_id": video_data["channel_id"],
"channel_indexed": video_data["channel_id"] in self.all_channels,
"channel_name": video_data["channel"],
"duration": get_duration_str(video_data.get("duration", 0)),
"published": self._extract_published(video_data),
"timestamp": int(datetime.now().timestamp()),
"title": video_data["title"],
"vid_thumb_url": self._extract_thumb(video_data),
"vid_type": self._extract_vid_type(video_data),
"channel_name": video_data["channel"],
"channel_id": video_data["channel_id"],
"channel_indexed": video_data["channel_id"] in self.all_channels,
"youtube_id": video_data["id"],
}
serializer = DownloadItemSerializer(data=to_add)
is_valid = serializer.is_valid()
if not is_valid:
print(f"{youtube_id}: serializer failed: {serializer.errors}")
self._notify_fail(403, youtube_id)
return None
return to_add
@ -393,18 +400,34 @@ class PendingList(PendingIndex):
return None
@staticmethod
def _extract_published(video_data) -> str | int | None:
def _extract_published(video_data) -> int | None:
"""build published date or timestamp"""
timestamp = video_data.get("timestamp")
if timestamp:
if timestamp and isinstance(timestamp, int):
return timestamp
if timestamp and isinstance(timestamp, float):
return int(timestamp)
if timestamp and isinstance(timestamp, str):
try:
# scientific string
return int(float(timestamp))
except (TypeError, ValueError):
pass
upload_date = video_data.get("upload_date")
if upload_date:
upload_date_time = datetime.strptime(upload_date, "%Y%m%d")
return upload_date_time.replace(
tzinfo=ZoneInfo(EnvironmentSettings.TZ)
).timestamp()
try:
upload_date_time = datetime.strptime(upload_date, "%Y%m%d")
except ValueError:
youtube_id = video_data["id"]
print(f"{youtube_id}: published date extraction failed.")
return None
tz = ZoneInfo(EnvironmentSettings.TZ)
timestamp = int(upload_date_time.replace(tzinfo=tz).timestamp())
return timestamp
return None

108
backend/download/src/thumbnails.py
View File

@ -4,9 +4,7 @@ functionality:
- check for missing thumbnails
"""
import base64
import os
from io import BytesIO
from time import sleep
import requests
@ -14,7 +12,7 @@ from common.src.env_settings import EnvironmentSettings
from common.src.es_connect import ElasticWrap, IndexPaginate
from common.src.helper import is_missing
from mutagen.mp4 import MP4, MP4Cover
from PIL import Image, ImageFile, ImageFilter, UnidentifiedImageError
from PIL import Image, ImageFile, UnidentifiedImageError
ImageFile.LOAD_TRUNCATED_IMAGES = True
@ -23,6 +21,7 @@ class ThumbManagerBase:
"""base class for thumbnail management"""
CACHE_DIR = EnvironmentSettings.CACHE_DIR
MEDIA_DIR = EnvironmentSettings.MEDIA_DIR
VIDEO_DIR = os.path.join(CACHE_DIR, "videos")
CHANNEL_DIR = os.path.join(CACHE_DIR, "channels")
PLAYLIST_DIR = os.path.join(CACHE_DIR, "playlists")
@ -217,6 +216,63 @@ class ThumbManager(ThumbManagerBase):
img_raw = img_raw.resize((336, 189))
img_raw.convert("RGB").save(thumb_path)
def embed_video_art(self, json_data: dict):
"""embed video artwork"""
file_path = os.path.join(self.MEDIA_DIR, json_data["media_url"])
if not os.path.exists(file_path):
print(f"{self.item_id}: skip art embed, file not found")
return
video = MP4(file_path)
thumb_path = self.vid_thumb_path(absolute=True)
if os.path.exists(thumb_path):
with open(thumb_path, "rb") as f:
cover_data = f.read()
video["covr"] = [
MP4Cover(cover_data, imageformat=MP4Cover.FORMAT_JPEG)
]
channel_id = json_data["channel"]["channel_id"]
banner_path = os.path.join(
self.CHANNEL_DIR, f"{channel_id}_banner.jpg"
)
self._embed_art_item(video, "channel_banner", art_path=banner_path)
channel_icon_path = os.path.join(
self.CHANNEL_DIR, f"{channel_id}_thumb.jpg"
)
self._embed_art_item(video, "channel_icon", art_path=channel_icon_path)
channel_tv_path = os.path.join(
self.CHANNEL_DIR, f"{channel_id}_tvart.jpg"
)
self._embed_art_item(video, "channel_tv", art_path=channel_tv_path)
playlist_ids = json_data.get("playlist", [])
for plalyist_id in playlist_ids:
playlist_path = os.path.join(
self.PLAYLIST_DIR, f"{plalyist_id}.jpg"
)
self._embed_art_item(
video, f"playlist_{plalyist_id}", art_path=playlist_path
)
video.save()
def _embed_art_item(self, video, key, art_path):
"""embed single item"""
if not os.path.exists(art_path):
return
with open(art_path, "rb") as f:
art_data = f.read()
video[f"----:com.tubearchivist:{key}"] = [
MP4Cover(art_data, imageformat=MP4Cover.FORMAT_JPEG)
]
def delete_video_thumb(self):
"""delete video thumbnail if exists"""
thumb_path = self.vid_thumb_path()
@ -228,10 +284,13 @@ class ThumbManager(ThumbManagerBase):
"""delete all artwork of channel"""
thumb = os.path.join(self.CHANNEL_DIR, f"{self.item_id}_thumb.jpg")
banner = os.path.join(self.CHANNEL_DIR, f"{self.item_id}_banner.jpg")
tv = os.path.join(self.CHANNEL_DIR, f"{self.item_id}_tvart.jpg")
if os.path.exists(thumb):
os.remove(thumb)
if os.path.exists(banner):
os.remove(banner)
if os.path.exists(tv):
os.remove(tv)
def delete_playlist_thumb(self):
"""delete playlist thumbnail"""
@ -239,20 +298,6 @@ class ThumbManager(ThumbManagerBase):
if os.path.exists(thumb_path):
os.remove(thumb_path)
def get_vid_base64_blur(self):
"""return base64 encoded placeholder"""
file_path = os.path.join(self.CACHE_DIR, self.vid_thumb_path())
img_raw = Image.open(file_path)
img_raw.thumbnail((img_raw.width // 20, img_raw.height // 20))
img_blur = img_raw.filter(ImageFilter.BLUR)
buffer = BytesIO()
img_blur.save(buffer, format="JPEG")
img_data = buffer.getvalue()
img_base64 = base64.b64encode(img_data).decode()
data_url = f"data:image/jpg;base64,{img_base64}"
return data_url
class ValidatorCallback:
"""handle callback validate thumbnails page by page"""
@ -283,9 +328,9 @@ class ValidatorCallback:
"""check if all channel artwork is there"""
for channel in self.source:
urls = (
channel["_source"]["channel_thumb_url"],
channel["_source"]["channel_banner_url"],
channel["_source"].get("channel_tvart_url", False),
channel["_source"].get("channel_thumb_url"),
channel["_source"].get("channel_banner_url"),
channel["_source"].get("channel_tvart_url"),
)
handler = ThumbManager(channel["_source"]["channel_id"])
handler.download_channel_art(urls, skip_existing=True)
@ -458,12 +503,12 @@ class ThumbFilesystem:
"""entry point"""
data = {
"query": {"match_all": {}},
"_source": ["media_url", "youtube_id"],
"_source": ["media_url", "youtube_id", "channel.channel_id"],
}
paginate = IndexPaginate(
index_name=self.INDEX_NAME,
data=data,
size=200,
size=100,
callback=EmbedCallback,
task=self.task,
total=self._get_total(),
@ -481,9 +526,7 @@ class ThumbFilesystem:
class EmbedCallback:
"""callback class to embed thumbnails"""
CACHE_DIR = EnvironmentSettings.CACHE_DIR
MEDIA_DIR = EnvironmentSettings.MEDIA_DIR
FORMAT = MP4Cover.FORMAT_JPEG
def __init__(self, source, index_name, counter=0):
self.source = source
@ -494,19 +537,4 @@ class EmbedCallback:
"""run embed"""
for video in self.source:
video_id = video["_source"]["youtube_id"]
media_url = os.path.join(
self.MEDIA_DIR, video["_source"]["media_url"]
)
thumb_path = os.path.join(
self.CACHE_DIR, ThumbManager(video_id).vid_thumb_path()
)
if os.path.exists(thumb_path):
self.embed(media_url, thumb_path)
def embed(self, media_url, thumb_path):
"""embed thumb in single media file"""
video = MP4(media_url)
with open(thumb_path, "rb") as f:
video["covr"] = [MP4Cover(f.read(), imageformat=self.FORMAT)]
video.save()
ThumbManager(video_id).embed_video_art(video["_source"])

105
backend/download/src/yt_dlp_base.py
View File

@ -7,9 +7,12 @@ functionality:
from datetime import datetime
from http import cookiejar
from io import StringIO
from os import path
import yt_dlp
from appsettings.src.config import AppConfig
from common.src.env_settings import EnvironmentSettings
from common.src.helper import deep_merge, rand_sleep
from common.src.ta_redis import RedisArchivist
from django.conf import settings
@ -17,12 +20,21 @@ from django.conf import settings
class YtWrap:
"""wrap calls to yt"""
BOT_MESSAGES = [
"not a bot",
]
BOT_ERROR_LOG = "YouTube bot detection, abort!"
OBS_BASE = {
"default_search": "ytsearch",
"quiet": True,
"socket_timeout": 10,
"extractor_retries": 3,
"retries": 10,
"cachedir": path.abspath(
path.join(EnvironmentSettings.CACHE_DIR, "ytdlp")
),
"plugin_dirs": [],
}
def __init__(self, obs_request, config=False):
@ -33,10 +45,10 @@ class YtWrap:
def build_obs(self):
"""build yt-dlp obs"""
self.obs = self.OBS_BASE.copy()
self.obs.update(self.obs_request)
deep_merge(self.obs, self.obs_request)
if self.config:
self._add_cookie()
self._add_potoken()
self._add_potoken_url()
if getattr(settings, "DEBUG", False):
del self.obs["quiet"]
@ -46,22 +58,29 @@ class YtWrap:
"""add cookie if enabled"""
if self.config["downloads"]["cookie_import"]:
cookie_io = CookieHandler(self.config).get()
self.obs["cookiefile"] = cookie_io
else:
cookie_io = CookieHandler(self.config).get("cookie_temp")
def _add_potoken(self):
"""add potoken if enabled"""
if self.config["downloads"].get("potoken"):
potoken = POTokenHandler(self.config).get()
self.obs.update(
self.obs["cookiefile"] = cookie_io
def _add_potoken_url(self):
"""add bgutils token url"""
if pot_provider_url := self.config["downloads"].get(
"pot_provider_url"
):
deep_merge(
self.obs,
{
"extractor_args": {
"youtube": {
"po_token": [potoken],
"player-client": ["mweb", "default"],
},
"youtubepot-bgutilhttp": {
"base_url": [pot_provider_url]
}
}
}
},
)
if EnvironmentSettings.APP_DIR == "/app":
# container internal only
self.obs["plugin_dirs"].append("/opt/yt_plugins/bgutil")
def download(self, url):
"""make download request"""
@ -73,6 +92,10 @@ class YtWrap:
print(f"{url}: failed to download with message {err}")
if "Temporary failure in name resolution" in str(err):
raise ConnectionError("lost the internet, abort!") from err
if any(m in str(err) for m in self.BOT_MESSAGES):
print(self.BOT_ERROR_LOG)
rand_sleep(self.config)
raise ConnectionError(self.BOT_ERROR_LOG) from err
return False, str(err)
@ -101,6 +124,10 @@ class YtWrap:
print(f"{url}: failed to get info from youtube: {err}")
if "Temporary failure in name resolution" in str(err):
raise ConnectionError("lost the internet, abort!") from err
if any(m in str(err) for m in self.BOT_MESSAGES):
print(self.BOT_ERROR_LOG)
rand_sleep(self.config)
raise ConnectionError(self.BOT_ERROR_LOG) from err
return None, str(err)
@ -111,26 +138,40 @@ class YtWrap:
def _validate_cookie(self):
"""check cookie and write it back for next use"""
if not self.obs.get("cookiefile"):
# empty in tests
return
new_cookie = self.obs["cookiefile"].read()
old_cookie = RedisArchivist().get_message_str("cookie")
self.obs["cookiefile"].seek(0)
new_cookie = self.obs["cookiefile"].read().strip("\x00")
if self.config["downloads"]["cookie_import"]:
cookie_key = "cookie"
expire = False
else:
cookie_key = "cookie_temp"
expire = 60 * 30 # 30 min
old_cookie = RedisArchivist().get_message_str(cookie_key)
if new_cookie and old_cookie != new_cookie:
print("refreshed stored cookie")
RedisArchivist().set_message("cookie", new_cookie, save=True)
print(f"refreshed stored {cookie_key}")
RedisArchivist().set_message(
cookie_key, new_cookie, expire=expire, save=True
)
class CookieHandler:
"""handle youtube cookie for yt-dlp"""
COOKIE_EMPTY = "# Netscape HTTP Cookie File\n"
def __init__(self, config):
self.cookie_io = False
self.config = config
def get(self):
def get(self, message_str: str = "cookie"):
"""get cookie io stream"""
cookie = RedisArchivist().get_message_str("cookie")
self.cookie_io = StringIO(cookie)
cookie = RedisArchivist().get_message_str(message_str)
self.cookie_io = StringIO(cookie or self.COOKIE_EMPTY)
return self.cookie_io
def set_cookie(self, cookie):
@ -196,27 +237,3 @@ class CookieHandler:
"validated_str": now.strftime("%Y-%m-%d %H:%M"),
}
RedisArchivist().set_message("cookie:valid", message, expire=3600)
class POTokenHandler:
"""handle po token"""
REDIS_KEY = "potoken"
def __init__(self, config):
self.config = config
def get(self) -> str | None:
"""get PO token"""
potoken = RedisArchivist().get_message_str(self.REDIS_KEY)
return potoken
def set_token(self, new_token: str) -> None:
"""set new PO token"""
RedisArchivist().set_message(self.REDIS_KEY, new_token)
AppConfig().update_config({"downloads": {"potoken": True}})
def revoke_token(self) -> None:
"""revoke token"""
RedisArchivist().del_message(self.REDIS_KEY)
AppConfig().update_config({"downloads": {"potoken": False}})

14
backend/download/src/yt_dlp_handler.py
View File

@ -196,15 +196,6 @@ class VideoDownloader(DownloaderBase):
}
)
if self.config["downloads"]["add_thumbnail"]:
postprocessors.append(
{
"key": "EmbedThumbnail",
"already_have_thumbnail": True,
}
)
self.obs["writethumbnail"] = True
self.obs["postprocessors"] = postprocessors
def _set_overwrites(self, obs: dict, channel_id: str) -> None:
@ -326,6 +317,9 @@ class DownloadPostProcess(DownloaderBase):
for channel_id, value in self.channel_overwrites.items():
if "autodelete_days" in value:
autodelete_days = value.get("autodelete_days")
if autodelete_days is None:
continue
print(f"{channel_id}: delete older than {autodelete_days}d")
now_lte = str(self.now - autodelete_days * 24 * 60 * 60)
must_list = [
@ -435,7 +429,7 @@ class DownloadPostProcess(DownloaderBase):
channel = YoutubeChannel(channel_id)
channel.get_from_es()
overwrites = channel.get_overwrites()
if "index_playlists" in overwrites:
if overwrites.get("index_playlists"):
channel.get_all_playlists()
to_add = [i[0] for i in channel.all_playlists]
RedisQueue(self.PLAYLIST_QUEUE).add_list(to_add)

12
backend/download/tests/test_src/test_pending_list.py
View File

@ -5,6 +5,18 @@ from datetime import datetime, timezone
from download.src.queue import PendingList
def test_returns_scientific_timestamp_if_present():
video_data = {"timestamp": 1.5135732e9}
result = PendingList._extract_published(video_data)
assert result == 1513573200
def test_returns_scientific_timestamp_string_if_present():
video_data = {"timestamp": "1.5135732e9"}
result = PendingList._extract_published(video_data)
assert result == 1513573200
def test_returns_timestamp_if_present():
video_data = {"timestamp": 1508457600}
result = PendingList._extract_published(video_data)

6
backend/playlist/serializers.py
View File

@ -11,7 +11,7 @@ class PlaylistEntrySerializer(serializers.Serializer):
youtube_id = serializers.CharField()
title = serializers.CharField()
uploader = serializers.CharField()
uploader = serializers.CharField(allow_null=True)
idx = serializers.IntegerField()
downloaded = serializers.BooleanField()
@ -22,7 +22,9 @@ class PlaylistSerializer(serializers.Serializer):
playlist_active = serializers.BooleanField()
playlist_channel = serializers.CharField()
playlist_channel_id = serializers.CharField()
playlist_description = serializers.CharField()
playlist_description = serializers.CharField(
allow_null=True, required=False
)
playlist_entries = PlaylistEntrySerializer(many=True)
playlist_id = serializers.CharField()
playlist_last_refresh = serializers.CharField()

17
backend/playlist/src/index.py
View File

@ -81,10 +81,13 @@ class YoutubePlaylist(YouTubeItem):
"playlist_channel": self.youtube_meta["channel"],
"playlist_channel_id": self.youtube_meta["channel_id"],
"playlist_thumbnail": playlist_thumbnail,
"playlist_description": self.youtube_meta["description"] or False,
"playlist_last_refresh": int(datetime.now().timestamp()),
"playlist_type": "regular",
}
if self.youtube_meta.get("description"):
self.json_data["playlist_description"] = self.youtube_meta[
"description"
]
def _ensure_channel(self):
"""make sure channel is indexed"""
@ -311,7 +314,10 @@ class YoutubePlaylist(YouTubeItem):
self.del_in_es()
def is_custom_playlist(self):
self.get_from_es()
"""check if is custom playlist"""
if not self.json_data:
self.get_from_es()
return self.json_data["playlist_type"] == "custom"
def delete_videos_metadata(self, channel_id=None):
@ -389,13 +395,16 @@ class YoutubePlaylist(YouTubeItem):
return True
def remove_playlist_from_video(self, video_id):
"""remove playlist id from video metadata"""
video = ta_video.YoutubeVideo(video_id)
video.get_from_es()
if video.json_data is not None and "playlist" in video.json_data:
video.json_data["playlist"].remove(self.youtube_id)
video.upload_to_es()
if self.youtube_id in video.json_data["playlist"]:
video.json_data["playlist"].remove(self.youtube_id)
video.upload_to_es()
def move_video(self, video_id, action, hide_watched=False):
"""move video within custion playlist based on action"""
self.get_from_es()
video_index = self.get_video_index(video_id)
playlist = self.json_data["playlist_entries"]

2
backend/requirements.plugins.txt Normal file
View File

@ -0,0 +1,2 @@
# install plugins in separate folder for runtime whitelisting
bgutil-ytdlp-pot-provider==1.3.1

25
backend/requirements.txt
View File

@ -1,14 +1,15 @@
apprise==1.9.5
celery==5.5.3
django-auth-ldap==5.2.0
django-celery-beat==2.8.1
apprise==1.9.9
celery==5.6.3
deepdiff==8.6.2
django-auth-ldap==5.3.0
django-celery-beat==2.9.0
django-cors-headers==4.9.0
Django==5.2.8
djangorestframework==3.16.1
drf-spectacular==0.29.0
Pillow==12.0.0
redis==7.0.1
requests==2.32.5
Django==6.0.3
djangorestframework==3.17.1
drf-spectacular==0.28.0 # rc:ignore
Pillow==12.1.1
redis==7.4.0
requests==2.33.0
ryd-client==0.0.6
uvicorn==0.38.0
yt-dlp[default]==2025.11.12
uvicorn==0.42.0
yt-dlp[default]==2026.3.17

12
backend/task/src/task_config.py
View File

@ -48,7 +48,7 @@ CHECK_REINDEX: TaskItemConfig = {
MANUAL_IMPORT: TaskItemConfig = {
"title": "Manual video import",
"group": "setting:import",
"api_start": True,
"api_start": False,
"api_stop": False,
}
@ -69,7 +69,7 @@ RESTORE_BACKUP: TaskItemConfig = {
RESCAN_FILESYSTEM: TaskItemConfig = {
"title": "Rescan your Filesystem",
"group": "setting:filesystemscan",
"api_start": True,
"api_start": False,
"api_stop": False,
}
@ -80,13 +80,6 @@ THUMBNAIL_CHECK: TaskItemConfig = {
"api_stop": False,
}
RESYNC_THUMBS: TaskItemConfig = {
"title": "Sync Thumbnails to Media Files",
"group": "setting:thumbnailsync",
"api_start": True,
"api_stop": False,
}
RESYNC_METADATA: TaskItemConfig = {
"title": "Sync Metadata to Media Files",
"group": "setting:thumbnailsync",
@ -125,7 +118,6 @@ TASK_CONFIG: dict[str, TaskItemConfig] = {
"restore_backup": RESTORE_BACKUP,
"rescan_filesystem": RESCAN_FILESYSTEM,
"thumbnail_check": THUMBNAIL_CHECK,
"resync_thumbs": RESYNC_THUMBS,
"resync_metadata": RESYNC_METADATA,
"index_playlists": INDEX_PLAYLISTS,
"subscribe_to": SUBSCRIBE_TO,

4
backend/task/src/task_manager.py
View File

@ -84,9 +84,9 @@ class TaskManager:
class TaskCommand:
"""run commands on task"""
def start(self, task_name):
def start(self, task_name, kwargs: dict | None = None):
"""start task by task_name, only pass task that don't take args"""
task = celery_app.tasks.get(task_name).delay()
task = celery_app.tasks.get(task_name).delay(**(kwargs or {}))
message = {
"task_id": task.id,
"status": task.status,

27
backend/task/tasks.py
View File

@ -19,7 +19,7 @@ from common.src.ta_redis import RedisArchivist
from common.src.urlparser import ParsedURLType, Parser
from download.src.queue import PendingList
from download.src.subscriptions import SubscriptionHandler, SubscriptionScanner
from download.src.thumbnails import ThumbFilesystem, ThumbValidator
from download.src.thumbnails import ThumbValidator
from download.src.yt_dlp_handler import VideoDownloader
from task.src.notify import Notifications
from task.src.task_config import TASK_CONFIG
@ -216,7 +216,7 @@ def check_reindex(self, data=False, extract_videos=False):
@shared_task(bind=True, name="manual_import", base=BaseTask)
def run_manual_import(self):
def manual_import(self, ignore_error, prefer_local):
"""called from settings page, to go through import folder"""
manager = TaskManager()
if manager.is_pending(self):
@ -225,7 +225,9 @@ def run_manual_import(self):
return
manager.init(self)
ImportFolderScanner(task=self).scan()
ImportFolderScanner(
task=self, ignore_error=ignore_error, prefer_local=prefer_local
).scan()
@shared_task(bind=True, name="run_backup", base=BaseTask)
@ -260,7 +262,7 @@ def run_restore_backup(self, filename):
@shared_task(bind=True, name="rescan_filesystem", base=BaseTask)
def rescan_filesystem(self):
def rescan_filesystem(self, ignore_error, prefer_local):
"""check the media folder for mismatches"""
manager = TaskManager()
if manager.is_pending(self):
@ -269,7 +271,9 @@ def rescan_filesystem(self):
return
manager.init(self)
handler = Scanner(task=self)
handler = Scanner(
task=self, ignore_error=ignore_error, prefer_local=prefer_local
)
handler.scan()
handler.apply()
thumbnail_check.delay()
@ -290,19 +294,6 @@ def thumbnail_check(self):
thumbnail.clean_up()
@shared_task(bind=True, name="resync_thumbs", base=BaseTask)
def re_sync_thumbs(self):
"""sync thumbnails to mediafiles"""
manager = TaskManager()
if manager.is_pending(self):
print(f"[task][{self.name}] thumb re-embed is already running")
self.send_progress(["Thumbnail re-embed is already running."])
return
manager.init(self)
ThumbFilesystem(task=self).embed()
@shared_task(bind=True, name="resync_metadata", base=BaseTask)
def re_sync_metadata(self):
"""resync metadata to media files"""

8
backend/user/migrations/0001_initial.py
View File

@ -39,7 +39,9 @@ class Migration(migrations.Migration):
"is_superuser",
models.BooleanField(
default=False,
help_text="Designates that this user has all permissions without explicitly assigning them.",
help_text=(
"Designates that this user has all permissions without explicitly assigning them."
),
verbose_name="superuser status",
),
),
@ -49,7 +51,9 @@ class Migration(migrations.Migration):
"groups",
models.ManyToManyField(
blank=True,
help_text="The groups this user belongs to. A user will get all permissions granted to each of their groups.",
help_text=(
"The groups this user belongs to. A user will get all permissions granted to each of their groups."
),
related_name="user_set",
related_query_name="user",
to="auth.group",

2
backend/user/src/user_config.py
View File

@ -40,7 +40,7 @@ class UserConfig:
_DEFAULT_USER_SETTINGS = UserConfigType(
stylesheet="dark.css",
page_size=12,
page_size=25,
sort_by="published",
sort_order="desc",
view_style_home="grid",

104
backend/video/serializers.py
View File

@ -4,6 +4,7 @@
from channel.serializers import ChannelSerializer
from common.serializers import PaginationSerializer
from drf_spectacular.utils import extend_schema_field
from rest_framework import serializers
from video.src.constants import OrderEnum, SortEnum, VideoTypeEnum, WatchedEnum
@ -15,6 +16,7 @@ class PlayerSerializer(serializers.Serializer):
watched_date = serializers.IntegerField(required=False)
duration = serializers.IntegerField()
duration_str = serializers.CharField()
progress = serializers.FloatField(required=False)
position = serializers.FloatField(required=False)
@ -43,32 +45,49 @@ class SponsorBlockSerializer(serializers.Serializer):
class StatsSerializer(serializers.Serializer):
"""serialize stats"""
like_count = serializers.IntegerField(required=False)
average_rating = serializers.FloatField(required=False)
view_count = serializers.IntegerField(required=False)
dislike_count = serializers.IntegerField(required=False)
like_count = serializers.IntegerField()
average_rating = serializers.FloatField()
view_count = serializers.IntegerField()
dislike_count = serializers.IntegerField()
class StreamItemSerializer(serializers.Serializer):
"""serialize stream item"""
index = serializers.IntegerField()
codec = serializers.CharField()
bitrate = serializers.IntegerField()
codec = serializers.CharField()
height = serializers.IntegerField(required=False)
index = serializers.IntegerField()
type = serializers.ChoiceField(choices=["video", "audio"])
width = serializers.IntegerField(required=False)
height = serializers.IntegerField(required=False)
class SubtitleFragmentSerializer(serializers.Serializer):
"""serialize subtitle fragment"""
subtitle_channel = serializers.CharField()
subtitle_channel_id = serializers.CharField()
subtitle_end = serializers.CharField()
subtitle_fragment_id = serializers.CharField()
subtitle_index = serializers.IntegerField()
subtitle_lang = serializers.CharField()
subtitle_last_refresh = serializers.IntegerField()
subtitle_line = serializers.CharField()
subtitle_source = serializers.ChoiceField(choices=["user", "auto"])
subtitle_start = serializers.CharField()
title = serializers.CharField()
youtube_id = serializers.CharField()
class SubtitleItemSerializer(serializers.Serializer):
"""serialize subtitle item"""
ext = serializers.ChoiceField(choices=["json3"])
name = serializers.CharField()
source = serializers.ChoiceField(choices=["user", "auto"])
ext = serializers.ChoiceField(choices=["json3", "vtt"])
lang = serializers.CharField()
media_url = serializers.CharField()
url = serializers.URLField()
name = serializers.CharField()
source = serializers.ChoiceField(choices=["user", "auto"])
url = serializers.URLField(allow_null=True)
class VideoSerializer(serializers.Serializer):
@ -76,19 +95,21 @@ class VideoSerializer(serializers.Serializer):
active = serializers.BooleanField()
category = serializers.ListField(child=serializers.CharField())
channel = ChannelSerializer()
comment_count = serializers.IntegerField(allow_null=True)
channel = ChannelSerializer(required=False)
comment_count = serializers.IntegerField(allow_null=True, required=False)
date_downloaded = serializers.IntegerField()
description = serializers.CharField()
description = serializers.CharField(allow_null=True, required=False)
media_size = serializers.IntegerField()
media_url = serializers.CharField()
player = PlayerSerializer()
playlist = serializers.ListField(child=serializers.CharField())
playlist = serializers.ListField(
child=serializers.CharField(), required=False
)
published = serializers.CharField()
sponsorblock = SponsorBlockSerializer(allow_null=True)
sponsorblock = SponsorBlockSerializer(allow_null=True, required=False)
stats = StatsSerializer()
streams = StreamItemSerializer(many=True)
subtitles = SubtitleItemSerializer(many=True)
subtitles = SubtitleItemSerializer(many=True, required=False)
tags = serializers.ListField(child=serializers.CharField())
title = serializers.CharField()
vid_last_refresh = serializers.CharField()
@ -123,37 +144,38 @@ class VideoListQuerySerializer(serializers.Serializer):
height = serializers.IntegerField(required=False)
class CommentThreadItemSerializer(serializers.Serializer):
"""serialize comment thread item"""
comment_id = serializers.CharField()
comment_text = serializers.CharField()
comment_timestamp = serializers.IntegerField()
comment_time_text = serializers.CharField()
comment_likecount = serializers.IntegerField()
comment_is_favorited = serializers.BooleanField()
comment_author = serializers.CharField()
comment_author_id = serializers.CharField()
comment_author_thumbnail = serializers.URLField()
comment_author_is_uploader = serializers.BooleanField()
comment_parent = serializers.CharField()
class CommentItemSerializer(serializers.Serializer):
"""serialize comment item"""
comment_id = serializers.CharField()
comment_text = serializers.CharField()
comment_timestamp = serializers.IntegerField()
comment_time_text = serializers.CharField()
comment_likecount = serializers.IntegerField()
comment_is_favorited = serializers.BooleanField()
comment_author = serializers.CharField()
comment_author_id = serializers.CharField()
comment_author_thumbnail = serializers.URLField()
comment_author_is_uploader = serializers.BooleanField()
comment_author_thumbnail = serializers.URLField()
comment_id = serializers.CharField()
comment_is_favorited = serializers.BooleanField()
comment_likecount = serializers.IntegerField()
comment_parent = serializers.CharField()
comment_replies = CommentThreadItemSerializer(many=True)
comment_text = serializers.CharField()
comment_time_text = serializers.CharField()
comment_timestamp = serializers.IntegerField()
comment_replies = serializers.SerializerMethodField()
@extend_schema_field(serializers.ListField())
def get_comment_replies(self, obj):
"""recursive replies"""
return CommentItemSerializer(
obj.get("comment_replies", []), many=True
).data
class CommentsSerializer(serializers.Serializer):
"""serialize comments as indexed"""
comment_channel_id = serializers.CharField()
comment_comments = CommentItemSerializer(many=True)
comment_last_refresh = serializers.IntegerField()
youtube_id = serializers.CharField()
class PlaylistNavMetaSerializer(serializers.Serializer):

33
backend/video/src/comments.py
View File

@ -25,7 +25,7 @@ class Comments:
self.is_activated = False
self.comments_format = False
def build_json(self):
def build_json(self, upload: bool = False):
"""build json document for es"""
print(f"{self.youtube_id}: get comments")
self.check_config()
@ -39,11 +39,13 @@ class Comments:
self.format_comments(comments_raw)
self.json_data = {
"youtube_id": self.youtube_id,
"comment_last_refresh": int(datetime.now().timestamp()),
"comment_channel_id": channel_id,
"comment_comments": self.comments_format,
"comment_last_refresh": int(datetime.now().timestamp()),
"youtube_id": self.youtube_id,
}
if upload:
self.upload_comments()
def check_config(self):
"""read config if not attached"""
@ -122,34 +124,33 @@ class Comments:
if not comment.get("author"):
comment["author"] = comment.get("author_id", "Unknown")
is_uploader = comment.get("author_is_uploader", False)
cleaned_comment = {
"comment_id": comment["id"],
"comment_text": comment["text"].replace("\xa0", ""),
"comment_timestamp": comment["timestamp"],
"comment_time_text": time_text,
"comment_likecount": comment.get("like_count", None),
"comment_is_favorited": comment.get("is_favorited", False),
"comment_author": comment["author"],
"comment_author_id": comment["author_id"],
"comment_author_is_uploader": is_uploader,
"comment_author_thumbnail": comment["author_thumbnail"],
"comment_author_is_uploader": comment.get(
"author_is_uploader", False
),
"comment_id": comment["id"],
"comment_is_favorited": comment.get("is_favorited", False),
"comment_likecount": comment.get("like_count", None),
"comment_parent": comment["parent"],
"comment_text": comment["text"].replace("\xa0", ""),
"comment_time_text": time_text,
"comment_timestamp": comment["timestamp"],
}
return cleaned_comment
def upload_comments(self):
"""upload comments to es"""
if not self.is_activated:
return
print(f"{self.youtube_id}: upload comments")
_, _ = ElasticWrap(self.es_path).put(self.json_data)
vid_path = f"ta_video/_update/{self.youtube_id}"
data = {"doc": {"comment_count": len(self.comments_format)}}
data = {
"doc": {"comment_count": len(self.json_data["comment_comments"])}
}
_, _ = ElasticWrap(vid_path).post(data=data)
def delete_comments(self):

99
backend/video/src/index.py
View File

@ -15,7 +15,7 @@ from common.src.helper import get_duration_sec, get_duration_str, randomizor
from common.src.index_generic import YouTubeItem
from django.conf import settings
from download.src.thumbnails import ThumbManager
from mutagen.mp4 import MP4
from mutagen.mp4 import MP4, MP4MetadataError
from playlist.src import index as ta_playlist
from ryd_client import ryd_client
from user.src.user_config import UserConfig
@ -198,31 +198,40 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
def process_youtube_meta(self):
"""extract relevant fields from youtube"""
self._validate_id()
# extract
self.channel_id = self.youtube_meta["channel_id"]
last_refresh = int(datetime.now().timestamp())
# build json_data basics
self.json_data = {
"title": self.youtube_meta["title"],
"description": self.youtube_meta.get("description", ""),
"category": self.youtube_meta.get("categories", []),
"vid_thumb_url": self.youtube_meta["thumbnail"],
"tags": self.youtube_meta.get("tags", []),
"published": self._build_published(),
"vid_last_refresh": last_refresh,
"date_downloaded": last_refresh,
"youtube_id": self.youtube_id,
# Using .value to make json encodable
"vid_type": self.video_type.value,
"active": True,
"category": self.youtube_meta.get("categories", []),
"date_downloaded": last_refresh,
"published": self._build_published(),
"tags": self.youtube_meta.get("tags", []),
"title": self.youtube_meta["title"],
"vid_last_refresh": last_refresh,
"vid_thumb_url": self.youtube_meta["thumbnail"],
"vid_type": self.video_type.value,
"youtube_id": self.youtube_id,
}
def _build_published(self):
if description := self.youtube_meta.get("description"):
self.json_data["description"] = description
def _build_published(self) -> int | str:
"""build published date or timestamp"""
timestamp = self.youtube_meta.get("timestamp")
if timestamp:
if timestamp and isinstance(timestamp, int):
return timestamp
if timestamp and isinstance(timestamp, float):
return int(timestamp)
if timestamp and isinstance(timestamp, str):
try:
# scientific string
return int(float(timestamp))
except (TypeError, ValueError):
pass
upload_date = self.youtube_meta["upload_date"]
if not upload_date:
raise ValueError(
@ -255,10 +264,10 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
def _add_stats(self):
"""add stats dicst to json_data"""
stats = {
"view_count": self.youtube_meta.get("view_count", 0),
"like_count": self.youtube_meta.get("like_count", 0),
"dislike_count": self.youtube_meta.get("dislike_count", 0),
"average_rating": self.youtube_meta.get("average_rating", 0),
"view_count": self.youtube_meta.get("view_count") or 0,
"like_count": self.youtube_meta.get("like_count") or 0,
"dislike_count": self.youtube_meta.get("dislike_count") or 0,
"average_rating": self.youtube_meta.get("average_rating") or 0,
}
self.json_data.update({"stats": stats})
@ -288,9 +297,9 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
self.json_data.update(
{
"player": {
"watched": False,
"duration": duration,
"duration_str": get_duration_str(duration),
"watched": False,
}
}
)
@ -379,8 +388,8 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
return
dislikes = {
"dislike_count": result.get("dislikes", 0),
"average_rating": result.get("rating", 0),
"dislike_count": result.get("dislikes") or 0,
"average_rating": result.get("rating") or 0,
}
self.json_data["stats"].update(dislikes)
@ -397,6 +406,10 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
self.json_data["subtitles"] = indexed
return
if not self.youtube_meta:
print(f"{self.youtube_id}: skip subtitle check without metadata")
return
handler = YoutubeSubtitle(self)
subtitles = handler.get_subtitles()
if subtitles:
@ -412,7 +425,7 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
subtitle_media_url = f"{base_name}.{lang}.vtt"
to_add = {
"ext": "vtt",
"url": False,
"url": None,
"name": lang,
"lang": lang,
"source": "file",
@ -424,10 +437,6 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
def embed_metadata(self):
"""embed metadata for video"""
if not self.config["downloads"].get("add_metadata"):
return
print(f"{self.youtube_id}: embed metadata")
if not self.json_data:
self.get_from_es()
@ -435,6 +444,16 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
print(f"{self.youtube_id}: skip embed, video not indexed")
return
if self.config["downloads"].get("add_metadata"):
try:
self._embed_text_data()
self._embed_artwork()
except MP4MetadataError as err:
print(f"{self.youtube_id}: embed failed: '{str(err)}'")
def _embed_text_data(self):
"""embed text metadata"""
print(f"{self.youtube_id}: embed metadata")
video_base = EnvironmentSettings.MEDIA_DIR
media_url = self.json_data.get("media_url")
file_path = os.path.join(video_base, media_url)
@ -444,14 +463,16 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
title = self.json_data["title"]
artist = self.json_data["channel"]["channel_name"]
description = self.json_data["description"]
description = self.json_data.get("description", "")
to_embed = self._get_to_embed()
video = MP4(file_path)
video["\xa9nam"] = [title] # title
video["\xa9ART"] = [artist] # artist
video["desc"] = [description] # description
video["ldes"] = [description] # synopsis
if description:
video["desc"] = [description] # description
video["ldes"] = [description] # synopsis
video["----:com.tubearchivist:ta"] = [to_embed.encode("utf-8")]
video.save()
@ -479,16 +500,30 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
"comments": comments,
"subtitles": subtitles,
"playlists": playlists,
"version": settings.TA_VERSION,
}
)
return to_embed
def _embed_artwork(self):
"""embed artwork"""
print(f"{self.youtube_id}: embed artwork")
ThumbManager(self.youtube_id).embed_video_art(self.json_data)
def index_new_video(youtube_id, video_type=VideoTypeEnum.VIDEOS):
"""combined classes to create new video in index"""
from appsettings.src.reindex import Reindex
video = YoutubeVideo(youtube_id, video_type=video_type)
video.build_json()
video.get_from_es(print_error=False)
if video.json_data:
# reindex only for force redownload
video = Reindex().reindex_single_video(youtube_id=youtube_id)
else:
video.build_json()
if not video.json_data:
raise ValueError("failed to get metadata for " + youtube_id)

26
backend/video/src/media_streams.py
View File

@ -12,7 +12,7 @@ class MediaStreamExtractor:
self.media_path = media_path
self.metadata = []
def extract_metadata(self):
def extract_metadata(self) -> list[dict]:
"""entry point to extract metadata"""
cmd = [
@ -38,17 +38,15 @@ class MediaStreamExtractor:
return self.metadata
def process_stream(self, stream):
def process_stream(self, stream) -> None:
"""parse stream to metadata"""
codec_type = stream.get("codec_type")
if codec_type == "video":
self._extract_video_metadata(stream)
elif codec_type == "audio":
self._extract_audio_metadata(stream)
else:
return
def _extract_video_metadata(self, stream):
def _extract_video_metadata(self, stream) -> None:
"""parse video metadata"""
if "bit_rate" not in stream:
# is probably thumbnail
@ -56,26 +54,26 @@ class MediaStreamExtractor:
self.metadata.append(
{
"type": "video",
"index": stream["index"],
"bitrate": int(stream.get("bit_rate", 0)),
"codec": stream["codec_name"],
"width": stream["width"],
"height": stream["height"],
"bitrate": int(stream["bit_rate"]),
"index": stream["index"],
"type": "video",
"width": stream["width"],
}
)
def _extract_audio_metadata(self, stream):
def _extract_audio_metadata(self, stream) -> None:
"""extract audio metadata"""
self.metadata.append(
{
"type": "audio",
"index": stream["index"],
"codec": stream.get("codec_name", "undefined"),
"bitrate": int(stream.get("bit_rate", 0)),
"codec": stream.get("codec_name", "undefined"),
"index": stream["index"],
"type": "audio",
}
)
def get_file_size(self):
def get_file_size(self) -> int:
"""get filesize in bytes"""
return stat(self.media_path).st_size

391
backend/video/src/meta_embed.py
View File

@ -1,7 +1,30 @@
"""bulk metadata embedding"""
"""
Functionality:
- bulk metadata embedding
- restore from embedded metadata
"""
import json
import os
import shutil
from appsettings.src.config import AppConfig, AppConfigType
from channel.serializers import ChannelSerializer
from channel.src.index import YoutubeChannel
from common.src.env_settings import EnvironmentSettings
from common.src.es_connect import ElasticWrap, IndexPaginate
from download.src.thumbnails import ThumbManager
from mutagen.mp4 import MP4, MP4FreeForm
from playlist.serializers import PlaylistSerializer
from playlist.src.index import YoutubePlaylist
from video.serializers import (
CommentsSerializer,
SubtitleFragmentSerializer,
VideoSerializer,
)
from video.src.comments import Comments
from video.src.index import YoutubeVideo
from video.src.subtitle import SubtitleParser, YoutubeSubtitle
class MetadataEmbed:
@ -21,10 +44,11 @@ class MetadataEmbed:
paginate = IndexPaginate(
index_name=self.INDEX_NAME,
data=data,
size=200,
size=100,
callback=MetadataEmbedCallback,
task=self.task,
total=self._get_total(),
pit_keep_alive=1000,
)
_ = paginate.get_results()
@ -49,3 +73,366 @@ class MetadataEmbedCallback:
for video in self.source:
youtube_id = video["_source"]["youtube_id"]
YoutubeVideo(youtube_id).embed_metadata()
class IndexFromEmbed:
"""restore from embedded metadata, potential untrusted"""
VIDEOS_BASE = EnvironmentSettings.MEDIA_DIR
CACHE_DIR = EnvironmentSettings.CACHE_DIR
HOST_UID = EnvironmentSettings.HOST_UID
HOST_GID = EnvironmentSettings.HOST_GID
def __init__(
self,
file_path: str,
use_user_conf: bool = True,
config: AppConfigType | None = None,
):
self.file_path = file_path
self.use_user_conf = use_user_conf
self.config = config
def run_index(self) -> None | dict:
"""run index"""
if not self.config:
self.config = AppConfig().config
json_embed = self._get_embedded()
if not json_embed:
return None
channel_data_clean = self.index_channel(json_embed)
video = self.index_video(json_embed, channel_data_clean)
self.index_subtitles(json_embed, video)
self.index_comments(json_embed)
self.restore_artwork(video)
self.index_playlists(json_embed, video)
self.archive_video(video)
return video.json_data
def _get_embedded(self) -> dict | None:
"""get embedded metadata"""
video_mutagen = MP4(self.file_path)
ta_data = video_mutagen.get("----:com.tubearchivist:ta")
if not ta_data:
return None
if not isinstance(ta_data, list):
raise ValueError(f"[{self.file_path}] unexpected embedded data")
to_decode = ta_data[0]
if not isinstance(to_decode, MP4FreeForm):
raise ValueError(f"[{self.file_path}] unexpected embedded data")
try:
json_embed = json.loads(to_decode.decode())
except Exception as exc: # pylint: disable=broad-exception-caught
err = f"[{self.file_path}] embedded decoding failed: {str(exc)}"
raise ValueError(err) from exc
if not json_embed.get("video"):
err = f"[{self.file_path}] embedded does not contain video key"
raise ValueError(err)
return json_embed
def index_channel(self, json_embed):
"""index channel"""
channel_data = json_embed["video"].get("channel")
if not channel_data:
raise ValueError(f"[{self.file_path}] missing channel metadata")
serializer = ChannelSerializer(data=channel_data)
is_valid = serializer.is_valid()
if not is_valid:
err = serializer.errors
raise ValueError(
f"[{self.file_path}] channel serializer failed: {err}"
)
channel_data_clean = dict(serializer.data)
if not self.use_user_conf:
if "channel_overwrites" in channel_data_clean:
channel_data_clean.pop("channel_overwrites")
channel_data_clean["channel_subscribed"] = False
channel = YoutubeChannel(youtube_id=channel_data_clean["channel_id"])
channel.get_from_es()
if not channel.json_data:
channel.json_data = channel_data_clean
channel.upload_to_es()
return channel.json_data
def index_video(self, json_embed, channel_data_clean):
"""index video"""
video_data = json_embed["video"]
video_data.pop("channel")
serializer = VideoSerializer(data=video_data)
is_valid = serializer.is_valid()
if not is_valid:
err = serializer.errors
raise ValueError(
f"[{self.file_path}] video serializer failed: {err}"
)
video_data_clean = dict(serializer.data)
if not self.use_user_conf:
video_data_clean["player"]["watched"] = False
video_data_clean["channel"] = channel_data_clean
video = YoutubeVideo(youtube_id=video_data_clean["youtube_id"])
video.get_from_es()
if not video.json_data:
video.json_data = video_data_clean
video.upload_to_es()
return video
def archive_video(self, video):
"""archive video file"""
channel_id = video.json_data["channel"]["channel_id"]
folder = os.path.join(self.VIDEOS_BASE, channel_id)
if not os.path.exists(folder):
os.makedirs(folder)
if self.HOST_UID and self.HOST_GID:
os.chown(folder, self.HOST_UID, self.HOST_GID)
new_path = os.path.join(folder, f"{video.youtube_id}.mp4")
if self.file_path == new_path:
# already archived
return
shutil.move(self.file_path, new_path, copy_function=shutil.copyfile)
if self.HOST_UID and self.HOST_GID:
os.chown(new_path, self.HOST_UID, self.HOST_GID)
def index_playlists(self, json_embed, video):
"""index playlists"""
playlist_data = json_embed.get("playlists")
if not playlist_data or not isinstance(playlist_data, list):
return
serializer = PlaylistSerializer(data=playlist_data, many=True)
is_valid = serializer.is_valid()
if not is_valid:
err = serializer.errors
raise ValueError(
f"[{self.file_path}] playlist serializer failed: {err}"
)
expected = video.json_data.get("playlist", [])
video_mutagen = MP4(self.file_path)
for playlist_data in serializer.data:
playlist_id = playlist_data["playlist_id"]
if playlist_id not in expected:
continue
json_data = self._process_embedded_playlist(playlist_data)
if not json_data:
continue
playlist_art = os.path.join(
self.CACHE_DIR, "playlists", f"{playlist_id}.jpg"
)
self._restore_art_item(
video_mutagen,
"----:com.tubearchivist:playlist_{plalyist_id}",
playlist_art,
)
def _process_embedded_playlist(self, playlist_data) -> dict | None:
"""process single embedded playlist"""
if not self.use_user_conf:
if playlist_data["playlist_type"] == "custom":
# custom playlist is user conf
return None
playlist = YoutubePlaylist(youtube_id=playlist_data["playlist_id"])
playlist.get_from_es()
if playlist.json_data:
# already indexed
return None
playlist_data_clean = dict(playlist_data)
if not self.use_user_conf:
playlist_data_clean["playlist_subscribed"] = False
playlist_data_clean["playlist_sort_order"] = "top"
playlist.json_data = playlist_data_clean
playlist.upload_to_es()
return playlist.json_data
def restore_artwork(self, video):
"""restore artwork if needed"""
video_mutagen = MP4(self.file_path)
thumb = ThumbManager(video.youtube_id).vid_thumb_path(absolute=True)
self._restore_art_item(video_mutagen, "covr", thumb)
channel_id = video.json_data["channel"]["channel_id"]
banner_path = os.path.join(
self.CACHE_DIR, "channels", f"{channel_id}_banner.jpg"
)
self._restore_art_item(
video_mutagen, "----:com.tubearchivist:channel_banner", banner_path
)
channel_icon = os.path.join(
self.CACHE_DIR, "channels", f"{channel_id}_thumb.jpg"
)
self._restore_art_item(
video_mutagen, "----:com.tubearchivist:channel_icon", channel_icon
)
tv_art_path = os.path.join(
self.CACHE_DIR, "channels", f"{channel_id}_tvart.jpg"
)
self._restore_art_item(
video_mutagen, "----:com.tubearchivist:channel_tv", tv_art_path
)
def _restore_art_item(
self, video_mutagen, mutagen_key: str, target_path: str
) -> None:
"""restore single art item"""
if os.path.exists(target_path):
# don't overwrite
return
art_item = video_mutagen.get(mutagen_key)
if not art_item and not isinstance(art_item, list):
# is not embedded
return
art_folder = os.path.dirname(target_path)
if not os.path.exists(art_folder):
os.mkdir(art_folder)
with open(target_path, "wb") as f:
f.write(bytes(art_item[0]))
def index_subtitles(self, json_embed, video):
"""index subtitles"""
subtitle_data = json_embed.get("subtitles")
if not subtitle_data:
return
serializer = SubtitleFragmentSerializer(data=subtitle_data, many=True)
is_valid = serializer.is_valid()
if not is_valid:
err = serializer.errors
raise ValueError(
f"[{self.file_path}] subtitle serializer failed: {err}"
)
self._process_embedded_subs(video, subtitle_data=serializer.data)
def _process_embedded_subs(self, video, subtitle_data):
"""process single embedded subtitle"""
embedded_subs = {
(i["subtitle_lang"], i["subtitle_source"]) for i in subtitle_data
}
subs = YoutubeSubtitle(video)
response = subs.get_es_subtitles()
indexed = {
(i["subtitle_lang"], i["subtitle_source"]) for i in response
}
for embedded_lang, embedded_source in embedded_subs:
needs_processing = self._process_subtitle(
indexed, embedded_lang, embedded_source
)
if not needs_processing:
continue
segments = [
i
for i in subtitle_data
if i["subtitle_lang"] == embedded_lang
and i["subtitle_source"] == embedded_source
]
to_index = sorted(segments, key=lambda d: d["subtitle_index"])
parser = SubtitleParser(
subtitle_str="{}", lang=embedded_lang, source=embedded_source
)
for segment in to_index:
parser.all_cues.append(
{
"start": segment["subtitle_start"],
"end": segment["subtitle_end"],
"text": segment["subtitle_line"],
"idx": segment["subtitle_index"],
}
)
subtitle_str = parser.get_subtitle_str()
query_str = parser.create_bulk_import(to_index)
subs.index_subtitle(query_str)
media_url = subs.get_media_url(lang=embedded_lang)
dest_path = os.path.join(self.VIDEOS_BASE, media_url)
subs.write_subtitle_file(dest_path, subtitle_str)
def _process_subtitle(
self, indexed, embedded_lang, embedded_source
) -> bool:
"""check if subtitle should be processed"""
for sub_indexed in indexed:
if (
sub_indexed.get("lang") == embedded_lang
and sub_indexed.get("source") == embedded_source
):
# already indexed
return False
if not self.use_user_conf:
return True
if not self.config:
return False
langs = self.config["downloads"]["subtitle"]
source = self.config["downloads"]["subtitle_source"]
if not langs or not source:
return False
lang_codes = [i.strip() for i in langs.split(",")]
if embedded_lang not in lang_codes:
return True
return False
def index_comments(self, json_embed):
"""index comments"""
comment_data = json_embed.get("comments")
if not comment_data:
return
serializer = CommentsSerializer(data=comment_data)
is_valid = serializer.is_valid()
if not is_valid:
err = serializer.errors
raise ValueError(
f"[{self.file_path}] comments serializer failed: {err}"
)
if self.use_user_conf:
if self.config and not self.config["downloads"]["comment_max"]:
return
comments = Comments(youtube_id=serializer.data["youtube_id"])
existing = comments.get_es_comments()
if existing:
return
comments.json_data = dict(serializer.data)
comments.upload_comments()

130
backend/video/src/subtitle.py
View File

@ -10,14 +10,25 @@ import os
import re
from datetime import datetime
from operator import itemgetter
from typing import TypedDict
import requests
from common.src.env_settings import EnvironmentSettings
from common.src.es_connect import ElasticWrap, IndexPaginate
from common.src.helper import rand_sleep, requests_headers
from download.src.yt_dlp_base import CookieHandler
from yt_dlp.utils import orderedSet_from_options
class SubtitleCue(TypedDict):
"""describe single subtitle queue"""
start: str
end: str
text: str
idx: int
class YoutubeSubtitle:
"""handle video subtitle functionality"""
@ -54,7 +65,9 @@ class YoutubeSubtitle:
)
]
except re.error as e:
raise ValueError(f"wrong regex in subtitle config: {e.pattern}")
raise ValueError(
f"wrong regex in subtitle config: {e.pattern}"
) from e
return relevant_subtitles
@ -74,8 +87,7 @@ class YoutubeSubtitle:
# not supported yet
continue
video_media_url = self.video.json_data["media_url"]
media_url = video_media_url.replace(".mp4", f".{lang}.vtt")
media_url = self.get_media_url(lang)
if not all_formats:
# no subtitles found
continue
@ -93,6 +105,12 @@ class YoutubeSubtitle:
return candidate_subtitles
def get_media_url(self, lang: str) -> str:
"""get media url"""
video_media_url = self.video.json_data["media_url"]
media_url = video_media_url.replace(".mp4", f".{lang}.vtt")
return media_url
def get_es_subtitles(self) -> list[dict]:
"""get subtitles from elastic"""
data = {
@ -113,39 +131,75 @@ class YoutubeSubtitle:
dest_path = os.path.join(videos_base, subtitle["media_url"])
source = subtitle["source"]
lang = subtitle.get("lang")
response = requests.get(
subtitle["url"], headers=requests_headers(), timeout=30
)
if not response.ok:
subtitle_key = f"{self.video.youtube_id}-{lang}"
print(f"{subtitle_key}: failed to download subtitle")
print(response.text)
rand_sleep(self.video.config)
response_text = self._make_request(subtitle["url"], lang)
if not response_text:
continue
if not response.text:
print(f"{subtitle_key}: skip empty subtitle")
rand_sleep(self.video.config)
continue
parser = SubtitleParser(response.text, lang, source)
parser = SubtitleParser(response_text, lang, source)
parser.process()
if not parser.all_cues:
rand_sleep(self.video.config)
continue
subtitle_str = parser.get_subtitle_str()
self._write_subtitle_file(dest_path, subtitle_str)
self.write_subtitle_file(dest_path, subtitle_str)
if self.video.config["downloads"]["subtitle_index"]:
query_str = parser.create_bulk_import(self.video, source)
self._index_subtitle(query_str)
documents = parser.create_documents(self.video, source)
query_str = parser.create_bulk_import(documents)
self.index_subtitle(query_str)
indexed.append(subtitle)
indexed.append(
{
"ext": "json3",
"name": subtitle["name"],
"source": subtitle["source"],
"lang": subtitle["lang"],
"media_url": subtitle["media_url"],
"url": subtitle["url"],
}
)
rand_sleep(self.video.config)
return indexed
def _write_subtitle_file(self, dest_path, subtitle_str):
def _make_request(self, url: str, lang: str | None) -> str | None:
"""make the request"""
request_kwargs: dict = {
"timeout": 30,
"headers": requests_headers(),
}
if self.video.config["downloads"].get("cookie_import"):
cookie = CookieHandler(self.video.config).get()
if cookie:
cookies_txt = cookie.read()
jar = requests.cookies.RequestsCookieJar()
for line in cookies_txt.split("\n"):
words = line.split()
if (len(words) == 7) and (words[0] != "#"):
jar.set(
words[5], words[6], domain=words[0], path=words[2]
)
request_kwargs["cookies"] = jar
response = requests.get(url, **request_kwargs)
if not response.ok:
subtitle_key = f"{self.video.youtube_id}-{lang}"
print(f"{subtitle_key}: failed to download subtitle")
print(response.text)
rand_sleep(self.video.config)
return None
if not response.text:
print(f"{subtitle_key}: skip empty subtitle")
rand_sleep(self.video.config)
return None
return response.text
def write_subtitle_file(self, dest_path, subtitle_str):
"""write subtitle file to disk"""
# create folder here for first video of channel
os.makedirs(os.path.split(dest_path)[0], exist_ok=True)
@ -158,7 +212,7 @@ class YoutubeSubtitle:
os.chown(dest_path, host_uid, host_gid)
@staticmethod
def _index_subtitle(query_str):
def index_subtitle(query_str):
"""send subtitle to es for indexing"""
_, _ = ElasticWrap("_bulk").post(data=query_str, ndjson=True)
@ -190,15 +244,14 @@ class YoutubeSubtitle:
class SubtitleParser:
"""parse subtitle str from youtube"""
def __init__(self, subtitle_str, lang, source):
def __init__(self, subtitle_str: str, lang: str, source: str) -> None:
self.subtitle_raw = json.loads(subtitle_str)
self.lang = lang
self.source = source
self.all_cues = False
self.all_cues: list[SubtitleCue] = []
def process(self):
def process(self) -> None:
"""extract relevant que data"""
self.all_cues = []
all_events = self.subtitle_raw.get("events")
if not all_events:
@ -213,12 +266,12 @@ class SubtitleParser:
print(f"skipping subtitle event without content: {event}")
continue
cue = {
"start": self._ms_conv(event["tStartMs"]),
"end": self._ms_conv(event["tStartMs"] + event["dDurationMs"]),
"text": "".join([i.get("utf8") for i in event["segs"]]),
"idx": idx + 1,
}
cue = SubtitleCue(
start=self._ms_conv(event["tStartMs"]),
end=self._ms_conv(event["tStartMs"] + event["dDurationMs"]),
text="".join([i.get("utf8") for i in event["segs"]]),
idx=idx + 1,
)
self.all_cues.append(cue)
@staticmethod
@ -272,9 +325,8 @@ class SubtitleParser:
return subtitle_str
def create_bulk_import(self, video, source):
def create_bulk_import(self, documents):
"""subtitle lines for es import"""
documents = self._create_documents(video, source)
bulk_list = []
for document in documents:
@ -288,18 +340,18 @@ class SubtitleParser:
return query_str
def _create_documents(self, video, source):
def create_documents(self, video, source):
"""process documents"""
documents = self._chunk_list(video.youtube_id)
channel = video.json_data.get("channel")
meta_dict = {
"youtube_id": video.youtube_id,
"title": video.json_data.get("title"),
"subtitle_channel": channel.get("channel_name"),
"subtitle_channel_id": channel.get("channel_id"),
"subtitle_last_refresh": int(datetime.now().timestamp()),
"subtitle_lang": self.lang,
"subtitle_last_refresh": int(datetime.now().timestamp()),
"subtitle_source": source,
"title": video.json_data.get("title"),
"youtube_id": video.youtube_id,
}
_ = [i.update(meta_dict) for i in documents]

2
docker-compose.yml
View File

@ -38,7 +38,7 @@ services:
depends_on:
- archivist-es
archivist-es:
image: bbilly1/tubearchivist-es # only for amd64, or use official es 8.18.2
image: bbilly1/tubearchivist-es # only for amd64, or use official es 8.19.0
container_name: archivist-es
restart: unless-stopped
environment:

3
docker_assets/nginx.conf
View File

@ -57,14 +57,13 @@ server {
location = /index.html {
add_header Cache-Control "no-store, no-cache, must-revalidate";
add_header Pragma "no-cache";
add_header Expires 0;
expires 0;
}
location / {
add_header Cache-Control "no-store, no-cache, must-revalidate";
add_header Pragma "no-cache";
add_header Expires 0;
expires 0;
try_files $uri $uri/ /index.html =404;
}
}

1
frontend/.nvmrc Normal file
View File

@ -0,0 +1 @@
22.13.0

1936
frontend/package-lock.json generated
View File

File diff suppressed because it is too large Load Diff

38
frontend/package.json
View File

@ -11,27 +11,27 @@
"preview": "vite preview"
},
"dependencies": {
"dompurify": "^3.2.6",
"react": "^19.1.1",
"react-dom": "^19.1.1",
"react-router-dom": "^7.8.0",
"zustand": "^5.0.7"
"dompurify": "^3.3.1",
"react": "^19.2.3",
"react-dom": "^19.2.3",
"react-router-dom": "^7.11.0",
"zustand": "^5.0.9"
},
"devDependencies": {
"@types/react": "^19.1.9",
"@types/react-dom": "^19.1.7",
"@typescript-eslint/eslint-plugin": "^8.39.0",
"@typescript-eslint/parser": "^8.39.0",
"@vitejs/plugin-react-swc": "^4.0.0",
"eslint": "^9.33.0",
"@types/react": "^19.2.7",
"@types/react-dom": "^19.2.3",
"@typescript-eslint/eslint-plugin": "^8.51.0",
"@typescript-eslint/parser": "^8.51.0",
"@vitejs/plugin-react-swc": "^4.2.2",
"eslint": "^9.39.2",
"eslint-config-prettier": "^10.1.8",
"eslint-plugin-react-hooks": "^5.2.0",
"eslint-plugin-react-refresh": "^0.4.20",
"globals": "^16.3.0",
"prettier": "3.6.2",
"typescript": "^5.9.2",
"typescript-eslint": "^8.39.0",
"vite": ">=7.1.1",
"vite-plugin-checker": "^0.10.2"
"eslint-plugin-react-hooks": "^7.0.1",
"eslint-plugin-react-refresh": "^0.4.26",
"globals": "^17.0.0",
"prettier": "3.7.4",
"typescript": "^5.9.3",
"typescript-eslint": "^8.51.0",
"vite": "^7.3.0",
"vite-plugin-checker": "^0.12.0"
}
}

1
frontend/src/api/actions/createTaskSchedule.ts
View File

@ -10,7 +10,6 @@ export type TaskScheduleNameType =
| 'restore_backup'
| 'rescan_filesystem'
| 'thumbnail_check'
| 'resync_thumbs'
| 'index_playlists'
| 'subscribe_to'
| 'version_check';

9
frontend/src/api/actions/deletePoToken.ts
View File

@ -1,9 +0,0 @@
import APIClient from '../../functions/APIClient';
const deletePoToken = async () => {
return APIClient('/api/appsettings/potoken/', {
method: 'DELETE',
});
};
export default deletePoToken;

9
frontend/src/api/actions/queueManualImport.ts Normal file
View File

@ -0,0 +1,9 @@
import APIClient from '../../functions/APIClient';
const queueManualImport = async (ignore_error: boolean, prefer_local: boolean) => {
return APIClient('/api/appsettings/manual-import/', {
method: 'POST',
body: { ignore_error, prefer_local },
});
};
export default queueManualImport;

9
frontend/src/api/actions/queueStartFilesystemRescan.ts Normal file
View File

@ -0,0 +1,9 @@
import APIClient from '../../functions/APIClient';
const queueStartFilesystemRescan = async (ignore_error: boolean, prefer_local: boolean) => {
return APIClient('/api/appsettings/rescan-filesystem/', {
method: 'POST',
body: { ignore_error, prefer_local },
});
};
export default queueStartFilesystemRescan;

10
frontend/src/api/actions/updatePoToken.ts
View File

@ -1,10 +0,0 @@
import APIClient from '../../functions/APIClient';
const updatePoToken = async (potoken: string) => {
return APIClient('/api/appsettings/potoken/', {
method: 'POST',
body: { potoken },
});
};
export default updatePoToken;

8
frontend/src/api/actions/updateTaskByName.ts
View File

@ -1,12 +1,6 @@
import APIClient from '../../functions/APIClient';
type TaskNamesType =
| 'download_pending'
| 'update_subscribed'
| 'manual_import'
| 'resync_thumbs'
| 'resync_metadata'
| 'rescan_filesystem';
type TaskNamesType = 'download_pending' | 'update_subscribed' | 'resync_metadata';
const updateTaskByName = async (taskName: TaskNamesType) => {
return APIClient(`/api/task/by-name/${taskName}/`, {

3
frontend/src/api/loader/loadAppsettingsConfig.ts
View File

@ -16,14 +16,13 @@ export type AppSettingsConfigType = {
format: string | null;
format_sort: string | null;
add_metadata: boolean;
add_thumbnail: boolean;
subtitle: string | null;
subtitle_source: string | null;
subtitle_index: boolean;
comment_max: string | null;
comment_sort: string;
cookie_import: boolean;
potoken: boolean;
pot_provider_url: string | null;
throttledratelimit: number | null;
extractor_lang: string | null;
integrate_ryd: boolean;

2
frontend/src/api/loader/loadPlaylistById.ts
View File

@ -12,7 +12,7 @@ export type PlaylistType = {
playlist_active: boolean;
playlist_channel: string;
playlist_channel_id: string;
playlist_description: string;
playlist_description: string | null;
playlist_entries: PlaylistEntryType[];
playlist_sort_order: 'top' | 'bottom';
playlist_id: string;

29
frontend/src/components/CommentBox.tsx
View File

@ -6,20 +6,6 @@ import Linkify from './Linkify';
import formatNumbers from '../functions/formatNumbers';
import Button from './Button';
export type CommentReplyType = {
comment_id: string;
comment_text: string;
comment_timestamp: number;
comment_time_text: string;
comment_likecount: number;
comment_is_favorited: false;
comment_author: string;
comment_author_id: string;
comment_author_thumbnail: string;
comment_author_is_uploader: boolean;
comment_parent: string;
};
export type CommentsType = {
comment_id: string;
comment_text: string;
@ -32,16 +18,17 @@ export type CommentsType = {
comment_author_thumbnail: string;
comment_author_is_uploader: boolean;
comment_parent: string;
comment_replies?: CommentReplyType[];
comment_replies: CommentsType[];
};
type CommentBoxProps = {
comment: CommentsType;
showThread?: boolean;
onTimestampClick?: (seconds: number) => void;
};
const CommentBox = ({ comment, onTimestampClick }: CommentBoxProps) => {
const [showSubComments, setShowSubComments] = useState(false);
const CommentBox = ({ comment, showThread = false, onTimestampClick }: CommentBoxProps) => {
const [showSubComments, setShowSubComments] = useState(showThread);
const hasSubComments =
comment.comment_replies !== undefined && comment.comment_replies.length > 0;
@ -93,10 +80,14 @@ const CommentBox = ({ comment, onTimestampClick }: CommentBoxProps) => {
<div className="comments-replies" style={{ display: 'block' }}>
{showSubComments &&
comment.comment_replies?.map(comment => {
comment.comment_replies.map(comment => {
return (
<Fragment key={comment.comment_id}>
<CommentBox comment={comment} onTimestampClick={onTimestampClick} />
<CommentBox
comment={comment}
onTimestampClick={onTimestampClick}
showThread={showSubComments}
/>
</Fragment>
);
})}

1
frontend/src/components/Filterbar.tsx
View File

@ -60,6 +60,7 @@ const Filterbar = ({
}
if (currentViewStyle === ViewStylesEnum.Table) {
// eslint-disable-next-line react-hooks/set-state-in-effect
setShowHidden(true);
} else {
setShowHidden(false);

21
frontend/src/components/MembershipAppsettings.tsx
View File

@ -24,6 +24,7 @@ type ProfileResponseType = {
sponsor_tier: SponsorTierType;
subscription_count: number;
subscription_is_max: boolean;
is_connected: boolean;
};
export default function MembershipAppsettings({ show_help_text }: { show_help_text: boolean }) {
@ -136,6 +137,13 @@ export default function MembershipAppsettings({ show_help_text }: { show_help_te
.
</li>
<li>Click on validate to verify everything is working.</li>
<li>
Setup the{' '}
<a href="https://github.com/tubearchivist/members" target="_blank">
Client Container
</a>
.
</li>
<li>
If you are subscribed to less channels than your sponsor tier allows, you can directly
sync all your subscriptions here.
@ -206,6 +214,19 @@ export default function MembershipAppsettings({ show_help_text }: { show_help_te
<br />
Subscriptions: {profileResponse.subscription_count}/
{profileResponse.sponsor_tier.max_subs}
<br />
Socket:{' '}
{profileResponse.is_connected ? (
'Established'
) : (
<>
Not established. Make sure the{' '}
<a href="https://github.com/tubearchivist/members" target="_blank">
Client Container
</a>{' '}
is connected.
</>
)}
</p>
</>
)}

4
frontend/src/components/SearchExampleQueries.tsx
View File

@ -104,6 +104,10 @@ const SearchExampleQueries = () => {
auto-generated subtitles only, or <i>user</i> to search through user-uploaded
subtitles only
</li>
<li>
<span>channel:</span> — limit subtitle search to a specific channel name (for
example: <code>full:javascript channel:corey schafer</code>)
</li>
</ul>
</li>
</ul>

3
frontend/src/components/VideoPlayer.tsx
View File

@ -146,6 +146,7 @@ const VideoPlayer = ({
}
if (setSeekToTimestamp) setSeekToTimestamp(undefined);
window.scroll(0, 0);
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [seekToTimestamp]);
const [searchParams] = useSearchParams();
@ -389,6 +390,7 @@ const VideoPlayer = ({
} else if (!theaterModePressed) {
setTheaterModeKeyPressed(false);
}
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [theaterModePressed, isTheaterMode, theaterModeKeyPressed]);
useEffect(() => {
@ -401,6 +403,7 @@ const VideoPlayer = ({
infoDialog('Normal mode');
}
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [escapePressed, isTheaterMode]);
return (

5
frontend/src/pages/ChannelAbout.tsx
View File

@ -8,7 +8,6 @@ import Routes from '../configuration/routes/RouteList';
import queueReindex, { ReindexType, ReindexTypeEnum } from '../api/actions/queueReindex';
import formatDate from '../functions/formatDates';
import PaginationDummy from '../components/PaginationDummy';
import FormattedNumber from '../components/FormattedNumber';
import Button from '../components/Button';
import updateChannelOverwrites from '../api/actions/updateChannelOverwrite';
import useIsAdmin from '../functions/useIsAdmin';
@ -145,10 +144,6 @@ const ChannelAbout = () => {
<div className="info-box-item">
<div>
{channel.channel_views > 0 && (
<FormattedNumber text="Channel views:" number={channel.channel_views} />
)}
{isAdmin && (
<>
<div className="button-box">

1
frontend/src/pages/Channels.tsx
View File

@ -41,7 +41,6 @@ export type ChannelType = {
channel_tags?: string[];
channel_thumb_url: string;
channel_tvart_url: string;
channel_views: number;
};
const Channels = () => {

1
frontend/src/pages/Home.tsx
View File

@ -89,7 +89,6 @@ export type DownloadsType = {
format: boolean;
format_sort: boolean;
add_metadata: boolean;
add_thumbnail: boolean;
subtitle: boolean;
subtitle_source: boolean;
subtitle_index: boolean;

3
frontend/src/pages/Playlist.tsx
View File

@ -108,7 +108,6 @@ const Playlist = () => {
}
setRefresh(false);
})();
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [
playlistId,
userConfig.hide_watched_playlist,
@ -331,7 +330,7 @@ const Playlist = () => {
</div>
</div>
{playlist.playlist_description !== 'False' && (
{playlist.playlist_description !== null && (
<div className="description-box">
<p
id={descriptionExpanded ? 'text-expand-expanded' : 'text-expand'}

15
frontend/src/pages/Search.tsx
View File

@ -65,6 +65,13 @@ const Search = () => {
const gridView = isGridView ? `boxed-${gridItems}` : '';
const gridViewGrid = isGridView ? `grid-${gridItems}` : '';
const fetchResults = async (searchQuery: string) => {
const searchResults = await loadSearch(searchQuery);
setSearchResults(searchResults);
setRefresh(false);
};
useEffect(() => {
const handler = setTimeout(() => {
setDebouncedSearchTerm(searchTerm);
@ -77,19 +84,13 @@ const Search = () => {
useEffect(() => {
if (debouncedSearchTerm.trim() !== '') {
// eslint-disable-next-line react-hooks/set-state-in-effect
fetchResults(debouncedSearchTerm);
} else {
setSearchResults(EmptySearchResponse);
}
}, [debouncedSearchTerm, refresh, videoId]);
const fetchResults = async (searchQuery: string) => {
const searchResults = await loadSearch(searchQuery);
setSearchResults(searchResults);
setRefresh(false);
};
return (
<>
<title>TubeArchivist</title>

79
frontend/src/pages/SettingsActions.tsx
View File

@ -7,16 +7,22 @@ import restoreBackup from '../api/actions/restoreBackup';
import Notifications from '../components/Notifications';
import Button from '../components/Button';
import { ApiResponseType } from '../functions/APIClient';
import ToggleConfig from '../components/ToggleConfig';
import queueStartFilesystemRescan from '../api/actions/queueStartFilesystemRescan';
import queueManualImport from '../api/actions/queueManualImport';
const SettingsActions = () => {
const [deleteIgnored, setDeleteIgnored] = useState(false);
const [deletePending, setDeletePending] = useState(false);
const [processingImports, setProcessingImports] = useState(false);
const [reEmbed, setReEmbed] = useState(false);
const [reSyncMeta, setReSyncMeta] = useState(false);
const [backupStarted, setBackupStarted] = useState(false);
const [isRestoringBackup, setIsRestoringBackup] = useState(false);
const [reScanningFileSystem, setReScanningFileSystem] = useState(false);
const [rescanIgnoreErrors, setRescanIgnoreErrors] = useState(false);
const [rescanPreferLocal, setRescanPreferLocal] = useState(false);
const [manualPreferLocal, setManualPreferLocal] = useState(false);
const [manualIgnoreErrors, setManualIgnoreErrors] = useState(false);
const [backupListResponse, setBackupListResponse] = useState<ApiResponseType<BackupListType>>();
@ -44,7 +50,6 @@ const SettingsActions = () => {
deleteIgnored ||
deletePending ||
processingImports ||
reEmbed ||
reSyncMeta ||
backupStarted ||
isRestoringBackup ||
@ -54,7 +59,6 @@ const SettingsActions = () => {
setDeleteIgnored(false);
setDeletePending(false);
setProcessingImports(false);
setReEmbed(false);
setReSyncMeta(false);
setBackupStarted(false);
setIsRestoringBackup(false);
@ -69,44 +73,48 @@ const SettingsActions = () => {
<h2>Manual media files import.</h2>
<p>
Add files to the <span className="settings-current">cache/import</span> folder. Make
sure to follow the instructions in the Github{' '}
sure to follow the instructions in the{' '}
<a
href="https://docs.tubearchivist.com/settings/actions/#manual-media-files-import"
target="_blank"
>
Wiki
Docs
</a>
.
</p>
<div id="manual-import">
<div className="settings-box-wrapper">
<div>
<p>Prefer embedded metadata</p>
</div>
<ToggleConfig
name="manual_prefer_local"
value={manualPreferLocal}
updateCallback={() => setManualPreferLocal(!manualPreferLocal)}
/>
</div>
<div className="settings-box-wrapper">
<div>
<p>Ignore missing metadata errors</p>
</div>
<ToggleConfig
name="manual_ignore_error"
value={manualIgnoreErrors}
updateCallback={() => setManualIgnoreErrors(!manualIgnoreErrors)}
/>
</div>
{processingImports && <p>Processing import</p>}
{!processingImports && (
<Button
label="Start import"
onClick={async () => {
await updateTaskByName('manual_import');
await queueManualImport(manualIgnoreErrors, manualPreferLocal);
setProcessingImports(true);
}}
/>
)}
</div>
</div>
<div className="settings-group">
<h2>Embed thumbnails into media file.</h2>
<p>Set extracted youtube thumbnail as cover art of the media file.</p>
<div id="re-embed">
{reEmbed && <p>Processing thumbnails</p>}
{!reEmbed && (
<Button
label="Start process"
onClick={async () => {
await updateTaskByName('resync_thumbs');
setReEmbed(true);
}}
/>
)}
</div>
</div>
<div className="settings-group">
<h2>Embed metadata into media file</h2>
<p>Embed metadata into media files as mp4 tags.</p>
@ -196,23 +204,42 @@ const SettingsActions = () => {
deleted videos from the filesystem.
</p>
<p>
Rescan your media folder looking for missing videos and clean up index. More info on the
Github{' '}
Rescan your media folder looking for missing videos and clean up index. More info on the{' '}
<a
href="https://docs.tubearchivist.com/settings/actions/#rescan-filesystem"
target="_blank"
>
Wiki
Docs
</a>
.
</p>
<div id="fs-rescan">
<div className="settings-box-wrapper">
<div>
<p>Prefer embedded metadata</p>
</div>
<ToggleConfig
name="prefer_local"
value={rescanPreferLocal}
updateCallback={() => setRescanPreferLocal(!rescanPreferLocal)}
/>
</div>
<div className="settings-box-wrapper">
<div>
<p>Ignore missing metadata errors</p>
</div>
<ToggleConfig
name="ignore_error"
value={rescanIgnoreErrors}
updateCallback={() => setRescanIgnoreErrors(!rescanIgnoreErrors)}
/>
</div>
{reScanningFileSystem && <p>File system scan in progress</p>}
{!reScanningFileSystem && (
<Button
label="Rescan filesystem"
onClick={async () => {
await updateTaskByName('rescan_filesystem');
await queueStartFilesystemRescan(rescanIgnoreErrors, rescanPreferLocal);
setReScanningFileSystem(true);
}}
/>

130
frontend/src/pages/SettingsApplication.tsx
View File

@ -17,8 +17,6 @@ import updateCookie from '../api/actions/updateCookie';
import loadCookie, { CookieStateType } from '../api/loader/loadCookie';
import deleteCookie from '../api/actions/deleteCookie';
import validateCookie from '../api/actions/validateCookie';
import deletePoToken from '../api/actions/deletePoToken';
import updatePoToken from '../api/actions/updatePoToken';
import { useUserConfigStore } from '../stores/UserConfigStore';
import MembershipAppsettings from '../components/MembershipAppsettings';
@ -33,6 +31,7 @@ const SettingsApplication = () => {
const { userConfig } = useUserConfigStore();
const [response, setResponse] = useState<SettingsApplicationReponses>();
const [refresh, setRefresh] = useState(false);
const [visibleSnapshotCount, setVisibleSnapshotCount] = useState(10);
const snapshots = response?.snapshots;
const appSettingsConfig = response?.appSettingsConfig;
@ -57,7 +56,6 @@ const SettingsApplication = () => {
const [downloadsFormatSort, setDownloadsFormatSort] = useState<string | null>(null);
const [downloadsExtractorLang, setDownloadsExtractorLang] = useState<string | null>(null);
const [embedMetadata, setEmbedMetadata] = useState(false);
const [embedThumbnail, setEmbedThumbnail] = useState(false);
// Subtitles
const [subtitleLang, setSubtitleLang] = useState<string | null>(null);
@ -71,8 +69,7 @@ const SettingsApplication = () => {
// Cookie
const [cookieFormData, setCookieFormData] = useState<string>('');
const [showCookieForm, setShowCookieForm] = useState<boolean>(false);
const [poTokenFormData, setPoTokenFormData] = useState<string>('web+');
const [showPoTokenForm, setShowPoTokenForm] = useState<boolean>(false);
const [potProviderUrl, setPotProviderUrl] = useState<string | null>(null);
// Integrations
const [showApiToken, setShowApiToken] = useState(false);
@ -115,7 +112,6 @@ const SettingsApplication = () => {
setDownloadsFormatSort(appSettingsConfigData?.downloads.format_sort || null);
setDownloadsExtractorLang(appSettingsConfigData?.downloads.extractor_lang || null);
setEmbedMetadata(appSettingsConfigData?.downloads.add_metadata || false);
setEmbedThumbnail(appSettingsConfigData?.downloads.add_thumbnail || false);
// Subtitles
setSubtitleLang(appSettingsConfigData?.downloads.subtitle || null);
@ -126,6 +122,9 @@ const SettingsApplication = () => {
setCommentsMax(appSettingsConfigData?.downloads.comment_max || null);
setCommentsSort(appSettingsConfigData?.downloads.comment_sort || '');
// Cookie
setPotProviderUrl(appSettingsConfigData?.downloads.pot_provider_url || null);
// Integrations
setDownloadDislikes(appSettingsConfigData?.downloads.integrate_ryd || false);
setEnableSponsorBlock(appSettingsConfigData?.downloads.integrate_sponsorblock || false);
@ -169,24 +168,14 @@ const SettingsApplication = () => {
setRefresh(true);
};
const handlePoTokenRevoke = async () => {
await deletePoToken();
setRefresh(true);
};
const handlePoTokenUpdate = async () => {
await updatePoToken(poTokenFormData);
setPoTokenFormData('web+');
setShowPoTokenForm(false);
setRefresh(true);
};
useEffect(() => {
// eslint-disable-next-line react-hooks/set-state-in-effect
fetchData();
}, []);
useEffect(() => {
if (refresh) {
// eslint-disable-next-line react-hooks/set-state-in-effect
fetchData();
setRefresh(false);
}
@ -473,9 +462,8 @@ const SettingsApplication = () => {
</li>
</ul>
</li>
<li>Embedding metadata adds additional metadata directly to the mp4 file.</li>
<li>
Embedding the thumbnail embeds the video thumbnail as a cover.jpg to the mp4
Embedding metadata adds additional metadata and thumbnails directly to the mp4
file.
</li>
</ul>
@ -530,16 +518,6 @@ const SettingsApplication = () => {
updateCallback={handleUpdateConfig}
/>
</div>
<div className="settings-box-wrapper">
<div>
<p>Embed Thumbnail</p>
</div>
<ToggleConfig
name="downloads.add_thumbnail"
value={embedThumbnail}
updateCallback={handleUpdateConfig}
/>
</div>
</div>
<div className="info-box-item">
<h2 id="subtitles">Subtitles</h2>
@ -630,17 +608,17 @@ const SettingsApplication = () => {
Download and index comments. Browsable on the video detail page. Example:
<ul>
<li>
<span className="settings-current">all,100,all,30</span>: Get 100
max-parents and 30 max-replies-per-thread.
<span className="settings-current">all,100,all,30,all</span>: Get 100
max-parents and 30 max-replies-per-thread at any depth.
</li>
<li>
<span className="settings-current">1000,all,all,50</span>: Get a total of
1000 comments over all, 50 replies per thread.
<span className="settings-current">1000,all,all,50,2</span>: Get a total
of 1000 comments over all, 50 replies per thread, only 2 levels of depth.
</li>
<li>
Values are in the format:{' '}
<span className="settings-current">
max-comments,max-parents,max-replies,max-replies-per-thread
max-comments,max-parents,max-replies,max-replies-per-thread,max-depth
</span>
.
</li>
@ -702,13 +680,13 @@ const SettingsApplication = () => {
.
</li>
<li>
The PO Token <i>(Proof of origin token)</i> can authenticate your request.
Make sure to read the{' '}
The PO Token Provider URL running external to tubearchivist. Make sure to
review{' '}
<a
target="_blank"
href="https://github.com/yt-dlp/yt-dlp/wiki/PO-Token-Guide"
href="https://docs.tubearchivist.com/settings/application/#po-token-provider-url"
>
PO guide
User Guide
</a>
</li>
</ul>
@ -763,52 +741,16 @@ const SettingsApplication = () => {
</div>
<div className="settings-box-wrapper">
<div>
<p>Add PO Token</p>
</div>
<div>
{response?.appSettingsConfig?.downloads.potoken ? (
<>
<p>PO Token enabled.</p>
<button onClick={handlePoTokenRevoke} className="danger-button">
Revoke
</button>
</>
) : (
<p>PO Token disabled</p>
)}
{showPoTokenForm ? (
<div>
<input
type="text"
value={poTokenFormData}
onChange={async e => {
setPoTokenFormData(e.target.value);
}}
/>
{poTokenFormData !== 'web+' && (
<button onClick={handlePoTokenUpdate}>Update</button>
)}
<button
onClick={() => {
setShowPoTokenForm(false);
setPoTokenFormData('web+');
}}
>
Cancel
</button>
</div>
) : (
<div>
<button
onClick={async () => {
setShowPoTokenForm(true);
}}
>
Update PO Token
</button>
</div>
)}
<p>PO Token Provider URL</p>
</div>
<InputConfig
type="text"
name="downloads.pot_provider_url"
value={potProviderUrl}
setValue={setPotProviderUrl}
oldValue={appSettingsConfig.downloads.pot_provider_url}
updateCallback={handleUpdateConfig}
/>
</div>
</div>
<div className="info-box-item">
@ -974,10 +916,9 @@ const SettingsApplication = () => {
</p>
<br />
{restoringSnapshot && <p>Snapshot restore started</p>}
{!restoringSnapshot &&
snapshots.snapshots &&
snapshots.snapshots.map(snapshot => {
return (
{!restoringSnapshot && snapshots.snapshots && (
<>
{snapshots.snapshots?.slice(0, visibleSnapshotCount).map(snapshot => (
<p key={snapshot.id}>
<Button
label="Restore"
@ -992,8 +933,17 @@ const SettingsApplication = () => {
<span className="settings-current">{snapshot.duration_s}s</span> to
create. State: <i>{snapshot.state}</i>
</p>
);
})}
))}
{visibleSnapshotCount < snapshots.snapshots.length && (
<Button
label="Load More"
onClick={() => {
setVisibleSnapshotCount(visibleSnapshotCount + 10);
}}
/>
)}
</>
)}
</>
)}
</div>

1
frontend/src/pages/SettingsScheduling.tsx
View File

@ -103,6 +103,7 @@ const SettingsScheduling = () => {
}, [refresh]);
useEffect(() => {
// eslint-disable-next-line react-hooks/set-state-in-effect
setRefresh(true);
}, []);

3
frontend/src/stores/AppSettingsStore.ts
View File

@ -23,14 +23,13 @@ export const useAppSettingsStore = create<AppSettingsState>(set => ({
format: null,
format_sort: null,
add_metadata: false,
add_thumbnail: false,
subtitle: null,
subtitle_source: null,
subtitle_index: false,
comment_max: null,
comment_sort: 'asc',
cookie_import: false,
potoken: false,
pot_provider_url: null,
throttledratelimit: null,
extractor_lang: null,
integrate_ryd: false,

17
requirements-dev.txt
View File

@ -1,10 +1,11 @@
-r backend/requirements.plugins.txt
-r backend/requirements.txt
ipython==9.7.0
pre-commit==4.4.0
pylint-django==2.6.1
pylint==3.3.9
pytest-django==4.11.1
pytest==9.0.1
python-dotenv==1.2.1
ipython==9.12.0
pre-commit==4.5.1
pylint-django==2.7.0
pylint==4.0.5
pytest-django==4.12.0
pytest==9.0.2
python-dotenv==1.2.2
requirementscheck==0.1.0
types-requests==2.32.4.20250913
types-requests==2.33.0.20260327