rainbownapkin/canopy
1
0
Fork 0
Code Issues 2 Pull requests Actions Packages Projects Releases Wiki Activity

Cross-Service YT-DLP Scraping Util #126

New issue
Closed
opened 2025-05-04 23:23:00 -04:00 by rainbownapkin · 15 comments
rainbownapkin commented 2025-05-04 23:23:00 -04:00 (Migrated from gitlab.com)
Copy link

Create a local YT-DLP Metadata Scraping Util.

Unfortunately reliably working piped/invidious instances with public IP's have pretty much vanished, the few that remain have disabled API and added captchas in desperation.

This is our quickest/easiest option for pulling YT data ourselves without registering some bullshit API key.

Otherwise, if our IP gets banned by google, we may have to resort to hosting our own piped/invidious instance locally(either on machine or LAN), while running it off of a separate global IP. Though it would still be good for other sources, or smaller canopy instances which might slip through the cracks.

Create a local YT-DLP Metadata Scraping Util. Unfortunately reliably working piped/invidious instances with public IP's have pretty much vanished, the few that remain have disabled API and added captchas in desperation. This is our quickest/easiest option for pulling YT data ourselves without registering some bullshit API key. Otherwise, if our IP gets banned by google, we may have to resort to hosting our own piped/invidious instance locally(either on machine or LAN), while running it off of a separate global IP. Though it would still be good for other sources, or smaller canopy instances which might slip through the cracks.
rainbownapkin commented 2025-05-04 23:23:00 -04:00 (Migrated from gitlab.com)
Copy link

added #124 as parent issue

added #124 as parent issue
rainbownapkin commented 2025-05-04 23:24:19 -04:00 (Migrated from gitlab.com)
Copy link

changed the description

changed the description
rainbownapkin (Migrated from gitlab.com) closed this issue 2025-05-06 07:50:16 -04:00
rainbownapkin commented 2025-05-06 07:50:17 -04:00 (Migrated from gitlab.com)
Copy link

Canopy Custom Media Object Creation from YT-DLP partially-implemented: 0ce0685fd5

Still need to refresh on queue from playlist, or upon playback if it was queued more than a couple of minutes ago.

Also need to fix queueing functions to wait until after media metadata has arrived to set the start date if starting now.

Canopy Custom Media Object Creation from YT-DLP partially-implemented: 0ce0685fd558e2f3ac1024a0bab3980463e6fded Still need to refresh on queue from playlist, or upon playback if it was queued more than a couple of minutes ago. Also need to fix queueing functions to wait until after media metadata has arrived to set the start date if starting now.
rainbownapkin (Migrated from gitlab.com) reopened this issue 2025-05-06 07:50:35 -04:00
rainbownapkin commented 2025-05-06 07:57:16 -04:00 (Migrated from gitlab.com)
Copy link

mentioned in task #127

mentioned in task #127
rainbownapkin commented 2025-05-06 07:57:49 -04:00 (Migrated from gitlab.com)
Copy link

changed title from YTDLP Backup Puller to YT-DLP Scraping Util

<p>changed title from <code class="idiff">YTDLP <span class="idiff left right deletion">Backup Puller</span></code> to <code class="idiff">YT<span class="idiff left addition">-</span>DLP <span class="idiff right addition">Scraping Util</span></code></p>
rainbownapkin commented 2025-05-06 07:57:58 -04:00 (Migrated from gitlab.com)
Copy link

changed title from YT-DLP Scraping Util to Cross-Service YT-DLP Scraping Util

<p>changed title from <code class="idiff">YT-DLP Scraping Util</code> to <code class="idiff"><span class="idiff left right addition">Cross-Service </span>YT-DLP Scraping Util</code></p>
rainbownapkin commented 2025-05-06 07:58:27 -04:00 (Migrated from gitlab.com)
Copy link

changed the description

changed the description
rainbownapkin commented 2025-05-06 08:01:50 -04:00 (Migrated from gitlab.com)
Copy link

changed the description

changed the description
rainbownapkin commented 2025-05-06 08:02:43 -04:00 (Migrated from gitlab.com)
Copy link

changed the description

changed the description
rainbownapkin commented 2025-05-06 08:02:58 -04:00 (Migrated from gitlab.com)
Copy link

changed the description

changed the description
rainbownapkin commented 2025-05-06 08:03:10 -04:00 (Migrated from gitlab.com)
Copy link

changed the description

changed the description
rainbownapkin commented 2025-05-06 21:15:41 -04:00 (Migrated from gitlab.com)
Copy link

Youtube videos now refresh raw file link 10 seconds before playback starts to ensure it's ready to be sent to the client. In the advent the pull takes longer than 10 seconds, the update will be sent out live to the clients so they can hotswap the video and play the remainder: 60cd21d938

Youtube videos now refresh raw file link 10 seconds before playback starts to ensure it's ready to be sent to the client. In the advent the pull takes longer than 10 seconds, the update will be sent out live to the clients so they can hotswap the video and play the remainder: 60cd21d9381efc9e7a4bb12442a6e171d82d4d86
rainbownapkin commented 2025-05-06 21:23:45 -04:00 (Migrated from gitlab.com)
Copy link

Still need to refresh raw links for:

  • Items re-hydrated from playlist or after server restart that begin in less than 10 seconds

  • Items re-hydrated after server restart that begin in less than 10 seconds or have already started

  • Items which where already queued, and then moved to start within 10 seconds

-- OR --

  • Re-write refreshRawLink() to check link expiration to only refresh when required, then add refreshRawLink() call to items scheduled to start within 10 seconds.

This last option is probably the cleanest, and solves other issues while also reducing the amount of youtube scrapes.

Still need to refresh raw links for: - Items re-hydrated from playlist or after server restart that begin in less than 10 seconds - Items re-hydrated after server restart that begin in less than 10 seconds or have already started - Items which where already queued, and then moved to start within 10 seconds -- OR -- - Re-write refreshRawLink() to check link expiration to only refresh when required, then add refreshRawLink() call to items scheduled to start within 10 seconds. This last option is probably the cleanest, and solves other issues while also reducing the amount of youtube scrapes.
rainbownapkin (Migrated from gitlab.com) closed this issue 2025-05-06 22:47:54 -04:00
rainbownapkin commented 2025-05-06 22:47:55 -04:00 (Migrated from gitlab.com)
Copy link

refreshRawLink() now checks link expiration to make sure link actually requires a refresh, scheduleMedia() refreshes any items that start before the preSwitchDelta: 2a3740dece

Looks like youtube scraping for YT-DLP is complete. I'll play around with Dailymotion and Vimeo before continuing...

refreshRawLink() now checks link expiration to make sure link actually requires a refresh, scheduleMedia() refreshes any items that start before the preSwitchDelta: 2a3740dece054117e7e89103207b42efa7110e11 Looks like youtube scraping for YT-DLP is complete. I'll play around with Dailymotion and Vimeo before continuing...
rainbownapkin (Migrated from gitlab.com) reopened this issue 2025-05-06 22:48:44 -04:00
rainbownapkin commented 2025-05-06 23:32:06 -04:00 (Migrated from gitlab.com)
Copy link

Dailymotion looks workable, though we'll need to get HLS playback working before we can get it playing proper. This would also benefit youtube playback, on top of being a requirement for livestreams, so that'll be a big improvement.

Vimeo seems to pull unreliably from YT-DLP, however it has an API that works just fine anonymously which should at least be enough to get their official embed working...

Dailymotion looks workable, though we'll need to get HLS playback working before we can get it playing proper. This would also benefit youtube playback, on top of being a requirement for livestreams, so that'll be a big improvement. Vimeo seems to pull unreliably from YT-DLP, however it has an API that works just fine anonymously which should at least be enough to get their official embed working...
rainbownapkin (Migrated from gitlab.com) closed this issue 2025-05-06 23:32:11 -04:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
dev
0.4-indev-hotfix-3
0.4-indev-hotfix-2
0.4-indev-hotfix-1
0.4-indev
0.3-indev-hotfix-1
0.3-indev
verbose-queue
0.2-indev
0.1-indev
No results found.
Labels
Clear labels   
Bug

   
Cleanup/Refactor

   
Core Feature

   
Documentation

   
Feature

   
Performance Improvement

   
Security Improvement

   
UX/Accessibility

   
Unreproducable Bug
No labels
Bug
Cleanup/Refactor
Core Feature
Documentation
Feature
Performance Improvement
Security Improvement
UX/Accessibility
Unreproducable Bug
Milestone
Clear milestone
No items
No milestone
Canopy v0.3-Indev
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: rainbownapkin/canopy#126
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?