Tools of-scraper now with DRM support

graytin6

graytin6

Lurker
Joined
Apr 1, 2023
Posts
21
Reaction score
0
ofscraper
ofscaper is a fork of onlyfans-scraper with additional features and fixes.
Many of these changes are in regards to file management. For example with the changes made you can control which types of files are downloaded, and can have more control of the final destination of these files. We've also added support for metadata files.
For those who are currently on digitalcriminals script. We are using the same user_data.db structure. So those files are completely compatible. Provided you use a proper setting for metadata in your config.json. Your user_data.db can be used to skip previously downloaded files.
Running
Code: 
ofscraper
Allows you to not only scrape content, batch likes and unlikes
It also allows you to manage most of the program. For example switching profiles, changing confing settings, also reseting auth file
For users that are comfortable with the commandline we provide the ability to enhance automation and skip any menus
For example a command I have been using has been
Code: 
ofscraper --posts all --purchased --renewal disabled --sub-status active --account-type paid
Which is meant to only scrape all wall content and paid content. For only paid accounts, that are active, and have renewal off
There is also a daemon available
Code: 
--daemon {minutes}
Changes
Many of the changes can be found at changes.md
However some of the ones I like are
- scraping messages, and paid content for specific models
- filtering by account type, subscription status, and renewal status
- fuzzy search during model selection
- ability to select specific type of posts to download
- background daemon support
This is meant to be a very straight forward program. Right now the plan is to only support onlyfans, as there are other scripts that can support fansly for example.
Links
Repo
Pypi
 
Quickly tried this script, and it works super well. I did run into four small bugs however:
Firstly I got run failed with exception: 'text' for a single profile, resulting in no downloads at all. I've not been able to replicate this on any other profiles however.
Code: 
? Which models do you want to scrape: auroraangel1
Status - UP
? Would you like to also download paid content Yes
Using profile: main_profile
Status - UP
Getting Selected post type(s) for auroraangel1
Subscription Active: True
? Which area(s) would you like to scrape? (Press ENTER to continue) ['Timeline', 'Archived', 'Highlights', 'Stories', '
Messages']
Name: aurora angel | Username: auroraangel1 | ID: 66748071 | Joined: 2020-09-03T00:00:00+00:00
- 501 posts
-- 452 photos
-- 207 videos
-- 0 audios
- 2 archived posts
Pages Progress: 12: : 12it [01:05, 5.49s/it]
run failed with exception: 'text'
Using profile: main_profile[01:03, 4.17s/it]
Status - UP
Getting paid content for auroraangel1
Scraping, Scraping isn't frozen. It takes time.
Scraped Page: 1

Username: auroraangel1
- paid content 0
-- photos 0
-- videos 0
-- audios 0

Done With Run
The second bug is that the script, for some reason, is skipping a single file when scraping. Trying both with and without --dupe doesn't change this. However, running the scrape 1 or 2 more times on the same profile seems to fix this and the mysteriously skipped file is downloaded. I'm not sure exactly what file it is, sadly.
The third bug is that the script sometimes doesn't download any of the archived content of a profile. --dupe doesn't change this, nor does selecting only archived content. :SadgeCry:
Code: 
? Which models do you want to scrape: avaxreyes
Status - UP
? Would you like to also download paid content Yes
Using profile: main_profile
Status - UP
Getting Selected post type(s) for avaxreyes
Subscription Active: True
? Which area(s) would you like to scrape? (Press ENTER to continue) ['Timeline', 'Archived', 'Highlights', 'Stories', '
Messages']
Name: Ava Reyes 💖 | Username: avaxreyes | ID: 125000045 | Joined: 2021-03-11T00:00:00+00:00
- 261 posts
-- 286 photos
-- 8 videos
-- 0 audios
- 30 archived posts
Pages Progress: 9: : 9it [00:05, 1.60it/s]
Pages Progress: 3: : 3it [00:46, 15.52s/it]
forcing all downloads

Downloading to ofscraper/{model_username}/{responsetype}/{mediatype}/

Progress: (300 photos, 8 videos, 0 skipped || 308/308||150.15 MB): : 308it [00:43, 7.08it/s]
Using profile: main_profile
Status - UP
Getting paid content for avaxreyes
Scraping, Scraping isn't frozen. It takes time.
Scraped Page: 1

Username: avaxreyes
- paid content 0
-- photos 0
-- videos 0
-- audios 0

Done With Run
And lastly, the fourth bug is that avatars are downloaded without a file extension. It's not an important thing for myself, but I thought I'd share it just in case someone needs it. 👍
Other than those four small bugs, the script works absolutely perfect and is super easy to use. Thanks for sharing! :MLADY:👍
 
Thanks for the kind words, and also the bug reports are very helpful
 
I've received a lot of help in improving this script like in the post quoted
With the release of 1.82
I thought I would go on here and see if some of he bugs here hand been dealt with
With that I've been able to fix a lot of issues with earlier releases, and implement ways to improve the script.
1. The text error can mean the config file is not correct or it can mean that the response from onlyfans does not have a expected value. Because this only happens with a specific model. It is probably the latter. I've revamped the script to do a better job at managing all the different types of responses onlyfans gives. I've received of reports of similar errors, but most of those seem to have been gone with later updates.
2. The failed downloads are most likely from onlyfans side, sometimes they will just send a bad response. Prior releases were meant to retry request. However this was not implemented correctly. With the new version I've corrected this. So now each download will be tried 5 times. This is done with a pause. Hopefully this will greatly reduce or remove skipped downloads
3. This I'm not sure about. There is a good chance that it has been fixed with all the changes made to the script.
4. I've implemened a fix for this in 1.82
 
I removed 1.82 because of a bug I introduced in that version
the latest version is 1.87
 
Thanks, hornyone! I never got around to using digitalcriminal and was downloading content manually... Tried out your script and it has made life so much easier.
 
Added a beta feature
It's a way to quickly visual information from the db, along with information gathered from the api
This is part of the upcoming 2.2 release
There will probably be a way to download individual posts that are unlocked this way, along with being able to do so directly from the command line
dp-Y8eWwsgY.png
 
Not sure if I'm doing something wrong or there's a way to do this but scraping DRM messages is not appending the quality to the filename (ex. _source). I have not tried with posts yet as I'm just using the tool for messages so far.
 
Only some files have _source added to them
It depends on the url that the API returns
 
I don't think this is true. Cause I used sim0n00ps scraper for the same messages and it returned with _source appended, while yours did not for the same files.
 
Yeah I've been using what the original url would state
I took a quick look at his code and he is appending that to the filename
folder + path + "/" + filename + "_source.mp4"
Some files do have it within what the url, but it is not consistent
I'll probably just do something similar to avoid confusion
 
Using ofscraper 2.4.5 on macOS. It downloaded everything in my purchases nicely... except for the DRM video isn't properly downloaded. It downloads the DRM video, except the DRM video can't be opened. MacOS QuickTime (the software used to play mp4s on macOS) says: "The file isn’t compatible with QuickTime Player.".. I also tried uploading that file to gofile to see if it would play on there, and it won't.
I'm using this for mp4decrypt: https://formulae.brew.sh/formula/bento4
And I also pointed my config at ffmpeg's executable (I already use ffmpeg regularly)
When inspecting the metadata of the DRM video that it downloaded that won't open... I noticed that the difference between it and the other non-DRM videos is that the DRM one has no information about dimensions, codec, etc. It's missing that metadata, which is my assumption to why it might be broken for me. Any ideas?
 
Also, how do I re-attempt downloads on previously downloaded content? I noticed that when I rerun on a model's messages, it won't download those that I've downloaded in the past?
 
The issue is with the extension
Some video players are able bypass this.
From the discord it seems vlc is one of them
however changing the names to .mkv or
redownloading with --dupe will fix the issue
https://github.com/datawhores/OF-Scraper/issues/85
you can run
ofscraper --dupe
This will skip the dupecheck and redownload every file
 
Redownloaded the DRM file, and it was successful and everything works. Thank you for your work!
 
Hello, I downloaded a DRM-paid video (message) and I would like to download it again, how I can do it? The file is marked downloaded by the DB.
 
You must log in or register to reply here.
Back
Top