36 replies to this topic

[fuzzyk]

I found the existing possibilities to export “My Follows” totally inadequate, so I rolled my own. Bad news first: It’s a Windows-only application.

It’s available on GitHub: https://github.com/fuzzykiller/batoto-grabber

You can also download prebuilt binaries from the Releases page.

Usage:

• Log in
• Select languages relevant to you on main page
• Start
• It crawls series
• It crawls groups (progress bar starts again)
• Select file name to save to
• It saves the database, then downloads cover images (progress bar starts again)
• A message box will appear when it's done

To make pages load faster, loading images is disabled. However, it will wait 100ms between page loads. All in all, it takes ~3 seconds per series/group on my super fast internet connection. YMMV

This whole process will get you an SQLite database with information on:

• Series (basically everything that’s visible on a series’ page, including the cover image
• All chapters (title, which group, which language, ...)
• Groups that translated those chapters (including their website URL if set)

System Requirements

• Windows 7 or higher
• .NET Framework 4.5.2
• Visual C++ 2013 Redistributable x86
• Lots of RAM (at least 2 GiB of free memory) for browser cache and whatnot
• Batoto reachable at https://vatoto.com(I could change this later, provided all the series data is still available)
• A reasonable amount of disk space (takes 90 MiB for the 279 series I follow, depends largely on the cover images)

Handled error conditions

• Cloudflare DDoS protection
• Cloudflare timeout pages
• Random timeouts
• Removed series (2 types)
• Removed translation groups
• Missing/otherwise unavailable cover images

If you want to modify it or whatever, you can use Visual Studio 2017 Community Edition (or higher). If you want to create something similar for different platforms, feel free to use the JavaScript snippets I created!

Disclaimer: I provide no warranties whatsoever.

Edited by fuzzyk, 21 January 2018 - 12:34 AM.

[ZdrytchX]

Edited by ZdrytchX, 16 January 2018 - 11:15 AM.

[James20]

Nice work fuzzyk, 73 follows and 164 groups exported without issue.

 

Though why not json? That way a website could display the data. Its fine though, there are plenty of converters for that purpose anyway.

[fuzzyk]

Though why not json? That way a website could display the data. Its fine though, there are plenty of converters for that purpose anyway.

Because it's relational data. That's why it's a database. You could also easily create a website or program to access this data directly.

---

Just a small heads-up: I found that contributors are currently not saved. So if that's important to you, you'll have to repeat the export when the big is fixed.

Edited by fuzzyk, 16 January 2018 - 07:43 PM.

[fuzzyk]

I just uploaded version 1.4 that fixes the issue reported above, correctly saves the contributor name and uses the other "My Follows" page to mark the chapters you last read (when possible).

[ZdrytchX]

Mine just finished after about 2 hours, dumping an sqlite file. I just realised that MS access isn't compatible because it's not the same type of database D:

What are your recommended ways of viewing the data?

[fuzzyk]

For quick checks, I usually use http://sqlitebrowser.org/

 

When everything is sorted, I’ll probably create a viewer.

 

That being said, if you install a SQLite3 ODBC driver, you should be able to make Access interact with the database.

[Xerx]

For quick checks, I usually use http://sqlitebrowser.org/

 

When everything is sorted, I’ll probably create a viewer.

 

That being said, if you install a SQLite3 ODBC driver, you should be able to make Access interact with the database.

Thank you.

Anyway you could work on a feature to import the follow list to something like mangaupdates?

[fuzzyk]

I'm not familiar with MangaUpdates. Generally, unless a service offers an API that allows third-party applications to modify your lists, this is next to impossible.

I understand it's very tedious to subscribe to hundreds or even thousands of series manually, but I don't see any other way. Sorry!

[salfredo]

Man this deserves a Medal

[Shadow Alpha]

Been trying to figure out how to actually run this program. I don't know much about javascript but would love to get this working. Is there a specific program i need to use to compile the list? Is there a file i am suppose to click to get it running? Does it require a specific browser to use it or need to be somehow loaded into the browser?

[fuzzyk]

Been trying to figure out how to actually run this program. I don't know much about javascript but would love to get this working. Is there a specific program i need to use to compile the list? Is there a file i am suppose to click to get it running? Does it require a specific browser to use it or need to be somehow loaded into the browser?

 

Just download the latest release from GitHub. It’s a ZIP file that contains "BatotoGrabber.exe". When you start this application, it looks like this:

 

 BatotoGrabber.png   25.09KB   0 downloads

 

Just log in, return to the start page (by clicking on the "Logo" image placeholder in the upper left part of the page), select the languages you’re interested in and start the process.

Edited by fuzzyk, 18 January 2018 - 08:45 AM.

[Shadow Alpha]

I can't actually find a .exe in the zip, that's why i was confused.

All i see is https://imgur.com/a/b7LUj in the folder, and when i look through github page i see no .exe in any of the folders. Am i just blind?

Edit: I checked the previous version and it has the .exe.

Edited by Shadow Alpha, 18 January 2018 - 08:59 AM.

[fuzzyk]

You downloaded the source code archive. You need to download BatotoGrabber-v1.4.zip (or a newer version).

[Shadow Alpha]

Is the file at a different link then the first on in your original post, because then i will need to get a previous version because i can’t find the exe in the version listed when i click that link.

Edit: spelling

Attached Files

•  6DAB199A-990B-4EE9-B40A-2EA37FFBD47A.png   308.86KB   0 downloads

Edited by Shadow Alpha, 18 January 2018 - 11:02 AM.

[ZdrytchX]

Is the file at a different link then the first on in your original post, because then i will need to get a previous version because i can’t find the exe in the version listed when i click that link.

Edit: spelling

In that screenshot, click "releases" tab
or click the link provided for prebuilt binaries (.exe are binaries) in the OP post and download BatotoGrabber-v1.4.zip from version 1.4

Also here are a few screenshots in regards the kind of data it takes

Spoiler

Edited by ZdrytchX, 18 January 2018 - 11:15 AM.

[Shadow Alpha]

In that screenshot, click "releases" tabor click the link provided for prebuilt binaries (.exe are binaries) in the OP post and download BatotoGrabber-v1.4.zip from version 1.4

Ahhhh ok, i see now thanks

[Shadow Alpha]

Ok so having tried running the grabber 3 times i have gotten the same issue all 3 times. While the program is running through the various groups warning pops up that says "CefSharp.BrowserSubProcessor has stopped working". It seems to happen at the same point every time at 236 out of 263 groups scanned. The scan did finish, but i haven't had the time to check to see if the file made works correctly. Any ideas? Is this a potentially serious issue? Is there any way i can provide more info to help fix this issue?

 

https://imgur.com/a/YtgUZ

[asdfblarg]

I guess I probably should have asked for the 64-bit first ...

I had 3900 follows exactly. And 3044 groups. 

 

Spoiler

 

Spoiler

[Jpwong]

Not sure that 64 bit would have worked any better.  It's trying to hold all the data in memory it looks like and you may have run out of available RAM or something.  Better if it were set to flush the data to disk every 100 manga or something so it can recover the stack space.