madmccoy

... What's holding us back?

The technical portions of a replicating bato.to are within the skillset of many developers, I like to think (I have several years experience, so I might be overestimating skill levels here). The bigger problems are nontechnical:

- DMCA form letters

- Cease and Desist orders from lawyers.

- The occasional lawsuit.

- Having to sort through server hosts to find one who won't fold at the first whiff of a legal problem.

Beyond those, you've got smaller but still niggling issues:

- Can you convince enough trustworthy people to be moderators?

- What happens if a moderator goes rogue? 

- Who has access to what? Why?

- Will anyone even use it?

I ran a website with user generated content for several years, and the number of spurious letters from law firms that showed up (literally showed up, in physical mail form) was amazing. Just because they're wrong, or the content being used is covered under fair use, doesn't mean you just ignore them. A good chunk of them have arbitrary deadlines; they'll even CC the legal department of your server provider the same half-assed printed-then-digitally-scanned letter in an effort to get them to immediately take your site down. And, depending on your provider, they will. Or maybe they won't. Or maybe they won't,for a while, until they get annoyed and are tired of dealing with your account.

You can automate a lot of things, but dealing with humans is still a manual process.

>  have a REALLY  BIG  follow list (maybe around a hundred), so i'm currently on a nightmare.

For now, you can select your entire follows list via your mouse cursor, copy (CTRL+C), paste into a new Google Document (CTRL+V), which will get you a newline delimited list of manga in that document. However, every entry will be a hyperlink to the associated bato.to page. If you want sanitized results -- say, you're going to feed the list of names into a software program like I am -- then select the entire list in Google Docs, copy the selection (CTRL+C), then paste that list without formatting by pressing CTRL+SHIFT+V. 

That will give you a simple, newline delimited list of manga you follow. You'll have to edit out the lines that are just headers ("#", "A", "B", etc), but that's only 26 lines you have to clean up by hand.

What you do with the resulting list is up to you, though. You can get the manga information at your leisure by looking them up one eat a time via baka updates, or scripting something yourself. Either way, this assures you have a complete list of all your follows.

well I don't think fully members only is what is gonna happen. Just old stuff.
 
I'm interested in your opinion as a professional, what are all the alternatives aside from implementing some sort of API to make bot work easier and lessen the server request load? Coz as it stands it is do or die.

Noting that this advice has no warranties, yadda yadda yadda, and putting on my "professional" hat and ignoring any ideologies or concepts about what it means to be a manga reader: 

Yank out the current reader and replace it with one that is run entirely on the client (eg, web browser). On the first visit to any page of the new reader, the initial page load cost would be ~100KB, assuming no fancy high-res assets for the reader itself. For each manga chapter viewed, there would be an additional ~20KB cost to communicate with the backend in order to retrieve a list of images that belong to the current chapter. From then on, when viewing a chapter, the only additional cost is downloading an image directly from the bato.to's CDN.

Judging by the number of assets queried on each page load in the reader, you'd also reduce HTTP requests to the server in general.

Pros:

* sharp reduction of resource utilization by returning visitors who view more than one manga chapter.

* mobile applications can use the same backend channel to display pages from bato.to

* this will break existing scrapers. When they investigate the site changes, they will discover how the new reader communicates with the backend, they may simply begin using that channel instead, reducing overall server utilization.

Cons:

* this will break existing scrapers. When they investigate the site changes, they will discover how the new reader communicates with the backend, they may simply begin using that channel instead, reducing their own resource utilization. (see intro statement)

* This is a suboptimal technique if you have a lot of hit and run viewers: new readers, with no cached assets, who load the viewer, view a single page of a chapter, and then never return to the site.

* requires a programmer with a strong knowledge of frontend engineering and backend engineering. No, I'm not available.

Disclaimers: any and all numbers above are based on my own experience migrating existing and developing greenfield sites to/with the described model of interaction. Since I don't know what the exact utilization numbers are for bato.to, I can't give any specific performance percentages impacted.