46 replies to this topic

[HimekoTachibana]

I'm curious about how this affects AMR now, what are you guys going to do? Is it possible to divert this in any way, aviar, sneezemonkey?

[Daktyl]

It's definitely possible to stop AMR from getting you auto-banned, but Aviar will have to change how often it scrapes and a few other behaviors. Honestly there's better ways of going about what it does in the first place

[HimekoTachibana]

It's definitely possible to stop AMR from getting you auto-banned, but Aviar will have to change how often it scrapes and a few other behaviors. Honestly there's better ways of going about what it does in the first place

The problem honestly though is that I use AMR for more than one manger reader lol Are there any good alternatives that does what I utilize AMR to do?

 

1. Real-time updates for Manga chapter uploads over multiple manga readers.

2. Loading images one-page format for seemless scrolling/reading.

3. Pre-fetching pages and chapters when reading so you don't need to wait for a load between chapters.

 

Those three things in one extension is basically what I need, if that exists then by all means I would be happy to switch over to it!~

 

Right now #1 is my biggest thing because #2 and #3 I already have a script that my friend made for me which is how I currently read Batoto in one-page. I still use AMR for #1 though.

Edited by HimekoTachibana, 02 November 2015 - 12:12 AM.

[aviar]

First of all, thanks for bringing this to my attention, I don't generally scan all the latest threads so I would have missed it otherwise.

 

Everything not NP is possible, and even for NP complete it is possible if the data set is small enough or there is a heuristic for it. Being a bit more serious, I'd assume it's completely possible, Grumpy himself has given a diagnoses and solution. An immediate thing that comes to mind is having a field that permits setting one of the following: number of times to retry a fetch, how much time should elapse between fetches, or how many times to retry before failing. This would mean it's not real-time, but I think for practical purposes it is enough.

 

As for prefetching logic, I don't think I've crossed the internal code that does it yet, but I would hope that it doesn't try to fetch all chapters and all pages for a manga. Instead it should probably fetch all pages for the current chapter, and n pages for each other chapter.

 

I'm reading up a bit (literally a bit, odd pages here and there) on JQuery and Chrome App/Extensions when I can, might be able to improve the code a bit (at the very least document what I go through). For Javascript I'm just reading the code, so I can only use whatever subset is used of the complete language. What I am saying is I'm probably not the ideal person to reprogram it, but from the little I know I think it's within the realm of things I can do.

 

P.S.: Thinking on it, its probably possible to identify error pages by their content and thus fail silently or present a message to the user.

 

P.S.S.: The whole update chapters list seems pretty overkill, I'd even consider getting the manga list for a site overkill. Might as well just let the user query the tool for a manga and it searches supported sites for that particular manga, offers up the user a list of sites that have it and permits the user to select which one to follow.

Edited by aviar, 02 November 2015 - 01:19 AM.

[sneezemonkey]

The problem honestly though is that I use AMR for more than one manger reader lol Are there any good alternatives that does what I utilize AMR to do?

 

1. Real-time updates for Manga chapter uploads over multiple manga readers.

2. Loading images one-page format for seemless scrolling/reading.

3. Pre-fetching pages and chapters when reading so you don't need to wait for a load between chapters.

 

Those three things in one extension is basically what I need, if that exists then by all means I would be happy to switch over to it!~

 

Right now #1 is my biggest thing because #2 and #3 I already have a script that my friend made for me which is how I currently read Batoto in one-page. I still use AMR for #1 though.

#2 You could try my script in signature (Shameless advertising).

 

#1 any alternative that does this will likely get you banned as well. They generally don't bother with timers (at least of sufficient intervals) so that updates appear faster.

 

@aviar

 

I have some ideas I'll PM you.

Edited by sneezemonkey, 02 November 2015 - 04:04 AM.

[HimekoTachibana]

#2/3 You could try my script in signature (Shameless advertising).

 

#1 any alternative that does this will likely get you banned as well. They generally don't bother with timers (at least of sufficient intervals) so that updates appear faster.

 

@aviar

 

I have some ideas I'll PM you.

AMR Does #1 though, unless I'm wrong? http://i.imgur.com/Qi8GnIJ.png

 

Thanks for all the effort you're putting into this guys, truly, I'm really grateful.

Edited by HimekoTachibana, 02 November 2015 - 03:54 AM.

[sneezemonkey]

AMR Does #1 though, unless I'm wrong? http://i.imgur.com/Qi8GnIJ.png

 

Thanks for all the effort you're putting into this guys, truly, I'm really grateful.

He was asking for alternatives since AMR seems to be getting people banned in current form.

[Daku]

Went ahead and fixed AMRs bato.to support, also added local file loading to make it easier to try and fix in the future.

 

Simply download & load this: https://github.com/DakuTree/AMR/tree/custom

It should replace your current install, although settings will still be the same.

 

Few things to note:

This uses my own fork of the mirrors repo since the main one seems to be inactive (The main dev was last active in late August). This can be found here: https://github.com/DakuTree/mirrors/tree/develop

To load local mirrors, stick the file in js/mirrors and add the name to the localMirrors array in mgEntry.

Any series you are following on bato.to via AMR will probably need to be removed & re-added. URL change kinda screwed with things, and there is no easy way to get the new URLs since it didn't properly store the main manga URL before.

This (probably) does not fix the ban issue pointed to in https://vatoto.com/forums/topic/23820-automated-ban-appeal-here. It should probably help though, since it's no longer endlessly failing when trying to load images. I'll try and properly fix it when I get the time.

 

[HimekoTachibana]

 

Went ahead and fixed AMRs bato.to support, also added local file loading to make it easier to try and fix in the future.

 

Simply download & load this: https://github.com/DakuTree/AMR/tree/custom

It should replace your current install, although settings will still be the same.

 

Few things to note:

This uses my own fork of the mirrors repo since the main one seems to be inactive (The main dev was last active in late August). This can be found here: https://github.com/DakuTree/mirrors/tree/develop

To load local mirrors, stick the file in js/mirrors and add the name to the localMirrors array in mgEntry.

Any series you are following on bato.to via AMR will probably need to be removed & re-added. URL change kinda screwed with things, and there is no easy way to get the new URLs since it didn't properly store the main manga URL before.

This (probably) does not fix the ban issue pointed to in https://vatoto.com/forums/topic/23820-automated-ban-appeal-here. It should probably help though, since it's no longer endlessly failing when trying to load images. I'll try and properly fix it when I get the time.

 

Oh my god... It actually works! It does everything that it used to do. I set the setting to update the chapter list for every 6 hours so hopefully this prevents the auto-ban. If not, I'll try 24 hours instead. You have no idea how happy I am that it's actually working now! 

 

The only caveat is like you said, I need to re-add the Batoto manga individually for it to update the URL but that's not a problem at all.

 

@Daku, @sneezemonkey, @aviar, PM me your PayPal emails, I'll give you each $5 if you would like.

 

EDIT: It seems the it's scraping the Non-English chapters in the pop-up but is not showing them when reading the manga which is how it's supposed to be. Is that able to be fixed? 

Edited by HimekoTachibana, 03 November 2015 - 07:19 AM.

[Daku]

 

EDIT: It seems the it's scraping the Non-English chapters in the pop-up but is not showing them when reading the manga which is how it's supposed to be. Is that able to be fixed? 

Just pushed an update which should fix that. Might be worth making sure your language settings are set in http://vatoto.com/myfollowsthough. 

[HimekoTachibana]

Just pushed an update which should fix that. Might be worth making sure your language settings are set in http://vatoto.com/myfollowsthough. 

Hm... I believe setting my language settings to English only for /myfollows broke AMR's ability to find the chapter lists. 

 

I'm going to enable all languages again and see what happens.

 

Also, your fork doesn't have the update on github I don't think.

 

EDIT: I think I broke it somehow >_> Going to uninstall and reinstall it.

Edited by HimekoTachibana, 03 November 2015 - 09:19 AM.

[Daku]

Hm... I believe setting my language settings to English only for /myfollows broke AMR's ability to find the chapter lists. 

 

I'm going to enable all languages again and see what happens.

 

Also, your fork doesn't have the update on github I don't think.

 

EDIT: I think I broke it somehow >_> Going to uninstall and reinstall it.

Weird, I'm having the same issue with that series. Re-adding the series twice seemed to fix it though.

 

Setting the language to english shouldn't cause any issues. All it does is have the site do the filtering so we don't have to.

The update is on github though. Try manually refreshing website implementations.

[HimekoTachibana]

Weird, I'm having the same issue with that series. Re-adding the series twice seemed to fix it though.

 

Setting the language to english shouldn't cause any issues. All it does is have the site do the filtering so we don't have to.

The update is on github though. Try manually refreshing website implementations.

Well'p. It broke. And this might be why, it seems Batoto is deactivated and grayed out. Functionality in the reader works, but scraping chapter lists isn't.

 

When it was working earlier it wasn't grayed out.

 

EDIT: Ahh.... basically every website that I have manga for is grayed out for some reason. 

Edited by HimekoTachibana, 03 November 2015 - 09:52 AM.

[Daku]

Well'p. It broke. And this might be why, it seems Batoto is deactivated and grayed out. Functionality in the reader works, but scraping chapter lists isn't.

 

When it was working earlier it wasn't grayed out.

 

EDIT: Ahh.... basically every website that I have manga for is grayed out for some reason. 

If it's grayed out it just means it's loaded & you have manga there.

 

As for chapter lists, I'm kind of out of ideas of what it could be. On my side I've had very little issue (The only one giving errors was the Accel World one).

Only two things I can suggest are: Remove/re-add the series and click "mark as read" when loading the new series.

[Bachal]

I tried the extension but it doesn't update the Batoto Manga.
I tried exporting and then importing the list again but it's stuck at "No chapters found yet..".
http://imgur.com/v44lstD
 
I also uploaded my manga list on dropbox in case you need to look into it.
https://www.dropbox.com/s/z9l8og7ypn3k39z/2015-10-25%20(1).txt?dl=0
 
edit: Also tried manually adding them, now that I read your note, but it still shows as ""No chapters found yet..".

edit 2: Reinstalled the extension without importing the list of my manga, went to a random manga on batoto added it to my list and it still shows as no chapters found.
Added the export list of that manga in the Spoiler. 

 

Spoiler

{"mangas":"[{\"mirror\":\"Batoto\",\"name\":\"4 Cut Hero\",\"url\":\"http://vatoto.com/comic/_/comics/4-cut-hero-r15771\",\"lastChapterReadURL\":\"http://vatoto.com/reader#eec9a0b9f3132dfd\",\"lastChapterReadName\":\"Ch.16\",\"read\":0,\"update\":1,\"ts\":1446559662,\"display\":0,\"cats\":\"[]\"}]","bookmarks":"[]"}

Edited by Bachal, 03 November 2015 - 02:21 PM.

[swimming95]

Thank you so much for doing this!  Really appreciate the work.  Granted I think Bachal just asked the same question but I am having trouble adding the local mirrors properly so that the chapter list can be found.

 

What I tried doing was:

1. Downloading your set of mirrors and adding all of the .js files to the js/mirrors folder

2. I tried both just listing the name of each website as well as using the json object in the localMirrors variable mgEntry

3. Reloading the unpacked extension in Chrome

4. Going to the manga page and adding to AMR

 

I am sure that I am just doing something stupid but any detailed step by step updating guide is appreciated.  Thank you again so much.

[aviar]

For local mirrors, you might want to just add a file select dialog to options.html (did that), add "file://" to permissions in manifest.json (thanks to sneezemonkey for linking me to the relevant material). Afterward you should be able to have read-only access to the filesystem, and in turn read from local files.The reason I recommend this is simply for user friendliness. 
 
P.S.: What one wants to do with the file would be to simply create the amrcObject structure that is used by amrcsql and execute the pertinent write. Data validation/sanitation optional.
P.S.S.: Sneezemonkey and I hang sometimes on the Batoto IRC, seems like a better way to communicate these things (looks ugly in posts).

 

options.html (add after closing </p> of options_ws_activation, line 656 in my editor) :

<p i18n="options_ws_file"><form id="localMirrorForm"><input type="file" accept=".js" id="mirrorFile"><input type="submit" value="Submit"></form>

 
manifest.json (in permissions property):

 

"file://*/*"

 

 
messages.json (anywhere)

"options_ws_file": {
    "message": "The following form is provided for using third-party Javascript mirror files. To use this, simply select the file that was provided by the third- party, click submit, then select the site from the list below.",
     "description": "JS File upload message"
}

If I program the function to read from the file, I'll pass it to you. Eitherway, it is essentially just repeating the parsing code that's there, except ending with a write to the webdb and site listing refresh instead of all the callbacks. Also is there a reason you forced languages? Was lang_Spanish, lang_English, etc not working?
 

<snip>

@Daku, @sneezemonkey, @aviar, PM me your PayPal emails, I'll give you each $5 if you would like.

<snip>

No need from my part, I'm doing it because I thought it might be helpful. Also Daku did all the work and Sneeze helped me out with reading so yeah, it's more them than me. 

Edited by aviar, 03 November 2015 - 04:47 PM.

[HimekoTachibana]

I still can't seem to get it working. The only thing I can think of is reverting whatever change you made during 

 

Just pushed an update which should fix that. Might be worth making sure your language settings are set in http://vatoto.com/myfollowsthough. 

because it worked perfectly fine prior to that I believe. If that update didn't break it then I have no idea what did. Hummm~ So close.

[Daku]

I've removed the english chapter check, so you'll need to set your language in /myfollows to avoid getting multiple languages in the AMR popup chapter list.

This "should" fix the no chapter error, apparently the class wasn't captilized properly so it couldn't find anything (strangly it sometimes worked and sometimes didn't...).

Importing still seems to be a bit picky, no idea what is going wrong there.

 

Series you are following which still uses the old URL format should now have a new chapter noting it's using the old format (after you refresh chapter lists). This should help when trying to figure out what needs manual re-adding.

AMR no longer tries to get said URLs either, which should help things.

 

Made a very small tweak to the AMR repo to use only git mirrors. It used to fallback to the old ones, but that probably isn't a good idea. You'll need to manually update for this.

 

I tried the extension but it doesn't update the Batoto Manga.
I tried exporting and then importing the list again but it's stuck at "No chapters found yet..".
http://imgur.com/v44lstD
 
I also uploaded my manga list on dropbox in case you need to look into it.
https://www.dropbox.com/s/z9l8og7ypn3k39z/2015-10-25%20(1).txt?dl=0
 
edit: Also tried manually adding them, now that I read your note, but it still shows as ""No chapters found yet..".

edit 2: Reinstalled the extension without importing the list of my manga, went to a random manga on batoto added it to my list and it still shows as no chapters found.
Added the export list of that manga in the Spoiler. 

 

Spoiler

{"mangas":"[{\"mirror\":\"Batoto\",\"name\":\"4 Cut Hero\",\"url\":\"http://vatoto.com/comic/_/comics/4-cut-hero-r15771\",\"lastChapterReadURL\":\"http://vatoto.com/reader#eec9a0b9f3132dfd\",\"lastChapterReadName\":\"Ch.16\",\"read\":0,\"update\":1,\"ts\":1446559662,\"display\":0,\"cats\":\"[]\"}]","bookmarks":"[]"}

Exporting > Importing won't work with old bato.to URLs, you need to manually re-add each one (So remove from your list, then re-add manually by loading a chapter & clicking mark as read/reading the chapter).

The old format saved the first chapter of the manga as the page to check for chapter lists, but since those URLs no longer work, they need to be manually changed. We can't automatically grab the new URL either which kinda sucks :\

 

Thank you so much for doing this!  Really appreciate the work.  Granted I think Bachal just asked the same question but I am having trouble adding the local mirrors properly so that the chapter list can be found.

 

What I tried doing was:

1. Downloading your set of mirrors and adding all of the .js files to the js/mirrors folder

2. I tried both just listing the name of each website as well as using the json object in the localMirrors variable mgEntry

3. Reloading the unpacked extension in Chrome

4. Going to the manga page and adding to AMR

 

I am sure that I am just doing something stupid but any detailed step by step updating guide is appreciated.  Thank you again so much.

To load a local mirror you should just stick the file in js/mirrors/, add the name to the localMirrors var in mgEntry, reload the extension (This will have to be done each time you edit the file).

 

For local mirrors, you might want to just add a file select dialog to options.html (did that), add "file://" to permissions in manifest.json (thanks to sneezemonkey for linking me to the relevant material). Afterward you should be able to have read-only access to the filesystem, and in turn read from local files.The reason I recommend this is simply for user friendliness. 
 
P.S.: What one wants to do with the file would be to simply create the amrcObject structure that is used by amrcsql and execute the pertinent write. Data validation/sanitation optional.
P.S.S.: Sneezemonkey and I hang sometimes on the Batoto IRC, seems like a better way to communicate these things (looks ugly in posts).

 

options.html (add after closing </p> of options_ws_activation, line 656 in my editor) :

 
manifest.json (in permissions property):

 
messages.json (anywhere)

If I program the function to read from the file, I'll pass it to you. Eitherway, it is essentially just repeating the parsing code that's there, except ending with a write to the webdb and site listing refresh instead of all the callbacks. Also is there a reason you forced languages? Was lang_Spanish, lang_English, etc not working?
 

No need from my part, I'm doing it because I thought it might be helpful. Also Daku did all the work and Sneeze helped me out with reading so yeah, it's more them than me. 

Only problem with using file:// is we'd need to avoid the remote mirrors overwriting it. Trying to get it to load from the extension folder was confusing enough xD

If you can get it working though, feel free to make a pull request to the repo :>

 

Languages were forced as I thought that was how it was before I changed things, apparently not :x. That change has been reverted though.

[Bachal]

So I installed the latest Version you have on Github and it didn't work with any website:

http://imgur.com/JKewFfK

 

I manually changed it back to the old mgentry.js you had on Github and now it works. 

It still shows up as "No chapter found yet" at first but after some time it finds the chapter, I clicked on the "Mark this chapter as latest read" button in the top bar that shows up, but I don't know if that has something to do with finding it.

 

Now while reading some Manga I noticed a couple of things that don't work properly. The "Next" and "Previous chapter" button on the top title bar thingy don't work properly when you are on a chapter, it won't load the next/previous chapters. 

 

The other things that don't work are on Mangastream, when a new chapter is released and you click on "View next chapter" in the Manga list it doesn't update automatically as read, as it does on other Manga sites.

And when you click on the "Mark this chapter as latest chapter read" in the top bar it marks the oldest chapter that there is as read.