Sign In
Create Account
This topic is locked
Tuesday roughly 3:30AM UTC - Server taken offline for emergency maintenance.
Tuesday 12:00PM UTC - Server Restored.
------------------- EDIT --------------------
Here's the delayed explanation (including why it was delayed). It gets pretty technical though.
Since around Jan 20th, we've been under a peculiar traffic. We could call it a "ddos" attack, but the situation is a bit odd to call it that. But the attacks were too small (relatively) to actually affect the server to go into denial, maybe slow down a bit (Be in mind, our server handles thousands of people, serving over 5000 pages per minute, which translates to 10K~100K files served per minute. That's out legitimate traffic. Small attacks doesn't really compare against the actual traffic). Also as a tidbit of a knowledge, a high traffic sites always get dos attacks from some stupid kid that thinks it's cool to (d)dos a site (then gets blocked by the server (usually automatically) and concludes the unreachable server "down"...). That... along with millions of random bots throughout the net that just goes around looking for exploits in servers. I think that part is an unforunate reality of operating a server. Back on topic, I had chosen to deal with the issue with a bit of leisure as it wasn't making a big effect (a mistake).
Jan 23rd comes a bit of a nightmare. The biggest culprit (I won't name because I believe it was not actually intentional because I know who it is) who was effectively ddosing out website got pretty large. They were requesting pretty much every single page in Batoto's existance at great speeds and in an... "ugly" way. Combined with other abusive usages that was found, it went past the tipping point making our server go into denial of service. This wouldn't have been that big of a deal normally, just reboot, block the problematic traffic and continue. But... I can't even connect to the server anymore.
;;I call the datacenter to ask my server to be manually rebooted. A machine says leave a message, they're temporarily out of office.
;;I call the biggest culprit's headquarters, have them forward me to their tech department. A machine says leave a message. They clearly went home already.
;;Well, I leave a message (a ticket actually) with the datacenter to reboot and a message to the culprit saying they're a freaking idiot with lot of servers.
Few minutes later, datacenter tells me they're rebooting.
Few minutes later, they say it refuses to reboot.
;; So, they're doing a hard reboot (power off, power on. Don't do this kids!! This is bad for your computer's health).A hard boot later, our server was online, I was busy at blocking off problematic traffic of quite a few sources. I may have blocked some users as well by accident... lol (They'll see my email then anyway). But, all the ones I banned were requesting like a 1000 page per minute, so I'm sure they're not real people... I can't read 1000 pages of comics in a minute, can you?
But our server was keep on going into denial. I didn't really get why until I found out that our RAID had failed after the hard reboot. This isn't all that likely to happen, but it happened to, not one, but two of our raid drives.
;; You can read more about what RAID is on wikipedia, but in short, it's like combining multiple hard drives to create a mega hard drive. Faster, bigger, awesomer. Awesomer is the most important part. So, without RAID functioning properly, it was trying to serve everyone (it's surprising how quickly people come back... they must be pressing F5 every minute instead of just stop procrastinating and doing homework) from few disk (instead of all) while rebuilding raid at the same time. Our disk just couldn't handle it (much of cache is also gone after a reboot, so everything comes from disk now). So, the choice was... let it rebuild and leave server on making a slow For some who might be wondering... don't you need to go to work? Meh, no one's gonna miss me because I'm gone for a day or two. This is the power of working in the tech industry!! Get your stuff done on time and no one gives a damn!
--------------- Part 2 - The Protest ----------------
The above was inevitable, so I chose to make the best of the situation. While I like to keep this site neutral as possible on many grounds, I felt this was the best choice. Batoto did not have any form of official protest against SOPA/PIPA. While they're utterly terrible bills, I did not feel the importance of taking this site down due to 1. the bother, I'm already loosing sleep. 2. felt it was already well in the knowledge of many people and 3. did not feel that it was my place to say it. The case of ACTA however a bit a different. The site was down anyway, not that many people know that it's on the brink of being passed and on the last point, I can be convinced. lol
The Internet censorship is a great deal. This site wouldn't exist if these bills were pushed to the full force which has not yet happened (but as we see with Megauploads, we can see how US would have used ACTA to keep a surveillance on MU owners without any court issued warrant). This site has far too much freedom of speech (to the level of what we wish) to really coexist with heavy surveillance and censorship. The Internet is the most creative forefront in the world today. Vast amount of software, media, and etc are being made and developed so fast and I believe that's largely due to licenses like Creative Commons, GNU, MIT, etc which allows sharing. The knowledge and the works are able to quickly build on top of each other creating a result much better than any single person could have created. This site houses hundreds, maybe thousands of programs made by other people who allowed it to be shared for free. I couldn't have made this site from scratch by myself, there's just too many things. And this is why I believe freedom is important for the sake of the growth of our future. Restricting this creativity would do nothing more than keep obsolete business models continue in their paths of inefficiency and restrict the growth of the Internet.
If you want to see the protest page, actually our Error 500 page is still availabe by just typing it out.
http://vatoto.com/50x.html
Back to top
