Skeptics Guide to the Universe Forums

General Discussions => General Discussion => Topic started by: Belgarath on December 02, 2014, 03:04:19 PM

Title: What Happened This Time?
Post by: Belgarath on December 02, 2014, 03:04:19 PM
Hi all,

I wanted to give the Forum members a detailed update on what happened this time with the crash and what I've done to ensure that this doesn't happen again.

tl;dr version: 

It broke.
It was the hosting company's fault. 
I fired them.
I hired a new and better hosting company and;
I have put in processes on the back end to ensure that this NEVER happens again.



(click to show/hide)

I am truly sorry that this happened a second time in a year and I have (I believe) made certain that this will NOT happen again.

P.S.  Thanks to TheLatinist and TLV for helping me select a new hosting site. 
Title: Re: What Happened This Time?
Post by: The Latinist on December 02, 2014, 07:05:28 PM
Glad to help.  And glad to be back!
Title: Re: What Happened This Time?
Post by: Caffiene on December 02, 2014, 07:10:28 PM
 :cheers:

Its good to be back.
Title: Re: What Happened This Time?
Post by: Harry Black on December 02, 2014, 07:14:31 PM
Well done everyone!
Might I suggest some sort of notification for new users for a while? Its going to suck getting old threads resurrected and having people think nothing has happened here since 2012!:D
Obviously restoring full functionality is a priority for now.
Title: Re: What Happened This Time?
Post by: The Latinist on December 02, 2014, 07:16:43 PM
Might I suggest some sort of notification for new users for a while? Its going to suck getting old threads resurrected and having people think nothing has happened here since 2012!:D

It might even be worth putting a notice at the top of the main page. If it's anything like VBulletin and PHPBB (with which I am much more familiar), it should be fairly easy to put a brief couple of sentences at the top of the main page.
Title: Re: What Happened This Time?
Post by: Belgarath on December 02, 2014, 08:31:13 PM
Might I suggest some sort of notification for new users for a while? Its going to suck getting old threads resurrected and having people think nothing has happened here since 2012!:D

It might even be worth putting a notice at the top of the main page. If it's anything like VBulletin and PHPBB (with which I am much more familiar), it should be fairly easy to put a brief couple of sentences at the top of the main page.

It's already there.  Mods just need to do something with it (i.e. like they already did  :) )
Title: Re: What Happened This Time?
Post by: The Latinist on December 02, 2014, 08:35:46 PM
Might I suggest some sort of notification for new users for a while? Its going to suck getting old threads resurrected and having people think nothing has happened here since 2012!:D

It might even be worth putting a notice at the top of the main page. If it's anything like VBulletin and PHPBB (with which I am much more familiar), it should be fairly easy to put a brief couple of sentences at the top of the main page.

It's already there.  Mods just need to do something with it (i.e. like they already did  :) )

Indeed, the forum seems to remember that I collapsed it at some point in the past.  I don't think I even knew there was anything under the News tab.  Wonder what other essential information I'be missed. ;-)
Title: Re: What Happened This Time?
Post by: c60Unit on December 02, 2014, 09:29:13 PM
Hi all,

I wanted to give the Forum members a detailed update on what happened this time with the crash and what I've done to ensure that this doesn't happen again.

Thank you all !!!   :woop:

Quote
tl;dr version: 

It broke.
It was the hosting company's fault. 
I fired them.
I hope you got a refund!!!

Quote
I hired a new and better hosting company and;
I have put in processes on the back end to ensure that this NEVER happens again.
So, we will not score with Andie MacDowell...  ::)  I can live with that!

Quote
... snip description of the activities of the sneaky bastard Murphy ...
(http://imgur.com/nRH4jV4.jpg)

This increases my liking of optical media as an alternative form of backup.   Out of curiosity, how much data is involved??  If it would fit(<25G), a monthly backup to an M-Disc BD-R (http://www.mdisc.com/) would seem like a good thing.  I'll even send you some if you have a writer.

Thanks Again!
Title: Re: What Happened This Time?
Post by: PANTS! on December 02, 2014, 09:52:42 PM
Thanks for all your hard work.  I needed my fix.
Title: Re: What Happened This Time?
Post by: xenu on December 02, 2014, 10:08:05 PM
I kept thinking that it was something that I was doing wrong on my end. Then I was thinking maybe the rouges got tired of all the bullshit on the BB and decided to pull the plug on it.
Good work all. I still can't believe this happened again.
Title: Re: What Happened This Time?
Post by: c60Unit on December 02, 2014, 10:19:45 PM
Strangely, my laptop finds the breadcrumb message to come here for both sguforums.com (http://sguforums.com) and www.sguforums.com (http://www.sguforums.com).  My computer downstairs gets what looks to be a hosting site message for sguforums.com (http://sguforums.com), but gets the breadcrumb for www.sguforums.com (http://www.sguforums.com).

The one thing I would like to restore is the lead post of the "Edward Lewis Tobinick, MD et al v. Novella et al" thread.  I spent a fair amount of time editing that post to have a lot of useful links.  Strangely, I can find thread pages in Google cache, but not that first page/post!  I also tried the Wayback Machine at Archive.org.  Sguforums.com was scanned several times this year, but when I tried
https://web.archive.org/web/20140815072925/http://sguforums.com/index.php/board,2.0.html (https://web.archive.org/web/20140815072925/http://sguforums.com/index.php/board,2.0.html)
things looked promising, but the wrong thread comes up when I click on the Tobinick thread.  So close...   Still, Archive.org may be of some use for people trying to recover content.  (Meantime, I'll keep poking around.)
Title: Re: What Happened This Time?
Post by: drwfishesman on December 02, 2014, 10:28:41 PM
Remember when someone suggested that the same thing that was happening over at JREF with their forums could happen here and we all made fun of them? LOLOL what a lark that was.
Title: Re: What Happened This Time?
Post by: amysrevenge on December 02, 2014, 11:16:32 PM
I...

it's...

we...


...


Glad to be back.
Title: Re: What Happened This Time?
Post by: Jay_One on December 03, 2014, 04:29:41 AM
Thank you for sorting this out (again), I can't imagine it's an easy process. :)
Title: Re: What Happened This Time?
Post by: starnado on December 03, 2014, 06:20:28 AM
Dark.
Cold.
Fuzzy indigo shapes appear and fade before his eyes.
Silence, punctuated only by random Youtube comments and adverts on facebook. However, he does not want an instant sixpack.

Then, far above him in the distance, a light sparkles, shifts, dances like a whisp. Grows closer, brighter.

He swims upwards, struggling against his own weight, reaches for the light that grows and envelops him. Gasping, his head breaks the surface of the water: he drags in breath, as if he hasn't breathed in days.

Well thank Jebus for that.

Hey guys, the rest of the internet is a bit shit, no? Like swimming in slurry. Nice to be back.


Thank you mods and Belg and all the others who worked so hard to bring back the light.
Title: Re: What Happened This Time?
Post by: Beleth on December 03, 2014, 06:50:24 AM
Thanks, Belg and Louie and everyone who had a part in restoring! And shame on the old hosting company.
Title: Re: What Happened This Time?
Post by: Guillermo on December 03, 2014, 08:05:25 AM
The worst thing about all this is that we can't call this "The Great Crash of 2014" due to the ambiguity.

Thanks for all the effort.
Title: Re: What Happened This Time?
Post by: Skulker on December 03, 2014, 08:28:46 AM
Well done gentlemen. :cheers: It feels good to be back in good old 2012 again. ;D
Title: Re: What Happened This Time?
Post by: Plastiq on December 03, 2014, 11:12:46 AM
Thanks, Belg and Louie and everyone who had a part in restoring! And shame on the old hosting company.

Outrageously bad service. I don't know, I guess shit happens, but, Christ, you had one job (the hosting company of course, not you)!
Title: Re: What Happened This Time?
Post by: Skulker on December 03, 2014, 12:30:21 PM
Thanks, Belg and Louie and everyone who had a part in restoring! And shame on the old hosting company.

Outrageously bad service. I don't know, I guess shit happens, but, Christ, you had one job (the hosting company of course, not you)!
Old hosting company = a couple of hippies running it from their mother's garage. ;)
Title: Re: What Happened This Time?
Post by: Jeremy's Sea on December 03, 2014, 01:10:10 PM
Thanks, Belg and Louie and everyone who had a part in restoring! And shame on the old hosting company.

Outrageously bad service. I don't know, I guess shit happens, but, Christ, you had one job (the hosting company of course, not you)!
Old hosting company = a couple of hippies running it from their mother's garage. ;)
Hey maaaaan, their mom is righteous and bakes cookies.  ;)

I almost lost a few hours of writing from my last airplane flight because like an idiot I didn't put my laptop back on a wifi and sync the files. That panic was nothing compared to what was lost here again. Thank god for the 2012 backup or we would have lost everything forever. That said, I got crashplan this year, and I'm extra thankful for that at this moment.
Title: Re: What Happened This Time?
Post by: drwfishesman on December 03, 2014, 03:02:04 PM
The worst thing about all this is that we can't call this "The Great Crash of 2014" due to the ambiguity.

Thanks for all the effort.

Is it just me misremembering or was the first crash of 2014 worse than this? In my mind, if there was a "Great Crash of 2014", that was it.
Title: Re: What Happened This Time?
Post by: Skulker on December 03, 2014, 03:23:34 PM
The worst thing about all this is that we can't call this "The Great Crash of 2014" due to the ambiguity.

Thanks for all the effort.

Is it just me misremembering or was the first crash of 2014 worse than this? In my mind, if there was a "Great Crash of 2014", that was it.
They were both equally bad but the lesson from the first crash was applied, i.e. don't waste time trying to recover threads, just restore the database, even if it is from 2012.
Title: Re: What Happened This Time?
Post by: bimble on December 03, 2014, 06:31:30 PM
Thanks for the hard work, glad the forums are back I did miss them.

Ahhhh, post #1, for the fourth time! :D
Title: Re: What Happened This Time?
Post by: Guillermo on December 03, 2014, 07:18:53 PM
Party :dance: like it's twenty twelve!

Oh NO! The Novellas are gone!!!
Title: Re: What Happened This Time?
Post by: Dan I on December 03, 2014, 08:26:44 PM
Whelp, back to square one.
Title: Re: What Happened This Time?
Post by: fonebone on December 04, 2014, 12:14:13 PM
Whelp, back to square one.

Just spam for a while.  Youll get your post count back up.   >:D
Title: Re: What Happened This Time?
Post by: Belgarath on December 04, 2014, 01:33:22 PM
So, I wanted to check that it's backing up and YAY we have a winner!

(https://lh4.googleusercontent.com/-YVUDWuSyh0k/VICnW-4NPvI/AAAAAAAApRY/jGwHEK2IXyc/w779-h766-no/Capture.JPG)

Generally every 8 hours, and I don't think I've heard any lag reports so we're good. 
Title: Re: What Happened This Time?
Post by: GhostOfPerdition on December 04, 2014, 05:01:18 PM
Oh man. Every time I decide to come back here and poke around the forums it's 2012 again :P
Title: Re: What Happened This Time?
Post by: The Latinist on December 04, 2014, 07:44:18 PM
Whelp, back to square one.

I would have almost 10,000 posts but for the two crashes.
Title: Re: What Happened This Time?
Post by: AQB24712 on December 04, 2014, 08:08:22 PM
Whelp, back to square one.

I would have almost 10,000 posts but for the two crashes.

 :raise:
Title: Re: What Happened This Time?
Post by: kendranoelle on December 05, 2014, 12:07:59 AM
Yay! I'm so glad things are back! Thank you so much you guise for getting this all fixed up and back and running!

Title: Re: What Happened This Time?
Post by: mathsoholic on December 05, 2014, 01:49:13 AM
Great job getting everything up and running again! :)

(I didn't realize how much I lurk here until I couldn't any longer. Oops)
Title: Re: What Happened This Time?
Post by: Morvis13 on December 05, 2014, 08:06:41 AM
Sure is good to be home again. Missed you guys.
Title: Re: What Happened This Time?
Post by: Skulker on December 05, 2014, 10:39:29 AM
Whelp, back to square one.

I would have almost 10,000 posts but for the two crashes.
Don't worry. Everything is made up and the posts don't matter.  ;)
Title: Re: What Happened This Time?
Post by: Skulker on December 05, 2014, 10:40:48 AM
Sure is good to be home again. Missed you guys.
If you really missed us you would have been in the Facebook group during this last event.  :P
Title: Re: What Happened This Time?
Post by: Morvis13 on December 05, 2014, 11:10:32 AM
Sure is good to be home again. Missed you guys.
If you really missed us you would have been in the Facebook group during this last event.  :P
Didn't miss you enough to qualify for that lobotomy.
Title: Re: What Happened This Time?
Post by: Friendly Angel on December 05, 2014, 12:42:28 PM
I crashed two times this year too - on my bicycle.  First time I broke my pelvis, second time I broke 7 ribs.  Things could be worse.
Title: Re: What Happened This Time?
Post by: Louie on December 05, 2014, 01:06:30 PM
Apart from the sad reality that many members are still having to re-register and a whole lot of content loss, we seem to be heading back towards normality. The right sub-forums seem to be in the right places, the 'like' button has made its triumphant brownish orange/orangeish brown return, and Evan once again rules 'Who's That Noisy?' with a firm but loving fist.

We need your help, though. There may be stuff we've missed. If you remember anything that used to be around and now isn't, please tell us about it here. Odds are that it's already being worked on, but it's best to make sure.
Title: Re: What Happened This Time?
Post by: fonebone on December 05, 2014, 02:48:44 PM
I crashed two times this year too - on my bicycle.  First time I broke my pelvis, second time I broke 7 ribs.  Things could be worse.

You need this.

(http://www.dirtybutton.com/media/db1571-bike-airbag.jpg)
Title: Re: What Happened This Time?
Post by: The Latinist on December 05, 2014, 10:36:56 PM
We need your help, though. There may be stuff we've missed. If you remember anything that used to be around and now isn't, please tell us about it here. Odds are that it's already being worked on, but it's best to make sure.

I would like to second this.  If there's anything strange or missing that you notice, post it here or in Tech Admin, or even PM me.  I don't bite unless asked nicely.
Title: Re: What Happened This Time?
Post by: c60Unit on December 05, 2014, 11:03:47 PM
Don't worry. Everything is made up and the posts don't matter.  ;)
Bobby, from the shower, at Dallas...
Title: Re: What Happened This Time?
Post by: c60Unit on December 05, 2014, 11:05:21 PM
I crashed two times this year too - on my bicycle.  First time I broke my pelvis, second time I broke 7 ribs.  Things could be worse.
Yeah, you could suffer a head crash.
Quote

You need this.

(http://www.dirtybutton.com/media/db1571-bike-airbag.jpg)
With no protection...
Title: Re: What Happened This Time?
Post by: c60Unit on December 05, 2014, 11:09:29 PM
So, I wanted to check that it's backing up and YAY we have a winner!

(https://lh4.googleusercontent.com/-YVUDWuSyh0k/VICnW-4NPvI/AAAAAAAApRY/jGwHEK2IXyc/w779-h766-no/Capture.JPG)

Generally every 8 hours, and I don't think I've heard any lag reports so we're good.
So about 2.5G of data total? 
Title: Re: What Happened This Time?
Post by: fonebone on December 06, 2014, 12:23:27 AM
So, I wanted to check that it's backing up and YAY we have a winner!

(https://lh4.googleusercontent.com/-YVUDWuSyh0k/VICnW-4NPvI/AAAAAAAApRY/jGwHEK2IXyc/w779-h766-no/Capture.JPG)

Generally every 8 hours, and I don't think I've heard any lag reports so we're good.
So about 2.5G of data total?
[/quote

I am sure the backups are incremental.  It probably isnt that much data day to day.
Title: Re: What Happened This Time?
Post by: PANTS! on December 06, 2014, 06:27:07 AM
That and I am not sure much of what accumulates here reaches the status of data.
Title: Re: What Happened This Time?
Post by: Belgarath on December 06, 2014, 09:54:53 AM
I only showed a small piece of the data, and CrashPlan highly compresses it's output, so total backup size (last I checked) is about 3GB
Title: Re: What Happened This Time?
Post by: c60Unit on December 07, 2014, 04:28:44 PM
I only showed a small piece of the data, and CrashPlan highly compresses it's output, so total backup size (last I checked) is about 3GB
So a snapshot would fit on a DVD... :D
Title: Re: What Happened This Time?
Post by: Dan I on December 07, 2014, 09:42:09 PM
Jeez if we can compress it down that much maybe we should just start distributing backups among users with the storage overhead. I've got 2TB of space over here.
Title: Re: What Happened This Time?
Post by: Friendly Angel on December 10, 2014, 01:25:45 PM
What is different for the members who didn't lose their post count and those of us who did (twice)? 
Title: Re: What Happened This Time?
Post by: The Latinist on December 10, 2014, 01:47:46 PM
What is different for the members who didn't lose their post count and those of us who did (twice)? 

If I understand what you're asking, the difference is in how long you have been a member.  Because we lost every registration and post since 28 Aug. 2012, counts were reset to what they were on that date.  Anyone who registered after that date had to recreate their account and start again from a post count of zero.  Others who had been members longer lost perhaps thousands of posts but still retained those posts they had made before that date.

I myself lost somewhere in the neighborhood of 6,000 posts in the two resets.
Title: Re: What Happened This Time?
Post by: xenu on December 11, 2014, 10:09:00 AM
Jeez if we can compress it down that much maybe we should just start distributing backups among users with the storage overhead. I've got 2TB of space over here.

That's not a bad idea. We could back up once a month. Have 12 people and that would just be once a year and hold it a year.
Title: Re: What Happened This Time?
Post by: The Latinist on December 11, 2014, 10:16:27 AM
I intend to start keeping a backup the first of every month and keeping it for at least one month.
Title: Re: What Happened This Time?
Post by: Skulker on December 11, 2014, 10:45:44 AM
I intend to start keeping a backup the first of every month and keeping it for at least one month.
I have access to vast amounts of storage so if you need another off site archive facility, I'd be happy to help.
Title: Re: What Happened This Time?
Post by: Harry Black on December 11, 2014, 03:01:19 PM
Hopefully its just my device but the site is sporadically failing to load today. Not having this issue with other sites.
I'll get a screenshot next time if that would help?
Title: Re: What Happened This Time?
Post by: 2397 on December 17, 2014, 04:48:03 PM
Shame about the per episode resources and similar content that is lost when this happens. But I wouldn't actually mind if some websites regularly self-destructed and erased all data. A lot of it doesn't need to stay up forever.
Title: Re: What Happened This Time?
Post by: Evil Eye on December 17, 2014, 08:48:04 PM
And people ask why notebooks, and photographs are still necessary.  :-\
Title: Re: What Happened This Time?
Post by: Hershele Ostropoler on January 21, 2015, 08:47:12 PM
I had to tell a DIRTY LIE to reregister (since my initial answer to the "had another account" was "yes" (though Jay's name is "Novella" as far as I know.))

Though I guess arguable it was the same account
Title: Re: What Happened This Time?
Post by: OhMyGecko on January 28, 2015, 10:55:53 AM
I am not as consistent as many on here but thank you for all the hard work getting the site back up.
Title: Re: What Happened This Time?
Post by: caue on March 08, 2015, 10:29:46 AM
internal comment: I guess I will be coming back from time to time after all! :P

I'm mostly sad for these news, for so many reasons... But I'm not sure if I should be happy. I must have been very hated here for my posts and that whole "bad karma" is now gone, I suppose. Still, I do hate to lose any kind of data (http://talk.cregox.com/t/how-to-keep-all-your-data-to-yourself/7468?u=cawas). Even this. I sure wished I had my backups there.

Jeez if we can compress it down that much maybe we should just start distributing backups among users with the storage overhead. I've got 2TB of space over here.

That's not a bad idea. We could back up once a month. Have 12 people and that would just be once a year and hold it a year.

Some of you will probably remember "my agenda" back then. Well, I just wanted to point it this once more (https://meta.discourse.org/t/download-archive-of-my-posts-implementation/12644/13?u=cawas). If you read that, you might agree maybe it'd be a good idea to do something like that to facilitate people making their own backups here as well. Another thing to learn from Discourse, they also offer a way to backup to Amazon S3. Quite a good idea, since that is very cheap, if you use the right tool there. And it's not too complicated to set up. It saved my ass already, although I was not losing as much.

@Belgarath:
I personally use CrashPlan for years, after I've researched many services starting with good advice from Leo: Carbonite. But I never had to restore a backup from them and I've got no idea if they're good for that. I hope they do a good job, but I don't trust any important data on them because of this reason. I highly suggest you try it if that's what you'll rely upon now. Both from their cloud and your machine. Also do keep an eye for their app, as it has bugged out many many times on me, in the past (it did get better in recent months, or maybe years). They do offer some kind of email reporting, so that might help to keep track as well.

In any case, making that backup available for anyone to download might be a good idea for adding redundancy there. Just throwing yet another idea. ;P
Title: Re: What Happened This Time?
Post by: Doctor Whom on April 03, 2015, 11:41:34 AM
I wanted to give the Forum members a detailed update on what happened this time with the crash and what I've done to ensure that this doesn't happen again.
Thanks for your hard work, it is appreciated!
Title: Re: What Happened This Time?
Post by: Anders on May 30, 2015, 03:02:36 PM
I actually think these crashes serve a great skeptical purpose. It forces us to look at our posts from a number of years ago. If you don't think "What the fuck was I thinking?" at least once, I would be worried that you're mind is not growing as it should.