Topic: Forget Terabytes, its all about Petabytes now (Read 5340 times)

hit_ny · « **on:** September 02, 2009, 10:34:15 am »

Soooooo..... can MC handle a PetaByte

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

Peter_T · « **Reply #1 on:** September 02, 2009, 12:58:53 pm »

holy crap.

Matt · « **Reply #2 on:** September 02, 2009, 01:00:21 pm »

Eat your heart out Benn600.

benn600 · « **Reply #3 on:** September 02, 2009, 01:26:10 pm »

That is a better price than what I ended up getting. However, it isn't like they got some serious deal. In other words, it probably doesn't do all that mine will--power, redundancy, etc. Notice simple SATA cards rather than a dedicated hardware raid card--so it very well may have a slower overall speed capability--not that I truly need the speed. I also like having the entire drive in one single volume, which could potentially be created from their methods too. For 31.5TB about 2 months ago I think the total was around $6,000. I did calculate building a petabyte--ya know, it'd be nice! It was over $200K

At my last check earlier today, I haven't had a single drive failure which is excellent.

Does anyone know if hot spares typically stay online and hot? They should be spun down to preserve life, right? The 24th drive in mine is a hot spare, so far unused by the array.

glynor · « **Reply #4 on:** September 02, 2009, 06:03:10 pm »

This is pretty cool. I don't know if you've ever heard of the company Backblaze, but they're a "cloud backup" company (mostly for commercial clients). They have a gazillion hard drives in a huge server room (or actually, a bunch of different server rooms in different places) on the internet somewhere, and you pay them a monthly fee to have them back your stuff up.

Anyway, they obviously need a huge, huge, huge pile of hard drives to pull this task off. Just like Amazon S3 and all the other big cloud storage facilities out there. They were apparently sick of paying millions of dollars for extremely expensive high-capacity storage servers, so they got down to it and designed their own which costs a tiny fraction of the cost of competitive solutions. And, they made it fire-engine red, because.... Why not!

And NOW, they're showing how they did it on their web site for free so you can build one yourself, if you want (and have the cash).

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

This shows you how to build a 67TB high-capacity storage "pod" (sever) for just under $8k, which uses 45 1.5TB Seagate SATA drives, Linux, and other off-the-shelf server parts. And it can, of course, be used together with other "pods" to scale to truly enormous capacities (how about a cool petabyte for all those BluRay movies, eh?). Of course, a Petabyte is way out of the price range for most average consumers, but still pretty darn cool. And, heck, the fire-engine red case will hide it when the thing turns red from the heat generated by those 45 hard drives!

Now, I don't have pockets deep enough for this (yet, of course, wink-wink)... But it is still VERY cool.

leezer3 · « **Reply #5 on:** September 02, 2009, 06:29:06 pm »

Hot spares will depend on the controller as to whether they're spun up constantly or not

What I've always found though, is that if a drive is going to die, it'll do it within the first few months.
After that, I've never seen any failures, and I abuse drives, and don't have any fancy stuff running

-Leezer-

JimH · « **Reply #6 on:** September 02, 2009, 06:32:32 pm »

The article reminded me of Servers in a Container.

There waw another interesting tangent this morning on public radio. John Gordon's Future Tense. He was reporting on the idea of "the singularity", where humans are supposed to donate their "neural network" to computers and then to reap the benefit. It's supposed to happen by 2030. I'll be 86 for most of that year.

newsposter · « **Reply #7 on:** September 02, 2009, 09:39:28 pm »

no 'enterprise' will spend that kind of money and not get redundancy and power.

There are a lot of smart minds working on storage appliances, some of them have multi-decades of successful design experience behind them

benn600 · « **Reply #8 on:** September 02, 2009, 11:33:56 pm »

I know, right. They don't even have redundant power supplies. I guess people's data would be mirrored on at least two or three of these boxes so they don't need redundant power--because even with redundant power you couldn't just replicate the data on a single of these. That wouldn't be safe enough, right?

MrHaugen · « **Reply #9 on:** September 03, 2009, 02:35:49 am »

Very similar approach to what I'm thinking about my self. I do NOT go for single SATA controllers though. That's not cool.

Bought my self a Lian Li Cube case who can handle up to 36 or 40 drives. Bought a Highpoint Rocketraid 4320 (PCIE x4) with 2 Mini SAS connectors. Each of this can be used for 4 SATA drives (mini SAS to x4 SATA or eSATA). With a rather simple multiplexer (Lycom ST-126) on each of the 8 cables I can connect yet another 5 disks on each SATA cable, without loosing much bandwidth they say

That's a total of 40 disks. Perfect for my cabinet, and my needs!

Going to start small with around 10 disks of Western Digital Green series 1.5 TB disks, and build it up from there. Looking forward to the day I have almost 60 GB array in my apartment! It might be more when bigger drives are supported as well. Only downside here is that the multiplexers don't support raid 6. Only up to raid 5. So I might end up having a few arrays, not one big one, just to be safe.

Peter_T · « **Reply #10 on:** September 03, 2009, 06:23:05 am »

Quote from: benn600 on September 02, 2009, 11:33:56 pm

I guess people's data would be mirrored on at least two or three of these boxes...

...yeah, it would have to be. I hope. I just signed up for their service last night. Cheaper and way less hassle than my current off-site backups. Which are medium, at best. I'll see how it goes.

benn600 · « **Reply #11 on:** September 03, 2009, 09:08:19 am »

Should I try uploading my 7TB? Wow, just checked and it is exactly 7.00TB used. I'm just afraid of downloading all that content again. The best idea would be to use DSL for normal browsing and then setup a cable internet line for continuous backup synchronizing in addition a very convenient emergency backup line if the other went out! So you'd be looking at a higher cost but I know I wouldn't want all the activity to interfere with my internet usage / voip calls / etc. I'd really like to backup everything. I've already tried another service. Just a little unsure about placing all that data with a third party.

Peter_T · « **Reply #12 on:** September 03, 2009, 10:44:20 am »

Quote from: benn600 on September 03, 2009, 09:08:19 am

Should I try uploading my 7TB? Wow, just checked and it is exactly 7.00TB used. I'm just afraid of downloading all that content again... Just a little unsure about placing all that data with a third party.

Yeah, I checked out how you download the info to recover. It's kind of like time machine (the functionality, not the interface) in that you can browse your file structure and then pick different backups (in time) from a drop-down list.

Then you can download that file, directory, branch, whatever as a zip file. Or you can have them send you one DVD's worth of data via FedEx, or a hard disk with your data.

If you were to download zip files, you'd probably end up downloading a chunk of stuff at a time... All your photos in the first chunk, all your documents in another. Music would probably be 25 chunks on it's own.

I am only backing up about 2TB, but restoring it would take weeks or months.

I hope it works out for me, but I would only ever restore if my main machine and my local backups failed. Like in a fire or something.

leezer3 · « **Reply #13 on:** September 03, 2009, 11:34:30 am »

Just FWIW, I've just read the T&Cs. Not a company I'd touch with a bargepole.
Notable highlights-
* Undefined fair use/ abuse policy.
* No support for files over 4gb
* Absolutely no SLA.
* No guarantee of data security whatsoever.
* Ability to change the T&Cs with no notice or notifications at any time.

Will probably work *most* of the time, but if anything goes wrong I'd say you're screwed.

-Leezer-

Peter_T · « **Reply #14 on:** September 03, 2009, 12:20:49 pm »

Hey leezer - I hear you. Some how I'm OK with most of the time. What are the odds that my house burns down AND these guys flake out on the same weekend? I know that a backup plan shouldn't play the odds, but hey. Between the convenience and the price, I may still use their service. I'm on a 15-day trial now, so I have a few days to chew on it.

Out of curiosity, why is the 4GB limit a red flag?

benn600 · « **Reply #15 on:** September 03, 2009, 12:46:25 pm »

Great marketing on their part to send out this tutorial which is surpising so many people when in reality it doesn't seem all that unknown. When I built my server I could have brought the price down but each step would be giving something up.

glynor · « **Reply #16 on:** September 03, 2009, 02:54:12 pm »

Quote from: MrHaugen on September 03, 2009, 02:35:49 am

Very similar approach to what I'm thinking about my self. I do NOT go for single SATA controllers though. That's not cool.

I imagine they went with simple SATA controllers primarily due to cost, but also since they were going to only access the storage through HTTPS (which is super-high latency anyway) the extra performance you gain from employing 10,000 RPM (or higher) SAS drives isn't really a huge concern (especially considering the enormous cost). In simple sustained throughput, SAS and SATA are identical, assuming you don't have a terrible SATA controller. The main difference between the two technologies is spindle speed (and that's only because of product segmentation, there's no reason you couldn't put a SATA connector on a 15,000 RPM drive other than marketing), which only impacts latency and burst transfer speed.

While on your average workstation, using a hardware RAID card is "important", this is mostly because the CPU in the system needs to be available to do other things. In a storage-server situation, it is actually far less essential. Obviously, the general-purpose CPU is much more powerful than whatever specialized RISC hardware processor they put in the RAID cards. Those hardware RAID chips mostly just serve to free up the CPU to do other things, but if the CPU has nothing else to do, well then, why bother paying for an extra RISC chip on-card? A hardware RAID card can also slightly speed up rebuild times, due mostly to parallel hardware XOR processors, but this only comes into play if the server goes down. Also not a giant concern when you're planning to have a room full of redundant servers (again, especially considering the cost differential).

However, since the SATA cards they were using are PCIE-1x cards, and they needed to use them with backplanes, it makes sense that they limited them to the 2 port varieties. That prevents the PCI-E bus (250 MB/s) from becoming a serious bottleneck (though it would still bottleneck the drives if all were active simultaneously). Obviously, a much better solution would be to use a bunch of 2 port PCIe 4x cards, but not very many of these are available, and most off-the-shelf server motherboards aren't going to have 4+ PCIe 4-lane slots.

newsposter · « **Reply #17 on:** September 03, 2009, 02:55:13 pm »

Any 'competent' internet-based (I refuse to use the word cloud) backup service should have an add-on offering where they will burn a set of DVDs with all of your data.

It's one thing to incrementally upload a ton 'o stuff, it's quite another to download it all at once.

JimH · « **Reply #18 on:** September 03, 2009, 03:01:15 pm »

Quote from: newsposter on September 03, 2009, 02:55:13 pm

Any 'competent' internet-based (I refuse to use the word cloud) ...

I prefer "Information Super Highway". It sounds classy.

glynor · « **Reply #19 on:** September 03, 2009, 03:16:29 pm »

Quote from: JimH on September 03, 2009, 03:01:15 pm

I prefer "Information Super Highway". It sounds classy.

I prefer "series of tubes". It sounds awesome.

JimH · « **Reply #20 on:** September 03, 2009, 03:29:31 pm »

It sounds like sausage.

benn600 · « **Reply #21 on:** September 03, 2009, 06:25:29 pm »

Quote from: JimH on September 03, 2009, 03:01:15 pm

I prefer "Information Super Highway". It sounds classy.

I'm going to use this more now.

JimH · « **Reply #22 on:** September 03, 2009, 06:50:10 pm »

Quote from: benn600 on September 03, 2009, 06:25:29 pm

I'm going to use this more now.

Points for succintness.

benn600 · « **Reply #23 on:** September 04, 2009, 08:09:47 am »

Quote from: JimH on September 03, 2009, 06:50:10 pm

Points for succintness.

1 Me
0 World

benn600 · « **Reply #24 on:** September 07, 2009, 10:57:56 am »

I think everyone is forgetting to answer the question of this thread. Yes, MC can handle a petabyte. See this thread where I asked for support. It was a long and grueling development process, but petabytes are now supported.

http://yabb.jriver.com/interact/index.php?topic=52968.msg361181

INTERACT FORUM

Author Topic: Forget Terabytes, its all about Petabytes now (Read 5340 times)