INTERACT FORUM

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1]   Go Down

Author Topic: Drobo- Gimmick or real world storage and backup?  (Read 4398 times)

Samson

  • Galactic Citizen
  • ****
  • Posts: 391
Drobo- Gimmick or real world storage and backup?
« on: August 07, 2013, 08:19:04 pm »

Hi All,

My usual way of backing up and storing was simply having media on a 2TB external drive and either using a sync tool or imaging program to another 2TB external drive.Now that I am spreading and potentially filling multiple drives it is getting a bit cumbersome.

Many threads like here, http://yabb.jriver.com/interact/index.php?topic=80400.msg547100#msg547100 , involving complicated RAID setups are 'above my pay grade'.

My choice seems to be 1) get 2 bigger drives but how big can you go..and what to do with the existing 2TB drives? or 2) Look at something like Drobo, http://www.drobo.com/ which seems to 'solve' all my problems. Is anyone actually using it and how much of "BeyondRAID" is just marketing hype?

Btw I have no idea what BeyondRAID, unRAID,FreeNAS,... or for that matter, what RAID really is (I used to think of it as redundant backup and storage but gather it isnt.Several years ago I did setup a RAID (cant remember what flavour or number) on my pc but found it a headache for all the reasons they say on the Drobo site)

Thanks,
David
Logged

InflatableMouse

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 3978
Re: Drobo- Gimmick or real world storage and backup?
« Reply #1 on: August 08, 2013, 02:20:31 am »

RAID is not backup. It is protection against drive failure. On a 3 disk RAID system, 2/3 is data, 1/3 is parity. Parity blocks are spread out over the 3 disks so that when any one of the drives fail, the system always has enough data + parity to calculate the missing part.

I never worked with a Drobo but I've read their website. From what I gather, a Drobo is the same as any other NAS in that respect except that they've written some smart software around it to make changes "on the fly".

The following is simplified, but imagine you have a RAID-5 with 5 2TB disks. Basically, 8TB of usable space and 2TB of parity data spread out over the 5 disks.

This is what happens on disk:
Code: [Select]
disk1|disk2|disk3|disk4|disk5
-----------------------------
data |data |data |data |Pari
data |data |data |Pari |data
data |data |Pari |data |data
data |Pari |data |data |data
pari |data |data |data |data

Drobo says they can change this to RAID-6 on the fly. RAID-6 protects against 2 drive failures, so per row of data/parity, we now need 2 parity blocks. RAID-6 will look like this:

Code: [Select]
disk1|disk2|disk3|disk4|disk5
-----------------------------
data |pari |data |data |Pari
pari |data |data |Pari |data
data |data |Pari |data |pari
data |Pari |data |pari |data
pari |data |data |data |pari
data |Pari |data |pari |data
pari |data |data|

Two things you should notice here. Each row now has 2 parity blocks and we have 2 more rows. The only way to move from RAID-5 to RAID-6 is to move a datablock to the end and replace it with an extra parity block. I don't think there is any other way because its not like you can set a flag that magically transforms a RAID-5 to a RAID-6, for each row of data you are going to need to calculate extra parity. This takes time and it takes space. To make space, they are going to have to move other data. I don't see any other way.

That is the added value of a Drobo I think, the fact that they wrote a piece of software that can transform a RAID-5 to a RAID-6 on the fly, without having to rebuild your raid manually.

From what I've heard and read about Drobo, they make excellent hardware and their stuff is supposed to work flawless and fast. But like I said, I never worked with one so I have no hands on experience with one.

As with all things, it comes down to what you are willing to pay and whether you see value in their software. The cheaper 5-bay Drobo will cost you 600 dollars, I think this is on par with the other NAS vendors out there, except that it only has an Ethernet connection, nothing else. A Synology comes with dual ethernet, USB 3, eSATA, etc, etc.

Personally, I don't really see the added value of Drobo, but maybe that is because I have never worked with one. If I had to advise a NAS, I would advise Synology simply because dozens of colleagues and friends have them, they are fast, reliable and quiet and there is a large community and a large software repository. Everyone I speak to rave about their support. I've heard stories more than once that Synology support technicians remotely login to help with drive failures and data recovery, even after warranty had ended for the product in question.

If you would use the NAS for backups only, I wouldn't go with extra fancy RAID-6 or RAID-10. I would say RAID-5 is sufficient. If you are going to use it as main storage, you might consider RAID-6, but remember this: no amount of redundancy can replace a proper backup! If data is valuable, back it up properly. And while I'm at it, a copy with some sync tool isn't a backup either ;).

By the way, when you buy a NAS cabinet, make sure your disk drives are compatible. I would look for drives that are low RPM, quiet, stay cool and low energy. Also if you need to buy new drives, do NOT buy all the drives at the same shop at the same time. Chances are they will come from the same batch and if one fails, they are more likely to all fail - around the same time! I've had that happen to me twice. Get them from different places and/or mix different brands/types of the same size, as long as they are compatible with the NAS you're buying you should be good.
Logged

Samson

  • Galactic Citizen
  • ****
  • Posts: 391
Re: Drobo- Gimmick or real world storage and backup?
« Reply #2 on: August 08, 2013, 11:25:40 pm »

Hi Inflatable,
Thank you for your fantastic reply. I have read it a few times and wrapping my head around the concepts with the aid of additional reading.Also great practical advice in there also.

Re NAS for storage only. I gather this would be spanning data and some parity files across a number of Hard discs. This is where I may have it screwed up.When a RAID 5 NAS device (with 5 x 2TB HDD) is [edit- used as a backup storage and is] plugged into the pc, the pc would see it as one volume...ie an 8 TB "X" Drive, or whatever drive letter? You would have a similar arrangement as a primary storage device (with perhaps a greater redundancy).You then backup one to the other. There would be no use of native server software on the NAS for file serving.

Re NAS as a file server. I gather that beyond the above implementation you could set it up as a file server probably utilizing DLNA or some such. I am wondering how this dovetails with JRiver MC? I gather  MC can "pull" the files from a NAS but this would be using the NAS server software rather than JRiver server software ?

Finally,on a different note, why isn't a copy with a sync tool (or perhaps imaging software ? ) a backup ? All the files are duplicated to another place.....and can be restored to the original location if required  ?

Thanks again
David

Logged

InflatableMouse

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 3978
Re: Drobo- Gimmick or real world storage and backup?
« Reply #3 on: August 09, 2013, 03:26:42 am »

Re NAS for storage only. I gather this would be spanning data and some parity files across a number of Hard discs.

Basically, yes but not as files. A RAID volume uses the harddisks as raw devices, not as formatted NTFS volumes your pc can work with. The entire array becomes a volume your pc can work with, but below that software will split up the data in data blocks and parity blocks, this process is transparent to the user.

When a RAID 5 NAS device (with 5 x 2TB HDD)with is plugged into the pc, the pc would see it as one volume...ie an 8 TB "X" Drive, or whatever drive letter?

A NAS is not plugged into your pc, its connected to your local network. It requires some setting up. Basically, when you turn it on for the first time it gets an IP address from your local router or you need to set it up with an ip address so you can access it. You can then access it through a webpage, setup storage volumes, create network shares and setup permissions. Here's an example of how you would setup volumes on a Synology. In fact, Synology has some nice articles you could browse to get an idea of how their software works. The only thing I couldn't find on their site was an initial setup. Once setup, you can create a mapped network drive on your pc(s) to the shares you created on the NAS. These shares can be an entire volume or individual folders you created on a volume.

Re NAS as a file server.

Whether you use it for backups or as a file server, the mechanism behind is the same. It is setup the same. You might choose to use no or less redundancy but that's about it.

An exception would be if the NAS vendor supplies special backup software that doesn't require shares but accesses the NAS 'directly' but I never worked with that and I don't even know if that still exists (I think my old QNAP had something like that, never used it. I could press a button on the NAS and a backup would automatically start on the pc it was linked with).

You would have a similar arrangement as a primary storage device (with perhaps a greater redundancy).You the backup one to the other.

I never meant to suggest you need to get 2 NAS devices, or maybe I'm not reading you correctly. It's possible but its also a little overkill if you ask me. A more practical solution would be to setup your NAS with 2 volumes, a 3 disk RAID 5 and either a 2 disk RAID-0 (no parity) or RAID-1 (mirrored) for backup purposes. You would have redundancy for whatever you store on the RAID 5 volume and place your backups on the other volume.

But I can already hear some people raging, because its not considered best practice to store your backups on the same device as the rest of your data. If that device burns out it could destroy both data and backups, or if it gets stolen, same story you loose both. Best practise dictates you store your backups offsite, but really, how far you want to go is up to you and how important you consider your backups. I'm aware of all the rules and regulations around making backups and storing them but that isn't practicle or affordable for home use so compromise is made. What and how exactly is up to you. It's your data, you know how important it is to you. Consider the concequences when it is lost and then you can determine how and where you want to store your backups.

There would be no use of server software on the NAS for file serving.

Correct. A NAS comes with everything you need to set it up for file sharing. Depending on the vendor, it may come with additional features such as DLNA server, print server, web server. Have a look at the product features from different vendors to get an idea what they can do.

I gather that beyond the above implementation you could set it up as a file server probably utilizing DLNA or some such. I am wondering how this dovetails with JRiver MC? I gather  MC can "pull" the files from a NAS but this would be using the NAS server software rather than JRiver server software ?

JRiver is both a client and a DLNA server. If you setup a NAS a DNLA server, MC can use it as a client and stream music and/or movies from it. Personally, I would rather use the NAS as a fileserver and let MC be the server and client for media, but its entirely up to you. I'm not sure how tagging and file management would work if you setup your NAS a DLNA server for MC though. I think the connection is read only.

But, I'm not really the best person to answer this, I have hardly any experience with DLNA. I suggest you search the Media Network topic and if you have any questions, post them there.

Finally,on a different note, why isn't a copy with a sync tool (or perhaps imaging software ? ) a backup ? All the files are duplicated to another place.....and can be restored to the original location if required  ?

Thanks again
David

Good question, glad you asked :). Imaging software is considered backup, a file sync is not.

When you copy a file or folder in Windows, a number of things can happen, depending on the source and destination location. But first you have to realize that a file is more than its contents. Agreed the contents is by far the most important aspect but that doesn't mean the rest should be ignored. A file has attributes, an owner and permissions. Attributes define basic parameters of a file or folder. It can define a file as a system file, as hidden, as read-only and whether its changed or not. This last one is an important one, its called the 'Archive' bit. The archive bit is used to tag a file as changed, the OS does this automatically when the contents or other attributes or of a file are changed (including permissions and owner). When a file is backed up or copied, the program resets the archive flag on the source and uses this bit to determine whether the source files have changed since the last time they were backed up or copied.

Another aspect of a file or folder are the owner and permissions. The owner is the user account that initially created the file or folder. Permissions determine who can read, write or modify files and folders, even if you have only setup a single user account on your pc (there are always multiple user accounts as the OS itself creates several).

When you copy a file, attributes, the owner and permissions can change depending on the source and destination. For instance, the read-only attribute is lost during a copy and the archive bit is set on newly created files by default. A regular copy will also ignore files and folders that have the system and hidden attribute set.

Quoting Microsoft Knowledgebase: By default, an object inherits permissions from its parent object, either at the time of creation or when it is copied or moved to its parent folder. The only exception to this rule occurs when you move an object to a different folder on the same volume. In this case, the original permissions are retained.

Copying to a share is handled the same as copying to another volume, I think. I'm lazy to look it up.

Last but not least, a copy also changes the owner. When you copy files created with a user account other than the one you're copying with, the owner changes to the one doing the copy with.

All this can lead to problems when you sync system folders or your profile folders, reinstall your pc and think you can simply copy the files back to their new locations. Later versions of Windows can handle this situation more gracefully than XP did, but you'd still get a few annoying popups and chances are that orphaned permissions remain on those files and folders. With system or program folders, you might even run into a situation that some programs won't work properly.

Other things a file copy will never include are system areas of disks and volumes. Notably, partition tables, Master Boot Record and a special system partition Vista and later versions create.

Now above I explained what happens when you do the copy from the command line or with Windows Explorer. A good sync tool will have options to change this default behavior. It can copy files and deal with attributes, owners and permissions. Most can deal with changed files and some can even keep a retention for changed and deleted files in a designated folder. It will have options to include system and/or hidden files and will set and reset the archive bit. So, for copying a bunch of user files to another drive or share, a sync tool may suffice or even offer features which are preferred for certain situations over backup programs. Strictly speaking, its still not a backup but a copy :P. System files, OS folders and Program File folders should never be synced with a file sync program.

Generally, setting up a folder sync is more complex because you need to think about ownership of files, permissions and attributes and configure the sync job to do the right thing. Last but not least, file sync tools can never recover an Operating System, recover (part of) the registry or do a full system recovery.

A good backup program is capable of a lot more than simply copying a few files. It can run on schedules, do full, incremental or differential backups, compress, verify, deal with attributes and permissions and can include the system partition, partition tables and master boot record. Some can even include some form of parity to protect against data corruption. Most backup programs offer features to restore to different hardware or resize destination partitions. They can 'image' complete drives and volumes with or without empty space (can be important). A backup program can often also be used to backup single files and folders. But more importantly, backup programs are typically more reliable and are easier to use than file sync programs are.

So backups and file syncs are different things and both have their purpose and should be used for the situation fitting their purpose. One does not replace the other. There is overlap, but there are also distinct differences that separate the two.

What I've deliberately not touched but is vital to how you backup your stuff are encrypted files and volumes (Windows file encryption/pgp/gpg/truecrypt and of course, bitlocker). When using any of these, they can completely mess up how you need to backup and sync/copy files.

Last but not least, I messed up splitting your message into quotes and when trying to correct it I only messed it up more. Then I started to run out of time so I rushed to complete it. Just ask again if I missed something or if its not clear.
Logged

Samson

  • Galactic Citizen
  • ****
  • Posts: 391
Re: Drobo- Gimmick or real world storage and backup?
« Reply #4 on: August 09, 2013, 07:48:35 pm »

Hi Inflatable,
Thanks for the full explanation of file copies vs true backup. I knew generally about this but not the specifics. I know it can get even more tricky with 'junction links' or whatever so always do an image based copy/backup of system or OS stuff. Also, thanks for the links to Synology, I will read up.



Quote
A RAID volume uses the harddisks as raw devices, not as formatted NTFS volumes your pc can work with. The entire array becomes a volume your pc can work with

Putting aside NAS for a moment, so RAID can group multiple HDD into one volume or split them into multiple volumes seen by the OS?

Putting aside RAID and NAS for the moment, can a HDD manager also group multiple drives into one volume as seen by the OS? I have Paragon Hard Disc manager (with virtualization) and it can obviously split or merge partitions...but afaik , not merge HDD into one volume. The reason I ask is this would be the easiest solution to my problem ie place 6 drives into an external enclosure, use 3 grouped as drive letter X as media storage and 3 drives grouped as drive Y as the backup.


Quote
A NAS is not plugged into your pc, its connected to your local network.

Can it not be physically attached to the HTPC  and then communicate with the HTPC through a wired usb connection? I imagined other pc's would connect to the NAS over the network, wired or wireless, for streaming. The reasoning here is that I would prefer a direct usb connection, music files to HTPC, and HTPC via ASIO to usb dac. This effectively is my current arrangement (except the exteranl usb media drive is a single 2TB HDD). I use MC library server/client to stream beyond the htpc, or can create a share to the media drive over the network etc.

Quote
Whether you use it for backups or as a file server, the mechanism behind is the same. It is setup the same. You might choose to use no or less redundancy but that's about it
 A NAS comes with everything you need to set it up for file sharing. Depending on the vendor, it may come with additional features such as DLNA server, print server, web server. Have a look at the product features from different vendors to get an idea what they can do..

JRiver is both a client and a DLNA server. If you setup a NAS a DNLA server, MC can use it as a client and stream music and/or movies from it. Personally, I would rather use the NAS as a fileserver and let MC be the server and client for media, but its entirely up to you. I'm not sure how tagging and file management would work if you setup your NAS a DLNA server for MC though. I think the connection is read only.

A slight confusion on my part. I am using "file server" generically as in any file (docs, audio,video,whatever).I gather the convention is to distinguish between ordinary files and media files...and even "UPnP AV" protocols ? Whatever the technical background I guess, as you say, a NAS will come with different server software/capabilities to suit its purpose.

More importantly, I'm not sure I need a UPnP/DLNA NAS setup per se as I use JRiver Server/client arrangement with the client having some write capability for tagging the server MC library. There is also the option of creating local libraries on pc's remote from the media drive and streaming from a mapped network drive.

So the issues for me seem to be getting a larger/more practical way of primary media storage and backup. Secondly, the possibility of streaming JRiver to non JRiver clients like a networked media player in another room. Can JRiver do this?....I am guessing that it can act as DLNA server and control point and then using something like GIZMO (I have windows not mac) to 'control the control point' ie 'see' and browse the library, control play etc .Inflatable, please don’t bother getting too detailed as its branching off into another topic and I will need to do further research anyway.

Thanks for all your help :-)
Cheers
David
Logged

InflatableMouse

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 3978
Re: Drobo- Gimmick or real world storage and backup?
« Reply #5 on: August 10, 2013, 07:10:26 am »

Putting aside NAS for a moment, so RAID can group multiple HDD into one volume or split them into multiple volumes seen by the OS?

A RAID controller (software or hardware) will create an array of 2 or more disks and present them to you as 1 large pool of 'raw disk space' called a logical unit which is distributed over all physical harddisks in the array. Depending on the hardware and software, you can create one or more "virtual" harddisks called logical disks from the space provided by the logical unit (although some (often cheaper) controllers automatically turn the logical unit into a logical disk visible to the OS). Once visible to the OS you can create volumes or partitions on them. Data written to the partition is distributed according to how the logical unit on which the volume resides was created.

If you're interested, you can read more here.

Putting aside RAID and NAS for the moment, can a HDD manager also group multiple drives into one volume as seen by the OS? I have Paragon Hard Disc manager (with virtualization) and it can obviously split or merge partitions...but afaik , not merge HDD into one volume. The reason I ask is this would be the easiest solution to my problem ie place 6 drives into an external enclosure, use 3 grouped as drive letter X as media storage and 3 drives grouped as drive Y as the backup.

Yes. Both Windows 7 and Windows 8 can do some form of pooling volume space together. With Windows 7 you need to convert the drives to dynamic disks and then you can expand a volume with the empty space on another (dynamic) drive. Now before you get excited, I **STRONGLY** advise against using dynamic disks. Just forget I ever mentioned it and pretend it doesn't exist. I even considered lying to you and simply tell you it's not possible, but I'm sure some other wiseguy here would jump in and tell you it is possible and I would have made a fool out of myself ;).

Windows 8 comes with something far more advanced than the relatively simple free-space combining technology Windows 7 offers. It might actually be worth looking into it if you have (or are considering) Windows 8. I have never worked with it and reviews I've read are positively mixed (if there is such a thing :P). Maybe Glynor if he's reading this can help you better with the pro's and con's of Storage Spaces. Google would also be very helpful as there is plenty to be found about it from respected websites like Arstechnica.com, Anandtech.com or Tomshardware.com.

Other options are Drive Bender and FlexRaid. I'm not advising either anymore as I've got no experience with FlexRaid and I had trouble with Drive Bender myself.

Can it not be physically attached to the HTPC  and then communicate with the HTPC through a wired usb connection? I imagined other pc's would connect to the NAS over the network, wired or wireless, for streaming. The reasoning here is that I would prefer a direct usb connection, music files to HTPC, and HTPC via ASIO to usb dac. This effectively is my current arrangement (except the exteranl usb media drive is a single 2TB HDD). I use MC library server/client to stream beyond the htpc, or can create a share to the media drive over the network etc.

Some NAS devices can be attached via USB or eSATA. It would however defeat the purpose of a NAS and downgrade it to a simple, dumb external harddrive cabinet. If that is what you want, you should look at external drive enclosures. Any sharing of the data on it you would have to do via the PC its connected to.

A NAS is actually a mini server. It has a CPU, memory, an OS and typically consumes much less power than a typical PC. Also expanding storage on a NAS is a lot easier and can be done online (often slow but still). I wouldn't really advise an external disk cabinet on USB unless you're absolutely sure that is what you want.

Secondly, the possibility of streaming JRiver to non JRiver clients like a networked media player in another room. Can JRiver do this?....I am guessing that it can act as DLNA server and control point and then using something like GIZMO (I have windows not mac) to 'control the control point' ie 'see' and browse the library, control play etc .Inflatable, please don’t bother getting too detailed as its branching off into another topic and I will need to do further research anyway.

Indeed, you can setup JRiver as a generic DLNA device and 3rd party DLNA clients (smart tv's, some receivers?) can work with that and you can always remote control your servers (and even clients I think?) with either a JRiver client or something like JRemote or Gizmo.

So the issues for me seem to be getting a larger/more practical way of primary media storage and backup.

As with all things, it depends on what you are willing to spend. If you want to keep the costs as low as possible, adding one or two drives to your current setup is the most economical thing to do. Your pc need to have room for them though, otherwise you can only upgrade the current ones with bigger drives.

A little more budget could give you an external cabinet, I would advise eSATA if you can, possibly add an PCI(-E) controller to give you eSATA if you don't have it. I believe some people here have done that so they might be in a better position to help you. USB would be my last choice as its too slow IMO. WIth USB you also need to be aware that not all devices power down on USB. I actually have several USB devices that continue to spin the harddrive even when I've shutdown the Pc they are connected to.

But if you can really spare the cash, I think a NAS device would actually give you what you need. You can add some of your own spare harddrives (data on them will be lost!) and you can expand the pools with new ones and easily replace/upgrade disks in a current pool. Note that not all vendors support all these features.

Another option if you feel adventurous is to build your own low power consumption home server :).
Logged

InflatableMouse

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 3978
Re: Drobo- Gimmick or real world storage and backup?
« Reply #6 on: August 10, 2013, 08:54:56 am »

Windows 8 comes with something far more advanced than the relatively simple free-space combining technology Windows 7 offers. It might actually be worth looking into it if you have (or are considering) Windows 8. I have never worked with it and reviews I've read are positively mixed (if there is such a thing :P). Maybe Glynor if he's reading this can help you better with the pro's and con's of Storage Spaces. Google would also be very helpful as there is plenty to be found about it from respected websites like Arstechnica.com, Anandtech.com or Tomshardware.com.

Maybe I should have been less optimistic. I've been re-reading some stuff and its not positive at all. People report a lot of problems with Storage Spaces, dissapearing pools, full pools while they are not full and incompatibilities.

Then, I read this.

Maybe you should forget about Storage Spaces as well.
Logged

Samson

  • Galactic Citizen
  • ****
  • Posts: 391
Re: Drobo- Gimmick or real world storage and backup?
« Reply #7 on: August 10, 2013, 07:10:06 pm »

Mr Mouse  :) Thank you so much for your excellent advice. I think you're right about non RAID pooled spaces/spanned volumes. Apart from anything else I also gather that one drive failure could potentially lose data on all discs. So some redundancy protection would be nice when pooling HDD's together.

[EDIT- RE NAS , INFLATABLE, I WILL POST A NEW THREAD ON THIS AS IS NO LONGER A DROBO QUESTION
http://yabb.jriver.com/interact/index.php?topic=82668.0]

The NAS thing seems to be a no brainer judging by its prevalence in forum posts.As you say it can do more than just protected storage (RAID) for large amounts of music/video storage but also acting as a server for those files.

As mentioned I would still need to assess whether I need this NAS server function if JRiver Library server is acting as my server to JRiver clients and/or JRiver is also setup as a DLNA server for DLNA clients.The HTPC stays running all the time anyway for recording TV shows (although I suppose it could be set to sleep in between TV recordings or for that matter when not serving a client??...haven't looked into that).

So assuming the HTPC is running all (or most) of the time and JRiver can provide all the server needs for streaming to clients and/or DLNA clients/renderers......the best storage option would appear to be a locally attached multi enclosure via eSATA (maybe USB3) utilizing  RAID redundancy/spanning multiple disks.

Having said that, I am guessing there are many JRiver users that are faced with a similar scenario and opt for a NAS. Some would be because they want their server located remote to the listening area and others may be realizing other server related benefits that are unknown to me at this point.
Logged
Pages: [1]   Go Up