Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[InflatableMouse]

I just bought an LSI 8708ELP 8 port SAS RAID controller for less than 50 bucks, complete with both break out cables but no BBU.

I have 3 2TB Seagate drives in my server which have all been RMA'd once for bad sectors. The new ones are also accumilating reallocated sectors but not as bad as before and not enough to RMA them again. Besides, by now warranty has long run out. These drives are used for redundant backups, all Media is stored on 3 4TB Hitachi's (which have been running completely flawless).

I've been wondering what to do, besides the obvious: replacing them . I also have 2 2TB drives in my own pc along with 2 1,5TB drives. I don't need that much space in my own PC so with the LSI card I decided to create a 5x2TB RAID-5 in the server to place those backups on.

This is now running (just finished ) and seems to work fine.

However, in the past I monitored SMART counters for each drive individually and got email notifications when (pre)failures occurred, or when temperatures got too high.

Since the LSI card is now offering a virtual drive to the OS, the OS no longer sees the individual drives and is unable to monitor SMART counters.

How is monitoring of individual drives normally done with a setup like this?

As this is my first "real" RAID card, is there anything else I need to keep in mind? I intend to watch out for another LSI card to keep as a spare, just in case.

Thanks!

[jmone]

I think HDS still pulls SMART data from RAID cards, give the trial a go.  I have an LSI card but flashed the infrastructure FW so all ports are presented as std SATA ports as I am no fan of RAID for home use.  Good price!

[InflatableMouse]

The server is running Linux  .

I managed though, I found an article describing how to install IBM/LSI MegaRAID Management tools on Debian. Seems to work fine and I was able to enable mail notifications for different kinds of failures and warnings. That is to say, I received the test notification but I have yet to receive any "real" notification (I've enabled them all by means of testing).

I hope it works, we'll see 

[InflatableMouse]

Funny.

So I had 4 harddisks in my main PC, took 2 out to create the RAID-5 in the server. 2 x 1,5TB Seagates left in my PC. That was 3 days ago.

Both disks in my own PC failed just now, within 30 minutes of each other.

What's nuts about this is they failed without warning. BIOS and OS sees them, one of them seems to work, but both continuously click.

I ran a short self test which should complete in about 2 minutes. It took over 10 minutes of clicking and the test completes succesfully  ?. The other one completed the self test normally without any excessive clicking.

This is the reason why you should never buy 2 same model harddisks together from the same shop. As crazy as it may sound this is the most likely reason they failed at the same time.

[InflatableMouse]

I found a ~3 year old Areca ARC-1231ML SFF-8087, 12 port SAS/SATA controller with JBOD support. It's got 1GB of ECC memory on the card too.

Seems to work fine under Debian, OS sees native disks and the card has an ethernet port built on to it for management. Pretty nifty.

Important thing is that the way it initializes disks in JBOD mode is compatible with how regular SATA controllers initialize disks (needs to be able to swap them around if needed).

The guy who sold me the Areca also had several 2TB disks laying around so I got 2 extra disks too. 1 spare.

It's coming around nicely to building my own ZFS server with the 6 2TB disks in there .

[Hendrik]

The problem with such old controllers is that it can easily happen that they don't like discs > 2GB. I have a neat 8-port controller around which I can't use for that reason anymore.

[InflatableMouse]

Its not that old  .

I made sure the controller supports larger than 2GB (64-bit LBA support) and also supports full EUFI systems (even Apple Mac OS X ).

This controller will be used with 2TB disks and will be a "playground" so to speak for learning ZFS. If I like it I'll expand the 4TB disk array and move that to ZFS too.

I think its a very nice controller. Here's the user manual if anyone's interested.

[InflatableMouse]

I'm a moron  .

Although it supports JBOD its not a good controller for use with ZFS at all because it simply drops a disk if a read failure occurs. This works well when using the controllers' RAID functionality, but with ZFS this is highly undesired as you'd want ZFS to be in control of the disks.

I probably sell it as quickly as I bought it but its still a nuicance swapping hardware all the time and reconfiguring stuff.

The controller I need is an IBM ServeRAID M1015 SAS/SATA. Turns out its cheaper new in the shop than what I just paid for that Areca 2nd hand .

You can stop laughing now .

[glynor]

How much you want for it, and do you know if it'll support:

1. Windows 8.1 x64 (without hassle)
2. RAID-6 (the manual is a bit vague and I'm too lazy to look through)

I'm currently looking to buy an Areca or LSI 2x 8088 or 8087 (I already have a two-port 8087>8088 passthrough bracket for my PC) with decent RAID-6 support.  A three port one would be even better, but new they were too rich for my blood.

I was really looking for something new, but I'd probably take it off your hands for a good price.  I'm in no rush, so the absurd shipping across the pond time doesn't much matter (though I don't know what the shipping cost would be for that)...

[glynor]

Also, are you sure you can't turn that off?  Lots of RAID cards have options for that kind of thing.  Sometimes there's weird procedures where you have to stand on one leg, flash the firmware with esoteric command line options, while murmuring the secret incantation or some such nonsense.

The good and modern ones, of course, put those options in the Web UI.  But my old Highpoint cards, for example, make you do weird stuff like described above.

[InflatableMouse]

I'll look up your questions and drop you a PM with the details tomorrow, I just got home and I'm tired. Sleepytime .

[InflatableMouse]

Just about finished copying and moving stuff back and forth ...

Code: [Select]

# zpool status
  pool: datapool
 state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
repaired.
  scan: none requested
config:

NAME                                            STATE     READ WRITE CKSUM
datapool                                        DEGRADED     0     0     0
  raidz2-0                                      DEGRADED     0     0     0
    ata-Hitachi_HDS5C3020ALA632_ML0220F30Z09GD  ONLINE       0     0     0
    ata-ST2000DM001-9YN164_W1E0L80S             FAULTED     43   252     0  too many errors
    ata-ST2000DM001-9YN164_W1E0KXJV             ONLINE       0     0     0
    ata-WDC_WD20EARS-00MVWB0_WD-WMAZA1448932    ONLINE       0     0     0
    ata-ST2000DM001-9YN164_W1E0HYEX             ONLINE       0     0     0
    ata-Hitachi_HDS5C3020ALA632_ML0220F30Z0A1D  ONLINE       0     0     0

errors: No known data errors


Code: [Select]

Mar  2 11:09:23 paradise smartd[10389]: Device: /dev/sdf [SAT], open() failed: No such device

I'm off to replace a disk ...

[InflatableMouse]

Well, that was easy.

Replace disk, type zpool replace [poolname] [old disk] [new disk] and we're back in business!

Code: [Select]

# zpool status
  pool: datapool
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sun Mar  2 12:24:29 2014
    183G scanned out of 8.45T at 317M/s, 7h35m to go
    30.4G resilvered, 2.11% done
config:

NAME                                            STATE     READ WRITE CKSUM
datapool                                        DEGRADED     0     0     0
  raidz2-0                                      DEGRADED     0     0     0
    ata-Hitachi_HDS5C3020ALA632_ML0220F30Z09GD  ONLINE       0     0     0
    replacing-1                                 UNAVAIL      0     0     0
      ata-ST2000DM001-9YN164_W1E0L80S           UNAVAIL      0     0     0
      ata-WDC_WD20EADS-11R6B1_WD-WCAVY2473346   ONLINE       0     0     0  (resilvering)
    ata-ST2000DM001-9YN164_W1E0KXJV             ONLINE       0     0     0
    ata-WDC_WD20EARS-00MVWB0_WD-WMAZA1448932    ONLINE       0     0     0
    ata-ST2000DM001-9YN164_W1E0HYEX             ONLINE       0     0     0
    ata-Hitachi_HDS5C3020ALA632_ML0220F30Z0A1D  ONLINE       0     0     0

errors: No known data errors

I'm beginning to really like how ZFS works .

[InflatableMouse]

Code: [Select]

Mar 2 17:42:23 hostname smartd[6778]: Device: /dev/sde [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 119 to 111
Mar 2 17:42:25 hostname smartd[6778]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 117 to 119

Please say a prayer that these 2 disks will hold out at least another 2 hours or I'll loose my array.

Rebuilding takes another 2,5 hours. I just picked up 2 extra spares but would you believe this? 3 disks in 1 day?

I suppose they didn't like me copying stuff all day long and now the rebuilding ... suppose that's all taking its toll  .

Edit: Ha! I just learned that a RRER that is rising is actually a good thing. It says the rate at which errors occur is getting lower. /dev/sde is going down, which is a bad thing, more errors during reads.

[InflatableMouse]

I knew it wouldn't be long:

Code: [Select]

    replacing-4                                 UNAVAIL      0     0     0
      ata-ST2000DM001-9YN164_W1E0HYEX           UNAVAIL      0     0     0
      ata-WDC_WD20EARS-00MVWB0_WD-WCAZAA800271  ONLINE       0     0     0  (resilvering)

One more Seagate left to explode ... 

[jmone]

Good luck with those WD20EARS drives... You're going to need it!

Good Q - over my "fleet" of 2TB HDD from WD it seems to be model dependant, eg
- The initial WD20EADS models are still error free and going strong (512K Sector model - oldest has been powered up for 978 days), but they were discontinued in favour of
- WD20EARS drive which have now all failed on me (first of the 4K sectors), and I also have some of the latest model
- WD20EARX drives which have not yet shown any issues (but could be to early to tell - oldest has only been powered up for 247 days)
.....

[Hendrik]

I have a couple of EARS in my server as well, and they are still running strong. One failed over the years, but after 2.5 years of service (it was one of the very first I got), thats not too bad.
I have been slowly replacing a bunch of the oldest 2TB drives with newer 3TB RED drives though.

3 EARS and one EARX left in my setup, everything else are 3TB models, a couple greens, mostly reds.
Might pick up a few 4TBs next to replace those last 4 2TBs.

When your case is full, replacing discs with bigger ones is your last choice... but its not too bad, they are all approaching 3 years of lifetime (except the EARX), I feel safer replacing them anyway.

[InflatableMouse]

In this pool there is 1 Seagate left, there's 1 EADS and 2 EARS. The other 2 are Hitachi's.

The WD disks I've been replacing the Seagates with are already a few years old and are in mint condition if I trust the SMART counters. No prefailures whatsoever. Their temperatures are the lowest in the fleet too. I trust they will run for a while, but I should be looking for another spare disk (1 WD EARS left on the plank).

The other array consists solely of Hitachi's, 3 deskstars and 2 ultrastars. They are absolutely flawless. Incidentally I read that Hitachi has been rated most reliable HDD brand by quite a large margin. I think WD was 2nd.

Edit: found it:

[Hendrik]

The article that was the source of this image has been debunked quite often, it should be taken with a shovel of salt.
Here is one of the articles that discusses and questions the results: http://www.tweaktown.com/articles/6028/dispelling-backblaze-s-hdd-reliability-myth-the-real-story-covered/index.html

Not saying I generally like the Seagate drives, there have been repeated reported of their problems, but I don't think the data they posted is accurate either.

[jmone]

Yup - at some point the # of available ports / bays become the driving factor and I've now replaced all my 2GB Data Drives.  I see that Hitachi have their 6TB drive out ($800) and Seagate is talking Q2 for their product.

I was a big fan of WD till the EARS line (the EADS and EARX are fine).  Not only did they all fail, they all failed around the same time.  Also here in Aust their "local" RMA depot is Singapore & with the falling price of new drives they were hardly worth the cost of the postage.  I had one of my HDS 4TB start to show SMART errors (but had not failed) so "removed" it from my pool and sent it back to HDS here in Aust.  Got a new one back within the week.  Pretty happy with that.

[Hendrik]

I've been wondering if its worth $200 to get a bigger case with room for 3 more drives.
My case already hosts 14 drives, while the one I've been looking at can host 13 itself + the 4x3.5-in-5.25 cage I have now.

But I would also need an additional controller, so thats another $100-150 on top.

I would get one of those 19" beasts with 24 slots, but I'm afraid of the noise, as i need to be able to sit next to it and work in peace.

[jmone]

I figure my next move for more drives will be to try one of those USB3 / esata drive enclosurers (eg http://www.hotway.com.tw/products/h82-su3s2.htm )

[Hendrik]

I've always wondered if its possible to build such an external enclosure but make every HDD in it individually addressable to the OS, so that I can put them seamlessly into my FlexRAID configuration.
With an external SAS enclosure that might work, but those are enterprise level and cost a fortune.

[jmone]

I think they are all seen by the OS as individual drives (these are not a Raid boxs).  You are just constrained by the bandwidth of the one connection but that should be no issue for most.

[InflatableMouse]

What you need is a SAS controller that can operate in IT mode, instead of the default IR mode.

IT mode basically means passthrough to the OS, and bypass all RAID functionality.

I think you need to look into IBM ServeRAID M1015, Dell H200, and similar. They are all the same rebranded card from LSI which can be flashed with an IT firmware. They are relatively cheap too, compared to 8 port SAS RAID controllers. I know Dell has models with external SAS connectors too, for enclosures.

[Hendrik]

I have a SAS controller which just works like this (a LSI SAS 9207-8i). I just use it as a 8-port SATA controller.
The problem is external enclosures, those are the expensive part. If I would find a decent external SAS enclosure, I would get a second SAS controller with external ports instead of internal ones and connect it that way .. but those enclosures are rare and expensive on consumer market.

[jmone]

Here is a review of these external 8 bay USB / e-sata encloser.  As I expected, you only need one port to connect the unit, all drives show up as indiviudal drives, and you are constrained by the bandwidth of the one IF across all drives

[Hendrik]

Hm, I would at least like something like 100MB/s per disc (ie. 1GBit per disc), otherwise running them under RAID is going to be painfully slow during full checks or rebuilds.
Maybe only 4 discs is a better idea so the interface isn't limiting so badly. Too bad it needs a Silicon Image eSATA card for port multiplier support - but even then cannot reach the desired speed.

[jmone]

I too have the IBM M1015 (LSI) running the IT Firmware and it works well expanding the SATA connections in a now fully populated 13 bay case.... but where to next?  I'm OK for some time as I still have 5TB free in my pool, but was thinking my options if I need more space is:
- Higher Capacity HDD (but these new 6TB HDD would need to be a lot cheaper than $800)
- Replace the 3 ODD Bays with a 5 in 3 Chassis (and add more Sata ports):  Note, I've already got one of these in my "Backup - WHS case).
- get one of the external drive bays.

[jmone]

I'm sure there are better devices than they one I linked to as these are now old (2011) but did not research any further than this at the time.

[Hendrik]

Next Gen Intel Chipset is rumored to have Thunderbolt in the mainstream, maybe it opens doors for new devices?
I would be open to rebuild my server for that.

[jmone]

Mmmm.... thinking about speed, by HW bottleneck for me would stay as is the dual port NIC Card anyway (not the shared USB/estata IF), and even then I'd need to have something like 40 clients reading a BD direct from disk to hit that. 

[Hendrik]

I'm not worried about clients consuming the data, as its going to be 2-3 max at best, and only for playback, so they don't even fill up a 1Gbit link.
For me its important that maintenance tasks like a RAID rebuild or a full validation (which I run at least once a month) still work decently fast. And especially such RAID maintenance tasks then tax all the discs at maximum, so a limiting interface would be bad.

[jmone]

Yup, I agree - so for you, more SATA Ports, Dumb Bays/Case (like the SNT-3051SAS), and lots of cables with your current gear, or ... one of those 24 bay rack mounted chassis like the unraiders like!

[InflatableMouse]

Wow ... I may have trashed that Seagate for nothing  .

The new drive is acting up as well so I looked more carefully this time and it turns out the SATA port or cable is acting up. In my kernel log I see this port disconnecting and connecting. I didn't look last time because I asumed it was the drive as it was already having errors.

Gonna hook it up to another port this afternoon. Or use anther cable. I've had bad cables before.

[jmone]

Good luck!  I've had drives fail, cables fail and my data trashed (twice) by just moving my old RAID card to a clean build of the OS.... it is all a PITA 

[InflatableMouse]

Thanks!

Yeh I've been through that myself as well. That's one of the reasons I moved to ZFS for my storage. Any regular SATA controller can read the disks and no matter what order you plug them in or what type or brand of controller it is, ZFS always figures out which disk was what. Its self healing with checksums and transaction groups so corruption in this case is virtually impossible.


But I still have an off box backup of the most important stuff, just in case 

[apgood]

Raid card failing and trashing my data was one of my reasons for moving to unraid with the raid card just using IT mode. Works well, just wish unraid would add double parity at some stage. I know I could use a Linux distro with zfs but just don't have the time lately to look into it.

[jmone]

+1 on the IT Mode  and a full backup!

[glynor]

I haven't been following along, but Hendrik...  Have you seen these:
http://www.newegg.com/Product/Product.aspx?Item=N82E16816117307

A SAS expander, without the dumb enclosure.  And, best of all, you can use them with two bonded input 8087's to connect 16 total drives (so each SATA equivalent channel only gets multiplied once).

There's a 360 version too which could split them in three, if you prefer.

Should work beautifully with your LSI SAS card.  Just saying... Great way to build your own SAS expander.

[glynor]

Further details:
http://www.intel.com/support/motherboards/server/sb/CS-033145.htm

I wish ThunderBolt systems weren't so dumb.  The Areca one looks interesting, but I'd want to see firm numbers and stuff first, and you still have to buy is as an all in one.  Then, there the ATTO SAS bridge thing, which would be perfect if it didn't cost 100 million dollars (ok, not that much, but they're absurdly expensive).  But ThunderBolt could be so perfect...  Optical cables (now finally shipping) can let you stash the storage box REALLY far away, without resorting to 1gbps NAS connections or flaky iSCSI (also slow).

But, I don't think it quite makes sense yet, and there are no decent BYO solutions for a good price.

[apgood]

+1 on the IT Mode  and a full backup!

Always for irreplaceable data like photos, docs and home videos.

[apgood]

Or you could get one of these off of eBay:  http://www.ebay.com.au/itm/IBM-46M0997-ServeRAID-16-Port-SAS-PCI-E-Expander-Card-60Y0309-/121289982350?pt=US_Disk_Array_Components&hash=item1c3d72418e&_uhb=1

Got one for AU$140 and works perfectly with my M1015 (LSI rebadge). Note: The pcie slot is only needed for power, so you can probably power it with some sort of adapter if necessary and you don't have spare slots.

[Hendrik]

I haven't been following along, but Hendrik...  Have you seen these:
http://www.newegg.com/Product/Product.aspx?Item=N82E16816117307

A SAS expander, without the dumb enclosure.  And, best of all, you can use them with two bonded input 8087's to connect 16 total drives (so each SATA equivalent channel only gets multiplied once).

There's a 360 version too which could split them in three, if you prefer.

Should work beautifully with your LSI SAS card.  Just saying... Great way to build your own SAS expander.

Still needs an enclosure with a power supply, and such extenders usually take PCIe power, so needs some kind of adapter to get it going as well... Its just not easy!
I would take a dumb enclosure :p

[glynor]

That one doesn't.  Molex power only.

That thing, an 8088 to 8087 passthrough bracket, a bunch of 8087 to SATA pigtails, and an Addonics case?

[Hendrik]

All the Addonics cases I see on their website are full blown NAS cases with choices of external connectivty. Or did I miss the case with nothing but room for HDDs and my own controller?

[glynor]

Darn.  I wrote up a response to this thread earlier today and apparently never hit Post.  I hate it when that happens.

Yes, they do.

* the entire Storage Tower series - I have the Storage Tower IV currently
* the Storage Rack - which I'm thinking about getting
* the 1U RAID Rack with MiniSAS - which just has a single 8088 to SATA breakout cable and is all-in-one for four drives (so you could stack a few of these instead of getting one big 4U rackmount thing, and adding the separate hotswap backplanes)

They also have a variety of backplane and snap-in modules to make them all nice and hot-swappable, though since the regular cases are just piles of 5.25" bays with the needed Power Supply jumper board, switches, and whatnot, you could use whatever you want.  StarTech, IcyDock, IStarUSA, and all kinds of other vendors sell simple hot-swap SATA backplane things that fit into 5.25" bays.

My current setup consists of:

* A ST4 base unit (no extra multipliers or anything, just the box, PSU, and whatnot)
* Two Disk Array 3SAs mounted in the four 5.25" bays in the ST4.  This gives me 6 3.5" hot-swap bays.  Those backplanes have some kind of built-in RAID support, I think, but I've never turned it on.  They ship by default in regular SATA mode, and they have three discreet SATA ports on the back.  There are bigger 5ST versions that fit 5 drives in 3 5.25" bays available too.
* An 8-port SF-8088 to SATA I/O adapter on a Centronics plate (mounted on the port provided on the back of the ST4)
* A two-port SFF-8087 to SFF-8088 passthrough adapter mounted in my PC (and attached to my RAID card's 8087 outputs, because buying the cards with 8088 included costs a heckuva lot more than $49).
* Some SFF-8088 cables from Monoprice.
* A crapload of very short SATA cables from Monoprice.
* Two of these to liberate the two extra SATA ports I have available in the ST4 but which I can't use because there isn't enough room in the box.  The ST4 has two punch-out spots for these on the back, along with the Centronics punch-out spots.
* A crappy old two-8087 Highpoint RAID card, which has been surprisingly rock-solid, but which doesn't support RAID-6 and doesn't perform anywhere near as well under Windows 8.1 as it did with Windows 7 (my typical sustained throughput rates dropped from around 400MB/s to ~300MB/s or so, and it seems to be worse with random I/O as well, though I haven't tested it thoroughly)

In general, I'm very happy with the setup, and it has been running for years now.  My issue is that... I'm maxed out in the case (using all 6 bays) and have been for a while.  So now those two extra eSATA ports are connected to junky external eSATA drive cases with two more drives stacked up on top (for a total of 8 drives, using all of the ports natively available on my RAID card).  That stinks, plus it is mega loud.

I replaced the two fans in the Disk Array 3SA back planes with Noctua fans, and they're much better now than how they shipped.  But the ST4 still has a junky little tiny fan (almost certainly not needed with the Noctuas on the 3SAs, but I've never disconnected it) and a loud, crappy fan on the included PSU (which is custom looking and I'm afraid to work on PSUs if I need to unscrew them substantially).  Its in the basement, so I don't really care much, but I would like it if it wasn't so loud.

I'm considering...

* Storage Rack DA (the only difference between the regular and the DA is that the DA has an extra bracket at the back for one more Disk Array 5SA if you want to use it, and they're they same price)
* A nice Zippy PSU.  The Addonics PSU in the ST4 has (surprisingly) held up for a long, long while.  I think it has been going close to 24/7 for... Four years?  Five?  I could look it up, but it is a long, long while.  So, it would probably be fine, but for the exact same price (because they sell the Storage Rack without a PSU for cheaper) I can get a nice, high-quality Zippy one from Newegg, and it'll probably have a better fan.
* Another one of these, with the Centronics bracket probably (though I think the Storage Rack has some regular PCI slot mounting spots so maybe not, I need to re-check that before I buy)
* The Intel thing I linked above.
* One Disk Array 5SA, Snap In Pro backplane, or maybe a third-party hot-swap backplane of some kind.  I've been moderately happy with the hot swap Disk Array things, once I replaced the fans.  They work pretty well.  The locking lever things are pretty fiddly though, so I might decide to go with some different ones if I can find ones that look decent.  Not sure on this, as I haven't decided yet.
* A new RAID card that is better and has RAID-6 support
* Some under-desk mounting brackets
* and, then, later when I rebuild my money supply, some 8-meter SFF-8088 cables, so I can move the Storage Rack to another room from my office.  I've measured it out, and could probably even pull it off with 6-meter cables, but they're so freaking expensive (and I'd need two).

That last piece is what has me peeved.  I'm extra peeved that my ASUS P8Z77-V Deluxe board was supposed to have the ability to add in a modestly-priced Thunderbolt EX card aftermarket to add in Thunderbolt ports, but apparently ASUS got in a fight with Intel and they wouldn't certify it.  Now it seems that Intel has backed down and has certified their next-gen Thunderbolt EX II card, but they're not back-porting it and making it work with the Z77 boards (I think, anyway, the specs list only the Z87 boards and ASUS has been totally silent on the screw up on the Z77 Thunderbolt headers).  Supposedly.  I'm hoping someone tries one of these on the older Z77 boards and it works.

Pisses me right off.  Part of the reason I bought this board, and not the competitors, was the possibility of upgrading to Thunderbolt for this exact project.

So, I don't know what to do.  My Laptop and Mac Mini both have Thunderbolt, so it would be convenient to use something like the ATTO ThunderStream box instead of getting a new LSI or Areca SAS RAID card.  If I did that, I could run it all over one cable.  And if the Server itself ever goes belly-up (again) then I could just unplug the ThunderBolt cable from my Server, plug it into the Mini, and I'd be back up and running (slightly crippled since the Mini wouldn't support my internal TV tuner cards).  Then instead of paying upwards of $800 for the two SFF-8088 8-meter cables, I could get a single fiber ThunderBolt cable and run it to the RAID box, and maybe upgrade that performance even further later on.  At the very least, I wouldn't have to pay two gouging prices for two long SFF-8088 cables, only the one fiber ThunderBolt cable, and that substantially eases the length-limitation and I could run the darn thing however I want.

But, I'd be getting doubly-gouged on the ATTO ThunderStream box, and I don't know how much I trust them compared to LSI or Areca (ATTO has a decent name, but I don't know much and the stuff has limited reviews outside of places like OWC).  I don't know... That seems clunky, especially since the current ThunderStream is Thunderbolt 1 only, still.

I've been hoping that some kind of mythical version of that ThunderStream box (preferably in a 5.25" bay form factor with 8087 connectors internally) would come out for $500-700.  That would be a bit more than I'd pay for a good LSI RAID card new.  Then, if the new ThunderBolt EX II card is confirmed to work perfectly with (though not certified for) my ASUS board it could work...  But I'm thinking that's not looking good (probably on either front).  I had a bunch of high-hopes that Areca would sell a version of the thingy they're using to power their ThunderBolt RAID-in-a-Box solution, but they're already on gen-2 and it's never appeared (I wonder if this too isn't caused by dumb Intel certification issues.)

It sure-as-heck isn't worth upgrading my otherwise-perfectly good (and rock-solid) server board just to get ThunderBolt, only to get gouged on the fiber ThunderBolt cables, and it certainly isn't worth it for a $1300 RAID card thing in a box with a pair of ThunderBolt 1 ports slapped onto it.

Sigh.

But still... My idea above could work brilliantly.  It'd let me immediately:

* Rid myself of the dumb external drive cases, which are loud, ugly, and have crappy-prone-to-failure power supplies (one of the two is only a hot-spare, so I can afford to lose both at once, but still)
* Give myself a ton of future expansion space
* I can quickly bump up to 10-12 drives and set up a RAID-6 volume
* I can do some of this in "phases".  I don't NEED to replace the existing SAS RAID card now, if I want to do it on the cheap, and I don't need the Intel thing at all unless I'm going to more than 8 drives (which I don't plan to do immediately)

[glynor]

More on my ThunderBolt-wishful (constantly searching for options) quest...

I've been hoping that some kind of mythical version of that ThunderStream box (preferably in a 5.25" bay form factor with 8087 connectors internally) would come out for $500-700.  That would be a bit more than I'd pay for a good LSI RAID card new.  Then, if the new ThunderBolt EX II card is confirmed to work perfectly with (though not certified for) my ASUS board it could work...  But I'm thinking that's not looking good (probably on either front).  I had a bunch of high-hopes that Areca would sell a version of the thingy they're using to power their ThunderBolt RAID-in-a-Box solution, but they're already on gen-2 and it's never appeared (I wonder if this too isn't caused by dumb Intel certification issues.)

It sure-as-heck isn't worth upgrading my otherwise-perfectly good (and rock-solid) server board just to get ThunderBolt, only to get gouged on the fiber ThunderBolt cables, and it certainly isn't worth it for a $1300 RAID card thing in a box with a pair of ThunderBolt 1 ports slapped onto it.

Sigh.

One thing I have considered, seriously, is getting one of these Sonnet ThunderBolt 2 PCIe boxes, and then building my own ThunderBolt RAID-in-a-box.  That would be perfect.  I could even use my existing Highpoint RAID card and 8088 to 8087 passthrough PCI slot thing with it (since the bigger Echo Express SE II has two usable PCIe slots).  But, the dumb thing is $499.  There's a one-low-profile PCIe slot version of the Sonnet thing available for $100 cheaper, but then I'm limited to buying a new RAID card with built-in SFF-8088.  The low-profile restriction shouldn't be that big of a deal, but that's annoying, and I can't "do it slow" and use my existing RAID card for now.

And, I'm not sure if those boxes are a bit flaky or not.  Initial reviews of the new ThunderBolt 2 versions have been positive, but they're new, and...? It shouldn't be flaky.  ThunderBolt is just PCIe in a different cable, so they just have to extract it without screwing up, but I know the original Echo Express boxes were a little picky about what devices they'd support.  I think most of the problems were people trying to use them with 16x GPUs and not storage adapters (which they're basically built for, along with audio cards and 10GBe adapters), but I haven't researched it super-thoroughly.

But...

None of that solves the problem that ASUS screwed me on the ThunderBolt headers thing.  Grrrr....

[glynor]

* the 1U RAID Rack with MiniSAS - which just has a single 8088 to SATA breakout cable and is all-in-one for four drives (so you could stack a few of these instead of getting one big 4U rackmount thing, and adding the separate hotswap backplanes)

I hadn't thought of this option until now, but...

With the 1U RAID Rack MiniSAS box, I wonder if you could Frankenstein on one of these in the back port cluster area without it being too junky (or maybe two Centronics plates side-by-side).  The one they're shipping it with is clearly a Centronics plate, but the actual port cluster cut-out is bigger (as you can see easily if you look at the other ones with all the USB and FireWire ports and built-in RAID).

Hmmm....

Then, you could mount the Intel SAS Expander thingy in the back part of the "top" U1 Rack case (there's a bunch of empty space on the right), and run 8088 cables OUT of it to daisy chain to additional single-8088 1U boxes.  But, I think you'd still be limited to three of the 1U boxes total (two in, and two out).  Might end up roughly the same price though, if that would meet your needs.  The 1U SAS-only boxes (R1MAS) from Addonics are only $259 each, so...

Probably not, and you'd be limited for expansion (it'd take a bunch of those to get to 24 drives, which I could stuff in the Storage Rack easily with some backplanes).  But, makes me go hmmmm....

[InflatableMouse]

I've been having trouble with disks dropping out on an almost daily basis. After replacing disk #3 I started thinking there must be another cause for this. With the first two I didn't really think about it as they were Seagates with bad SMART counters, it made sense. I'm thinking I might have trashed those disks for nothing though 

I replaced the PSU today and so far, things are looking good. Hopefully this was the cause for all those problems in the past few weeks.