INTERACT FORUM

More => Old Versions => Media Center 13 (Development Ended) => Topic started by: JimH on March 26, 2009, 07:11:56 pm

Title: Our day today, or, why things may have seemed a little slow lately
Post by: JimH on March 26, 2009, 07:11:56 pm
We had a big day today.  We've been working on a project with MediaNet (http://www.mndigital.com/), formerly Musicnet.  MusicNet was formed almost ten years ago by Real Networks, EMI, Warner Music, and BMG, three of the then-five Major Record Labels.

We've worked with MediaNet for about five years.  Our players power many of their clients.  About a year ago, we were asked to take on some of the server work they had previously done.  Since July, we've been working on that project.  It has been slow-going, tedious, and time-consuming.  It has been a formidable challenge.

In the last month, we've begun to enjoy some success.  A few weeks ago, we moved some of the customers onto our server(s) (http://yabb.jriver.com/interact/index.php?topic=50403.0) for some of the functions they needed from MediaNet.

Today we moved the rest.

To describe what this would take a while, but you might imagine that some of us haven't slept well for a few weeks.

Today, we moved to the new servers.  It worked.  Then it didn't.  Then it did, but that was a mistake.  It wasn't working.  We found we had a small internal machine that wasn't able to keep up.  It was a server we've relied on for our daily bread for years, but today it saw a flood of new transactions and, lifting its skirts, tried to climb to higher ground.  She was getting hundreds of request a second, and the load average began to climb higher and higher.  What customers saw (and there were thousands enjoying the view) was a player that loaded pages in minutes, not seconds. 

We spent about 10 minutes considering possible solutions.  I was for going home and seeing what happened next.  Others said we could switch a language to gain speed.  Bob, our network/hardware go-to person, finally said, "Let's put it on that [under-the-table 64 bit AMD] machine."   

Three minutes later, we had decided to follow Bob and the new machine.  We carried the carcass into the server room, pulled the trusty server from her slot, powered her down, unplugged the cables, unscrewed the top and surgically removed her drives, controller, and Ethernet card, and transferred them to the under-the-table machine.

Seven minutes later, the new server booted.  Some fiddling with the Ethernet card, a reboot, and the loveable old server was running in new gear. 

It was fast.  It was beautiful.  256MB to 2GB of RAM.  7 year old processor to 2 year old state-of-the-art processor.  Jet lag peeled off.  Everything began to hum.

So now, I can write this, relax a little, and think of the superb team I work with.  And I can look forward to a good night's sleep.

That was our day today.  Tomorrow will be another.
Title: Re: Our day today, or, why things may have seemed a little slow lately
Post by: fitbrit on March 26, 2009, 09:49:28 pm
Thanks for sharing, Jim. Part of the reason I love MC is because you and your team work hard to keep us users as happy as you can.
Title: Re: Our day today, or, why things may have seemed a little slow lately
Post by: benn600 on March 26, 2009, 09:54:31 pm
Thanks for sharing, Jim. Part of the reason I love MC is because you and your team work hard to keep us users as happy as you can.

Thanks and ditto!
Title: Re: Our day today, or, why things may have seemed a little slow lately
Post by: p7389 on March 26, 2009, 09:55:37 pm
Sounds like you earned a good night of sleep!
Title: Re: Our day today, or, why things may have seemed a little slow lately
Post by: phalanthus on March 27, 2009, 02:49:05 am
sounds like you didn't even resort to alcohol - well done!
Title: Re: Our day today, or, why things may have seemed a little slow lately
Post by: hit_ny on March 27, 2009, 04:42:23 am
Kudos to you all  :D
Title: Re: Our day today, or, why things may have seemed a little slow lately
Post by: gappie on March 27, 2009, 09:21:29 am
always good to hear things are going well on other fronts to... hope everything was still running well this morning.

 :)
gab
Title: Re: Our day today, or, why things may have seemed a little slow lately
Post by: KingSparta on March 27, 2009, 04:58:06 pm
Quote
3.  Too much chocolate and too many animal crackers

I feel that way when I open a new box of girl scout cookies
Title: Re: Our day today, or, why things may have seemed a little slow lately
Post by: glynor on March 27, 2009, 05:34:25 pm
Great story, Jim.  I imagine it was rather more enjoyable to read about it than to live through it, but now that it is over (and you didn't fail) I imagine it comes with a huge dose of relief and accomplishment.

Isn't it fantastic to work with a group of energetic, dedicated, and creative people?  I know it is where I work!

Great work to you all, and here's hoping that Opteron keeps up the good work!
Title: Re: Our day today, or, why things may have seemed a little slow lately
Post by: ThoBar on March 27, 2009, 07:56:49 pm
Gratz for gettting through it alive.


Been there done that, happy to have moved on. Although it was fun for a while.

Our motto used to be "Good servers = good service" -  we used to overspec everything (almost stupidly so) - just because you never know when you'll need it. Keep all your hardware up to date and your system migratable. Of course, all this isnt cheap :(
Title: Re: Our day today, or, why things may have seemed a little slow lately
Post by: benn600 on March 27, 2009, 07:58:17 pm
It's (often) so much better to over spec, though, than to buy the bare minimum and then shortly later you want to add some new functionality.  Discarding or re-purposing equipment can be very expensive on all fronts!