INTERACT FORUM

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1]   Go Down

Author Topic: Our day today, or, why things may have seemed a little slow lately  (Read 3028 times)

JimH

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 72438
  • Where did I put my teeth?

We had a big day today.  We've been working on a project with MediaNet, formerly Musicnet.  MusicNet was formed almost ten years ago by Real Networks, EMI, Warner Music, and BMG, three of the then-five Major Record Labels.

We've worked with MediaNet for about five years.  Our players power many of their clients.  About a year ago, we were asked to take on some of the server work they had previously done.  Since July, we've been working on that project.  It has been slow-going, tedious, and time-consuming.  It has been a formidable challenge.

In the last month, we've begun to enjoy some success.  A few weeks ago, we moved some of the customers onto our server(s) for some of the functions they needed from MediaNet.

Today we moved the rest.

To describe what this would take a while, but you might imagine that some of us haven't slept well for a few weeks.

Today, we moved to the new servers.  It worked.  Then it didn't.  Then it did, but that was a mistake.  It wasn't working.  We found we had a small internal machine that wasn't able to keep up.  It was a server we've relied on for our daily bread for years, but today it saw a flood of new transactions and, lifting its skirts, tried to climb to higher ground.  She was getting hundreds of request a second, and the load average began to climb higher and higher.  What customers saw (and there were thousands enjoying the view) was a player that loaded pages in minutes, not seconds. 

We spent about 10 minutes considering possible solutions.  I was for going home and seeing what happened next.  Others said we could switch a language to gain speed.  Bob, our network/hardware go-to person, finally said, "Let's put it on that [under-the-table 64 bit AMD] machine."   

Three minutes later, we had decided to follow Bob and the new machine.  We carried the carcass into the server room, pulled the trusty server from her slot, powered her down, unplugged the cables, unscrewed the top and surgically removed her drives, controller, and Ethernet card, and transferred them to the under-the-table machine.

Seven minutes later, the new server booted.  Some fiddling with the Ethernet card, a reboot, and the loveable old server was running in new gear. 

It was fast.  It was beautiful.  256MB to 2GB of RAM.  7 year old processor to 2 year old state-of-the-art processor.  Jet lag peeled off.  Everything began to hum.

So now, I can write this, relax a little, and think of the superb team I work with.  And I can look forward to a good night's sleep.

That was our day today.  Tomorrow will be another.
Logged

fitbrit

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 4887
Re: Our day today, or, why things may have seemed a little slow lately
« Reply #1 on: March 26, 2009, 09:49:28 pm »

Thanks for sharing, Jim. Part of the reason I love MC is because you and your team work hard to keep us users as happy as you can.
Logged

benn600

  • Citizen of the Universe
  • *****
  • Posts: 3849
  • Living: Santa Monica CA Hometown: Cedar Rapids IA
Re: Our day today, or, why things may have seemed a little slow lately
« Reply #2 on: March 26, 2009, 09:54:31 pm »

Thanks for sharing, Jim. Part of the reason I love MC is because you and your team work hard to keep us users as happy as you can.

Thanks and ditto!
Logged

p7389

  • World Citizen
  • ***
  • Posts: 171
Re: Our day today, or, why things may have seemed a little slow lately
« Reply #3 on: March 26, 2009, 09:55:37 pm »

Sounds like you earned a good night of sleep!
Logged
[img width= height= alt=My Top Albums]http://lastfm.obsessive-media.de/3month/7x1/p7389.jpeg[/img]

phalanthus

  • Galactic Citizen
  • ****
  • Posts: 461
Re: Our day today, or, why things may have seemed a little slow lately
« Reply #4 on: March 27, 2009, 02:49:05 am »

sounds like you didn't even resort to alcohol - well done!
Logged

hit_ny

  • Citizen of the Universe
  • *****
  • Posts: 3310
  • nothing more to say...
Re: Our day today, or, why things may have seemed a little slow lately
« Reply #5 on: March 27, 2009, 04:42:23 am »

Kudos to you all  :D
Logged

gappie

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 4580
Re: Our day today, or, why things may have seemed a little slow lately
« Reply #6 on: March 27, 2009, 09:21:29 am »

always good to hear things are going well on other fronts to... hope everything was still running well this morning.

 :)
gab
Logged

KingSparta

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 20063
Re: Our day today, or, why things may have seemed a little slow lately
« Reply #7 on: March 27, 2009, 04:58:06 pm »

Quote
3.  Too much chocolate and too many animal crackers

I feel that way when I open a new box of girl scout cookies
Logged
Retired Military, Airborne, Air Assault, And Flight Wings.
Model Trains, Internet, Ham Radio, Music
https://MyAAGrapevines.com
https://centercitybbs.com
Fayetteville, NC, USA

glynor

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 19608
Re: Our day today, or, why things may have seemed a little slow lately
« Reply #8 on: March 27, 2009, 05:34:25 pm »

Great story, Jim.  I imagine it was rather more enjoyable to read about it than to live through it, but now that it is over (and you didn't fail) I imagine it comes with a huge dose of relief and accomplishment.

Isn't it fantastic to work with a group of energetic, dedicated, and creative people?  I know it is where I work!

Great work to you all, and here's hoping that Opteron keeps up the good work!
Logged
"Some cultures are defined by their relationship to cheese."

Visit me on the Interweb Thingie: http://glynor.com/

ThoBar

  • Citizen of the Universe
  • *****
  • Posts: 992
  • Was confishy
Re: Our day today, or, why things may have seemed a little slow lately
« Reply #9 on: March 27, 2009, 07:56:49 pm »

Gratz for gettting through it alive.


Been there done that, happy to have moved on. Although it was fun for a while.

Our motto used to be "Good servers = good service" -  we used to overspec everything (almost stupidly so) - just because you never know when you'll need it. Keep all your hardware up to date and your system migratable. Of course, all this isnt cheap :(
Logged

benn600

  • Citizen of the Universe
  • *****
  • Posts: 3849
  • Living: Santa Monica CA Hometown: Cedar Rapids IA
Re: Our day today, or, why things may have seemed a little slow lately
« Reply #10 on: March 27, 2009, 07:58:17 pm »

It's (often) so much better to over spec, though, than to buy the bare minimum and then shortly later you want to add some new functionality.  Discarding or re-purposing equipment can be very expensive on all fronts!
Logged
Pages: [1]   Go Up