INTERACT FORUM

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1]   Go Down

Author Topic: Servers Compromised [Now Repaired]  (Read 15168 times)

bob

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 13931
Servers Compromised [Now Repaired]
« on: August 20, 2018, 06:48:07 pm »

We are very sorry to say that at about 16:00 GMT our merchant server was taken offline followed by our webserver.
We took them down because there was a break-in.

EDIT on August 27:  We're mostly back up and running.  Still a few rough edges.  Thank you for your patience.

We will tell you more when we know more, but it looks like the first point of entry was rover, our merchant server.

What we lost:
No Credit card information was lost because we made the decision many years ago not to store credit cards.

We don't know for certain yet, but you should assume that password information from our servers may have been stolen. Change your password now on our servers and on any others where you have used the same password.

[Edit -- We now believe that the passwords were well protected with strong encryption.]

JRiver

Logged

JimH

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 72534
  • Where did I put my teeth?
Re: Servers Compromised
« Reply #1 on: August 21, 2018, 12:16:20 am »

I moved the questions to:
Security Questions and Answers

and started a new thread for discussion of stolen passwords.  There is a good chance yours is in that database already since some very large sites like Twitter have been compromised in the past.

I want to keep this thread clean and easy to read.  We will report everything we know here.

Please use those threads for discussion.

We're working on the servers first.  I'll try to keep you posted here, but the he information here won't come as fast as you might like. 

Thanks.  And my apologies for any problems we've caused.
Logged

JimH

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 72534
  • Where did I put my teeth?
Re: Servers Compromised
« Reply #2 on: August 21, 2018, 05:58:20 am »

In email, Bob described the method used:

"The license server got hacked, probably through the buy.cgi script which must have a flaw in it somewhere that allowed the hacker to use a sql injection script to get access as the postgresql user.

"They then set up a shell as that user to a server in romania and another in holland, so anything that the
postgresql user could see is probably compromised.

"Anyone can see the users in the /etc/passwd file (but not passwords since they are not stored there).

"They then downloaded software into /tmp to allow them to do a brute force ssh attack on the only machine the license server can reach which is webserver.

"They got into webserver as user "nobody" this morning some time. The user "nobody" runs the web server at a low privilege level."
Logged

JimH

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 72534
  • Where did I put my teeth?
Re: Servers Compromised
« Reply #3 on: August 24, 2018, 12:16:47 pm »

I just posted this in another thread:

"The server that hosts the Access Keys was re-built.  It's coming up, piece by piece today."

It's more than that.  It includes www.jriver.com and the forum.  I believe both are now running on the new server.  It seems faster.
Logged

JimH

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 72534
  • Where did I put my teeth?
Re: Servers Compromised
« Reply #4 on: August 24, 2018, 12:34:05 pm »

Just to add a little more detail, both our license server and our web server were thoroughly torched, so they have both been rebuilt on new hardware.

Both Bob, John, and Brad have put a lot of work into this so far and we're probably only about half done.  Thanks to them.
Logged

bob

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 13931
Re: Servers Compromised
« Reply #5 on: August 24, 2018, 06:36:58 pm »

The license server is back online so you can restore licenses, install and purchase new products with the exception of Paypal which is still being worked on.
Logged

JimH

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 72534
  • Where did I put my teeth?
Re: Servers Compromised
« Reply #6 on: August 27, 2018, 01:42:42 am »

There are still a few rough edges, but we're mostly back to normal, back in business.

The last big piece that was fixed was Paypal purchasing.   https://yabb.jriver.com/interact/index.php/topic,117210.0.html

Thanks to the team for all the work.  We're all better people now.
Logged

JimH

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 72534
  • Where did I put my teeth?
Re: Servers Compromised [Mostly Solved]
« Reply #7 on: August 28, 2018, 04:57:59 am »

YADB is still down, so lookups fail for cover art and track lookup.
Logged

bob

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 13931
Re: Servers Compromised [Mostly Solved]
« Reply #8 on: August 31, 2018, 03:12:18 pm »

YADB is back online for cover art lookups.
Logged

JimH

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 72534
  • Where did I put my teeth?
Re: Servers Compromised [Mostly Solved]
« Reply #9 on: September 15, 2018, 06:23:24 am »

Pix01 is up and secure, but you'll need MC 24.0.52 or higher  to upload, and you'll need to use the "forgot password" option to set a new one.

I believe everything is running as it should be now.  Please report any problems.

We also believe that, even if the password file was taken, the encryption was strong enough that it's unlikely that passwords were obtained.  It's still a good idea to change your password.

I apologize again for the disruption.  We failed.  We learned a lot.
Logged
Pages: [1]   Go Up