Username: Password: lost p/w?
home | help | subscribe | search | register
Dmusic partial outage report
Posted by Otherleflaw in on January 29, 2008 at 3:19 PM

http://gallery.brawl-hall.com/data/media575322/7/loveyourjob.gif



SEE LAST THREAD FOR MORE RECENT UPDATE:


_______________

Update: We are testing streaming and downloading functionality of our new file servers today, with partial service expected to return tonite and tomorrow. We still have a way to go (migrating the file server only so far, webserver, email and othr functions still performing on old servers until total changeover of DNS. We apparently lost some data when a RAID 5 server failed, but we have have sent the drives to a recovery service. Backup failed because an ex- employee was asleep at the switch.

Dmusic is 10 years old this year, and we started on 2 Pentium 1's with 60 gig hard drives! We are now a terabyte of data, and it is only recently possible to put a site this size on ONE LARGE FAST SERVER, with another large fast server as mirror and backup. That's the plan. Sometimes you need a near catastrophe to get moving, and we see a silver lining in all this - a fast, modern, develiper friendly, reliable platform, which has eluded us so far.

____________________________________

___________________________


We were in the midst of a server migration to bigger faster servers, so we could enable video, the shopping cart for all artists(paypal required), protools support, and a bunch of new features.

BOOM - our file server quit ( Dell poweredge 2650 with Raid 5, if your interested.) We were in the midst of setting up the new servers when this happenned, so we had no place to restore the data to, except to set up new linux boxes ( debian 4.0 if you are interested, and buy a bunch of 750 gig drives to move the 500 GIGS of files ( takes DAYS!).

We debated about taking the site down temporarily (like a week), but since some functions are working ( forums, news, etc) we figured we would tough it out. We are still in progress, and most data appears healthy, although we might have to recover a scsi drive or two (which will take a week. Hopefully, you should see a faster and more reliable Dmusic in a week or so.


PLEASE BEAR WITH US! WE VALUE YOUR PATRONAGE AND WILL MAKE AMENDS WHEN WE ARE FULLY FUNCTIONAL --- FASTER AND EASIER TO DEVELOP ( boy could we could tell you stories...)

Thanks and stay healthy this new year!

_________________


User Comments

Otherleflaw
Date: January 3, 2008 @ 4:03 PM
Here at Dmusic it is imperative that we benchmark next-generation solutions by reinventing synergistic communities and empowering revolutionary metrics.

We take delivering enterprise web-readiness very seriously because in order to adapt to the growing markets, we must always realize that to visualize seamless systems is a very important part of any corporate mandate.

It has been an on going problem that most entities have been unable to synthesize open-source web services to the necessary scale of scale integrated e-markets.

To that end, we need to make it our mission to evolve compelling portals and streamline B2B portals to have any hope of ever driving impactful e-commerce.

Thanx!!!

_____________

courtesy of the bullshit generator ...
http://dack.com/web/bullshit.html
Electronicleedsquietman
Date: January 3, 2008 @ 4:09 PM
Thanks for the information Larry. Much appreciated.
AlternativeDistilled1
Date: January 3, 2008 @ 4:22 PM
So good to read something here and not just all the rants in the forums. Cant wait to see it in action:D (Big Grin)
OtherMojosnake
Date: January 3, 2008 @ 8:15 PM
Bowing Down thanks for the update larry !!!!!!!
Alternativeebarbarella
Date: January 3, 2008 @ 9:14 PM
Pop i believe, DMusic is the best place to be on the net Typing At Computer
Thanks for explaining whats happening to the site Heartbeat
Go DMusic
e.barbarella
Pop Dolls
Rockdsindel
Date: January 3, 2008 @ 10:06 PM
it's all good Larry...
We here at DM always weather the storm!
Sailboat
ElectronicChillinBuzz
Date: January 3, 2008 @ 10:09 PM
If you'd been a spokesperson for Bush after 9/11, I'd have bought the whole terrorist story. :D (Big Grin)

You'd be mad to let this place go down or into someone else's hands, Larry... Remember that. Nodding
Alternativehate9wicket
Date: January 3, 2008 @ 11:48 PM
Thanks for thinking "outside the box" and maximizing your throughput. Where is rokatie when you need her?
AdminShadowMom
Date: January 4, 2008 @ 12:17 AM
Okay, the second part made as much sense to me as the first part Hiding but I get the idea. AND my house is going to be really squeaky clean!! Thanks much for the update! Big Grin
DMemberpessimist
Date: January 4, 2008 @ 6:00 AM

rokatie January 4, 2008 @ 5:00 AM

box?

Folkvzeye
Date: January 4, 2008 @ 12:05 PM
Donald Duck Mickey Mouse
Donald & Mickey are now here to answer any questions...
Folkjoycedavis
Date: January 4, 2008 @ 6:20 PM
information like this is very very good. it diminishes my sense of loneliness and confusion.
peacePeace
Coderjack
Date: January 4, 2008 @ 8:51 PM
courtesy of the bullshit generator ...
http://dack.com/web/bullshit.html

damn it, foiled again..
AlternativeDistilled1
Date: January 4, 2008 @ 9:45 PM
deploy bricks-and-clicks vortals
AlternativeDistilled1
Date: January 4, 2008 @ 9:47 PM
harness cross-platform communities hmmm Dmusic crossing with Secondlife ;) (Wink) YAY!
AlternativeViciousKitty
Date: January 5, 2008 @ 12:50 AM
Tx for the effort I'm looking forward to a even better and stronger Dmusic!
R&B/Soul/Urbanimsproductions
Date: January 5, 2008 @ 12:57 AM
I understand the frustration you are going through and you have my full support all the way .... Dmusic is worth the wait.....
Otherleflaw
Date: January 5, 2008 @ 4:41 PM
I promise a rockin site when we are done with this migration. We could no longer develop on the old platform.

We are also setting up mirros, development servers, new load balancers, video server, new streaming radio, ....

all powered by dual AMD 64 bit processors with A FLUX CAPACITOR built in!!!
AlternativeDistilled1
Date: January 5, 2008 @ 4:43 PM
new streaming Radio????? yes yes yes yes!!!!
AdminShadowMom
Date: January 5, 2008 @ 7:06 PM
Eek! I think I want a raise! Faints Now I have to learn a whole new vocabulary, don't I? Laughing My Arse Off
AdminShadowMom
Date: January 5, 2008 @ 8:05 PM
Rolling On Floor Laughing! And...... back to the future!!! Razz
RockBOWILLIE
Date: January 6, 2008 @ 5:03 PM
Rockaorpete
Date: January 6, 2008 @ 7:07 PM
Big Grin MAN!!! THIS PLACE RULES!!! High Five!
I love this site - it's my home on the net... but with all these new functions this place is really gonna KICK-ASS!!! Electroshock
I can't wait for everything to be 'up & running' again so I can start directing all traffic from my other sites back here as before!!! Nodding
Looking forward to upping some new vids too!!! Hyper
AWESOME - CAN'T WAIT!!!
Thumbs Up
OtherIsrafel
Date: January 6, 2008 @ 8:50 PM
All of this sounds pretty advanced...will we have to wear helmets while surfing? Will there be new DM helmets? COOL!! Bouncy
OtherMojosnake
Date: January 7, 2008 @ 4:15 PM
i already wear a helmet most of the time Rolls Eyes url
but a dmusic helmet would be COOL!!!!
maybe one that holds my beer cans with a straw type thingy
Devil
AlternativeDistilled1
Date: January 11, 2008 @ 2:43 PM
http://88.198.24.213:8030/listen.pls

me DJing Live 320GB of DMusic and Indie Music only as well as some chatter and news!!!
1/11/08 2 pm EST until???
Alternativeebarbarella
Date: January 11, 2008 @ 3:26 PM
Pop Groovin definitely everyone should be listening to Distilled1 show ... great DMusic sounds, and what a DJ baby! Nodding Thumbs Up

e.barbarella
Pop Dolls
"Revolutionize back-end systems
Brand efficient methodologies
Cultivate next-generation content"
Wink that generator rocks! Laughs Out Loud
ElectronicChillinBuzz
Date: January 11, 2008 @ 3:35 PM
did this partial outage occur in the staff hiring program? :P (Razz)
Folkvzeye
Date: January 11, 2008 @ 5:19 PM
The problem all started when the monkey got too close to the camera.....
AlternativeDistilled1
Date: January 12, 2008 @ 3:06 PM
1/12/08 3 pm EST or -0600GMT live DMUSIC AND INDIE ONLY FOR YOU PLEASURE!
Live now http://88.198.24.213:8030/listen.pls
ElectronicMonnieB
Date: January 13, 2008 @ 7:51 AM
You know we love you... We'd never, ever leave this place.. Group Hug
AlternativeRikki27
Date: January 13, 2008 @ 12:43 PM
Personally, I blame the RIAA Laughing My Arse Off
ClassicalSaadya
Date: January 13, 2008 @ 4:18 PM
Leflaw sir, your first post sounds so corporate it scared me :( (Frown)
OtherBritmick
Date: January 14, 2008 @ 8:22 PM
Serves you right for buying a Dell.

:-)) (Very Happy)
Metalsedonarecording
Date: January 15, 2008 @ 5:54 AM
I was wondering why I could not listen to my music. I have been sending myspacers here to check my tunes and they have been saying that nothing worked.
Redundant Array, means server, Multiple SCSI, plus external tape, DLT/DAT and network copy. Even if you had a Pentium 100 with Redhat Linux Server and a 500gb hard drive, you can transfer the entire server over local FTP just as a backup.
Metalbillrhynes
Date: January 15, 2008 @ 8:11 PM
Flux capacitor??? Homer Simpson mmmmmmmmmmmmmmmmmmm, I like the sound of that
Otherleflaw
Date: January 16, 2008 @ 10:05 AM
17 gigs a day over a t1. Thats the backup speed.

Otherleflaw
Date: January 18, 2008 @ 10:41 AM
ETA ONE WEEK- I am told that we will be on line in with new faster servers and reconstructed cleaner code (10 years of crap removed) by January 26th.
AlternativeDistilled1
Date: January 18, 2008 @ 10:53 AM
Wooooo!
Rockceltaidd
Date: January 18, 2008 @ 3:05 PM
Thinking
OtherMojosnake
Date: January 18, 2008 @ 8:22 PM
BIRTHDAY JAN 27...DMUSIC BACK ONLINE JAN 26TH ...HAPPY BIRTHDAY TO ME INDEED!!!!!!!!
AdvancedThaspian
Date: January 20, 2008 @ 9:20 PM
I hope the stuff the video is hosted on is faster than the 50-55 KBps stuff the music is hosted on.

What's this about pro tools support?
Otherleflaw
Date: January 21, 2008 @ 3:23 PM
STATUS.

WE ARE SHIPPING A USB BACKUP of over 400 gigs of music files to the new servers. It took two weeks to make.


Otherleflaw
Date: January 21, 2008 @ 3:25 PM
Protools support:


Upload your session files....
download yours and other session files, add a track , do a mix,
and reupload.



HiphopEvolOne
Date: January 22, 2008 @ 6:08 PM
Alright, so we may be down for a minute, that just gives us all time to focus a bit on our music...good lookin' on the whole ordeal. Good luck with the transition, much love!

Peace,
One!

Evol 1 - 1 lovE
RockSupertzar
Date: January 23, 2008 @ 12:01 PM
OK,I was worried there for a minute.Good luck with the upgrade.
Othercantstoprunning
Date: January 25, 2008 @ 3:20 AM
any idea when things will be back up and running?
AlternativeDistilled1
Date: January 25, 2008 @ 7:31 AM
Today? or the 28th?
Otherleflaw
Date: January 26, 2008 @ 6:13 AM
We copied 400 gigs of data to new servers yesterday. I will know more at 3:00 today.
Otherleflaw
Date: January 30, 2008 @ 8:37 AM
Streaming and uploading is returning site wide as DNS changes propagate across web!!!
AdminCodeWarrior
Date: January 30, 2008 @ 12:39 PM
High Five Lef!
~Code
PopRichLum
Date: January 30, 2008 @ 6:14 PM
So once we can access the File Manager it is OK to upload new sings?
Adminrhthesinner
Date: January 31, 2008 @ 2:55 AM
Jak wrote in the Site Outage thread, "And another thing - if you can get in to the file manager (which means your DNS has updated, you can also stream!) - uploading works but DON'T upload anything, as it will be lost when we move over. Well, you can upload, but you wont be able to stream it and I'm not getting it back for you when we move!..."
ClassicalSaadya
Date: January 31, 2008 @ 3:32 AM
When you move over to where? I thought you already moved over. Why won't we be able to stream if we upload? Getting it back from where?

When will we be able to stream -all- files as well as upload new stuff?
Adminrhthesinner
Date: January 31, 2008 @ 3:39 AM
We are moving over. The DNS changes could take anything up to 48 hours. Which gives us quite a few more yet Nodding
Otherleflaw
Date: January 31, 2008 @ 9:37 AM
THIS TOP 20 STREAM WORKS for me!



http://music.dmusic.com//top/20/hifi/stream.m3u
Otherleflaw
Date: February 4, 2008 @ 2:49 PM
STATUS REPORT.


We have moved files.dmusic.com to new servers. They are 16 gigs of RAM. However they only had the 32 bit kernel of Linux installed, which sees only 3 gigs. We are updating the kernel today.

DNS move is mostly done, so we know are concentrating on moving the following:

webserver
file server
mail server
image server.
scripts, etc.

They have to be troubleshot as we go, so expect some rough seas for a few days.

Streaming seems ok for most of you, but uploading and downloading is still spotty. This shall improve as we go along.

We did not plan to do a live vivisection of the site, but when the RAID 5 crashed with aonly a partial backup, we made the decision to go full speed ahead with the migration, damn the torpedos.

I predict (gulp) that Dmusic will now be in a position to be more easily developed, to gain substantial traffic, since the old legacy system was full off DOS attacks, weird memory leaks, not enought RAM, etc.

Imagine how frustrating it has been on this end. I am sure its not been a picnic for you guys either.

Anybody want to buy a Dell Poweredge server?


Ps: we are also recovering the missing files from the Raid 5, and we believe that most will be recovered, so that you won't have to re- upload.

In the interim, I would wait a a few days before uploading a lot of files. Try 1 or 2 first, and let us know how it goes.

Your help and patience have been and will be appreciated.




Otherleflaw
Date: February 4, 2008 @ 6:44 PM
here's more:

Greetings,

I have installed the same set of packages as are installed on web1. I was able to pull a list of packages from dpkg and installed them on web2. This should give you the same base system as web1 to get DMusic going on the 64-bit server for testing.

Unfortunately, the data center technician didn't indicate which server he was working on, but I specified web2, as discussed during the conversation regarding 32 versus 64 bit.

All you need to do now is get the DMusic site on web2 and make sure it works so that we can eventually get web1 on a 64-bit OS.

Thank you for choosing us. Should you need to update this ticket, please log into your account and click on the Support tab, or you may call us anytime at one of the below numbers. We're glad to do what we can to help in any way possible!

Best Regards,

Sam C.
Linux Systems Administrator II
AlternativeDistilled1
Date: February 4, 2008 @ 8:02 PM
I'm having no luck with uploads as of yet.

Streaming was working but as of today it stopped as well as Downloading :( (Frown)
will test some later as well.
AdminShadowMom
Date: February 5, 2008 @ 12:57 PM
I don't speak Greek or geek... Rolling On Floor Laughing!
Electronicsilentpair
Date: February 5, 2008 @ 3:55 PM
granted, i AM incompetent -but it still seems funny from my end. when i try to delete the files in my file manger, it says "deleted" but also a "smarty error". they disappear from the list only to reappear when i sign out and look from a "visitor" perspective. i've also tried changing properties (no dl especially) and no dice there either.

not complaining, just seeing if there's anything useful i can provide. good luck.
Metalmilesov
Date: February 5, 2008 @ 4:35 PM
i have 4 or 5 songs that currently don't appear that should, file manager somehow was set to invisible, which i didn't do...but even when i put it back to visible setting, songs still don't appear on my music page...don't know if its related to all this fun stuff or what, but thought i'd point it out as silentpair did.
You must be logged in to post replies to news articles.
Log in or register with the form at the top of the page.

 

 

 

search

news tree


advertising



 

 
© DMusic LLC - Advertising | Employment | TOS | Subscribe