1. If you would like to get rid of these banners and the ads within the posts, all you have to do is sign up and they will go away!
  2. Welcome to Rc-Help, are you here looking for the PDF Plans? They are located in the store in the tab above, but you must be signed in to access that part of the site. It only takes a second to sign up!
  3. Like Us On Facebook!
    Hello Guest, it occurred to us during that last server failure that we had no way of informing the members of the site of the failure and I would like to invite you to like us on facebook so that you can get updates if the site happens to go down again. We don't post much over there unless we are updating the members so we will not flood your feed like some places do. Just click on Like and then allow notifications and that's it. See you there soon!

    Click Here To Rc-Help's Facebook page!
    Dismiss Notice
Dismiss Notice
Hey Guest, I see you are not signed up on our forum yet. Did you know Registration is FREE and is only a couple clicks away? You can even sign in from Facebook, Twitter or Google+ for your convenience! So what are you waiting for, CLICK HERE to join in on the conversation!

Corrupted Server, That Sucks...

Discussion in 'News & Announcements' started by Tony, Jul 14, 2018.

< Site Down Time | No Sidebar Member List >
  1. Tony

    Tony Administrator Staff Member

    Blog Posts:
    14
    Joined:
    Feb 21, 2011
    Messages:
    36,152
    Likes Received:
    1,409
    Trophy Points:
    113
    I do apologize guys. I was working in my email program and all of a sudden I get a "no permission" error. I come to this site and get the same error. I go to Broke Auto Forums and the same message. No email, no sites, but I could still get into the server. I started out with restarting httpd (Apache), and it doesn't restart. It fails. I restart MariaDB, restarts perfectly. I try starting httpd again, nothing. So I decide to go in and look for the issues, starting with looking at the files for the sites.

    I click on File Manager under the BrokeAuto url and it comes back "nothing found". Okay, that is a little weird. I back out and try RCH, same message. Okay, so I log in as root, and I log into the directory under the root directory and the /home folder is completely gone. /home is where ALL the files are for ALL of my websites, public and private. It is all GONE...

    I had just eaten taco's and my heart sank to badly and the anxiety of losing the sites was so great that they almost came back up. I figured there was one more thing I could do and that is to just reboot the entire server. I fired up a command prompt and typed in "reboot" and a second later, The command window printed out the same "reboot" like an echo, but not a linux echo.... I waited 10 minutes and the server didn't come back up.

    It was then that I put in a ticket with my server company to go and pull the cord on the back of the server and manually power cycle it. Less than 5 minutes later, I refreshed the server page and it was back up. I quickly went into the file directory and /home was back... I have never in my life had a server selectively delete file locations. Not the files, but the locations and cause these issues. I have asked my Linux buddy if he has seen this hoping we can find a fix to the issue so it doesn't happen again.

    Sometimes, you just have to hate Linux. My uptime on the server was only 168 days. Not very long at all.
     


  2. Geena

    Geena Moderator Staff Member

    Blog Posts:
    0
    Joined:
    Feb 3, 2017
    Messages:
    889
    Likes Received:
    68
    Trophy Points:
    28
    Gender:
    Female
    Occupation:
    Certified Aircraft Welder
    Location:
    Sebring, Florida
    I have been trying to get in here all day, but I kinda figured you took it down for a bit for maintenance or something. I`m glad you were able to get it back up and running. That had to be a bad feeling to have that happen. But it`s back, and you can breath easy again. :)
     
  3. Tony

    Tony Administrator Staff Member

    Blog Posts:
    14
    Joined:
    Feb 21, 2011
    Messages:
    36,152
    Likes Received:
    1,409
    Trophy Points:
    113
    It was a horrible feeling! If we are planning on extended maintenance I will always let you all know ahead of time. This one was instant while I was working on an email. I don't even remember what email it was lol. But yea, when I saw that /home was gone, holy hell that was not a good feeling.
     
  4. murankar

    murankar Moderator Staff Member Armed Forces

    Blog Posts:
    9
    Joined:
    Dec 4, 2011
    Messages:
    8,954
    Likes Received:
    361
    Trophy Points:
    83
    Gender:
    Male
    Occupation:
    Soldier
    Location:
    Eastlake, Ohio
    In 20 years of linux I have never heard of that.

    The only thing I can think of is the "ntfs" portion got hosed. Basically if you use something like ext4 then your file system is journaled . What I surmised happened is that the journal got hosed and needed refreshed. I don't think you can command line a refresh of the journal.
     
  5. rdsok

    rdsok Well-Known Member

    Blog Posts:
    0
    Joined:
    Sep 11, 2011
    Messages:
    1,583
    Likes Received:
    60
    Trophy Points:
    48
    Location:
    Norman, Ok USA
    NTFS is a journaling file system... still, anytime anything is being written to the disk it's at risk of corruption whenever that process gets interupted.
     
  6. Tony

    Tony Administrator Staff Member

    Blog Posts:
    14
    Joined:
    Feb 21, 2011
    Messages:
    36,152
    Likes Received:
    1,409
    Trophy Points:
    113
    We were finally able to get in and do a SMART test on the drive (well, Ken did it. I have no idea about this stuff lmao) and there is definitely some errors on the drive. So we are working out how to get the files on the new drive since my host doesn't offer any kind of cloning service. So this may be a lengthy process, and this site will go down again, so be ready for it. I'm super busy for a while and Ken is also super busy so it is taking an abnormal amount of time to get this resolved. But it will get resolved. Please be patient while we work on this.
     
  7. murankar

    murankar Moderator Staff Member Armed Forces

    Blog Posts:
    9
    Joined:
    Dec 4, 2011
    Messages:
    8,954
    Likes Received:
    361
    Trophy Points:
    83
    Gender:
    Male
    Occupation:
    Soldier
    Location:
    Eastlake, Ohio
    I was having a Brian fart and couldn't remember ext4 so i said ntfs.


    Glad you found the issue. So you can't ghost the drive and replicate it. How long would a back up take and a reinstall of CentOs.
     
  8. Tony

    Tony Administrator Staff Member

    Blog Posts:
    14
    Joined:
    Feb 21, 2011
    Messages:
    36,152
    Likes Received:
    1,409
    Trophy Points:
    113
    the swap of the drive is simple and pretty quick. It's getting all the info on the new drive since they do not do anything at my datacenter. They just make sure the server is running, that's it.
     
  9. Tony

    Tony Administrator Staff Member

    Blog Posts:
    14
    Joined:
    Feb 21, 2011
    Messages:
    36,152
    Likes Received:
    1,409
    Trophy Points:
    113
    Hey everyone, as you can see the site is finally back up and running. A HUGE thanks goes to Ken on this one as he is the one that did EVERYTHING on the site to get it going. If my host would have just took 20 minutes and cloned the old drive, it would have taken about an hour total to get the site up and running. Instead, Ken had to do it the hard way. Dude is awesome!

    He did make a post under my name in this section explaining what he had done briefly. Man, this was a nightmare! We have never had this kind of issue before. Now I'm thinking about having a second HDD installed as a clone just in case... We will see what that costs.

    I want to thank you all for your patience!
     
  10. Geena

    Geena Moderator Staff Member

    Blog Posts:
    0
    Joined:
    Feb 3, 2017
    Messages:
    889
    Likes Received:
    68
    Trophy Points:
    28
    Gender:
    Female
    Occupation:
    Certified Aircraft Welder
    Location:
    Sebring, Florida
    I`m happy you got it all sorted, and I hope you never have to go through that again. It`s good to have RC-Help back online. :)
     
  11. D.O.G.

    D.O.G. I Support Rc-Help! Rc-Help Supporter Goblin 380 Supporter

    Blog Posts:
    0
    Joined:
    Jul 9, 2013
    Messages:
    2,804
    Likes Received:
    101
    Trophy Points:
    73
    Gender:
    Male
    Occupation:
    Disable
    Location:
    Inverness,Florida
    I agree with Geena post. Good luck from here on in :). Shout out to Ken.
     
  12. Tony

    Tony Administrator Staff Member

    Blog Posts:
    14
    Joined:
    Feb 21, 2011
    Messages:
    36,152
    Likes Received:
    1,409
    Trophy Points:
    113
    yea that was a nightmare to say the least. Definitely not something that I (or Ken) want to go through again.
     
  13. rdsok

    rdsok Well-Known Member

    Blog Posts:
    0
    Joined:
    Sep 11, 2011
    Messages:
    1,583
    Likes Received:
    60
    Trophy Points:
    48
    Location:
    Norman, Ok USA
    There was some good that came of it...

    ... I did backup's on my two websites JUST IN CASE of this exact issue... I guess that's no help though ( for you or Ken )
     
  14. Tony

    Tony Administrator Staff Member

    Blog Posts:
    14
    Joined:
    Feb 21, 2011
    Messages:
    36,152
    Likes Received:
    1,409
    Trophy Points:
    113
    If it is just a file backup, that is what we do every single day with a script Ken wrote. We had all the backups, we had all of the files from the old server (after he figured out how to get to them without killing the new drive...), but what we needed was a system image. I just talked to him about this and he is running single parody on his server, and quite honestly, I don't have a need for that much space or the money to pay for all of those drives or what they would charge me to set them up. So I'm just going to do a monthly system image backup. Unfortunately while it does the backup, it will have to put the site down for a while as it is doing the backup.
     
  15. rdsok

    rdsok Well-Known Member

    Blog Posts:
    0
    Joined:
    Sep 11, 2011
    Messages:
    1,583
    Likes Received:
    60
    Trophy Points:
    48
    Location:
    Norman, Ok USA
    The term is "parity" not parody....

    I did a complete WP backup which does the files and mysql backup. Still not the same as what you'd need since the forum db is changed on each post, so as you mentioned you need the forum down while backing it up.

    The backups that I do are actually redundant to the ones my server guy does and I believe that includes an offsite copy as well. In short, I'd only need them if the servers had to be completely reloaded AND the backups he does were also unuseable. If he had to put me back up on a fresh install, it'd only take me a few hours at most to resetup ( if that ) since my sites are relatively static ( no constant changes ).
     
  16. Tony

    Tony Administrator Staff Member

    Blog Posts:
    14
    Joined:
    Feb 21, 2011
    Messages:
    36,152
    Likes Received:
    1,409
    Trophy Points:
    113
    I know how to spell it, I just know it drives you nuts hahahaha.
     
< Site Down Time | No Sidebar Member List >

Share This Page