Let’s Debug It Live: Recovering My Hacked Blog
Tuesday, April 13th, 2010 | Author: lisaksimone

I’m writing this post in Word because I can’t access my blog. It was hacked this weekend to provide you dear readers with low cost designer shoes and a delightful Trojan virus thrown in for free.

It appears the Wordpress and Network Solutions combo had a design flaw, and Network Solutions sent step-by-step instructions to recover (most) of our content. Like a good engineer (ahem) I followed the directions carefully. Didn’t work. But according to Network Solutions tech support, I am a special case. (Oh, the irony.) Alex expedited my request and then Elmer prioritized my case and now I’ll receive help in 1-3 business days. Grrr.

Thing is, I dug around on the server and found some strange goings-on with my blog files. I think the current account hack fix upset the carefully balanced apple cart Network Solutions built during my last trouble ticket mess.

Symptoms

  • Email from Network Solutions “Alert: Incident Regarding Your Network Solutions Hosting & WordPress“   “Dear Valued Network Solutions Customer: We have confirmed that your site was among several hosted WordPress sites that were targeted by an unauthorized user …”
  • Followed by my immediate visit to my blog, which reported in big bold letters, “Error Establishing Database Connection.”
  • Followed by my immediate visit to the “Tips and Info for Network Solutions WordPress Customers“  provided in said “You’ve Been Hacked” email.

Aside - I love the post title, “Tips and Info.”  Implying a level of urgency and importance akin to “Plant a Beautiful Garden in Half the Time.”

I dutifully FTP’ed over to my content directory and changed my password in the wp-config.php file. (I’ve FTP’ed for 25 years. I imagine most blog owners who use canned templates were freaked out at this step.)

Anyway, the symptoms changed. Below is the config file followed by the new error.  (Yes, I changed the password for this post!)


Hypothesis 1

I don’t know PHP but I’ve coded C for years. They look similar, so I tried to track the error. I see nothing obvious like mismatches, etc.  But  I’m a little confused by the use of both /* */ and // for comments. Is the first #define for DB_NAME supposed to be commented out or not?

I wasn’t sure, so I removed all the comments and tried again.

The next error, shown below, keeps moving - it’s nested. And I don’t know PHP.

So I spent a little time on the Internet and started with index.php … which went to wp_blog_header.php which accessed wp_config.php which defined the path to Wordpress and my blog’s goodies. All lots of nice PHP files in the same directory.

But I found one disturbing reference among all these PHP files. While they’re all in the same directory, index.php (which I assume is the first one executed) refers to a parent directory.  And that file doesn’t exist there.  Ruh Roh.

But, hmmmm.

Hypothesis 2

PHP expert that I am (Think Neo “Now I know PHP”), I start thinking about problems with relative addresses like this. Is it referring to www.lisaksimone.com/stuff/morestuff or just www.lisaksimone.com/stuff ?

Then I remember a Really Big Change between two websites with Network Solutions a few months ago.

And it involved moving some entire website directories up and down a level.

** groans and grabs head **

Oh no, I thought we had that mess figured out.

Backing up in Time

Now, because my blog is still down, I don’t really know if this is related at all, but there’s enough of a suspicion for me to track this one down in real time and see how close we get when the dust settles.

Kinda like the Reality Debugging TV.

A Unix hosting package with Network Solutions comes with a website creator tool with some decent templates that can be used for *1* registered domain, and a Wordpress blog. But I figured out (legitimately) that you can host as many sites as you want in subdirectories, but they’re limited to HTML or anything that doesn’t require extensions.

So a little while back, my main website was for consulting - www.simoneconcepts.com. I used the website creator tool. Later, I got www.lisaksimone.com and wrote it in HTML because I mostly used it to point to other stuff. As per Network Solutions, the content resided in a subdirectory under simoneconcepts. Since I wanted my “Real Life Debugged” blog as part of www.lisaksimone.com, the blog content was located in the lisaksimone directory as such: lisaksimone/phoneonfire..

Here’s a “before” picture of the two websites and blog. Pretty clean.

Then last December 2009 I wanted to swap them so lisaksimone was my main site so I could use the web builder tool, and simoneconcepts would be relegated off to a subdirectory in HTML-land. Network Solutions told me no problem - just go to the control panel for each domain name and set the appropriate directory. In other words, make one the root and the other the subdirectory. I don’t recall when I thought about the blog, but I moved it up a level myself after talking with them. lisaksimone and phoneonfire were promoted appropriately, and simoneconcepts was demoted.

Mostly everything was fine. Some picture links were broken because of how Wordpress stores image files, but I thought I had it all worked out.

Until now.

When I dug around this morning, here’s the directory structure I found. Not so clean.

It SHOULD be like the “before” picture with lisaksimone at the top level, and simoneconcepts and phoneonfire as two folders within lisaksimone.  But … in addition to the appropriate directory changes, an extra partially populated combo of lisaksimone/lisaksimone/phoneonfire was still tagging along for the ride.

It looks like a disaster.

The extra phoneonfire content is partial - no PHP files, but there’s some content and images from January, and a plugin I installed about a month ago.

The blog has been working with string and duct tape.

That’s why I got really suspicious when I found the “./directory” reference this morning. Because I’d been off dot-slashing in the past.

Testing the Hypothesis

I want desperately to clean up the hangers-on’ers and delete the extra lisaksimone and phoneonfire directory. But I’m afraid to, because SOMEWHERE in this mess PART of the server code still points to the wrong place. And unlike NEO, I really DON’T know PHP and the rest of server side architecture.

So I have to wait 1-3 business days to see if *they* can figure it out. I’d like to give them a call and point them in the dot-slash direction, but I’m afraid to do that too, because some well-intentioned (being polite here) customer service representative may just do the same thing - copy content that’s “clearly in the wrong place.”

Escalating the escalated escalation

Network Solutions posted an email address (listen@networksolutions.com) specifically for people still having this problem, and I outlined what I’d tried here.  Within 4 hours, I got an email back from James, who told me my wp-config.php file had lost all it’s formatting.  After correcting it, the blog was back up.

Turns out I was on the right path with Hypothesis #1 - my suspicion about the commenting.  I’d noticed that a missing CR caused a define() to be commented out.  In testing the hypothesis, however, I removed not just what appeared to be comments, but some of the actual code itself.

Here’s what the file *should* have looked like.  Comparing this with the first screenshot above, any PHP expert could have figured this out right away.  In what LOOKED LIKE a massive block of comments, actual useful stuff resided.  Especially the line below about “wp_” and the second error code above that the function wp() was undefined.  Nice confirmation.

How did this happen?

So, to allay some of the pain, I looked at each file with a hex editor.  Turns out all CR-LF (carriage return-line feed, or 0×0D 0×0A) had been stripped from the file.  However, if  a lone LF occurred (like at the top of the file that looks kinda normal), it was left alone.  Some “text” editors perform this wondrous operation on save, leaving you with a blob of ASCII (as shown at the bottom of the original file, as Penguin Pete describes nicely.  Microsoft’s Notepad does this nastiness.

Another cause is FTPing using text mode vs. binary mode.  Even if it’s text, use binary mode so it doesn’t make assumptions about the whole CR-LF issue when moving files between PC and UNIX based systems.  I always use binary.

How did this happen?  I tried both Network Solutions FTP program and my own (CoreFTP).  I can’t tell you if it was screwed up to begin with, but after that, it looked like a train wreck.

Testing the solution

Pretty obvious.  Try old file, no blog.  Try new file, blog.  Looks fixed to me.

But what about second hypothesis about multiple not-quite-duplicate directory entries for the blog?  Seems to be unrelated, but a little disaster of its own.  So now that we’ve eliminated one problem, I feel free to work on the other one since all else appears fine.

So what about the the dot-slashing? (added content)

The real bug fix was part of the first hypothesis.  But if you recall (as bradeyh does in his comment below), what dragged me over into the second hypothesis was a reference in index.php … not to “wp-config.php” but to “./wp-config.php”.  And because all other PHP files were referenced directly, my mind noticed the difference and misread “./” as “../”  The former means “current directory” and the latter means “parent directory.”

Okay, my bad.  But, it’s really double my bad, because I forgot to loop back around and re-ponder the whole directory structure hypothesis.  I should have asked myself … how could referencing a non-existent file in a parent directory be overcome by fixing a simple formatting error in the config file?

So bradeyh convinced me to finish the job.  The directory structure problem I (just found out) I have because I moved website directories is not related to either 1) the bug for this post or 2) my misreading dot-slash as dot-dot-slash.

Take home

I did this debugging “live”.  I woke up this morning, kinda bent that I’d wasted a couple hours chasing that particular hypothesis when I could have been out riding a horse or something.

But I did the right thing.  A majority of the time, debugging is collecting symptoms, looking for patterns, making educated guesses and testing hypotheses.  That’s what we did here.  Just because what appears logical wasn’t the bug, the effort wasn’t wasted.

AND - a hypothesis based on an invalid assumption isn’t necessarily terrible either.

Because as it always seems, finding one bug and chasing it often uncovers its friends.  And now I’ve discovered why my blog acts a little flaky and now I have solid data for another bug-fix project.