I suspected my antivirus/firewall software had a virus. Ya see, out of nowhere at 7am over a nice cup of coffee, BitDefender screamed at me, “Your system is NO LONGER PROTECTED!

Huh?!? I renewed it last month. It’s been running fine!

BitDefender further admonished, “The specified key is not valid. Please enter a valid license key.” After much cajoling and growing ire on my part, it refused to yield, and dragged me way down the rabbit hole.

This is one of those debugging scenarios where you DON’T get access to the source code. OR access to any tool to debug it faster than real time. So, we’re stuck debugging by permutation, with the results of our testing presented as snippy little error messages. But pictures are so much fun, so let’s have a go at it.

(And one of those big hints… Party Like It’s December 31, 1969.)

Symptoms and Stuff I Noticed

Well, here is the second and more concerning symptom-thingie:

So I figured I’d just re-enter the new license key I got last month for Internet Security 2009. While it happily accepted it when I first received it, BitDefender now tells me my license key is for a DIFFERENT product? (I have no other product with this company. Pretty sure that’s not gonna change.)

It was at this point I suspected a virus.

But at the bottom of this screen (not shown) I have the option to log into “My Account.” Another brandy-new error.

Sigh. No soup for you.

My suspicion of a virus grows. “try again later to CREATE the account”? First, I have a valid account. Second, it tells me it can’t Phone Home…but it knows I have to CREATE an account when I’ve selected the “Sign in” on the left and not the “Create a new” on the right?

As my MS adviser said years ago, “How do it know?” Hmmmm.

So I kept digging, eventually redirected to the following page which informed me that my license key was invalidated because it had been pirated. (But I AM invited to purchase a new license… heh heh.)

Perhaps my virus paranoia is still valid.

Danger Will Robinson

It just seems the rabbit hole gets deeper and deeper. So let’s run kicking and screaming away from a suspected virus and see what the company’s website can tell me.

At www.bitdefender.com directly, I found a different login screen, continuing my exponentially growing concern for infection.

But, I could log in!!!

Anyway, I blasted over to the “My Products” tab to check my registration status. Whew! My registration is still valid. Quickly, I grabbed a screen shot in case fights with BitDefender ensued. Part of my freakout was effectively quelled with the following now tucked nicely in my back pocket (external hard drive):

YES! I am still GOOD!

But … I still suspect a virus. After all, there is a significant difference between what my computer is telling me, and what my account online is telling me.

Virus Paranoia

I knew folks who got hit by the April 1 virus, so my paranoia continued. Even though this problem didn’t start for me until April 4th. But I ran off to get the latest Windows Updates, then to run Spybot and Malwarebyte’s Anti-Malware (an excellent program)! No problems found. But the entire process took a couple hours and required a couple reboots.

A Big Clue

But … after I’d finished with the updates and virus checks, the sad little grey BitDefender icon turned happily back to red.

Red. Which means BitDefender feels I am worth talking to again. WTF?

So with that, I was able to launch the program and actually navigate around to check my registration status in the program itself. Please note that this information is NOT available in ANY WAY unless you have a valid key. BitDefender essentially removes vital information to help you when you have a problem.

Here’s what I found:

So - some good brandy-new information here. What do you see here? Anything funny?

New Information

  • BitDefender got happy again after I did Windows updates and ran a couple malware programs. And then rebooted.
  • After BitDefender got happy, I was able to access more information about my registration history.
  • The “Name of the action” above confirms BitDefender’s fickle opinions of me. The bottom two lines dated 3/28 are real - when I had to buy another year. The top two are the first incorrect expiration at 3AM, with successive goodness and happiness restored at nearly noon.
  • ….and…coolness in a way… my license key expired when? December 31, 1969 at 7:00:00 PM.

Let’s pause and let that sink in.

Well of COURSE my license is expired. It apparently expired nearly 40 years ago! Clearly I have several back-registrations due before BitDefender will happily return me to its good graces.

Anyone recognize this date? December 31, 1969 7:00:00 PM.

This is obviously a massive clue, and finally a clear symptom we can latch on to.

If you’ve been around for a while, this date is familiar. UNIX time started on January 1, 1970 UTC. Time has been incrementing from 0 every second since that time, and this number can be converted into the familiar Date/Time format. Here’s what 0 translates into using this nice little converter.

UTC (coordinated universal time) is the same as Greenwich mean time (GMT) so you will see these used interchangeably. I write this in the Eastern standard time (EST) zone, which is 5 hours behind UTC. Therefore, the second that UNIX time started on 1/1/1970 00:00:00 AM UTC, the time at my house would have been 12/31/1969 07:00:00 PM.

As a current example, at the time I write this sentence for the first time, UNIX time is 1239048720, which translates into Mon, 06 Apr 2009 20:12:00 GMT.

Got it? Ahhhhhh.

That means any time you see the magic date December 31, 1969, you know that somehow a UNIX time of 00:00:00 is involved. Accidentally.

Sound like one of those boundary conditions? Accidentally using a variable that hasn’t been initialized?

SEEM LIKE IT COULD BE? ;-)

How could this happen? UNIX time = zero?

1) A common cause of this happens when the battery supplying power to the real time clock in your computer goes dead. When you turn your computer back on, it reads the current date stored with power supplied by that little battery. But with a dead battery, the current date (in UNIX format) is zero, so the Date/Time on your computer shows up with the magic December date. And your computer thinks Party Like Its 1999 is just a grand bash 30 years in the future.

So, one cause is dead battery.

2) Another cause. UNIX time is just a number counting from zero. “Time” has no colons, AM/PM, etc. So, if any operation on time results in a zero value, it is interpreted as the magic December date.

What can cause this? Bad initializations. Subtraction to determine differences in time are especially suspect.

Hypothesis

1) Is my computer clock battery dead? Could be, but this problem never happens during normal operation, and somehow only magically happens at the EXACT TIME BitDefender wants to update my virus signatures and check my registration?

Ahhh, no.

2) I vote for #2. I think it’s a subtraction problem because license keys are typically valid for 1 year, or 365 days. How do you compute that? Sheesh. Let’s check it out.

BitDefender has a history log file that shows all communications (I hope) with the Mothership. On my machine, it’s located at:

C:\Documents and Settings\All Users\Application Data\BitDefender\Desktop\Events\history.xml

This section of the log file shows three separate events: 1) an update, 2) registration check, 3) an update.

Event 1 - Calling the Mothership to get updates.
Event 2 - Registration expires on the magic December date.

And very suspiciously, I see two identical timestamps for these sequential events … 1238935453.

1238935453 = Sun, 5 Apr 2009 12:44:13 UTC

Hmmm. Did someone accidentally change the

days left = current event time - original registration time
expiration date = current event time

with

days left = current event time - last event time
expiration date = days left

And with zero being the result of that operation, assuming that NO days were left, and using the difference rather than the base number to report the expiration date?

When I look throughout the entire history file, I see the same thing. So, COULD BE!

Summary of Findings to Date

  • License key expired every day early in the morning
  • License key UN-expired after reboot (does this force a registration check that is not logged?)
  • December 31, 1969 is a clear indication of a zeroed variable accidentally assumed to be a UNIX timestamp value (0) which is then converted back to Date/Time timestamp.
  • My log file shows obvious places where the result of an inappropriate UNIX timestamp subtraction is zero, and the corresponding Date/Time is December 31, 1969.

Grrrrr, BitDefender is trashing me because some software guru decided to do a date check using an improperly uninitialized or zeroed variable. And I use the term guru with significant sarcasm. But, I have to prove this, Type A engineer that I am.

A major smoking gun, though.

So now what to do?

At this point, BitDefender has gotten temperamental with me three nights in a row. Pardon my frustration, but I don’t need this crap every morning. I am pretty damned convinced that

  1. This is not a virus
  2. This is the December 31, 1969 issue
  3. The issue is NOT in my computer, but with BitDefender’s software

So now what? Armed with this information, of course I go to the internet. For two days, I can find nothing related to this issue. But …. every morning to accompany my morning coffee … I get more delightful news after the newly-obligatory reboot.

Oh please just shoot me now.

So, over the course of three days (insert Vent-Rant here), I did the following:

  • Submitted trouble tickets to variously advertised help email addresses, including one called invalidkey@bitdefender.com. I currently have 4 open tickets, with responses to none. I completely answered multiple questions, provided log file dumps, and attached screenshots at their request. ALL ignored.
  • I initiated the Live Chat and waited, eventually “Bob” appeared and then immediately closed the connection.
  • Sat on hold on the customer support line, on my own nickel. Bailed out after 25 minutes.

Grrr.

Not working in a vacuum - talk to other people

I found online that people frustrated with BitDefender were not offered refunds, so you are generally out the money. Well, I can’t tolerate my antivirus/firewall software turning itself off every day in the middle of the night, and my having to reboot to get it working every morning. So I did a final search and began looking for a replacement program.

Then I found my exact issue on the BidDefender forum.

Here is the “solution” to the problem offered by a support person named “Diana Bele”

The issue reported in this thread is due to a recent product update that was released last week - the resolution is to right click on the BitDefender icon, select “Update now”, retrieve all the latest updates and restart the system. Then everything should be back on.

After trying this (didn’t work) for me or anyone else on the same forum. She never returned to the forum page.

So, I asked others on the same thread to do a little test with me. I posted the following proposal:

So, I’m going to alter my starting conditions - I am turning OFF all scanning tonight. I will allow the updates to occur automatically every hour all night. Since my issues start with a scan, I’ll report back tomorrow if I still have the problem.

So, I ask YOU to report back as well, but DON’T change anything with your configurations. That way we know that if I am successful and you are not, that the problem is related directly to scanning and not because they happened to fix the problem at the time we are running this test.

This morning, two people reported that they still had the same registration failure on 12/31/1969 problem. But I DID NOT!

I had turned OFF all scanning, and updates occurred normally throughout the night. So clearly scans are involved.

Because I am That Way, I prepared the following table to compare the last 5 days of behavior to find similarities, etc.

Lots more evidence

I found the following: Issues I see

* Scans either finished or aborted before the failure occurs (BAD)
* Updates failed AFTER finished or aborted scans say that the scan is still in progress (REALLY BAD)
* Many similarities in sequence of events

Similarities in event progression for all FAILED cases:

* Scan are always first, starting the failure sequence, followed by
* Many update errors
* Real time protection is disabled
* Several iterations of files downloaded and mostly successful updates
* Registration expires
* Alternating update successes/failures
* Firewall enabled
* Lots of update successes/failures
* Registration new license key (via reboot the next morning)

Grrr.

Ahhh, BitDefender Customer Support

Please recall that creating an account with BitDefender provides the following …

Please note the 3rd and 4th items…

  • “GAIN IMPROVED RESPONSIVENESS FROM THE CUSTOMER CARE TEAM.”
  • “ATTAIN HIGHER PRIORITY ON CUSTOMER CARE TASKS LISTS

5 tickets submitted, all with increasing levels of debugging information from me. One live chat aborted by BitDefender. 25 minutes on the phone, on my nickel, aborted. Only ONE acknowledgment that a problem exists on the FORUM with no followup. (Which has nothing to do with “My Account.”)

THIS IS IMPROVED RESPONSIVENESS?????

Final Thoughts on Testing

I wasn’t convinced until the very end that this didn’t involve a virus. Sadly, it was simply a case of some software developer somewhere not bothering to thoroughly test his/her work. Or, maybe that passed unit testing but certainly system test would have failed. I have special ire reserved for people who don’t bother to test. Especially when “fixes” are rolled out to thousands of people.

Now, if he/she is reading this post, I am sure the response will be, “But I DID test my code.”

Yeah?

  • Did you test for boundary conditions?
  • Did you test for permutations like scheduled scans?
  • Scans that may have overlapping start times?
  • Did you look at the history log file to see if your fixes actually worked?
  • If you DID and were able to reproduce this problem, did you even see the 12/31/1969 timestamp?
  • Did you look at error conditions … every one of these failures involved problems with updates failing.
  • Problems with updates occurring during scans that were apparently were not happening? (Or perhaps they were and the history log file has screwed up log info?)
  • Do you have a Test Plan? Do you USE it?

These are not “out there in left field” tests.

It seems fairly clear that a New Release or Update should have a basic test plan that includes permutations of how users actually USE their product. We aren’t doing anything strange. We just want automatic updates and to do scans when we’re away from the computer.

Shame on BitDefender for this debacle and how they are handling it. (It is not yet solved, or even really acknowledged.) Who knows how many people this affected and how many hours of time all of us wasted trying to figure out the source of this problem. Virus? Dead battery? Screwed up company software?

Yup, a completely-preventable “product update” that BitDefender didn’t bother to test.

They coded, compiled, submitted, and went home for the weekend.

AND AS I WRITE THIS, THE PROBLEM STILL EXISTS AND BITDEFENDER HAS IGNORED IT. INCLUDING ALL OF THE EXPLICIT DEBUGGING INFORMATION I HAVE PROVIDED.

<Expletive deleted>

[April 17 update] Part II of the Saga: Giving up on BitDefender - Lisa, Your Claims are ABSURD

≈ ♦ ≈

See the comprehensive list of All the “Let’s Debug It” Mysteries

Let’s Debug It: A Rabid War between Technology and Nature (Jet Skis vs Mangroves)

Let’s Debug It: My Internet Security Software HATES me, TOLERATES me, REVILES me. Evil BitDefender

Let’s Debug It: Alaskan Scuba Stuff and iPods Don’t Exercise

Let’s Debug It: A Cat’s Plea to Microsoft - “Can You Hear Me Now?”

Let’s Debug It: “No Ma’am (Idiot), You’re Calling from Line 2″

Let’s Debug It: Keyboards and Endothermic Mammals

Let’s Debug It: Hacksaws are Your Friend

Let’s Debug It: My Computer Plays “I’m Ignoring You Now”