Midland/Permian Basin

Texas(432) 235-1776

Farmington/San Juan Basin

NM(505) 327-7300

Comcast down again

Summary

Yesterday (May 14th, 2014) Comcast in San Juan County sufferred 2 hard outages. The first was from about 3:30pm-4:15pm. The second was considerably longer lasting from about 5:40pm-9:10pm. Yesterday's total downtime was approximately 4 hours 15 minutes.

 

This is following an outage back on March 17th that lasted from about 6:04am-9:10am (3 hrs, 6 mins).

 

Cause

Regarding the cause of yesterday's outage, this morning I was told by Mark Johnson (a local supervisor) that a piece of equipment in their Farmington infrastructure had failed.

 

Business Impact

Yesterday's outage affected many Impala Networks clients adversly. The hardest hit were our clients using business-critical cloud-based services. This included internet-based VoIP phone services, internet-based credit card terminals, and remote/VPN-based business software.

 

We had customers calling us seeking an ETA, as well as any solution to help them regain some limited functionality. Some of these we were able to help and others simply had no alternative but to wait out the outage. Unfortunately some of these included restaurants and retail sales establishments where their critical business hours extend past 7pm each day. So the 2nd outage actually hurt them just as badly if not worse than the first.

 

Overview of local ISPs

Our customers who use Cedar Networks for their Internet service had no disruption yesterday. We also use Cedar Networks and were able to continue working productively while so many of our customers could not.

 

Comcast has truly helped push Farmington's internet connectivity forward over the past few years, primarily by bringing much faster speeds at a very competitive price. This has prompted competitors like CenturyLink and CedarNetworks to step their game and increase their speeds at a competitive price as well.

 

Unfortunately Comcast's reliability in San Juan County has not been stellar. The outages from March and yesterday will now leave Comcast with no better than 99.916% uptime in San Juan County for the year of 2014. This may seem high. But mid-day outages are not easily forgotten by customers. Cisco Systems and other industry leaders set the bar for network availability to 99.999% (a.k.a. "five 9s"). This is equivelant to 5 minutes downtime per year.

 

Comparing Comcast's outages to Cedar Networks, just this year Comcast's downtime has been near equivelant of Cedar Networks aggregate downtime in San Juan County over the past 7 years. This is referring to the general service outages affecting the entire region, as opposed to individual line issues that may affect only 1 customer.

Email Greylisting

What is greylisting?

Greylisting is a common anti-spam technique. It is an internet standard defined in RFC 6647 (http://tools.ietf.org/html/rfc6647). It complies and fits in nicely with the standard SMTP RFC 5321. Particularly the Retry Strategies section of the SMTP RFC (http://tools.ietf.org/html/rfc5321#section-4.5.4).

 

How does it work?

Greylisting is done by the receiving server. It simply issues a temporary deferral to the sending server. This means that the receiving server tells the sending server "I'm sorry. But I can't accept that message from you right now. But I may be able to later. So please retry delivery later. Don't bounce the message because I am not rejecting it. I'm just deferring it".

 

This is the same response (deferral) that a receiving server sends when:

  1. A user's mailbox is full & cannot receive the message at this time.
  2. The receiving server is overloaded & can't handle processing any more messages at the moment.
  3. The receiving server is having some other problem, etc. and is not able to process the message presently but the problem is of a nature that it is likely that the message should be able to be received & processed later. Therefore rather than bouncing (permanently rejecting) the message, it simply defers it.

 

Greylisting only applies to strangers:

Greylisting is typically only done when an email is coming from an IP address & sender address (or domain) that we have never seen before (or at least not in the recent past). Odds are that 95-99% of the emails that you receive are coming from people that you have previously received email from. Greylisting engines maintain a database of IP addresses associated with domain names/senders & recipients. The greylisting engine refers to this database each time an inbound email is received. If the combination of IP/IP block and sender/sender domain has never before been seen, then it is not considered a "known sender". In this case, the message is deferred. Once that the sending server retries the message, the greylisting engine allows it through. So greylisting is usually only a 1-time deferral asking for the sender to retry again shortly.

 

Retry:

The sending server should be able to handle the deferral without any problem. Internet SMTP standard (referenced above) states that emails servers must retry delivery of deferred messages, generally for up to 4-5 days. Additionally, the first retry should happen less than an hour from the deferral. So in most cases, a person should end up receiving the email less than 1 hour after it was sent. As long as the sender retries the message from the same source IP address, the greylisting server will let it through.

 

Future messages are not delayed/deferred:

Greylisting engines do not delay future emails, so long as they are coming from a previously seen combination of IP address/block and sender address/domain. Only the initial such email is delayed. Then you're home free.

 

Why does it work?

A few reasons:

  1. Some spammers are too lazy to retry sending the spam. Or their email engine just doesn't because they really don't care about following SMTP rules in the first place. They treat a deferral as a failure & just move on.
  2. It gives spammers more time to get on a blacklist. If they are sending a mass spam from a newly infected PC, it is likely that the PC's IP won't immediately be on any email blacklists. But give it 5 minutes, and it's likely that it will be. Hence by delaying the message reception by a few minutes, it is much more likely that the receiving server will be able to correctly identify the message as spam because the sender IP addres is now on blacklists.

 

Does it work with IP addresses or blocks? And is it using sender addresses, or domains? Which is it?

Both. Ultimately it depends on the greylisting implementation. However, ours takes both the IP block into account as well as the IP address. It also takes both the sender's email address & the sender's email domain into account. If multiple emails from different addresses (but in the same domain) have been received within a certain period of time (60 days as of this writing), then we whitelist the sender's entire email domain for the associated IP address.

Also, we whitelist an entire class C IP block rather than just the single IP, so long as the sender IP has a reverse DNS record & doesn't appear to be a home internet DNS record.

 

What problems does greylisting cause?

  1. Temporary delays which can be annoying.
  2. Bounced or missing emails due to:
    1. Some email servers don't follow the rules (referenced at the top of this blog). These emails may be completely bounced immediately, or never bounced or delivered (the treacherous black hole). Note that the receiving/greylisting server does NOT bounce the message. It defers it. However, if the sending server doesn't adhere to internet standards, or it is misconfigured, then one of the above could occur. Note: If your email provider doesn't sent you bounce notifications for any email that it determines will not delivered, you should probably consider changing to a new email provider. The black hole should never happen and is a sign of an email service you probably don't want to use.
    2. Some larger email providers may have a pool of outbound servers. And some may actually retry transmission from a different IP address every time. In such instances, the email may never make it through the greylisting engine. The solution here is to notify your email provider and they can probably make an adjustment for the greylisting engine to completely whitelist the IP block (not just a single IP) the message is coming from. We can do this with our system.
    3. A novice web designer decides to use a PHP or similar email function that just sends email (like from a feedback form) directly to the destination servers, rather than to its own internal email server. The result is that such a PHP email function is NOT an email server and has no concept or mechanism to queue a message for retry. It fails to be delivered and will never be tried again. Instead, the developer should have configured his web script to send/relay all such emails through his own email server so that deferrals can be handled correctly.

 

What to tell senders who don't retry transmission:

 

They should fix their email server configuration (or replace their server) so that it is in compliance with internet email standards.

Not responding correctly to Greylisting is in violiation of:
RFC 6647 (Greylisting):
http://tools.ietf.org/html/rfc6647
and
RFC 5321 (SMTP [sec. 4.5.4 - Retry Strategies]):
http://tools.ietf.org/html/rfc5321#section-4.5.4
The specific violations via the SMTP RFC are:
"mail that cannot be transmitted immediately MUST be queued and periodically retried by the sender."

"The sender MUST delay retrying a particular destination after one attempt has failed. In general, the retry interval SHOULD be at least 30 minutes; however, more sophisticated and variable strategies will be beneficial when the SMTP client can determine the reason for non-delivery. Retries continue until the message is transmitted or the sender gives up; the give-up time generally needs to be at least 4-5 days."

 

Is greylisting worth the hassle?

Yes. When we consider the security risks associated with viruses/malware, as well as phishing scams, etc., you really have to take every reasonable measure to protect yourself. Greylisting can be a bit inconvenient at times, but it works quite well and is a part of a good anti-spam system. We recommend that everyone use an anti-spam system supporting greylisting. Turning it back off just allows too much spam through.

Not all online backups are created equal

 

Although 4 years have passed since this incident, I think it is still worth mentioning. Carbonite who is a popular home & small business online backup vendor lost the backups of over 7,500 customers. This resulted from hardware failure across different parts of their online backup data storage cloud.

 

I.T. professionals will recognize why this happened. Carbonite was using "Promise Technologies" as the manufacturer of their storage (RAID) hardware. Promise is the cheapest of the cheap. So much so that they have not been considered a real storage controller vendor in the business world for many years (this is my opinion which I share with other contemporaries). Although their products do exist and are and have been used over the years by very, very cheap motherboards with integrated RAID, etc., Promise is absent from nearly all circles where enterprise storage is discussed.

Yet here was Carbonite, with very possibly tens or hundreds of thousands of customers’ backups being trusted to a company who many in the I.T. industry would just regard as junk (when compared to real storage controllers made by LSI & the likes).

 

Bottom line is that if something seems too good to be true, it probably is. Be wary of online backup vendors with rock bottom prices. In addition to this likely translating to a lower level of customer service, it may also mean that they make compromises like this.

 

Impala Networks uses multiple online backup & disaster recovery solutions from multiple reputable vendors. Our solutions are price competitive with many in the industry. But more importantly, they are using companies who are known to do things right & not put their reputation (and customers' data) on the line by compromising the quality of their products.

 

 

http://www.boston.com/business/technology/articles/2009/03/21/data_backup_firm_sues_2_hardware_suppliers/?page=full

 

Please call us today to discuss strategic methods to backup your data & provide disaster recovery to keep your business functional in the wake of data or computer loss.

New Intel CPU and its effect on mobile computing.

 

Providing IT services in Farmington, New Mexico is a bit of a challenge sometimes, due to the rugged terrain and hot climate. We have clients that do all sorts of interesting activities with their computers and they always come to ask us for advice on how to make their systems more efficient and user friendly. For example in San Juan County we have several oil companies and insurance companies that constantly send people out into the field to survey and report the status of issues. Many of these people are accustomed to taking their clunky old laptops out into the field with them and having the battery die, or the mobile internet car fail or just being so embarrassed to even bring that old clunky thing out. A few companies have started to issue their field employees IPads or equip them with apps so that they can do their reporting from a mobile device while in the field. While Ipads are great, no one can best the flexibility and compatibility of the windows operating system. Many of the popular survey and reporting software systems only run on windows or mac desktop environments and are only barely getting wise to the mobile shift that is happening in the world of technology. Enter “Haswell” the newest of Intel’s multi core processors. This CPU is designed for a huge amount of battery life increase and also for more power. Now companies like Dell and Acer are plugging these Haswells” into tablets and ultralight laptops to create the ultimate mobile computing experience. Now our clients can get the power of a desktop on a tablet, while still having enough battery life to forget the charger when their out in the field. We insist on providing the top in IT services for our clients in Farmington , New Mexico and all of San Juan county and now we can help provide a better and more enjoyable experience by recommending these new CPU’s. Using them you will be able to quickly search through hundreds of emails in outlook on a tablet without the lag. Faster Processing means that the reporting programs will function much better so the employee isn’t wasting precious time watching progress bars load. These new CPU’s seem to be what the mobile world is needing right now and we just hope they can deliver all that they promise.

Here is an article all about the new haswell processors on the apcmag.com website.

 

http://apcmag.com/intel-haswell-cpu-how-it-works.htm

 

Impala Networks now selling Comcast

By Douglas Mortensen

As Comcast is showing forth that they are a strong player in the small business phone & internet market in Farmington & Aztec, NM, we have decided to partner with them in order to help interested customers get the most out of Comcast's offerrings, as well as to help facilitate the sign up process, and ensure that the customer selects the package that meets their needs most appropriately.

Here's a quick overview of Comcast Business Class:

Read more: Impala Networks now selling Comcast

New Spyware Found in the Wild - "Rogue Utility Products"

By Douglas Mortensen

There is a new piece of spyware in the wild that Symantec Endpoint Protection is not catching.  I have dubbed it “Rogue Utility Product”.

The Current rendition is “Hard Drive Diagnostics”. 

What does it do?

  • It displays a window that looks a lot like the fake security malware that essentially take your computer hostage.  Instead, this new spyware pops up a window stating that your hard drive “may be fragmented” or something very similar.

Read more: New Spyware Found in the Wild - "Rogue Utility Products"

Impala Networks getting a head start with SBS 2011

By Douglas Mortensen

Just a quick update. Just over two weeks back on Thursday April 7th, Doug Mortensen (me) and Jason Heuser (Impala Networks Service Manager) attended a Microsoft Partner Technical Seminar on the latest & greatest Microsoft Server product for small & medium sized businesses. Namely Small Business Server 2011 Standard and Small Business Server 2011 Essentials.

We just wanted to let our customers & community know that we are taking much effort to keep on top of the Microsoft server technologies that so many businesses will inevitably be using, as old servers are upgraded, phased out, etc.

Read more: Impala Networks getting a head start with SBS 2011

Which smartphone should you choose?

By Douglas Mortensen

This is a question that we are frequently asked. I just typed up a response to a customer and figured I'd put it in our blog, and maybe we can refer to it in the future, and possibly help others who are just searching online & happen to stumble upon our blog. Here's the email, which started off as a request for a recommendation between the iPhone and Android:

Hmm. It’s a tough call. Both have their strengths & (minor) weaknesses. But both are worlds ahead of the smart phones from 5 years ago.

Read more: Which smartphone should you choose?

Android Market Security Farce - Trusted CAs not required

By Douglas Mortensen

Today I came across some very disturbing information regarding what appears to be a major security compromise on the side of the Android market app submission requirements. It was so disturbing to me that I sent the following email to my technical staff:

Subject: Signing Your Applications | Android Developers

Androids don't care if apps use a self-signed / self-issued digital certificates.

Kind of pathetic in my opinion. What's the point of requiring apps to be digitally signed, if you just accept all digital signatures including "no name" self-signed/self-issued certs? Kinds of defeats the whole purpose of digitally signing apps in the first place.

Read more: Android Market Security Farce - Trusted CAs not required