You are here: Home Blog
Yesterday (May 14th, 2014) Comcast in San Juan County sufferred 2 hard outages. The first was from about 3:30pm-4:15pm. The second was considerably longer lasting from about 5:40pm-9:10pm. Yesterday's total downtime was approximately 4 hours 15 minutes.
This is following an outage back on March 17th that lasted from about 6:04am-9:10am (3 hrs, 6 mins).
Regarding the cause of yesterday's outage, this morning I was told by Mark Johnson (a local supervisor) that a piece of equipment in their Farmington infrastructure had failed.
Yesterday's outage affected many Impala Networks clients adversly. The hardest hit were our clients using business-critical cloud-based services. This included internet-based VoIP phone services, internet-based credit card terminals, and remote/VPN-based business software.
We had customers calling us seeking an ETA, as well as any solution to help them regain some limited functionality. Some of these we were able to help and others simply had no alternative but to wait out the outage. Unfortunately some of these included restaurants and retail sales establishments where their critical business hours extend past 7pm each day. So the 2nd outage actually hurt them just as badly if not worse than the first.
Our customers who use Cedar Networks for their Internet service had no disruption yesterday. We also use Cedar Networks and were able to continue working productively while so many of our customers could not.
Comcast has truly helped push Farmington's internet connectivity forward over the past few years, primarily by bringing much faster speeds at a very competitive price. This has prompted competitors like CenturyLink and CedarNetworks to step their game and increase their speeds at a competitive price as well.
Unfortunately Comcast's reliability in San Juan County has not been stellar. The outages from March and yesterday will now leave Comcast with no better than 99.916% uptime in San Juan County for the year of 2014. This may seem high. But mid-day outages are not easily forgotten by customers. Cisco Systems and other industry leaders set the bar for network availability to 99.999% (a.k.a. "five 9s"). This is equivelant to 5 minutes downtime per year.
Comparing Comcast's outages to Cedar Networks, just this year Comcast's downtime has been near equivelant of Cedar Networks aggregate downtime in San Juan County over the past 7 years. This is referring to the general service outages affecting the entire region, as opposed to individual line issues that may affect only 1 customer.
Greylisting is a common anti-spam technique. It is an internet standard defined in RFC 6647 (http://tools.ietf.org/html/rfc6647). It complies and fits in nicely with the standard SMTP RFC 5321. Particularly the Retry Strategies section of the SMTP RFC (http://tools.ietf.org/html/rfc5321#section-4.5.4).
Greylisting is done by the receiving server. It simply issues a temporary deferral to the sending server. This means that the receiving server tells the sending server "I'm sorry. But I can't accept that message from you right now. But I may be able to later. So please retry delivery later. Don't bounce the message because I am not rejecting it. I'm just deferring it".
Greylisting is typically only done when an email is coming from an IP address & sender address (or domain) that we have never seen before (or at least not in the recent past). Odds are that 95-99% of the emails that you receive are coming from people that you have previously received email from. Greylisting engines maintain a database of IP addresses associated with domain names/senders & recipients. The greylisting engine refers to this database each time an inbound email is received. If the combination of IP/IP block and sender/sender domain has never before been seen, then it is not considered a "known sender". In this case, the message is deferred. Once that the sending server retries the message, the greylisting engine allows it through. So greylisting is usually only a 1-time deferral asking for the sender to retry again shortly.
The sending server should be able to handle the deferral without any problem. Internet SMTP standard (referenced above) states that emails servers must retry delivery of deferred messages, generally for up to 4-5 days. Additionally, the first retry should happen less than an hour from the deferral. So in most cases, a person should end up receiving the email less than 1 hour after it was sent. As long as the sender retries the message from the same source IP address, the greylisting server will let it through.
Greylisting engines do not delay future emails, so long as they are coming from a previously seen combination of IP address/block and sender address/domain. Only the initial such email is delayed. Then you're home free.
A few reasons:
Both. Ultimately it depends on the greylisting implementation. However, ours takes both the IP block into account as well as the IP address. It also takes both the sender's email address & the sender's email domain into account. If multiple emails from different addresses (but in the same domain) have been received within a certain period of time (60 days as of this writing), then we whitelist the sender's entire email domain for the associated IP address.
Also, we whitelist an entire class C IP block rather than just the single IP, so long as the sender IP has a reverse DNS record & doesn't appear to be a home internet DNS record.
They should fix their email server configuration (or replace their server) so that it is in compliance with internet email standards.
Not responding correctly to Greylisting is in violiation of:
RFC 6647 (Greylisting):
RFC 5321 (SMTP [sec. 4.5.4 - Retry Strategies]):
The specific violations via the SMTP RFC are:
"mail that cannot be transmitted immediately MUST be queued and periodically retried by the sender."
"The sender MUST delay retrying a particular destination after one attempt has failed. In general, the retry interval SHOULD be at least 30 minutes; however, more sophisticated and variable strategies will be beneficial when the SMTP client can determine the reason for non-delivery. Retries continue until the message is transmitted or the sender gives up; the give-up time generally needs to be at least 4-5 days."
Yes. When we consider the security risks associated with viruses/malware, as well as phishing scams, etc., you really have to take every reasonable measure to protect yourself. Greylisting can be a bit inconvenient at times, but it works quite well and is a part of a good anti-spam system. We recommend that everyone use an anti-spam system supporting greylisting. Turning it back off just allows too much spam through.
Although 4 years have passed since this incident, I think it is still worth mentioning. Carbonite who is a popular home & small business online backup vendor lost the backups of over 7,500 customers. This resulted from hardware failure across different parts of their online backup data storage cloud.
I.T. professionals will recognize why this happened. Carbonite was using "Promise Technologies" as the manufacturer of their storage (RAID) hardware. Promise is the cheapest of the cheap. So much so that they have not been considered a real storage controller vendor in the business world for many years (this is my opinion which I share with other contemporaries). Although their products do exist and are and have been used over the years by very, very cheap motherboards with integrated RAID, etc., Promise is absent from nearly all circles where enterprise storage is discussed.
Yet here was Carbonite, with very possibly tens or hundreds of thousands of customers’ backups being trusted to a company who many in the I.T. industry would just regard as junk (when compared to real storage controllers made by LSI & the likes).
Bottom line is that if something seems too good to be true, it probably is. Be wary of online backup vendors with rock bottom prices. In addition to this likely translating to a lower level of customer service, it may also mean that they make compromises like this.
Impala Networks uses multiple online backup & disaster recovery solutions from multiple reputable vendors. Our solutions are price competitive with many in the industry. But more importantly, they are using companies who are known to do things right & not put their reputation (and customers' data) on the line by compromising the quality of their products.
Please call us today to discuss strategic methods to backup your data & provide disaster recovery to keep your business functional in the wake of data or computer loss.
Providing IT services in Farmington, New Mexico is a bit of a challenge sometimes, due to the rugged terrain and hot climate. We have clients that do all sorts of interesting activities with their computers and they always come to ask us for advice on how to make their systems more efficient and user friendly. For example in San Juan County we have several oil companies and insurance companies that constantly send people out into the field to survey and report the status of issues. Many of these people are accustomed to taking their clunky old laptops out into the field with them and having the battery die, or the mobile internet car fail or just being so embarrassed to even bring that old clunky thing out. A few companies have started to issue their field employees IPads or equip them with apps so that they can do their reporting from a mobile device while in the field. While Ipads are great, no one can best the flexibility and compatibility of the windows operating system. Many of the popular survey and reporting software systems only run on windows or mac desktop environments and are only barely getting wise to the mobile shift that is happening in the world of technology. Enter “Haswell” the newest of Intel’s multi core processors. This CPU is designed for a huge amount of battery life increase and also for more power. Now companies like Dell and Acer are plugging these Haswells” into tablets and ultralight laptops to create the ultimate mobile computing experience. Now our clients can get the power of a desktop on a tablet, while still having enough battery life to forget the charger when their out in the field. We insist on providing the top in IT services for our clients in Farmington , New Mexico and all of San Juan county and now we can help provide a better and more enjoyable experience by recommending these new CPU’s. Using them you will be able to quickly search through hundreds of emails in outlook on a tablet without the lag. Faster Processing means that the reporting programs will function much better so the employee isn’t wasting precious time watching progress bars load. These new CPU’s seem to be what the mobile world is needing right now and we just hope they can deliver all that they promise.
Here is an article all about the new haswell processors on the apcmag.com website.
As Comcast is showing forth that they are a strong player in the small business phone & internet market in Farmington & Aztec, NM, we have decided to partner with them in order to help interested customers get the most out of Comcast's offerrings, as well as to help facilitate the sign up process, and ensure that the customer selects the package that meets their needs most appropriately.
Here's a quick overview of Comcast Business Class:
There is a new piece of spyware in the wild that Symantec Endpoint Protection is not catching. I have dubbed it “Rogue Utility Product”.
The Current rendition is “Hard Drive Diagnostics”.
What does it do?
Just a quick update. Just over two weeks back on Thursday April 7th, Doug Mortensen (me) and Jason Heuser (Impala Networks Service Manager) attended a Microsoft Partner Technical Seminar on the latest & greatest Microsoft Server product for small & medium sized businesses. Namely Small Business Server 2011 Standard and Small Business Server 2011 Essentials.
We just wanted to let our customers & community know that we are taking much effort to keep on top of the Microsoft server technologies that so many businesses will inevitably be using, as old servers are upgraded, phased out, etc.
This is a question that we are frequently asked. I just typed up a response to a customer and figured I'd put it in our blog, and maybe we can refer to it in the future, and possibly help others who are just searching online & happen to stumble upon our blog. Here's the email, which started off as a request for a recommendation between the iPhone and Android:
Hmm. It’s a tough call. Both have their strengths & (minor) weaknesses. But both are worlds ahead of the smart phones from 5 years ago.
Today I came across some very disturbing information regarding what appears to be a major security compromise on the side of the Android market app submission requirements. It was so disturbing to me that I sent the following email to my technical staff:
Subject: Signing Your Applications | Android Developers
Androids don't care if apps use a self-signed / self-issued digital certificates.
Kind of pathetic in my opinion. What's the point of requiring apps to be digitally signed, if you just accept all digital signatures including "no name" self-signed/self-issued certs? Kinds of defeats the whole purpose of digitally signing apps in the first place.