Imagine if you were a traditional shop keeper and your high street store was visited by over 1,000,000 shoplifters every day; do you think you’d do something about it? Or if over 60% of visitors to your premises were actually sent by your competitors, to deliberately block the way to the shelves and tills so your customers couldn’t actually see or buy anything? Would you do anything?
How about if your competitors got your pricing models and made sure they undercut you on every single item? Would you do anything about that? Or finally, how about if your competitors escorted customers around the store and made purchases for them, adding a little bit of profit for themselves? Would you think it sensible not to do anything about it? And what do you think is worse? Choosing to turn a blind eye to the activities, or not even noticing that all that was going on in and around your store in the first place?
What if I told you, that all that was commonplace in commerce? Are you very glad you are not responsible for the management or marketing of a physical store?
How about if I told you that this activity was actually happening online and that it is, more than likely, happening to your web site?
Ignorance isn’t really blissful
Don’t believe me?
You think you would know about that sort of loss, surely?
Well, it is Sentor’s experience that a lot of companies simply do not know that they are haemorrhaging their Intellectual Property to competitors and/or cyber criminals, until we show them. I quoted real life examples from some sites we were brought on board to investigate and protect: One client did not know its site was being scraped 1,000,000 times a day; or that 60% of the traffic on one site was scrapers and that legitimate customers were abandoning transactions because the scrapers were slowing it all down. Another client hired us because they knew third parties were scraping its site and making purchases on behalf of consumers, which meant our client was losing sell-on opportunities as well as no longer being in control of the brand relationship.
The wholesale theft of web site content (known as web or screen scraping, data harvesting, or web data extraction) is a growing form of cybercrime and it is staggering how many businesses are oblivious to the risk. We know, as probably the first security consultancy to spot the behaviour, and to coin the term web scraping back in 1996, that nobody then had the audacity to actually advertise their web scraping services, but now if you type the term into Google, the majority of the 3 million+ hits returned are companies offering to scrape sites for a fee. And I have no doubt that this is a problem that is only going to grow, so choosing to do nothing about it is not a good idea.
Who are the Bad Guys?
Looking at what data is being stolen, you can take an educated guess as to who are the bad guys. Some competitors scrape sites for price sensitive information and adjust their own prices accordingly, to ensure they have the competitive edge. Other scrapers are wholesale thieves who either use your content themselves, possibly even posing as you to customers, or sell it on to others to use however they want.
Protect yourself and your brand
So what can you do about it?
Depending on what your web site is used for, there are some simple steps you can take.
You could implement a captcha request for visitors, before they complete transactions. But that is a bit like asking a customer to whisper the magic word before you’ll ring up the item in the register. It is a bit annoying for all concerned, could cause people to abandon their purchases, and it really is not necessary in most cases.
Or you could put up in the terms of use on your web site that you don’t want to be scraped. But WebBots don’t tend to read the Ts&Cs.
Or you could interrogate every visitor to your web site, and check their IP address against a list of known offenders – assuming you had that information available - and then decide if you want them on your site.
You could increase your IT budgets and buy more servers to allow the scrapers to carry on their DDOS-like activities regardless, without impacting the on-site experience of your legitimate customers.
Sometimes, you might chose to allow scrapers onto your site and possibly, direct them to some irrelevant or misleading information. All’s fair in love and war, they say.
It might be that you could cut a deal with the people scraping your site and actually turn it into a revenue earning situation for you both. If you know who to talk to, that is.
Think of it as an opportunity
There are numerous solutions available to you and often there is not a one- size- fits-all answer to the problem of being scraped. There are subtleties and nuances in reasons for scraping, perhaps confusingly, and your decision on how to handle them may change depending on the, who, what, when, why and how you are being scraped. The important thing for your brand is that you remain in control of your own information and the online experience of your customers.
So like a lot of things in business, the best solution is probably to talk to the experts. The kinds of companies that deal with blocking scrapers every day. Find someone you trust to protect your business and its property.
And if you don’t know what existing security measures your company has in place to protect your Intellectual Property and revenues from web scraping, then maybe now is the time to ask.
By Martin Zetterlund, Founding Partner of Sentor, www.sentor.se,
PrivSec Conferences will bring together leading speakers and experts from privacy and security to deliver compelling content via solo presentations, panel discussions, debates, roundtables and workshops.
For more information on upcoming events, visit the website.
comments powered by Disqus