Web Scraping for Data Science — Is it legal?
Keep this in mind when scraping websites for your data science projects.
As a data scientist, you won’t always have a clean dataset ready to use for new projects. In most cases, the first step in your data science project would be extracting data from websites with scraping tools that imitate human surfing behavior on the internet.
However, these web scraping tools might be seen as a threat to most websites. In fact, companies like Facebook seemed to be against web scraping, even for research purposes.
To start off your data science project on the right foot, we’ll analyze in which scenarios scraping data would be considered either legal or illegal.
Web Scraping for research projects
Although projects with research purposes like those involving sentiment analysis, fake news detections, or model predictions might seem harmless, it might raise alarms of some companies if you obtained that data from their website without permission.
Let’s consider what happened between Facebook and New York University as an example.
On October 16th, Facebook wrote a letter demanding a New York University (NYU) research project stop collecting data without permission. The NYU project consisted of tracking how political ads were targeted by collecting ad targeting data from users. Facebook wrote:
“Scraping tools, no matter how well-intentioned, are not a permissible means of collecting information from us. We understand the intent behind your tool. However, the browser plugin scrapes information in violation of our terms, which are designed to protect people’s privacy.”
Although companies like Facebook don’t allow web scraping, scraping isn’t considered a big issue when it comes to research. In fact, web scraping is legal for UK researchers, after introducing, in 2014, clear legislation to define exceptions to copyright for non-profit research.
That being said, Facebook’s response to the New York University project might be only an inevitable overreaction after the Cambridge Analytica scandal. In fact, Facebook seemed to back down from its threat to shut down the project after receiving public criticism.
Web Scraping for commercial purposes
When you scrape data from websites, you’re accessing data that might be protected by copyright. If you scrape websites and use the data obtained for commercial purposes, you could get into trouble.
Let’s consider the HiQ and LinkedIn case as an example.
HiQ is a data analytics firm that provides business intelligence based on publicly-available data scraped from LinkedIn. For this reason, LinkedIn invoked the Computer Fraud and Abuse Act (CFAA) in a cease-and-desist letter to HiQ.
Although the US Court of Appeals denied LinkedIn’s request saying that web scraping public sites does not violate the CFAA, it should be noted that the decision doesn’t grant HiQ the freedom to use data obtained by scraping for commercial purposes.
Some of the techniques HiQ used for scraping LinkedIn and avoiding IP ban are explained in the following article.
So projects with commercial purposes, like selling the data scraped, are not safe. Instead, you could make your project profitable without selling data with the following option.
Web Scraping for trading projects
Trading involves buying and selling things. The goal is to profit off of any buying and selling activity.
Web scraping for trading usually works in the following way.
Some people build a scraper that returns the price of a specific product, so when the price drops, the program automatically buys the product before it’s sold out. Once the demand for the product grows and the price rises, they resell the product to make a profit.
However, one of the best types of trading is arbitrage because you only take advantage of the price difference between two or more markets, so you buy in one market and simultaneously sell in another market for a higher price.
Arbitrage occurs in exchange rates, sports betting, cryptocurrencies and more. On top of that, arbitrage trading is legal in the United States and is encouraged, as it contributes to market efficiency.
If you like the idea of making extra money with arbitrage trading, check the article below:
Conclusion
In this article, we analyzed the legality of web scraping. We found that research projects shouldn’t have legal issues, especially in countries like the UK, which clearly stated that web scraping is legal for researchers. Projects with commercial purposes, however, might infringe copyright. That being said, it’s still possible to profit off of the data obtained with arbitrage trading.
Above all, it seems that “what you’re gonna do with the data” would determine the legality of scraping a website.