THE CEO INTERVIEW www.ceotodaymagazine.com 20 Let’s start from the beginning. How has web scraping evolved to its current state? Web scraping, in some sense of the word, began with the first attempts to index pages scattered across the internet. While it wasn’t automated public data collection in the way we understand now, crawling across the web does much of the same as scraping. So, I would say that web scraping began almost with the advent of the World Wide Web, likely sometime in the early 90s, depending on who you ask. It may look surprising to some as web scraping still seems like something completely novel, but the technology and its underlying core principles have been with us for an incredibly long time. Naturally, many people would then ask about the changes that must have occurred for web scraping to go from something few know about to a nearly household name. There are a few important steps in evolution, I think, that both, directly and indirectly, impacted scraping. First, the digitalisation of businesses has probably had the most significant impact. Businesses thrive on competition, and competition is all about knowing what to do next for the best results. It would follow that any information on industry peers would be of great benefit. As such, there has always been an implicit drive for data acquisition technologies to develop. Second, as these technologies developed, they changed from being something accessible to the select few to a much wider audience. Looking through the same competition lens, demand for public data acquisition spurred companies to improve these technologies and lower operating costs. In fact, we were surprised by how deeply web scraping has entrenched itself in some industries. We recently conducted a survey, inquiring about data management practices in the UK and US finance sectors, which revealed that web scraping is about as popular as internal data acquisition practices. Finally, the prominence and importance of data in all fields, even outside business, has greatly increased. As a side effect, any collection method that purports to ease a usually complicated and drawn out process gets attention. So, could you expand upon how the perception of web scraping has changed over the years? Of course, as I’ve mentioned, web scraping had long been in the shadows as doing anything similar had been extremely resourceintensive and complicated. In fact, many people, even today, don’t know that search engines are founded upon web scraping. They couldn’t exist otherwise. As it moved more towards mainstream consciousness, there had been a certain air of suspicion as with all new technologies. Unfortunately, it still hasn’t passed, partly because there is no industry- It may look surprising to some as web scraping still seems like something completely novel, but the technology and its underlying core principles have been with us for an incredibly long time. “ “
RkJQdWJsaXNoZXIy Mjk3Mzkz