Improving Agent Performance and Reliability

It is important to understand that the complexity of web scraping increases significantly when extracting larger amounts of data. The time it takes to perform certain operations may take so long that the whole web scraping process becomes impractical. Web scraping is unreliable in nature, and the longer a web scraping project runs, the higher the chance something fails and interrupts the web scraping process.

The following topics list a few steps you can take to increase performance and reliability. Performance tuning is not for the novice user, and if you have no IT experience, you may not be able to implement the recommendations in this topic.

• Using MS SQL or MySQL as Internal Database

• Using the HTML Parser

• Using the JSON Parser

• Using the XML Parser

• Optimizing Agent Commands

• Performance Sessions

• Multi-threading