Sequentum often receives requests for proposal (RFPs) for large-scale data collection with a narrow content focus, for example, for news, people data, or document downloads. The RFP issuer is often concerned that writing automation for each particular website will be cumbersome, expensive, and hard to maintain. They hope that we have some "auto-magic" way of solving their ambitious data collection requirements. In fact, there are many companies that will promise to solve this problem using Artificial Intelligence or Machine Learning but the solution inevitably falls short of handling all the edge cases. We often hear from teams who tried the AI/ML route only to suffer loss of reputation, budget overruns, and low team morale. This is because the AI/ML solution never truly "understands" the content that needs to be collected and while the team may successfully automate 60% of the content collection tasks, the ongoing investment attempting to identify and course-correct quality problems in the underlying model is overwhelming and unmanageable.
At Sequentum we have designed and perfected our platform for operating Intelligent Agents dating back to our first release in 2010. Our guiding design focus was to make it easy to author and maintain agents on a very large scale. We do this by providing a point and click interface that automates 95% of what the engineer would otherwise need to hand code. We provide modular building blocks that are re-usable across agents, projects, and software versions for common tasks like iterating through a list, running data collection across sessions in parallel, tracking changes between runs or de-duplicating data collected. We make it easy to vary input data sets, output formats and delivery endpoints. We include a quality monitoring component that is quick to configure and that alerts the team in real-time when a website implements a change that is not already detected and handled in the agent workflow. We even automate the documentation of the agents so your business analysts, data engineers and compliance team have transparency into how the agent is built and scheduled. We include a ticketing system that streamlines communication between stakeholders on any given agent. And we have frequent releases to support new protocols that emerge on our constantly evolving web. Automation covers the IT operation to manage request rate limits, agent version control, deployments, clustering, load balancing, queuing of jobs, performance management and more. Rather than rely on notoriously faulty AI models, we provide the data operations team with the tools necessary to stay on top of tens of thousands of agents with a small nimble human operation that consistently delivers high quality data feeds in a reliably and transparent way.
In these linked demos, we show a handful of examples of how easy and quick it is to author (or maintain) site-specific agents for your news, people, and document sources. We start by setting a custom rundate in a customer specific format using python, then go ahead and collect the data and output to CSV and XML formats.
Please reach out with questions or if you want an in-person demo.