Manipulating the Proxy Pool at Runtime
Sequentum Enterprise supports various types of proxy sources. We refer to the collection of all these proxy sources configured in the agent as "Proxy Pool". A single type of proxy source is normally selected at design-time, and automatic proxy rotation is enabled by default, however, for complex proxy rotation requirements, the Scripting Interface in Sequentum Enterprise enables users to control the agent's proxy pool during run-time.
Overriding the Agent's Proxy Configuration
The Scripting Interface exposes the GlobalData.ProxyList object, which takes precedence over the Agent's proxy configuration. This means that if this object is populated with proxies at run-time, the Agent's design-time proxy configuration is ignored, and proxies in the GlobalData.ProxyList are used instead.
Ideally, this custom proxy list must be populated in the Agent Initialization Script, since this script is executed before the agent's "Start URL" gets loaded. It is still possible to manipulate this custom proxy list anywhere in the agent at run-time.
C#
Python
using System;
using Sequentum.ContentGrabber.Api;
public class Script
{
//See help for a definition of AgentInitializationArguments.
public static bool InitializeAgent(AgentInitializationArguments args)
{
// Add proxy servers
args.GlobalData.ProxyList.AddProxyServer("123.45.67.12:6000", "user", "pass");
args.GlobalData.ProxyList.AddProxyServer("123.45.67.34:6000", "user", "pass");
args.GlobalData.ProxyList.AddProxyServer("123.45.67.56:6000", "user", "pass");
args.GlobalData.ProxyList.AddProxyServer("123.45.67.78:6000", "user", "pass");
args.GlobalData.ProxyList.AddProxyServer("123.45.67.90:6000", "user", "pass");
args.GlobalData.ProxyList.IsRotateAutomatically = true;
args.GlobalData.ProxyList.IsVerifyProxies = false;
return true;
}
import clr
from Sequentum.ContentGrabber.Api import *
def InitializeAgent(args):
# Add proxy servers
args.GlobalData.ProxyList.AddProxyServer("123.45.67.12:6000", "user", "pass")
args.GlobalData.ProxyList.AddProxyServer("123.45.67.34:6000", "user", "pass")
args.GlobalData.ProxyList.AddProxyServer("123.45.67.56:6000", "user", "pass")
args.GlobalData.ProxyList.AddProxyServer("123.45.67.78:6000", "user", "pass")
args.GlobalData.ProxyList.AddProxyServer("123.45.67.90:6000", "user", "pass")
args.GlobalData.ProxyList.IsRotateAutomatically = True
args.GlobalData.ProxyList.IsVerifyProxies = False
return True
Sample code that shows how to populate the custom proxy list in the Agent Initialization Script
You will then have to call the args.RotateProxies method in an Execute script command to activate the proxy list. See the section below on how to Rotate Proxies via Scripting.
Another helper method is SetProxy(). This method sets a single proxy in global data and makes that the active proxy. All current proxies in global data will be removed.
C#
Python
using System;
using Sequentum.ContentGrabber.Api;
public class Script
{
//See help for a definition of CustomScriptArguments.
public static CustomScriptReturn CustomScript(CustomScriptArguments args)
{
args.SetProxy("123.45.67.12:6000", "user", "pass");
return CustomScriptReturn.Empty();
}
}
import clr
from Sequentum.ContentGrabber.Api import *
def CustomScript(args):
args.SetProxy("123.45.67.12:6000", "user", "pass")
return CustomScriptReturn.Empty()
SetProxy() helper method sample usage in Execute Script command
Changing Proxy Source via Scripting
An Agent can switch to another proxy source at run-time, but this can only be done within the Agent Initialization Script.
C#
Python
using System;
using Sequentum.ContentGrabber.Api;
public class Script
{
//See help for a definition of AgentInitializationArguments.
public static bool InitializeAgent(AgentInitializationArguments args)
{
//Set ACC Proxy Pool
args.Agent.Proxies.AddAccPool().ProxyAddress="http://acc-test.sequentum.com/sequentum/testproxy1";
//Enable the Proxy Rotation
args.Agent.Proxies.ProxyPools.Pools[0].IsRotateAutomatically=true;
//Set Proxy Rotation interval
args.Agent.Proxies.ProxyPools.Pools[0].RotationInterval=200;
return true;
}
}
import clr
from Sequentum.ContentGrabber.Api import *
def InitializeAgent(args):
#Set ACC Proxy Pool
args.Agent.Proxies.AddAccPool().ProxyAddress="http://acc-test.sequentum.com/sequentum/testproxy1";
#Enable the Proxy Rotation
args.Agent.Proxies.ProxyPools.Pools[0].IsRotateAutomatically=True
#Set Proxy Rotation interval
args.Agent.Proxies.ProxyPools.Pools[0].RotationInterval=200
return True
Sample code for changing proxy source in the Agent Initialization Script
Additionally, a proxy source's configuration can be accessed and modified from within the Agent Initialization Script.
NOTE:
To use the following code in the agent, use this additional library. By default the library is not added in Agent Initialization Script.
C# : using Sequentum.ContentGrabber.Commands;
Python: from Sequentum.ContentGrabber.Commands import *
C#
Python
using System;
using Sequentum.ContentGrabber.Api;
using Sequentum.ContentGrabber.Commands;
public class Script
{
//See help for a definition of AgentInitializationArguments.
public static bool InitializeAgent(AgentInitializationArguments args)
{
// Set "Proxy List" as proxy source
args.Agent.Proxies.ProxyType=AgentProxyType.ProxyPools;
// Modify Proxy List's items.
args.Agent.Proxies.AddProxyListPool().ProxyServers.Clear();
args.Agent.Proxies.AddProxyListPool().ProxyServers.Add(new ProxyServer("123.45.67.12:6000", null, "user", "pass"));
args.Agent.Proxies.AddProxyListPool().ProxyServers.Add(new ProxyServer("123.45.67.34:6000", null, "user", "pass"));
args.Agent.Proxies.AddProxyListPool().ProxyServers.Add(new ProxyServer("123.45.67.56:6000", null,"user", "pass"));
args.Agent.Proxies.AddProxyListPool().ProxyServers.Add(new ProxyServer("123.45.67.78:6000", null, "user", "pass"));
args.Agent.Proxies.AddProxyListPool().ProxyServers.Add(new ProxyServer("123.45.67.90:6000", null, "user", "pass"));
args.Agent.Proxies.AddProxyListPool().ProxyServers.Shuffle();
return true;
}
}
import clr
import random
from Sequentum.ContentGrabber.Api import *
from Sequentum.ContentGrabber.Commands import *
def InitializeAgent(args):
# Set "Proxy List" as proxy source
args.Agent.Proxies.ProxyType=AgentProxyType.ProxyPools
#Modify Proxy List's items.
args.Agent.Proxies.AddProxyListPool().ProxyServers.Clear()
args.Agent.Proxies.AddProxyListPool().ProxyServers.Add(ProxyServer("123.45.67.12:6000", None, "user", "pass"))
args.Agent.Proxies.AddProxyListPool().ProxyServers.Add(ProxyServer("123.45.67.34:6000", None, "user", "pass"))
args.Agent.Proxies.AddProxyListPool().ProxyServers.Add(ProxyServer("123.45.67.56:6000", None,"user", "pass"))
args.Agent.Proxies.AddProxyListPool().ProxyServers.Add(ProxyServer("123.45.67.78:6000", None, "user", "pass"))
args.Agent.Proxies.AddProxyListPool().ProxyServers.Add(ProxyServer("123.45.67.90:6000", None, "user", "pass"))
random.shuffle(args.Agent.Proxies.AddProxyListPool().ProxyServers)
return True
Modifying the Proxy List in the Agent Initialization Script
Proxy Rotation via Scripting
The following sample code rotates proxies, and logs the currently active proxy from an Execute Script command.
C#
Python
using System;
using Sequentum.ContentGrabber.Api;
public class Script
{
//See help for a definition of CustomScriptArguments.
public static CustomScriptReturn CustomScript(CustomScriptArguments args)
{
args.RotateProxies(false);
// Log currently active proxy
var currentActiveProxy = args.ScriptUtilities.GetCurrentActiveProxy().GetProxyAddress();
args.WriteDebug("Current Proxy: " + currentActiveProxy);
return CustomScriptReturn.Empty();
}
}
import clr
from Sequentum.ContentGrabber.Api import *
def CustomScript(args):
args.RotateProxies(False)
#Log currently active proxy
currentActiveProxy = args.ScriptUtilities.GetCurrentActiveProxy().GetProxyAddress()
args.WriteDebug("Current Proxy: " + currentActiveProxy)
return CustomScriptReturn.Empty()
Sample proxy rotation script
Manipulate proxies at run-time on any node or command in the agent
To disable the proxy in initialization script use below method:
args.Agent.Proxies.ProxyType=AgentProxyType.NoProxy;
2. To add and set a particular proxy pool using initialization script use below method:
args.Agent.Proxies.SetAccPool().ProxyAddress="http://acc-test.sequentum.com/sequentum/testproxy1";
args.Agent.Proxies.AddAccPool(true).ProxyAddress=http://acc-test.sequentum.com/sequentum/testprox2";
3. To disable the proxy in elsewhere in the agent like as navigate link, execute script, navigate url and other capture commands use below method:
args.ScriptUtilities.DisableProxies()
4. To enable the proxy in elsewhere in the agent like as navigate link, execute script, navigate url and other capture commands use below method:
args.ScriptUtilities.EnableProxies()
args.ScriptUtilities.RefreshProxyPools(args.Agent.Proxies)