Manipulating the Proxy Pool at Runtime

Sequentum Enterprise supports various types of proxy sources. We refer to the collection of all these proxy sources configured in the agent as "Proxy Pool". A single type of proxy source is normally selected at design-time, and automatic proxy rotation is enabled by default, however, for complex proxy rotation requirements, the Scripting Interface in Sequentum Enterprise enables users to control the agent's proxy pool during run-time.

Overriding the Agent's Proxy Configuration

The Scripting Interface exposes the GlobalData.ProxyList object, which takes precedence over the Agent's proxy configuration. This means that if this object is populated with proxies at run-time, the Agent's design-time proxy configuration is ignored, and proxies in the GlobalData.ProxyList are used instead.

Ideally, this custom proxy list must be populated in the Agent Initialization Script, since this script is executed before the agent's "Start URL" gets loaded. It is still possible to manipulate this custom proxy list anywhere in the agent at run-time.

using System;
using Sequentum.ContentGrabber.Api;

public class Script
{
    //See help for a definition of AgentInitializationArguments.
    public static bool InitializeAgent(AgentInitializationArguments args)
    {
        // Add proxy servers
        args.GlobalData.ProxyList.AddProxyServer("123.45.67.12:6000", "user", "pass");
        args.GlobalData.ProxyList.AddProxyServer("123.45.67.34:6000", "user", "pass");
        args.GlobalData.ProxyList.AddProxyServer("123.45.67.56:6000", "user", "pass");
        args.GlobalData.ProxyList.AddProxyServer("123.45.67.78:6000", "user", "pass");
        args.GlobalData.ProxyList.AddProxyServer("123.45.67.90:6000", "user", "pass");
 
        args.GlobalData.ProxyList.IsRotateAutomatically = true;
        args.GlobalData.ProxyList.IsVerifyProxies = false;
 
        return true;
    }
}
import clr
from Sequentum.ContentGrabber.Api import *

def InitializeAgent(args):
	# Add proxy servers
	args.GlobalData.ProxyList.AddProxyServer("123.45.67.12:6000", "user", "pass")
	args.GlobalData.ProxyList.AddProxyServer("123.45.67.34:6000", "user", "pass")
	args.GlobalData.ProxyList.AddProxyServer("123.45.67.56:6000", "user", "pass")
	args.GlobalData.ProxyList.AddProxyServer("123.45.67.78:6000", "user", "pass")
	args.GlobalData.ProxyList.AddProxyServer("123.45.67.90:6000", "user", "pass")
	
	args.GlobalData.ProxyList.IsRotateAutomatically = True
	args.GlobalData.ProxyList.IsVerifyProxies = False
	return True

Sample code that shows how to populate the custom proxy list in the Agent Initialization Script

You will then have to call the args.RotateProxies method in an Execute script command to activate the proxy list. See the section below on how to Rotate Proxies via Scripting.

Another helper method is SetProxy(). This method sets a single proxy in global data and makes that the active proxy. All current proxies in global data will be removed.

using System;
using Sequentum.ContentGrabber.Api;

public class Script
{
    //See help for a definition of CustomScriptArguments.
    public static CustomScriptReturn CustomScript(CustomScriptArguments args)
    {
        args.SetProxy("123.45.67.12:6000", "user", "pass");
return CustomScriptReturn.Empty(); } }
import clr
from Sequentum.ContentGrabber.Api import *

def CustomScript(args):
	args.SetProxy("123.45.67.12:6000", "user", "pass")
	return CustomScriptReturn.Empty()

SetProxy() helper method sample usage in Execute Script command

Changing Proxy Source via Scripting

An Agent can switch to another proxy source at run-time, but this can only be done within the Agent Initialization Script.

using System;
using Sequentum.ContentGrabber.Api;

public class Script
{
    //See help for a definition of AgentInitializationArguments.
    public static bool InitializeAgent(AgentInitializationArguments args)
    {
        //Set ACC Proxy Pool 
         args.Agent.Proxies.AddAccPool().ProxyAddress="http://acc-test.sequentum.com/sequentum/testproxy1";
         //Enable the Proxy Rotation 
args.Agent.Proxies.ProxyPools.Pools[0].IsRotateAutomatically=true;
//Set Proxy Rotation interval
args.Agent.Proxies.ProxyPools.Pools[0].RotationInterval=200; return true; } }
import clr
from Sequentum.ContentGrabber.Api import *

def InitializeAgent(args):
	#Set ACC Proxy Pool 
	args.Agent.Proxies.AddAccPool().ProxyAddress="http://acc-test.sequentum.com/sequentum/testproxy1";
        #Enable the Proxy Rotation         
args.Agent.Proxies.ProxyPools.Pools[0].IsRotateAutomatically=True
#Set Proxy Rotation interval
args.Agent.Proxies.ProxyPools.Pools[0].RotationInterval=200 return True

Sample code for changing proxy source in the Agent Initialization Script

Additionally, a proxy source's configuration can be accessed and modified from within the Agent Initialization Script.

using System;
using Sequentum.ContentGrabber.Api;
using Sequentum.ContentGrabber.Commands;

public class Script
{
    //See help for a definition of AgentInitializationArguments.
    public static bool InitializeAgent(AgentInitializationArguments args)
    {
       // Set "Proxy List" as proxy source
      args.Agent.Proxies.ProxyType=AgentProxyType.ProxyPools;
      // Modify Proxy List's items.
      args.Agent.Proxies.AddProxyListPool().ProxyServers.Clear();
      args.Agent.Proxies.AddProxyListPool().ProxyServers.Add(new ProxyServer("123.45.67.12:6000", null, "user", "pass"));
      args.Agent.Proxies.AddProxyListPool().ProxyServers.Add(new ProxyServer("123.45.67.34:6000", null, "user", "pass"));
      args.Agent.Proxies.AddProxyListPool().ProxyServers.Add(new ProxyServer("123.45.67.56:6000", null,"user", "pass"));
      args.Agent.Proxies.AddProxyListPool().ProxyServers.Add(new ProxyServer("123.45.67.78:6000", null, "user", "pass"));
      args.Agent.Proxies.AddProxyListPool().ProxyServers.Add(new ProxyServer("123.45.67.90:6000", null, "user", "pass"));

      args.Agent.Proxies.AddProxyListPool().ProxyServers.Shuffle();

      return true;
    }
}
import clr
import random
from Sequentum.ContentGrabber.Api import *
from Sequentum.ContentGrabber.Commands import *

def InitializeAgent(args):
	# Set "Proxy List" as proxy source
	args.Agent.Proxies.ProxyType=AgentProxyType.ProxyPools
	#Modify Proxy List's items.
	args.Agent.Proxies.AddProxyListPool().ProxyServers.Clear()
	args.Agent.Proxies.AddProxyListPool().ProxyServers.Add(ProxyServer("123.45.67.12:6000", None, "user", "pass"))
	args.Agent.Proxies.AddProxyListPool().ProxyServers.Add(ProxyServer("123.45.67.34:6000", None, "user", "pass"))
	args.Agent.Proxies.AddProxyListPool().ProxyServers.Add(ProxyServer("123.45.67.56:6000", None,"user", "pass"))
	args.Agent.Proxies.AddProxyListPool().ProxyServers.Add(ProxyServer("123.45.67.78:6000", None, "user", "pass"))
	args.Agent.Proxies.AddProxyListPool().ProxyServers.Add(ProxyServer("123.45.67.90:6000", None, "user", "pass"))
	
	random.shuffle(args.Agent.Proxies.AddProxyListPool().ProxyServers)
	return True

Modifying the Proxy List in the Agent Initialization Script

Proxy Rotation via Scripting

The following sample code rotates proxies, and logs the currently active proxy from an Execute Script command.

using System;
using Sequentum.ContentGrabber.Api;

public class Script
{
    //See help for a definition of CustomScriptArguments.
    public static CustomScriptReturn CustomScript(CustomScriptArguments args)
    {
        args.RotateProxies(false);

        // Log currently active proxy
        var currentActiveProxy = args.ScriptUtilities.GetCurrentActiveProxy().GetProxyAddress();
        args.WriteDebug("Current Proxy: " + currentActiveProxy);
        
        return CustomScriptReturn.Empty();
    }
}
import clr
from Sequentum.ContentGrabber.Api import *

def CustomScript(args):
	args.RotateProxies(False)
	#Log currently active proxy
	currentActiveProxy = args.ScriptUtilities.GetCurrentActiveProxy().GetProxyAddress()
	args.WriteDebug("Current Proxy: " + currentActiveProxy)
	return CustomScriptReturn.Empty()

Sample proxy rotation script

Was this article helpful?
0 out of 0 found this helpful

Comments

0 comments

Please sign in to leave a comment.