Manipulating the Proxy Pool at Runtime

Sequentum Enterprise supports various types of proxy sources. We refer to the collection of all these proxy sources configured in the agent as "Proxy Pool". A single type of proxy source is normally selected at design-time, and automatic proxy rotation is enabled by default, however, for complex proxy rotation requirements, the Scripting Interface in Sequentum Enterprise enables users to control the agent's proxy pool during run-time.

Overriding the Agent's Proxy Configuration

The Scripting Interface exposes the GlobalData.ProxyList object, which takes precedence over the Agent's proxy configuration. This means that if this object is populated with proxies at run-time, the Agent's design-time proxy configuration is ignored, and proxies in the GlobalData.ProxyList are used instead.

Ideally, this custom proxy list must be populated in the Agent Initialization Script, since this script is executed before the agent's "Start URL" gets loaded. It is still possible to manipulate this custom proxy list anywhere in the agent at run-time.

C#
Python

using System;
using Sequentum.ContentGrabber.Api;

public class Script
{
    //See help for a definition of AgentInitializationArguments.
    public static bool InitializeAgent(AgentInitializationArguments args)
    {
        // Add proxy servers
        args.GlobalData.ProxyList.AddProxyServer("123.45.67.12:6000", "user", "pass");
        args.GlobalData.ProxyList.AddProxyServer("123.45.67.34:6000", "user", "pass");
        args.GlobalData.ProxyList.AddProxyServer("123.45.67.56:6000", "user", "pass");
        args.GlobalData.ProxyList.AddProxyServer("123.45.67.78:6000", "user", "pass");
        args.GlobalData.ProxyList.AddProxyServer("123.45.67.90:6000", "user", "pass");
 
        args.GlobalData.ProxyList.IsRotateAutomatically = true;
        args.GlobalData.ProxyList.IsVerifyProxies = false;
 
        return true;
    }
}

import clr
from Sequentum.ContentGrabber.Api import *

def InitializeAgent(args):
	# Add proxy servers
	args.GlobalData.ProxyList.AddProxyServer("123.45.67.12:6000", "user", "pass")
	args.GlobalData.ProxyList.AddProxyServer("123.45.67.34:6000", "user", "pass")
	args.GlobalData.ProxyList.AddProxyServer("123.45.67.56:6000", "user", "pass")
	args.GlobalData.ProxyList.AddProxyServer("123.45.67.78:6000", "user", "pass")
	args.GlobalData.ProxyList.AddProxyServer("123.45.67.90:6000", "user", "pass")
	
	args.GlobalData.ProxyList.IsRotateAutomatically = True
	args.GlobalData.ProxyList.IsVerifyProxies = False
	return True

Sample code that shows how to populate the custom proxy list in the Agent Initialization Script

You will then have to call the args.RotateProxies method in an Execute script command to activate the proxy list. See the section below on how to Rotate Proxies via Scripting.

Another helper method is SetProxy(). This method sets a single proxy in global data and makes that the active proxy. All current proxies in global data will be removed.

C#
Python

using System;
using Sequentum.ContentGrabber.Api;

public class Script
{
    //See help for a definition of CustomScriptArguments.
    public static CustomScriptReturn CustomScript(CustomScriptArguments args)
    {
        args.SetProxy("123.45.67.12:6000", "user", "pass");
        return CustomScriptReturn.Empty();
    }
}

import clr
from Sequentum.ContentGrabber.Api import *

def CustomScript(args):
	args.SetProxy("123.45.67.12:6000", "user", "pass")
	return CustomScriptReturn.Empty()

SetProxy() helper method sample usage in Execute Script command

Changing Proxy Source via Scripting

An Agent can switch to another proxy source at run-time, but this can only be done within the Agent Initialization Script.

C#
Python

using System;
using Sequentum.ContentGrabber.Api;

public class Script
{
    //See help for a definition of AgentInitializationArguments.
    public static bool InitializeAgent(AgentInitializationArguments args)
    {
        //Set ACC Proxy Pool 
         args.Agent.Proxies.AddAccPool().ProxyAddress="http://acc-test.sequentum.com/sequentum/testproxy1";
         //Enable the Proxy Rotation 
         args.Agent.Proxies.ProxyPools.Pools[0].IsRotateAutomatically=true;
         //Set Proxy Rotation interval
         args.Agent.Proxies.ProxyPools.Pools[0].RotationInterval=200;
        return true;
    }
}

import clr
from Sequentum.ContentGrabber.Api import *

def InitializeAgent(args):
	#Set ACC Proxy Pool 
	args.Agent.Proxies.AddAccPool().ProxyAddress="http://acc-test.sequentum.com/sequentum/testproxy1";
        #Enable the Proxy Rotation         
        args.Agent.Proxies.ProxyPools.Pools[0].IsRotateAutomatically=True
        #Set Proxy Rotation interval
        args.Agent.Proxies.ProxyPools.Pools[0].RotationInterval=200
	return True

Sample code for changing proxy source in the Agent Initialization Script

Additionally, a proxy source's configuration can be accessed and modified from within the Agent Initialization Script.

NOTE: To use the following code in the agent, use this additional library. By default the library is not added in Agent Initialization Script.
C# : using Sequentum.ContentGrabber.Commands;
Python: from Sequentum.ContentGrabber.Commands import *

C#
Python

using System;
using Sequentum.ContentGrabber.Api;
using Sequentum.ContentGrabber.Commands;

public class Script
{
    //See help for a definition of AgentInitializationArguments.
    public static bool InitializeAgent(AgentInitializationArguments args)
    {
       // Set "Proxy List" as proxy source
      args.Agent.Proxies.ProxyType=AgentProxyType.ProxyPools;
      // Modify Proxy List's items.
      args.Agent.Proxies.AddProxyListPool().ProxyServers.Clear();
      args.Agent.Proxies.AddProxyListPool().ProxyServers.Add(new ProxyServer("123.45.67.12:6000", null, "user", "pass"));
      args.Agent.Proxies.AddProxyListPool().ProxyServers.Add(new ProxyServer("123.45.67.34:6000", null, "user", "pass"));
      args.Agent.Proxies.AddProxyListPool().ProxyServers.Add(new ProxyServer("123.45.67.56:6000", null,"user", "pass"));
      args.Agent.Proxies.AddProxyListPool().ProxyServers.Add(new ProxyServer("123.45.67.78:6000", null, "user", "pass"));
      args.Agent.Proxies.AddProxyListPool().ProxyServers.Add(new ProxyServer("123.45.67.90:6000", null, "user", "pass"));

      args.Agent.Proxies.AddProxyListPool().ProxyServers.Shuffle();

      return true;
    }
}

import clr
import random
from Sequentum.ContentGrabber.Api import *
from Sequentum.ContentGrabber.Commands import *

def InitializeAgent(args):
	# Set "Proxy List" as proxy source
	args.Agent.Proxies.ProxyType=AgentProxyType.ProxyPools
	#Modify Proxy List's items.
	args.Agent.Proxies.AddProxyListPool().ProxyServers.Clear()
	args.Agent.Proxies.AddProxyListPool().ProxyServers.Add(ProxyServer("123.45.67.12:6000", None, "user", "pass"))
	args.Agent.Proxies.AddProxyListPool().ProxyServers.Add(ProxyServer("123.45.67.34:6000", None, "user", "pass"))
	args.Agent.Proxies.AddProxyListPool().ProxyServers.Add(ProxyServer("123.45.67.56:6000", None,"user", "pass"))
	args.Agent.Proxies.AddProxyListPool().ProxyServers.Add(ProxyServer("123.45.67.78:6000", None, "user", "pass"))
	args.Agent.Proxies.AddProxyListPool().ProxyServers.Add(ProxyServer("123.45.67.90:6000", None, "user", "pass"))
	
	random.shuffle(args.Agent.Proxies.AddProxyListPool().ProxyServers)
	return True

Modifying the Proxy List in the Agent Initialization Script

Proxy Rotation via Scripting

The following sample code rotates proxies, and logs the currently active proxy from an Execute Script command.

C#
Python

using System;
using Sequentum.ContentGrabber.Api;

public class Script
{
    //See help for a definition of CustomScriptArguments.
    public static CustomScriptReturn CustomScript(CustomScriptArguments args)
    {
        args.RotateProxies(false);

        // Log currently active proxy
        var currentActiveProxy = args.ScriptUtilities.GetCurrentActiveProxy().GetProxyAddress();
        args.WriteDebug("Current Proxy: " + currentActiveProxy);
        
        return CustomScriptReturn.Empty();
    }
}

import clr
from Sequentum.ContentGrabber.Api import *

def CustomScript(args):
	args.RotateProxies(False)
	#Log currently active proxy
	currentActiveProxy = args.ScriptUtilities.GetCurrentActiveProxy().GetProxyAddress()
	args.WriteDebug("Current Proxy: " + currentActiveProxy)
	return CustomScriptReturn.Empty()

Sample proxy rotation script

Manipulate proxies at run-time on any node or command in the agent

1. To disable the proxy in initialization script use below method:

 args.Agent.Proxies.ProxyType=AgentProxyType.NoProxy;

2. To add and set a particular proxy pool using initialization script use below method:

args.Agent.Proxies.SetAccPool().ProxyAddress="http://acc-test.sequentum.com/sequentum/testproxy1";

args.Agent.Proxies.AddAccPool(true).ProxyAddress=http://acc-test.sequentum.com/sequentum/testprox2";

3. To disable the proxy in elsewhere in the agent like as navigate link, execute script, navigate url and other capture commands use below method:

args.ScriptUtilities.DisableProxies()

4. To enable the proxy in elsewhere in the agent like as navigate link, execute script, navigate url and other capture commands use below method:

args.ScriptUtilities.EnableProxies()

args.ScriptUtilities.RefreshProxyPools(args.Agent.Proxies)

Manipulating the Proxy Pool at Runtime

Overriding the Agent's Proxy Configuration

Changing Proxy Source via Scripting

Proxy Rotation via Scripting

Manipulate proxies at run-time on any node or command in the agent

Articles in this section

Comments