Skip to main content

Agent Config Files

Agent config files can be used to enforce specific configuration for a single agent or all agents in a folder. This makes it possible to alter an agent's behavior by simply editing a text file.

The configuration files must be named config.txt and are INI files that have a number of sections followed by properties for each section.

To apply a configuration file to multiple agents, the agents must have the same Shared Folder and the config file must be placed in that Shared Folder. By default, the config file will be applied to all agents with the same Shared Folder, but the config file can specify exactly which agents it should apply to. This is done by adding a Config section with an agent property that specifies a comma-separated list of agent names. The Config section must be the first section in a config file. You can see an example of this below in Example 2.

To apply the configuration to a single agent, place the config file in the agent's folder.

Configuration files correspond tightly to the API. A section in a configuration file specifies the instance of a class to modify, and the properties specify the class properties to modify.

Configuration files can only set agent properties, and cannot add new class instances. However, you can call methods on instances that may allow you to add new class instances.

A configuration file can contain an Include section that specifies paths to other configuration files that will be executed at the point where the include section is encountered.

Configuration files can contain a Parameters section, which is an easy way of setting input parameters.

Special Content Sections can be used to define long content, such as scripts, which can then be assigned to properties. A Content Section must start with content and have the format [content.name] where the name can be any unique name. To assign a Content Section to a property use the format property = [content.name].

NOTE:

The [Include] section of the config file settings can take multiple comma-separated values and both "path" and "paths" keywords can be used for adding multiple config files.

[Include]
path|paths =c:\\test\config1.txt,c:\\test\config2.txt

Example 1 

[Include]
path|paths=c:\\test\config.txt
[Parameters]
test=working 

[Set]
ObeyRobotsRules=Always 

[InternalDatabase.Set]
DatabaseType=SqlServer

[InternalDatabase.DatabaseReference.Set]
DatabaseConnectionName=myconnection

The above configuration file does the following:

  1. Loads and executes the config file c:\test\config.txt

  2. Sets the input parameter test to working.

  3. Sets the agent property ObeyRobotsRules to Always.

  4. Sets the agent property InternalDatabase.DatabaseType to SqlServer.

  5. Sets the agent property InternalDatabase.DatabaseReference.DatabaseConnectionName to myconnection.

Example 2

[config]
agents=test1,test2

[FindOrAdd("longitude", "CalculatedValue").Set]
IsAllowEmptyValue = true
DataType=Float
ChangeTracking=Exclude

The above configuration file does the following:

  1. Restricts the configuration file to the agents test1 and test2

  2. Finds a command "longitude". If the command does not exist, a new Calculated Value command is added. 

  3. Sets the command property IsAllowEmptyValue to true.

  4. Sets the command property DataType to Float.

  5. Sets the command property ChangeTracking to Exclude.

Example 3

[Set]
Environment = Custom
CustomEnvironment = [content.env]

[content.env]
PATH = C:\Program Files\Python37
TEMP = %SystemRoot%\TEMP

The above configuration file does the following:

  1. Sets the agent property Environment to Custom.

  2. Sets the agent property CustomEnvironment to a content section that's defined separately.

  3. Adds a content section that specifies two environment variables.

Agent Configuration Window

You can also use Sequentum Enterprise Desktop's Toolbar menu option 'Agent Configuration' to write the agent configuration.

AgentConfiguration.gif

Using Agent Configuration

The new data export pipeline, Named Connections, and Remote Storage can also be added using the config file settings, refer to Adding Data Export Pipeline Using Config File section for detailed reference.

Use the below config file setting for the old data  Export Targets Configuration (Compatibility Mode)

[Agent.Set]
UseCompatibilityExport=Compatibility

Config File Reference

Sequentum Enterprise  Property Settings

Options

Config File Settings Examples

Description

Internal Database
DatabaseType=SqlServer/PostgreSql/SqLite/MySql
[InternalDatabase.Set]
DatabaseType=SqlServer/PostgreSql/SqLite/MySqlIsAutomaticOverwrite=True
IsEmbedFiles=falseUseDefaultDownloadPath=false
DownloadDirectoryName=C:\Data\ExportDownloadedFileExistHandling=DiscardNewFile/Overwrite/RandomNameIsDeleteDownloadedFiles=true
[InternalDatabase.DatabaseReference.Set]DatabaseConnectionName=<MY_CONNECTION_NAME>
Use this config file setting
to set the agent property
InternalDatabase.DatabaseType
to SqlServer or PostgreSql or MySql or SQLite
and also it will set the agent property InternalDatabase.DatabaseReference
.DatabaseConnectionName to <MY_CONNECTION_NAME>(add your preferred already created connection name in here.)
Separate Export Database
ExportDatabaseType=SqlServer/PostgreSql/MySql
[InternalDatabase.set]
IsUseSeparateExportDatabase=false ExportDatabaseType=SqlServer/PostgreSql/MySql [InternalDatabase.ExportDatabaseReference.Set] DatabaseConnectionName=<Seperate_Export_Database_Name>
Use this config setting to add a separate export database for the agent.
Input Parameters
name/value pairs to use as agent input parameters.
[Parameters]
run_id_ =new
production_run_=true
Use this config to set the input parameters in your agent like run_id_ which will generate an increasing number every time you run your agent and can be used as an output column in the agent to identify the run history. Similarly, production_run_ can be used to set the agent to always run in production mode.
Database Export Target
ExportDestination=DatabaseExport
[ExportDestination.DatabaseExport.Set]
IsEnabled=true
DatabaseConnectionName=<MY_CONNECTION_NAME> IsKeepHistoricalData=true
IsSeparateHistoricalData=true
IsSharedDatabaseTables=true
AllowDropSchema=Newer
Use this config to set Export Target to DatabaseExport.
Export Target
ExportDestination=CsvExport/ParquetExport
[ExportDestination.CsvExport.Set] (for csv export) [ExportDestination.ParquetExport.Set] (for parquet export) IsEnabled=True
IsTimeStampFile=true
Use this config to set Export Target csv or parquet
Script Export Target
ExportDestination=Script
[ExportDestination.ScriptExport.Set]
IsEnabled=true
ScriptLanguage=Csharp
CsharpScript=[Content.ExportData]
[Content.ExportData]
using System; using System.IO;
using System.Text.RegularExpressions;
using System.Collections.Generic; using Sequentum.ContentGrabber.Api;
using Sequentum.ContentGrabber.Commands;
public class Script
{
public static bool ExportData(DataExportArguments args)
{
var dt = DateTime.UtcNow;
var dtFormatted = dt.ToString("yyyyMMdd");
var exportedFiles = args.ExportToCsv(null, true);
var csvFiles = exportedFiles.DataFiles;
var newExportedFilesAll = new List<string>();
var newExportedFiles = new List<string>();
foreach (var file in csvFiles)
{
var filename = Path.GetFileName(file);
var filenameWithoutExt = Path.GetFileNameWithoutExtension(file);
var ext = Path.GetExtension(file);
var newFile = Regex.Replace(file, filename, string.Format("{0}-{1}{2}", filenameWithoutExt, dtFormatted, ext));
if (File.Exists(newFile)) File.Delete(newFile);
File.Move(file, newFile);
newExportedFilesAll.Add(newFile);
}
exportedFiles.DataFiles = newExportedFilesAll;
exportedFiles.DataFiles = newExportedFiles; return true;
}
}
Use this config to set the Export Target to Script
CsvExport FilenameTransformation using Csharp script
 
[ExportDestination.CsvExport.Set]
IsEnabled=True
IsTimeStampFile=false [ExportDestination.CsvExport.FilenameTransformationScript.Set] IsEnabled=true
ScriptLanguage=Csharp
CsharpScript=[Content.TransformFilename] [Content.TransformFilename]
using System;
using Sequentum.ContentGrabber.Api;
public class Script
//See help for a definition of FilenameTransformationArguments.
{ 
public static string TransformFilename(FilenameTransformationArguments args)
{
//Place your transformation code here. //This example just returns the input data var dt = DateTime.UtcNow;
var dtFormatted = dt.ToString("yyyyMMdd");
var outputFileName=args.FileName+"-"+dtFormatted+".csv";
return outputFileName;
}
}
use this config setting to transform the output/export file name using c# script in the preferred name and format.
Data Distribution to AWS S3
 
[ExportDestination.AwsS3BucketDelivery.Set]
IsEnabled = True
BucketName=<s3bucketName>
CredentialsFilePath=<S3_credentials_File_Path>
Use this config to deliver the data on S3 Browser.
Data Distribution to Email
 
[ExportDestination.EmailDelivery.Set]
IsEmailDeliveryEnabled=True Server=mail.authsmtp.comPort= 25Username= <username>Password= <password>FromEmailAddress=<from_email_address>RecipientEmailAddresses= <recipent_email_addresses> BccEmailAddresses=<bcc_email_address>IsDeliverDataFiles=TrueIsDeliverExtractedFiles=True
IsCompressFiles=False
 
Use this config to deliver the data on Email.
Data Distribution to FTP
 
[ExportDestination.FtpDelivery.Set]
IsFtpDeliveryEnabled=true
Server=1.0.0.0 Port=21
Username=<USERNAME>
Password=<PASSWROD>
IsDeliverDataFiles=true
IsCompressFiles=true
RemoteDirectory=test
Use this config to deliver the data on FTP server.
FTP FileNameTransformation for Delivering Compressed file
 
[ExportDestination.FtpDelivery.FilesTransformationScript.Set] IsEnabled=True
CsharpScript=[Content.CompressedFilename]
[Content.CompressedFilename]
using System;
using Sequentum.ContentGrabber.Api;
public class Script
{ 
public static ExportedFiles TransformFiles(DataDeliveryTransformationArguments args)
{
var dtFormatted = DateTime.Now.ToString("yyyy.MM.dd_HH.mm.ss"); args.Agent.ExportDestination.FtpDelivery.CompressedFilename= dtFormatted +"_"+args.Agent._name+ ".zip"; return args.ExportedFiles; } }
Use this config setting to change the compressed file name while delivering a compressed file to the FTP server.
Success Criteria
SuccessCriteria=Min DataCount /MaxPageError/Min export rows
[SuccessCriteria.Set]
IsDataCountCriteriaEnabled=true
IsDataCountCriteriaPercentage=true
DataCountCriteria=90
IsPageErrorCriteriaEnabled=true
PageErrorCriteria=0
IsExportRowCountCriteriaEnabled=true
ExportRowCriteria=1
Use this config to set the Success Criteria in your agent which will define the criteria for a successful data extraction. A success criteria can be defined to tell an agent when it should consider an agent run successful.
Session
SessionSupport=Multiple
Performance
Single
[Agent.Set]
SessionSupport=Multiple
Use this config to configure an agent to support sessions
Robot rules
 ObeyRobotsRules=Always/Never
 [Agent.Set]
ObeyRobotsRules=Always
 Use this config to configure an agent to obey robot rules for the target website.
 
Max Run Time
MaxRunTime=hh:mi:ss MaxRunTime=d.hh:mi:ss
 [Agent.Set]
MaxRunTime=05:00:00
MaxRunTime=2.00:00:00
 This config is used to Limit the max execution time In the given example first is limit to a maximum of 5 hours and for the second example, it is 2 days.
 Max Active Parsers
 
 [Agent.Set]
MaxActiveParsers=7
 This config is used to set max active parsers.
 
Empty Export Row Handling
 EmptyExportRowHandling=Remove
ValidationErrorHandling
None
 [Agent.Set]
EmptyExportRowHandling=Remove
 Use this config to set export empty row handling.
 
Export Validation Error Handling
 ValidationErrorHandling=RemoveWithError
Remove
None
 [Agent.Set]
ValidationErrorHandling=RemoveWithError
 Use this config to set remove the row with error handling.
 
Export Name
 
 [Agent.Set]
ExportTableName=<FileName>
 Use this config to set Export File Name
 
Export ID Name
 
 [Agent.Set]
ExportIdColumnName=<uniqueIdName>
 Use this config to set Export ID Name
 
Export Column (Multiple export columns having same name)
 IsAllowEmptyValue = false/true
IsKeyColumn=true/false
ChangeTracking=Exclude/include
Validation=Export/Runtime Datatype=integer
ShortText
Decimal
DateTime
LongText
BigInteger
Float
Boolean
 
[FindAll("ColumnName").Set]
IsAllowEmptyValue = false
IsKeyColumn=true
ChangeTracking=Exclude
Validation=Export Datatype=integer
 
Use this config to set data validations on all output columns having the same name.
 
Export Column
 IsAllowEmptyValue = false/true
IsKeyColumn=true/false
ChangeTracking=Exclude/include
Validation=Export/Runtime Datatype=integer
ShortText
Decimal
DateTime
LongText
BigInteger
Float
Boolean
 
[FindCapture("ColumnName").Set]
IsAllowEmptyValue = false
IsKeyColumn=true
ChangeTracking=Exclude
Validation=Export Datatype=integer
This config is used in special cases where a single column (for example PRODUCT_NAME) is captured at three different tables/nodes then FindCapture will only be applicable to the first instance while second and third instance will be unaffected.
Increase Data Count(Multiple export columns having the same name)
 
IsIncreaseDataCount=true/false
[FindAll("ColumnName").Set]
IsIncreaseDataCount=true
Use this config to set the Increase data count on all the output columns having the same name which will show the total number of output results.
 
Logging
 
DefaultLogLevel=Low/High/Medium
 [Agent.AgentLogging.Set]
DefaultLogLevel=Low
IsDefaultLogTofile=true
Use this config to set Log Level.
Centralized Database Logging
LogLevel = 1/2/3
[AgentLogging.CentralizedLogging.Set]
ConnectionName = SQLServerNamedConnection
EnableCentralizedLogging = true
TableName = AgentLogging
LogLevel = 1
RetentionDays=30
Environment=Prod
Use this config to enable centralized logging for an agent.
The ConnectionName needs to be created through the Application Settings → Managed Named Connections.
 
Error Handling
 ErrorHandling=RetryCommand
 ExitCommand
 StopAgent
 RestartAgent
 [Agent.Action.Set]
ErrorHandling=RetryCommand
ErrorRetryCount=7
ErrorRetryDelay=10000
ErrorRetryClearOnSuccess=false
 Use this config for error handling.
 
Data List- Data Provider
 Provider Type=Simple CSV
Excel
Parquet
Selection Script
Simple Script
Number Range
Date Range
SQL Server
MySQL
Oracle
OleDB
ODBC
[Find("Data List").DataProvider.Set]
ProviderType=SqlServer
[Find("Data List").DataProvider.DatabaseProvider.Set] DatabaseConnectionName=sequentum_data
Sql=Select * from yellowpagesTest;
 
Use this config setting to set the SQL SERVER database connection and SQL for the input data provider in the Data List Command
 
Proxy Configuration
Proxy Pools=Agent Control Center, Proxy List, Proxy API, Proxy List,
No Proxy
Application
 
 [Proxies.SetAccPool().Set]
ProxyAddress = https://acc-test.sequentum.com/sequentum/testproxy1
IsRotateAutomatically=true
RotationInterval= 200
[Proxies.AddProxyApiPool().Set]
ProxyAddress=https://acc-test.sequentum.com/Sequentum/testproxy2
[Proxies.ProxyPools.Set]
IsCyclePools=true
IsRotatePoolsOnError=true
RotateErrorCount=2
RotateErrorSetSize=1
Use the below config setting to Set and Add Multiple Proxy Pools
[Proxies.SetAccPool().Set]
ProxyAddress = https://acc-test.sequentum.com/sequentum/testproxy1
IsVerifyProxies = True
[Proxies.AddAccPool(false).Set]
ProxyAddress = https://acc-test.sequentum.com/sequentum/testproxy2
IsVerifyProxies = True
Use the below config setting to set NoProxy 
[Agent.Proxies.Set]
ProxyType=NoProxy
Use the below config setting to set Application Proxy 
[Agent.Proxies.Set]
ProxyType=Application

Use the below config setting to add/modify the proxy addresses within the Proxy List 
[Proxies.AddProxyListPool(true).GetOrAddServer(0).Set]
[Proxies.ProxyPools.Pools.Get(0).GetOrAddServer(0).Set]
ProxyAddress=125.0.0.1
Username = uname
Password = pswd
[Proxies.ProxyPools.Pools.Get(0).GetOrAddServer(1).Set]
ProxyAddress=125.0.0.2
Username = uname
Password = pswd
[Proxies.ProxyPools.Pools.Get(0).GetOrAddServer(2).Set]
ProxyAddress=125.0.0.3
Username = uname
Password = pswd

Use the below config setting to add/modify multiple Proxy Lists  

[Proxies.AddProxyListPool(true).GetOrAddServer(0).Set]
[Proxies.ProxyPools.Pools.Get(0).GetOrAddServer(0).Set]
ProxyAddress=125.0.0.1
Username = uname
Password = pswd
[Proxies.AddProxyListPool(false).GetOrAddServer(1).Set]
[Proxies.ProxyPools.Pools.Get(1).GetOrAddServer(0).Set]
ProxyAddress=125.0.0.2
Username = uname
Password = pswd
[Proxies.AddProxyListPool(false).GetOrAddServer(2).Set]
[Proxies.ProxyPools.Pools.Get(2).GetOrAddServer(0).Set]
ProxyAddress=125.0.0.3
Username = uname
Password = pswd
Use the below config setting to verify the proxies - 

[Agent.Proxies.ProxyPools.Pools.Get(0).Set]
IsVerifyProxies = True
Use this config setting to set the Proxy server configuration for your agent.





































AddProxyListPool(true) - is used to clear the already added proxies.
Increase Data Count
IsIncreaseDataCount=true/false
[FindCapture("ColumnName").Set] IsIncreaseDataCount=true
This config is used in special cases where a single column (for example PROD_NAME) is captured at three different tables/nodes then FindCapture will only be applicable to the first instance while the second and third instance will be unaffected.
Email Notifications
 
[EmailNotification.Set]
IsNotificationEnabled=True
Server=mail.authsmtp.com
Port=25
Username=<USERNAME>
Password=<PASSWORD>
FromEmailAddress=<from_email_address>
RecipientEmailAddresses=<recipent_email_addresses>
SuccessEmailAddresses=<success_email_address>
IsNotifyAlways=True
IsNotifyOnCriticalError=True
IsNotifyOnSuccessfullExtrcation=True
IsNotifyOnUnsuccessfullExtrcation=True
IsUseDefaultSettings=False
IsUseDefaultEmailAddresses=False
Enabled=Always|Never|Environment
Environment=Prod|Dev|QA
Note: Please note that you need to add the Enabled parameter either at the end or after the IsNotificationEnabled parameter only.
Use this config setting to enable and configure Email Notifications available under the Agent Settings menu. Email Notifications can be sent when an agent completes a run and certain specified conditions are met.
XPATH
 
[Agent.FindOrIgnore("ColumnName").Selection.SelectionPaths.Get(0).Set]
Xpath = html/div[0]
Use this config setting to add XPath for a command
Thumbnail
 
[FindCapture("ColumnName").ConvertToThumbnail.Set]
IsConvertToThumbnail = true
MaximumWidth = 90
MaximumHeight = 90
Use this config to create a thumbnail for an image
Multiple Config files
 
[include]
paths=C:\Data\AgentControlCenter\Sequentum\ExportConfig Agents\AllExportTargets\Shared\config1.txt,
C:\Data\AgentControlCenter\Sequentum\ExportConfig Agents\AllExportTargets\Shared\config2.txt
Use comma separated values to load and execute multiple config files from two or more different paths in the [include] section.
JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.

Contact Support Close