Agent Config Files

Agent config files can be used to enforce specific configuration for a single agent or all agents in a folder. This makes it possible to alter an agent's behavior by simply editing a text file.

The configuration files must be named config.txt and are INI files that have a number of sections followed by properties for each section.

To apply a configuration file to multiple agents, the agents must have the same Shared Folder and the config file must be placed in that Shared Folder. By default, the config file will be applied to all agents with the same Shared Folder, but the config file can specify exactly which agents it should apply to. This is done by adding a Config section with an agent property that specifies a comma-separated list of agent names. The Config section must be the first section in a config file. You can see an example of this below in Example 2.

To apply the configuration to a single agent, place the config file in the agent's folder.

Configuration files correspond tightly to the API. A section in a configuration file specifies the instance of a class to modify, and the properties specify the class properties to modify.

Configuration files can only set agent properties, and cannot add new class instances. However, you can call methods on instances that may allow you to add new class instances.

A configuration file can contain an Include section that specifies paths to other configuration files that will be executed at the point where the include section is encountered.

Configuration files can contain a Parameters section, which is an easy way of setting input parameters.

Special Content Sections can be used to define long content, such as scripts, which can then be assigned to properties. A Content Section must start with content and have the format [content.name] where the name can be any unique name. To assign a Content Section to a property use the format property = [content.name].

NOTE:

The [Include] section of the config file settings can take multiple comma-separated values and both "path" and "paths" keywords can be used for adding multiple config files.

[Include]
path|paths =c:\\test\config1.txt,c:\\test\config2.txt

Example 1

[Include]
path|paths=c:\\test\config.txt
[Parameters]
test=working

[Set]
ObeyRobotsRules=Always

[InternalDatabase.Set]
DatabaseType=SqlServer

[InternalDatabase.DatabaseReference.Set]
DatabaseConnectionName=myconnection

The above configuration file does the following:

Loads and executes the config file c:\test\config.txt
Sets the input parameter test to working.
Sets the agent property ObeyRobotsRules to Always.
Sets the agent property InternalDatabase.DatabaseType to SqlServer.
Sets the agent property InternalDatabase.DatabaseReference.DatabaseConnectionName to myconnection.

Example 2

[config]
agents=test1,test2

[FindOrAdd("longitude", "CalculatedValue").Set]
IsAllowEmptyValue = true
DataType=Float
ChangeTracking=Exclude

The above configuration file does the following:

Restricts the configuration file to the agents test1 and test2
Finds a command "longitude". If the command does not exist, a new Calculated Value command is added.
Sets the command property IsAllowEmptyValue to true.
Sets the command property DataType to Float.
Sets the command property ChangeTracking to Exclude.

Example 3

[Set]
Environment = Custom
CustomEnvironment = [content.env]

[content.env]
PATH = C:\Program Files\Python37
TEMP = %SystemRoot%\TEMP

The above configuration file does the following:

Sets the agent property Environment to Custom.
Sets the agent property CustomEnvironment to a content section that's defined separately.
Adds a content section that specifies two environment variables.

Example 4

[Set]
TemplateLibraryOptionIsEncrypt = True

TemplateLibraryOptionIsShared = True

The above configuration file does the following:

Enables encryption for the Script Template Library, ensuring that scripts are encrypted by default .
Enables the sharing of scripts between agents, allowing scripts to be easily shared and accessed across different agents.

Agent Configuration Window

You can also use Sequentum Enterprise Desktop's Toolbar menu option 'Agent Configuration' to write the agent configuration.

Using Agent Configuration

The new data export pipeline, Named Connections, and Remote Storage can also be added using the config file settings, refer to Adding Data Export Pipeline Using Config File section for detailed reference.

Use the below config file setting for the old data Export Targets Configuration (Compatibility Mode)

[Agent.Set]
UseCompatibilityExport=Compatibility

Config File Reference

Sequentum Enterprise Property Settings	Options	Config File Settings Examples	Description
Internal Database	DatabaseType=SqlServer/PostgreSql/SqLite/MySql	[InternalDatabase.Set] DatabaseType=SqlServer/PostgreSql/SqLite/MySqlIsAutomaticOverwrite=True IsEmbedFiles=falseUseDefaultDownloadPath=false DownloadDirectoryName=C:\Data\ExportDownloadedFileExistHandling=DiscardNewFile/Overwrite/RandomNameIsDeleteDownloadedFiles=true [InternalDatabase.DatabaseReference.Set]DatabaseConnectionName=<MY_CONNECTION_NAME>	Use this config file setting to set the agent property InternalDatabase.DatabaseType to SqlServer or PostgreSql or MySql or SQLite and also it will set the agent property InternalDatabase.DatabaseReference .DatabaseConnectionName to <MY_CONNECTION_NAME>(add your preferred already created connection name in here.)
Separate Export Database	ExportDatabaseType=SqlServer/PostgreSql/MySql	[InternalDatabase.set] IsUseSeparateExportDatabase=false ExportDatabaseType=SqlServer/PostgreSql/MySql [InternalDatabase.ExportDatabaseReference.Set] DatabaseConnectionName=<Seperate_Export_Database_Name>	Use this config setting to add a separate export database for the agent.
Input Parameters	name/value pairs to use as agent input parameters.	[Parameters] run_id_ =new production_run_=true	Use this config to set the input parameters in your agent like run_id_ which will generate an increasing number every time you run your agent and can be used as an output column in the agent to identify the run history. Similarly, production_run_ can be used to set the agent to always run in production mode.
Database Export Target	ExportDestination=DatabaseExport	[ExportDestination.DatabaseExport.Set] IsEnabled=true DatabaseConnectionName=<MY_CONNECTION_NAME> IsKeepHistoricalData=true IsSeparateHistoricalData=true IsSharedDatabaseTables=true AllowDropSchema=Newer	Use this config to set Export Target to DatabaseExport.
Export Target	ExportDestination=CsvExport/ParquetExport	[ExportDestination.CsvExport.Set] (for csv export) [ExportDestination.ParquetExport.Set] (for parquet export) IsEnabled=True IsTimeStampFile=true	Use this config to set Export Target csv or parquet
Script Export Target	ExportDestination=Script	[ExportDestination.ScriptExport.Set] IsEnabled=true ScriptLanguage=Csharp CsharpScript=[Content.ExportData] [Content.ExportData] using System; using System.IO; using System.Text.RegularExpressions; using System.Collections.Generic; using Sequentum.ContentGrabber.Api; using Sequentum.ContentGrabber.Commands; public class Script { public static bool ExportData(DataExportArguments args) { var dt = DateTime.UtcNow; var dtFormatted = dt.ToString("yyyyMMdd"); var exportedFiles = args.ExportToCsv(null, true); var csvFiles = exportedFiles.DataFiles; var newExportedFilesAll = new List<string>(); var newExportedFiles = new List<string>(); foreach (var file in csvFiles) { var filename = Path.GetFileName(file); var filenameWithoutExt = Path.GetFileNameWithoutExtension(file); var ext = Path.GetExtension(file); var newFile = Regex.Replace(file, filename, string.Format("{0}-{1}{2}", filenameWithoutExt, dtFormatted, ext)); if (File.Exists(newFile)) File.Delete(newFile); File.Move(file, newFile); newExportedFilesAll.Add(newFile); } exportedFiles.DataFiles = newExportedFilesAll; exportedFiles.DataFiles = newExportedFiles; return true; } }	Use this config to set the Export Target to Script
CsvExport FilenameTransformation using Csharp script		[ExportDestination.CsvExport.Set] IsEnabled=True IsTimeStampFile=false [ExportDestination.CsvExport.FilenameTransformationScript.Set] IsEnabled=true ScriptLanguage=Csharp CsharpScript=[Content.TransformFilename] [Content.TransformFilename] using System; using Sequentum.ContentGrabber.Api; public class Script //See help for a definition of FilenameTransformationArguments. { public static string TransformFilename(FilenameTransformationArguments args) { //Place your transformation code here. //This example just returns the input data var dt = DateTime.UtcNow; var dtFormatted = dt.ToString("yyyyMMdd"); var outputFileName=args.FileName+"-"+dtFormatted+".csv"; return outputFileName; } }	use this config setting to transform the output/export file name using c# script in the preferred name and format.
Data Distribution to AWS S3		[ExportDestination.AwsS3BucketDelivery.Set] IsEnabled = True BucketName=<s3bucketName> CredentialsFilePath=<S3_credentials_File_Path>	Use this config to deliver the data on S3 Browser.
Data Distribution to Email		[ExportDestination.EmailDelivery.Set] IsEmailDeliveryEnabled=True Server=mail.authsmtp.comPort= 25Username= <username>Password= <password>FromEmailAddress=<from_email_address>RecipientEmailAddresses= <recipent_email_addresses> BccEmailAddresses=<bcc_email_address>IsDeliverDataFiles=TrueIsDeliverExtractedFiles=True IsCompressFiles=False	Use this config to deliver the data on Email.
Data Distribution to FTP		[ExportDestination.FtpDelivery.Set] IsFtpDeliveryEnabled=true Server=1.0.0.0 Port=21 Username=<USERNAME> Password=<PASSWROD> IsDeliverDataFiles=true IsCompressFiles=true RemoteDirectory=test	Use this config to deliver the data on FTP server.
FTP FileNameTransformation for Delivering Compressed file		[ExportDestination.FtpDelivery.FilesTransformationScript.Set] IsEnabled=True CsharpScript=[Content.CompressedFilename] [Content.CompressedFilename] using System; using Sequentum.ContentGrabber.Api; public class Script { public static ExportedFiles TransformFiles(DataDeliveryTransformationArguments args) { var dtFormatted = DateTime.Now.ToString("yyyy.MM.dd_HH.mm.ss"); args.Agent.ExportDestination.FtpDelivery.CompressedFilename= dtFormatted +"_"+args.Agent._name+ ".zip"; return args.ExportedFiles; } }	Use this config setting to change the compressed file name while delivering a compressed file to the FTP server.
Success Criteria	SuccessCriteria=Min DataCount /MaxPageError/Min export rows	[SuccessCriteria.Set] IsDataCountCriteriaEnabled=true IsDataCountCriteriaPercentage=true DataCountCriteria=90 IsPageErrorCriteriaEnabled=true PageErrorCriteria=0 IsExportRowCountCriteriaEnabled=true ExportRowCriteria=1	Use this config to set the Success Criteria in your agent which will define the criteria for a successful data extraction. A success criteria can be defined to tell an agent when it should consider an agent run successful.
Session	SessionSupport=Multiple Performance Single	[Agent.Set] SessionSupport=Multiple	Use this config to configure an agent to support sessions
Robot rules	ObeyRobotsRules=Always/Never	[Agent.Set] ObeyRobotsRules=Always	Use this config to configure an agent to obey robot rules for the target website.
Max Run Time	MaxRunTime=hh:mi:ss MaxRunTime=d.hh:mi:ss	[Agent.Set] MaxRunTime=05:00:00 MaxRunTime=2.00:00:00	This config is used to Limit the max execution time In the given example first is limit to a maximum of 5 hours and for the second example, it is 2 days.
Max Active Parsers		[Agent.Set] MaxActiveParsers=7	This config is used to set max active parsers.
Empty Export Row Handling	EmptyExportRowHandling=Remove ValidationErrorHandling None	[Agent.Set] EmptyExportRowHandling=Remove	Use this config to set export empty row handling.
Export Validation Error Handling	ValidationErrorHandling=RemoveWithError Remove None	[Agent.Set] ValidationErrorHandling=RemoveWithError	Use this config to set remove the row with error handling.
Export Name		[Agent.Set] ExportTableName=<FileName>	Use this config to set Export File Name
Export ID Name		[Agent.Set] ExportIdColumnName=<uniqueIdName>	Use this config to set Export ID Name
Export Column (Multiple export columns having same name)	IsAllowEmptyValue = false/true IsKeyColumn=true/false ChangeTracking=Exclude/include Validation=Export/Runtime Datatype=integer ShortText Decimal DateTime LongText BigInteger Float Boolean	[FindAll("ColumnName").Set] IsAllowEmptyValue = false IsKeyColumn=true ChangeTracking=Exclude Validation=Export Datatype=integer	Use this config to set data validations on all output columns having the same name.
Export Column	IsAllowEmptyValue = false/true IsKeyColumn=true/false ChangeTracking=Exclude/include Validation=Export/Runtime Datatype=integer ShortText Decimal DateTime LongText BigInteger Float Boolean	[FindCapture("ColumnName").Set] IsAllowEmptyValue = false IsKeyColumn=true ChangeTracking=Exclude Validation=Export Datatype=integer	This config is used in special cases where a single column (for example PRODUCT_NAME) is captured at three different tables/nodes then FindCapture will only be applicable to the first instance while second and third instance will be unaffected.
Increase Data Count(Multiple export columns having the same name)	IsIncreaseDataCount=true/false	[FindAll("ColumnName").Set] IsIncreaseDataCount=true	Use this config to set the Increase data count on all the output columns having the same name which will show the total number of output results.
Logging	DefaultLogLevel=Low/High/Medium	[Agent.AgentLogging.Set] DefaultLogLevel=Low IsDefaultLogTofile=true	Use this config to set Log Level.
Centralized Database Logging	LogLevel = 1/2/3	[AgentLogging.CentralizedLogging.Set] ConnectionName = SQLServerNamedConnection EnableCentralizedLogging = true TableName = AgentLogging LogLevel = 1 RetentionDays=30 Environment=Prod	Use this config to enable centralized logging for an agent. The ConnectionName needs to be created through the Application Settings → Managed Named Connections.
Error Handling	ErrorHandling=RetryCommand ExitCommand StopAgent RestartAgent	[Agent.Action.Set] ErrorHandling=RetryCommand ErrorRetryCount=7 ErrorRetryDelay=10000 ErrorRetryClearOnSuccess=false	Use this config for error handling.
Data List- Data Provider	Provider Type=Simple CSV Excel Parquet Selection Script Simple Script Number Range Date Range SQL Server MySQL Oracle OleDB ODBC	[Find("Data List").DataProvider.Set] ProviderType=SqlServer [Find("Data List").DataProvider.DatabaseProvider.Set] DatabaseConnectionName=sequentum_data Sql=Select * from yellowpagesTest;	Use this config setting to set the SQL SERVER database connection and SQL for the input data provider in the Data List Command
Proxy Configuration	Proxy Pools=Agent Control Center, Proxy List, Proxy API, Proxy List, No Proxy Application	[Proxies.SetAccPool().Set] ProxyAddress = https://acc-test.sequentum.com/sequentum/testproxy1 IsRotateAutomatically=true RotationInterval= 200 [Proxies.AddProxyApiPool().Set] ProxyAddress=https://acc-test.sequentum.com/Sequentum/testproxy2 [Proxies.ProxyPools.Set] IsCyclePools=true IsRotatePoolsOnError=true RotateErrorCount=2 RotateErrorSetSize=1 Use the below config setting to Set and Add Multiple Proxy Pools [Proxies.SetAccPool().Set] ProxyAddress = https://acc-test.sequentum.com/sequentum/testproxy1 IsVerifyProxies = True [Proxies.AddAccPool(false).Set] ProxyAddress = https://acc-test.sequentum.com/sequentum/testproxy2 IsVerifyProxies = True Use the below config setting to set NoProxy [Agent.Proxies.Set] ProxyType=NoProxy Use the below config setting to set Application Proxy [Agent.Proxies.Set] ProxyType=Application Use the below config setting to add/modify the proxy addresses within the Proxy List [Proxies.AddProxyListPool(true).GetOrAddServer(0).Set] [Proxies.ProxyPools.Pools.Get(0).GetOrAddServer(0).Set] ProxyAddress=125.0.0.1 Username = uname Password = pswd [Proxies.ProxyPools.Pools.Get(0).GetOrAddServer(1).Set] ProxyAddress=125.0.0.2 Username = uname Password = pswd [Proxies.ProxyPools.Pools.Get(0).GetOrAddServer(2).Set] ProxyAddress=125.0.0.3 Username = uname Password = pswd Use the below config setting to add/modify multiple Proxy Lists [Proxies.AddProxyListPool(true).GetOrAddServer(0).Set] [Proxies.ProxyPools.Pools.Get(0).GetOrAddServer(0).Set] ProxyAddress=125.0.0.1 Username = uname Password = pswd [Proxies.AddProxyListPool(false).GetOrAddServer(1).Set] [Proxies.ProxyPools.Pools.Get(1).GetOrAddServer(0).Set] ProxyAddress=125.0.0.2 Username = uname Password = pswd [Proxies.AddProxyListPool(false).GetOrAddServer(2).Set] [Proxies.ProxyPools.Pools.Get(2).GetOrAddServer(0).Set] ProxyAddress=125.0.0.3 Username = uname Password = pswd Use the below config setting to verify the proxies - [Agent.Proxies.ProxyPools.Pools.Get(0).Set] IsVerifyProxies = True	Use this config setting to set the Proxy server configuration for your agent. AddProxyListPool(true) - is used to clear the already added proxies.
Increase Data Count	IsIncreaseDataCount=true/false	[FindCapture("ColumnName").Set] IsIncreaseDataCount=true	This config is used in special cases where a single column (for example PROD_NAME) is captured at three different tables/nodes then FindCapture will only be applicable to the first instance while the second and third instance will be unaffected.
Email Notifications		[EmailNotification.Set] IsNotificationEnabled=True Server=mail.authsmtp.com Port=25 Username=<USERNAME> Password=<PASSWORD> FromEmailAddress=<from_email_address> RecipientEmailAddresses=<recipent_email_addresses> SuccessEmailAddresses=<success_email_address> IsNotifyAlways=True IsNotifyOnCriticalError=True IsNotifyOnSuccessfullExtrcation=True IsNotifyOnUnsuccessfullExtrcation=True IsUseDefaultSettings=False IsUseDefaultEmailAddresses=False Enabled=Always\|Never\|Environment Environment=Prod\|Dev\|QA Note: Please note that you need to add the Enabled parameter either at the end or after the IsNotificationEnabled parameter only.	Use this config setting to enable and configure Email Notifications available under the Agent Settings menu. Email Notifications can be sent when an agent completes a run and certain specified conditions are met.
Email Formatting		[EmailNotification.Set] IsEmailFormattingEnabled=True EmailSubject = <AGENT_NAME> has been <STATUS> on <DATE_TIME_yyyy-MM-dd> EmailBody = Agent name is <AGENT_NAME> Data Count of the agent is <DATA_COUNT> Export Row Count of the agent is <EXPORT_ROW_COUNT> Input Parameter used in the agent is <INPUT_PARAMETER(Username)>	This configuration code enables Email Notifications with the Email Formatting feature, allowing users to customize both the Email Subject and Body based on their specified configuration.
XPATH		[Agent.FindOrIgnore("ColumnName").Selection.SelectionPaths.Get(0).Set] Xpath = html/div[0]	Use this config setting to add XPath for a command
Thumbnail		[FindCapture("ColumnName").ConvertToThumbnail.Set] IsConvertToThumbnail = true MaximumWidth = 90 MaximumHeight = 90	Use this config to create a thumbnail for an image
Multiple Config files		[include] paths=C:\Data\AgentControlCenter\Sequentum\ExportConfig Agents\AllExportTargets\Shared\config1.txt, C:\Data\AgentControlCenter\Sequentum\ExportConfig Agents\AllExportTargets\Shared\config2.txt	Use comma separated values to load and execute multiple config files from two or more different paths in the [include] section.