Agent Config Files
Agent config files can be used to enforce specific configuration for a single agent or all agents in a folder. This makes it possible to alter an agent's behavior by simply editing a text file.
The configuration files must be named config.txt and are INI files that have a number of sections followed by properties for each section.
To apply a configuration file to multiple agents, the agents must have the same Shared Folder and the config file must be placed in that Shared Folder. By default, the config file will be applied to all agents with the same Shared Folder, but the config file can specify exactly which agents it should apply to. This is done by adding a Config section with an agent property that specifies a comma-separated list of agent names. The Config section must be the first section in a config file. You can see an example of this below in Example 2.
To apply the configuration to a single agent, place the config file in the agent's folder.
Configuration files correspond tightly to the API. A section in a configuration file specifies the instance of a class to modify, and the properties specify the class properties to modify.
Configuration files can only set agent properties, and cannot add new class instances. However, you can call methods on instances that may allow you to add new class instances.
A configuration file can contain an Include section that specifies paths to other configuration files that will be executed at the point where the include section is encountered.
Configuration files can contain a Parameters section, which is an easy way of setting input parameters.
Special Content Sections can be used to define long content, such as scripts, which can then be assigned to properties. A Content Section must start with content and have the format [content.name] where the name can be any unique name. To assign a Content Section to a property use the format property = [content.name].
NOTE:
The [Include] section of the config file settings can take multiple comma-separated values and both "path" and "paths" keywords can be used for adding multiple config files.
[Include]
path|paths =c:\\test\config1.txt,c:\\test\config2.txt
Example 1
[Include]
path|paths=c:\\test\config.txt
[Parameters]
test=working
[Set]
ObeyRobotsRules=Always
[InternalDatabase.Set]
DatabaseType=SqlServer
[InternalDatabase.DatabaseReference.Set]
DatabaseConnectionName=myconnection
The above configuration file does the following:
Loads and executes the config file c:\test\config.txt
Sets the input parameter test to working.
Sets the agent property ObeyRobotsRules to Always.
Sets the agent property InternalDatabase.DatabaseType to SqlServer.
Sets the agent property InternalDatabase.DatabaseReference.DatabaseConnectionName to myconnection.
Example 2
[config]
agents=test1,test2
[FindOrAdd("longitude", "CalculatedValue").Set]
IsAllowEmptyValue = true
DataType=Float
ChangeTracking=Exclude
The above configuration file does the following:
Restricts the configuration file to the agents test1 and test2
Finds a command "longitude". If the command does not exist, a new Calculated Value command is added.
Sets the command property IsAllowEmptyValue to true.
Sets the command property DataType to Float.
Sets the command property ChangeTracking to Exclude.
Example 3
[Set]
Environment = Custom
CustomEnvironment = [content.env]
[content.env]
PATH = C:\Program Files\Python37
TEMP = %SystemRoot%\TEMP
The above configuration file does the following:
Sets the agent property Environment to Custom.
Sets the agent property CustomEnvironment to a content section that's defined separately.
Adds a content section that specifies two environment variables.
Example 4
[Set]
TemplateLibraryOptionIsEncrypt = True
TemplateLibraryOptionIsShared = True
The above configuration file does the following:
Enables encryption for the Script Template Library, ensuring that scripts are encrypted by default .
Enables the sharing of scripts between agents, allowing scripts to be easily shared and accessed across different agents.
Agent Configuration Window
You can also use Sequentum Enterprise Desktop's Toolbar menu option 'Agent Configuration' to write the agent configuration.
Using Agent Configuration
The new data export pipeline, Named Connections, and Remote Storage can also be added using the config file settings, refer to Adding Data Export Pipeline Using Config File section for detailed reference.
Use the below config file setting for the old data Export Targets Configuration (Compatibility Mode)
[Agent.Set]
UseCompatibilityExport=Compatibility
Config File Reference
Sequentum Enterprise Property Settings | Options | Config File Settings Examples | Description |
---|---|---|---|
Internal Database | DatabaseType=SqlServer/PostgreSql/SqLite/MySql | [InternalDatabase.Set]DatabaseType=SqlServer/PostgreSql/SqLite/MySqlIsAutomaticOverwrite=TrueIsEmbedFiles=falseUseDefaultDownloadPath=false | Use this config file setting |
Separate Export Database | ExportDatabaseType=SqlServer/PostgreSql/MySql | [InternalDatabase.set]IsUseSeparateExportDatabase=false ExportDatabaseType=SqlServer/PostgreSql/MySql [InternalDatabase.ExportDatabaseReference.Set] DatabaseConnectionName=<Seperate_Export_Database_Name> | Use this config setting to add a separate export database for the agent. |
Input Parameters | name/value pairs to use as agent input parameters. | [Parameters]run_id_ =newproduction_run_=true | Use this config to set the input parameters in your agent like run_id_ which will generate an increasing number every time you run your agent and can be used as an output column in the agent to identify the run history. Similarly, production_run_ can be used to set the agent to always run in production mode. |
Database Export Target | ExportDestination=DatabaseExport | [ExportDestination.DatabaseExport.Set]IsEnabled=trueDatabaseConnectionName=<MY_CONNECTION_NAME> IsKeepHistoricalData=trueIsSeparateHistoricalData=trueIsSharedDatabaseTables=trueAllowDropSchema=Newer | Use this config to set Export Target to DatabaseExport. |
Export Target | ExportDestination=CsvExport/ParquetExport | [ExportDestination.CsvExport.Set] (for csv export) [ExportDestination.ParquetExport.Set] (for parquet export) IsEnabled=TrueIsTimeStampFile=true | Use this config to set Export Target csv or parquet |
Script Export Target | ExportDestination=Script | [ExportDestination.ScriptExport.Set]IsEnabled=true | Use this config to set the Export Target to Script |
CsvExport FilenameTransformation using Csharp script | [ExportDestination.CsvExport.Set]IsEnabled=TrueIsTimeStampFile=false [ExportDestination.CsvExport.FilenameTransformationScript.Set] IsEnabled=trueScriptLanguage=CsharpCsharpScript=[Content.TransformFilename] [Content.TransformFilename]using System;using Sequentum.ContentGrabber.Api;public class Script//See help for a definition of FilenameTransformationArguments.{public static string TransformFilename(FilenameTransformationArguments args){//Place your transformation code here. //This example just returns the input data var dt = DateTime.UtcNow;var dtFormatted = dt.ToString("yyyyMMdd");var outputFileName=args.FileName+"-"+dtFormatted+".csv";return outputFileName;}} | use this config setting to transform the output/export file name using c# script in the preferred name and format. | |
Data Distribution to AWS S3 | [ExportDestination.AwsS3BucketDelivery.Set]IsEnabled = TrueBucketName=<s3bucketName>CredentialsFilePath=<S3_credentials_File_Path> | Use this config to deliver the data on S3 Browser. | |
Data Distribution to Email | [ExportDestination.EmailDelivery.Set]IsEmailDeliveryEnabled=True Server=mail.authsmtp.comPort= 25Username= <username>Password= <password>FromEmailAddress=<from_email_address>RecipientEmailAddresses= <recipent_email_addresses> BccEmailAddresses=<bcc_email_address>IsDeliverDataFiles=TrueIsDeliverExtractedFiles=True | Use this config to deliver the data on Email. | |
Data Distribution to FTP | [ExportDestination.FtpDelivery.Set]IsFtpDeliveryEnabled=trueServer=1.0.0.0 Port=21Username=<USERNAME>Password=<PASSWROD>IsDeliverDataFiles=trueIsCompressFiles=trueRemoteDirectory=test | Use this config to deliver the data on FTP server. | |
FTP FileNameTransformation for Delivering Compressed file | [ExportDestination.FtpDelivery.FilesTransformationScript.Set] IsEnabled=TrueCsharpScript=[Content.CompressedFilename][Content.CompressedFilename]using System;using Sequentum.ContentGrabber.Api;public class Script{public static ExportedFiles TransformFiles(DataDeliveryTransformationArguments args){var dtFormatted = DateTime.Now.ToString("yyyy.MM.dd_HH.mm.ss"); args.Agent.ExportDestination.FtpDelivery.CompressedFilename= dtFormatted +"_"+args.Agent._name+ ".zip"; return args.ExportedFiles; } } | Use this config setting to change the compressed file name while delivering a compressed file to the FTP server. | |
Success Criteria | SuccessCriteria=Min DataCount /MaxPageError/Min export rows | [SuccessCriteria.Set]IsDataCountCriteriaEnabled=trueIsDataCountCriteriaPercentage=trueDataCountCriteria=90IsPageErrorCriteriaEnabled=truePageErrorCriteria=0IsExportRowCountCriteriaEnabled=trueExportRowCriteria=1 | Use this config to set the Success Criteria in your agent which will define the criteria for a successful data extraction. A success criteria can be defined to tell an agent when it should consider an agent run successful. |
Session | SessionSupport=MultiplePerformanceSingle | [Agent.Set]SessionSupport=Multiple | Use this config to configure an agent to support sessions |
Robot rules | ObeyRobotsRules=Always/Never | [Agent.Set]ObeyRobotsRules=Always | Use this config to configure an agent to obey robot rules for the target website. |
Max Run Time | MaxRunTime=hh:mi:ss MaxRunTime=d.hh:mi:ss | [Agent.Set]MaxRunTime=05:00:00MaxRunTime=2.00:00:00 | This config is used to Limit the max execution time In the given example first is limit to a maximum of 5 hours and for the second example, it is 2 days. |
Max Active Parsers | [Agent.Set]MaxActiveParsers=7 | This config is used to set max active parsers. | |
Empty Export Row Handling | EmptyExportRowHandling=RemoveValidationErrorHandlingNone | [Agent.Set]EmptyExportRowHandling=Remove | Use this config to set export empty row handling. |
Export Validation Error Handling | ValidationErrorHandling=RemoveWithErrorRemoveNone | [Agent.Set]ValidationErrorHandling=RemoveWithError | Use this config to set remove the row with error handling. |
Export Name | [Agent.Set]ExportTableName=<FileName> | Use this config to set Export File Name | |
Export ID Name | [Agent.Set]ExportIdColumnName=<uniqueIdName> | Use this config to set Export ID Name | |
Export Column (Multiple export columns having same name) | IsAllowEmptyValue = false/trueIsKeyColumn=true/falseChangeTracking=Exclude/includeValidation=Export/Runtime Datatype=integerShortTextDecimalDateTimeLongTextBigIntegerFloatBoolean | [FindAll("ColumnName").Set]IsAllowEmptyValue = falseIsKeyColumn=trueChangeTracking=ExcludeValidation=Export Datatype=integer | Use this config to set data validations on all output columns having the same name. |
Export Column | IsAllowEmptyValue = false/trueIsKeyColumn=true/falseChangeTracking=Exclude/includeValidation=Export/Runtime Datatype=integerShortTextDecimalDateTimeLongTextBigIntegerFloatBoolean | [FindCapture("ColumnName").Set]IsAllowEmptyValue = falseIsKeyColumn=trueChangeTracking=ExcludeValidation=Export Datatype=integer | This config is used in special cases where a single column (for example PRODUCT_NAME) is captured at three different tables/nodes then FindCapture will only be applicable to the first instance while second and third instance will be unaffected. |
Increase Data Count(Multiple export columns having the same name) | IsIncreaseDataCount=true/false | [FindAll("ColumnName").Set]IsIncreaseDataCount=true | Use this config to set the Increase data count on all the output columns having the same name which will show the total number of output results. |
Logging | DefaultLogLevel=Low/High/Medium | [Agent.AgentLogging.Set]DefaultLogLevel=LowIsDefaultLogTofile=true | Use this config to set Log Level. |
Centralized Database Logging | LogLevel = 1/2/3 | [AgentLogging.CentralizedLogging.Set]ConnectionName = SQLServerNamedConnectionEnableCentralizedLogging = trueTableName = AgentLoggingLogLevel = 1RetentionDays=30Environment=Prod | Use this config to enable centralized logging for an agent.The ConnectionName needs to be created through the Application Settings → Managed Named Connections. |
Error Handling | ErrorHandling=RetryCommandExitCommandStopAgentRestartAgent | [Agent.Action.Set]ErrorHandling=RetryCommandErrorRetryCount=7ErrorRetryDelay=10000ErrorRetryClearOnSuccess=false | Use this config for error handling. |
Data List- Data Provider | Provider Type=Simple CSVExcelParquetSelection ScriptSimple ScriptNumber RangeDate RangeSQL ServerMySQLOracleOleDBODBC | [Find("Data List").DataProvider.Set]ProviderType=SqlServer[Find("Data List").DataProvider.DatabaseProvider.Set] DatabaseConnectionName=sequentum_dataSql=Select * from yellowpagesTest; | Use this config setting to set the SQL SERVER database connection and SQL for the input data provider in the Data List Command |
Proxy Configuration | Proxy Pools=Agent Control Center, Proxy List, Proxy API, Proxy List,No ProxyApplication | [Proxies.SetAccPool().Set]ProxyAddress = https://acc-test.sequentum.com/sequentum/testproxy1IsRotateAutomatically=trueRotationInterval= 200[Proxies.AddProxyApiPool().Set]ProxyAddress=https://acc-test.sequentum.com/Sequentum/testproxy2[Proxies.ProxyPools.Set] | Use this config setting to set the Proxy server configuration for your agent.
|
Increase Data Count | IsIncreaseDataCount=true/false | [FindCapture("ColumnName").Set] IsIncreaseDataCount=true | This config is used in special cases where a single column (for example PROD_NAME) is captured at three different tables/nodes then FindCapture will only be applicable to the first instance while the second and third instance will be unaffected. |
Email Notifications | [EmailNotification.Set] | Use this config setting to enable and configure Email Notifications available under the Agent Settings menu. Email Notifications can be sent when an agent completes a run and certain specified conditions are met. | |
XPATH | [Agent.FindOrIgnore("ColumnName").Selection.SelectionPaths.Get(0).Set] | Use this config setting to add XPath for a command | |
Thumbnail | [FindCapture("ColumnName").ConvertToThumbnail.Set] | Use this config to create a thumbnail for an image | |
Multiple Config files | [include] | Use comma separated values to load and execute multiple config files from two or more different paths in the [include] section. |