1. Sequentum
  2. Sequentum Enterprise Manual
  3. Agent Commands

Agent Config Files

Agent config files can be used to enforce specific configuration for a single agent or all agents in a folder. This makes it possible to alter an agent's behavior by simply editing a text file.

The configuration files must be named config.txt and are INI files that have a number of sections followed by properties for each section.

To apply a configuration file to multiple agents, the agents must have the same Shared Folder and the config file must be placed in that Shared Folder. By default, the config file will be applied to all agents with the same Shared Folder, but the config file can specify exactly which agents it should apply to. This is done by adding a Config section with an agent property that specifies a comma-separated list of agent names. The Config section must be the first section in a config file. You can see an example of this below in Example 2.

To apply the configuration to a single agent, place the config file in the agent's folder.

Configuration files correspond tightly to the API. A section in a configuration file specifies the instance of a class to modify, and the properties specify the class properties to modify.

Configuration files can only set agent properties, and cannot add new class instances. However, you can call methods on instances that may allow you to add new class instances.

A configuration file can contain an Include section that specifies paths to other configuration files that will be executed at the point where the include section is encountered.

Configuration files can contain a Parameters section, which is an easy way of setting input parameters.

Special Content Sections can be used to define long content, such as scripts, which can then be assigned to properties. A Content Section must start with content and have the format [content.name] where the name can be any unique name. To assign a Content Section to a property use the format property = [content.name].

NOTE

The [Include] section of the config file settings can take multiple comma-separated values and both "path" and "paths" keywords can be used for adding multiple config files.

[Include]
path|paths =c:\\test\config1.txt,c:\\test\config2.txt

Example 1 

[Include]
path|paths=c:\\test\config.txt

[Parameters]
test=working 

[Set]
ObeyRobotsRules=Always 

[InternalDatabase.Set]
DatabaseType=SqlServer

[InternalDatabase.DatabaseReference.Set]
DatabaseConnectionName=myconnection

The above configuration file does the following:

  1. Loads and executes the config file c:\test\config.txt
  2. Sets the input parameter test to working.
  3. Sets the agent property ObeyRobotsRules to Always.
  4. Sets the agent property InternalDatabase.DatabaseType to SqlServer.
  5. Sets the agent property InternalDatabase.DatabaseReference.DatabaseConnectionName to myconnection.

Example 2

[config]
agents=test1,test2

[FindOrAdd("longitude", "CalculatedValue").Set]
IsAllowEmptyValue = true
DataType=Float
ChangeTracking=Exclude

The above configuration file does the following:

  1. Restricts the configuration file to the agents test1 and test2
  2. Finds a command "longitude". If the command does not exist, a new Calculated Value command is added. 
  3. Sets the command property IsAllowEmptyValue to true.
  4. Sets the command property DataType to Float.
  5. Sets the command property ChangeTracking to Exclude.

Example 3

[Set]
Environment = Custom
CustomEnvironment = [content.env]

[content.env]
PATH = C:\Program Files\Python37
TEMP = %SystemRoot%\TEMP

The above configuration file does the following:

  1. Sets the agent property Environment to Custom.
  2. Sets the agent property CustomEnvironment to a content section that's defined separately.
  3. Adds a content section that specifies two environment variables.

Agent Configuration Window

You can also use Sequentum Enterprise Desktop's Toolbar menu option 'Agent Configuration' to write the agent configuration.

AgentConfiguration.gif

Using Agent Configuration

The new data export pipeline, Named Connections, and Remote Storage can also be added using the config file settings, refer to Adding Data Export Pipeline Using Config File section for detailed reference.

Use the below config file setting for the old data  Export Targets Configuration (Compatibility Mode)

[Agent.Set]
UseCompatibilityExport=Compatibility

Config File Reference

Sequentum Enterprise  Property Settings Options Config File Settings Examples Description
Internal Database DatabaseType=SqlServer/PostgreSql/SqLite/MySql

[InternalDatabase.Set]

DatabaseType=SqlServer/PostgreSql/SqLite/MySql
IsAutomaticOverwrite=True

IsEmbedFiles=false
UseDefaultDownloadPath=false
DownloadDirectoryName=C:\Data\Export
DownloadedFileExistHandling=DiscardNewFile/Overwrite/RandomName
IsDeleteDownloadedFiles=true

[InternalDatabase.DatabaseReference.Set]
DatabaseConnectionName=<MY_CONNECTION_NAME>

Use this config file setting
to set the agent property
InternalDatabase.DatabaseType
to SqlServer or PostgreSql or MySql or SQLite
and also it will set the agent property InternalDatabase.DatabaseReference
.DatabaseConnectionName to <MY_CONNECTION_NAME>(add your preferred already created connection name in here.)

Separate Export Database ExportDatabaseType=SqlServer/PostgreSql/MySql

[InternalDatabase.set]

IsUseSeparateExportDatabase=false ExportDatabaseType=SqlServer/PostgreSql/MySql [InternalDatabase.ExportDatabaseReference.Set] DatabaseConnectionName=<Seperate_Export_Database_Name>

Use this config setting to add a separate export database for the agent.

Input Parameters name/value pairs to use as agent input parameters.

[Parameters]

run_id_ =new

production_run_=true

Use this config to set the input parameters in your agent like run_id_ which will generate an increasing number every time you run your agent and can be used as an output column in the agent to identify the run history. Similarly, production_run_ can be used to set the agent to always run in production mode.

Database Export Target ExportDestination=DatabaseExport

[ExportDestination.DatabaseExport.Set]

IsEnabled=true

DatabaseConnectionName=<MY_CONNECTION_NAME> IsKeepHistoricalData=true

IsSeparateHistoricalData=true

IsSharedDatabaseTables=true

AllowDropSchema=Newer

Use this config to set Export Target to DatabaseExport.

Export Target ExportDestination=CsvExport/ParquetExport

[ExportDestination.CsvExport.Set] (for csv export) [ExportDestination.ParquetExport.Set] (for parquet export) IsEnabled=True

IsTimeStampFile=true

Use this config to set Export Target csv or parquet
Script Export Target ExportDestination=Script

[ExportDestination.ScriptExport.Set]

IsEnabled=true
ScriptLanguage=Csharp

CsharpScript=[Content.ExportData]

[Content.ExportData]

using System; using System.IO;

using System.Text.RegularExpressions;

using System.Collections.Generic; using Sequentum.ContentGrabber.Api;

using Sequentum.ContentGrabber.Commands;

public class Script

{

public static bool ExportData(DataExportArguments args)

{

var dt = DateTime.UtcNow;

var dtFormatted = dt.ToString("yyyyMMdd");

var exportedFiles = args.ExportToCsv(null, true);

var csvFiles = exportedFiles.DataFiles;

var newExportedFilesAll = new List<string>();

var newExportedFiles = new List<string>();

foreach (var file in csvFiles)

{

var filename = Path.GetFileName(file);

var filenameWithoutExt = Path.GetFileNameWithoutExtension(file);

var ext = Path.GetExtension(file);

var newFile = Regex.Replace(file, filename, string.Format("{0}-{1}{2}", filenameWithoutExt, dtFormatted, ext));

if (File.Exists(newFile)) File.Delete(newFile);

File.Move(file, newFile);

newExportedFilesAll.Add(newFile);

}

exportedFiles.DataFiles = newExportedFilesAll;

exportedFiles.DataFiles = newExportedFiles; return true;

}

}

Use this config to set the Export Target to Script
CsvExport FilenameTransformation using Csharp script  

[ExportDestination.CsvExport.Set]

IsEnabled=True

IsTimeStampFile=false [ExportDestination.CsvExport.FilenameTransformationScript.Set] IsEnabled=true

ScriptLanguage=Csharp

CsharpScript=[Content.TransformFilename] [Content.TransformFilename]

using System;

using Sequentum.ContentGrabber.Api;

public class Script

//See help for a definition of FilenameTransformationArguments.

{ 

public static string TransformFilename(FilenameTransformationArguments args)

{

//Place your transformation code here. //This example just returns the input data var dt = DateTime.UtcNow;

var dtFormatted = dt.ToString("yyyyMMdd");

var outputFileName=args.FileName+"-"+dtFormatted+".csv";

return outputFileName;

}

}

use this config setting to transform the output/export file name using c# script in the preferred name and format.
Data Distribution to AWS S3  

[ExportDestination.AwsS3BucketDelivery.Set]

IsEnabled = True

BucketName=<s3bucketName>

CredentialsFilePath=<S3_credentials_File_Path>

Use this config to deliver the data on S3 Browser.
Data Distribution to Email  

[ExportDestination.EmailDelivery.Set]

IsEmailDeliveryEnabled=True 
Server=mail.authsmtp.com
Port= 25
Username= <username>
Password= <password>
FromEmailAddress=<from_email_address>
RecipientEmailAddresses= <recipent_email_addresses> BccEmailAddresses=<bcc_email_address>
IsDeliverDataFiles=True
IsDeliverExtractedFiles=True
IsCompressFiles=False

 

 Use this config to deliver the data on Email.
Data Distribution to FTP  

[ExportDestination.FtpDelivery.Set]

IsFtpDeliveryEnabled=true

Server=1.0.0.0 Port=21

Username=<USERNAME>

Password=<PASSWROD>

IsDeliverDataFiles=true

IsCompressFiles=true

RemoteDirectory=test

Use this config to deliver the data on FTP server.
FTP FileNameTransformation for Delivering Compressed file  

[ExportDestination.FtpDelivery.FilesTransformationScript.Set] IsEnabled=True

CsharpScript=[Content.CompressedFilename]

[Content.CompressedFilename]

using System;

using Sequentum.ContentGrabber.Api;

public class Script

{ 

public static ExportedFiles TransformFiles(DataDeliveryTransformationArguments args)

{

var dtFormatted = DateTime.Now.ToString("yyyy.MM.dd_HH.mm.ss"); args.Agent.ExportDestination.FtpDelivery.CompressedFilename= dtFormatted +"_"+args.Agent._name+ ".zip"; return args.ExportedFiles; } }

Use this config setting to change the compressed file name while delivering a compressed file to the FTP server.
Success Criteria SuccessCriteria=Min DataCount /MaxPageError/Min export rows

[SuccessCriteria.Set]

IsDataCountCriteriaEnabled=true

IsDataCountCriteriaPercentage=true

DataCountCriteria=90

IsPageErrorCriteriaEnabled=true

PageErrorCriteria=0

IsExportRowCountCriteriaEnabled=true

ExportRowCriteria=1

Use this config to set the Success Criteria in your agent which will define the criteria for a successful data extraction. A success criteria can be defined to tell an agent when it should consider an agent run successful.
Session

SessionSupport=Multiple

Performance

Single

[Agent.Set]

SessionSupport=Multiple

Use this config to configure an agent to support sessions
Robot rules  ObeyRobotsRules=Always/Never

 [Agent.Set]

ObeyRobotsRules=Always

 Use this config to configure an agent to obey robot rules for the target website.
 Max Run Time MaxRunTime=hh:mi:ss MaxRunTime=d.hh:mi:ss

 [Agent.Set]

MaxRunTime=05:00:00

MaxRunTime=2.00:00:00

 This config is used to Limit the max execution time In the given example first is limit to a maximum of 5 hours and for the second example, it is 2 days.
 Max Active Parsers  

 [Agent.Set]

MaxActiveParsers=7

 This config is used to set max active parsers.
 Empty Export Row Handling

 EmptyExportRowHandling=Remove

ValidationErrorHandling

None

 [Agent.Set]

EmptyExportRowHandling=Remove

 Use this config to set export empty row handling.
 Export Validation Error Handling

 ValidationErrorHandling=RemoveWithError

Remove

None

 [Agent.Set]

ValidationErrorHandling=RemoveWithError

 Use this config to set remove the row with error handling.
 Export Name  

 [Agent.Set]

ExportTableName=<FileName>

 Use this config to set Export File Name
 Export ID Name  

 [Agent.Set]

ExportIdColumnName=<uniqueIdName>

 Use this config to set Export ID Name
 Export Column (Multiple export columns having same name)

 IsAllowEmptyValue = false/true

IsKeyColumn=true/false

ChangeTracking=Exclude/include

Validation=Export/Runtime Datatype=integer

ShortText

Decimal

DateTime

LongText

BigInteger

Float

Boolean

 [FindAll("ColumnName").Set]

IsAllowEmptyValue = false

IsKeyColumn=true

ChangeTracking=Exclude

Validation=Export Datatype=integer

 Use this config to set data validations on all output columns having the same name.
 Export Column

 IsAllowEmptyValue = false/true

IsKeyColumn=true/false

ChangeTracking=Exclude/include

Validation=Export/Runtime Datatype=integer

ShortText

Decimal

DateTime

LongText

BigInteger

Float

Boolean

 [FindCapture("ColumnName").Set]

IsAllowEmptyValue = false

IsKeyColumn=true

ChangeTracking=Exclude

Validation=Export Datatype=integer

This config is used in special cases where a single column (for example PRODUCT_NAME) is captured at three different tables/nodes then FindCapture will only be applicable to the first instance while second and third instance will be unaffected.
Increase Data Count(Multiple export columns having the same name)  IsIncreaseDataCount=true/false

[FindAll("ColumnName").Set]

IsIncreaseDataCount=true

Use this config to set the Increase data count on all the output columns having the same name which will show the total number of output results.
 Logging  DefaultLogLevel=Low/High/Medium

 [Agent.AgentLogging.Set]

DefaultLogLevel=Low

IsDefaultLogTofile=true

Use this config to set Log Level.
Centralized Database Logging LogLevel = 1/2/3

[AgentLogging.CentralizedLogging.Set]

ConnectionName = SQLServerNamedConnection

EnableCentralizedLogging = true

TableName = AgentLogging

LogLevel = 1

RetentionDays=30

Environment=Prod

Use this config to enable centralized logging for an agent.

The ConnectionName needs to be created through the Application Settings → Managed Named Connections.
 Error Handling

 ErrorHandling=RetryCommand

 ExitCommand

 StopAgent

 RestartAgent

 [Agent.Action.Set]

ErrorHandling=RetryCommand

ErrorRetryCount=7

ErrorRetryDelay=10000

ErrorRetryClearOnSuccess=false

 Use this config for error handling.
 Data List- Data Provider

 Provider Type=Simple CSV

Excel

Parquet

Selection Script

Simple Script

Number Range

Date Range

SQL Server

MySQL

Oracle

OleDB

ODBC

[Find("Data List").DataProvider.Set]

ProviderType=SqlServer

[Find("Data List").DataProvider.DatabaseProvider.Set] DatabaseConnectionName=sequentum_data

Sql=Select * from yellowpagesTest;

 Use this config setting to set the SQL SERVER database connection and SQL for the input data provider in the Data List Command

 Proxy Configuration

Proxy Pools=Agent Control Center | Proxy API | No Proxy | Application

 

 [Proxies.SetAccPool().Set]

ProxyAddress = https://acc-test.sequentum.com/sequentum/testproxy1

IsRotateAutomatically=true

RotationInterval= 200

[Proxies.AddProxyApiPool().Set]

ProxyAddress=https://acc-test.sequentum.com/Sequentum/testproxy2

[Proxies.ProxyPools.Set]
IsCyclePools=true
IsRotatePoolsOnError=true
RotateErrorCount=2
RotateErrorSetSize=1

Use the below config setting to set NoProxy 

[Agent.Proxies.Set]
ProxyType=NoProxy

Use the below config setting to set Application Proxy 

[Agent.Proxies.Set]
ProxyType=Application

 Use this config setting to set the Proxy server configuration for your agent.
Increase Data Count IsIncreaseDataCount=true/false [FindCapture("ColumnName").Set] IsIncreaseDataCount=true This config is used in special cases where a single column (for example PROD_NAME) is captured at three different tables/nodes then FindCapture will only be applicable to the first instance while the second and third instance will be unaffected.
Email Notifications  

[EmailNotification.Set]
IsNotificationEnabled=True
Server=mail.authsmtp.com
Port=25
Username=<USERNAME>
Password=<PASSWORD>
FromEmailAddress=<from_email_address>
RecipientEmailAddresses=<recipent_email_addresses>
SuccessEmailAddresses=<success_email_address>
IsNotifyAlways=True
IsNotifyOnCriticalError=True
IsNotifyOnSuccessfullExtrcation=True
IsNotifyOnUnsuccessfullExtrcation=True
IsUseDefaultSettings=False
IsUseDefaultEmailAddresses=False
Enabled=Always|Never|Environment
Environment=Prod|Dev|QA

Note: Please note that you need to add the Enabled parameter either at the end or after the IsNotificationEnabled parameter only.

Use this config setting to enable and configure Email Notifications available under the Agent Settings menu. Email Notifications can be sent when an agent completes a run and certain specified conditions are met.

XPATH   [Agent.FindOrIgnore("ColumnName").Selection.SelectionPaths.Get(0).Set]
Xpath = html/div[0] 		Use this config setting to add XPath for a command
Thumbnail   [FindCapture("ColumnName").ConvertToThumbnail.Set]
IsConvertToThumbnail = true
MaximumWidth = 90
MaximumHeight = 90 		Use this config to create a thumbnail for an image
Multiple Config files   [include]
paths=C:\Data\AgentControlCenter\Sequentum\ExportConfig Agents\AllExportTargets\Shared\config1.txt,
C:\Data\AgentControlCenter\Sequentum\ExportConfig Agents\AllExportTargets\Shared\config2.txt 		Use comma separated values to load and execute multiple config files from two or more different paths in the [include] section.

Articles in this section

Was this article helpful?
0 out of 0 found this helpful

Comments

0 comments

Please sign in to leave a comment.