Data Input Scripts

Data Input scripts can be used to generate data for data provider commands. Data provider commands include the following types of commands:

Script

A Data Input script can be added to a command by selecting the data provider Script from the Data configuration tab:

The following example generates a list of URLs that could be used in an Agent command to provide start URLs:

C#
Python

using System;
using System.Collections.Generic;
using System.Data;
using Sequentum.ContentGrabber.Api;
public class Script
{
public static CustomDataStorage ProvideData(DataProviderArguments args)
{
     List<string> urls = new List<string>();
     for(int i=1; i<100; i++)
     {
         urls.Add("http://www.domain.com/page.php?ID=" + i.ToString());
     }
     return new CustomDataStorage(urls.ToArray());
}
}

import clr
from Sequentum.ContentGrabber.Api import *

def ProvideData(args):
cds = CustomDataStorage()
cds.AddStringColumn('URLs')
for i in range(1,100,1):
cds.AddRowValues("("http://www.domain.com/page.php?ID="+str(i))
return cds

The C# script must have a static method with the following signature:

public static CustomDataStorage ProvideData(DataProviderArguments args)

The function must return a CustomDataStorage that contains the input data. The CustomDataStorage can have one or more data columns, and all columns will be available to the data provider command that is using the script.

If you want Web Selection to be available in your script please check the checkbox "Use web selection in script". This option will make sure that whatever you have selected on the web page that will be available in the script. You can use the selected web content value within the script using property "Content" of the class DataProviderArguments using the object args.

To use Web Selection in your script select a web content on the downloaded page as shown below.

Now select the type of Web Content. This can be one of the "HTML", "Clean HTML", "Styled HTML", "Inner HTML", "Text", "Formatted Text", "Tag Text", "URL". There are other options available against this option.

After selecting Web Element on the Web Page edit the script and use the selected value within the script as shown below. The "agrs.Content" returns the selected value in the script.

C#
Python

using System;
using System.Collections.Generic;
using System.Data;
using Sequentum.ContentGrabber.Api;
public class Script
{
public static CustomDataStorage ProvideData(DataProviderArguments args)
{
DateTime dt = Convert.ToDateTime(args.Content);
CustomDataStorage cds = new CustomDataStorage(dt);
return cds;
}
}

import clr
import datetime
from Sequentum.ContentGrabber.Api import *

def ProvideData(args):
dt = datetime.datetime.strptime(args.Content,'%B %Y')
return CustomDataStorage(str(dt.strftime("%m/%d/%Y %H:%M:%S")))

An instance of the DataProviderArguments class is provided by Sequentum Enterprise and has the following functions and properties:

Property or Function	Description
Agent Agent	The current agent.
ScriptUtils ScriptUtilities	A script utility class with helper methods. See Script Utilities for more information.
Command Command	The current agent command being executed.
IContainer ParentContainer	The parent container command of the current command.
IConnection DatabaseConnection	The current internal database connection used by the agent. This connection is already open and should not be closed by your script.
IHtmlNode HtmlNode	The extracted HTML node.
IInternalDataRow DataRow	The current internal data row containing the data that has been extracted so far in the current container command.
bool IsDebug	True if the agent is running in debug mode.
bool IsSchemaOnly	If true, only the data schema is required, so you can optimize processing by only returning the data schema with no data.
IInputData InputDataCache	All input data available to the current command.
bool IsParentCommandMissingSelectionOrData	True if the parent container's selection was not found, or is iterating through an empty data list.
bool IsParentCommandActionError	True if the parent action command encountered an error while executing the action command.
BrowserResponse LastErrorResponse	The last error response from the web browser. This will contain any errors messages when loading a new web page.
bool HasDoneAction	True if the parent action command triggered an action in the web browser.
string Content	An extracted attribute from the selected web element.
IHtmlDocument HtmlDocument	The current HTML document.
void WriteDebug(string debugMessage, DebugMessageType messageType = DebugMessageType.Information)	Writes log information to the agent log. This method has no effect if agent logging is disabled, or if called during design time.
void WriteDebug(string debugMessage, bool showMessageInDesignMode, DebugMessageType messageType = DebugMessageType.Information)	Writes log information to the agent log. This method has no effect if agent logging is disabled, or if called during design time.
void Notify(bool alwaysNotify)	Triggers notification at the end of an agent run. If alwaysNotify is set to false, this method only triggers a notification if the agent has been configured to send notifications on critical errors.
void Notify(string message, bool alwaysNotify)	Triggers notification at the end of an agent run, and adds the message to the notification email. If alwaysNotify is set to false, this method only triggers a notification if the agent has been configured to send notifications on critical errors.
GlobalDataDictionary GlobalData	Global data dictionary that can be used to store data that needs to be available in all scripts and after agent restarts. Input Parameters are also stored in this dictionary.
IConnection GetDatabaseConnection(string connectionName)	Returns the specified database connection. The database connection must have been previously defined for the agent or be a shared connection for all agents on the computer. Your script is responsible for opening and closing the connection by calling the OpenDatabase and CloseDatabase methods.
IInputDataRow GetInputData()	If the current command is a data provider, the data for that command is returned. Otherwise, this function searches the command's parents and returns the first found input data.
IInputDataRow GetInputData(Command command)	If the specified command is a data provider, the data for that command is returned. Otherwise, this function searches the command's parents and returns the first found input data.
IInputDataRow GetInputData(string commandName)	If the specified command is a data provider, the data for that command is returned. Otherwise, this function searches the command's parents and returns the first found input data.
IInputDataRow GetInputData(Guid commandId)	If the specified command is a data provider, the data for that command is returned. Otherwise, the function throws an error.
IRuntimeData RuntimeData	Provides access to the internal database.
void WriteInfo(string debugMessage, params object[] pars)	Writes log information to the agent log. This method has no effect if agent logging is disabled, or if called during design time. debugMessage - The log message. pars - A variable number of parameters used in the debug message.
void WriteError(string debugMessage, params object[] pars)	Writes log error to the agent log. This method has no effect if agent logging is disabled, or if called during design time. debugMessage - The log message. pars - A variable number of parameters used in the debug message.
void WriteWarning(string debugMessage, params object[] pars)	Writes log warning to the agent log. This method has no effect if agent logging is disabled, or if called during design time. debugMessage - The log message. pars - A variable number of parameters used in the debug message.
void SetPageLoadError(string errorMessage)	Logs a page load error and mark the page for retry if the agent errors are later retried. errorMessage - The error message to log.
void AddNotificationMessage(string message)	Adds a message to the notification message, but does not trigger a notification. message - The message to add to the notification email.
CustomDataStorage GetOrAddDataStorage(string storageName, string key, CustomDataColumns columns)	Returns a custom database table storage that can be used to store persistent data while running an agent. A new storage is created if it does not already exist. The storage is created in the internal database. storageName - The name of the storage. key - A value identifying a sub-storage. columns - Data field definitions for the storage.
ContainerDataDictionary ContainerData	Container data dictionary that can be used to store data that needs to be available in the current container command and all sub-commands of the current container.

Simple Script

Similar to the Script the Simple Script option enables you to you use the script as the input data source. The only difference is that Simple Script returns an array of string instead of CustomDataStorage as in case of Script.

A Data Input script can be added to a command by selecting the data provider Simple Script from the Data configuration tab:

Here is the sample code using the Simple Script option

C#
Python

using System;
using System.Collections.Generic;
using System.Data;
using Sequentum.ContentGrabber.Api;
public class Script
{
public static string[] ProvideSimpleData(SimpleDataProviderArguments args)
{
string[] months2018 = new string[] {"Jan 2018", "Feb 2018", "Mar 2018", "Apr 2018",
"May 2018", "Jun 2018", "Jul 2018",
"Aug 2018", "Aug 2018", "Sep 2018",
"Oct 2018", "Nov 2018", "Dec 2018"};

return months2018;

}
}

import clr
from Sequentum.ContentGrabber.Api import *
def ProvideSimpleData(args):
months2018 = ["Jan 2018", "Feb 2018", "Mar 2018", "Apr 2018",
"May 2018", "Jun 2018", "Jul 2018",
"Aug 2018", "Aug 2018", "Sep 2018",
"Oct 2018", "Nov 2018", "Dec 2018"]
return months2018

TheSimpleDataProviderArguments object has the exact same properties and functions as the instance of the DataProviderArguments object.

For more information on List of Data Providers and Data Input Script, you can follow our article

Using Data Input