Skip to main content

Crawl Website

The Crawl Website command Crawls a website by following all links found on the website. The number of links can be limited by domain name, link depth or number of URLs. The command selects all links by default, but the web selection can be modified, so it only selects specific links.

The Crawl Website command is often used to scan a website for certain information, such as emails, addresses, and phone numbers.

CrawlWebsite.png

Crawl Website configuration screen.

The Crawl Website command provides the data fields URL and Depth, which can be captured with a Data Value command.

The Crawl Website command uses an HTML Parser, not a Dynamic Browser, by default. This can be changed in the Action tab, but an HTML Parser is recommended to ensure good performance. 

Command Properties

Action

URL Action: The action to perform on the selected web element or URL.

  • Action Type: Specifies the action to perform when clicking on a web element.

  • Activities: Specifies how this action should wait for the browser activities to complete.

    • Wait for Content: The default value is set to Optional. Waits for web selections and URLs specified by this property value required, optional, no wait.

    • Wait for External Sub-Page Load: The default value is set to No Wait or Parse. Specifies if the command should wait for one or more page loads to occur that originate from different domains than the main page.

    • Wait for External Sub-Page Scripts: The default value is set to False. Specifies if the command should wait for one or more async scripts loads to complete on subpages from different domains than the main page.

    • Wait for Internal Sub-Page AJAX: The default value is set to True. Specifies if the command should wait for one or more AJAX requests to complete on sub-pages from the same domain as the main page.

    • Wait for Internal Sub-Page Load: The default value is set to Optional. Specifies if the command should wait for one or more page loads to occur in browser frames. Will only wait for pages from the same domain as the main page.

    • Wait for Internal Sub-Page Scripts: The default value is set to False. Specifies if the command should wait for one or more async scripts loads to complete on subpages from the same domain as the main page.

    • Wait for Main Page AJAX: The default value is set to True. Specifies if the command should wait for one or more AJAX requests to complete on the main page in the browser.

    • Wait for Main Page Load: The default value is set to Required. Specifies if the command should wait for one or more full page loads to occur in the browser.

  • Add Force Refresh Header: Adds an “If-Modified-Since” header to the web request to make sure the web page is not retrieved from cache. The default value is set to false.

  • Block Known Ad Servers: The web page will not load content from known ad servers, such as “ad.doubleclick.net”. This speeds up processing slightly.

  • Block Popup: Default value is set to False. This property set to True allows the user to blocks any popup opened by the action.

  • Break HTML Area: Specifies whether the action breaks an HTML area. Actions that load a new page always break HTML areas.

  • Browser Mode: This property specifies the different types of browser that can be used to run an agent·

    • Dynamic Browser: The browser functions as a standard web browser, and it download images and execute JavaScript.

    • HTML Parser: The HTML Parser doesn’t execute JavaScript and does not load frames, so it is faster and more reliable than a Dynamic Browser. However, the parser doesn’t work on websites that rely on JavaScript, and the parser may also be unable to some web forms (even when they don’t rely on JavaScript).

    • J SON Parser: This property does not start a new browser. Instead, it parses J SON content returned by a web server, and lets you easily extract content elements from the J SON content.

    • XML Parser: This property does not start a new browser. Instead, it parses XML content returned by a web server, and lets you easily extract content elements from the XML content.

  • Capture Requests(Regex): A regular expression matching URLs which requests should be captured and made available for scripting. Multiple regular expressions separated by line breaks can be specified.

  • Clear Storage: This property allows users to clear the website’s cookies and local storage. The Default value indicates that no storage will be cleared. This property when set, works on the commands such as Navigate_URL, Navigate_Link and not on the overall agent.

    • Default: A value of default will delete all cookies and local storage if using in-memory cache, but only site cookies and local storage is using a persistent cache.

    • None: Indicates that will not delete cookies and storage.

    • All Cookies & Storage: It will delete all cookies and local storage if using in-memory cache, but only site cookies and local storage is using a persistent cache.

    • Site Cookies & Storage: It will delete only website cookies and local storage if using in-memory cache, but only site cookies and local storage is using a persistent cache.

    • All Cookies: It will delete all cookies only.

    • Site Cookies: It will delete website cookies only.

    • Site Session Cookies: It will delete site session cookies only.

  • Close Browser After Use: Default value is set to False. This property set to True allows user to close any browser after use. This option is only applicable if the action opens a new browser.
    Command Returning Max Scrolls: Limit the number of scrolls to a number captured by the specified command. This option can be useful when a page can scroll indefinitely.

  • Custom Request Headers: Custom headers are sent with all requests made by a browser. Headers must be sent on each new browser.

  • Detect Encoding: This property specifies how a new parser should detect the encoding of content received from a web server. If the option is set to Default, this action uses the same detection method as the parent parser. If there’s no parent action, The default detection method is “Content & Server”. You must reopen the browser window or the agent for this change to take effect.

  • Discover Action: This property when set to True, configures action properties automatically when the command is first executed. Default value is False.

  • Editor Action: Specifies the web element or URL to use when performing the action in the design browser. Default value is “Default” which is indicated by “Use Specific URL” value False. To use the Specific URL, we set the “Use Specific URL” value True which allows users to use a specific URL.

    • URL: Specifies the URL to load.

    • Use Specific URL: Specifies whether to load a direct URL. This value is set to False.

  • Error Handling: This property specifies how the agent should react when an error occurs while executing the command action. The default reaction is to exit the command. Use the option “No Error Handling” if you want the agent to continue executing sub-commands after an error . You can handle the error in sub-commands by using the script parameter “IsParentActionError”. This property specifies the different ways of Error Handling.

    • Exit Command: The agent command will exit the action command and continue executing the next command. The agent will skip all the sub-commands of the action command.

    • Retry Command: The agent command will retry the action command a specified number of times, and if the action command does not succeed, it will skip all sub-commands of the action command and continue executing the next command. Set the property Retry Count to specify the number of retries. If Retry Count is set to zero, the agent will keep retrying the command indefinitely.

    • Retry With No Error Handling: The agent command will retry the specified number of times, but then continue with no error handling.

    • Restart and Resume Agent: The agent command will restart and resume where it left off. This option is useful if an error puts the website into a state where the agent cannot continue.

    • Restart Agent and Retry Command : The agent command will retry the action command a specified number of times and if the action command does not succeed, then it will Restart the Agent.

    • Stop Agent: The agent will stop.

  • Error Retry Clear On Success: This property clears the counter if the action succeeds. The default value is set to True.

  • Error Retry Count: This property allows users to specify the number of times the agent should retry the command when an error occurs while executing the command action. This property is only applicable if “Error Handling ” is set to “Retry Command”. The default value is set to 5, which indicates that the agent will retry the command 5 times.

  • Error Retry Delay: This property allows users to specify the number of milliseconds the agent will delay a retry. This property is only applicable if “Error Handling ” is set to “Retry Command”. The default value is set to 5000, which indicates that the agent will delay a retry 5000 milliseconds.

  • Error Retry Proxy Rotation: The default value is set to “Keep and Rotate”. This property allows users to specify if the proxy should rotate before retrying the action and if the current proxy should be removed. This property is only applicable if “Error Handling ” is set to “Retry Command”.

    • None: Indicates that will not Rotate proxy before or after retrying the action.

    • Remove and Rotate: Indicates that current proxy will be removed and Rotate proxy before retrying the action.

    • Remove and Rotate After Retry: Indicates that current proxy will be removed and Rotate proxy after retrying the action.

    • Keep and Rotate: Indicates that current proxy will remain same and Rotate proxy before retrying the action.

  • Fixed Encoding: This property specifies the encoding when “Detect Encoding” is set to “Fixed”. You must reopen the browser window or the agent for this change to take effect.
    Ignore Error Codes: Some websites return incorrect status codes, so this property allows users to ignore status codes by default unless the web server is not returning any content.

  • JSON Transformation: Regular expression used to transform JSON content loaded with a static parser.

  • Limit Number Of Scrolls: This property set to true allows users to limit the number of scrolls to a specified number. This property can be useful when a page can scroll indefinitely. The default value is set to false.

  • Max AJAX Calls: Stops waiting for AJAX after the specified number of AJAX Calls. The default value is set to 10.

  • Maximum Number Of Scrolls: This property allows users to limit the number of scrolls to a specified number. This property can be useful when a page can scroll indefinitely. The default value is set to “50”. This property is only applicable if “Limit Number Of Scrolls ” is set to “True”.

  • Never Open New Browser: This property set to True indicates never configure an action to open a web page in a new web browser when discovering an action. Some webpage may not function correctly when opening a link in a new browser. The default value is set to False.

  • No Parse: The web pages are not parsed if existing parsed pages exist. The default value is set to “False”.

  • Page Not Found Handling: Specifies the action to take if a web page was not found. The default is to do nothing and let normal error handling deal with the error.

    • Continue: If a web page not found, it continues executes to the agent.

    • ExitCommand: If a web page not found, it will exit from the agent.

  • Redirect First Request: This property allows users to redirect the first request to a new browser window when Target Browser is set to “New”, even if the first request is coming from a frame within the current browser window. If this property is set to “False”, requests from frames within the current browser window will not be redirected.

  • Rotate Proxies: The property set to “True” allows users to rotate the proxy before executing the action. The default value is set to False.

  • Scroll Steps: The number of pixels the page will be scrolled in each step. The default value of “o” will scroll the page all the way to the bottom and all the way to the right in each step.

  • Scroll Until End of Page: This property set to True allows users to scroll to the end of the web page after an action. Scrolls repeatedly until unable to scroll any further, and wait for AJAX calls to complete if scrolling trigger AJAX calls. The default value is set to “False”.

  • Separate Parser Proxies: Creates a separate parser proxy loop if the action opens in a parser. The default value is set to “False”.

  • Target Browser: Specifies the web browser where a new web page should be loaded. These are the different options which can be used by a user:

    • New: Default value is set to New which specifies to load a web page in a new browser window.

    • Current: This specifies to load a web page in the same browser window.

    • Parent: This specifies to load a web page in the parent browser window.

    • Popup: This specifies to load a web page in a popup window.

  • Timeouts: Specifies timeout values for the action. Timeouts specified in activities override these values.

    • Ajax Completed: The default number of milliseconds to wait for an AJAX call to complete.

    • Ajax Content Render Delay: The default number of milliseconds to wait for ajax loaded content to render on a web page.

    • Ajax Content Render Delay After Scroll: The default number of milliseconds to wait for ajax loaded content to render on a web page after triggering a scroll. The command will stop waiting as soon as it can scroll the page further down. This property is different from “Ajax Content Render Delay” which always waits a fixed amount of time. A fixed timeout is slower but is required on some websites. 

    • Asynchronous Completed: The default number of milliseconds to wait for an asynchronous action to complete.

    • Discover First Activity: The default number of milliseconds to wait for the first activity when discovering new activities.

    • Discover First URL  Activity: The default number of milliseconds to wait for the first URL to start loading

    • Discover Next Activity: The default number of milliseconds to wait for the next activity when discovering new activities.

    • File Download Completed: The default number of milliseconds to wait for a file download to complete.

    • File Download Started: The default number of milliseconds to wait for a previous file download to complete to start downloading the next file.

    • Frame Completed: The default number of milliseconds to wait for frame content to complete loading. This timeout applies to internal frames, and only to external frames if the property WaitForExternalFrames is set to True.

    • JavaScript Parser Timeout: The number of milliseconds to wait for JavaScript DOM operations  to complete

    • Main Page Redirect: The number of milliseconds to wait for the main page to redirect to another page.

    • Page Completed: The default number of milliseconds to wait for a page load to complete.

    • Wait For Content Timeout: The default number of milliseconds to wait for web content to appear on a web page, or a URL to load that matches a specified  Regex.

    • Wait Times: The default timeout values are multiplied by this value. This can be a quick way to test if issues with action are caused by timeout values being too short. Default timeouts are used when discovering activities, and when scrolling a page.

  • Transformation Script: A script used to transform the URL attribute when using the URL action type.

    • C# Script: Specifies C# script.

    • Enabled: To use the Script we need to set this Property as True. The default value is set to False, which indicates that the script is disabled.

    • Library Assembly File: The name of a custom assembly file when "Use Default Library" is set to false.

    • Library Method Name: The method to execute when using the default script library.

    • Library Method Parameter: A custom parameter passed to the script library method.

    • Python Script: Specifies Python script.

    • Regex Script: Specifies Regex script.

    • Script Language: Specifies the scripting language which you want to use e.g C#, http://VB.NET , Python, Script Library, Regular Expressions.

    • Template Name: The template name of the referenced template.

    • Template Reference: Loads this script from a template when the agent is loaded.

    • Use Default Library: Uses the default script library when Script Language is set to Script Library.

    • Use Selection: The script is provided with the selected web element. The script will not be provided with the selected web element if this value is False.

    • Use Shared Library: Uses a script library that is shared among all agents.

    • VB.NET Script: Specifies VB.NET script.

Command

Command Description: A custom description for the command. Default value is Empty.

Command Transformation Script:  A script used to change command properties at run time. The default value is disabled.

Disabled: This property set to True allows user to disable the command. A disabled command will be ignored. The default value is set to False.

ID:  This property indicates the internal unique ID of the command and is always auto- generated e.g. 58c8e4ac-e4c0-48f7-a63d-77064945380b.

Increase Data Count: This property indicates the data count every time this command is processed. The default value is set to False. Set it to TRUE if you want to get the count of the number of times a specific command is executed to get the data. The data count value is increased during data extraction, so it is used to measure agent progress and basis this increased data count, the agent  decides the success criteria.

Name:  This property specifies the name of the command.

Notify On Critical Error:  A notification email is sent at the end of an agent run if the command encounters a critical error, and the agent has been configured to notify on critical errors. Critical errors include page load errors and missing required web selections. Default value is set to False.

Container

Always Execute:  Set this property value as True, if you want  the container is always executed if encountered on retry or continue. List Commands are only executed if they processes a single input. Always Execute can be used to always execute container commands when retrying, even when they have already been completed. Default Value is set to False.

Command Link:  Links to another container command where processing will continue. The targeted container command will be executed, so it’s normally best to link to a group command  that does nothing, so it’s clear what happens after the link.

Dependent Command:  The action of the dependent command will come into effect only when the agent has a supporting parent container command, in which case, it will be executed before the parent container command is processed.

Repeat While Selection is Valid:  Set this property to TRUE if you want to repeatedly process the command while the command selection is valid. Default value is set to False.

Data List

Data Missing Action: This property specifies the action to take when data is missing. 

  • Optional: This property specifies to use the optional command when data is missing.

  • Ignore Command when Data is Missing: This property specifies to ignore command when data is missing.

Data Provider: The Default value is set to Simple . This property is used to provide input data to the command and all sub-commands. You can assign a data provider to any  Data List command and can load data from the different  data sources such as Simple Provider, CSV Provider, Database Provider, Date Provider, Number Provider, Parquet Provider, Selection Provider etc.Data List  commands include the following in which we can use this property:  Agent, Data List, Navigate URL, Set Form Field.

 The value under the ‘Data Provider ’ text area get set according to what specific ‘Provider Type’  value selected.

  • CSV Data Provider: Choosing CSV for the Data Provider is similar to the Simple data provider, but uses an external CSV file. We recommend that you choose the CSV Data Provider for large CSV files, since the CSV data provider will perform much better than the Simpledata provider for large quantities of data.For a CSV provider, you can choose the value Separator and the text Encoding of the CSV file. You can place CSV files anywhere on your computer, but we recommend you place them in the default input data folder for your agent. Later, if you want to export your agent, then you can include these files along with the export. The value under the ‘CSV Provider ’ text area reflects the same value which is specified in “File Name or Path” option.

    • Encoding: The File encoding used by the CSV file. We can set encoding like utf-8, BOM, ASCII, Unicode, Unicode BOM etc. .

    • File Name Or Path: Specifies file name or path where your input csv/excel file is placed .

    • Has Header Fields: Default value is  set to True specifies whether the first data row contains column names.

    • Ignore Parse Errors: Ignores parse errors.

    • Value Separator: A character used to separate values in the CSV File. Default value is set to Comma. We can use other characters such as Tab, Semicolon, Pipe. 

  • Database Provider: Choose a Database for the Data Provider to work any of these database connections: SQL Server, Oracle, MySQL, OleDB etc. .In Sequentum Enterprise, you can share database connections among all agents on a computer.

  • Date Provider: Choosing Date Range for the Data Provider specifies the date range .

    • Add Delayed Date: Adds a second date to the data provide. The second date will be delayed with a specified number of days, weeks or months.

    • Convert to Strings: Converts the dates to strings.

    • Date Format: Format of each date in the date range when converting dates to strings.

    • Days From End Date: Number of days, weeks or months from end date when end date is set to "Days From Today", "Weeks From Today" or "Months From Today".

    • Days From Start Date: Number of days, weeks or months from start date when start date is set to "Days From Today", "Weeks From Today" or "Months From Today".

    • Delay: Specifies of the second date is delayed by days, weeks or months.

    • Delay Days: Number of days, weeks or months the second date is delayed.

    • End Date: Last date in the date range.

    • Specific End Date: Specific last date in the date range.

    • Specific Start Date: Specific first date in the date range. 

    • Start Date: First date in the date range.

    • Step: Steps between dates in the date range.

  • Design Row Index: This property specifies the index of the data row used at design time.

  • Discover More Data: Waits for asynchronous sub commands to complete if no more cached data is available. Use this option if asynchronous sub commands add more data to the data provider.

  • Hide Data in Editor: Hides the data in the Sequentum Enterprise editor. This can be used to hide sensitive data, such as passwords, in the Sequentum Enterprise editor.

  • Load More Data: Tries to load more data from the data source when no more cached data is available.

  • Number Provider: Choosing Number Range for the Data Provider specifies the number range which is used for any calculation or other purpose. To extract the Number range as a DataValue  in a specific order with specific interval we can use the Number Provider. 

    • End Number: Specifies the last number in the number range. Default value is 1000.

    • Start Number:  Specifies the first number in the number range. Default value is 1.

    • Step:  Specifies the steps between numbers in the number range. Default value is 1.

  • Parquet Provider: Specify to choose Parquet  for the Data Provider .

    • File Name Or Path: Specifies file name or path where your input parquet  file is placed . The value under the ‘Parquet Provider ’ text area reflects the same value  which is specified in “File Name or Path” option.

  • Provide Data As List: The provided data is a list of values. This can be used to set multiple values in a HTML list box.

  • Provider Type: This property specifies the type of data sources. Default value is “Simple”. You can use other data sources such as CSV, Parquet, Script, Number Range etc. .

  • Public Provider: If this is a Simple data provider.

  • Public Provider Name: The name of the data provider. The name of the command is used if this value is left empty.

  • Script Data Provider: Choose Script for the Data Provider for full customization of the agent input data. This option provides a .NET data table containing the input data, which may contain multiple data columns and rows. Sequentum Enterprise provides some standard .NET libraries which makes it easier to generate .NET data tables for a variety of input data. The value under the ‘Script Data Provider ’ text area reflects the same value  which is selected in “Script Language” option

    • C# Script: Specifies C# script.

    • Enabled: To use the Script we need to set this Property as True. Default value is set to False, which indicates that script is disabled.

    • HtmlAttribute: An Html Attribute to extract from the selected web element. This attribute will be available in the script.

    • Library Assembly File: The name of a custom assembly file when "Use Default Library" is set to false.

    • Library Method Name: The method to execute when using the default script library.

    • Library Method Parameter: A custom parameter passed to the script library method.

    • Python Script: Specifies  Python script.

    • Regex Script: Specifies Regex script.

    • Script Language: Specifies the scripting language which you want to use e.g  C#, VB.NET, Python ,Script Library, Regular Expressions.

    • Template Name: The template name of the referenced template.

    • Template Reference: Loads this script from a template when the agent is loaded.

    • Use Default Library: Uses the default script library when Script Language is set to Script Library.

    • Use Selection: The script is provided  with the selected web element. The script will not be provided with the selected web element if this value is False.

    • Use Shared Library: Uses a script library that is shared among all agents.

    • VB.NET Script: Specifies VB.NET script.

  • Selection Provider: Choose Selection for the Data Provider . The value under the ‘Selection Provider ’ text area reflects the same value  which is specified in “HTML Attributes” option.

    • HTML Attributes: Specifies number of HTML Attributes, which is reflected as “HTML Attribute#1”, “HTML Attribute#2” etc..

    • Relative Xpath: Specifies the relative Xpath.

  • Test Set Count: The Test Set Count refers to the number of data rows to be utilized for testing when working with a test set. By default, the Test Set Count is set to 1.

    This setting allows users to control the size or quantity of data rows that are considered during the testing process. By adjusting the Test Set Count, users can specify how many data rows from the test set should be included in the testing phase. This flexibility enables targeted and efficient testing based on specific requirements or sample sizes.

    The Test Set functionality is designed to operate exclusively in the QA and Dev environments. It is specifically programmed to automatically disable itself when running in the Prod environment.

    This approach ensures that the Test Set is not active in the live production environment where real users interact with the system. This helps maintain the stability and reliability of the production environment, ensuring that it operates smoothly for end users.

  • Test Set Start Value: The Test Set Start Value represents the index of the initial data row to be used for testing when utilizing a test set. By default, the Test Set Start Value is set to 1.

    This setting enables users to specify the starting point within the test set from which the testing process should begin. By adjusting the Test Set Start Value, users can determine the index of the first data row to be included in the testing phase. This feature provides flexibility in selecting a specific subset of data for testing purposes, allowing for targeted and controlled testing scenarios 

  • Use Test Set: Uses "Test Set Start Value" and "Test Set End Value" to limit the number of data rows provided. Default value is set to False.

Debug

Debug BreakPoint: Debugging will break at this command if the break point is set.Default value is set to False.

Debug Disabled: A disabled command will be ignored during debugging. Default value is set to False.

Debug Error Option: This property specifies what action to take when an error occurs in the debugger. Default value is set to Notify which indicates that when an error occurs at debugging time , then it will be notified. If we want to ignore the error at debug time , then we need to set this property value as Ignore.

Debug Set: This property specifies the set of list elements to process when debugging.

  • ListCount:-Specifies the number of list items to use when debugging. A value of zero means all items are used.

  • ListStartIndex:-Specifies the index of first item to use in a list when debugging.

Export

Data Validation Script: A script is used to validate a single row of export data.

Duplicate Export Row Handling: This property specifies what action to take when duplicate data export rows are detected. All  child data of a removed row will be assigned to the existing duplicate row. The duplicate check can be performed on values extracted by  Capture commands with the key property, or it can be performed on hash keys calculated from all values in an export data row. Default value is set to “None” .

  • None: This will not remove Duplicate rows.

  • Remove(SHA-512):  This is used to  remove Duplicate rows when an agent run in single session.

  • Remove(Key Values): This is used to  remove Duplicate rows when an agent run in single session.

  • Remove(Key Values Across Sessions) :  This is used to  remove Duplicate rows when an agent run in Performance Sessions.

Empty Export Row Handling: This property specifies what action to take when an export data  row is empty. If removing empty rows, all child data of an empty row will also be removed, even if the child data is not empty. Capture commands with the property “Act as System” are ignored when checking if a data row is empty. Default value is set to “None” .

  • None: This will not remove Empty rows.

  • Remove Row: Removes Empty rows.

  • Remove Row and Increase Error: Remove Empty rows and increase errors.

  • Trigger Export Failure: Export fails.

Empty Export Table Handling: This property specifies what action to take when an export data table is empty. Default value is set to “None”.

  • None: This will not remove Empty table.

  • Remove: Removes Empty table.

  • Remove Table and Trigger Error: Remove Empty table and trigger error message.

  • Trigger Export Failure: Export fails.

Export Empty Row If No Data: Default value is set to True indicates that exports a single empty data row if this container extracts no data. Parent and Sibling data will be lost if merged with an empty data set, so this option ensures that parent and sibling data is exported when this container extracts no data. This option only has an effect when “Export Method” is set to “Add Columns and Rows”. If we don’t want to export a single empty data row if the container extracts no data , then we need to set is False.

Export Enabled: Default value is set to True. A command with Export Enabled set to False will not save any data to data output. This includes all sub-commands if the command is a container command.

Export ID Name: This property specifies the name of the primary key column in the exported table (database table, spreadsheet, CSV file, XML Node), if this container generates a new table. The export name postfixed with “ID” is used if this property is empty. If multiple agents are exporting data to the same table,  then you must set this option to the same value for all those agents.

Export ID Sort Order: This property specifies the sort order  of the primary key column in the exported table (database table, spreadsheet, CSV file, XML Node), if this container generates a new table. Default value is set to 1000.

Export Keys: This property specifies how keys are exported for this container. This property applies only if the agent is configured to export to file.

  • Default: Default indicates that Primary key exported for the container. 

  • None: Indicates that no any key will be exported.

  • Primary: This option allows users to export Primary key only.

  • Parent: This option allows users to export Parent key only.

  • Both: This option allows users to export both Primary and Parent keys.

In addition, there is another 'Export Keys' property available under Data->Export Target->Export Keys. This must be set to either "Row Counter" or "Always"

Export Method: This property specifies how to  export data from this container. Default value is set to “Add Columns and Rows” which allows users to add columns and rows. There are different ways in which we can extract data:-

  • Separate Output: This property allows users to export data in separate file.

  • Add  Columns And Rows: This property is used to add columns and rows.

  • Add Columns Only: This property is used to add columns only.

  • Add Columns and Merge Rows: This property is used to add columns and merge rows.

  • Convert Rows Into Columns: This property allows users to convert rows into columns

Export Name: This property specifies the name of the exported table (database table, spreadsheet, CSV file, XML Node), if this container generates a new table. The command name is used if this property is empty. If multiple agents are exporting data to the same table, then you must set this option to the same value for all those agents.

Export Parent ID Name: Specifies the name of the parent key column in the exported table (database table, spreadsheet, CSV file, XML Node), if this container generates a new table. The parent ID column name is used if this property is empty. If multiple agents are exporting data to the same table, then you must set this option to the same value for all those agents.

Export Parent Sort Order: Specifies the sort order of the parent key column in the exported table (database table, spreadsheet, CSV file, XML Node), if this container generates a new table. Default Value is 1000.

Export Rows to Columns Name Command: When "Export Method" is set to "Convert Rows Into Columns" this command provides the names for the new columns.

Export Rows to Columns Value Command: When "Export Method" is set to "Convert Rows Into Columns" this command provides the values for the new columns.

Export Validation Error Handling: Specifies what action to take when data validation fails on an export data row. Default value is set to “ Remove Row and Increase error”.

  • Remove Row and Increase Error: Remove Row and increase error when data validation fails.

  • None: This will not remove any row when data validation fails.

  • Remove Row:  Removes row when data validation fails.

  • Trigger  Failure: Trigger Export fails when data validation fails.

History Table Name: Specifies the name of the database table containing historical data if this container generates a new table and the agent is configured to store historical data in separate database tables. The command name appended with "History" is used if this property is empty. If multiple agents are exporting data to the same table, then you must set this option to the same value for all those agents.

Plural Export Name: Specifies the plural name of the exported table if this container generates a new table. The "Export Table Name" with the added character "s" is used if this property is empty. This property can be used to control the name of XML nodes when exporting to XML.

Download Document Command: Attempts to download a document when the form field value changes, and uses the selected Download Document command to capture and export the downloaded document.

List

Process in Sessions: This property specifies that  list will be split and processed in multiple sessions if the agent supports session ranges and a proper session range is specified. The input list associated with the Agent command (start command) will be divided by default, but you can specify any list command in an agent by setting the Process in Sessions  property as TRUE on the list command. You can only set this option on one command in an agent. Default value is set to False.

Navigation

Depth: Only follows URLs to the specified depth. Default value is set to 1.

Max. URLs: Only follows this number of URLs. Default max value is set to 20.

Stay Inside Domain: Only follows URLs inside the current domain.

Templates

Template Name:  The template path of the referenced template.

Use Template Reference:  Loads this container from a template when the agent is loaded. Default value is False.

URL

Data Consumer: Specifies the input data to use when processing this command.

  • Captured Data Command: Specifies the previously captured data column name which you want to use as  input data .

  • Data Source:  The source of the data consumed.

  • Data Transformation Script: Data transformation script. Default value is set to Disabled which is reflected by "Enabled" property value False. If you want to enable the data transformation script then you need to set to "Enabled" property value as True.

  • Input Parameter Name : Specifies the input parameter name to use .

  • Provider Column Name: Specifies a column from the data source that should provide the data to this command. specifies a command that provides data to the agent. A command can provide data to itself.

  • Provider Container: Specifies a command that provides data to this command.

Follow Current Page URL:  Follows a URL even if it's the exact same as the current page URL.

Ignore Invalid URLs:  This property specifies to ignores invalid URLs without logging an error.

Processing Mode:  Specifies if this command should open the link and process sub-commands synchronously or asynchronously. Sequentum Enterprise will try to work out the most appropriate mode if this value is set to Default.

Web Selection

Selection:  The selection XPaths of the web elements associated with this command.

  • Paths:  List of selection XPaths.

    • Path:  The selection XPath.

  • Select Hidden Elements:  Selects visible and disabled elements when true. Otherwise selects only visible and enabled web elements.

  • Selection Missing Option:  Specifies what happens if this selection does not exist in the current page.

    • Default: Specifies if  this selection does not exist in the current page then logs error. 

    • Ignore Command but Execute Sub-Commands: Specifies if  this selection does not exist in the current page then it ignores the current command , but executes sub-commands of the command.

    • Ignore Command: Specifies if  this selection does not exist in the current page then it ignores the current command as  well as sub-commands.

    • Log Error and Ignore Command: Specifies if this selection does not exist in the current page then it ignores the current command as  well as sub-commands and logs an error message.

    • Log Warning and Ignore Command: Specifies if this selection does not exist in the current page then it ignores the current command as  well as sub-commands and logs a warning message. Note: Warning message will be logged if , Log level is set to either ‘Low’ or ‘High’.

    • Log PageLoad Error and Ignore Command: Specifies if  this selection does not exist in the current page then it ignores the current command as  well as sub-commands and logs a Page Load error.

JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.