Remove Duplicate
The Remove Duplicate command checks extracted data against previously extracted data and discards the extracted data if it already exists.
Example
Below screenshot shows the simplest example where the Remove Duplicate command can be used.
The Remove Duplicate command should be added after the capture commands that extract the data that should be checked, and the Remove Duplicate command must have the same parent command as the capture commands.
When a Remove Duplicate command detects a duplicate it will exit a specified parent command. By default, a Remove Duplicate command exits the current list entry of the nearest parent list command or the nearest parent command that exports data to a separate data table.
If the target data is displayed by descending order, then you can use a Remove Duplicate command to extract only new data by exiting the Agent command when a duplicate is detected. For example, if the target data is ordered by newest date to oldest date and the Remove Duplicate command detects that a specific date has already been extracted, then you know that all data after that date has already been extracted and you can exit the agent.
Command Properties
Command
Command Description: A custom description for the command. The default value is Empty.
Command Transformation Script: A script used to change command properties at run time. The default value is disabled.
Disabled: This property set to True allows the user to disable the command. A disabled command will be ignored. The default value is set to False.
ID: This property indicates the internal unique ID of the command and is always auto- generated e.g. 58c8e4ac-e4c0-48f7-a63d-77064945380b.
Increase Data Count: This property indicates the data count every time this command is processed. The default value is set to False. Set it to TRUE if you want to get the count of the number of times a specific command is executed to get the data. The data count value is increased during data extraction, so it is used to measure agent progress and basis this increased data count, the agent decides the success criteria.
Name: This property specifies the name of the command.
Notify On Critical Error: A notification email is sent at the end of an agent run if the command encounters a critical error, and the agent has been configured to notify on critical errors. Critical errors include page load errors and missing required web selections. The default value is set to False.
Debug
Debug Break Point: Debugging will break at this command if the breakpoint is set. The default value is set to False.
Debug Disabled: A disabled command will be ignored during debugging. The default value is set to False.
Debug Error Option: This property specifies what action to take when an error occurs in the debugger. The default value is set to Notify which indicates that when an error occurs at debugging time, then it will be notified. If we want to ignore the error at debug time, then we need to set this property value as ignore.
Export
Export Enabled: A command with Export Enabled set to false will not save any data to data output. The default value is set to True which indicates that any data will be saved to a data output.
Remove Duplicate
Capture Commands: Checks the data extracted by the specified capture commands to see if that data has already been extracted.
Check Method: This property specifies the Search Method for Duplicate.
New Data: Only data extracted in the current agent run is searched.
Old Data: Only data extracted in the previous agent run is searched.
Old and New Data: Data extracted in the current agent run and all previous agent runs are searched. This is the Default value.
Exit Container: The command to action on.
Default: Exits from Parent List Entry.
Agent: Exits from Agent.
Parent Command: Exit from Parent Command.
Parent List Command: Exit from Parent List Command.
Parent List Entry: Exit from Parent List Entry.
Parent Pagination Command: Exit from Parent Pagination Command.
Specific Command: Exit from a specific command which is specified in “Specific command”.
Specific command: A specific command to action on.