The Internal Database

Sequentum Enterprise uses two layers of internal data, internal data and export data. While the agent is running, it continuously saves data in an internal format. When the agent finishes, it converts the internal data into export data, and then it sends this data to the export target so it becomes external data.

The default internal database is a SQLite database, but this can be changed to SQL Server or MySQL. Oracle and OleDB are not currently supported as internal databases.

Internal Data

Sequentum Enterprise stores internal data in the database - which contains all the data extraction elements for any agent, but also contains the data which corresponds to the settings for the agent properties. For example, if an agent stops, crashes, or otherwise fails, the internal data is used to start the agent again exactly at the point of failure.  Also, the internal data will always contain a data table for each container command in the agent and at least one data field for each capture command in the agent. You can always view the internal data by clicking the Data > View Internal Data menu.

If you configure an agent to add new data to the existing data, then the internal data store will continue to grow larger. Otherwise, the internal data store will be recreated every time the agent is run.

Export Data

The agent stores the export data in the internal database, and this data doesn't contain any of the property data for the agent itself. Also, the export data is more readable than the internal data. The export data will usually not contain a data table for each container command in the agent, and some capture commands may not have corresponding data fields.

An agent can be configured to use a separate export database. Please refer to the Use Separate Export Database option below.

Export data is always overwritten, so you cannot add data to existing export data. You can only add data to existing internal data. You can always view the internal data by clicking the Data > View Export Data menu.

External Data

External data is the same as the export data, and the agent delivers it to an external data store chosen by the user. Typically, external data is overwritten, except when you choose Export last data segment only and you are exporting to a database. In that case, the operation simply adds the data to the external database tables.

You can also use a Data Export Script to customize the export process which will allow you to update or add to existing data. The export data is set in a fixed format, but you have some options for changing this format. You need to use a Data Export Script if you want to automatically create or configure your own data structures within this data.

Internal Database

The Internal Database configuration window can be opened from the Data > Internal Database Settings menu button in the Sequentum Enterprise editor.

You can use this window to change the internal database. The default internal database is a SQLite file database, but you can change it to either a SQL Server or MySQL database. Changing the internal database from SQLite to SQL Server or MySQL can increase the run-time performance of agents significantly.

Internal database settings.

The Data retention option on the Internal Database window allows you to control how long extracted data is kept in the internal database. The following options are available.

The option Allow automatic overwrite of internal database can be used to control whether an agent is allowed to recreate the database tables. An error occurs if a change is made to an agent that requires the database tables to be recreated, and the agent is not allowed to do so. In this case, a user may be able to manually make changes to the internal database tables to accommodate the modified agent without losing existing data.

 If you uncheck the Allow automatic overwrite of internal database option and choose the option Keep All and Export or Keep Some and Don't Export  for the Data Retention settings, and you want to add a new column to a specific container node of the agent, then you will need to manually alter the internal SWSIR table associated with that container

Let us understand this by referring to the following screenshot of a sample agent.

As per the screenshot below, the CapturedData2 command is being used to extract data by navigating through both Container3 and Container4 link commands.

Next, if you want to navigate only through the Container3 command to extract the same data as shown in the screenshot below, then you will need to alter the internal SWSIR table associated with the Container3 navigate command, which would be [SWSIR Container3].

The reason for this alteration is required is because the previous table structure had the Container3 column in the SWSIR Container4 table, but now it has been moved to the Container3 column hence it should be added to the [SWSIR Container3].

Sample alter query

If you need to add multiple columns to different nodes, then you will have to alter the relevant internal tables for each new column.

Please note that every container type, including Agent Command, Web Element List, Navigate URL, Navigate Link, Group command, Branch Command, and Data List Command, has its own corresponding internal table as the SWSIR table followed by the container name.

The Embed files in database option is used to control whether extracted files are stored in the internal database or on disk.

• If this option is enabled, downloaded or extracted files are embedded in the selected internal database.

• Otherwise, additional configuration will be shown that will let the user choose whether to store extracted files to a custom folder location, or to the default location.

• The File exist handling option determines which action to take when an extracted/downloaded file already exists with the same filename. The following handling actions are available:

  • Overwrite Existing File

  • Discard New File (This option simply discards the new duplicate file and keeps the old one)

  • Create File With Random Name

The Separate Export Database option, when enabled, allows the user to configure a separate export database for the agent. This is particularly useful when you intend to have a centralized export database if running different instances of multiple performance sessions of the same agent across multiple servers.

The Track latest changes option can be used to keep track of the latest changes that have been made to extracted data. The agent will mark extracted data as deleted, modified or added. See the Change Tracking topic for more information.

When changes are made to an agent, the internal database tables may need to change as well. For example, if the agent is modified to extract more data fields, the internal database tables need more columns to store the new data. Sequentum Enterprise needs to recreate the internal database tables when such agent changes are made, and this will remove all existing data in those tables. This will reset change tracking and could have other serious consequences if an agent is configured to use previously extracted data.

Important Notice About SQL Server and MySQL

If two agents use the same internal database, it's important they don't use the same agent name. When an agent recreates the internal database it first removes all tables for that agent by deleting tables starting with the name of the agent.

For example, if the agent uses the following internal tables, it would remove all tables starting with SWSCR [Agent Name].

SWSCR [Agent Name] AGENT PROCESS

SWSCR [Agent Name] AGENT PROCESS

SWSCR [Agent Name] AGENT BROWSER

SWSCR [Agent Name]

SWSCR [Agent Name] [Command Name 1]

SWSCR [Agent Name] [Command Name 2]

This approach creates an obvious problem when two agents use the same internal database and also start with the same name because Sequentum Enterprise would remove the tables for both projects when it should only remove tables for one project. It is important to ensure that no two agents that use the same internal database have the same agent name.