Data Export Scripts

Data Export scripts can be used to customize the export process. An export script could be used to export extracted data to custom data structures in a database, or it could export data to a custom data source.

A Data Export script can be added from the Data Export Configuration screen:

The following example writes data from one or more export tables to one or more CSV and Parquet files:

NOTE: To use the following code in the agent, use this additional library. By default the library is not added in the Export Script.

C# :
using System.Text.RegularExpressions;
using System.Collections.Generic;

Python:
from System.Collections.Generic import *
import datetime
import os
import shutil
import re

C#
Python

using System;
using System.IO;
using System.Text.RegularExpressions;
using System.Collections.Generic;
using Sequentum.ContentGrabber.Api;
using Sequentum.ContentGrabber.Commands;
public class Script
{
public static bool ExportData(DataExportArguments args)
{
var dt = DateTime.UtcNow;
var dtFormatted = dt.ToString("yyyy.MM.dd_HH.mm.ss") + "Z";

var exportedFiles = args.ExportToParquet(null, true);
var exportedFiles1 = args.ExportToCsv(null, true);
var newExportedFiles = new List();
//args.WriteDebug( dt +"for my ref only" , DebugMessageType.Information);
var files = exportedFiles.DataFiles;
foreach (var file in files)
{

files = exportedFiles1.DataFiles;
foreach (var file in files)
{
var filename = Path.GetFileName(file);
var filenameWithoutExt = Path.GetFileNameWithoutExtension(file);
var ext = Path.GetExtension(file);
var newFile = Regex.Replace(file, filename, string.Format("{0}_{1}{2}", dtFormatted, filenameWithoutExt, ext));
File.Move(file, newFile);
newExportedFiles.Add(newFile);
}

exportedFiles.DataFiles = newExportedFiles;
//args.BackupData(exportedFiles);
args.DistributeData(exportedFiles);
return true;
}
}

import clr
import datetime
from Sequentum.ContentGrabber.Api import *
from Sequentum.ContentGrabber.Commands import *
from datetime import datetime
#from System.Collections.Generic import List, Dictionary
from System.Collections.Generic import *
import re
import os
import shutil

def ExportData(args):
dtFormatted = (datetime.now().strftime("%Y.%m.%d_%H.%M.%S")) +"Z"
exportedFiles = args.ExportToParquet(None, True)
exportedFiles1 = args.ExportToCsv(None, True)
newExportedFiles =List[str]()#list[str]();
files = exportedFiles.DataFiles

for file in files:
file_name = os.path.basename(file)
filename_withouext = os.path.splitext(file_name)[0]
filename_ext = os.path.splitext(file_name)[1]

newFile = file.replace(file_name, '{0}_{1}{2}'.format(dtFormatted, filename_withouext, filename_ext))
shutil.move(file, newFile)
newExportedFiles.Add(newFile)

files = exportedFiles1.DataFiles
for file in files:
file_name = os.path.basename(file)
filename_withouext = os.path.splitext(file_name)[0]
filename_ext = os.path.splitext(file_name)[1]

newFile = file.replace(file_name, '{0}_{1}{2}'.format(dtFormatted, filename_withouext, filename_ext))
shutil.move(file, newFile)
newExportedFiles.Add(newFile)

exportedFiles.DataFiles = newExportedFiles
#args.BackupData(exportedFiles)
args.DistributeData(exportedFiles)
return True

The script must have a static method with the following signature:

public static bool ExportData(DataExportArguments args)

The function should return True if it succeeds and False if it fails. In this example, an instance of the DataExportArguments class will be given by Sequentum Enterprise and has the following functions and properties:

Property or Function	Description
Agent Agent	The current agent.
ScriptUtils ScriptUtilities	A script utility class with helper methods. See Script Utilitiesfor more information.
IExportData Data	The data to export.
bool IsDebug	True if the agent is running in debug mode.
ExtendedSortedList<string, string> InputParemeters	All input parameters defined by the agent.
void WriteDebug(string debugMessage, DebugMessageType messageType = DebugMessageType.Information)	Writes log information to the agent log. This method has no effect if agent logging is disabled, or if called during design time.
void WriteDebug(string debugMessage, bool showMessageInDesignMode, DebugMessageType messageType = DebugMessageType.Information)	Writes log information to the agent log. This method has no effect if agent logging is disabled, or if called during design time.
void Notify(bool alwaysNotify)	Triggers notification at the end of an agent run. If alwaysNotify is set to false, this method only triggers a notification if the agent has been configured to send notifications on critical errors.
void Notify(string message, bool alwaysNotify)	Triggers notification at the end of an agent run, and adds the message to the notification email. If alwaysNotify is set to false, this method only triggers a notification if the agent has been configured to send notifications on critical errors.
GlobalDataDictionary GlobalData	Global data dictionary that can be used to store data that needs to be available in all scripts and after agent restarts. Input Parameters are also stored in this dictionary.
IConnection GetDatabaseConnection(string connectionName)	Returns the specified database connection. The database connection must have been previously defined for the agent or be a shared connection for all agents on the computer. Your script is responsible for opening and closing the connection by calling the OpenDatabase and CloseDatabase methods.
string[] ExportData(ExportTarget exportTarget, string sessionId = null)	Exports data to the specified export target. Returns the names of any exported data files if exporting to a file format.
void ExportToDatabase(string databaseConnectionName, string sessionId = null)	Exports data to the specified database.
string[] ExportToCsv(string sessionId = null)	Exports data to a CSV file using the default export folders and default options. Use the ExportData method to specify CSV export options. Returns the names of all exported CSV files.
string ExportToExcel(string sessionId = null)	Exports data to an Excel file using the default export folders and default options. Use the ExportData method to specify Excel export options. Returns the name of the exported Excel file.
string ExportToXml(string sessionId = null)	Exports data to an XML file using the default export folders and default options. Use the ExportData method to specify XML export options. Returns the name of the exported XML file.
void DistributeData(string file)	Distributes data to the target configured in the agent.
void DistributeData(string[] files)	Distributes data to the target configured in the agent.

The IExportData interface has the following functions and properties:

Property or Function	Description
IExportReader GetTable()	Returns a data reader that reads data from the main table containing exported data. All other data tables will be child tables of this table.
IExportReader this[string name] { get; }	Returns a data reader that reads data from a specified data table.
IExportReader GetTable(string name)	Returns a data reader that reads data from a specified data table.
bool ValidateData()	Validates the export data and returns True if the data is valid and False if the data is invalid. If the agent that extracted this data has changed since the data was extracted, then the data may have become invalid.

The IExportReader interface has the following functions and properties:

Property or Function	Description
bool Read()	Reads the next data row. Returns False if there are no more data rows available.
void Close()	Closes the data reader.
string GetStringValue(string columnName)	Gets a String value from the current row.
int GetIntValue(string columnName)	Gets an Integer value from the current row.
Guid GetGuidValue(string columnName)	Gets a GUID value from the current row.
string GetStringValue(int index)	Gets a String value from the current row.
int GetIntValue(int index)	Gets an Integer value from the current row.
Guid GetGuidValue(int index)	Gets a GUID value from the current row.
Type GetFieldType(int index)	Gets the data type of a specified column.
object GetFieldValue(string columnName)	Return a data value from the current data row.
object GetFieldValue(int index)	Return a data value from the current data row.
object this[string columnName]	Return a data value from the current data row.
object this[int index]	Return a data value from the current data row.
IExportReader GetChildTable(string tableName)	Return a data reader that reads data from a specified child table of the table associated with this data reader. The returned data reader will only read data rows that are child data to the current data row.
string[] ChildTableNames	An array containing the names of all child tables of the table associated with this data reader.