ProvideDataAsList Overview
● Function: ProvideDataAsList (a property on DataProvider ) toggles whether a provider should present the entire result table as a single "list payload" instead of as multiple row items.
● Propagation: The flag is propagated to provider load methods and ultimately stored as CustomDataStorage.IsProvideDataAsList .
● Effect: This changes row counting and how input rows are exposed to consumers.
How it Flows in Code (High Level)
DataProvider.ProvideDataAsList is read in
DataProviderExtensions.LoadData(...) and passed into each provider-specific loadData(..., provideDataAsList) call.
Data-loading helpers convert their DataTable to CustomDataStorage using DataTableExtensions.ToDataStorage(dataTable, provideDataAsList) . This simply calls new CustomDataStorage(dataTable, isProvideDataAsList) .
CustomDataStorage.IsProvideDataAsList controls:
○ The logic for CustomDataStorage.Count() .
○ How consumers retrieve input rows via helpers in DataStorageExtensions .
Concrete Behavior (What Consumers See)
Mode | ProvideDataAsList Value | CustomDataStorage .Count() | GetInputDataRow(... ) Result |
Normal Mode | false | Returns the number of rows. | Returns a row mapping each column to the value from the specific row index. |
List Mode | true | Returns 1 (as long as the table has $\ge$1 row). | Returns one InputDataRow where each column maps to a List<object> (all values from that column in the original table, preserved in row order). The whole table becomes a single logical item. |
Key Implementation Points / Locations
● Propagation: DataProviderExtensions.LoadData(...) (passes dataProvider.ProvideDataAsList into provider loader calls).
● Conversion: ToDataStorage(...) calls new CustomDataStorage(dataTable, isProvideDataAsList) .
● Storage Flag & Count: CustomDataStorage — constructor stores IsProvideDataAsList.
○ Count() logic: return IsProvideDataAsList && dataTable.Rows.Count > 0 ? 1 : dataTable.Rows.Count;
● Consumer Assembly: GetInputDataRow(...) builds the row differently based on provideDataAsList .
Example (Conceptual)
Column | A | B |
Row 1 | 1 | "x" |
Row 2 | 2 | "y" |
Row 3 | 3 | "z" |
ProvideDataAsList = false (Normal Mode):
● Count() $\to$ 3
● GetInputDataRow(2, false) $\to$ { A: 2, B: "y" }
ProvideDataAsList = true (List Mode):
● Count() $\to$ 1
● GetInputDataRow(1, true) $\to$ { A: [1, 2, 3], B: ["x", "y", "z"] }
Practical Notes / Gotchas
● When set, ProvideDataAsList changes the semantics for downstream commands—they must expect lists as column values.
● CustomDataStorage.GetUnsafeDataTable() still exposes the raw DataTable
(unchanged). The list mode is a logical presentation layer only.
● CustomDataStorage.Count() returning 1 in list mode is critical: consumers that
iterate input rows will loop exactly once and must treat column values as lists.
● Most providers consistently support the flag because loaders always pass provideDataAsList into ToDataStorage(...) .
InputDataRow Class Documentation
Overview
InputDataRow is a class in the Sequentum.ContentGrabber.Internal namespace that
represents a single row of input data used by Content Grabber. It implements two interfaces:
● Api.IInputDataRow
● IDisposable
The class stores column names and values for a row of data, provides accessors to retrieve typed values, and manages an associated CustomDataStorage instance.
Key Responsibilities
● Represents a row of input data where each column has a name and value.
● Supports typed retrieval of column values (string, int, bool, double, DateTime, binary, list of strings).
● Handles special cases such as empty rows and case‑insensitive column access.
● Provides utility methods to get the first non‑empty or first string column.
● Ensures resource cleanup through the Dispose() method.
Properties
DataStorage
Returns the associated CustomDataStorage instance.
RowCount
The total number of rows available in the data source.
RowIndex
The index of the current row.
IsEmpty
Indicates whether this is an empty placeholder row.
Constructors
InputDataRow(CustomDataStorage dataTable, int rowIndex, int rowCount)
Creates a row backed by a data table.
InputDataRow()
Creates an empty row (no data, no storage).
Public Methods
Add(string key, object theValue) Return Type: void
Adds a column name and value to the input data row.
GetColumnValues()
Return Type: IList<object>
Returns all values of all columns of a data source in sorted order.
Example - Input data source is csv.
Name | Company | Address | Zip Code |
Peter | ABC | KPC | 123 |
David | XYZ | BVB | 456 |
GetColumnValues() will return the values of all columns in sorted order, which means
[0] -> Address - [KPC, BVB]
[1] -> Company - [ABC, XYZ]
[2] -> Name - [Peter, David]
[3] -> Zip Code - [123, 456]
GetColumnNames()
Return Type: IList<string>
Returns all column names of a data source.
Example - Input data source is csv which contains columns in the following order.
Name | Company | Address | Zip Code |
XXX | XXX | XXX | XXX |
GetColumnNames() will return name of all columns, which means
Column[0] - Name
Column[1] - Company
Column[2] - Address
Column[3] - Zip Code
Note : column names are not in sorted order.
GetValue(string columnName, bool isReturnNullIfnotFound = false)
Return Type: object
This method retrieves the value associated with a given column name. It includes several layers of logic to safely and flexibly handle different cases:
Empty Row Handling
If the row is marked as IsEmpty , the method immediately returns an empty string ( "" ). This ensures that empty placeholder rows do not cause exceptions when accessed.
Column Existence Check If the column does not exist:
○ If isReturnNullIfnotFound is true , the method returns null instead of throwing an exception.
○ If isReturnNullIfnotFound is false , it throws an InputDataColumnNotFoundException .
This provides flexibility depending on whether the caller wants strict enforcement or graceful fallback.
Special Case: "default" Column
If the caller requests a column named "default" (case-insensitive), and the row contains values, the method returns the first value in the row.
This acts as a shortcut for retrieving a primary or default value when a specific column name is not important.
Normal Value Retrieval
If none of the special conditions apply, the method simply returns the value for the specified column name.
Example - Input data source is csv.
Name | Company | Address | Zip Code |
Peter | ABC | KPC | 123 |
David | XYZ | BVB | 456 |
For the column “Name” it will return -
[0] - Peter
[1] - David
ContainsKey(string columnName)
Return Type: bool
Returns true if the specified column exists or if the special column name "default" is used.
GetFirstNoneEmptyStringValue()
Return Type: string
Check if the row is empty:
● If IsEmpty is true , the method immediately returns an empty string.
Case 1 — provideDataAsList = false . Value is a string :
● If the value is a string and it is not null or whitespace , the method returns it immediately.
● If the column value is not string type then it will return an empty string.
Case 2 — provideDataAsList = true . Value is a List<object> :
● If the value is a list, the method checks whether the list contains any string entries.
● If yes, it returns the first string from the list .
GetFirstStringValueOrEmpty()
Return Type: string
● Returns the first string column value or the first column converted to string.
● Does not check whether the string is empty or whitespace.
● If no string value exists, returns the string representation of the first stored value.
Typed Getters
GetIntegerValue(string columnName) Return Type: int
Case 1. provideDataAsList = false .
● If IsEmpty is true , the method returns 0 immediately because there is no data to interpret.
● If the value is indeed an integer, it will return the integer value.
● If the retrieved value is not an int , the method throws an
InputDataFormatException indicating that the column is not of type integer.
Case 2. provideDataAsList = true
● This method will throw an Exception.
GetDoubleValue(string columnName) Return Type: double
Case 1. provideDataAsList = false .
● If IsEmpty is true , the method immediately returns 0 because no data is available to read.
● If the value is confirmed to be a double, it will return a double value.
● If the returned value is not of type double , the method throws an
InputDataFormatException
Case 2. provideDataAsList = true
● This method will throw an Exception.
GetDateTimeValue(string columnName) Return Type: DateTime
Case 1. provideDataAsList = false .
● When IsEmpty == true , the method returns: DateTime.MinValue
● When IsEmpty == false, t his method returns the actual DateTime value stored in the column.
Case 2. provideDataAsList = true
● This method will throw an Exception.
GetBinaryValue(string columnName) Return Type: byte[]
Case 1. provideDataAsList = false .
● This method retrieves the value of a column as a byte array .
● If the value is not a byte[] , it throws an InputDataFormatException . Case 2. provideDataAsList = true
● This method will throw an Exception.
GetBooleanValue(string columnName) Return Type: bool
Case 1. provideDataAsList = false .
This method retrieves a column's value as a Boolean . It accepts two types only:
● bool → returned directly
● int → converted to Boolean ( > 0 = true, otherwise false) Anything else triggers an InputDataFormatException . Case 2. provideDataAsList = true
● This method will throw an Exception.
GetStringValue(string columnName, bool isReturnBlankIfnotFound = false) Return Type: string
Case 1. provideDataAsList = false .
● When IsEmpty = true, the method will return "" (empty string).
● If the value is string → returns the string
● If the value is any other type → it will be converted into a string value. Case 2. provideDataAsList = true
● It converts the list items to strings.
● Returns a comma-separated string of all list elements.
GetStringList(string columnName)
Return Type: List<string>
Case 1. provideDataAsList = false .
● If IsEmpty = true, then it returns list by converting stored value (supports List<object> and List<string>)
● Returns list only if stored value is List<object> of strings; otherwise throws exception
● If the value is not a list, it will always throws exception
Case 2. provideDataAsList = true
● Returns string list if all list items are strings; otherwise throws exception
ToString()
Return Type: string
Serializes the row into a #value sequence using the first item if the value is a list.
Exception Handling
This class throws the following exceptions:
● InputDataColumnNotFoundException — When requesting a missing column.
● InputDataFormatException — When a column contains an invalid type.
Usage Summary
The InputDataRow class is designed to:
● Provide flexible typed access to data extracted from input sources.
● Maintain column order and case‑insensitive lookup.
● Handle both simple values and list‑based values.
● Safely manage underlying data storage objects.
It is commonly used internally by Content Grabber to process structured data provided through input sources such as CSV, Excel, or databases.
CustomDataStorage Class Documentation
Overview
The CustomDataStorage class provides a flexible and thread-safe data container built on top of the .NET DataTable . It is designed to:
● Store tabular data in various formats.
● Dynamically add columns and rows.
● Load data from primitive values, arrays, DataTable , IDataReader , and custom storage providers.
● Support thread-safety through ReaderWriterLockSlim .
● Optionally interact with an external database via the ICustomDatabaseSubStorage interface.
● Provide selective data retrieval and filtering capabilities.
● Conditionally return data as a list (flattened) when IsProvideDataAsList is enabled.
This class is part of the Sequentum.ContentGrabber.Api namespace and commonly used in data extraction workflows.
Key Features
Flexible Data Initialization
The class provides multiple constructors allowing initialization using:
● A single primitive value ( string , int , double , DateTime ).
● Arrays of primitive values.
● Custom column definitions via SortedList<string, Type> .
● Existing DataTable .
● Data loaded from IDataReader or a custom storage provider ( ICustomDatabaseSubStorage ).
Dynamic Schema Management
Methods available for adding columns:
● AddColumn(string, Type)
● AddStringColumn(string)
● AddIntegerColumn(string)
● AddDoubleColumn(string)
● AddDateTimeColumn(string)
● AddBooleanColumn(string)
● AddStringColumns(string[])
Data Row Manipulation
Supports adding rows in various formats:
● Adding using column names.
● Adding based on fixed column positions.
● Adding via CustomDataValues collection.
● Adding via IList<object> .
Data Querying and Filtering
Provides filtering and existence checks using:
● Raw filter string ( SelectRows(string) )
● Column name and value
● Multiple column/value pairs ( SortedList<string, object> )
Thread Safety
The entire class uses ReaderWriterLockSlim to ensure safe concurrent reading and writing.
External Storage Support
When initialized with ICustomDatabaseSubStorage , it can:
● Save updated data ( UpdateStorage )
● Delete the storage ( DeleteStorage )
Data Limiting and Pagination
Supports:
● Limiting rows using session ranges.
● Limiting rows by offset and count.
Disposal Handling
Cleans up internal DataTable and locking resources. Can optionally remove external storage when disposing.
Properties
bool IsProvideDataAsList
Indicates whether row count should be treated as a single list item ( useful for certain integrations).
Important Methods
Data Adding Methods
● AddRow(string name, object value)
● AddRow(string name1, object value1, string name2, object value2)
● AddRow(string name1, object value1, string name2, object value2, string name3, object value3)
● AddRowValues(object value)
● AddRowValues(object v1, object v2)
● AddRowValues(object v1, object v2, object v3)
● AddRow(CustomDataValues values)
● AddRow(IList<object> values)
Selection Methods
● SelectRows(string filter)
● SelectRows(string columnName, object value)
● SelectRows(SortedList<string, object> conditions)
● ContainsData(...) (multiple overloads)
Storage Methods
● UpdateStorage(IConnection connection)
● DeleteStorage(IConnection connection)
Management Methods
● Count()
● HasData()
● Limit(offset, count)
● Limit(rangeSessionId)
Internal Behavior
The class internally:
● Uses DataTable.Select() for filtering.
● Protects operations using locker.EnterReadLock() and locker.EnterWriteLock() .
● Handles primitive type filters safely, escaping special characters.
● Provides a helper method to build filter expressions.
Use Cases
● Temporary in-memory storage for extracted content.
● Passing structured data between ContentGrabber commands.
● Storing and filtering lookup data.
● Paginating large data sets.
● Synchronizing data to external custom databases.
Conclusion
CustomDataStorage is a robust, thread-safe, and extremely flexible data container designed for ContentGrabber's internal and external integrations. Its versatility in loading, storing, filtering, and managing data makes it a key component for data handling scenarios.
If you want, I can also generate:
● A UML diagram
● XML documentation comments for the class
● A shorter summary version