Error Logs and Notifications

You can use error notifications to warn when an agent encounters an error, so that the administrator can take appropriate action. The administrator can use the logs to get more information about the errors.

Error Notifications

Notifications are email messages that are sent to a specific email address when certain conditions occur. The notification configuration screen is available by clicking the Agent Settings > Notifications menu.

Email_Notifications.gif

This agent sends email notifications if it encounters any critical errors

 

An email notification can be sent when an agent finishes, but never during agent execution or when debugging an agent. You can choose to send email notifications when one or more of these conditions are met:

Condition Description

Always send notifications when an agent completes a run

A notification email will always be sent when the agent finishes, whether an error occurs or not.

Send notification on critical errors

An email notification will be sent when critical errors occur. A critical error will appear colored red on the log screen. These critical errors include page load errors and also situations where required web selections are missing from a web page.

Send notification on unsuccessful data extraction

A notification email will be sent when an agent run doesn't meet defined success criteria.

Send notification on successful data extraction

A notification email will be sent when an agent run meets defined success criteria. The notification is sent to Success email addresses if specified.

 

A script can trigger notifications by calling one of these methods on the script parameter object:

Method Description

void Notify(bool alwaysNotify = true)

Triggers notification at the end of agent execution. If alwaysNotify is set to False, this method only triggers a notification if you configure the agent to Send notification on critical errors (see above).

void Notify(string message, bool alwaysNotify = false)

Triggers notification at the end of an agent execution, and adds a message to the notification email. If alwaysNotify is set to False, this method only triggers a notification if you configure the agent to Send notification on critical errors (see above).

 

Success Criteria

A success criterion can be defined to tell an agent when it should consider an agent run successful. For example, success criteria could include that the agent must load a minimum number of pages.

Define criteria for a successful agent run.

The Success Criteria window can be opened from the Run menu in the Sequentum Enterprise editor, or from the Notification window.

The following four success criteria can be defined.

Criteria Description

Min. page loads

The minimum number of pages the agent must load.

Min. data count

The minimum number of data entries the agent must extract. A data entry is defined by setting the option Increase data count on any command. The data count is increased every time a command with Increase data count is executed.

Max. page errors

The maximum number of page errors allowed

Min. export rows

It defines the minimum number of rows exported to a file. A minimum number should be specified here so that it can prevent exporting blank files.

 
Success criteria can be specified as absolute numbers or as a percentage of the value from the last successful agent run. For example, if Min. page loads are set to 95% it means the agent must load a minimum 95% of the number of pages it loaded at the last successful run. A Percentage criterion always succeeds if the agent has not been successfully run before.

Success criteria can be used to decide when email notifications are sent but are also used in Change Tracking to decide when it's safe to mark data as deleted.

Email Troubleshooting

Most email servers are very restrictive in regard to who or what is allowed to send emails. If you are getting an error when sending emails from Sequentum Enterprise, then it's most likely a problem with the configuration of your account. Here is a list of things you can check:

  • Make sure your email account is configured to allow you to send emails from the email address specified in From email address.
  • Make sure your username and password are correct, and that login to your account is not restricted. Use another email client or your web browser to login to your email account to make sure you can login.
  • Use SSL if required. Most online services, such as Gmail, require SSL.
  • Use the correct port. Gmail uses port 587 for example.
  • Most online services, such as Gmail, don't allow just any application to login to your email account. Many online services will have a special option to allow any application to login to your account, and you must set this option to allow Sequentum Enterprise to login. In Gmail you can set this option on the following page:
    https://www.google.com/settings/security/lesssecureapps

Error Logs

Logging is set to low by default when running an agent, and high by default when debugging an agent. Debug logging is always saved to a database table, but runtime logging can be saved to a database table or a text file.

The database table supports the log viewer in the Sequentum Enterprise editor, which you can access by clicking the View Log button in either the Run ribbon menu or the Debug ribbon menu.

Sequentum Enterprise supports 3 log levels, High, Medium and Low. The High log level will log everything. The Medium log level will log errors and warnings, and the Low log level will only log errors.

The critical errors in this log are red in color

URL Column

The URL column in the Agent Debug Log window shows links to all web pages that the agent has processed, and you can double-click on any of these links to open the web page in your default browser. For any failures on data extraction, this is an easy way to open the web page and check if the web page layout is different from the expected layout.

NOTICE

Double-clicking on a link to a sub-page only works if the website allows direct links into the sub-pages. Many websites don't allow direct links into sub-pages without going through other pages first. Read below to learn how to directly access the HTML for sub-pages.

HTML Column

If a website does not allow direct links into sub-pages, you can configure an agent to save the entire HTML of all processed pages. When you configure the agent to log all HTML, the HTML column will contain buttons that you can click to view the raw HTML in the Sequentum Enterprise HTML viewer. The raw HTML does not include style sheets and other support files that are necessary to render a complete web page properly, but it will often show you enough information to determine what is causing an error (such as a CAPTCHA page).

Always Log Proxy Information

If you use multiple proxy servers for anonymization purposes, it might be useful to log information about the proxy servers. This would help you find information about what proxy server is used for a specific page load or request, how your proxies rotate, and diagnose potentials issues.
The Always Log Proxy Information property can be set to true to always log proxy information irrespective of a log level set.

Always_Log_Proxy_Information.png

Read the following article IP Blocking & Proxy Servers about proxy servers and why you may want to use them.

Was this article helpful?
0 out of 0 found this helpful

Comments

0 comments

Please sign in to leave a comment.