You can use error notifications to warn when an agent encounters an error, so that the administrator can take appropriate action. The administrator can use the logs to get more information about the errors.
Error Notifications
Notifications are email messages that are sent to a specific email address when certain conditions occur. The notification configuration screen is available by clicking the Agent Settings > Notifications menu.
This agent sends email notifications if it encounters any critical errors
An email notification can be sent when an agent finishes, but never during agent execution or when debugging an agent. You can choose to send email notifications when one or more of these conditions are met:
| Condition | Description |
|---|---|
|
Always send notifications when an agent completes a run |
A notification email will always be sent when the agent finishes, whether an error occurs or not. |
|
Send notification on critical errors |
An email notification will be sent when critical errors occur. A critical error will appear colored red on the log screen. These critical errors include page load errors and also situations where required web selections are missing from a web page. |
|
Send notification on unsuccessful data extraction |
A notification email will be sent when an agent run doesn't meet defined success criteria. |
|
Send notification on successful data extraction |
A notification email will be sent when an agent run meets defined success criteria. The notification is sent to Success email addresses if specified. |
A script can trigger notifications by calling one of these methods on the script parameter object:
| Method | Description |
|---|---|
|
void Notify(bool alwaysNotify = true) |
Triggers notification at the end of agent execution. If alwaysNotify is set to False, this method only triggers a notification if you configure the agent to Send notification on critical errors (see above). |
|
void Notify(string message, bool alwaysNotify = false) |
Triggers notification at the end of an agent execution, and adds a message to the notification email. If alwaysNotify is set to False, this method only triggers a notification if you configure the agent to Send notification on critical errors (see above). |
Success Criteria
A success criterion can be defined to tell an agent when it should consider an agent run successful. For example, success criteria could include that the agent must load a minimum number of pages.
Define criteria for a successful agent run.
The Success Criteria window can be opened from the Run menu in the Sequentum Enterprise editor, or from the Notification window.
The following four success criteria can be defined.
| Criteria | Description |
|---|---|
|
Min. page loads |
The minimum number of pages the agent must load. |
|
Min. data count |
The minimum number of data entries the agent must extract. A data entry is defined by setting the option Increase data count on any command. The data count is increased every time a command with Increase data count is executed. |
|
Max. page errors |
The maximum number of page errors allowed |
|
Min. export rows |
It defines the minimum number of rows exported to a file. A minimum number should be specified here so that it can prevent exporting blank files. |
Success criteria can be used to decide when email notifications are sent but are also used in Change Tracking to decide when it's safe to mark data as deleted.
Email Troubleshooting
Most email servers are very restrictive in regard to who or what is allowed to send emails. If you are getting an error when sending emails from Sequentum Enterprise, then it's most likely a problem with the configuration of your account. Here is a list of things you can check:
- Make sure your email account is configured to allow you to send emails from the email address specified in From email address.
- Make sure your username and password are correct, and that login to your account is not restricted. Use another email client or your web browser to login to your email account to make sure you can login.
- Use SSL if required. Most online services, such as Gmail, require SSL.
- Use the correct port. Gmail uses port 587 for example.
- Most online services, such as Gmail, don't allow just any application to login to your email account. Many online services will have a special option to allow any application to login to your account, and you must set this option to allow Sequentum Enterprise to login. In Gmail you can set this option on the following page:
https://www.google.com/settings/security/lesssecureapps- Refer to this article to troubleshoot SMTP connection to Yahoo Mail.
Error Logs
Logging is set to low by default when running an agent, and high by default when debugging an agent. Debug logging is always saved to a database table, but runtime logging can be saved to a database table or a text file.
The database table supports the log viewer in the Sequentum Enterprise editor, which you can access by clicking the View Log button in either the Run ribbon menu or the Debug ribbon menu.
Sequentum Enterprise supports 3 log levels, High, Medium and Low. The High log level will log everything. The Medium log level will log errors and warnings, and the Low log level will only log errors.
The critical errors in this log are red in color
URL Column
The URL column in the Agent Debug Log window shows links to all web pages that the agent has processed, and you can double-click on any of these links to open the web page in your default browser. For any failures on data extraction, this is an easy way to open the web page and check if the web page layout is different from the expected layout.
NOTICE
Double-clicking on a link to a sub-page only works if the website allows direct links into the sub-pages. Many websites don't allow direct links into sub-pages without going through other pages first. Read below to learn how to directly access the HTML for sub-pages.
HTML Column
If a website does not allow direct links into sub-pages, you can configure an agent to save the entire HTML of all processed pages. When you configure the agent to log all HTML, the HTML column will contain buttons that you can click to view the raw HTML in the Sequentum Enterprise HTML viewer. The raw HTML does not include style sheets and other support files that are necessary to render a complete web page properly, but it will often show you enough information to determine what is causing an error (such as a CAPTCHA page).
Always Log Proxy Information
If you use multiple proxy servers for anonymization purposes, it might be useful to log information about the proxy servers. This would help you find information about what proxy server is used for a specific page load or request, how your proxies rotate and diagnose potentials issues.
The Always Log Proxy Information property can be set to true to always log proxy information irrespective of a log level set.
Read the following article IP Blocking & Proxy Servers about proxy servers and why you may want to use them.
Centralized Database Logging
Centralized database logging is used to add the agent logs to a centralized database in addition to the agent file logging. By default, the Centralized Database Logging is disabled and needs a Database connection to write the agent logs to a database. This feature is only supported for SQL Server, MySQL and PostgreSQL database.
- Connection Name: Database connection for centralized agent logging. The Database connection name is used from the connection created through the Application Settings → Managed Named Connections. The database connection can also be uploaded to the Agent Control Center for Centralized logging.
- Enable Centralized Logging: By default, this property is set to False, and setting it to True enables the centralized agent logging.
- Environment: Only logging from agents running in the specified environment will be written to the centralized database. The default value is set to Prod which means logs written to a centralized database when the agent runs with the Prod environment.
- Log Level: It is used to specify the maximum level of logging written to the centralized database. The default value is set to 1 which is used to write only error logs on a centralized database. Below are the log level values which is used for logging :
1: Default value is set to 1 which is used to write only error logs on a centralized database
2: Log Level value set to 2 is used to write only Warning and Error logs(One additional Info message will also be there in Log Level = 2 where it will add Agent started information.) on a centralized database.
3: Log Level value set to 3 is used to write Info, Waning, and Errors logs. - Retention Days: It is used to specify the number of days to keep logging for an agent. The latest log for each agent is never deleted. The default value is set to 0 which means only the latest 1-day log will be retained.
- Table Name: Database table name for centralized agent logging.
Log Files Export Target
In order to export the log files to your desired location, you can add a Log Files export target by going to the Data → Export Targets Configuration and select the Log Files Export Target and it will be added at the end of the Export Targets Pipeline.
By default it has the agent Logs folder path and can be modified to add the desired custom path and the log files will start getting copied to the new custom path.
You can also choose to Cleanup the Logs folder every time the agent runs when using the Log Files export target so that you do not the same files present over two locations.
Log Files: It is used to export the log files and Log HTML Files on either the default folder or a defined custom folder.
Log Folder: It is used to export the log files on either the default folder or a defined custom folder.
Log HTML Folder: It is used to export the log HTML files on either the default folder or a defined custom folder.
Log Path: It is used to export the log files to a particular path. There are 3 ways to export the log files for all agents using this one common setting which is available under the Application Settings→Folders.
Agent Folder: This is the default option that is used to deliver the log files in the individual agent’s Log folder for every agent.
Custom Folder: It is used to export the log files to the custom path which is defined by the user for all the agents.
Agent Group Folder: It is used to export the log files in the Agents folder for all the agents.
Cleanup: The cleanup logs option is set to False/Unchecked by default. Setting the Cleanup option to True will delete all the existing log files from the default log folder every time the agent starts.
Cleanup Log Folders: This property is set to "Application" by default which behaves as per the setting applied on the Application Settings→Folder→Log Path →Cleanup
The cleanup logs option is set to False/Unchecked by default under the Application Settings, however, if the same is to True then all the log files from the default log folder will get deleted every time the agent starts.
Comments
0 comments