Norgate Data Products Support Corporate
General Support
- Downloads
- How to Download
- Installation
- Downloading / Installation FAQ
> Data File Corruption
- Antivirus Programs
- Submit Problem Report
- Premium Data Converter
- Smart Charts Copier
- The MetaStock Data Format
- Virtual Environments
Sales Support
- Order Form
- Refunds Policy
- My Account
- Forgot Password
- Price
- Free Trial
- History Only Purchases
- Installing on a Second Machine
- How to Upgrade an ASX Subscription
- Privacy Policy
- Licence Agreement
Premium Data/Stock Market Support
- Premium Data FAQ
- ASX FAQ
DataTools/Futures Support
- DataTools FAQ
- DataTools Service Information
- Understanding Futures Data
- Futures Coverage Notes
- Futures Database Notes
- Futures Contract Detail (Spreadsheet)
- Futures - Continuous Contracts
Forex Support
- Premium Forex FAQ
- Premium Forex Service Information
Charting Programs & Our Data
- Switching To Us From Another Vendor
- Advanced GET FAQ
- AmiBroker FAQ
- Dynamic Trader FAQ
- FCharts FAQ
- Fibotrader FAQ
- Gannalyst FAQ
- Guppytraders Essentials FAQ
- HotTrader FAQ
- Market Analyst FAQ
- MetaStock™ FAQ
- MTPredictor FAQ
- Omnitrader FAQ
- Stock Neuromaster FAQ
- Trading Blox FAQ
- TradeGuider FAQ
- TradeStation FAQ
Market Resources
- Alternative Market Glossary
- Links to Other Sites
Notices
- dataHQ Service Notice

Data File Corruption



Data file corruption can be indicated by error messages appearing during an update or by the inappropriate appearance of one or more charts. For instance, if the file order in a folder becomes jumbled with respect to its "master" file, the data for one security will start to be appended to the history for another. This anomaly should show up on a price chart even though the file is being updated successfully.

Some known causes of "MetaStock" format data file corruption are discussed below. Go to the end of the discussion for ways of recovering.

Simultaneous accessing of data files

If you have two or more charting applications, make sure you do not use them to view data at the same time, or file corruption may ensue.
Also, make sure that no charting or analysis application that can access the data is open when you are performing a data update.

Interference from Real-time Anti-Virus Scanning

Careful analysis of repeated file corruption events has shown that the problem could be attributed to real-time anti-virus scanning. When real-time scanning was disabled, daily updates proceeded without error. When scanning was resumed, file corruption invariably recurred.

During an update, numerous files need to be opened and closed. Real-time anti-virus scanning attempts to check all of these files at approximately the same time as the update is dealing with them. If the timing is slightly out, then the two applications are attempting to access the files at the same time and corruption occurs.

Interference from Cloud-based file hosting systems

Cloud-based file hosting systems (such as DropBox, Google Drive, Microsoft OneDrive) periodically lock files when they are synchronizing them to the Cloud. Such file locking interferes with the normal operation of our software and charting software, and can lead to corruption.

If you experience repeated instances of file corruption, it is strongly recommended that you exclude your data folders from real-time anti-virus scanning, disable scanning entirely before an update commences, or even consider changing your Anti-Virus software.

Power failures during data updates

Any power failure/spontaneous reboot of your system whilst a data update is underway will also cause data corruption. Purchasing an Uninterruptible Power Supply (UPS) should help to prevent power failures causing data corruption in the future.

Computer component issues

Computers that spontaneously reboot usually have hardware component issues and this will cause corruption when a data update is underway. Other component failures can cause issues such as faulty RAM and failing hard drives that may not immediately present any other obvious issues. If you have repeated computer issues you should consider obtaining professional IT support to help diagnose and subsequently resolve your hardware problems.

Recovering from File Corruption

The best way to insure against potential corruption is to perform regular backups of your data folders with a Backup utility. Failing that, Windows also has some built-in protection which you might be able to use as a fall-back.

Windows Vista / Windows 7

If you notice that a data folder has become corrupted, simply right-click on that folder, select "Restore previous versions" and then choose an earlier-dated version to restore.

Click here for an illustration

Windows System Protection should be enabled by default for your c: drive. If your data is kept on another drive, such as the d: drive, configure System Protection to cover that drive by accessing System > System Protection from the Windows Control Panel.

Windows 8

By default, Windows File History only backs up the Desktop, Favorites, Contacts, and the local OneDrive folder (if you have one). It also backs up the Music, Pictures and Videos "Libraries".
To get around this limitation, you can add the "Trading Data" folder as a "Library" so that it is included in the File History backup process. In File Explorer, browse to the "C:\Trading Data" folder, right-click on it, select "Include in library" and then "Create new library".
To access the backups, open the Control Panel and select Security/File History then click on the "Restore Personal Files" link in the left pane.

Windows 10

By default, Windows File History may not be enabled on your machine.
Before you can modify any settings, you will need to plug in an external hard drive or a large USB memory stick with lots of free space on it, then go Start and type "File History Settings" or go Start > Settings > Update & Security > Backup > Add Folder and browse to the "C:\Trading Data" folder.

What to Restore

If you restore an entire database, either from a backup or from a Windows System Protection restore point, the corresponding updating program will "know" where to update from.

For Premium Data/Stocks - restore the entire "ASX" or "US" folders.
For DataTools/Futures - restore the entire "Futures" folder.
For Premium Forex/Forex - restore the entire "Forex" folder.

Installing a fresh Data History

It may be that no sound backup can be found to restore, or only a very old one. Or it may be that the corruption occurred quite a long time ago, which would require the restoration of a very old back up and necessitate a very long update.

In these situations, you can always recover by installing a fresh history (by obtaining the relevant history installer from our Downloads area). Notes on history re-installation are available here.

History installers are always available to current subscribers who purchased the corresponding history at some stage.