amibroker

HomeKnowledge Base

Wrong close price in Yahoo data (no more?)

July 22, 2017 UPDATE :
Yahoo today changed the order of columns (swapped “Close” with “Adjusted Close”).
To fix, you need to edit aqh.format file (in Formats subdirectory).

Old format line


$FORMAT Date_DMY,Open,High,Low,Skip,Close,Volume

should be changed to:


$FORMAT Date_DMY,Open,High,Low,Close,Skip,Volume

A ready to use aqh.format file can be downloaded from here (right click and choose “Save target as…”). Place it into “Formats” subdirectory.

You may have noticed over last few days that on some symbols (stocks and ETF that pay dividends) close price is below open, high, low prices.

This is new Yahoo error due to the fact that they added “dividend adjustments” incorrectly
http://forum.amibroker.com/t/amiquote-3-15-is-coming/171/18

In the period May 17 – June 10, Yahoo was NOT adjusting for dividends so OHLC prices were adjusted for splits only.

Charts looked correct (albeit without dividend adjustments).

Then people sent complaints about missing dividend adjustments and Yahoo started adjusting for Dividends, but… only Close price, leaving High, Low, Open adjusted for Splits only.

Here is example of EWJ data downloaded from Yahoo:

02-08-2010, 39.000000, 39.360001, 38.880001, 9.840000, 35.239758, 4488800

The columns are as follows:

Date, AdjOpen, AdjHigh, AdjLow, RawClose, AdjCloseForSplitsAndDividends, Volume

As you can see adjusted for splits&dividends close price (35.239758) is lower than adjusted low (38.880001)

So here is where the mess is coming from.

Yahoo is adjusting High, Low, Open fields for splits but at the same time they are adjusting Close field for splits and dividends.

That is why you are getting Close field below OHL fields and strange looking charts.

Another example – this time straight from Yahoo web site.

Link:
https://finance.yahoo.com/quote/IP/history?period1=804549600&period2=807055200&interval=1d&filter=history&frequency=1d

What you get:

Wrong data on Yahoo

Clearly visible Adjusted Close is lower than Adjusted Low, because they subtract dividends from Close price, but don’t do that for High, Low, Open prices.

This needs to be fixed by Yahoo on their site. You need to complain to Yahoo to fix their mess.

There is already a thread that describes this data error on Yahoo web site https://forums.yahoo.net/t5/Yahoo-Finance-help/Historical-Data-Split-and-Dividend-Adjustment-Errors-in-EEM/td-p/279800 but apparently Yahoo does not see / understand that yet.

The are two possible ways to fix that on Yahoo side:
1. Yahoo needs to adjust Open, High, Low fields the same way as they are adjusting Close.
..or..
2. Yahoo needs to send RawOpen, RawHigh, RawLow instead of adjusted ones. Then adjusted OHL can be calculated on the fly by deriving split ratio from AdjClose/Close

Method 2 was used in the past (prior to May 17, 2017 changes). It is best solution as it gives access to both adjusted and unadjusted OHLC easily. But any method would do as long as OHLC fields are adjusted the same way (i.e. for both splits AND dividends)

How to install AmiQuote 3.14 correctly

UPDATE: AmiQuote 3.21 public version is available now.

AmiQuote 3.21 has been announced here: http://www.amibroker.com/devlog/2017/07/11/amiquote-3-21-released/

AmiQuote 3.21 is not a standalone installer, therefore – it requires AmiBroker installed on your computer and the installation path should point to the folder where AmiBroker is installed.

To run the installer – use the link at:
http://www.amibroker.com/bin/aq3210.exe (119KB) – 32 bit version
http://www.amibroker.com/bin/aq3210x64.exe (143KB) – 64 bit version

Make sure to download correct version. If you have 32-bit AmiBroker, use 32-bit AmiQuote. If you have 64-bit AmiBroker, use 64-bit AmiQuote. If you have both, install both.

32-bit version of AmiBroker

The correct folder is automatically detected if AmiBroker is installed so you don’t need to change it. The default installation path is the following:

Image1

64-bit version of AmiBroker

The correct folder is automatically detected if AmiBroker is installed so you don’t need to change it. The default installation path is the following:

Image1

TROUBLESHOOTING:

  1. If you are getting “Failed to update registry, use REGEDIT” message, then you need to run AmiQuote just once with administrator rights – to do so click on AmiQuote icon with right mouse button and select “Run As Administrator”. Do this just once. Don’t run as admin all the time because automatic import won’t work.
  2. If you are not able to download more than 2 years worth of data it means you entered too early “From” date. For example if you enter 1900 as “from” date you won’t get more than recent 2 years. But if you enter more resonable starting date such as year 2000, then you will be able to download 17 years worth of data
  3. AmiQuote 3.20 uses new method that should be independent from IE version installed, however if the download hangs on Windows XP – new Yahoo Finance pages are incompatible with Windows XP, and specifically with Internet Explorer 9 or lower that Windows XP shipped with. Windows XP is obsolete and not supported anymore by Yahoo. You need at least Internet Explorer 10 to be installed (Windows 7)
  4. If it still does not work – you did something wrong. The program works fine, as long above steps are PRECISELY done. Rinse and repeat until it clicks.

AmiQuote Yahoo Historical stopped working

In May 2017 Yahoo Finance started making changes to their web services. During this time certain services may be interrupted or broken.

Three AmiQuote functionalities are affected by recent Yahoo changes

  1. Yahoo Historical download – the old CSV download API is broken at Yahoo Finance. Fix is available in v3.14 and higher
  2. Yahoo Fundamental Extra – the API has been changed. Fix is available in v3.14 and higher
  3. Yahoo Intraday download – API does not respond. No fix available at the moment

You need to update to the most recent AmiQuote version announced here http://www.amibroker.com/devlog/ and available in the download section http://www.amibroker.com/download.html to be able to use Yahoo data again.

Purchases from Hong Kong

Customers from Hong Kong sometimes have problems ordering because there are no ZIP codes in Hong Kong and SWREG system that we are using requires ZIP code. If you are buying using credit card, entering 0000 or NA in place of ZIP code field is accepted. But if you are buying using PayPal, it may reject it if your PayPal account does not have matching ZIP code. Therefore, if you are using PayPal you need to follow these instructions:

  1. Update your PayPal account billing address with ‘0000’ in the Postal Code/ZIP field
  2. Now you can use 0000 as ZIP code in SWREG ordering page

Support response times

AmiBroker’s technical support staff everyday faces with very wide scope of subjects ranging from simple installation, lost registration details, password reminders questions to complex things like C++ programming or esoteric issues that occur say once a month or only when program is loaded with dozens of gigabytes of data.

Support response times to those different inquiries obviously vary a lot.

Technically we answer basic questions in 24 hours on week days (Monday-Friday).

Very simple questions get answered even in minutes if you happen to ask them when our support staff is in the office (we are in GMT+1 timezone). If you are in different time zone, we may be currently sleeping so you may need to wait for next day.

This response time applies to questions that are covered already in our Official Knowledge Base, Users’ Knowledge Base, Users’ Manual or internal documentation/resources. It is quite good idea to check those resources yourself as you are very likely to find the answer much quicker.

For more complex questions that need some formulas to be written or checked/verified the response time may be higher (48 hours), as long as this check can be done by our regular support staff.

Some complex issues/questions can not be solved/answered by support staff alone and then they are escalated to development. You need to keep in mind that development is 100% busy all the time, we are not sitting here doing nothing. It is all-day development job that is on-going and those complex support issues must wait in the queue. Also since some of issues/questions require lots of work (setting up environment to mimic customer’s setup, testing, single-step debugging sessions, going through millions of lines of code), it may easily take days or even weeks to complete. If development finds out that the issue is due to software problem, then the problem is either fixed at once or scheduled for fixing. This is a process. So please do not expect “next day response” for those kind of issues. You are also not going to get constant e-mails/updates like “we are working on it”, because we are always working on something in the queue. Please be patient, things are being worked on constantly.

How to register AmiQuote and AFL Code Wizard

AmiQuote and AFL Wizard are separate applications, therefore the registration process is also separate from registering AmiBroker and requires to enter the unlock code into Help->Register menu in AmiQuote or AFL Wizard respectively. The unlock codes are delivered in the transaction receipt generated after the purchase (sent from SWREG, ShareIt or other payment processor)

In order to register these programs, it is necessary to launch AmiQuote or AFL Wizard first.

AmiQuote can be launched e.g. from the Windows Start menu or by double-clicking on Quote (Quote.exe) program in AmiBroker/AmiQuote folder.

register

Once the program is running, we need to enter the unlock codes into Help->Register AmiQuote menu:

register

Then we can enter our name and the unlock code, then press Update button.

register

AFL Code Wizard can be launched from Analysis menu inside AmiBroker:

register

After the program is launched it’s necessary to select Help->Registration details item from the menu.

register

Then we can enter our name and the unlock code, then press Update button.

register

Calling custom user functions in our code

AFL language allows us to define reusable functions that can be used in our formulas. The following chapter of the manual explains the procedure in details: http://amibroker.com/guide/a_userfunctions.html

When we want to call such function in our formula, we should add function definition into our code, so AmiBroker could identify and interpret custom keyword properly. Consequently, if we use the function in multiple chart panes, each of the formulas should contain the function definition first.

// custom function definition
function myMACD( array, fastslow )
{
   return 
EMA( array, fast ) - EMA( array, slow );
}

// use of function
PlotmyMACDHigh1226 ), "Open MACD"colorRed );

Since we may potentially define a large group of our own functions, pasting the definitions manually may not be very convenient. To avoid that, we can use #include statement and group our definitions in a separate AFL file which will be called with a single statement from our main code.

To create such file we should do the following:

  1. Create a new formula. The preferred location is in Include folder in chart windows, we can in fact choose any custom location of the file.
    include

  2. We can also rename the file to a descriptive name, for example myfunctions.afl:
    include

  3. Now we can edit the file and paste our function definitions, then save the file:
    function myMACD( array, fastslow )
    {
       return 
    EMA( array, fast ) - EMA( array, slow );
    }
  4. Now in our main file we can use only a reference to myfunctions.afl file:
    // include our definitions
    #include <myfunctions.afl>

    // use of function
    PlotmyMACDHigh1226 ), "Open MACD"colorRed )

We don’t have to specify the path, because we saved our formula in the folder, which is specified as a ‘default include path’ in Tools–>Preferences–>AFL:

include

In other cases we should provide full path to the file – #include is a pre-processor command, therefore this time we use single backslashes in the path:

#include “C:\Program Files\AmiBroker\AFL\common.afl”

More information about include command can be found at:

http://www.amibroker.com/guide/afl/_include.html

Where does AmiQuote save downloaded data?

AmiQuote is a companion program shipped with AmiBroker, which allows data from free resources, such as Yahoo Finance, Google Finance and others. Since it is a separate application, then it can work independently from AmiBroker and it saves data in text files stored in Destination Folder defined in Tools->Settings window:

aq download folder

AmiQuote can also communicate with AmiBroker using OLE automation and automatically import downloaded data into AmiBroker if Automatic Import option is selected:

aq download folder

AmiQuote will import data to the database, which is opened in AmiBroker at the time of import.

Additionally, if more than one instance of AmiBroker is opened at the same time with different databases loaded, then AQ will communicate with the instance that was launched first and will import data into the database opened in this instance of AmiBroker.

Do not exceed real-time symbol limit

When we subscribe to a real-time datasource, such as eSignal or IQFeed – our subscription package determines how many symbols we can access in realtime at the same time. The plugin configuration in File->Database Settings->Configure should match the subscription limit.

IQFeed:

rt config

eSignal:

rt config

As it is explained in the users guide here: http://www.amibroker.com/guide/h_rtsource.html – although AmiBroker is able to handle more symbols in the database than the streaming limit, we should not really exceed the RT subscription limits in continuous screening during session hours.

This is because if we do otherwise and try to access more symbols than our subscription covers, then it would requires lengthy process that includes:

  1. removing the oldest symbol from the streaming list
  2. adding the new one
  3. triggering backfill for the newly added stock to fill the historical data from last valid update that we already have
  4. streaming and displaying RT data.

Then such process will be repeated for each new symbol that is included in screening. As a result, that might cause various problems with the data source not able to handle that many backfill requests in a short time, additionally data-vendors may be pro-actively protecting their servers from abusing the streaming limits this way.

Therefore – it is highly recommended to stay within the subscription limits for real-time operation and scanning to avoid problems.

Long-only rotational back-test

Rotational trading is a kind of backtest where you trade by switching positions between various symbols based on their relative score instead of traditional buy/sell/short/cover signals.

Since there are no signals used, only PositionScore assigned to given symbol matters.

It is worth noting that in case of rotational test, the Positions field in General tab of the Analysis settings is ignored. It is only used for regular backtests that use actual buy/sell/short/cover signals.

In the rotational mode the trades are driven by values of PositionScore variable alone.

In particular:

  • higher positive score means better candidate for entering long trade
  • lower negative score means better candidate for entering short trade

As you can see the SIGN of PositionScore variable decides whenever it is long or short.

Therefore – if we want to test long-only system in rotational backtesting mode, then we should use only positive values in PositionScore variable. For example – if we are trading a system, which uses 252-bar rate of change for scoring purposes:

SetBacktestModebacktestRotational );
SetOption("MaxOpenPositions",5);
SetOption("WorstRankHeld",5);
SetPositionSize20spsPercentOfEquity );
PositionScore ROCClose252 );

Then, to trade only long positions, we should change PositionScore defintion for example to:

PositionScore 1000 ROCClose252 ); // make sure it is positive by adding big constant

This way our scores will remain positive and that will effectively disable short trades.

More information about the rotational mode of the backtester can be found in the manual: http://www.amibroker.com/guide/afl/enablerotationaltrading.html

Next Page »