Dear Konrad,
Categories are indeed constrained to be global, partly by technical reasons, but partly for computational practicality.
Indeed, imagine that you have a 5 sequence tree (with 7 branches) and you you let each branch have it's own omega (or any other category variable). Further suppose, that each category is discretized into 4 rate classes. In order to compute the likelihood of this tree, one needs to compute the expected value of the likelihood function over all omegas. But even in this simple case, the computation involves 4^7=16384 likelihood evaluations, i.e. it very quickly becomes impratical.
I could be misunderstanding what you are proposing though
What you can do instead, is assign one category to some branches and another to others (e.g. internal versus terminal branches) etc.
The way to do this in HyPhy is as follows:
(1). Define two category variables (c_1 and c_2)
(2). Define two models (say, M_1 which depends on 'c_1' and M_2 which depends on 'c_2')
(3). Build a tree with explicit model assignments
Code:Tree t = (a{M_1},b{M_1},c{M_2});
Cheers,
Sergei