Market Microstructure Patterns Powering Trading and Surveillance Agents

Text-only Preview

Journal of Universal Computer Science, vol. 14, no. 14 (2008), 2288-2308
submitted: 30/9/07, accepted: 30/4/08, appeared: 28/7/08
Market Microstructure Patterns
Powering Trading and Surveillance Agents

Longbing Cao
(University of Technology, Sydney, Australia
[email protected])

Yuming Ou
(University of Technology, Sydney, Australia
[email protected])

Abstract: Market Surveillance plays important mechanism roles in constructing market
models. From data analysis perspective, we view it valuable for smart trading in designing legal
and profitable trading strategies and smart regulation in maintaining market integrity,
transparency and fairness. The existing trading pattern analysis only focuses on interday data
which discloses explicit and high-level market dynamics. In the mean time, the existing market
surveillance systems available from large exchanges are facing crucial challenges of
diversified, dynamic, distributed and cyber-based misuse, mis-disclosure and misdealing of
information, announcement and orders in one market or crossing multiple markets. Therefore,
there is a crucial need to develop innovative and workable methods for smart trading and
surveillance. To deal with such issues, we propose the innovative concept microstructure
pattern analysis
and corresponding approaches in this paper. Microstructure pattern analysis
studies trading behaviour patterns of traders in market microstructure data by utilizing market
microstructure knowledge. The identified market microstructure patterns are then used for
powering market trading and surveillance agents for automatically detecting/designing
profitable and legal trading strategies or monitoring abnormal market dynamics and trader’s
behaviour. Such trading/surveillance agent-driven market trading/surveillance systems can
greatly enhance the analytical, discovery and decision-support capability of market
trading/surveillance than the current predefined rule/alert-based systems.

Keywords: market microstructure pattern, data mining, agents, market surveillance
Categories: I.2.6, H.1.1, M.0, M.1
In many types of markets such as capital and electricity markets, market surveillance
plays important mechanism roles in designing market models and business rules, as
well as regulation roles in maintaining the market integrity, transparency and fairness
[Dehdashti, 05; UN, 05; O’Hara, 01]. Many current program trading systems are
based on predefined trading strategies. The existing research on trading pattern
analysis mainly focus on interday data. One the other hand, the existing market
surveillance systems usually rely on surveillance rules for alerting of suspect findings
in the market. Such rules are predefined and based on business rules. Additional
surveillance rules come from statistics and reporting results, which can capture more
sophisticated abnormal trading behaviour and market movement. These rules play

Cao L., Ou Y.: Market Microstructure Patterns ...
important roles in filtering obvious offences against market business rules, regulation
rules, and explicitly exceptional market dynamics.
However, the existing trading pattern analysis loses the in-depth information
hidden in the market microstructure. With regard to market surveillance rules, the
existing systems available from large exchanges are facing crucial challenges of
identifying diversified, dynamic, distributed and cyber-based misuse, mis-disclosure
and misdealing of information, announcement and orders in one market or crossing
multiple markets. Such challenges cannot be handled by the existing systems and
techniques usually used in exchanges.
The current trading pattern analysis and price movement analysis mainly focus
on interday data, in particular, closing prices. The resulting analytical results are not
workable for real-time market surveillance because they cannot catch and filter the
microstructure behaviour every second of every day. In fact, currently, there is no
analytical work reported on analyzing tick-by-tick data for scrutinizing either
profitable trading strategies or abnormal trading behaviour. There is a crucial need to
develop breakthrough methodologies and techniques to discover hidden knowledge
on the market microstructure data under the increasing financial and trading
In this paper, to deal with the above issues, we study market microstructure
behaviour surrounded by market microstructure data. We propose the innovative
concepts and corresponding approaches to identifying market microstructure patterns.
Market microstructure behaviour consists of investor’s actions and interactions
among investors in one market or crossing multiple markets, as well as their
embodiment in market dynamics. Microstructure pattern analysis studies trader’s
behaviour patterns in market microstructure data by following and involving market
microstructure theories.
General market microstructure patterns consist of positive and negative market
microstructure patterns in time series and activity sequences. In addition, hybrid
patterns and more advanced activity microstructure patterns may be identified in
market microstructure time-series and sequences.
Microstructure patterns are very useful for smart trading or surveillance. If the
patterns legally make sense, they are helpful for smart trading. Some other
exceptional microstructure patterns may reflect abnormal trading behaviour in the
market. The identified microstructure patterns can then be used for powering market
trading/surveillance agents that automatically detect/monitor the market dynamics and
trader’s behaviour patterns. For instance, market surveillance agents can be developed
for market surveillance officers and management teams to present them alerts and
indicators of abnormal market movements. Such market microstructure pattern-driven
market trading/surveillance systems can greatly enhance the analytical, discovery and
decision-support capability of market trading/surveillance than the current predefined
strategy/alert-based systems.
In fact, microstructure behaviour can be seen in many financial applications, for
instance, derivative market, foreign currency exchange market, and index exchange
market. The innovative methodology of microstructure pattern analysis presents new
and powerful approaches to enhancing the existing market surveillance systems and
trading performance.

Cao L., Ou Y.: Market Microstructure Patterns ...
Market Microstructure and Data
Market Microstructure
Market microstructure [Madhavan, 00; Harris, 03] is a branch of finance concerned
with the details of how exchange occurs in markets. The major thrust of market
microstructure research examines the ways in which the working processes of a
market affects determinants of transaction costs, prices, quotes, volume, and trading
behaviour1. Microstructure theory focuses on how specific trading mechanisms affect
the price formation process2. It is devoted to theoretical, empirical, and experimental
research on the economics of securities markets, including the role of information in
the price discovery process, the definition, measurement, control, and determinants of
liquidity and transactions costs, and their implications for the efficiency, welfare, and
regulation of alternative trading mechanisms and market structures.”3 The theory of
market microstructure applies to the exchange of real or financial assets in financial
markets. Market microstructure deals with issues of market structure and design, price
formation and discovery, transaction and timing cost, information and disclosure, and
market maker and investor behaviour.
Market Microstructure Data
Transactional data recording the investor behaviour in markets obeying market
microstructure theory present a unique structure. We call such data market
microstructure data
. On the one hand, market microstructure data presents syntactic
components and representation nothing special from transactions normally
accumulated in many applications such as e-commerce data, retail data and telecom
data. On the other hand, market microstructure data does present differences. In
summary, market microstructure data presents some major characteristics that are not
usually seen in many other applications.
• The data indicates rich semantics. The semantics is led by market microstructure
theories and somehow embodied through the syntactic representation. For
instance, any order is associated with either ask or bid, and orders are likely
partnered from ask and bid sides. An ask-order transaction distinguishes from a
bid-order one.
• The data presents time frame and gradient. Market microstructure data is
associated with timeframe. Further, it also presents time gradient. For instance,
opening and closing prices are daily data, while a trading price, belonging to
intraday data, is usually associated with a particular time point in a hundredth of
one second.
• The data presents granularity dynamics. There are both interday and intraday
data in markets. Both interday and intraday data presents different granularities.
For example, stock market index consists of both intraday and interday
categories, its intraday elements also include various catalogues. Some are broad-
based while others are sector-based.

2 O'Hara, Maureen, Market Microstructure Theory, Blackwell, Oxford, 1995

Cao L., Ou Y.: Market Microstructure Patterns ...
• The data is heterogeneous. Market data consists of numerical, categorical,
sequential/serial, and textual elements. If more than one market must be involved,
then the data structures are likely different. Processing and mining such mixed
data needs effective methods.
The above characteristics of market microstructure data distinguish it from the
data usually seen in the mainstream data mining research. Such semantic information
(1) brings challenges to the existing pattern mining approaches, and also (2) provides
guideline or hints for itemset construction and pattern mining.
Typical market microstructure data can be described by the data model in Figure
1. It shows a three-dimensional model: time (t ), multiple levels of ask/bid prices
(P /
) and volumes (
) associated with a particular point of time .
Asi PBii
VAsi VBii

Figure 1: Market microstructure data
In financial markets, market microstructure data consists of basic data categories
including orders (O), trades (T), indices (I) and market data (M). It involves several
dimensions: time (t), value (v) and direction (d). Basic trading types can be classified
into buy (B), sell (S) and hold (K). To represent a general element in market
microstructure data, we use the following model:

Cao L., Ou Y.: Market Microstructure Patterns ...

where, a∈{S, I, …} is the asset traded: security (S), index (I), etc.; x∈{p, v, q} refers
to the target attribute: price (p), volume (v) and value (q) of a at time t, y ∈{O, T}; d
refers to the order direction, if y = ‘T’, d∈{B, S, K} refers to buy (B), sell (S) and hold
(K), while d = {Bi, As} if y = ‘Q’ indicating the data comes from either bid (Bi) or ask
(As) side; j = {1, …, J} refers to the number of x corresponding to time t. In addition,
investor’s actions u∈{n, l, m, w} on an order can be: add (n), delete (l), amend (m) or
withdraw (w). For instance,
refers to the price of no. 54 bid quote on the
security CBA.
Processing Market Microstructure Data
Before the mining can be undertaken on market microstructure data, the data
characteristics described in the above sections have to be properly catered through
data preparation. The semantics, granularity, timeframe and heterogeneity
surrounding market microstructure data determine the preparation tasks and guideline
for itemset construction. We discuss a few general techniques here.
The semantic information is helpful to guide us to build more meaningful series
and sequences. For instance, with order types and directions, we can construct
sequences by extracting investor’s actions in markets. As an example, in the
following, we show how to represent and construct microstructure order sequences
guided by domain knowledge.
Vector-Based Microstructure Order Representation
Considering the fact that every order follows market microstructure theory and
indicates information about order holder’s intention, the representation of such order
sequences should reflect them accordingly. In stock markets, although the values of a
particular order attribute vary from order to order, they actually reflect a trader’s
intentions, and cater for the particular stage of their lifecycles. For instance, for a
single time point, an order may present in one of the following states (s) in its
lifecycle: s∈{new, traded partly, traded entirely, deleted, outstanding}. Further, even
for the same values of a particular order attribute, they may indicate divided
circumstances that reflect investor’s varying motivation and behaviour [Shleifer, 00].
Therefore, the proper representation of an order should reflect order holder’s
intention, actions and the order’s lifecycle. For this purpose, we propose vector-based
order representation
, namely a multi-dimensional vector O represents an order in
terms of attributes that describe the above aspects. For instance, in stock market, a
five-dimension vector O(d, δ, ρ, φ, ε) is defined to represent an order. Dimension d
reflects the trade direction of an order, δ stands for the probability that an order is
traded, ρ measures the size of an order, φ represents how many trades the order leads
to, and ε reflects the balance of an order at the time of market close.
These five dimensions are defined as follows:

Cao L., Ou Y.: Market Microstructure Patterns ...






where p is the order price and is the last trade price in the market when the order is
The order vector O(d, δ, ρ, φ, ε) encloses plenty of semantics: (1) indicating the
direction, probability and size of an order to be traded, (2) reflecting an order’s
dynamics during its lifecycle. The vector actually provides a mechanism to transform
microstructure-based orderbook into vector-based order sequences. For example, the
following Table 1 and Table 2 show the orderbook data and trade data related to order

O100 28/06/2005
A123 S123
B 10.00 1000
Table 1: Orderbook data related to order O100
O078 28/06/2005
A348 S123 S
10.10 500
O102 28/06/2005
A980 S123 B
10.10 500
O067 28/06/2005
A690 S123 S
10.00 200
O100 28/06/2005
A123 S123 B
10.00 200
O089 28/06/2005
A531 S123 S
10.00 300
O100 28/06/2005
A123 S123 B
10.00 300
Table 2: Trade data related to order O100

Cao L., Ou Y.: Market Microstructure Patterns ...
Following the O(d, δ, ρ, φ, ε) specification, the above order lifecycle can be
transformed into the following order sequence: (B, δ
M, ρS, φN, ε1 .
Constructing Vector-Based Microstructure Order Sequences
Through data processing, market microstructure transactional data can be expressed in
terms of microstructure series and microstructure sequences.
Microstructure series refer to both interday and intraday time-series data of
numerical attributes, for instance, closing price series and trading price series. Time
series consist of both original and derived series. Derived time series data is of those
attributes aggregated on top of original time series, for instance, volatility series and
sharpe ratio series. For example, the following shows a buy-order price series.
{ 1
p ,..., m
p }
Microstructure sequences refer to ordinal and categorical data that is related to
investor’s behaviour. In markets, investor actions may be placing, amending, deleting
and withdrawing orders. Corresponding sequences can be constructed for each of
them. In addition, trades consist of sequences as well. For instance, we can construct
all sell-side put-order sequences as per investor.
{ 1
,..., j
S , A
S ,
Based on the above vector-based order representation O(d, δ, ρ, φ, ε),
microstructure orderbook can be transformed into order vectors. For intraday
microstructure data, an order at most lasts for one day since its generation. This
indicates the length of an order lifecycle is maximally one trading day. In addition,
orders placed by different investors indicate different intentions, beliefs and desires.
Thus it is domain-friendly to construct order sequences in terms of trading day and
order investors.
A microstructure order sequence consists of sequences of orders in vectors for
a trader within a trading day,
Ω = {O1(d1, δ1, ρ1, φ1, ε1), O2(d2, δ2, ρ2, φ2, ε2), …, Oi(dj, δj, ρj, φj, εj), …}
in which O is an order vector. Therefore, an order sequence systematically reflects
an investor’s intention, his/her order lifecycles and trading activities in a market. For
example, if a trader entered three orders (B, δL, ρL, φ0, ε-1), (B, δL, ρL, φ0, ε-1), and (S,
δH, ρM, φ1, ε0) on July 16, 2004, then the corresponding order sequence can be
expressed as:
Ω = {(B, δL, ρL, φ0, ε-1), (B, δL, ρL, φ0, ε-1), (S, δH, ρM, φ1, ε0)}.
Market Microstructure Patterns
Market microstructure patterns are identified in market microstructure time-series and
sequences. In microstructure time-series, general microstructure patterns may consist
of the following types:
• Microstructure time-series patterns
Microstructure sequential patterns identified in microstructure sequences may
consist of

Cao L., Ou Y.: Market Microstructure Patterns ...
• Positive/negative microstructure patterns:
• Sequential microstructure patterns
• Exceptional microstructure patterns
• Microstructure activity patterns
Further, in data combining time series and sequences, combined patterns may be
• Combined microstructure patterns
We interpret them respectively in the following subsections.
Microstructure Time-series Pattern Analysis
Financial market data is typical time series data. On microstructure time-series data,
the following microstructure time-series patterns may be identified.
(1) Microstructure single time-series patterns
In financial markets, basic microstructure time-series consist of price series,
volume series, value series, index series. Based on these series, derived series can be
generated such as profit series, abnormal return series, and sharpe ratio series.
Pattern analysis on such series includes outliers and trends. Outliers indicate
exceptional trading patterns, while trends present the dynamics of the normal
(2) Microstructure cross-time-series patterns
With the involvement of domain knowledge, cross-time-series analysis has
potential to disclose much richer information about market movement and suspect
exceptional activities. The aim is to find microstructure cross-time-series patterns.
We don’t expand these patterns’ discovery in this paper. Rather, our focus is on
microstructure activity pattern analysis, which to the best of our knowledge, has never
been investigated before, and has potential to in-depth understanding of market
dynamics from the perspective of market microstructure.
Microstructure Activity Pattern Analysis
By contrast to microstructure time-series pattern analysis, microstructure activity
pattern analysis is a totally new topic. We here focus only on microstructure trading
behaviour, namely an investor’s sequences of actions in one or cross markets.
Sequences of investor’s trading actions can be constructed in terms of various
strategies by introducing domain experts’ guide, for instance,
• S1: The lifecycle of an order,
• S2: The lifecycle or selected investment life interval of an investor,
• S3: The lifecycle of an investor on selected assets in one market,
• S4: The lifecycle of an investor on selected assets cross markets.
Correspondingly, microstructure activity pattern analysis intends to discover
behaviour patterns in the activity sequences. There are two ways to identify such
patterns. One is to ignore the order in the sequences, and focus on positive, negative
or hybrid activity associations. The other is to identify sequential activity patterns.
Many innovative types of sequential activity patterns may be identified.
Investor’s activity patterns may be exceptional if they are against average
people’s expectation. If this is the case, it likely indicates some abnormal behaviour in
the market. Algorithms and exception monitoring indicators can be carried over by

Cao L., Ou Y.: Market Microstructure Patterns ...
market surveillance agents. They can then detect and alert exceptional trading in real-
time and automatic manner. On the other hand, those constructive and legal trading
behaviour patterns (including exceptionally well-performed patterns satisfying legal
and regulatory rules) are useful for powering trading agents to better the investment
performance. Such trading agents can autonomously detect the market movement and
make smart trading decisions on behalf of their investors.
Let D be the microstructure data, f
, …, } be the original attributes in
j ∈ {f1
after the vector-based transformation, we get the new attributes
A = {a1, a2, ..., am} are the activity set extracted from the orderbook data by following
the vector-based order representation specification.
(1) Positive/negative/hybrid microstructure activity associations
We then define positive microstructure activity associations as {a , } (where ‘,’
i aj
indicates a simple combination without ordering) indicating the occurrence of some
activities in markets, while negative microstructure activity patterns {¬a , ¬ }
indicate non-occurrence of the itemsets. Further, combined associations may present
in form of {¬a ,
i ai+k
(2) Sequential microstructure activity patterns
Sequential microstructure activity patterns consist of trading activities in some
order, the ordering is represented by ‘;’. Similar to association pattern analysis,
positive {a ;
; ¬
} and
hybrid sequentialai
microstructure activity patterns may be found in ordinal activity set.
(3) Exceptional microstructure activity patterns
Both microstructure activity associations and sequences can be exceptional if
they are out of normal cycle. Exceptional activity patterns may present as outliers. Let
Ψ be the exceptions, exceptional activity patterns can be generalized in terms of
activity algebra: .
(4) Impact-targeted microstructure activity patterns
If the impacts of activities (denoted by the set Φ) can be defined and categorized
by domain experts, then impact-targeted microstructure activity patterns [Cao, 081;
Cao, 082] can be identified. For instance, the following forms of impact-targeted
microstructure activity patterns may be examined.
• Positive and negative impact-targeted activity pattern mining

Target occurred (
Target disappeared ( )
Pattern appearing (A)

Pattern disappearing ( )

Table 3: Positive and negative impact-targeted activity pattern mining
• Sequential impact-contrasted activity patterns
Let A be a sequence of activities, sequential impact-contrasted activity patterns
consist of a pair of patterns that are associated with A as follows (D and refer to
two datasets):


Cao L., Ou Y.: Market Microstructure Patterns ...
• Sequential impact-reversed activity patterns
Sequential impact-reversed activity patterns consist of a pair of patterns, in which
one of it is an underlying impact-targeted pattern associated with an impact, the other
is a derivative impact-targeted activity pattern that is associated with another impact,
the impact is opposite to that of the underlying pattern. Sequential impact-reversed
activity patterns can be described as follows, in which D and could be the same or
different datasets.

In fact, certain outcome-targeted exceptional activity patterns can be viewed as
one type of impact-driven activity patterns, in which the impacts refer to the
exceptions, which may be predefined or ad hoc.
Approaches for Identifying Microstructure Patterns
In this section, we briefly illustrate two approaches and corresponding case studies in
identifying market microstructure patterns. One is to detect announcement pre-
disclosure-associated volatility deviations on time series data by segmentation, the
other is to identify exceptional trading activity patterns related to stock price
Segmenting Volatility Deviation Associated with Announcement Pre-

Finance literatures have disclosed that the announcement arrival and the resolution of
its informational impact are directly related to the dynamics of the market volatility
[Andersen, 97; Mitchell, 94; Rahman, 02]. On top of such domain findings, we
develop methods [Ou, 07; Yu, 07] to segment microstructure time-series for finding
volatility deviations. Such deviations, if identified, may indicate announcement pre-
disclosure. Rules can then be defined and loaded to market surveillance agents for
automated detection of potentially exceptional company announcements on market
movements. Thus this section discusses the detection of movement changes (denoted
by turning points) on microstructure time-series.
Volatility and segmentation
The market return volatility is defined as follows.