Jun Chen - Detecting Regime Change in Computational Finance: Data Science, Machine Learning and Algorithmic Trading

Jun Chen
Edward P K Tsang

The book Detecting Regime Change in Computational Finance is the first book of its kind to build on the framework of Directional Change. The concept of Directional Change is a new field of research that rethinks the way in which data is represented and sampled. The methodology opens a whole new area of research, which raises fundamental questions about model-design and proposes an alternative approach to it. Instead of sampling data according to the pulse of physical time, the new methodology uses the Directional Change operator to filter the data and pick out the truly relevant data points. The new approach was developed because established methodologies have not rendered satisfactory results and are not fulfilling expectations. Market prices are not recorded at fixed intervals and time does not flow at the same pace during the course of 24 hours and over a week, where different time periods are not equally meaningful. The Directional Change framework is a methodology to select the time points that are significant.

I am deeply grateful for the many years of friendship and support of Professor Edward Tsang of the Centre for Computational Finance and Economic Agents, Essex University. We had a great working relationship! Thank you, Edward, for sharing your enthusiasm with the students, and encouraging them to explore the new framework, and to dig deeper to achieve a better understanding of the benefits of the new approach. Many thanks to you, Jun Chen, for embarking on an exciting research project and making your work accessible in a book.

The great technological innovations in natural sciences in the last century were possible because the scientists were data driven and ready to embrace new theories. This openness led to the development of computers, the Internet technology and mobile telephony. Technological innovation has had a transformative impact on modern society that can be experienced day by day. The ability to make telephone calls with a mobile from anywhere in the world, is just one example of a technical feat that was inconceivable only 50 years ago. Scientific progress in social sciences did not match the progress made in natural sciences. The disappointing performance of social sciences is painfully obvious in the many shortcomings of the modern economy and the fallout of the Coronavirus pandemic. The depletion of natural resources and the evermore threatening news of global warming are additional evidence of the failure of the economic system to allocate resources in a sustainable way, as well as the lack of progress in social sciences.

The catalyst for progress in natural science is the data-centric methodology that has led to unexpected discoveries. The Michelson Morley experiment conducted in 1887, which paved the way for the discovery of the theory of relativity illustrates how data-driven natural scientists are, as one example. At a time when technology was in its infancy and exact measurements were extremely difficult, the physicists Albert. A. Michelson and Edward W. Morley went about measuring the speed of light in different directions. The outcome of the MichelsonMorley experiment was a big surprise. The results showed beyond doubt that the speed of light is a constant and invariant.

This discovery debunked existing theories. Scientists were open to accept defeat and ready to question their basic assumptions. Their openness paved the way for the discovery of relativity theory and, indirectly, quantum theory.

The MichelsonMorley experiment is an example of the benefits of a data-centric research methodology and the necessity of putting in established theories when empirical evidence does not align with the inferences of existing theories. We need to embrace new theories even though they may challenge existing assumptions that are deeply ingrained in the public perception.

I was first introduced to the deficiencies of economic theories while studying at Oxford in 1980. James Mirrlees, who would later receive a Nobel Prize in economics, explained in an economics class that established growth models were unsatisfactory and did not answer the real problem. I still hear his words explaining that growth models derail if one screwdriver breaks and is unstable. This was deeply troubling. If growth models cannot account for machines breaking down, or other things going wrong, then these models miss out on an essential feature of economic and social progress. The history of humankind has one constant: things go wrong and unexpected negative events are a daily occurrence. Wars and human errors are omnipresent. Economic theory has to account for and be able to incorporate setbacks into its modelling framework.

_______________

Encouraged by the intellectual openness of the Oxford academic environment, I started a long journey of investigation and research, going back to the first principles of economics in search of a more stable model framework. In the process, I started to develop a relativistic framework that I called system theory to explain economic phenomena. On my journey of developing a new theory, I started to investigate the nature of time and its definition, and eventually realised that there are some very basic open questions.

Economic and finance models use historical data as inputs to test and validate the models. The validation process is an essential steppingstone in model building. It is there that basic questions can arise. Take a simple example of computing the price average of the EUR vs. the USD since the beginning of the year. How should the data be sampled? Should prices be sampled every day, including Saturdays and Sundays, or only on weekdays? How should public holidays such as Christmas and Easter be dealt with? How should one account for the holiday schedules of the different countries? At what time of the day should the data be sampled? What frequency of sampling is appropriate? How should one deal with intra-day price data? Should the researcher include in the computation every price recorded, or restrict the analysis to a fixed time interval of minutes or hours? Some financial instruments are traded very actively and have price updates every second or more, whereas other instruments are traded infrequently and have only a few updates per day. The end result of the computation depends on the specific decision of how to sample data.