Anthony C. Ijeh (editor) - Crime Science and Digital Forensics: A Holistic View

---

The editors acknowledge and thank all authors and contributors for their effort and contributions.

As editors, we would also like to thank CRC Press Publications; especially the team that worked for the book to be completed.

Finally, the editors thank all readers of Crime Science and Digital Forensics: A Holistic View, for their interest in the book and encourage them to continue to send their invaluable feedback and ideas for further improvement of the book.

Rafael Padilha, Fernanda A Andal, Lus A M Pereira and Anderson Rocha

---

Institute of Computing, University of Campinas

Through our devices, we are able to record and share content, fomenting the generation of information about what happens around us. An example is a massive blaze that struck Notre Dame Cathedral, in April 2019, during which millions of images and videos were recorded and shared on the Internet by the media at large and everyday citizens. All content is potentially important to document our history and also as a source of information for investigations. However, in this vast pool of information, we do not promptly know how the different items connect to each other to unravel the whole story. This chapter presents how to achieve the X-coherence for an event, i.e., sort an unstructured collection of images and videos, in space and time, allowing the analysis of the corresponding event as a whole. To this end, we present machine learning techniques to automatically position each media piece in space and time, considering the Notre Dame fire as the backdrop. We employ convolutional neural networks to capture visual clues in the Cathedrals structure and its surroundings and how the scenes appearance alters as time progresses. Finally, this chapter also presents a comprehensive way of visualizing and exploring the organized data.

We live in a connected world, where events taking place across the globe often have the power to impact our daily lives. Such events reach us and are broadcasted by us in the form of visual and textual content, generating a massive unstructured pool of data. A paramount example is a recent blaze that struck the Notre-Dame Cathedral, an ancient Parisian architectural and religious symbol. In April 2019, a fire tore through the cathedral, devastating large parts of its structure and spire (). People worldwide followed the tragic event through millions of images and videos that were shared by the media and everyday citizens, at the same time that part of the structure turned to dust.

The content generated from unprecedented events, such as the Notre-Dame fire, is potentially important to document our history and also as a source of information for investigations. This content, however, naturally comes from heterogeneous sources, often times lacking proper structure, as to where and when it was captured and how it is connected with other pieces of information. An effective way of understanding an event is to create a structure for this type of data by constructing a unified space where all pieces of related information can be coherently organized.

Figure 1: Notre-Dame Cathedral during a massive blaze that destroyed part of its structure and spire, in April 2019. Credit: Milliped [CC BY-SA 4.0] and Francois Guillot/AFP/Getty.

The process of synchronizing data by positioning its items in a common consistent system is called X-coherence (). The idea is that by properly organizing an events data, it is possible to navigate through it, understand how it unravels in time and space, and even observe the timeline for a specific person or object within the same context. X-Coherence could aid, for instance, fact-checking and the mining of suspects in forensic investigations.

Although achieving X-coherence for an event is beneficial, performing it manually might be an unfeasible task, given the large volume of data to be processed within a reasonable timeframe. Even worse is the possibility of a person interfering with the process by introducing unconscious bias, consequently invalidating the constructed space and any conclusions drawn from it. One such example is an unprecedented manhunt that followed the Boston Marathon Bombings, in April 2013. After two bombs exploded near the marathon finish line, several people tried to find the bombers by analyzing thousands of images and videos captured at the event, which turned into a major failure ().

A way to overcome these problems is to automate the X-coherence process as much as possible, allowing the effective organization of all data generated from an event, while also mitigating possible biases. To this end, in this chapter, we show how state-of-the-art machine learning techniques can be used to achieve the X-coherence, by automatically sorting an unstructured collection of images, in space and time, while also providing a joint visualization system to understand the event as a whole.

We consider the Notre-Dame fire as the backdrop of our research and, for this, we gathered images and videos of the tragic event from social and mainstream media. For a given subset of annotated imagery from the same event, we train convolution neural networks (CNNs) to capture important visual clues to place each data item in space and time. The trained models can then be used to organize new images as well as placing them in a common coherent space for visualization.

Works in the literature focus on different tasks related to X-coherence. Considering the spatial ordering of images, presented an interface to explore unstructured collections of photographs, by computing the viewpoint of each photograph as well as a sparse 3D model of the scene.

For temporal ordering, proposed a method to splice together sets of near-duplicate shots, thus aligning them temporally, in order to reconstruct a complete video sequence for an event.

Some works focus specifically on how to visualize data from events (), avoiding information overload and failure to indicate overall trends when analyzing events.

Different from these works, our goal is to explicitly and thoroughly tackle the problem of achieving the X-coherence for an event. For this, our main contributions are:

• A method for positioning images of an event in space, assigning them to different cardinal directions in relation to the event center;
• A method for placing images of an event to specific timeframes determined by important sub-events;
• A visualization approach to coherently represent images of an event in space and time.

Although we demonstrate these contributions on a specific eventthe Notre-Dame Cathedral fire, we emphasize that the proposed methods and methodology can be applied to similar events, in which images are captured in different positions and angles around and throughout an event.

To solve the X-coherence for the Notre-Dame fire event, we divide the process into three steps: spatial classification, temporal classification, and visualization ().

In the spatial and temporal classification steps, which can occur simultaneously, the input image is positioned in space and time by the analysis of its content. For spatial classification, a cardinal direction is assigned to the image depending on the location where the image was captured in relation to the cathedral. During temporal classification, our method detects when the image was taken in relation to important sub-events, such as the spire collapsing and the fire being extinguished.