Mapping Layers of Abstraction¶

Real-world data, like the data sets we are using for the project, can be quite complicated, often involving combinations of lists and dictionaries. It is easy to get lost in the details of how the data is organized and how it can be accessed. This is often why the data is first filtered to produce a simpler data set to be used for further computation or visualization.

An aid to understanding the organization of a complex data set is a graphical depiction - a data map. The data map can be thought of an outline of the data set, showing the structure of the data without the detail of the actual data. The data map also does not show the detailed syntax of the data structure; only the structure of the data set is shown. Concretely, the map gives guidance on how to access the parts of the data of interest.

Before dealing with a project-scale degree of complexity, a simpler example will be used to illustrate the basic form of a data map. Two different styles of data maps will be shown. Next, two tools for constructing a data map for a more complex data set will be shown.

Data Map for a Simple Structure

Two different styles of data maps will be drawn for a simple example. The example is the following list of dictionaries that represent a multi-city forecast of temperature and precipitation for several U.S. cities. The Multi_City_Forecast data is organized as a list of 5 elements. Each element of the list is a dictionary with two keys: 'place' and 'forecast'. The values of these two keys are themselves dictionaries with two keys. Notice that all the keys in this example are written as strings with quote marks (e.g., 'place' and 'forecast').


Multi_City_Forecast = [
 {'place' : {'city' : 'Blacksburg, VA' , 'zip code' : 24061 } , 'forecast': {'temperature' : 76 , 'precipitation' : 0.1} } ,
 
 {'place' : {'city' : 'Seattle, WA' , 'zip code' : 98101 } , 'forecast': {'temperature' : 62 , 'precipitation' : 1.1} } ,
 
 {'place' : {'city' : 'Miami, FL' , 'zip code' : 33101 } , 'forecast': {'temperature' : 86 , 'precipitation' : 0.0} } ,
 
 {'place' : {'city' : 'San Jose, CA' , 'zip code' : 95103 } , 'forecast': {'temperature' : 79 , 'precipitation' : 0.0} } ,
 
 {'place' : {'city' : 'New York, NY' , 'zip code' : 10036 }, 'forecast': {'temperature' : 85 , 'precipitation' : 0.5} } 
]

One way to represent the structure of this data is a graphical drawing as shown in the figure below. At the top of the diagram is a rectangle "list of" indicting that the data is organized into a list. The arrow leading out from this list will show one (a typical) element of the list. Only one element of the list needs to be shown because all elements of the list have the same structure. The arrow leading from the list box points to a typical list elment. In this diagram a dictionary is represented by a box with rounded corners within which are written the dictionary's keys. As shown, the typical list element is a dictionary with two keys: 'place' and 'forecast'. There is an arrow from each key leading to a graphical description of the value associated with that key. As shown, the value of the 'place' key is also a dictionary having the two keys 'city' and 'zip code'. Similarly, the arrow from the key 'forecast' shows that its value is a dictionary having the two keys 'temperature' and 'precipitation'. finally, basic types (string, int, and float) are shown as text. This means that the arrow leading from the key 'city' to the text string means that the value associated with the 'city' key is a string. For example, in the first element of the list shown above the value associated with the key 'city' is the string 'Blacksburg, VA'. As shown in the diagram the values associated with the keys 'zip code', 'temperature', and 'precipitation' are values of type int, int, and float, respectively.

Data Map in diagram form

Another way to represent the structure of the Multi_City_Forecast data is a text outline as shown below. In this case indentation is used to show the structure of the data. The first line indicates that the data is organized as a list where each element of the list is a dictionary. The notation "{}" is used to indicate a dictionary. One level of indentation is used to show that each list element dictionary has two keys, 'place' and 'forecast' each of which is itself a dictionary. A second level indentation is used to show that the dictionary associated with the 'place' key has itself two keys: the key 'city' which has a string vaue and the key 'zip code' which has an int value. Similarly, a second level indentation is used to show that the dictionary associated with the 'forecast' key has itself two keys: the key 'temperature' which has an int vaue and the key 'precipitation' which has a float value.

Data Map in outline form

There are two important comments about the data maps. First, there is nothing special about the way these data maps have been drawn. A perfectly suitable data map might be drawn using other kinds of diagrams. The important thing is that we have some kind of representation to act as a guide when dealing with complex data structures. Second, the data map can be "read" to show exactly how to access a part of the data. For example, suppose that we wanted to output all of the zip codes in the data. An iterationis used to accesses each element of the Multi_City_Forecast list one a time. Each such element is a complex entity, but reading the data map from top to bottom - following the arrows - yields this code.

for forecast in Multi_City_Forecast: 
    where = forecast['place']
    zip = where['zip code'] 
    print(zip)

In this code, the variable where is assigned the value associated with the key 'place'. This value is also a dictionary as shown in the data map. The zip code value can be accessed using the key 'zip code' in the where dictionary.

It is also possible to write code with a sequence of keys, one after the other. The following code shows how this is done to access the zip code value.

for forecast in Multi_City_Forecast: 
    zip = forecast['place']['zip code'] 
    print(zip)

Mapping a Complex Data Set

Trying to examine the complete data set directly as was done in the simple example above is not practical for a large, complex, and realistic data set. Drawing a data map for such data sets requires the aid of some tools to examine the data set. Each of the data sets in the Corgis collection has an Explore Structure option that can be used to view and map the structure of the data set. The figure below the Corgis page for the Earthquake data set. Just below the Explore Structure heading, a red arrow is shown pointing to the button "Explore earthquake data". When this button is pushed a new window appears showing the top level structure of the data set. The read arrow labelled 1 points to this new window in the figure.

Constructing a Data Map - Step 1

The Explore Structure window shows:

Title (at the top): "List" at the top indicating that the top level structure is a list of values.
Index : the value 0 indicates that the first element of the list will be shown as the "typical" element.
Type: indicating that each element in the list is a dictionary.
Example Value: clicking on the icon will show an example value.

To build the data map we need to know the structure of each of the elements of the list. We know that each element is a dictionary from the "Type" field but we also need to know the keys in this dictionary. Clicking on the icon in the "Example Value" field produces a new window that shows the structure of the dictionary. This is illustrated in the following figure.

Constructing a Data Map - Step 2

The new window shows an element of the list. As can be seen, each element is a dictionary (note the title of the window is "Dictionary". The window displays the 4 keys (note the number in the title of the widow). These four keys are: 'impact',' 'id', 'location', and 'time'. In the Type column we can see that the value of the key 'id' is a string and that the value of each of the other keys is a dictionary. We can now use this information to build the intial part of the data map. The figure below shows this initial, though incomplete, data map in both the graphical and outline forms.

Initial Data Map of Earthquake structure

We can continue using the Explore Structure to show the structure of the value associated with the key 'impact' by clicking on the icon in the "Example Value" column. This is shown in the next figure. The numbered red arrows shows the sequence of events. The newest window shows that the 'magnitude' key has a float value, the 'significance' key has an int value, and the 'gap' value has a float value. Notice also that there is a "Comment" column giving a description of what the value means. For example, the "Comment" for the "magnitude" explains that this is a measure of the size of the earthquake at its source and continues with a discussion of the magntiude scale used to measure earthquake magnitudes.

Constructing a Data Map - Step 3

All of this information seen so far can be collected into a more complete data map as shown in the following figures.

Second Data Map of Earthquake structure

We can continue using the Explore Structure to map the Earthquake data structure until all the entire structure has been mapped.