GPX, QGIS, and a Bit of Creativity
- Jonathan
- Jan 6, 2020
- 9 min read
Updated: Jan 7, 2020
To give credit where credit is due, this first started with this Reddit post: https://www.reddit.com/r/dataisbeautiful/comments/dp5tda/oc_i_cycled_through_all_the_streets_central_london/
My version after some hours of trial and error:
davisvilums had recorded all of their rides in London with the goal of covering all of the streets that they could, and judging by the final product, they had done a pretty good job. This caught my eye because I have been religiously recording my runs over the past 3 years and wanted to share my data as well. I started out on a quest to try and recreate what he had done to the best of my ability. Coming into this project I had some unofficial Python programming knowledge including using the Strava API and a general love of data. I hadn't worked with GIS at all (and didn't know what it was).
Retrieve and Prepare Data with Python
To recreate what they had done, I first had to collect every GPX file from over the past year from my own activities. I didn't know of a way via the Python Strava API to collect full GPX files, so I manually downloaded each GPX file from the Garmin Connect website. If you pull up an individual activity, there is a gear icon in the top right corner of the screen and an option to export it as a GPX file. You could do this with Strava as well, I don't see why that would be an issue. It's important to note I only exported activities that would be relevant to my map and to not export activities outside of my wanted "field of view".
This by far was the most tedious step in the entire process, but it was the best way I knew to get the data. Luckily each activity downloads as activity_1234567890.gpx and they appear in chronological order when sorted alphabetically. This helped out greatly.
After installing QGIS and being completely confused, I tried some of the GPX plugins. None of these could do what I needed them to, so I created a python script which would pull a directory full of individual GPX files and would spit out a .csv file which I could work with. The python script would use as few or as many files as it could find and would return a single .csv
Make sure to change the relative directories used, but this should accomplish what we are looking for. I started using some of these libraries back when I was graphing a lot of GPX files collected after hikes and measured them up against potential new hikes.
There are some added values calculated which will come in handy later. These are mainly the "activity" int which correlates the points to the activity they are from in numerical order, the "display_time" string which displays the points time in "%Y-%m-%d" for output to the screen and the "point_no" int which holds the total point number across all of the activities. Once you have run and created your .csv file you are ready to head back over to QGIS.
QGIS and Importing Data
There are a couple of plugins I used which turned out to be very helpful, they can be managed by navigating to the top menu bar Plugins > Manage and Install Plugins...
TimeManager - used to display portions of the csv file at a time
QuickMapServices - used to download maps from the web for display
Openlayers - alternative maps through Open Street Maps
I wanted to have a very dark background on my map compared to what I had first seen, to do that I went to
Web > QuickMapServices > Search QMS
If you search for "Dark Matter" you should be able to double click the first entry to add the map layer to QGIS which displays in the bottom left hand window. I found that QGIS is similar to GIMP/Photoshop in that you work a lot with layers and the layer order determines what order you see information. The Dark Matter map is not used in my final products, but is used in my more recent projects. There are many maps that you could use - particularly through Open Street Maps and the OpenLayers plugin.
To add a layer which contains all of your data points, first go to
Layer > Add Layer > Add Delimited Text Layer
Change layer name to something like "2019" and repeat the process to add a second duplicate layer. Name this one "2019 HL". 2019 will serve as a base layer to show all of our aggregated data, the 2019 HL layer will just show the most recent run which is being shown on the screen - HL stands for highlight.
Now you should see your map with all of your data on it in a jumbled mess. If you can see the map but not your runs, you can right click on the layer name and select "Zoom to Layer" to properly display the map. What you see on the map is what will be on the output GIF/MP4 from TimeManager, you can resize your window by dragging to change the aspect ratio of the output.
Right click "2019" layer > Properties. Under "Symbology" (left) at the top there is a drop down for the type of coloring you want to use. The example uses "Heatmap". Radius should be set to something like 2-4 depending on taste and you can play with the "Maximum Value" based on how you would like the map to look. I also changed the Rendering Quality to "Best" to get the best picture possible - which still left some to be desired. The Color Ramp is what took me the longest to figure out.
The below file is an example of the color ramp that I had used in my above mp4 file.
Settings > Style Manager > Import/Export
<!DOCTYPE qgis_style>
<qgis_style version="2">
<symbols/>
<colorramps>
<colorramp type="gradient" tags="Colorful" name="Heatmap - Blue">
<prop k="color1" v="8,48,107,0"/>
<prop k="color2" v="247,251,255,255"/>
<prop k="discrete" v="1"/>
<prop k="rampType" v="gradient"/>
<prop k="stops" v="0.0013624;11,68,148,255:0.13079;8,81,156,255:0.3;33,113,181,255:0.4;66,146,198,255:0.5;107,174,214,255:0.6;158,202,225,255:0.7;198,219,239,255:0.8;222,235,247,255:0.9;247,251,255,255"/>
</colorramp>
</colorramps>
<textformats/>
<labelsettings/>
</qgis_style>
Save the above code as an XML file and you should be able to import the color ramp that I had created. This color ramp has a transparent section in the very start which indicates what color overlay the entire map will have - I'm not sure why this is but it took some time to figure out. The rest of the color ramp progresses from 1 repeat (dark blue) to many repeats (white). I found this was easiest to see over time rather than the other way around.
At the bottom of the Symbology page, select the "Layer Rendering" triangle and change the "Layer" dropdown to "Multiply". This will give you the heatmap coloring you are expecting.
Now you have set up your 2019 layer style, you need to change the 2019 HL style. This HL layer will basically only show the last/current track that you had done so it is easier to see on top of all of the other data.
You can style this one much more freely, but in the symbology page I would select a small dot type icon and make sure that you turn the "Stroke Style" to "No Pen" and then change the fill color to one that you would like (white for me). Also, I would make the line width of the 2019 layer at least 1 larger than the 2019 HL layer.
Make sure that the 2019 HL layer is ABOVE the 2019 layer in the bottom left, and of course that they are both above the Dark Matter layer (or other map layer if you'd like).
Animating Our Data
Next we are going to use the Time Manager Plugin. If you do not see the slide bar title Time Manager at the bottom of the screen, you can hide/show the plugin through
Plugins > TimeManager > Toggle Visibility
Now on the Time Manager Plugin navigate to Settings > Add Layer. Select layer 2019 and set the start time to "Activity" which should be chosen as default. This chooses which data field will be used to track time. Each incremental number of 1 correlates to 1 second in the TimeManger plugin, this means each activity will show up at once since all of the datapoints for the first activity have a value of 1 in the "activity" field.
Also, for the 2019 layer, make sure to check the "Accumulate features" checkbox near the bottom of the window. This will ensure that as the Time Manager goes through different activities, it will build upon the previous activities.
Add a second layer to Time Manager the same way for 2019 HL, but do not check the "Accumulate features" box as you only want this layer to display the most recent activity.
Once that is done, make sure Time Manager is enabled by looking at the power symbol in the top left of the Time Manager window. By default the time frame size is set to 1 minute. Since each of our activities take up the place of 1 second, the map will now be showing your first 60 activities. You want to change this to 1 second and you should be looking at a map with your first activity highlighted in white with a blue stroke around it.
If you want to resize your window now that would be a good idea, simply toggle the power icon in the Time Manager to see how large the picture will grow. This will not show the end product as it is showing the 2019 HL layer in full and with the plugin you will only see one activity at a time from the HL layer. If you want to see the finished product, turn Time Manager on and slide the slide bar all the way to the right. This will show you what the output will be at the last frame.
Once you are satisfied, make sure the slide bar is all the way to the left and that the Time Manager is enabled. Select the Export Video option. Choose an output folder that you know and select "Frames Only". There is a built in creator for an animated gif, but I found that it was difficult to get working. Also, make sure to clear previous frame files in directory to remove old files.
This will process one frame (activity) at a time and create a .png file for each activity in your chosen folder. This will take a while to run depending on the strength of your computer and the number of activities that you have.
MP4 Creation
Once completed, you should have a directory filled with frames of your potential MP4 file. You could also uses these frames to create a GIF, but I found that format to be far too large to be practical. I would go over these and make sure they look correct. You can always go back to your layers to tweak settings to change the look of your final heat map.
There are many ways to stitch all of these files together, I am using OSX and chose to write a script which utilizes the ffmpeg tool. The script is written in bash.
The above script will need some modification to fit your folder structure, but it will essentially ask for a filename and FPS to display your map at. The higher the FPS, the faster the final product will be. There are a couple of other flags used in here, mostly based on situations I have run into. This accounts for your png files having an odd number of pixels as well as if your png file names have 4 or 5 characters. This could clearly be written better, but until now it was just for my purposes.
Some of the other flags are used to make it an "Instagram-able" MP4 file. I'm not exactly sure why this is needed, but it makes it into a nice format that my iPhone can see without issues as long as I keep it below 90FPS. This took some research into the ffmpeg tool, but I'm happy overall with the final product.
And that's it! You should now have an MP4 file which shows all of your had work and determination. Feel free to comment below with what you can come up with!
This below version is from my 2018 running data. Instead of creating an animation with a static interval between activities (1/second) this one uses the display time as the time variable. This creates an animation which has pauses which represent my actual pauses in run days. This allows me to use the time indicator as the time codes in Time Manager are correct. I believe this is more along the lines of what was done in the Reddit post, but I personally prefer the more fluid look of using a static interval.
I did run into a few issues with some data where Time Manager would fail to export the png file and would repeat itself over and over around 3 activities. I found the only way around this was to find the activity causing the issue by looking at where Time Manager was failing and removing that activity entirely. I haven't been able to figure out why the activity was rejected or what actually causes the issue.
I do have some additional modifications I have made to make some different visuals which I will post about more in depth in Part 2.
Thanks for reading!
Comments