Generating KML for Japan (or any country) for thematic mapping from a Natural Earth shapefile

KML file of Japan prefectures

Raw KML file available to download for free and to be used in any way is here.


As some of my previous blogs have pointed out I am interested in the thematic mapping concept and using these methods in SAP and other applications with KML.

http://sap-in-japan.blogspot.com/2011/01/sap-thematic-mapping-with-kml.html

The blog above covers the "simple transformation" of data into a KML file. I used the Natural Earth datasets as a base to generate the KML for SAP.

I did have intentions to cover the method I used to generate the KML earlier but have been delayed by my work related activities and the fact I am not currently at my home in Japan.

However I have gone through the details now and here is ONE way to generate KML from the Natural Earth Datasets. (How to use this KML in SAP will follow in another blog)

*As another blog entry pointed out there are Google fusion tables generated over at the thematic mapping engine available online. When I commented on Mr Sandvik's blog he pointed out that the datasets can be used but also google terms of use are also referred to. I do not like reading google's "terms of use" for other services as it always  dissapoints me when commercial aspects are not allowed. I have not read fusion tables terms of use in detail. However reading that the Natural Earth datasets are public domain and can be used for any purpose, I detail below the method I used to generate KML by prefecture for Japan.

*My Google Fusion blog for KML http://sap-in-japan.blogspot.com/2011/03/japan-kml-by-prefecture.html

Also to point out if you want online community access to geographic data and maps then the geocommons website is well worth a visit. I found lots of data related to Japan (and prefectures) that could be used, however commercial aspects are restricted.
Geocommons link searching for Japan prefectures.
 http://geocommons.com/search?mh_query=japan+prefectures

So the intention with the blog is to show a way to get KML that can be used for ANY purpose.

KML from the Natural Earth Datasets.

Summary of the software/data used.

*) Natural Earth datasets from the 10m cultural download section
The shapefile used to generate the KML by prefecture of Japan is from the Admin 1 section shown below.

I chose the "Download without large lakes" shapefile. I have not checked whether it actually contains large lakes or not however the name of the download made me select it. The actual name of the shapefile "10m_admin_1_states_provinces_lakes_shp.shp"  and the fact that the download is a larger file than the one without large lakes - does make me doubt the description! However the KML at the end is fine.

*) QGIS
To quote "Quantum GIS (QGIS) is a user friendly Open Source Geographic Information System"

The QGIS software is used to extract the Japan prefectures from the Natural Earth shapefile
Download here http://www.qgis.org/wiki/Download

I have QGIS installed locally on my desktop.

*) Excel or Open Office spreadsheet.
Excel is used to modify the dbf file from the shapefile downloaded from Natural Earth. The data in the shapefile relates to FIPS standard notation where I add the ISO related data from wikipedia for the prefectures of Japan.
I used this link to cross reference the data to the wikipedia page http://www.statoids.com/ujp.html

*) Geoserver
To quote again "GeoServer is an open source software server written in Java that allows users to share and edit geospatial data"

The Geoserver is used to generate the KML from the modified shapefile.
Download here http://geoserver.org/display/GEOS/Download

I have an instance of Geoserver running in the cloud on an free micro instance from Amazon.
Here is my EC2 instance showing the Japan prefectures with openlayers.
As its a test server I may stop it at some point and therefore it will not display.



QGIS Process

First objective is to delete all data that is not related to Japan from the Natural Earth shapefile.
Open QGIS and take the menu options "Layer" -> "Add vector layer".
Select the "10m_admin_1_states_provinces_lakes_shp.shp" file downloaded earlier.


QGIS Query option to select Japan from the ISO code JPN

Attribute table will show the remaing data for Japan (or county of choice).

Save the changes to the shapefile

 

Merged Polygons

While editing the natural earth shapefile I came across a very small  polygon with only 3 vertices in the Shimane prefecture,
As this would not be seen in my use of the KML file I merged the features using QGIS's merge icon. Also I edited the merged data to match that of the Shimane prefecture, as shown below.

Save the project and save the results back to the shapefile. Answer yes to the prompt.

Excel/Open Office Spreadsheet Process

Once the Japan only shapefile has been created then the dbf file needs to be edited. (in this example I add the ISO codes for the prefectures but this step is optional).
Open "10m_admin_1_states_provinces_lakes_shp.dbf" with Excel/Open Office.

I removed all columns apart from the ones shown below. I added the ISOP_1 column for the Japanese ISO prefecture codes.

ISO related data from wikipedia 
FIPS and ISO related data found at http://www.statoids.com/ujp.html

I noted the following while editing the natural earth shapefile data.


Osaka and Nagasaki share the same FIPS code. I changed Nagasaki FIPS to JA27
Okinawa and Tokyo share the same FIPS code. I changed Okinanawa FIPS to JA47

OBJECTID	VERTEXCOU	ISO	NAME_0	NAME_1	REGION	PROVNUMBER	ISOP_1	FIPS_1
1872	311.00000000000	JPN	Japan	Hiroshima	Chugoku	11	JP-34	JA11

*the same procedure for all rows in the file.


Save this file in the SAME dbase format (*.dbf files).
Geoserver Process

Once the shapefile is ready then it needs to be converted into KML.

There is a Geoserver quickstart guide which covers the Natrual Earth datasets.
Follow this guide to import the edited shapefile into Geoserver
http://live.osgeo.org/en/quickstart/geoserver_quickstart.html

However as we only want Japan (or any country of choice) then the bounding box needs to be adapted for Japan only.

As per this screen shot

I found the bounding box data from here http://nobmob.blogspot.com/2011/04/14-open-street-map-2-geoserver.html
Native Bounding Box:
Min X: 122.94
Min Y: 24.322
Max X: 145.82
Max Y: 45.522
Lat/Lon Bounding Box: 
Min X: 122.94
Min Y: 24.322
Max X: 145.82
Max Y: 45.522

Finally Export to KML

Option "Layer Preview"

Export to plain KML

To be continued....

KML file result is here http://bit.ly/japankml

Using the KML file in SAP
Using the KML file in Excel

Google maps can read online KML files , so heres a screen shot of my Geoserver serving up the KML on Google maps.

Here is a screenshot of the KML loaded into Google Earth