Weather data inputs

Section for FAQs regarding i-Tree Hydro

Moderators: i-Tree Support, i-Tree Team

Post Reply
rcoville
i-Tree Team
Posts: 64
Joined: Fri May 08, 2015 9:29 am
Location: Syracuse, NY

Weather data inputs

Post by rcoville » Mon Aug 31, 2015 1:43 pm

Weather information is required for i-Tree Hydro simulations. Of the 3 options i-Tree Hydro provides for weather input (users can pick from a map; use raw data; or use processed data), the most common and practical route is to pick from a map. If the map doesn't have data for the area or time period you're interested in, the next best option is to use raw data.

Picking a weather station from a map:
Weather data for many locations around the United States is pre-loaded into i-Tree Hydro for the convenience of users, after having been screened for completeness. Such stations are available through the map selection option, limited to time frames between 2005 and 2012.

Using a raw weather file (updated August 2020):
Standard weather data in the raw format (including stations pre-loaded into i-Tree Hydro) come from the National Oceanic and Atmospheric Administration (NOAA)'s Integrated Surface Database (ISD). Users seeking weather data beyond what is preloaded in i-Tree Hydro can follow the procedure below to obtain and prepare that data for use in i-Tree Hydro or Hydro+ models.

1) Download a raw weather archive (.gz) from NOAA FTP (ftp://ftp.ncei.noaa.gov/pub/data/noaa/), by year then WBAN-USAF station ID.
One way to find the WBAN-USAF ID of a weather station is to pick a weather station from the Hydro v6 Graphical User Interface (GUI) and record its USAF-WBAN code (in the format 123456-12345). This can be done without setting up your own Hydro project, by opening the sample project and editing the location information in Step 1 to match your project (or reference) location, and then perusing the map that appears when you pick "Select a weather station from map". Picking a weather station from Hydro this way should ensure you end up with a station that records all the parameters you need, such as sky cover and cloud ceiling.
2) Unzip the .gz raw data archive you obtained from NOAA FTP. There are many utilities available for handling .gz archives, and while we do not endorse any, one option we are aware is widely used is the “7-Zip” program.
3) Download the i-Tree Research Suite weather preprocessor from https://www.itreetools.org/documents/63 ... sor_r2.zip, unzip it, and find ishapp2.exe in the /scripts/ folder. That is the only file you need from this archive if proceeding with the Hydro GUI. If you are not using the Hydro GUI, you will skip step 5 and proceed to step 6 below.
4) From a Windows Command Line Interface (CLI) (e.g. go to Windows > Run > type "cmd" and hit enter), navigate to the directory with ishapp2.exe (e.g. type "cd /d <full directory path>" to jump to a new directory) and then enter: ishapp2.exe <raw-input-file-path> <output-name-for-raw-formatted-output.txt>
The terminal should respond with the input and output file names, and you should find the output file generated. If ishapp2.exe seems to run but generated no output file, try using shorter file paths and/or ensuring ishapp2.exe and your raw data input and output are all in the same directory.
Here are example commands for this step, with all files in same directory:

Code: Select all

C:\Users\covil>cd /d C:\Users\covil\Documents\iTree\SVN\WeatherPrep\trunk\scripts
C:\Users\covil\Documents\iTree\SVN\WeatherPrep\trunk\scripts>ishapp2.exe 724060-93721-2018 Formatted_93721.txt
5) The output file produced by step 4 is ready to use in the Hydro GUI as raw weather data. This output will have the same format as the "raw" weather data previously available from NOAA’s Climate Data Online data source, where Hydro obtained raw weather prior to 2020.
6) If not using the Hydro GUI, the weather preprocessor obtained in step 3 above includes a utility to preprocess raw weather for use in Hydro+ models. The compiled weather preprocessor utility is included in \WeatherDataExtractor\WeatherDataExtractor\bin\Release\ as WeatherDataExtractor.exe and brief instructions for each field is included WeatherPreprocessorDiagram.PDF to parameterize this utility. Note that the required LocationSpecies.mdb is available as part of i-Tree Eco installation, for example at C:\Program Files (x86)\i-Tree\EcoV6\Data\LocationSpecies.mdb.

Here is an example formatted raw weather file (headers may differ slightly): http://www.itreetools.org/hydro/resourc ... xample.txt
Here are definitions of abbreviations in the raw weather file: http://www.itreetools.org/hydro/resourc ... viated.txt

International weather data:
As of January 2017 we are no longer processing international weather data for use in i-Tree Hydro. We are working to develop the i-Tree Database tool for importing data into i-Tree Eco. At this time, we cannot provide support for international i-Tree Hydro projects. The below information is for reference only.

Downloading raw weather data works for stations worldwide, but stations outside of the United States we have tested often do not have sufficient data for i-Tree Hydro simulations. In those cases, international users would need to provide their own data time-series for their projects, merging that with other weather data from the most proximate viable NOAA ISD station. Using ISD data is necessary because Hydro is designed to work with that format, and ISD data includes variables which are not easy to acquire otherwise but are needed by Hydro.

International users can attempt to use proximate ISD data that will process in Hydro v6, then merge their own data into the processed weather file. The processed weather files in Hydro v6 use comma-separated values (CSV) formatting, which makes it possible to edit in spreadsheet software. Guidelines on how to modify processed weather data files are available in the following FAQ: Customizing timeseries inputs (e.g. weather, observed flow) in Hydro v6. Note that the i-Tree Team cannot currently provide any support for this process.

In choosing a proximate ISD station, consider your Geographic Reference Location. That is a location which represents your project but which is located in the United States, since Hydro's location database only includes data for US based locations. Your project's Geographic Reference Location should be a city that has a similar climate in terms of rain, snow, temperature, vegetation abundance, elevation, leaf-on & leaf-off days, and latitude. As an example of how this method can work: if the Geographic Reference Location you decide is most proximate to your study location is Syracuse, NY, then you could obtain processed weather data for Syracuse during your desired project period, and then customize that processed weather data to include the localized data you have available and want to simulate with.

Valid project periods depend on what NOAA ISD data is available. For example, if the nearest Hourly Global ISD station to your Geographic Reference Location only has observations for 2010 to September 2014, you can only merge in hourly precipitation data from within that period as you would need to pair your data with that ISD data.

Note that this method assumes your reference location and reference weather data is useful for making predictions about your study area and that the reference weather is compatible with the local weather data you merge into it. These assumptions are unlikely to be completely true, so it is important to recognize those limitations when assessing your simulation results. For example, at a time when it's raining in your local study area, the atmospheric conditions may not be representative of rain in your reference area's weather data. Due to the complex interactions between location data and different weather variables, each project's use of this method will have different strengths and weaknesses, and it is up to users to determine how valid this is for project needs.

Multi-year weather data
For multiple years of weather data, perform the above steps for each year individually, and then append the time periods of interest together in an un-interrupted timeseries that matches your intended simulation period.

Related FAQs:
A member of the i-Tree Team

Please provide feedback on our technical support services by clicking here to take our 5 minute survey
MattLeconey
Posts: 5
Joined: Mon Nov 02, 2020 3:34 pm

Re: Weather data inputs

Post by MattLeconey » Wed Nov 11, 2020 3:09 pm

I want to add to "Using raw weather file 6)" for those that are less computer science inclined. To "Compile the included /WeatherDataExtractor/WeatherDataExtractor.sln C# program," you need to use something like Visual Studio to do so. Once you download this, you just have to open the sln file using Visual Studio, go to "Build" and "Build WeatherDataExtractor". I had an error with Form1.resx being "blocked," so I had to go into properties for it in Windows Explorer and check "Unblock."

This was my first time compiling a program like this, and it wasn't hard once I figured out what I actually had to do. I figured I'd add on to help clarify the confusions I had.
rcoville
i-Tree Team
Posts: 64
Joined: Fri May 08, 2015 9:29 am
Location: Syracuse, NY

Re: Weather data inputs

Post by rcoville » Wed Nov 11, 2020 4:04 pm

Thanks for sharing that clarification and feedback Matt. Glad you were able to navigate compiling that program, despite a build error you needed to troubleshoot.

To make this easier for other users, the weather preprocessor package now includes a compiled EXE, and the instructions above have been updated to reflect that.

Sorry that compiled EXE was not available for your use earlier Matt. We are in the midst of transitioning from the Hydro GUI to Hydro+ and are working on procedures to make certain features more accessible, including weather preprocessing and autocalibration.
A member of the i-Tree Team

Please provide feedback on our technical support services by clicking here to take our 5 minute survey
MattLeconey
Posts: 5
Joined: Mon Nov 02, 2020 3:34 pm

Re: Weather data inputs

Post by MattLeconey » Thu Nov 12, 2020 5:04 pm

No problem, I'm glad I can help highlight things to make the process easier for others.

As a follow up, why does the Weather Data Extractor restrict the data to 2005-2015? While I'm using other years for my analysis is fine, 2004 is a year used in another analysis (deemed a "typical" year in NJ) that I'd like to use if possible to try to compare some results.

Also, I'm working through the Hydro+ procedure now, and I can let you know any more stumbling blocks I run into to help improve the accessibility of the process if it's helpful. What would be the best way to do so since this thread probably isn't the right place?
rcoville
i-Tree Team
Posts: 64
Joined: Fri May 08, 2015 9:29 am
Location: Syracuse, NY

Re: Weather data inputs

Post by rcoville » Fri Nov 13, 2020 2:21 pm

Matt,

I've checked in with the weather preprocessor developer to get a clearer answer to your question. In the interim, I'll share that I was able to manually enter a year outside of the drop-down menu range and successfully preprocess weather and run it in Hydro+. Specifically, I clicked on the year in that drop-down field and typed "2018" and was able to move forward with that entry.

My assessment of that utility's code is that weather year is flexible as long as it matches the year in your raw weather file. If that is confirmed, we will update our instructions for the tool and, to the extent our developer capacity allows, we will change the year field to be a type-in form rather than a drop-down menu.

Your feedback on Hydro+ is greatly appreciated! You can send further feedback directly to us at info@itreetools.org and robert.coville@davey.com. Topics we see fit for broader audiences can be summarized on here or in the Technical Manual.

Thanks for your interest and efforts.
A member of the i-Tree Team

Please provide feedback on our technical support services by clicking here to take our 5 minute survey
Post Reply