Handle Climate Data from DWD (German Weather Service)

Description

• find, select, download + read data from the German weather service DWD
  

• vectorized, progress bars, no re-downloads
  

• index of files + meta data
  

• observational time series from 6k meteorological recording stations (2.5k active)
  -> rain, temperature, wind, sunshine, pressure, cloudiness, humidity, snow, ...
  

• gridded raster data from radar + interpolation
  

• european data stock slowly growing
  For an introduction to the package, see https://bookdown.org/brry/rdwd.
  

Searchability Terms

Weather Data Germany download with R, Climate Data Germany
Deutscher Wetterdienst R Daten download Klimastationen
DWD Daten mit R runterladen, Wetter und Klimadaten in R

Author(s)

Berry Boessenkool, berry-b@gmx.de

See Also

related weather data processing references at https://bookdown.org/brry/rdwd/see-also.html

add country and Bundesland borders to a map

Description

add country and Bundesland borders to a map

Usage

addBorders(de = "grey80", eu = "black", add = TRUE, ...)

Arguments

Details

# Use the SpatVectors directly with:
DEU <- terra::vect(system.file("extdata/DEU.gpkg", package="rdwd"))
EUR <- terra::vect(system.file("extdata/EUR.gpkg", package="rdwd"))

# Obtained with the code:
url <- "https://gisco-services.ec.europa.eu/distribution/v2/nuts/shp/NUTS_RG_03M_2021_4326_LEVL_1.shp.zip"
tf <- tempfile(fileext=".zip")
download.file(url, tf) # 0.9 MB # in 2023-06 error 'Transferred a partial file'
unzip(tf, exdir="misc/vign") ; rm(url, tf)

DEU <- terra::vect("misc/vign/NUTS_RG_03M_2021_4326_LEVL_1.shp")
library(terra) # for bracket method
DEU <- DEU[DEU$CNTR_CODE=="DE","NUTS_NAME"]
terra::writeVector(DEU, "inst/extdata/DEU.gpkg", overwrite=TRUE)

url <- "https://gisco-services.ec.europa.eu/distribution/v2/nuts/shp/NUTS_RG_03M_2021_4326_LEVL_0.shp.zip"
tf <- tempfile(fileext=".zip")
download.file(url, tf) # 0.7 MB # in 2023-06 error 'Transferred a partial file'
unzip(tf, exdir="misc/vign") ; rm(url, tf)

EUR <- terra::vect("misc/vign/NUTS_RG_03M_2021_4326_LEVL_0.shp")
EUR <- terra::crop(EUR, c(-11,25, 40,60))
EUR <- EUR[,"NUTS_NAME"]
terra::writeVector(EUR, "inst/extdata/EUR.gpkg", overwrite=TRUE)

Value

invisible list with DEU and EUR

Author(s)

Berry Boessenkool, berry-b@gmx.de, Aug 2019, June 2023

See Also

plotRadar, website raster chapter

Examples

if(requireNamespace("terra", quietly=TRUE)){
plot(1, xlim=c(2,16), ylim=c(47,55))
addBorders()
addBorders(add=FALSE)
plot(1, xlim=c(2,16), ylim=c(47,55))
addBorders(de="orange", eu=NA)
}

Launch interactive weather analysis app

Description

Launch interactive analysis of weather period comparison for different RDWD stations. The R session is blocked during usage, close the app to re-enable console usage.

Usage

app(...)

Arguments

Author(s)

Berry Boessenkool, berry-b@gmx.de, July 2018 + April 2023

See Also

shiny::runApp(), rdwd

Examples

# app()

check indexes

Description

check indexes. Mainly for internal usage in createIndex(). Not exported, so call it as rdwd:::checkIndex() if you want to run tests yourself. Further test suggestions are welcome!

Usage

checkIndex(
  findex = NULL,
  mindex = NULL,
  gindex = NULL,
  excludefp = TRUE,
  fast = FALSE,
  warn = !quiet,
  logfile = berryFunctions::packagePath(file = "misc/ExampleTests/warnings.txt"),
  quiet = rdwdquiet()
)

Arguments

Value

Charstring with issues (if any) to be printed with cat().

Author(s)

Berry Boessenkool, berry-b@gmx.de, May 2019

See Also

createIndex

Examples

data(fileIndex) ; data(metaIndex) ; data(geoIndex)
# ci <- rdwd:::checkIndex(findex=fileIndex, mindex=metaIndex, gindex=geoIndex)
# cat(ci)

check suggested package for availability

Description

check suggested package for availability, yielding an instructive error message if not

Usage

checkSuggestedPackage(package, functionname)

Arguments

Value

invisible success logical value from requireNamespace()

Author(s)

Berry Boessenkool, berry-b@gmx.de, Aug 2019

See Also

requireNamespace()

Create file and meta index of the DWD CDC FTP Server

Description

This is mainly an internal function. Create data.frames out of the vector index returned by indexFTP(). For fileIndex (the first output element) createIndex tries to obtain res, var, per, file, id, start and end from the paths. If meta=TRUE, metaIndex and geoIndex are also created. They combine all Beschreibung files into a single data.frame.
If you create your own index as suggested in selectDWD (argument findex), you can read the produced file as shown in the example section.

Usage

createIndex(
  paths,
  base = dwdbase,
  dir = "DWDdata",
  fname = "fileIndex.txt",
  meta = FALSE,
  metadir = "meta",
  mname = "metaIndex.txt",
  gname = "geoIndex.txt",
  overwrite = FALSE,
  checkwarn = TRUE,
  checklog = tempfile(),
  quiet = rdwdquiet(),
  ...
)

Arguments

Value

invisible data.frame (or if meta=TRUE, list with two data.frames) with a number of columns inferred from the paths. Each is also written to disc.

Author(s)

Berry Boessenkool, berry-b@gmx.de, Oct-Nov 2016, June 2017

See Also

indexFTP(), updateIndexes(), index, selectDWD(), website index chapter

Examples

## Not run:  # Not tested with R CMD check because of file writing
link <- "daily/kl/historical/tageswerte_KL_00699_19490101_19580630_hist.zip"
ind <- createIndex(link, dir=tempdir())
ind
# res  var        per  id    start          end ismeta
# daily kl historical 699 1949-01-01 1958-06-30 FALSE
link2 <- "daily/kl/historical/KL_Tageswerte_Beschreibung_Stationen.txt"
link3 <- "daily/kl/recent/KL_Tageswerte_Beschreibung_Stationen.txt"
ind2 <- createIndex(c(link,link2,link3), dir=tempdir(), meta=TRUE, checkwarn=FALSE)
lapply(ind2, head)
link4 <- "1_minute/precipitation/meta_data/Meta_Daten_ein_min_rr_00755.zip"
ind <- createIndex(link4, dir=tempdir())
ind

## End(Not run)

Download data from the DWD CDC FTP Server

Description

Get climate data from the German Weather Service (DWD) FTP-server. The desired dataset is downloaded into dir. If read=TRUE, it is also read and processed.
dataDWD handles vectors of URLs, displays progress bars (if the package pbapply is available) and by default does not re-download data already in dir (but see argument force to update files).
To solve "errors in download.file: cannot open URL", see https://bookdown.org/brry/rdwd/fileindex.html.

Usage

dataDWD(
  url,
  base = dwdbase,
  joinbf = FALSE,
  dir = locdir(),
  force = FALSE,
  overwrite = !isFALSE(force),
  read = TRUE,
  dbin = TRUE,
  method = getOption("download.file.method"),
  dfargs = NULL,
  sleep = 0,
  progbar = !quiet,
  browse = FALSE,
  ntrunc = 2,
  file = NULL,
  quiet = rdwdquiet(),
  ...
)

Arguments

Value

Presuming downloading and processing were successful: if read=TRUE, the desired dataset (as returned by readDWD()), otherwise the filename as saved on disc (may have "_n" appended in name, see newFilename()).
If length(file)>1, the output is a list of outputs / vector of filenames.
The output is always invisible.

Author(s)

Berry Boessenkool, berry-b@gmx.de, Jun-Oct 2016

See Also

selectDWD(). readDWD(), download.file().
https://bookdown.org/brry/rdwd
Helpful for plotting: berryFunctions::monthAxis(), see also berryFunctions::climateGraph()

Examples

## Not run:  ## requires internet connection
# find FTP files for a given station name and file path:
link <- selectDWD("Fuerstenzell", res="hourly", var="wind", per="recent")
# download file:
fname <- dataDWD(link, dir=locdir(), read=FALSE) ; fname
# dir="DWDdata" is the default directory to store files
# unless force=TRUE, already obtained files will not be downloaded again

# read and plot file:
wind <- readDWD(fname, varnames=TRUE) ; head(wind)
metafiles <- readMeta(fname)          ; str(metafiles, max.level=1)
column_names <- readVars(fname)       ; head(column_names)

plot(wind$MESS_DATUM, wind$F, main="DWD hourly wind Fuerstenzell", col="blue",
     xaxt="n", las=1, type="l", xlab="Date", ylab="Hourly Wind speed  [m/s]")
berryFunctions::monthAxis(1)


# current and historical files, keep historical in the overlap time period:
link <- selectDWD("Potsdam", res="daily", var="kl", per="hr"); link
potsdam <- dataDWD(link, dir=locdir(), hr=4)
plot(TMK~MESS_DATUM, data=tail(potsdam,1500), type="l")


# With many files (>>50), use sleep to avoid getting kicked off the FTP server
#links <- selectDWD(res="daily", var="solar")
#sol <- dataDWD(links, sleep=20) # random waiting time after download (0 to 20 secs)

# Real life examples can be found in the use cases section of the vignette:
# browseURL("https://bookdown.org/brry/rdwd")

## End(Not run)

directory management for rdwd

Description

Manage directories with useful messages in the rdwd package.

Usage

dirDWD(dir = "DWDdata", quiet = rdwdquiet())

Arguments

Value

dirDWD invisibly returns the prior working directory as per setwd().

Author(s)

Berry Boessenkool, berry-b@gmx.de, Oct 2016

See Also

dataDWD()

Examples

# see source code of dataDWD and metaDWD

DWD FTP Server base URL

Description

base URLs to the DWD FTP Server

dwdbase: observed climatic records at ftp:// variant of
https://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/
See overview of available datasets and usage suggestions.


gridbase: spatially interpolated gridded data at ftp:// variant of
https://opendata.dwd.de/climate_environment/CDC/grids_germany/
See usage suggestions

Usage

dwdbase

Format

An object of class character of length 1.

DWD parameter explanations

Description

Short German parameter explanations for the DWD abbreviations on the CDC FTP server.
These are manually created by me and might need to be expanded if the DWD adds more abbreviations.
readVars() maps them to the variable abbreviations in the "Metadaten_Parameter.*txt" file in any given zip folder and will warn about missing entries.

Usage

dwdparams

Format

An object of class data.frame with 176 rows and 2 columns.

Author(s)

Berry Boessenkool, berry-b@gmx.de, Jun 2018

See Also

readVars(), readDWD()

Examples

head(dwdparams)

determine DWD file type

Description

determine which subfunction to call in readDWD() from the file extension (ext).

The first block is for observational data (overview), the second for gridded data (overview).
Click on the type for the subfunction documentation, e.g. data for readDWD.data().

Usage

fileType(file)

Arguments

Value

Character (vector)

Author(s)

Berry Boessenkool, berry-b@gmx.de, Jul 2020

See Also

readDWD()

Examples

ft <- read.table(header=TRUE, stringsAsFactors=FALSE, text="
type    filename
data    daily_kl_recent_tageswerte_KL_03987_akt.zip
meta    daily_kl_recent_KL_Tageswerte_Beschreibung_Stationen.txt
multia  multi_annual_mean_81-10_Temperatur_1981-2010_aktStandort.txt
multia  multi_annual_mean_81-10_Temperatur_1981-2010.txt
stand   subdaily_standard_format_kl_10381_00_akt.txt
deriv   derived_germany_soil_daily_historical_3987.txt.gz
pdf     DESCRIPTION_obsgermany_climate_monthly_kl_historical_en.pdf

radar   radolan_recent_bin_raa01-rw_10000-1802020250-dwd---bin.gz
binary  daily_radolan_historical_bin_2017_SF201712.tar.gz
raster  16_DJF_grids_germany_seasonal_air_temp_mean_188216.asc.gz
nc      daily_Project_TRY_humidity_RH_199509_daymean.nc.gz
asc     radolan_historical_asc_2018_RW-201809.tar
rklim   5_minutes_radolan_reproc_2017_002_bin_2020_YW2017.002_202006.tar
grib2   ftp_weather_nwp_cosmo-d2_005_T_2M.grib2.bz2
")
fileType(ft$filename)

stopifnot(fileType(ft$filename)==ft$type)
berryFunctions::is.error(fileType("random_stuff.odt"), force=TRUE)

stopifnot(validFileTypes %in% ft$type)
stopifnot(ft$type %in% validFileTypes)

find DWD weather station ID from name

Description

Identify DWD weather station ID from station name

Usage

findID(
  name = "",
  exactmatch = TRUE,
  mindex = metaIndex,
  failempty = FALSE,
  quiet = rdwdquiet()
)

Arguments

Value

Character string (vector) with ID(s)

Author(s)

Berry Boessenkool, berry-b@gmx.de, Oct-Nov 2016

See Also

used in selectDWD(), metaInfo()

Examples

# Give weather station name (must be existing in metaIndex):
findID("Potsdam")
findID("potsDam") # capitalization is ignored
# all names containing "Hamburg":
findID("Hamburg", exactmatch=FALSE)
findID("Potsdam", exactmatch=FALSE)

# vectorized:
findID(c("Potsdam","Berlin-Buch"))

# German Umlauts are changed to ue, ae, oe, ss
findID("Muenchen", FALSE)
berryFunctions::convertUmlaut("M?nchen") # use this to convert umlauts in lists

Indexes of files and metadata on the DWD CDC FTP server

Description

Created with indexFTP() and createIndex() used in updateIndexes().
In functions, you can access them with rdwd:::fileIndex etc.
fileIndex: A data.frame with the filenames (and derived information) at the default base value dwdbase.
metaIndex: A data.frame with the contents of all the station description files (..._Beschreibung_Stationen.txt) under dwdbase.
geoIndex: metaIndex distilled to geographic locations.
gridIndex: Vector of file paths at gridbase.
formatIndex: (modified) table from https://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/subdaily/standard_format/formate_kl.html

Format

fileIndex: data.frame with character strings. ca 260k rows x 8 columns:
res, var, per (see selectDWD()), station id, time series start and end, and ismeta information, all according to path.
metaIndex: data.frame with ca 97k rows for 12 columns:
⁠Stations_id, von_datum, bis_datum, Stationshoehe, geoBreite, geoLaenge, Stationsname, Bundesland, res, var, per, hasfile⁠
geoIndex: data.frame with ca 6k rows for 11 columns:
⁠id, name, state, lat, lon, ele, nfiles, nonpublic, recentfile, display, col⁠
gridIndex: Vector with ca 50k file paths at gridbase
formatIndex: data.frame with 140 rows for 12 columns:
⁠Ke_Ind, Kennung, Label, Beschreibung, Einheit, Code-Tabellen, Zusatzinfo, Typ, Pos, Erlaubt, Fehlk, dividebyten⁠

Author(s)

Berry Boessenkool, berry-b@gmx.de, June-Nov 2016, June 2017, Oct 2019

Source

Deutscher WetterDienst / Climate Data Center FTP Server

See Also

createIndex(), indexFTP(), selectDWD(), findID(), metaInfo(), website index chapter

Examples

data(fileIndex)
data(metaIndex)
data(geoIndex)
head(fileIndex)
head(metaIndex)
head(geoIndex)

# in functions, you can use head(rdwd:::fileIndex) etc, but I don't export them
# because Hadley says 'Never @export a data set' in
# browseURL("http://r-pkgs.had.co.nz/data.html#data-data")

# To use a custom index, see
# browseURL("https://bookdown.org/brry/rdwd/fileindex.html")

Create a recursive index of an FTP Server

Description

Create a list of all the files (in all subfolders) of an FTP server. Defaults to the German Weather Service (DWD, Deutscher WetterDienst) OpenData server at https://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/.
The R package RCurl must be available to do this.

It's not suggested to run this for all folders, as it can take quite some time and you may get kicked off the FTP-Server. This package contains an index of the climatic observations at weather stations (fileIndex) and gridded datasets (gridIndex). If they are out of date, please let me know!

Getting banned from the FTP Server
Normally, this shouldn't happen anymore: since Version 0.10.10 (2018-11-26), a single RCurl handle is used for all FTP requests and since version 1.0.17 (2019-05-14), the file tree provided by the DWD is used to obtain all folders first, eliminating the recursive calls.
There's a provision if the FTP server detects bot requests and denies access. If RCurl::getURL() fails, there will still be an output which you can pass in a second run via folder to extract the remaining dirs. You might need to wait a bit and set sleep to a higher value in that case. Here's an example:

gridindex <- indexFTP("", gridbase)
gridindex <- indexFTP(gridindex, gridbase, sleep=15)

Of course, with a higher sleep value, the execution will take longer!

Usage

indexFTP(
  folder = "currentfindex",
  base = dwdbase,
  is.file.if.has.dot = TRUE,
  exclude.latest.bin = TRUE,
  fast = TRUE,
  sleep = 0,
  dir = "DWDdata",
  filename = folder[1],
  overwrite = FALSE,
  quiet = rdwdquiet(),
  progbar = !quiet,
  verbose = FALSE
)

Arguments

Value

a vector with file paths

Author(s)

Berry Boessenkool, berry-b@gmx.de, Oct 2016

See Also

createIndex(), updateIndexes(), website index chapter

Examples

## Not run:  ## Needs internet connection
sol <- indexFTP(folder="/daily/solar", dir=tempdir())
head(sol)

# mon <- indexFTP(folder="/monthly/kl", dir=tempdir(), verbose=TRUE)

## End(Not run)

distance between lat-long coordinates

Description

Great-circle distance between points at lat-long coordinates. Mostly a copy of OSMscale::earthDist Version 0.5.3 (2017-04-19). https://github.com/brry/OSMscale/blob/master/R/earthDist.R#L57-L102. Copied manually to avoid dependency hell. Does not check coordinates. Not exported.

Usage

lldist(lat, long, data, r = 6371, i = 1L)

maxlldist(lat, long, data, r = 6371, fun = max, each = TRUE, ...)

Arguments

Value

Vector with distance(s) in km (or units of r, if r is changed)

Author(s)

Berry Boessenkool, berry-b@gmx.de, Aug 2016 + Jan 2017. Angle formula from Diercke Weltatlas 1996, Page 245

local data directory

Description

This can be used to set a directory for DWD data across projects, thus avoiding multiple downloads of the same file.
Set the default for all subsequent calls with options(rdwdlocdir="YOUR/PATH").
Currently, the dataDWD() dir defaults to a project specific folder at getwd. In the future, this may change to locdir().
locdir() is used especially for the website, local tests and examples.

Usage

locdir(dir = getOption("rdwdlocdir"), file = NULL, quiet = rdwdquiet())

Arguments

Value

charstring (directory)

Author(s)

Berry Boessenkool, berry-b@gmx.de, Apr 2019, Jun 2021

See Also

runLocalTests()

Examples

locdir()
oldopt <- options(rdwdlocdir="~")
locdir()
stopifnot(locdir() == path.expand("~"))
options(oldopt) ; rm(oldopt)

Information for a station ID on the DWD CDC FTP server

Description

Information for a station ID on the DWD CDC FTP server

Usage

metaInfo(id, mindex = metaIndex, hasfileonly = TRUE)

Arguments

Value

invisible data.frame. Also prints the output nicely formatted.

Author(s)

Berry Boessenkool, berry-b@gmx.de, Nov 2016

See Also

metaIndex

Examples

metaInfo(2849)

Find DWD stations close to given coordinates

Description

Select DWD stations within a given radius around a set of coordinates

Usage

nearbyStations(
  lat,
  lon,
  radius,
  res = NA,
  var = NA,
  per = NA,
  mindate = NA,
  hasfileonly = TRUE,
  statname = "nearbyStations target location",
  quiet = rdwdquiet(),
  ...
)

Arguments

Value

metaIndex subset with additional columns "dist" and "url"

Author(s)

Berry Boessenkool, berry-b@gmx.de, Mar 2017

See Also

selectDWD(), metaIndex, website use case with nearbyStations

Examples

m <- nearbyStations(49.211784, 9.812475, radius=30,
    res=c("daily","hourly"), var= c("precipitation","more_precip","kl") ,
    mindate=as.Date("2016-05-30"), statname="Braunsbach catchment center")
# View(m)

# for a continued example of this, see the vignette in chapter
# use case: plot all rainfall values around a given point
# browseURL("https://bookdown.org/brry/rdwd")

Enhance readDWD column names

Description

Add short German parameter descriptions to the DWD abbreviations. This uses dwdparams() to create column names like "TT_TU.Lufttemperatur" and "RSK.Niederschlagshoehe." Column names not in the abbreviation list will be left untouched.

Usage

newColumnNames(dataframe, variables = dwdparams, separator = ".")

Arguments

Value

The dataframe with new column names

Author(s)

Berry Boessenkool, berry-b@gmx.de, Apr 2019

See Also

dwdparams, readVars(), readDWD() argument varnames, newColumnNames()

Examples

# mainly for internal usage

Quickly plot time series

Description

plot rdwd time series from data.frames

Usage

plotDWD(
  x,
  cn,
  monthaxis = TRUE,
  line0 = FALSE,
  xlab = "",
  ylab = cn,
  main = "",
  type = "l",
  lwd = 3,
  col = "blue",
  las = 1,
  mar = c(2.6, 3.1, 2.5, 0.5),
  mgp = c(1.9, 0.7, 0),
  keeppar = TRUE,
  ...
)

Arguments

Value

Nothing

Author(s)

Berry Boessenkool, berry-b@gmx.de, Sep 2021

See Also

readDWD()

Examples

link <- selectDWD("Potsdam", res="daily", var="kl", per="r")
clim <- dataDWD(link, dir=locdir(), varnames=TRUE)
plotDWD(clim, "TMK.Lufttemperatur", line0=TRUE, main="Potsdam")

plot radar products on a pretty map

Description

Convenience function to plot radar products on a pretty map. Creates a separate plot for each layer, a selection is possible.

Usage

plotRadar(
  x,
  layer = NULL,
  main = names(x),
  land = "gray80",
  sea = "cadetblue1",
  de = "grey80",
  eu = "black",
  col = berryFunctions::seqPal(),
  xlim = NULL,
  ylim = NULL,
  zlim = NULL,
  axes = TRUE,
  las = 1,
  mar = c(2.5, 3.5, 2.5, 5),
  keeppar = TRUE,
  project = TRUE,
  proj = "radolan",
  extent = "radolan",
  adjust05 = FALSE,
  targetproj = "ll",
  quiet = rdwdquiet(),
  ...
)

Arguments

Value

terra object, (re)projected (if project=TRUE). If length(layer)==1, only that selected layer is returned.

Author(s)

Berry Boessenkool, berry-b@gmx.de, Feb 2020, June 2023

See Also

projectRasterDWD(), addBorders(), readDWD(), website raster chapter

Examples

# See https://bookdown.org/brry/rdwd/raster-data.html
## Not run:  ## Excluded from CRAN checks: requires internet connection
link <- "seasonal/air_temperature_mean/16_DJF/grids_germany_seasonal_air_temp_mean_188216.asc.gz"
rad <- dataDWD(link, base=gridbase, joinbf=TRUE)
radp <- plotRadar(rad, proj="seasonal", extent=NULL, main="plotRadar ex")
plotRadar(radp, ylim=c(52,54), project=FALSE)

# plotRadar equivalent, map only country borders:
radpm <- projectRasterDWD(rad[[1]], proj="seasonal", extent=NULL)
terra::plot(radpm)
addBorders()

# several layers
url <- "daily/Project_TRY/pressure/PRED_199606_daymean.nc.gz"  #  5 MB
nc <- dataDWD(url, base=gridbase, joinbf=TRUE)

ncp3 <- plotRadar(nc, main=paste(terra::longnames(nc), terra::time(nc)), layer=1:3,
                  col=terrain.colors(100), proj="nc", extent="nc")
plotRadar(ncp3, layer=3:4, project=FALSE) # still has all layers
plotRadar(ncp3, layer=4:5, project=FALSE, zlim="ind") # individual zlims per layer
plotRadar(ncp3, layer=1, project=FALSE, zlim=c(1016,1020))

ncp1 <- plotRadar(nc, layer=1, proj="nc", extent="nc") # much faster projection
# no longer has layers 2-4:
berryFunctions::is.error(plotRadar(ncp1, layer=1:4, project=FALSE), TRUE, TRUE)

## End(Not run)

project DWD raster data

Description

Set projection and extent for DWD raster data. Optionally (and per default) also reprojects to latlon data.

WARNING: reprojection to latlon changes values slightly. For the tested RX product, this change is significant, see: https://github.com/brry/rdwd/blob/master/misc/ExampleTests/Radartests.pdf
In terra::plot, use range=zlim with the original range if needed.

Usage

projectRasterDWD(
  r,
  proj = "radolan",
  extent = "radolan",
  adjust05 = FALSE,
  targetproj = "ll",
  threads = TRUE,
  quiet = rdwdquiet()
)

Arguments

Details

The internal defaults are extracted from the Kompositformatbeschreibung at https://www.dwd.de/DE/leistungen/radolan/radolan.html, as provided 2019-04 by Antonia Hengst.
The nc extent was obtained by projecting Germanys bbox to EPSG 3034 (specified in the DWD documentation). Using that as a starting point, I then refined the extent to a visual match, see developmentNotes.R

Value

terra raster object with projection and extent, invisible

Author(s)

Berry Boessenkool, berry-b@gmx.de, May 2019, June 2023

See Also

plotRadar()
⁠terra::⁠crs / ext / project
readDWD.binary / raster / asc / radar / nc
website raster chapter

Examples

# To be used after readDWD.binary etc

global quiet option for rdwd

Description

global quiet option. The default rdwdquiet() is FALSE.
Just write the following in your code and all subsequent calls will be quiet:
options(rdwdquiet=TRUE)

Usage

rdwdquiet()

Process data from the DWD CDC FTP Server

Description

Read climate data that was downloaded with dataDWD(). The data is unzipped and subsequently, the file(s) are read, processed and returned as a data.frame / terra raster object.

For observational data, new users are advised to set varnames=TRUE to obtain more informative column names.

readDWD will call internal (but documented) subfunctions depending on the argument type, see the overview in fileType().

Not all arguments to readDWD are used for all subfunctions, e.g. fread is used only by readDWD.data, while dividebyten is used in readDWD.raster and readDWD.asc.

file can be a vector with several filenames. Most other arguments can also be a vector and will be recycled to the length of file.

Usage

readDWD(
  file,
  type = fileType(file),
  varnames = FALSE,
  fread = NA,
  format = NA,
  tz = "GMT",
  hr = 0,
  dividebyten = TRUE,
  var = "",
  progbar = !quiet,
  quiet = rdwdquiet(),
  quietread = quiet,
  ...
)

Arguments

Value

For observational data, an invisible data.frame of the desired dataset, or a named list of data.frames if length(file) > 1.
For gridded data, terra raster objects.

Author(s)

Berry Boessenkool, berry-b@gmx.de, Jul-Oct 2016, Winter 2018/19

See Also

dataDWD(), readVars(), readMeta(), selectDWD(), fileType()
https://bookdown.org/brry/rdwd

Examples

# see dataDWD and readDWD.* subfunctions

read dwd gridded radolan asc data

Description

read grid-interpolated radolan asc data. Intended to be called via readDWD().
All layers (following selection if given) in all .tar.gz files are combined into a terra raster with terra::rast().
To project the data, use projectRasterDWD()

Usage

readDWD.asc(
  file,
  exdir = NULL,
  dividebyten = TRUE,
  selection = NULL,
  quiet = rdwdquiet(),
  progbar = !quiet,
  ...
)

Arguments

Value

data.frame

Author(s)

Berry Boessenkool, berry-b@gmx.de, April 2019

See Also

readDWD()

Examples

## Not run:  # Excluded from CRAN checks, but run in localtests
# File selection and download:
radbase <- paste0(gridbase,"/hourly/radolan/historical/asc/")
radfile <- "2018/RW-201809.tar" # 25 MB to download
file <- dataDWD(radfile, base=radbase, joinbf=TRUE, read=FALSE)
#asc <- readDWD(file) # 4 GB in mem. ~ 20 secs unzip, 10 secs read, 2 min divide
asc <- readDWD(file, selection=1:5, dividebyten=TRUE)
plotRadar(asc[[1]], main=names(asc)[1])

viddir <- paste0(tempdir(),"/RadolanVideo")
dir.create(viddir)
png(paste0(viddir,"/Radolan_%03d.png"), width=7, height=5, units="in", res=300)
plotRadar(asc, layer=1:3, main=names(asc)) # 3 secs per layer
dev.off()
berryFunctions::openFile(paste0(viddir,"/Radolan_001.png"))

# Time series of a given point in space:
plot(unlist(asc[800,800,]), type="l", xlab="Time [hours]")

# if dividebyten=FALSE, terra stores things out of memory in the exdir.
# by default, this is in tempdir, hence you would need to save asc manually:
# terra::writeRaster(asc, tempfile(fileext="/RW2018-09.gpkg"), overwrite=TRUE)

## End(Not run)

read dwd gridded radolan binary data

Description

read gridded radolan binary data. Intended to be called via readDWD().

Usage

readDWD.binary(
  file,
  exdir = sub(".tar.gz$", "", file),
  toraster = TRUE,
  quiet = rdwdquiet(),
  progbar = !quiet,
  selection = NULL,
  ...
)

Arguments

Value

list depending on argument toraster, see there for details

Author(s)

Berry Boessenkool, berry-b@gmx.de, Dec 2018. Significant input for the underlying dwdradar::readRadarFile() came from Henning Rust & Christoph Ritschel at FU Berlin.

See Also

readDWD(), especially readDWD.radar()
https://wradlib.org for much more extensive radar analysis in Python
Kompositformatbeschreibung at https://www.dwd.de/DE/leistungen/radolan/radolan.html for format description

Examples

## Not run:  # Excluded from CRAN checks, but run in localtests

# SF file as example: ----

SF_link <- "/daily/radolan/historical/bin/2017/SF201712.tar.gz"
SF_file <- dataDWD(url=SF_link, base=gridbase, joinbf=TRUE,   # 204 MB
                     dir=locdir(), read=FALSE)
# exdir radardir set to speed up my tests:
SF_exdir <- "C:/Users/berry/Desktop/DWDbinarySF"
if(!file.exists(SF_exdir)) SF_exdir <- tempdir()
# no need to read all 24*31=744 files, so setting selection:
SF_rad <- readDWD(SF_file, selection=1:10, exdir=SF_exdir) #with toraster=TRUE
if(length(SF_rad)!=2) stop("length(SF_rad) should be 2, but is ", length(SF_rad))

SF_radp <- plotRadar(SF_rad$dat, layer=1:3, main=SF_rad$meta$date)
plotRadar(SF_radp, layer=1, project=FALSE)

# RW file as example: ----

RW_link <- "hourly/radolan/reproc/2017_002/bin/2017/RW2017.002_201712.tar.gz"
RW_file <- dataDWD(url=RW_link, base=gridbase, joinbf=TRUE,   # 25 MB
                  dir=locdir(), read=FALSE)
RW_exdir <- "C:/Users/berry/Desktop/DWDbinaryRW"
if(!file.exists(RW_exdir)) RW_exdir <- tempdir()
RW_rad <- readDWD(RW_file, selection=1:10, exdir=RW_exdir)
RW_radp <- plotRadar(RW_rad$dat[[1]], main=RW_rad$meta$date[1], extent="rw")

# ToDo: why are values + patterns not the same?

# list of all Files: ----
data(gridIndex)
head(grep("historical", gridIndex, value=TRUE))

## End(Not run)

read regular dwd data

Description

Read regular dwd data. Intended to be called via readDWD().

Usage

readDWD.data(
  file,
  fread = FALSE,
  varnames = FALSE,
  format = NA,
  tz = "GMT",
  hr = 0,
  quiet = rdwdquiet(),
  ...
)

Arguments

Value

data.frame

Author(s)

Berry Boessenkool, berry-b@gmx.de

See Also

readDWD(), Examples in dataDWD()

read derived dwd data

Description

Read dwd data from /CDC/derived_germany/. Intended to be called via readDWD().

Usage

readDWD.deriv(file, gargs = NULL, todate = TRUE, quiet = rdwdquiet(), ...)

Arguments

Value

data.frame

Author(s)

Berry Boessenkool, berry-b@gmx.de

See Also

readDWD(), https://bookdown.org/brry/rdwd/use-case-derived-data.html

read nwp forecast data

Description

read gridded numerical weather prediction data. Intended to be called via readDWD().

Usage

readDWD.grib2(file, pack = "terra", bargs = NULL, quiet = rdwdquiet(), ...)

Arguments

Value

terra or stars object, depending on pack

Author(s)

Berry Boessenkool, berry-b@gmx.de, Jan 2021.

See Also

readDWD()
https://www.dwd.de/EN/ourservices/nwp_forecast_data/nwp_forecast_data.html
https://www.dwd.de/EN/aboutus/it/functions/Teasergroup/grib.html

Examples

## Not run:  # Excluded from CRAN checks, but run in localtests
nwp_t2m_base <- "ftp://opendata.dwd.de/weather/nwp/icon-d2/grib/15/soiltyp"
nwp_urls <- indexFTP("", base=nwp_t2m_base, dir=tempdir())
# for p instead of soiltyp, icosahedral_model-level files fail with GDAL errors,
# see https://github.com/brry/rdwd/issues/28
# regular-lat-lon_pressure-level files work with pack="terra" or "stars"

nwp_file <- dataDWD(tail(nwp_urls,1), base=nwp_t2m_base, dir=tempdir(), 
                    joinbf=TRUE, dbin=TRUE, read=FALSE)
nwp_data <- readDWD(nwp_file)
terra::plot(nwp_data)
addBorders() # the projection seems to be perfectly good :)

# index of GRIB files
if(FALSE){ # indexing takes about 6 minutes!
grib_base <- "ftp://opendata.dwd.de/weather/nwp/icon-d2/grib"
grib_files <- indexFTP("", base=grib_base, dir=tempdir())
for(f in unique(substr(grib_files, 1,3))) print(grib_files[which(substr(grib_files, 1,3)==f)[1]])
View(data.frame(grep("regular",grib_files, value=TRUE)))
}

## End(Not run)

read dwd metadata (Beschreibung*.txt files)

Description

read dwd metadata (Beschreibung*.txt files). Intended to be called via readDWD().
Column widths for read.fwf() are computed internally.
if(any(meta)), readDWD() tries to set the locale to German (to handle Umlaute correctly). It is hence not recommended to call rdwd:::readDWD.meta directly on a file!
Names can later be changed to ascii with berryFunctions::convertUmlaut().

Usage

readDWD.meta(file, quiet = rdwdquiet(), ...)

Arguments

Value

data.frame

Author(s)

Berry Boessenkool, berry-b@gmx.de

See Also

readDWD()

Examples

## Not run:  # Excluded from CRAN checks, but run in localtests

link <- selectDWD(res="daily", var="kl", per="r", meta=TRUE)
link <- link[!grepl("mn4", link)] # for mn4 file May 2022
link <- grep(".txt$", link, value=TRUE)
if(length(link)!=1) stop("length of link should be 1, but is ", length(link),
                ":\n", berryFunctions::truncMessage(link,prefix="",sep="\n"))

file <- dataDWD(link, dir=locdir(), read=FALSE)
meta <- readDWD(file)
head(meta)

cnm <- colnames(meta)
if(length(cnm)!=8) stop("number of columns should be 8, but is ", length(cnm),
                        ":\n", toString(cnm))

## End(Not run)

read multi_annual dwd data

Description

read multi_annual dwd data. Intended to be called via readDWD().
All other observational data at dwdbase can be read with readDWD.data(), except for the multi_annual and subdaily/standard_format data.

Usage

readDWD.multia(
  file,
  fileEncoding = "latin1",
  comment.char = "\032",
  quiet = rdwdquiet(),
  ...
)

Arguments

Value

data.frame

Author(s)

Berry Boessenkool, berry-b@gmx.de, Feb 2019

See Also

readDWD()

Examples

## Not run:  # Excluded from CRAN checks, but run in localtests

# Temperature aggregates (2019-04 the 9th file, 2022-05 the 8th):
durl <- selectDWD(res="multi_annual", per="mean_81-10")[8]
murl <- selectDWD(res="multi_annual", per="mean_81-10", meta=TRUE)[8]
ma_temp <- dataDWD(durl, fileEncoding="")
ma_meta <- dataDWD(murl, fileEncoding="")

head(ma_temp)
head(ma_meta)

ma <- merge(ma_meta, ma_temp, all=TRUE)
berryFunctions::linReg(ma$Stationshoehe, ma$Jahr, main="annual average ~ elevation")
op <- par(mfrow=c(3,4), mar=c(0.1,2,2,0), mgp=c(3,0.6,0))
for(m in colnames(ma)[8:19])
  {
  berryFunctions::linReg(ma$Stationshoehe, ma[,m], xaxt="n", xlab="", ylab="", main=m)
  abline(h=0)
  }
par(op)

par(bg=8)
berryFunctions::colPoints(ma$geogr..Laenge, ma$geogr..Breite, ma$Jahr, add=F, asp=1.4)

DEU <- terra::vect(system.file("extdata/DEU.gpkg", package="rdwd"))
pdf("MultiAnn.pdf", width=8, height=10)
par(bg="grey90")
for(m in colnames(ma)[8:19])
  {
  terra::plot(DEU, border="grey40")
  berryFunctions::colPoints(ma[-262,]$geogr..Laenge, ma[-262,]$geogr..Breite, ma[-262,m],
                            asp=1.4, # Range=range(ma[-262,8:19]),
                            col=berryFunctions::divPal(200, rev=TRUE), zlab=m, add=T)
  }
dev.off()
berryFunctions::openFile("MultiAnn.pdf")

## End(Not run)

read dwd netcdf data

Description

Read netcdf data. Intended to be called via readDWD().
Note that R.utils and ncdf4 must be installed to unzip and read the .nc.gz files.

Usage

readDWD.nc(
  file,
  gargs = NULL,
  var = "",
  toraster = TRUE,
  quiet = rdwdquiet(),
  ...
)

Arguments

Value

terra::rast() object. Alternatively, if toraster=FALSE, a list with time, lat, lon, var, varname, file and cdf. cdf is the output of ncdf4::nc_open().

Author(s)

Berry Boessenkool, berry-b@gmx.de, Aug 2019

See Also

readDWD()

Examples

## Not run:  # Excluded from CRAN checks, but run in localtests

library(berryFunctions) # for seqPal and colPointsLegend

url <- "daily/Project_TRY/pressure/PRED_199606_daymean.nc.gz"  #  5 MB
url <- "daily/Project_TRY/humidity/RH_199509_daymean.nc.gz"    # 25 MB
file <- dataDWD(url, base=gridbase, joinbf=TRUE, dir=locdir(), read=FALSE)
nc <- readDWD(file)
ncp <- plotRadar(nc, main=paste(terra::longnames(nc), terra::time(nc)), layer=1:3,
                 col=seqPal(), proj="nc", extent="nc")

str(terra::values(nc[[1]])) # obtain actual values into memory

terra::plot(nc[[1]]) # axes 0:938 / 0:720, the number of grid cells
terra::plot(ncp[[1]])# properly projected, per default onto latlon

rng <- range(terra::global(nc[[1:6]], "range", na.rm=TRUE))
terra::plot(nc, col=seqPal(), zlim=rng, maxnl=6)

# Array instead of terra rast:
nc <- readDWD(file, toraster=FALSE)
image(nc$var[,,1], col=seqPal(), asp=1.1)
colPointsLegend(nc$var[,,1], title=paste(nc$varname, nc$time[1]))

# interactive selection of variable:
# nc <- readDWD(file, toraster=FALSE, var="-") # commented out to not block automated tests
str(nc$var)

## End(Not run)

open pdf data

Description

open pdf file. This leads to less failures in the new meta=TRUE

Usage

readDWD.pdf(file, quiet = rdwdquiet(), ...)

Arguments

Value

berryFunctions::openFile() output

system in selectDWD().

Intended to be called via readDWD().

Author(s)

Berry Boessenkool, berry-b@gmx.de, May 2022.

See Also

readDWD()

Examples

## Not run:  # Excluded from CRAN checks, but run in localtests
link <- selectDWD(res="hourly", var="sun", per="r", meta=TRUE)[2]
file <- dataDWD(link, dir=locdir(), read=FALSE)
readDWD(file)

## End(Not run)

read dwd gridded radolan radar data

Description

read gridded radolan radar data. Intended to be called via readDWD().

Usage

readDWD.radar(
  file,
  gargs = NULL,
  toraster = TRUE,
  dividebyten = TRUE,
  quiet = rdwdquiet(),
  ...
)

Arguments

Value

Invisible list with dat (matrix or raster, depending on toraster) and meta (list with elements from header)

Author(s)

Berry Boessenkool, berry-b@gmx.de, Aug 2019. Significant input for the underlying dwdradar::readRadarFile() came from Henning Rust & Christoph Ritschel at FU Berlin.

See Also

readDWD(), especially readDWD.binary()
https://wradlib.org for much more extensive radar analysis in Python
Kompositformatbeschreibung at https://www.dwd.de/DE/leistungen/radolan/radolan.html for format description

Examples

## Not run:  # Excluded from CRAN checks, but run in localtests
# recent radar files
rrf <- indexFTP("hourly/radolan/recent/bin", base=gridbase, dir=tempdir())
lrf <- dataDWD(rrf[773], base=gridbase, joinbf=TRUE, dir=tempdir(), read=FALSE)
r <- readDWD(lrf)
plotRadar(r$dat, main=paste("mm in 24 hours preceding", r$meta$date))

## End(Not run)

read dwd gridded raster data

Description

Read gridded raster data. Intended to be called via readDWD().
Note that R.utils must be installed to unzip the .asc.gz files.

Usage

readDWD.raster(file, gargs = NULL, dividebyten, quiet = rdwdquiet(), ...)

Arguments

Value

terra::rast object

Author(s)

Berry Boessenkool, berry-b@gmx.de, Dec 2018

See Also

readDWD()

Examples

## Not run:  # Excluded from CRAN checks, but run in localtests

rasterbase <- paste0(gridbase,"/seasonal/air_temperature_mean")
ftp.files <- indexFTP("/16_DJF", base=rasterbase, dir=tempdir())
localfiles <- dataDWD(ftp.files[1:2], base=rasterbase, joinbf=TRUE,
                      dir=locdir(), read=FALSE)
rf <- readDWD(localfiles[1])
rf <- readDWD(localfiles[1]) # runs faster at second time due to skip=TRUE
terra::plot(rf)

plotRadar(rf, proj="seasonal", extent=NULL)

testthat::expect_equal(c(terra::minmax(rf)), c(-8.2,4.4))
rf10 <- readDWD(localfiles[1], dividebyten=FALSE)
terra::plot(rf10)
testthat::expect_equal(c(terra::minmax(rf10*1)), c(-82,44))

## End(Not run)

read dwd gridded radklim binary data

Description

read gridded radklim binary data. Intended to be called via readDWD().
Note: needs dwdradar >= 0.2.6 (2021-08-08)

Usage

readDWD.rklim(
  file,
  exdir = NULL,
  unpacked = NULL,
  selection = NULL,
  toraster = TRUE,
  quiet = rdwdquiet(),
  progbar = !quiet,
  ...
)

Arguments

Value

list depending on argument toraster, see there for details

Author(s)

Berry Boessenkool, berry-b@gmx.de, Aug 2021.

See Also

readDWD.binary(), radar locations from https://www.dwd.de/DE/leistungen/radarklimatologie/radklim_kompositformat_1_0.pdf?__blob=publicationFile&v=1

Examples

## Not run:  # Excluded from CRAN checks, but run in localtests
yw_link <- "/5_minutes/radolan/reproc/2017_002/bin/2022/YW2017.002_202203.tar"
# 202006 has untar error on Mac, 2023-04, maybe due to incomplete download
yw_file <- dataDWD(url=yw_link, base=gridbase, joinbf=TRUE, read=FALSE) # 207 MB
x <- readDWD(yw_file, selection=3641:3644)
# 00:30 for tar files, 01:40 for unpacking. 
# If you need a preselection argument, let me know.
terra::plot(x$dat)
plotRadar(x$dat[[1]], extent="rw") # better extent below

f <- system.file("tests//raa01-yw2017.002_10000-2006131525-dwd---bin", package="dwdradar")
# https://stackoverflow.com/a/72207233/1587132 on how to install with tests folder
if(!file.exists(f)){
# Clone from https://github.com/brry/dwdradar:
f <- locdir(file="binary_testfile")
download.file(paste0("https://github.com/brry/dwdradar/raw/master/tests/",
                     "raa01-yw2017.002_10000-2006131525-dwd---bin"), f, mode="wb")
}
x <- dwdradar::readRadarFile(f)
x$dat <- terra::rast(x$dat)
terra::plot(x$dat)
plotRadar(x$dat, extent=c(-360, 380, -4730 ,-3690))

radloc <- read.table(header=T, sep=",", text="
ND, NM, NS  ,   ED, EM, ES
53, 33, 50.4,   06, 44, 53.9
51, 07, 26.5,   13, 45, 48.5
51, 24, 18.5,   06, 57, 49.8
47, 52, 21.3,   08, 00, 24.6
54, 10, 23.2,   12, 06, 25.3
52, 28, 40.3,   13, 23, 13.0
54, 00, 15.8,   10, 02, 48.7
51, 07, 28.7,   13, 46, 07.1
49, 32, 26.4,   12, 24, 10.0
53, 20, 19.4,   07, 01, 25.5
51, 24, 20.2,   06, 58, 01.6
47, 52, 25.0,   08, 00, 13.0
51, 20, 06.0,   08, 51, 09.0
51, 18, 40.3,   08, 48, 07.2
50, 03, 06.0,   08, 34, 05.0
50, 01, 20.8,   08, 33, 30.7
53, 37, 16.5,   09, 59, 47.6
52, 27, 47.0,   09, 41, 53.9
52, 27, 36.2,   09, 41, 40.2
48, 10, 28.9,   12, 06, 06.3
48, 02, 31.7,   10, 13, 09.2
48, 20, 10.9,   11, 36, 42.1
50, 30, 00.4,   11, 08, 06.2
50, 06, 34.7,   06, 32, 53.9
49, 59, 05.1,   08, 42, 46.6
52, 38, 55.2,   13, 51, 29.6
54, 10, 32.4,   12, 03, 29.1
48, 35, 07.0,   09, 46, 58.0
52, 09, 36.3,   11, 10, 33.9")
radloc$x <- radloc$ED + radloc$EM/60 + radloc$ES/3600
radloc$y <- radloc$ND + radloc$NM/60 + radloc$NS/3600
for(i in 1:29) berryFunctions::circle(radloc$x[i], radloc$y[i], 0.9)

## End(Not run)

read subdaily/standard_format dwd data

Description

read subdaily/standard_format dwd data. Intended to be called via readDWD().
All other observational data at dwdbase can be read with readDWD.data(), except for the multi_annual and subdaily/standard_format data.

Usage

readDWD.stand(
  file,
  fast = TRUE,
  fileEncoding = "latin1",
  formIndex = formatIndex,
  quiet = rdwdquiet(),
  ...
)

Arguments

Value

data.frame with column names as per formatIndex. "Q"-columns have "_parameter" appended to their name. A "Date" column has been added. NA-indicators have been processed into NAs.

Author(s)

Berry Boessenkool, berry-b@gmx.de, Oct 2019

See Also

readDWD()

Examples

## Not run:  # Excluded from CRAN checks, but run in localtests

link <- selectDWD(res="subdaily", var="standard_format", per="r")
link <- link[grepl("10381", link, fixed=TRUE)]
# Not ID, according to meta data, hence no longer in column id (2023-04).
file <- dataDWD(link, dir=locdir(), read=FALSE)
sf <- readDWD(file)

sf2 <- readDWD(file, fast=FALSE) # 20 secs!
stopifnot(all.equal(sf, sf2))

plot(sf$Date, sf$SHK, type="l")

# Plot all columns:
if(FALSE){ # not run in any automated testing
tmp <- tempfile(fileext=".pdf")
char2fact <- function(x)
 {
 if(all(is.na(x))) return(rep(-9, len=length(x)))
 if(!is.numeric(x)) as.factor(x) else x
 }
pdf(tmp, width=9)
par(mfrow=c(2,1),mar=c(2,3,2,0.1), mgp=c(3,0.7,0), las=1)
for(i in 3:ncol(sf)-1) plot(sf$Date, char2fact(sf[,i]), type="l", main=colnames(sf)[i], ylab="")
dev.off()
berryFunctions::openFile(tmp)
}

## End(Not run)

Process data from the DWD CDC FTP Server

Description

Read climate meta info textfiles in zip folders downloaded with dataDWD().

Usage

readMeta(file, progbar = TRUE, ...)

Arguments

Value

Invisible named list of data.frames; or a list of lists, if length(file)>1.

Author(s)

Berry Boessenkool, berry-b@gmx.de, 2016 + March 2019

See Also

dataDWD(), readVars(), readDWD()

Examples

# see dataDWD

Process data from the DWD CDC FTP Server

Description

Read climate variables (column meta data) from zip folders downloaded with dataDWD(). The metadata file "Metadaten_Parameter.*txt" in the zip folder file is read, processed and returned as a data.frame.
file can be a vector with several filenames.

Usage

readVars(file, params = dwdparams, quiet = rdwdquiet(), progbar = TRUE)

Arguments

Value

data.frame of the desired dataset, or a named list of data.frames if length(file) > 1.

Author(s)

Berry Boessenkool, berry-b@gmx.de, Jun 2018

See Also

dataDWD(), readDWD(), dwdparams, newColumnNames()
readMeta() for complete Metadaten_Parameter file.
website use case

Examples

# see dataDWD

Reminders when using devtools::release

Description

Reminders when using devtools::release.

Usage

release_questions()

Create leaflet map popup from data.frame rows

Description

Create display character string for leaflet map popup from data.frame rows. This function is not exported, as it is only internally useful. A generic version is available in berryFunctions::popleaf().

Usage

rowDisplay(x)

Arguments

Value

Vector of character strings, one for each row in x.

Author(s)

Berry Boessenkool, berry-b@gmx.de, Feb 2017

See Also

geoIndex

run local tests of rdwd

Description

Run rdwd tests on local machine. Due to time-intensive data downloads, these tests are not run automatically on CRAN.

Usage

runLocalTests(
  dir_data = locdir(),
  dir_exmpl = berryFunctions::packagePath(file = "misc/ExampleTests"),
  start = 1,
  quiet = rdwdquiet()
)

Arguments

Value

Time taken to run tests in minutes

Author(s)

Berry Boessenkool, berry-b@gmx.de, Apr-Oct 2019

See Also

locdir()

Select data from the DWD CDC FTP Server

Description

Select data files for downloading with dataDWD().
The available res/var/per folders with datasets are listed online.
Set ⁠res="", var="", per=""⁠ to avoid the default interactive selection.
The arguments name/id and res/var/per can be vectors.

Usage

selectDWD(
  name = "",
  res = NA,
  var = NA,
  per = NA,
  expand = TRUE,
  id = findID(name, exactmatch = exactmatch, mindex = mindex, quiet = quiet, failempty =
    failempty),
  exactmatch = TRUE,
  mindex = metaIndex,
  failempty = TRUE,
  findex = fileIndex,
  current = FALSE,
  base = dwdbase,
  meta = FALSE,
  quiet = rdwdquiet(),
  ...
)

Arguments

Value

Character string with file path and name(s) in the format "base/res/var/per/filename.zip"

Author(s)

Berry Boessenkool, berry-b@gmx.de, Oct 2016, rewritten May 2022

See Also

dataDWD(), metaIndex, website station selection chapter

Examples

# Give weather station name (must exist in metaIndex):
selectDWD("Potsdam", res="daily", var="kl", per="historical")

# all files for all stations matching "Koeln":
tail(selectDWD("Koeln", res="", var="", per="", exactmatch=FALSE)) # 686 files
findID("Koeln", FALSE)

## Not run:  # Excluded from CRAN checks to save time

# selectDWD("Potsdam") # interactive selection of res/var/per

# directly give station ID:
selectDWD(id="00386", res="daily", var="kl", per="historical")
selectDWD(id=537, "", "", "", "") # 8 files

# period can be abbreviated:
selectDWD(id="5419", res="daily", var="kl", per="h")

# selectDWD is vectorizable!
# since version 1.5.28 (2022-05-12) outer product, not elementwise comparison:
selectDWD("Freiburg", res="daily", var="kl", per="rh")
selectDWD("Freiburg", res=c("daily","monthly"), var="kl", per="r")
selectDWD("Freiburg", res=c("daily","monthly"), var="kl", per="hr")
# get old behaviour (needed e.g. in nearbyStations):
ids <- c(3761,3761, 3603)
# all combinations:
selectDWD(id=ids, res="daily", var="kl", per=c("h","r","r")) # 4
# only given combinations:
selectDWD(id=ids, res="daily", var="kl", per=c("h","r","r"), expand=FALSE) # 3

# all files in all paths matching id:
head( selectDWD(id=c(1050, 386), res="",var="",per="") ) # 277 files
# all files in a given path (if ID is empty):
head(  selectDWD(id="", res="daily", var="kl", per="recent")   ) # 585 files

selectDWD(id=386, res="monthly", var="kl", per="h")

# Meta  data - Description and Beschreibung txt/pdf files.:
# manually select .txt (not pdf) files for automated opening with readDWD.
link <- selectDWD(res="monthly", var="kl", per="h", meta=TRUE) # omit ID/Name!
link
link2 <- grep("\\.txt$", link, value=TRUE)   ;   link2
m <- dataDWD(link2, dir=locdir())
head(m)
# 
# Open PDF files with your system's default Viewer:
dataDWD(link[1], dir=locdir())

## End(Not run)

update rdwd indexes

Description

This is meant to be called with getwd at the rdwd package source code directory to update the indexes with one single call.
To use custom or current indexes, see https://bookdown.org/brry/rdwd/fileindex.html

Usage

updateIndexes(dwdlocal = FALSE, grdlocal = FALSE, metaforce = NA)

Arguments

Value

checkIndex() results

Author(s)

Berry Boessenkool, berry-b@gmx.de, Oct 2019

See Also

createIndex()

Examples

# number of files at dwdbase
#  25'757 (2017-03-14)
# 218'593 (2018-03-25)
# 228'830 (2018-11-26)
# 240'737 (2019-02-19)
# 242'584 (2019-03-11)
# 266'860 (2019-05-15)
# 254'446 (2019-05-30)
# 255'252 (2019-07-31)
# 254'925 (2019-09-17)
# 254'943 (2019-10-26)
# 266'841 (2020-03-16)
# 265'712 (2020-04-10)
# 265'712 (2020-04-24)
# 266'106 (2020-06-01)
# 266'216 (2020-07-06)
# 266'216 (2020-07-28)
# 267'175 (2020-09-21)
# 269'561 (2020-12-03)
# 286'306 (2021-04-02)
# 286'189 (2021-04-08)
# 285'246 (2021-04-23)
# 285'972 (2021-06-02)
# 321'477 (2022-04-07)
# 477'236 (2022-04-28) # 5_minutes files added
# 482'907 (2022-04-29)
# 497'190 (2022-05-13)
# 576'768 (2023-04-06) # lots of duplicate historical files in transition period
# 577'160 (2023-04-12)
# 571'667 (2023-04-14)
# 572'583 (2023-05-10)
# 574'397 (2023-06-16)

# gridbase
#  49'247 (2019-05-26)
#  49'402 (2019-05-30)
#  54'314 (2019-07-31)
#  56'759 (2019-09-17)
#  58'656 (2019-10-26)
#  30'320 (2020-03-16)
#  31'787 (2020-04-10)
#  32'478 (2020-04-24)
#  34'203 (2020-06-01)
#  35'953 (2020-07-06)
#  37'038 (2020-07-28)
#  39'791 (2020-09-21)
#  43'435 (2020-12-03)
#  31'698 (2021-04-02)
#  32'015 (2021-04-08)
#  32'736 (2021-04-23)
#  34'708 (2021-06-02)
#  34'854 (2022-04-07)
#  35'874 (2022-04-28)
#  35'937 (2022-04-29)
#  36'630 (2022-05-13)
#  36'059 (2023-04-06)
#  36'356 (2023-04-12)
#  36'454 (2023-04-14)
#  37'763 (2023-05-10)
#  39'670 (2023-06-16)

Update rdwd development version

Description

Update rdwd to the latest development version on github, if necessary. If the version number or date is larger on github, remotes::install_github() will be called.

Usage

updateRdwd(
  pack = "rdwd",
  user = "brry",
  vignette = NA,
  quiet = rdwdquiet(),
  ...
)

Arguments

Value

data.frame with version information

Author(s)

Berry Boessenkool, berry-b@gmx.de, Nov 2019

See Also

remotes::install_github()

Examples

# updateRdwd()

valid fileType values

Description

fileType values that have a reading subfunction readDWD.ftype().

Usage

validFileTypes

Format

An object of class character of length 13.