The following demonstrates how to grab files from the EDI Repository. These functions offer similar services to those from the ‘EDIUtils’ package but are opinionated to work similarly to other workflows in ‘deltadata’.
Reading in entities from EDI
The getEDI() function is a one-stop shop function that
allows us to download entities from an EDI repository. To get started,
provide the function with a URL to the data package of interest.
# Data package of the CDFW IEP SLS Survey
possibleEntities <- getEDI("https://portal.edirepository.org/nis/mapbrowse?packageid=edi.534.9")## Specify files to download:
## name extension size
## 1 Catch.csv csv 331.2 KiB
## 2 FishCodes.csv csv 2.3 KiB
## 3 Lengths.csv csv 7.3 MiB
## 4 MeterCorrections.csv csv 15.8 KiB
## 5 SLS.csv csv 7.5 MiB
## 6 Station_Lookup.csv csv 7.1 KiB
## 7 TowInfo.csv csv 384.9 KiB
## 8 WaterInfo.csv csv 645.7 KiB
## 9 SLSIntegrateEDI.R R 21.8 KiB
## 10 SLSTables.rds rds 824.7 KiB
## 11 SLS_Metadata.pdf pdf 855.7 KiB
## description
## 1 Fish catch data from the Smelt Larval Survey
## 2 Taxa and corresponding fish codes
## 3 Data related to fish size, including fork length, yolk sac, and adipose fin of measured individuals
## 4 Instrument information for flow meters used throughout the seasons
## 5 joined long formatted data frame of the base tables. This table is produced from the LTMRdata package
## 6 theoretical GPS coordinates of all SLS stations
## 7 metrics relating directly to the tow
## 8 metrics relating directly to water measured during the tow
## 9 R code to join 6 relational tables together (excluding "FishCodes.cs") to output the "SLS.csv" file.
## 10 Compressed data file of the 7 relational tables uploaded in csv format (containing exactly the same data). This file is for use in R. Once read, the 7 relational tables are structured in a list form, with each table being a named element of the list
## 11 Formatted metadataWhen only a URL is provided, the function will return a list of
entities available to download. The name is the file name
of the entity, extension is the file extension,
size is the file size, and description is a
short description of the entity provided by the creators. This printed
table contains only a subset of the entire data.frame, which can be
accessible if you assign the function to a variable.
# Full data.frame has more information than what is printed
head(possibleEntities)## name extension size sizeBytes
## 1 Catch.csv csv 331.2 KiB 339194
## 2 FishCodes.csv csv 2.3 KiB 2406
## 3 Lengths.csv csv 7.3 MiB 7642912
## 4 MeterCorrections.csv csv 15.8 KiB 16230
## 5 SLS.csv csv 7.5 MiB 7856964
## 6 Station_Lookup.csv csv 7.1 KiB 7265
## description
## 1 Fish catch data from the Smelt Larval Survey
## 2 Taxa and corresponding fish codes
## 3 Data related to fish size, including fork length, yolk sac, and adipose fin of measured individuals
## 4 Instrument information for flow meters used throughout the seasons
## 5 joined long formatted data frame of the base tables. This table is produced from the LTMRdata package
## 6 theoretical GPS coordinates of all SLS stations
## link
## 1 https://pasta.lternet.edu/package/data/eml/edi/534/9/81f62ff73abedbc542ee5e890f527af9
## 2 https://pasta.lternet.edu/package/data/eml/edi/534/9/d80798177ea89ffbe78bcaee38dc51be
## 3 https://pasta.lternet.edu/package/data/eml/edi/534/9/4653e53fa60d50832adb576d721abde5
## 4 https://pasta.lternet.edu/package/data/eml/edi/534/9/c29d02c289d2a3cd78e15b12135e2ff3
## 5 https://pasta.lternet.edu/package/data/eml/edi/534/9/bdcccddc9053eebadded31ec92040128
## 6 https://pasta.lternet.edu/package/data/eml/edi/534/9/f1a7a06f59e795bfae649619fdf84e1d
## id
## 1 81f62ff73abedbc542ee5e890f527af9
## 2 d80798177ea89ffbe78bcaee38dc51be
## 3 4653e53fa60d50832adb576d721abde5
## 4 c29d02c289d2a3cd78e15b12135e2ff3
## 5 bdcccddc9053eebadded31ec92040128
## 6 f1a7a06f59e795bfae649619fdf84e1dTo download file(s), we can provide the values of interest in the
name column to the function.
entities <- getEDI("https://portal.edirepository.org/nis/mapbrowse?packageid=edi.534.9",
files = c("Catch.csv", "SLSTables.rds", "SLS_Metadata.pdf"))## Downloading Catch.csv ; file size: 331.2 KiB
## Downloading SLSTables.rds ; file size: 824.7 KiB
## Downloading SLS_Metadata.pdf ; file size: 855.7 KiBFiles are stored differently
The function download and stores the entities based on what the file extension. Specifically, ‘.csv’ and ‘.rds’ files are read directly into R, while all other file types are downloaded to the temporary folder and a link to the file is provided.
# csv files are read in directly
head(entities$Catch.csv)## Date Station Tow FishCode Catch X1.4.Subsampled X1.2.Subsampled CatchID
## 1 2009-01-05 902 1 49 7 0 0 4
## 2 2009-01-05 906 1 2 1 0 0 5
## 3 2009-01-05 912 1 49 1 0 0 6
## 4 2009-01-05 914 1 49 1 0 0 7
## 5 2009-01-05 915 1 49 6 0 0 8
## 6 2009-01-05 918 1 49 1 0 0 9
# rds files are read in directly, outputted as a list
lapply(entities$SLSTables, head)## $Catch
## Date Station Tow FishCode Catch X1.4.Subsampled X1.2.Subsampled CatchID
## 1 2009-01-05 902 1 49 7 0 0 4
## 2 2009-01-05 906 1 2 1 0 0 5
## 3 2009-01-05 912 1 49 1 0 0 6
## 4 2009-01-05 914 1 49 1 0 0 7
## 5 2009-01-05 915 1 49 6 0 0 8
## 6 2009-01-05 918 1 49 1 0 0 9
##
## $Lengths
## Date Station Tow FishCode Length EntryOrder YolkSacOrOilPresent
## 1 2009-01-05 902 1 49 6 274 FALSE
## 2 2009-01-05 902 1 49 6 275 FALSE
## 3 2009-01-05 902 1 49 6 276 FALSE
## 4 2009-01-05 902 1 49 6 277 FALSE
## 5 2009-01-05 902 1 49 6 278 FALSE
## 6 2009-01-05 902 1 49 6 279 FALSE
##
## $MeterCorrections
## StudyYear MeterSerial CalibrationDate kFactor Notes
## 1 1994 7539 1993-10-01 0.026858 <NA>
## 2 1994 9794 1994-08-31 0.026974 <NA>
## 3 1994 9887 1994-08-31 0.025497 <NA>
## 4 1994 10369 1994-08-31 0.026630 <NA>
## 5 1994 10378 1994-08-31 0.026577 <NA>
## 6 1994 11228 1994-08-30 0.026947 <NA>
##
## $TowInfo
## Date Station Tow Time Tide BottomDepth CableOut Duration
## 1 2009-01-05 902 1 1899-12-30 02:12:00 4 26 110 10
## 2 2009-01-05 906 1 1899-12-30 06:23:00 2 39 140 10
## 3 2009-01-05 910 1 1899-12-30 04:49:00 4 40 155 10
## 4 2009-01-05 912 1 1899-12-30 05:26:00 4 36 140 10
## 5 2009-01-05 914 1 1899-12-30 04:08:00 4 35 110 10
## 6 2009-01-05 915 1 1899-12-30 02:41:00 4 21 90 10
## NetMeterSerial NetMeterStart NetMeterEnd NetMeterCheck
## 1 21920 103454 124337 20883
## 2 21920 247554 267406 19852
## 3 21920 188797 211813 23016
## 4 21920 225510 247557 22047
## 5 21920 167578 188797 21219
## 6 21920 124337 146419 22082
## Comments
## 1 crew: KE, KF, LD, VA
## 2 crew: KE, KF, LD, VA
## 3 crew: KE, KF, LD, VA. Sample splashed when pouring into formalin jar. Less than 5% of sample.
## 4 crew: KE, KF, LD, VA. Re-tow due to incorrect cable length
## 5 crew: KE, KF, LD, VA. Net will come up to 25 feet
## 6 crew: KE, KF, LD, VA
##
## $WaterInfo
## Survey Date Station TopTemp TopEC BottomEC Secchi FNU NTU StartLat
## 1 1 2009-01-05 902 7.4 858 777 92 NA NA <NA>
## 2 1 2009-01-05 906 8.0 469 478 129 NA NA <NA>
## 3 1 2009-01-05 910 8.2 548 587 77 NA NA <NA>
## 4 1 2009-01-05 912 8.2 997 991 63 NA NA <NA>
## 5 1 2009-01-05 914 7.8 604 601 157 NA NA <NA>
## 6 1 2009-01-05 915 7.5 803 801 140 NA NA <NA>
## StartLatDeg StartLatMin StartLatSec StartLong StartLongDeg StartLongMin
## 1 <NA> <NA> <NA> <NA> <NA> <NA>
## 2 <NA> <NA> <NA> <NA> <NA> <NA>
## 3 <NA> <NA> <NA> <NA> <NA> <NA>
## 4 <NA> <NA> <NA> <NA> <NA> <NA>
## 5 <NA> <NA> <NA> <NA> <NA> <NA>
## 6 <NA> <NA> <NA> <NA> <NA> <NA>
## StartLongSec EndLat EndLatDeg EndLatMin EndLatSec EndLong EndLongDeg
## 1 <NA> <NA> <NA> <NA> <NA> <NA> <NA>
## 2 <NA> <NA> <NA> <NA> <NA> <NA> <NA>
## 3 <NA> <NA> <NA> <NA> <NA> <NA> <NA>
## 4 <NA> <NA> <NA> <NA> <NA> <NA> <NA>
## 5 <NA> <NA> <NA> <NA> <NA> <NA> <NA>
## 6 <NA> <NA> <NA> <NA> <NA> <NA> <NA>
## EndLongMin EndLongSec Comments
## 1 <NA> <NA> <NA>
## 2 <NA> <NA> <NA>
## 3 <NA> <NA> <NA>
## 4 <NA> <NA> <NA>
## 5 <NA> <NA> <NA>
## 6 <NA> <NA> <NA>
##
## $Station_Lookup
## ID Station
## 1 1 340
## 2 2 342
## 3 3 343
## 4 4 344
## 5 5 345
## 6 6 346
## Description
## 1 Napa River. Along Vallejo seawall and park @ 10-12 feet depth
## 2 Napa River. Tow across South Slough on west bank @ 22 feet depth
## 3 Napa River. Tow between lights #9 and #11 @ 10 Feet depth
## 4 Napa River. Just past railroad bridge across Fagan Sl. @ 15 Feet depth
## 5 Napa River. Just past entrance to marina up to Cuttings Wharf @ 15 feet depth
## 6 Napa River. Just past last bend before Hwy 12 bridge @ 17 feet depth
## Lat Long
## 1 38 05 57.2 122 15 47.8
## 2 38 08 46.5 122 17 19.5
## 3 38 10 56.5 122 18 33.4
## 4 38 12 45.7 122 18 31.3
## 5 38 13 25.8 122 18 32.3
## 6 38 14 11.0 122 17 13.8
##
## $FishCodes
## Fish.Code Taxa
## 1 18 Ameiurus melas
## 2 76 Symphurus atricauda
## 3 72 Notropis lutrensis
## 4 73 Pimephales promelas
## 5 74 Tridentiger barbatus
## 6 75 Tridentiger spp.
# All other file types are temporarily downloaded and a file path provided
entities$SLS_Metadata## [1] "/tmp/Rtmp821uvh/SLS_Metadata.pdf"You can use shell.exec() to open any downloaded
files.
shell.exec(entities$SLS_Metadata)