RchivalTag Tutorial | Part 2 (Interactive) Time Series Data Plots

RchivalTag ggplot dygraphs time-series daytime-shading

In this second part of the RchivalTag tutorial series, we will learn how to read, analyze and visualize depth and temperature time series data.

Robert K. Bauer http://www2.hawaii.edu/~rkbauer (Hawai’i Institute of Marine Biology
University of Hawai’i at Mānoa)https://scholar.google.com/citations?hl=en&user=J-0_tdbR2tgC
11-20-2020

Getting started | Requirements

To run this tutorial we will need RchivalTag version >= 0.1.5 and oceanmap version >= 0.13. You can find the newest versions of both packages on github

## install or load package
# install.packages("RchivalTag") # from CRAN
# library(xfun)
# install_github("rkbauer/oceanmap") # newest version from github
# install_github("rkbauer/RchivalTag") # newest version from github
library("RchivalTag")

## Package overview and version:
?RchivalTag 
help(package="RchivalTag") ## list of functions

How to load Time Series data from archival and Pop-up archival tags

Recovered Archival Tags provide researchers with detailed information about the vertical behavior of fish and other aquatic animals such as turtles and mammals. Pop-Up archival tags can also transmit (subsets of) such data (e.g. miniPATs from Wildlife Computers) to the ARGOS satellite system, and thus do not need to be recovered. We therefore distinguish between high resolution time series data (3-10s sampling resolution from recovered tags, e.g. ###-Archive.csv files from recovered miniPATs, mk10, mk9 etc.) and and low resolution time series data (150-600s sampling resolution from transmitted data sets, e.g. ###-Series.csv files from miniPATs). Time Series data is incredibly powerful and may include data from different sensors (depth, temperature, acceleration, etc.). Here we will focus on Depth Time Series data. To read this data, you might be tempted to read in the data via R’s read.table or read.csv functions:

## read example transmitted Depth Temperature Time Series data:
ts_file <- system.file("example_files/104659-Series.csv",package="RchivalTag")
ts_df <- read.csv(ts_file)
str(ts_df,1)

'data.frame':   4163 obs. of  14 variables:
 $ DeployID       : Factor w/ 1 level "15P1019": 1 1 1 1 1 1 1 1 1 1 ...
 $ Ptt            : int  104659 104659 104659 104659 104659 104659 104659 104659 104659 104659 ...
 $ DepthSensor    : logi  NA NA NA NA NA NA ...
 $ Source         : Factor w/ 1 level "Transmission": 1 1 1 1 1 1 1 1 1 1 ...
 $ Instr          : Factor w/ 1 level "MiniPAT": 1 1 1 1 1 1 1 1 1 1 ...
 $ Day            : Factor w/ 15 levels "07-Aug-2016",..: 1 1 1 1 1 1 1 1 1 1 ...
 $ Time           : Factor w/ 288 levels "00:00:00","00:05:00",..: 99 100 101 102 103 104 105 106 107 108 ...
 $ LocationQuality: logi  NA NA NA NA NA NA ...
 $ Latitude       : logi  NA NA NA NA NA NA ...
 $ Longitude      : logi  NA NA NA NA NA NA ...
 $ Depth          : num  105 85 72 23 46 16.5 36 23 23 23 ...
 $ DRange         : num  14 7.25 7.25 3.5 7 3.5 3.75 3.5 3.5 3.5 ...
 $ Temperature    : num  14.2 14.2 14.2 16.3 15.1 18 15.5 18 18 17.6 ...
 $ TRange         : num  0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 ...
head(ts_df,3)

  DeployID    Ptt DepthSensor       Source   Instr         Day
1  15P1019 104659          NA Transmission MiniPAT 07-Aug-2016
2  15P1019 104659          NA Transmission MiniPAT 07-Aug-2016
3  15P1019 104659          NA Transmission MiniPAT 07-Aug-2016
      Time LocationQuality Latitude Longitude Depth DRange
1 08:10:00              NA       NA        NA   105  14.00
2 08:15:00              NA       NA        NA    85   7.25
3 08:20:00              NA       NA        NA    72   7.25
  Temperature TRange
1        14.2   0.25
2        14.2   0.25
3        14.2   0.25

This works out perfectly, but I do not encourage you to do so since some transmitted data sets have gaps. Such gaps are not indicated in the transmitted data sets. In addition, we should convert the Day and Time columns to a datetime format so that we can plot and analyze our data with ease. Fortunately, RchivalTag has it’s own function to read time series data from different manufacturers (e.g. Wildlife Computers and LOTEK) that fills gaps with NAs and does the datetime conversion automatically.

## read example transmitted Depth Temperature Time Series data via RchivalTag:
ts_file <- system.file("example_files/104659-Series.csv",package="RchivalTag")
ts_df <- read_TS(ts_file)
str(ts_df,1)

'data.frame':   4320 obs. of  11 variables:
 $ DeployID   : chr  "15P1019" "15P1019" "15P1019" "15P1019" ...
 $ Ptt        : int  104659 104659 104659 104659 104659 104659 104659 104659 104659 104659 ...
 $ date       : Date, format: "2016-08-07" ...
 $ datetime   : POSIXct, format: "2016-08-07 00:00:00" ...
 $ DepthSensor: logi  NA NA NA NA NA NA ...
 $ Source     : chr  NA NA NA NA ...
 $ Instr      : chr  NA NA NA NA ...
 $ Depth      : num  NA NA NA NA NA NA NA NA NA NA ...
 $ DRange     : num  NA NA NA NA NA NA NA NA NA NA ...
 $ Temperature: num  NA NA NA NA NA NA NA NA NA NA ...
 $ TRange     : num  NA NA NA NA NA NA NA NA NA NA ...

We can see that the time series data has been converted. We have a date and datetime column in Date and POSIXct format (instead of the unformatted Day and Time columns before). In addition, the dataframe now starts with the “2016-08-07 00:00:00” while before it was “2016-08-07 08:10:00”, so the gaps have been completed even those before the tags deployment. This is an important step, also for other conversions (e.g. to histogram data) that we will discuss later.

I would like to further highlight the advanced date time conversion capabilities of RchivalTag since this can be a tricky problem (e.g. when the data is in a time and or local language format). For example, this is the same data but with another date time format. Actually, the time format in the different tagging data files often differs (e.g. in case of Wildlife Computers), so the next code is actually pretty interesting.

Let’s try to read the same data in another time format.

## other date_format:
ts_file2 <- system.file("example_files/104659-Series_date_format2.csv",package="RchivalTag")
ts_miniPAT2 <- try(read_TS(ts_file2)) # run to see error message

Error in .fact2datetime(ts_df$datetime_tmp, date_format = date_format,  : 
  date concersion failed! Please revise current'date_format': %d-%b-%Y %H:%M:%S to 07-08-2016 08:10:00

So RchivalTag tells us what to do. We need to adjust the standard date format to ‘07-08-2016 08:10:00’. Let’s do that:

## changing the date format and 
ts_miniPAT2 <- read_TS(ts_file2, date_format = "%d-%m-%Y %H:%M:%S")
str(ts_miniPAT2,1)

'data.frame':   4320 obs. of  11 variables:
 $ DeployID   : chr  "15P1019" "15P1019" "15P1019" "15P1019" ...
 $ Ptt        : int  104659 104659 104659 104659 104659 104659 104659 104659 104659 104659 ...
 $ date       : Date, format: "2016-08-07" ...
 $ datetime   : POSIXct, format: "2016-08-07 00:00:00" ...
 $ DepthSensor: logi  NA NA NA NA NA NA ...
 $ Source     : chr  NA NA NA NA ...
 $ Instr      : chr  NA NA NA NA ...
 $ Depth      : num  NA NA NA NA NA NA NA NA NA NA ...
 $ DRange     : num  NA NA NA NA NA NA NA NA NA NA ...
 $ Temperature: num  NA NA NA NA NA NA NA NA NA NA ...
 $ TRange     : num  NA NA NA NA NA NA NA NA NA NA ...

This worked out! However, sometimes the data is formatted in another language format as indicated earlier:

## other language format and separator:
ts_file_ES <- system.file("example_files/104659-Series_date_format_ES.csv",package="RchivalTag") # spanish date format
ts_miniPAT_ES <- try(read_TS(ts_file_ES)) # run to see error message

                                                                           WC.miniPAT.data.with.spanish.date.format
1 DeployID;Ptt;DepthSensor;Source;Instr;Day;Time;LocationQuality;Latitude;Longitude;Depth;DRange;Temperature;TRange
2                                       15P1019;104659;;Transmission;MiniPAT;07/abr/16;08:10:00;;;;105;14;14.2;0.25
3                                      15P1019;104659;;Transmission;MiniPAT;07/abr/16;08:15:00;;;;85;7.25;14.2;0.25
4                                      15P1019;104659;;Transmission;MiniPAT;07/abr/16;08:20:00;;;;72;7.25;14.2;0.25
5                                       15P1019;104659;;Transmission;MiniPAT;07/abr/16;08:25:00;;;;23;3.5;16.3;0.25
6                                         15P1019;104659;;Transmission;MiniPAT;07/ene/16;08:30:00;;;;46;7;15.1;0.25
Error in `$<-.data.frame`(`*tmp*`, "datetime_nm", value = numeric(0)) : 
  replacement has 0 rows, data has 11

In this case we need to change the separator, the date and language format to ‘07/ene/16;08:30:00’. Turns out this is a spanish format.

ts_miniPAT_ES <- read_TS(ts_file_ES,skip=1,sep=";",header = TRUE, 
                         date_format = "%d/%b/%y %H:%M:%S",lang_format = "es") # spanish date format
head(ts_miniPAT_ES,2)

  DeployID    Ptt       date            datetime DepthSensor Source
1  15P1019 104659 2016-01-07 2016-01-07 00:00:00          NA   <NA>
2  15P1019 104659 2016-01-07 2016-01-07 00:05:00          NA   <NA>
  Instr Depth DRange Temperature TRange
1  <NA>    NA     NA          NA     NA
2  <NA>    NA     NA          NA     NA

As mentioned earlier, read_TS also works with data from other manufacturers (currently only LOTEK). Here is the same data in the LOTEK PSAT Dive format:

## load same data in LOTEK format
ts_file <- system.file("example_files/104659_PSAT_Dive_Log.csv",package="RchivalTag")
ts_df <- read_TS(ts_file,date_format="%m/%d/%Y %H:%M:%S")
head(ts_df,2) ## attention no identifier (Ptt, Serial, DeployID) included!

  DeployID Ptt       date            datetime Validity Temperature
1       NA  NA 2016-08-07 2016-08-07 00:00:00       NA          NA
2       NA  NA 2016-08-07 2016-08-07 00:05:00       NA          NA
  Depth
1    NA
2    NA

This worked out, however, LOTEK does not include the DeployID and Ptt information in their time series data. We therefore need to add this information manually. This will be important when further converting and visualizing our data.

## add missing DeployID and PTT information to LOTEK depth temperature time series data:
ts_df$DeployID <- ts_df$Ptt <- "104659"

How to plot time series data

Now let’s plot this data. RchivalTag has several plotting functions for time series data. These include:

In the first part of this tutorial we have seen how to read and plot histogram data, which can also be obtained from depth time series (DepthTS) data. Although histogram data is an efficient data product to provide an overall overview of general depth layer preferences, its value to analyze diel vertical behavior is limited. On the other side, analyzing DepthTS data from one ore more individual can be tricky, because of the duration and variability in these data sets. Still there are some handy functions in RchivalTag that can help to perform this job, that we will learn in a sec. For now, let’s say we are interested in getting a general idea of the diel vertical behavior depth preferences (which is a good starting point of a more in-depth analysis of DepthTS data). For this purpose, RchivalTag includes the function ggboxplot_DepthTS_by_hour.

ts_file <- system.file("example_files/104659-Series.csv",package="RchivalTag")
ts_df <- read_TS(ts_file)
ggboxplot_DepthTS_by_hour(ts_df)

Let’s add position data to obtain twilight and nighttime shadings. We could add here fixed position values.

ts_df$Lon <- 5; ts_df$Lat <- 43
ts_df2 <- get_DayTimeLimits(ts_df)
ggboxplot_DepthTS_by_hour(ts_df2,ylim=c(0,100))

However, it is more accurate to use the maximum likelihood positions from the GPE3 model runs, especially if the animal is highly migratory and thus moving over large distances. This is particularly important when we analyze the raw depth time series data. By contrast, the ggboxplot_DepthTS_by_hour function will calculate the average timing of sunrise and sunset.

We can read in the maximum likelihood positions via the get_geopos function. We will learn more about this function and the visualizing of the tracks in the 4th part of this tutorial.

library(dplyr)
gpe3_file <- system.file("example_files/15P1019-104659-1-GPE3.csv",package="RchivalTag")
tracks <- get_geopos(gpe3_file)

Loading tagging tracks from file: /home/work/R/x86_64-pc-linux-gnu-library/3.6/RchivalTag/example_files/15P1019-104659-1-GPE3.csv
add <- tracks %>% select(DeployID,Ptt,Lat,Lon,date) %>% 
                  group_by(DeployID,Ptt,date) %>%
                  summarise(Lat=mean(Lat),Lon=mean(Lon)) %>%
                  mutate(datetime=RchivalTag:::.date2datetime(date))
add <- classify_DayTime(add) %>% select(-datetime)

dat <- ts_df %>% select(-Lon,-Lat)
ts_df2 <- add %>% select(-DeployID) %>% inner_join(dat,by=c("date","Ptt","DeployID"))

ggboxplot_DepthTS_by_hour(ts_df2,ylim=c(0,100))

Now let’s checkout the underlying, raw depth time series data. We can directly plot the entire data set with the plot_DepthTS function:

plot_DepthTS(ts_df)

This function has a bunch of arguments that we will go through. Some basic arguments like xlim to select the dates to be plotted:

plot_DepthTS(ts_df,xlim = "2016-08-10")

Another way to produce this figure is via the empty.plotTS function that will draw an empty plot first and then gives us a line.

# example for empty.plotTS and adding time series data as line:
empty.plot_TS(xlim="2016-08-10",ylim=c(100,0))
lines(ts_df$datetime, ts_df$Depth)

Many aquatic animals show diurnal migration patterns so we might be interested in shading the night-time and twilight periods. To do so we need to add at least the Longitude and Latitude information since the sunrise and sunset timings change differ with location. Let’s do this manually.

# plot also day night time information:
ts_df$Lon <- 5; ts_df$Lat <- 43 # manual example, please take Lon/Lat data from (GPE3) model outputs for your analysis instead. (check get_geopos-function from RchivalTag)
ts_df <- classify_DayTime(ts_df)
plot_DepthTS(ts_df,xlim = c("2016-08-10","2016-08-15"),plot_DayTimePeriods = T)

We can do the same figure with an empty plot and the vertical track as line on top:

# alternative:
plot_DepthTS(ts_df, xlim=c("2016-08-10","2016-08-12"), plot_DayTimePeriods = TRUE, type='n')
lines(ts_df$datetime, ts_df$Depth)

Interactive time series data plots

The dy_DepthTS function allows us to to the same figures but with the interactive interface of the dygraphs package.

ts_file <- system.file("example_files/104659-Series.csv",package="RchivalTag")
ts_df <- read_TS(ts_file)
ts_df$Serial <- ts_df$DeployID

dy_DepthTS(ts_df)

Adding night and twilight shadings:

ts_df$Lon <- 5; ts_df$Lat <- 43
dy_DepthTS(ts_df)

Some further arguments:

dy_DepthTS(ts_df, xlim = unique(ts_df$date)[2:3], plot_DayTimePeriods = FALSE, doRangeSelector= FALSE)

Remove grid, plot points:

library(dygraphs)
dg <- dy_DepthTS(ts_df, xlim = unique(ts_df$date)[2:3], plot_DayTimePeriods = FALSE, drawPoints = TRUE)
dg <- dyOptions(dg,drawGrid=FALSE)
dg

Gaps in the Time Series Data

RchivalTag fills gaps with NAs. However, both plot_DepthTS and dy_DepthTS function can also deal with data data gaps:

ts_gaps <- ts_df
ts_gaps$Depth[c(300:800)] <- NA
ts_cut <- ts_df[-c(300:800),]

plot_DepthTS(ts_gaps)
plot_DepthTS(ts_cut) # same result


dy_DepthTS(ts_gaps) ## same figure with dy_DepthTS

dy_DepthTS(ts_cut) # same result

Depth-Temperature Time Series Data

plot_DepthTS is a base-graph. So it comes with almost all the functionalities of the base plots. We could for example also add another line and axis for the temperature data.

par(mar=c(5,4,4,5)) ## change margins to add second axis label
ts_sub <- plot_DepthTS(ts_df,xlim = c("2016-08-10","2016-08-15"),plot_DayTimePeriods = T, Return=T)

par(new=T) ## plot on top of first depth time series plot
ylim <- range(pretty(range(ts_sub$Temperature)))
plot(ts_sub$datetime, ts_sub$Temperature, xlim=range(ts_sub$datetime), ylim = ylim, type='n',axes=F,xlab="",ylab="") # create empty plot
lines(ts_sub$datetime, ts_sub$Temperature,col="orange") # add line
axis(4, at = pretty(ylim)) # add axis
mtext(side=4,"Temperature (°C)",las=0,line=2.3) # add axis label

Let’s create a second line for the temperature time series data with dy_DepthTS and dygraphs:

dg <- dy_DepthTS(ts_df) # run via dy_DepthTS to get shadings

## manually create the same figure with a second line
dat <- ts_df[,c("datetime","Depth","Temperature")]
dat$datetime <- as.POSIXct(dat$datetime,tz = "UTC")
dat_xts <- xts::xts(dat[,2:3],order.by=dat$datetime)
dg2 <- dygraph(dat_xts) # create initial dygraph object
dg2$x$shadings <- dg$x$shadings ## assign shadings

 # define limits of y-axes
ylim1 <- c(max(ts_df$Depth,na.rm=T),-1)
ylim2 <- c(12,30)

dg2 <- dg2 %>% dyRangeSelector() %>% 
  dySeries("Temperature", axis = "y2") %>% ## set up second y-axis
  dyOptions(labelsUTC = TRUE, strokeWidth = 1) %>% ## set datetime format to UTC (default is local time zone)
  dyAxis("y", label = "Depth", valueRange = ylim1) %>% ## set limits and label of y1-axis
  dyAxis("y2", label = "Temperature", valueRange = ylim2) ## set limits and label of y2-axis
dg2

Well this worked, but it was quite tricky and it’s not that pretty either. RchivalTag has some additional functions to illustrate depth-temperature time series data in a much nicer way. One option is plot_DepthTempTS. In this case, adjacent depth-temperature records will be linearly interpolated. The outcome looks good, but is not recommended for low resolution time series data (>= 300 s sampling resolution), because the potential interpolation between distant records might be misleading (e.g. 30 degrees at the surface and 10 degrees at 400m leads to a false temperature profile, with 20 degrees at around 200 m)… .

library(oceanmap)
data(cmap)
par(mar=c(5,4,4,0)) ## change margins back to default
plot_DepthTempTS(ts_df,xlim = c("2016-08-10","2016-08-15"))

Another option is to interpolate the temperature based on daily resamples (average profiles) via the plot_DepthTempTS_resampled. Both of these functions are adaptations of plot_DepthTS, which means we have similar arguments (e.g. to produce night-time and twilight shadings).

plot_DepthTempTS_resampled(ts_df,xlim = c("2016-08-10","2016-08-15"),plot_DayTimePeriods = T,tz="HST")

Another option is to estimate the temperature profiles from external data sets. PDT-data (PAT-Style Depth-Temperature profiles) from Wildlife Computers are one example for that.

ts_df$Temperature <- c()
pdt_file <- system.file("example_files/104659-PDTs.csv",package="RchivalTag")
PDT <- read_PDT(pdt_file)

running pdt_file /home/work/R/x86_64-pc-linux-gnu-library/3.6/RchivalTag/example_files/104659-PDTs.csv 
plot_DepthTempTS_resampled_PDT(ts_df,PDT,xlim = c("2016-08-10","2016-08-15"),plot_DayTimePeriods = T,tz="HST",do_interp = F)

We will discuss this the different types of temperature profiles in the 3rd part of this tutorial.

Citation

For attribution, please cite this work as

Bauer (2020, Nov. 20). Marine Biologging & Data Science | Blog: RchivalTag Tutorial | Part 2 (Interactive) Time Series Data Plots. Retrieved from http://oceantags.com/posts/RchivalTag_Tutorials_Part2_Time_Series_Data/

BibTeX citation

@misc{bauer2020rchivaltag,
  author = {Bauer, Robert K.},
  title = {Marine Biologging & Data Science | Blog: RchivalTag Tutorial | Part 2 (Interactive) Time Series Data Plots},
  url = {http://oceantags.com/posts/RchivalTag_Tutorials_Part2_Time_Series_Data/},
  year = {2020}
}