loading seer data into R - r

Question

I am trying to load SEER data from ASCII files. There is only a .sas load file which I am trying to convert into an R load command.
the .sas load file looks like this:
filename seer9 './yr1973_2015.seer9/*.TXT';
data in;
infile seer9 lrecl=362;
input
# 1 PUBCSNUM $char8. /* Patient ID */
# 9 REG $char10. /* SEER registry */
# 19 MAR_STAT $char1. /* Marital status at diagnosis */
# 20 RACE1V $char2. /* Race/ethnicity */
# 23 NHIADE $char1. /* NHIA Derived Hisp Origin */
# 24 SEX $char1. /* Sex */
I have the following code to try an replicate a similar loading process:
data <- read.table("OTHER.TXT",
col.names = c("pubcsnum", "reg", "mar_stat", "race1v", "nhaide", "sex"),
sep = c(1, 9, 19, 20, 23, 24))
If I use the separegument I get the following error:
Error in read.table("OTHER.TXT", col.names = c("pubcsnum", "reg", "mar_stat",
:invalid 'sep' argument
If I dont use the sep argument I get the following error:
Error in scan(file = file, what = what, sep = sep, quote = quote, dec = dec,
:
line 1 did not have 133 elements
Does anyone have experience loading seer data? Does anyone have a suggestion why this isn't working?
*of note when I use the fill = TRUE argument, the second error line 1 did not have 133 elements doesn't occur anymore, BUT the data is not correct when I evaluate the first few observations. I further confirmed by evaluating a known variable sex :
> summary(data$sex)
Min. 1st Qu. Median Mean 3rd Qu. Max.
0.000e+00 2.000e+00 3.020e+03 7.852e+18 9.884e+13 2.055e+20
where the values are 1/2 and the summary is nonsensical

Answer 1

So the other comments and answers point out most of this but here's a more complete answer for your exact problem. I have heard of many people struggling with these ASCII files (including many related but, not very simple packages) and I wanted to answer these for anyone else searching.
Fixed width files
These SEER "ASCII" files are actually fixed width text files (ASCII is an encoding standard not a file format). This means that there is no delimiter character (e.g. "," or "\t") that separates the fields (in a .csv or .tsv).
Instead, each field is defined by a start and end position in the line (sometimes a start position and the field width/length). This is what we see the in .sas file that you summarize:
input
# 1 PUBCSNUM $char8. /* Patient ID */
# 9 REG $char10. /* SEER registry */
...
What does this mean?
the first Patient ID field starts at position 1 and has a length of 8 (from $char8, similar to precision in SQL schemas etc.) which means it ends at position 8.
the second field, SEER registry ID, starts at position 9 (1 + 8 from the previous field) and has a length of 10 (again from $char10) which means it ends at position 18.
etc.
Where the # number consistently increases so the fields don't overlap.
Reading fixed width files
I find the readr::read_fwf() function nice and simple, mostly because it has a couple of helper functions, namely fwf_positions() that tell it how to define each field by start and end (or widths, with fwf_widths()).
So, to read just these two fields from the file we can do:
read_fwf(<file>, fwf_positions(start=c(1, 9), end=c(8, 18), col_names=c("patient_id", "registry_id")))
Where col_names is only there to rename the columns.
Helper script.
I have struggled with these before so I actually wrote some code that reads that .sas file and extracts the start positions, widths, column names and descriptions.
Here is the entire thing, just replace the file name:
## Script to read the SEER file dictionary and use it to read SEER ASCII data files.
library(tidyverse)
library(stringr)
#### Reading the file dictionary ----
## https://seer.cancer.gov/manuals/read.seer.research.nov2017.sas
sas.raw <- read_lines("https://seer.cancer.gov/manuals/read.seer.research.nov2017.sas")
sas.df <- tibble(raw = sas.raw) %>%
## remove first few rows by insisting an # that defines the start index of that field
filter(str_detect(raw, "#")) %>%
## extract out the start, width and column name+description fields
mutate(start = str_replace(str_extract(raw, "# [[:digit:]]{1,3}"), "# ", ""),
width = str_replace(str_extract(raw, "\\$char[[:digit:]]{1,2}"), "\\$char", ""),
col_name = str_extract(raw, "[[:upper:]]+[[:upper:][:digit:][:punct:]]+"),
col_desc = str_trim(str_replace(str_replace(str_extract(raw, "\\/\\*.+\\*\\/"), "\\/\\*", ""), "\\*\\/", "" )) ) %>%
## coerce to integers
mutate_at(vars(start, width), funs(as.integer)) %>%
## calculate the end position
mutate(end = start + width - 1)
column_mapping <- sas.df %>%
select(col_name, col_desc)
#### read the file with the start+end positions----
## CHANGE THIS LINE
file_path = "data/test_COLRECT.txt"
## read the file with the fixed width positions
data.df <- read_fwf(file_path,
fwf_positions(sas.df$start, sas.df$end, sas.df$col_name))
## result is a tibble
Hope that helps!

Answer 2

Fixed width files such as those described by that .sas file are read with read.fwf function in the foreign package. I'm afraid that nicely formatted webpage hosted by Princeton is simply wrong about how to use read.table for this purpose. There are really no separators, just positions. In the case in point you could have used (assuming you have a directory named "yr1973_2015.seer9" in your working directory):
library(utils) #not really needed, just correcting my faulty memory
inputdf <- read.fwf( "yr1973_2015.seer9/OTHER.TXT",
widths= c(1, 9, 19, 20, 23, 24),
col.names = c("pubcsnum", "reg", "mar_stat", "race1v", "nhaide", "sex"))
You would lose most of the information since the lrecl value tells us there are 362 characters per line, but this would be a good test case and you could then switch to the SAScii functions.... and thanks to #AnthonyDamico:
packageDescription("SAScii")
#---------------
Package: SAScii
Type: Package
Title: Import ASCII files directly into R using only a SAS
input script
Version: 1.0
Date: 2012-08-18
Authors#R: person( "Anthony Joseph" , "Damico" , role = c(
"aut" , "cre" ) , email = "ajdamico#gmail.com" )
Description: Using any importation code designed for SAS
users to read ASCII files into sas7bdat files, the
SAScii package parses through the INPUT block of a
(.sas) syntax file to design the parameters needed
for a read.fwf function call. This allows the user
to specify the location of the ASCII (often a .dat)
file and the location of the .sas syntax file, and
then load the data frame directly into R in just one
step.
License: GPL (>= 2)
URL: https://github.com/ajdamico/SAScii
Depends: R (>= 2.14)
LazyLoad: Yes
Packaged: 2012-08-17 08:35:18 UTC; AnthonyD
Author: Anthony Joseph Damico [aut, cre]
Maintainer: Anthony Joseph Damico <ajdamico#gmail.com>
Repository: CRAN
Date/Publication: 2012-08-17 10:55:15
Built: R 3.4.0; ; 2017-04-20 18:55:31 UTC; unix
-- File: /Library/Frameworks/R.framework/Versions/3.4/Resources/library/SAScii/Meta/package.rds
I wasn't abolutely sure that the trailing info would be effectively ignored on those long lines but checked with this slight mod of athe first example on the ?read.fwf page:
> ff <- tempfile()
> cat(file = ff, "12345689", "98765489", sep = "\n")
> read.fwf(ff, widths = c(1,2,3))
V1 V2 V3
1 1 23 456
2 9 87 654
>unlink(ff)
I checked my memory that using Anthony's name as a search term could be helpful and find that his website has been updated. Check out:
http://asdfree.com/surveillance-epidemiology-and-end-results-seer.html