Media Type Detection from Bytes and File Names

Tells what a file is from its first bytes and from its name, with no system library. Implements the Web Hypertext Application Technology Working Group (WHATWG) Multipurpose Internet Mail Extensions (MIME) Sniffing Standard < https://mimesniff.spec.whatwg.org/> over raw vectors and paths. Reads inside containers to tell 'Office' documents from plain 'ZIP' archives, audio from video in 'MP4' files, and 'Word' from 'Excel' in legacy 'Office' files, and recognises the data files 'R' users hold, such as files saved by 'R', 'Parquet', 'SQLite' and 'Stata' files. Every type has a parent in a hierarchy, and an extension table generated from the Internet Assigned Numbers Authority (IANA) media types registry and the 'Apache' 'httpd' type map answers from the name. One call reports both answers, whether they agree, and whether a file is compressed, encrypted or truncated.

zumime

A file’s name says what someone hopes it is; its bytes say what it is. An upload named invoice.pdf can be a PNG, a .docx is a ZIP archive inside, and a .csv.gz holds a CSV you cannot see until you open the gzip. Before a parser starts, you want to know which of these you have, and whether the file is encrypted or cut short.

zumime answers from both signals in one call and says whether they agree. It needs no other package at run time and no system library, so it gives the same answer on every machine.

• The browser’s algorithm. The byte answer is the WHATWG MIME Sniffing Standard, with its supplied type, no-sniff flag and contexts, checked against Go’s net/http sniff cases.
• Containers and R’s own files. ZIP entry names tell a Word document from a JAR, and ISO base media brands tell an M4A from an MP4, checked against the fixtures of the JavaScript file-type package. RDS, RData, Parquet, SQLite, Stata and SPSS files are known by their headers.
• A hierarchy. Every type has a parent, ending at application/octet-stream, so an unknown ZIP-based format is still a ZIP, and an allowlist of "image" accepts every image.
• What to know before opening. Encryption, completeness, line endings and a script’s interpreter, read from the file’s bytes, and the type inside a compression wrapper, from the name or, with zukomp, from the decompressed bytes.
• The same answer everywhere. No system mime.types, no Windows registry, no libmagic: the extension table is generated from IANA’s registry and Apache httpd’s mime.types.

Installation

Install zumime from CRAN:

install.packages("zumime")

Or install the development version from GitHub:

# install.packages("pak")
pak::pak("pedrobtz/zumime")

Usage

library(zumime)

Take a folder of uploads: a real PDF, a PNG that claims to be a PDF, a compressed CSV and an R data file.

uploads <- tempfile("uploads")
dir.create(uploads)
writeLines("%PDF-1.7", file.path(uploads, "report.pdf"))
writeBin(as.raw(c(0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a)),
  file.path(uploads, "invoice.pdf"))
con <- gzfile(file.path(uploads, "sales.csv.gz"), "w")
writeLines(c("region,total", "north,10"), con)
close(con)
saveRDS(mtcars, file.path(uploads, "cars.rds"))

paths <- list.files(uploads, full.names = TRUE)

mime_type() types each file from its bytes and from its name. agree is FALSE for the PNG named invoice.pdf, and inner names what the gzip holds, guessed from the .csv before the .gz:

files <- mime_type(paths)
files$path <- basename(files$path)
files[c("path", "type", "by_name", "agree", "inner")]
#> <zumime_result: 4 rows>
#>           path              type           by_name agree    inner
#> 1     cars.rds application/x-rds application/x-rds  TRUE     <NA>
#> 2  invoice.pdf         image/png   application/pdf FALSE     <NA>
#> 3   report.pdf   application/pdf   application/pdf  TRUE     <NA>
#> 4 sales.csv.gz  application/gzip  application/gzip  TRUE text/csv

mime_check() asks only the allowlist question, from the bytes alone, since that is the question you ask because names lie. Of the two files named .pdf, one is:

setNames(mime_check(paths, "application/pdf"), basename(paths))
#>     cars.rds  invoice.pdf   report.pdf sales.csv.gz 
#>        FALSE        FALSE         TRUE        FALSE

mime_sniff() runs the standard on bytes in hand, as a browser would on a response. A text/plain label is one some servers send for anything, so the standard checks it and finds binary bytes; under nosniff the label stands:

png <- readBin(file.path(uploads, "invoice.pdf"), "raw", 8)
mime_sniff(png, type = "text/plain")$type
#> [1] "application/octet-stream"
mime_sniff(png, type = "text/plain", no_sniff = TRUE)$type
#> [1] "text/plain"

Types form a hierarchy, so a question about a family covers its members:

docx <- "application/vnd.openxmlformats-officedocument.wordprocessingml.document"
mime_parent(docx)
#> [[1]]
#> [1] "application/zip"          "application/octet-stream"
mime_is(docx, "application/zip")
#> [1] TRUE

Every function but zumime_info() starts with mime_. Every type zumime finds is a canonical name from the registry that mime_types() lists; a supplied type that stands comes back as its lower-cased essence.

Learn more

• Checking an upload before you open it walks through an allowlist, the agree, encrypted and complete columns, and the incremental sniffer on a stream’s first chunk.
• The reference documents every function, grouped by task.

Related work

The byte answer follows the WHATWG MIME Sniffing Standard and is tested against Go’s net/http sniff cases; mime_parse() is tested against the web-platform-tests mimesniff fixtures. Container inspection follows file-type, filetype and mimetype; the extension table is generated from the IANA media types registry and Apache httpd’s mime.types. zumime’s code is written from these references, not copied from them. The test suite uses Go’s sniff cases and file-type’s answers as oracles, credited where they are used.