Workshop on Missing Warc Features

Yesterday I tweeted:

Considering a session on crawl artifacts that don't yet fit in WARCs for #iipcGA16. If you're interested in helping organize, let me know.

— Kristinn Sigurðsson (@kristsi) November 11, 2015

Possible topics include screenshots, SSL certs, HTTP 2.X and AJAX interactions. #iipcGA16

— Kristinn Sigurðsson (@kristsi) November 11, 2015

I thought I'd expand a bit on this without a 140 character limit.

The idea for this session came to me while reviewing the various issues set forward for the WARC 1.1 review. Several of the issues/proposals were clearly important as they addressed real needs but at the same time they were nowhere near ready for standardization.

It is important that before we enshrine a solution in the standard that we mature it. This may include exploring multiple avenues to resolve the matter. At minimum it requires implementing it in the relevant tools as proof of concept.

The issues in question include:

• Screenshots – Crawling using browsers is becoming more common. A very useful by product of this is a screenshot of the webpage as rendered by the browser. It is, however, not clear how best to store this in a WARC. Especially not in terms of making it easily discoverable to a user of a Wayback like replay. This may also affect other types of related resources and metadata.
  See further on WARC specification issue tracker: #13 #27
• HTTP 2 – The new HTTP 2 standard uses binary headers. This seems to breaks one of the expectations of WARC response records containing HTTP responses.
  See further on WARC specification issue tracker: #15
• SSL certificates – Store the SSL certificates used during an HTTPS session. #12
• AJAX Interactions – #14

The above list is unlikely to be exhaustive. It merely enumerates the issues that I'm currently aware of.

I'd like to organize a workshop during the 2016 IIPC GA to discuss these issues. For that to become a reality I'm looking for people willing to present on one or more of these topics (or a related one that I missed). It will probably be aimed at the open days so attendance is not strictly limited to IIPC members.

The idea is that we'd have a ~10-20 minute presentation where a particular issues's problems and needs were defined and a solution proposed. It doesn't matter if the solution has been implemented in code. Following each presentation there would then be a discussion on the merits of the proposed solution and possible alternatives.

A minimum of three presentations would be needed to "fill up" the workshop. We should be able to fit in four.

So, that's what I'm looking for, volunteers to present one or more of these issues. Or, a related issue I've missed.

To be clear, while the idea for this workshop comes out of the WARC 1.1 work it is entirely separate from that review. By the time of the GA the WARC 1.1 revision will be all but settled. Consider this, instead, as a first possible step on the WARC 1.2 revision.