Having screenshots of web pages can be a useful augmentation to a web archive by providing a record of how the website looked in browsers contemporary to the capture. With browser based crawling becoming more common, creating these is, likewise, becoming easier.
Currently, any screenshots stored in WARCs are invisible in replay tools, significantly reducing their usefulness and it is not entirely clear how these screenshots should be stored in WARCs. It is important that a standard way be defined for this type of 'associated resource' so that replay tools can consistently provide consistent access to them.
I've used the term 'associated resource' rather than just 'screenshot' as there are other types of 'data associated' with an URL that we may wish to store. An obvious example might be a video that is embedded in a webpage in a manner that can not be easily crawled or replayed. That video might be extracted through side channels and associated with the original web page via the same mechanism. Replay tools would then, at minimum, provide links to open the video in their 'header'. More advanced replay tools might use advanced rewriting rules to inject it into, e.g. YouTube pages.
There are likely a number of other uses for this type of mechanism. Another example might be to attach some type of annotation/documentation that is attached in this manner to a URL, either via manual curation or an automatic process.
As these are not 'just' metadata that is available for 'big data' style processing, it is important to carefully consider this from the perspective of the replay tools. Notably, how we store this in WARC must facilitate easy discovery using typical web archive indexing (CDX/URL based indexing).
We must also consider that each 'associated resource' will require some amount of metadata to provide suitable context for the replay tool. This goes beyond just type (screenshot, video), but also, e.g. type of capture (is the screenshot based on the same HTTP transaction as the primary resource, was is created in parallel, or was it perhaps created via a replay mechanism), which browser was used etc.
An initial idea might be to store each of these 'associated resources' in a WARC resource record using the URL of the primary resource with a special prefix. E.g. PREFIX:http://example.org. Metadata could then be stored in the record's header using a set of custom fields, a single 'metadata' field containing a JSON 'payload' or some mix of the two.
I'm unsure if this is the best approach, but it serves as a starting point. Over the next few months I'm hoping, with broad input from the people who are building the tools that create and use this data, to write up a proposal for standardizing this that the IIPC could endorse. The document might also include some 'best practice' guidance to how replay tools should handle this data.
If you would like to be a part of that conversation, please get in touch.
Showing posts with label warc. Show all posts
Showing posts with label warc. Show all posts
November 12, 2018
May 17, 2016
WARC MIME Media Type
A curious thing came up during the WARC 1.1 review process. In version 1.0, section 8 talked about what MIME media types should be used when exchanging WARCs over the Internet. During the review process, however, it was pointed out that this is actually outside the scope of the standard. 1.1 consequently drops section 8.
For now we should regard the instructions from 1.0 section 8 as best practices. But it isn't part of any official standard.
That's not to say that it isn't important to have a standard set of MIME types for WARC content. Only that the WARC ISO standard isn't the place for it. This is actually something that IANA is responsible for, with specification work going through the IETF if I'm understanding this correctly.
I'm not at all familiar with this process. But it is clear that if we wish to have this standardized then going through this process is the only option. If anyone can offer further insight into how we could move this forward please get in touch.
For now we should regard the instructions from 1.0 section 8 as best practices. But it isn't part of any official standard.
That's not to say that it isn't important to have a standard set of MIME types for WARC content. Only that the WARC ISO standard isn't the place for it. This is actually something that IANA is responsible for, with specification work going through the IETF if I'm understanding this correctly.
I'm not at all familiar with this process. But it is clear that if we wish to have this standardized then going through this process is the only option. If anyone can offer further insight into how we could move this forward please get in touch.
March 18, 2016
Declaring WARR on "CDX Server" API
Work is currently ongoing to specify a "CDX Server API" for OpenWayback. The name of this API has, however, caused an unfortunate amount of confusion. Despite the name, the data served via this API needn't be in CDX files!
The core purpose of this API is to respond to a query containing an URL and optionally a timestamp or timerange with a set of records that fall within those parameters. This is meant to support two basic functionalities. One, replay of captured web content and, two, discovery of capture web content.
CDXs need not enter into it. It is just that the most common way (by far) to manage such an index is to use sorted CDX files. Thus the unfortunate name. Nothing prevents alternative indexing solutions being used. You could use a relational database, Lucene or whatever tool allows lookups of strings!
So, this API desperately needs a new name. My suggestion is "Web Archive Resource Resolution Service" or WARR Service for short. Yes, I did torture that until it produced a usable acronym.
In my last post I discussed changes to the CDX file format itself. Those changes should facilitate WARR servers running on CDX indexes. But ultimately, the development of the WARR Service API is not directly coupled to those changes. We should focus on developing the WARR Service API with respect to the established use cases.
In truth, the exact scope and nature of this new API remains debated. You can find some lively discussion in this Github issue. More on that topic another day.
The core purpose of this API is to respond to a query containing an URL and optionally a timestamp or timerange with a set of records that fall within those parameters. This is meant to support two basic functionalities. One, replay of captured web content and, two, discovery of capture web content.
CDXs need not enter into it. It is just that the most common way (by far) to manage such an index is to use sorted CDX files. Thus the unfortunate name. Nothing prevents alternative indexing solutions being used. You could use a relational database, Lucene or whatever tool allows lookups of strings!
So, this API desperately needs a new name. My suggestion is "Web Archive Resource Resolution Service" or WARR Service for short. Yes, I did torture that until it produced a usable acronym.
In my last post I discussed changes to the CDX file format itself. Those changes should facilitate WARR servers running on CDX indexes. But ultimately, the development of the WARR Service API is not directly coupled to those changes. We should focus on developing the WARR Service API with respect to the established use cases.
In truth, the exact scope and nature of this new API remains debated. You can find some lively discussion in this Github issue. More on that topic another day.
March 17, 2016
Rewriting the CDX file format
CDX files are used to support URL+timestamp searching of web archives. They've been around for a long time, having first been used to catalog the contents of ARC files. Despite the advent of the WARC file format, they haven't changed much. I think it is past due that we reconsider the format from the ground up.
The current specification lists a large number of possible fields. Many are not used in typical scenarios.
The first field is a canonicalized URL. I.e. an URL with trivial elements (such as protocol) removed so that equivalent URLs end up with the same canonical URL here. This serves as the primary search key.
The only problem with this is that searching for content in all subdomains is not possible without scanning the entire CDX. This is because the subdomain comes before the domain. Instead, we should use a SURT (Sort-friendly URI Reordering Transform) form of the canonical URL instead. SURT URLs turn the domain/sub-domain structure around, making such queries fairly straightforward. There is essentially no downside to doing this and, in fact, a number of CDXs have been built in this manner, regardless of any "formal" standardization (as there isn't really any formal standard).
I suggest that any revised CDX format mandate the use of SURT URLs for the first field. Furthermore, we should utilize the correct SURT format. In most (probably all) current CDXs with SURT URLs, an annoying mistake has been made where the closing comma is missing. An URL that should read:
instead reads:
The protocol prefix has been removed as unnecessary along with the opening ellipse.
The second field should remain the timestamp with whatever precision is available in the ARC/WARC. I.e. an w3c-iso8601 of varying accuracy as per this proposed revision the WARC standard (the revision is extremely likely to be included in WARC 1.1).
The third field would remain the original URL.
The fourth field should be a content digest including the hashing algorithm. Presently, this field is missing the algorithm.
The fifth field would be the WARC record type (or a special value to indicate an ARC response record). This is the most significant change as it allows us to capture additional WARC record types (such as metadata and conversion) while also handling the existing fields in a more targeted manner (e.g. response vs revisit). It might be argued that this should be the second field to facilitate searches of a specific record type. I believe that, probably implemented, this field would allow replay tools to effectively surface any content "related" to the URL currently being viewed, a problem that I know many are trying to tackle.
The next two fields would be the WARC (or ARC) filename (this is supposed to be unique) of the file containing the record and offset at which the record exists within the (W)ARC. This is as it works currently. Some would argue for a more expressive resource locator here, but I believe that is best handled be a separate (W)ARC resolution service. Otherwise you may have to substantially rebuild your CDX index just because you moved your (W)ARCs to a new disk or service.
Lastly, there should be a single line JSON "blob" containing record type relevant additional data. For response records, this would include HTTP status code and content type which I've excluded from the "base" fields in the CDX. This part would be significantly more flexible due to the JSON format, allowing us to include optional data where appropriate etc. The full range of possible values is beyond the scope of this blog post.
There is clearly more work to be done on the JSON aspect, plus some adjustments may be necessary to the base data, but I believe that, at minimum, this is the right direction to head in. Of course, this means we have to rebuild all our CDX files in order to implement this. That's a tall order, but the benefits should be more than enough to justify that one-time cost.
The current specification lists a large number of possible fields. Many are not used in typical scenarios.
The first field is a canonicalized URL. I.e. an URL with trivial elements (such as protocol) removed so that equivalent URLs end up with the same canonical URL here. This serves as the primary search key.
The only problem with this is that searching for content in all subdomains is not possible without scanning the entire CDX. This is because the subdomain comes before the domain. Instead, we should use a SURT (Sort-friendly URI Reordering Transform) form of the canonical URL instead. SURT URLs turn the domain/sub-domain structure around, making such queries fairly straightforward. There is essentially no downside to doing this and, in fact, a number of CDXs have been built in this manner, regardless of any "formal" standardization (as there isn't really any formal standard).
I suggest that any revised CDX format mandate the use of SURT URLs for the first field. Furthermore, we should utilize the correct SURT format. In most (probably all) current CDXs with SURT URLs, an annoying mistake has been made where the closing comma is missing. An URL that should read:
com,example,www,)instead reads:
com,example,www)The protocol prefix has been removed as unnecessary along with the opening ellipse.
The second field should remain the timestamp with whatever precision is available in the ARC/WARC. I.e. an w3c-iso8601 of varying accuracy as per this proposed revision the WARC standard (the revision is extremely likely to be included in WARC 1.1).
The third field would remain the original URL.
The fourth field should be a content digest including the hashing algorithm. Presently, this field is missing the algorithm.
The fifth field would be the WARC record type (or a special value to indicate an ARC response record). This is the most significant change as it allows us to capture additional WARC record types (such as metadata and conversion) while also handling the existing fields in a more targeted manner (e.g. response vs revisit). It might be argued that this should be the second field to facilitate searches of a specific record type. I believe that, probably implemented, this field would allow replay tools to effectively surface any content "related" to the URL currently being viewed, a problem that I know many are trying to tackle.
The next two fields would be the WARC (or ARC) filename (this is supposed to be unique) of the file containing the record and offset at which the record exists within the (W)ARC. This is as it works currently. Some would argue for a more expressive resource locator here, but I believe that is best handled be a separate (W)ARC resolution service. Otherwise you may have to substantially rebuild your CDX index just because you moved your (W)ARCs to a new disk or service.
Lastly, there should be a single line JSON "blob" containing record type relevant additional data. For response records, this would include HTTP status code and content type which I've excluded from the "base" fields in the CDX. This part would be significantly more flexible due to the JSON format, allowing us to include optional data where appropriate etc. The full range of possible values is beyond the scope of this blog post.
There is clearly more work to be done on the JSON aspect, plus some adjustments may be necessary to the base data, but I believe that, at minimum, this is the right direction to head in. Of course, this means we have to rebuild all our CDX files in order to implement this. That's a tall order, but the benefits should be more than enough to justify that one-time cost.
January 28, 2016
To ZIP or not to ZIP, that is the (web archiving) question
Do you use uncompressed (W)ARC files?
It is hard to imagine why you would want to store the material uncompressed. After all, web archives are big. Compression saves space and space is money.
While this seems straightforward it is worth examining some of the assumptions made here and considering what trade-offs we may be making.
Lets start by consider that a lot of the files on the Internet are already compressed. Images, audio and video files as well as almost every file format for "large data" is compressed. Sometimes simply by wrapping everything in a ZIP container (e.g. EPUB). There is very little additional benefit gained from compressing these files again (it may even increase the size very slightly).
For some, highly specific crawls, it is possible that compression will accomplish very little.
But it is also true that compression costs very little. We'll get back to that point in a bit.
For most general crawls, the amount of HTML, CSS, JavaScript and various other highly compressible material will make up a substantial portion of the overall data. Those files may be smaller, but there are a lot more of them. Especially, automatically generated HTML pages and other crawler traps that are impossible to avoid entirely.
In our domain crawls, HTML documents alone typically make up around a quarter of the total data downloaded. Given that we then deduplicate images, videos and other, largely static, file formats, HTML files' share in the overall data needing to be stored is even greater. Typically approaching half!
Given that these text files compress heavily (by 70-80% usually), tremendous storage savings can be realized using compression. In practice, our domain crawls compressed size is usually about 60% of the uncompressed size (after deduplication).
More frequently run crawls (with higher levels of deduplication) will benefit even more. Our weekly crawls' compressed size is usually closer to 35-40% of the uncompressed volume (after deduplication discards about three quarters of the crawled data).
So you can save anywhere from ten to sixty percent of the storage needed, depending on the types of crawling you do. But at what cost?
On the crawler side the limiting factor is usually disk or network access. Memory is also sometimes a bottleneck. CPU cycles are rarely an issue. Thus the additional overhead of compressing a file, as it is written to disk, is trivial.
On the access side, you also largely find that CPU isn't a limiting factor. The bottleneck is disk access. And here compression can actually help! It probably doesn't make much difference when only serving up one small slice of a WARC but when processing entire WARCs it will take less time to lift them off of the slow HDD if the file is smaller. The additional overhead of decompression is insignificant in this scenario except in highly specific circumstances where CPU is very limited (but why would you process entire WARCs in such an environment).
So, you save space (and money!) and performance is barely affected. It seems like there is no good reason to not compress your (W)ARCs.
But, there may just be one, HTTP Range Requests.
To handle an HTTP Range Request a replay tool using compressed (W)ARCs will have to access a WARC record and then decompress the entire payload (or at least from start and as far as needed). If uncompressed, the replay tool could simply locate the start of the record and then skip the required number of bytes.
This only affects large files and is probably most evident when replaying video files. User's may wish to skip ahead etc. and that is implemented via range requests. Imagine the benefit when skipping to the last few minutes of a movie that is 10 GB on disk!
Thus, it seems to me that a hybrid solution may be the best course of action. Compress everything except files whose content type indicates an already compressed format. Configure it to compress when in doubt. It may also be best to compress files under a certain size threshold due to headers being compressed. That would need to be evaluated.
Unfortunately, you can't alternate compressed and uncompressed records in a GZ file such as (W)ARC. But it is fairly simple to configure the crawler to use separate output files for these content types. Most crawls generate more than one (W)ARC anyway.
Not only would this resolve the HTTP Range Request issue, it would also avoid a lot of pointless compression/uncompression work being done.
It is hard to imagine why you would want to store the material uncompressed. After all, web archives are big. Compression saves space and space is money.
While this seems straightforward it is worth examining some of the assumptions made here and considering what trade-offs we may be making.
Lets start by consider that a lot of the files on the Internet are already compressed. Images, audio and video files as well as almost every file format for "large data" is compressed. Sometimes simply by wrapping everything in a ZIP container (e.g. EPUB). There is very little additional benefit gained from compressing these files again (it may even increase the size very slightly).
For some, highly specific crawls, it is possible that compression will accomplish very little.
But it is also true that compression costs very little. We'll get back to that point in a bit.
For most general crawls, the amount of HTML, CSS, JavaScript and various other highly compressible material will make up a substantial portion of the overall data. Those files may be smaller, but there are a lot more of them. Especially, automatically generated HTML pages and other crawler traps that are impossible to avoid entirely.
In our domain crawls, HTML documents alone typically make up around a quarter of the total data downloaded. Given that we then deduplicate images, videos and other, largely static, file formats, HTML files' share in the overall data needing to be stored is even greater. Typically approaching half!
Given that these text files compress heavily (by 70-80% usually), tremendous storage savings can be realized using compression. In practice, our domain crawls compressed size is usually about 60% of the uncompressed size (after deduplication).
More frequently run crawls (with higher levels of deduplication) will benefit even more. Our weekly crawls' compressed size is usually closer to 35-40% of the uncompressed volume (after deduplication discards about three quarters of the crawled data).
So you can save anywhere from ten to sixty percent of the storage needed, depending on the types of crawling you do. But at what cost?
On the crawler side the limiting factor is usually disk or network access. Memory is also sometimes a bottleneck. CPU cycles are rarely an issue. Thus the additional overhead of compressing a file, as it is written to disk, is trivial.
On the access side, you also largely find that CPU isn't a limiting factor. The bottleneck is disk access. And here compression can actually help! It probably doesn't make much difference when only serving up one small slice of a WARC but when processing entire WARCs it will take less time to lift them off of the slow HDD if the file is smaller. The additional overhead of decompression is insignificant in this scenario except in highly specific circumstances where CPU is very limited (but why would you process entire WARCs in such an environment).
So, you save space (and money!) and performance is barely affected. It seems like there is no good reason to not compress your (W)ARCs.
But, there may just be one, HTTP Range Requests.
To handle an HTTP Range Request a replay tool using compressed (W)ARCs will have to access a WARC record and then decompress the entire payload (or at least from start and as far as needed). If uncompressed, the replay tool could simply locate the start of the record and then skip the required number of bytes.
This only affects large files and is probably most evident when replaying video files. User's may wish to skip ahead etc. and that is implemented via range requests. Imagine the benefit when skipping to the last few minutes of a movie that is 10 GB on disk!
Thus, it seems to me that a hybrid solution may be the best course of action. Compress everything except files whose content type indicates an already compressed format. Configure it to compress when in doubt. It may also be best to compress files under a certain size threshold due to headers being compressed. That would need to be evaluated.
Unfortunately, you can't alternate compressed and uncompressed records in a GZ file such as (W)ARC. But it is fairly simple to configure the crawler to use separate output files for these content types. Most crawls generate more than one (W)ARC anyway.
Not only would this resolve the HTTP Range Request issue, it would also avoid a lot of pointless compression/uncompression work being done.
November 12, 2015
Workshop on Missing Warc Features
Yesterday I tweeted:
The idea for this session came to me while reviewing the various issues set forward for the WARC 1.1 review. Several of the issues/proposals were clearly important as they addressed real needs but at the same time they were nowhere near ready for standardization.
It is important that before we enshrine a solution in the standard that we mature it. This may include exploring multiple avenues to resolve the matter. At minimum it requires implementing it in the relevant tools as proof of concept.
The issues in question include:
I'd like to organize a workshop during the 2016 IIPC GA to discuss these issues. For that to become a reality I'm looking for people willing to present on one or more of these topics (or a related one that I missed). It will probably be aimed at the open days so attendance is not strictly limited to IIPC members.
The idea is that we'd have a ~10-20 minute presentation where a particular issues's problems and needs were defined and a solution proposed. It doesn't matter if the solution has been implemented in code. Following each presentation there would then be a discussion on the merits of the proposed solution and possible alternatives.
A minimum of three presentations would be needed to "fill up" the workshop. We should be able to fit in four.
So, that's what I'm looking for, volunteers to present one or more of these issues. Or, a related issue I've missed.
To be clear, while the idea for this workshop comes out of the WARC 1.1 work it is entirely separate from that review. By the time of the GA the WARC 1.1 revision will be all but settled. Consider this, instead, as a first possible step on the WARC 1.2 revision.
Considering a session on crawl artifacts that don't yet fit in WARCs for #iipcGA16. If you're interested in helping organize, let me know.
— Kristinn Sigurðsson (@kristsi) November 11, 2015
Possible topics include screenshots, SSL certs, HTTP 2.X and AJAX interactions. #iipcGA16
— Kristinn Sigurðsson (@kristsi) November 11, 2015
I thought I'd expand a bit on this without a 140 character limit.The idea for this session came to me while reviewing the various issues set forward for the WARC 1.1 review. Several of the issues/proposals were clearly important as they addressed real needs but at the same time they were nowhere near ready for standardization.
It is important that before we enshrine a solution in the standard that we mature it. This may include exploring multiple avenues to resolve the matter. At minimum it requires implementing it in the relevant tools as proof of concept.
The issues in question include:
- Screenshots – Crawling using browsers is becoming more common. A very useful by product of this is a screenshot of the webpage as rendered by the browser. It is, however, not clear how best to store this in a WARC. Especially not in terms of making it easily discoverable to a user of a Wayback like replay. This may also affect other types of related resources and metadata.
See further on WARC specification issue tracker: #13 #27 - HTTP 2 – The new HTTP 2 standard uses binary headers. This seems to breaks one of the expectations of WARC response records containing HTTP responses.
See further on WARC specification issue tracker: #15 - SSL certificates – Store the SSL certificates used during an HTTPS session. #12
- AJAX Interactions – #14
I'd like to organize a workshop during the 2016 IIPC GA to discuss these issues. For that to become a reality I'm looking for people willing to present on one or more of these topics (or a related one that I missed). It will probably be aimed at the open days so attendance is not strictly limited to IIPC members.
The idea is that we'd have a ~10-20 minute presentation where a particular issues's problems and needs were defined and a solution proposed. It doesn't matter if the solution has been implemented in code. Following each presentation there would then be a discussion on the merits of the proposed solution and possible alternatives.
A minimum of three presentations would be needed to "fill up" the workshop. We should be able to fit in four.
So, that's what I'm looking for, volunteers to present one or more of these issues. Or, a related issue I've missed.
To be clear, while the idea for this workshop comes out of the WARC 1.1 work it is entirely separate from that review. By the time of the GA the WARC 1.1 revision will be all but settled. Consider this, instead, as a first possible step on the WARC 1.2 revision.
August 24, 2015
Writing WARCs
We started doing deduplication four years before we started using WARC. As the ARC format had no revisit concept, the only record of the deduplicated items from that era lies in the crawl logs.
When we put our collection online back in 2009 we built our own indexer that consumed these crawl logs so we could include these items. It worked very well at the time.
As our collection grew, we eventually hit scaling issues with the custom indexer. Ultimately, it was abandoned when we upgraded to Wayback 1.8 two years ago. We moved to use a CDX index instead. The only casualty was the early deduplication/revisit records from our ARC days which were no longer included.
So, for the last two years, I've had a task on my todo list. Create a program that consumes all of those crawl logs and spits out WARC files containing the revisit records that are missing from the ARC files.
For a while I toyed with the idea of incorporating this in a wider ARC to WARC migration effort. But it's become clear that such a migration just isn't worthwhile. At least not yet.
Recently I finally made some time to address this. I figured it shouldn't take much time to implement. Basically, the program needs to do two things:
Boom, done. Can knock that out in an hour, easy.
Or, I should be able to.
It turns out that writing WARCs is painfully tedious. The libraries require you to understand the structure of a WARC file very well and do nothing to guide you. There is also no useful documentation on this subject. You best bet is to find code examples, but even those can be hard to find and may not directly address your needs.
I tried both the webarchive-commons library and JWAT. Both were tedious, but JWAT was less so. Both require you to understand exactly what header fields need to be written for each record type. To know that you need to write a warcinfo field first and so on. At least JWAT made it fairly simple to configure the header fields.
Consulting both existing WARC files and the WARC spec, I was able to put all the pieces together in about half a day using JWAT.
And that's when I realized that JWAT does not track the number of bytes written out. That means I can't split the writing up to make "even sized" WARCs like Heritrix does (usually 1 GiB/file).
Darn, I need to rewrite this, after all, using webarchive-commons.
**Sigh**
It seems to me that such an important format should have better library support. A better library would save a lot of time and effort.
By saving effort, we may also wind up saving the format itself. The danger that someone will create a widely used program that writes invalid WARCs is very real. If such invalid WARCs proliferate that can greatly undermine the usefulness of the format.
It is important to make it easy to do the right thing. Right now, even someone, like myself, who is very familiar with WARC needs to double and triple check every step of a very simple WARC writing program. Never mind if you need to do something a little advanced.
Someone with less knowledge and, perhaps, less motivation to "get it right" could all too easily write something "good enough".
It is important to demystify the WARC format. Good library support would be an ideal start.
When we put our collection online back in 2009 we built our own indexer that consumed these crawl logs so we could include these items. It worked very well at the time.
As our collection grew, we eventually hit scaling issues with the custom indexer. Ultimately, it was abandoned when we upgraded to Wayback 1.8 two years ago. We moved to use a CDX index instead. The only casualty was the early deduplication/revisit records from our ARC days which were no longer included.
So, for the last two years, I've had a task on my todo list. Create a program that consumes all of those crawl logs and spits out WARC files containing the revisit records that are missing from the ARC files.
For a while I toyed with the idea of incorporating this in a wider ARC to WARC migration effort. But it's become clear that such a migration just isn't worthwhile. At least not yet.
Recently I finally made some time to address this. I figured it shouldn't take much time to implement. Basically, the program needs to do two things:
- Ingest and parse a crawl.log file from Heritrix, making it possible to iterate over its lines and access specific fields. As the DeDuplicator already does this, it was a minor task to adapt the code.
- For each line that represented a deduplicated URI, write a revisit record in a WARC file.
Boom, done. Can knock that out in an hour, easy.
Or, I should be able to.
It turns out that writing WARCs is painfully tedious. The libraries require you to understand the structure of a WARC file very well and do nothing to guide you. There is also no useful documentation on this subject. You best bet is to find code examples, but even those can be hard to find and may not directly address your needs.
I tried both the webarchive-commons library and JWAT. Both were tedious, but JWAT was less so. Both require you to understand exactly what header fields need to be written for each record type. To know that you need to write a warcinfo field first and so on. At least JWAT made it fairly simple to configure the header fields.
Consulting both existing WARC files and the WARC spec, I was able to put all the pieces together in about half a day using JWAT.
And that's when I realized that JWAT does not track the number of bytes written out. That means I can't split the writing up to make "even sized" WARCs like Heritrix does (usually 1 GiB/file).
Darn, I need to rewrite this, after all, using webarchive-commons.
**Sigh**
It seems to me that such an important format should have better library support. A better library would save a lot of time and effort.
By saving effort, we may also wind up saving the format itself. The danger that someone will create a widely used program that writes invalid WARCs is very real. If such invalid WARCs proliferate that can greatly undermine the usefulness of the format.
It is important to make it easy to do the right thing. Right now, even someone, like myself, who is very familiar with WARC needs to double and triple check every step of a very simple WARC writing program. Never mind if you need to do something a little advanced.
Someone with less knowledge and, perhaps, less motivation to "get it right" could all too easily write something "good enough".
It is important to demystify the WARC format. Good library support would be an ideal start.
August 17, 2015
The WARC Format 1.1
The WARC Format 1.0 is an ISO standard describing the container format that web archives use to store their data. WARCs contain not only the actual file resources (HTML, images, JavaScript etc.) but also request and response headers, metadata about the resources and the overall collection, deduplication records and conversion records.
It's a pretty flexible format. It has served us quite well, but it is not perfect.
While it is an ISO standard, most of it was written by IIPC members. Indeed, it is heavily influenced the ARC format developed by The Internet Archive. So, now that the WARC format is being revisited it is only natural that the IIPC community, again, writes the first draft.
At the IIPC GA this year, in Stanford, there was a workshop where the pain points of the current specification were brought to light. There was a lot of energy in the room and people were excited. But, as everyone got back home a lot of that energy went away.
It is a lot easier to talk about change, than making it happen. Making things more difficult, few of us know much about the standards process. It all felt very inscrutable.
To help with the procedural aspect we came up with an approach that involves using the tools we are familiar with (software development). Consequently, we (and by "we" I mean Andy Jackson of the British Library) set up a GitHub project around the WARC specification.
Any problems with the existing specification could be raised there as "issues" (you'll find all the ones discussed in Stanford on there!). The existing spec could be included as markdown formatted text and any proposed changes could be submitted as "pull requests" acting on the text of the existing spec.
Currently there are two pull requests, each representing a proposed set of changes to address one specific shortcoming of the existing spec.
One of the pull requests comes from yours truly. It address the concerns of "uri agnostic revisit records". This was previously dealt with via an advisory on the subject adopted by the IIPC. This allows us to promote what has been a defacto standard into the actual standard.
The other pull request centers on improving the resolution of timestamps in WARC headers.
Neither pull request has been merged, meaning that both are up for comment and may change or be rejected altogether. There are also many issues that still need to be addressed.
I would like to encourage all interested parties (IIPC members and non-members alike) to take advantage of the GitHub venue if the WARC format is important to you. You can do this by opening issues if you have a problem with the spec that hasn't been brought up. You can do this by commenting on existing issues and pull requests, suggesting solutions or pointing out pitfalls.
And you can do this be suggesting actual edits to the standard via pull requests (let us know if you need help with the technical bits).
Ultimately, the draft thus generated will be passed on to the relevant ISO group for review and approval. This will happen (as I understand it) next year.
So grab the opportunity while it presents itself and have your say on The WARC Format 1.1.
It's a pretty flexible format. It has served us quite well, but it is not perfect.
While it is an ISO standard, most of it was written by IIPC members. Indeed, it is heavily influenced the ARC format developed by The Internet Archive. So, now that the WARC format is being revisited it is only natural that the IIPC community, again, writes the first draft.
At the IIPC GA this year, in Stanford, there was a workshop where the pain points of the current specification were brought to light. There was a lot of energy in the room and people were excited. But, as everyone got back home a lot of that energy went away.
It is a lot easier to talk about change, than making it happen. Making things more difficult, few of us know much about the standards process. It all felt very inscrutable.
To help with the procedural aspect we came up with an approach that involves using the tools we are familiar with (software development). Consequently, we (and by "we" I mean Andy Jackson of the British Library) set up a GitHub project around the WARC specification.
Any problems with the existing specification could be raised there as "issues" (you'll find all the ones discussed in Stanford on there!). The existing spec could be included as markdown formatted text and any proposed changes could be submitted as "pull requests" acting on the text of the existing spec.
Currently there are two pull requests, each representing a proposed set of changes to address one specific shortcoming of the existing spec.
One of the pull requests comes from yours truly. It address the concerns of "uri agnostic revisit records". This was previously dealt with via an advisory on the subject adopted by the IIPC. This allows us to promote what has been a defacto standard into the actual standard.
The other pull request centers on improving the resolution of timestamps in WARC headers.
Neither pull request has been merged, meaning that both are up for comment and may change or be rejected altogether. There are also many issues that still need to be addressed.
I would like to encourage all interested parties (IIPC members and non-members alike) to take advantage of the GitHub venue if the WARC format is important to you. You can do this by opening issues if you have a problem with the spec that hasn't been brought up. You can do this by commenting on existing issues and pull requests, suggesting solutions or pointing out pitfalls.
And you can do this be suggesting actual edits to the standard via pull requests (let us know if you need help with the technical bits).
Ultimately, the draft thus generated will be passed on to the relevant ISO group for review and approval. This will happen (as I understand it) next year.
So grab the opportunity while it presents itself and have your say on The WARC Format 1.1.
August 12, 2014
JWAT WARC reading speed doubled
This is a follow up to my last post about inefficiencies in WARC readers.
As I noted there, reading a WARC file straight (via á GZIP reader, so uncompressing it, but not parsing the content) takes about 10 seconds, whereas iterating over the same file, using the Java tools available (webarchive-commons, JWAT) takes about 40 seconds.
More specifically:
JWAT GzipReader: 10s
webarchive-commons WARCReader: 42s
JWAT WarcReader: 45s
There is a variability there of a few hundred milliseconds between runs so I've rounded to the nearest second. Note also, that the WARC readers were run immediately after the GzipReader and would have benefited from any OS caching.
In my earlier post I speculated that adding a read buffer to JWAT's ByteCountingPushbackInputStream would likely improve the efficiency considerably. I proceeded to test this hypothesis. New run time:
JWAT WarcReader for GZ: 21s
So, JWAT goes from being slightly slower than webarchive-commons, to being twice as fast.
The class still passes all its unit tests and as far as I can tell it is still functionally equivalent.
I've forked the JWAT project to my account on GitHub. You can see the modified file here: ByteCountingPushBackInputStream.java
There are no doubt greater gains to be had, but they'll require a deeper understanding of the code than I possesses at this moment.
The downside to this change is, of course, a slight uptick in the amount of memory used, as a 10K buffer is assigned every time an instance of the ByteCountingPushBackInputStream is created (and that is done more frequently than just to wrap the core gzip read). Still, it seems a small price to pay for the speed increase.
I have no doubt that improvements can also be made in webarchive-commons, but it is far less clear to me where those changes should be made
As I noted there, reading a WARC file straight (via á GZIP reader, so uncompressing it, but not parsing the content) takes about 10 seconds, whereas iterating over the same file, using the Java tools available (webarchive-commons, JWAT) takes about 40 seconds.
More specifically:
JWAT GzipReader: 10s
webarchive-commons WARCReader: 42s
JWAT WarcReader: 45s
There is a variability there of a few hundred milliseconds between runs so I've rounded to the nearest second. Note also, that the WARC readers were run immediately after the GzipReader and would have benefited from any OS caching.
In my earlier post I speculated that adding a read buffer to JWAT's ByteCountingPushbackInputStream would likely improve the efficiency considerably. I proceeded to test this hypothesis. New run time:
JWAT WarcReader for GZ: 21s
So, JWAT goes from being slightly slower than webarchive-commons, to being twice as fast.
The class still passes all its unit tests and as far as I can tell it is still functionally equivalent.
I've forked the JWAT project to my account on GitHub. You can see the modified file here: ByteCountingPushBackInputStream.java
There are no doubt greater gains to be had, but they'll require a deeper understanding of the code than I possesses at this moment.
The downside to this change is, of course, a slight uptick in the amount of memory used, as a 10K buffer is assigned every time an instance of the ByteCountingPushBackInputStream is created (and that is done more frequently than just to wrap the core gzip read). Still, it seems a small price to pay for the speed increase.
I have no doubt that improvements can also be made in webarchive-commons, but it is far less clear to me where those changes should be made
August 6, 2014
Inefficiencies in WARC readers
There is a hard limit to how fast you can read a WARC file, dictated by the storage mechanism. You can't process it faster than you can read it off of an HDD or SSD.
Considering how glacially slow those devices are, compared to everything else you might expect that processing a WARC (to build a CDX for example) would take about as long as it takes to read it from disk. After all, nothing overly special is being done. GZIP decompression is fairly good and parsing the WARC header fields is simple enough.
This is, unfortunately, not the case.
I've suspected as much ever since I generated CDXs for our entire collection some months ago. I ran into this issue again last week as I was trying to extract data from WARC files to build a deduplication index. So I set out to do some testing.
Using a randomly selected 1 GB gzipped WARC file I tried running it through the WarcReader that is in the webarchive-commons library. It averaged about 40 seconds to process this one WARC file.
I then tried reading it using the GZIPInputStream provided by the Java Standard Library and found that it took about 10 seconds to read the file. This, by the way, is consistent with the amount of time needed to read a file of that size from disk.
So, why does it take the webarchive-commons tools four times as long?
I looked under the hood and saw that it was using its own input stream named GZIPMembersInputStream. However, when I tried to use that directly, I also got a run time of about 10 seconds.
Digging a bit further I noticed something interesting. Whereas I was reading the file like so:
GZIPMembersInputStream gin = new GZIPMembersInputStream(
new FileInputStream(file),BUFFER);
byte[] buf = new byte[BUFFER];
int bytesRead = in.read(buf);
while (bytesRead!=-1) {
bytesRead = in.read(buf);
}
I.e. in 10KB blocks, the webarchive-commons tools were (at least mostly) reading it a byte at a time. This can be seen, for example in LaxHttpParser, line 84. Calls to read()accounted for 20 of the 40 second run time.
I changed my tests on the GZIPInputStream and GZIPMembersInputStream to read a byte at a time and, sure enough, it now took about 40 seconds to read the WARC files.
Clearly, there is an overhead to each read action from a compressed stream, unrelated to the size of the read.
Unfortunately, the GZIPMembersInputStream and its parent classes are quite convoluted, so adding an output buffer is complicated. I barely know where to begin.
For the sake of completeness, I also tested the JWAT WARC reader. It has the same issue, although I suspect it may be easier to address there in either the ByteCountingPushBackInputStream or PushbackInputStream as they aren't nearly as convoluted as GZIPMembersInputStream in webarchive-commons.
Bottom line. Byte-at-a-time reading from a compressed stream is very inefficient. A buffer (of about 7-8 KB according to my tests) provides massive improvements. Implementing this in webarchive-commons is going to be tricky, given the legacy code there, but the performance gains would seem to justify the effort.
Considering how glacially slow those devices are, compared to everything else you might expect that processing a WARC (to build a CDX for example) would take about as long as it takes to read it from disk. After all, nothing overly special is being done. GZIP decompression is fairly good and parsing the WARC header fields is simple enough.
This is, unfortunately, not the case.
I've suspected as much ever since I generated CDXs for our entire collection some months ago. I ran into this issue again last week as I was trying to extract data from WARC files to build a deduplication index. So I set out to do some testing.
Using a randomly selected 1 GB gzipped WARC file I tried running it through the WarcReader that is in the webarchive-commons library. It averaged about 40 seconds to process this one WARC file.
I then tried reading it using the GZIPInputStream provided by the Java Standard Library and found that it took about 10 seconds to read the file. This, by the way, is consistent with the amount of time needed to read a file of that size from disk.
So, why does it take the webarchive-commons tools four times as long?
I looked under the hood and saw that it was using its own input stream named GZIPMembersInputStream. However, when I tried to use that directly, I also got a run time of about 10 seconds.
Digging a bit further I noticed something interesting. Whereas I was reading the file like so:
GZIPMembersInputStream gin = new GZIPMembersInputStream(
new FileInputStream(file),BUFFER);
byte[] buf = new byte[BUFFER];
int bytesRead = in.read(buf);
while (bytesRead!=-1) {
bytesRead = in.read(buf);
}
I.e. in 10KB blocks, the webarchive-commons tools were (at least mostly) reading it a byte at a time. This can be seen, for example in LaxHttpParser, line 84. Calls to read()accounted for 20 of the 40 second run time.
I changed my tests on the GZIPInputStream and GZIPMembersInputStream to read a byte at a time and, sure enough, it now took about 40 seconds to read the WARC files.
Clearly, there is an overhead to each read action from a compressed stream, unrelated to the size of the read.
Unfortunately, the GZIPMembersInputStream and its parent classes are quite convoluted, so adding an output buffer is complicated. I barely know where to begin.
For the sake of completeness, I also tested the JWAT WARC reader. It has the same issue, although I suspect it may be easier to address there in either the ByteCountingPushBackInputStream or PushbackInputStream as they aren't nearly as convoluted as GZIPMembersInputStream in webarchive-commons.
Bottom line. Byte-at-a-time reading from a compressed stream is very inefficient. A buffer (of about 7-8 KB according to my tests) provides massive improvements. Implementing this in webarchive-commons is going to be tricky, given the legacy code there, but the performance gains would seem to justify the effort.
Subscribe to:
Posts (Atom)